T2. Learning from past technologies: reference cases, precaution, permissionless innovation and responsible innovation#

A thematic synthesis of Andrew Maynard’s thinking, 2014 to September 2026.

Scope and evidence. This file maps one thread: how Maynard uses earlier technologies (chemicals, nanomaterials, GMOs, recombinant DNA, nuclear, the Industrial Revolution, implants, social media and others) as reference cases; what he means by precaution, permissionless innovation and responsible innovation; and what he thinks transfers between technologies, and how. It does not compare his work with anything outside it.

Only his own prose counts as evidence. - Excluded: Modem Futura podcast posts (at the user’s request), AI-generated text (including the o1-pro report in 2025-04-06, the Perplexity and ChatGPT summary links, and the Fable drafts) and guest posts. - Weaker evidence, flagged where used: co-written work, namely 2019-08-13 responsible-innovation (with Elizabeth Garbee), the 2023 Nature Nanotechnology commentary with Sean Dudley (known here only through his posts) and the 2010 Davos proposal with Tim Harper (quoted in 2023-12-22). - Mixed provenance, flagged where used: 2026-07-16 orphan-risks-frontier-ai-maynard (his rewrite of a Fable-assisted paper) and 2026-09-24 being-an-academic-in-an-age-of-ai (his lecture, drafted into prose by Claude and line-edited by him).

Posts are cited by date and slug, with long slugs shortened. Films from the Future (2018) is cited as FFTF with page numbers. Republished text is dated by when it was written: 2018-12-15 is a 2015 Conversation piece, and the 2018 Ex Machina chapter reappears in 2023-04-16 and 2025-03-02. Interpretation is marked “my reading”.

1. The position in brief#

1. He reasons from past technologies constantly, but almost never by literal hazard analogy. What he carries from one technology to another falls into three kinds: - a method of risk reasoning, taken from chemical and nanomaterial risk science; - process lessons about how transformative technologies succeed or fail in society (engagement, trust, transdisciplinarity, who decides), mostly from nanotechnology and GMOs; - recurring human and institutional patterns: well-meaning myopia, hubris, uncertainty that suits incumbents, the inertia of the status quo, and resistance that protects what people value.

His own statement of the move is “an algorithm is not a chemical”, yet once the different mechanisms are set aside, “the analogy between algorithms and chemicals becomes intriguingly compelling” (2019-03-05 should-we-be-treating-algorithms-the-same-way-we-treat-hazardous-chemicals).

2. His oldest lesson from the past is that the past’s frameworks don’t fit. Risk innovation starts from the claim that regulations are “inevitably built around previous technologies”. New technologies get shoehorned into frameworks that are “not remotely the right shape” (2016-01-11 thinking-innovatively-about-the-risks-of-tech-innovation). In the book: “the new wine of technological innovation” squeezed “into the old wineskins of conventional risk thinking” (FFTF p.23). So learning from the past has two halves for him: reuse the hard-won lessons, and do not trust the old tools, categories or track records.

3. Precaution is a qualifier, never a creed. He endorses a proportionate, participatory formulation (UNESCO COMEST) as “a sound philosophy” for catastrophic uncertainty (2020-07-30 life-on-mars). He scales caution to irreversibility. He treats the costs of precautionary action, and the risks of not innovating, as real harms.

4. Permissionless innovation is his most consistent critical target. He refines the idea: it “isn’t necessarily reckless innovation”; the problem is responsibility judged by the innovator alone (FFTF p.162). He restates this in 2025, “more relevant now than it was then”, and adds a reversibility test (2025-03-02 the-lure-of-permissionless-innovation).

5. Responsible, “socially responsible and responsive”, innovation is his lifelong frame. He also treats it as the main thing that did transfer, from nanotechnology. His confidence in it falls visibly after 2024.

6. The emphasis moves from continuity to discontinuity. Past cases are central from 2014 to 2023, peaking in the 2023 AI-governance posts. From 2024 he says more and more strongly that AI breaks analogy. Frontier AI “defies analogy” (2026-01-22 think-you-know-ai-think-again). Judging it “within past frameworks” yields “categorical errors” (2026-09-24, Claude-drafted). He still uses analogies, but as “a mindset” rather than “a playbook” (2024-05-05 blackberry-or-iphone-educational-ai), and often names where they fail.

How firmly he holds these. The method and process lessons are firm and have been restated for more than a decade. The discontinuity claims are recent and growing, strongly worded by 2026 but hedged (“I would argue”). He never works out how they fit with his continued use of history (§8).

2. The repertoire: where his reference cases come from#

His reference cases are largely autobiographical: - 13 years of workplace aerosol research at the UK Health and Safety Executive and NIOSH, and a grandfather who died of coal miner’s pneumoconiosis (FFTF pp.118–120); - collaborating on the 2008 study showing long carbon nanotubes can act like asbestos in mice (2016-02-01 we-dont-talk-much-about-nanotechnology-risks-anymore); - first co-chair of the NNI’s environment and health group, and science adviser to the Project on Emerging Nanotechnologies (FFTF p.215 fn; 2023-10-02 responsible-ai-lessons-from-nanotechnology); - the WEF emerging-technology councils (2023-12-22 un-governing-ai-for-humanity); - Asilomar 2017 (2023-04-04 what-are-the-alternatives-to-calling).

He came to AI through nanotechnology debates. In 2008 “AI wasn’t even on my radar” (FFTF p.168).

Batch readers tagged about half of his pre-2019 posts with past technologies, a third of his 2023 posts, and a tenth in 2025–26 (timeline.md).

One fact needs recording. In his 2015 defence of Musk, Hawking and Gates against ITIF’s “Luddite” label (2018-12-15 if-elon-musk-is-a-luddite-count-me-in), he cites the European Environment Agency’s catalogue of innovations “that damaged lives and environments because early warnings of possible harm were either ignored or overlooked”. It is the only such citation in the corpus. Per the brief, I record the citation and do not discuss the reports.

3. How the transfer works: method, structure and metaphor, rarely literal hazard#

3.1 Transferring a method: the chemical-risk template#

The fullest statement is 2019-03-05, prompted by a self-driving “predictive inequity” paper. He argues that five concepts from chemical risk assessment “are directly applicable” to algorithms: - hazard versus risk; - the events that turn potential harm into actual harm; - consequences; - “how much ‘stuff’” causes harm; - checks and balances on how evidence is used.

He coins “algorithmic exposure” (anyone affected by an algorithm’s decisions is exposed to it) and an “algorithmic exposure-response relationship”. Three features matter.

• The limits are marked. “Of course, an algorithm is not a chemical. It doesn’t have a physical form, nor a chemical identity.” What carries over is the logic of assessment, not the mechanism of harm.
• The lesson cuts both ways. Chemical risk communities “had to learn the hard way”. Their evidential rules “save lives” by preventing “well-meaning but potentially disastrous decisions being made on the flimsiest of evidence”. His examples of fear-driven misreadings are vaccines and GM food. So the chemical lesson is as much against over-reaction as against neglect: “knee-jerk reactions to seemingly-startling results rarely result in socially beneficial outcomes”. He also carries over “there’s no such thing as zero risk” and “there’s always a risk of not developing or using a product”.
• Harms are widened to livelihood and “liberty, dignity, and self-respect”, as his threat-to-value framing predicts.

He closes with what could stand as the motto of this thread: treat algorithms “with the same rigor and respect” as chemicals, “Unless, that is, we’re content to repeat the mistakes of the past as we reinvent the future.”

2023: the template used as a probe. In the addendum to 2023-11-26 he sets out risk as a function of hazard and exposure. He stresses non-linear dose-response, “threshold responses, hormesis, and other low-dose responses”. He maps AI hazard from “disrupting financial services” to “influencing human behavior”, and exposure from agency over critical systems to “hints of ideas encountered over hours of social media use”. He finds “the lack of even the beginnings of a framework”. He had left the paradigm out of the main post deliberately, not wanting to imply “that zero exposure — as in no AI — is a default risk management strategy.” The chemical template is tested here, and a precautionary reading of it is explicitly refused.

2024: a rung on a ladder. Chemicals are no longer the template. Acceptable safety is socially set even for bridges. It “gets infinitely more complex” with “harmful chemicals and biological substances”, and more complex again with emerging technologies and superintelligence (2024-06-20 ilya-sutskevers-safe-superintelligence-rethink).

3.2 Behaviour, not labels#

A second principle from his materials career governs how he thinks lessons should move. In a 2020 talk (published 2022-02-10) he separates “terms of art” from “terms of science”. “Nanomaterial” and “advanced material” are terms of art, and standards built on them are fragile, because “nature doesn’t care what we call a material, it just cares about how it behaves.” Two corollaries follow: - “existing materials that are used in new ways can lead to unexpected risks just as readily as new materials”; - “there is no reason to assume that new materials, by default, present new risks.”

He warns against building new standards on nanotechnology ones “without fully understanding the limitations of these foundations”. He asks: “What evidence is there that nanotechnology-related standards have led to measurable positive outcomes?”

The same principle appears earlier. Quantum dots and Vantablack judged on hazard alone are “taken in isolation” and therefore “misleading” (2015-01-10). Vantablack S-VIS “is not comparable to asbestos”, while long nanotubes may be (2016-02-01).

My reading: this is the underlying rule of his cross-technology reasoning. A lesson transfers when the mechanism or behaviour recurs, not when the category label matches. It explains why he rejects both “nothing really new” and “new therefore dangerous”. On brain–machine interfaces, experts who see “nothing really new here” are “misguided”, because what has changed is “a synergistic scaling of ability, accessibility, and use”, not new science (2020-10-15). The same logic underlies his doubt that the nano-era rule of regulating use, not technology, fits general-purpose AI (§4.2).

3.3 Transferring process and governance lessons#

For AI governance, the lessons he presses hardest are about process.

• GMOs as the failure case. “Monsanto’s ‘it’s complicated, leave it to us’ approach backfired spectacularly” (2023-05-15 erik-schmidt-ai-regulation). GMOs were “a masterclass in how naivety, hubris, greed, and a lack of broad engagement, can create near-insurmountable roadblocks to progress” (2023-10-02). The reading is pro-innovation: the damage he stresses is to the technology’s progress. In 2016 GM foods were his example of a “wicked” problem (2016-03-12). In 2023 he read opposition to GM crops as driven “as much by concerns around corporate power and social equity” as by the technology itself (2023-05-12).
• Nanotechnology as a qualified success. “we did dodge a bullet there. And we did this by engaging early and often”, though “it was touch and go at some points” (2023-05-15). “early investment in responsible nanotech drew heavily on lessons learned from the GMO debacle” (2023-10-02). The success depended on the arts, humanities and social sciences, and on engagement that is “absolutely critical to success” (2023-10-02). NISE Net, which came out of nanotechnology funding, is his model of engagement infrastructure (2023-09-04 why-public-engagement-is-so-important).
• The distilled rule. “the one big lesson from previous advanced technologies is that if we don’t listen and engage early and often, we’ll come to regret it” (2023-05-17 ai-senate-hearing-may-2023).
• Nano governance as a literal model (2025). Against the “Build, Baby, Build” AI Action Plan, early-2000s nanotechnology governance was “a balanced, proactive, and above all collaborative approach that placed wellbeing above dominance”. He adds that “we have lots of theories, studies, and evidence from multiple systems” about what happens “with the guardrails down” (2025-07-23 americas-ai-action-plan).
• A governance genealogy. His case for moving AI from an “ethics framing to one around risk and socially responsible/beneficial innovation” is built as a history: recombinant DNA (bioethics), the Human Genome Project (ELSI), then nanotechnology and synthetic biology (agile, anticipatory, “soft law” governance, driven by the “pacing gap”). AI swung the field back to ethics, partly because its framers “haven’t necessarily been aware of the sophistication of conversations going on elsewhere” (2023-04-04). These past technologies are stages in how governance matured, not analogues for hazard.
• Ignoring the accumulated expertise, a charge he repeats for a decade:
• Schwab’s Fourth Industrial Revolution “reads as if it were written in a vacuum” (2016-01-11 the-fourth-industrial-revolution).
• AI advocates are “blissfully unaware of lessons learned from past technology transitions”, and each wave tends “to metaphorically re-invent the wheel” (2023-04-12 navigating-advanced-technology-transitions).
• The Executive Order shows “a lack of evidence for learning from past technology transitions” (2023-10-30 white-house-goes-all-in-on-responsible-ai).
• AI developers act “as if they’re the first people to notice” long-studied risks (2026-09-15 will-ai-really-kill-us-all).

My reading: the “lesson” is often that an expert community and its toolkit exist and are being ignored. His use of history is in part an argument about whose expertise counts.

3.4 Metaphor, used knowingly, with stated breakpoints#

When he borrows a vivid image from another technology, he usually says whether he means it literally.

• Self-replication. Gray goo is “deeply flawed” physically: “Taken literally, I’m not sure that physical self-replication is our biggest concern with emerging technologies. But as a metaphor, it’s a powerful one—especially when applied to AI.” The metaphor maps onto capability outrunning understanding, and onto “self-replicating” beliefs spread by machines “adroit at manipulating language”. Speculation about out-of-control nanobots “resonate[s] deeply with concerns around value-misalignment in modern-day AI” (2023-04-26).
• The virus. Moltbook agents look life-like, but “a virus doesn’t instinctively know how to use every cognitive trick in the book to make us believe it’s alive”. The same post borrows biosafety for governance: “the digital equivalent of biosafety level 4 containment” (2026-01-31, main text).
• Evolutionary mismatch. Mismatches between evolved risk responses and new technologies “are actually quite commonplace”, with “synthetic chemicals, vaccines” as examples, and humans compensate by thinking. The break: “what if the mismatch impacts the very cognitive abilities we rely on” (2026-01-10 is-ai-a-cognitive-trojan-horse). He cites the chemical case to show where AI departs from it.
• The drug. “if AI was a drug, we’d probably be thinking carefully about how access is overseen” (2026-05-10 do-not-do-this-with-ai). This is hypothetical; the post’s remedy is risk communication.
• BlackBerry and iPhone. “I try and stay clear of analogies to describe the emergence and impact of artificial intelligence.” The calculator, the internet, the printing press and the industrial revolution “all fail to capture the sheer uniqueness and profundity” of AI. He keeps the iPhone “not as a playbook … but as a mindset” (2024-05-05).

3.5 Lineages and literal comparisons#

Some links to the past are lineages, a real descent of ideas or methods rather than a resemblance. - Phrenology through eugenics and fMRI to machine-learning criminality classifiers: “the intent is remarkably similar” (FFTF p.83; 2020-09-26). - Aitken’s Victorian dust counter to modern nanoparticle monitoring. The lesson he quotes from his 2015 paper: “seemingly novel challenges don’t always demand novel solutions, and sometimes, the key to moving forward safely, is to look back at what’s already known” (2024-12-01 geoengineering-aerosol-monitoring-john-aitken). - Asilomar 1975 for gene-based AI, “a landmark in establishing the foundations of responsible and beneficial genetic manipulation” (2025-02-23). - Risk quantification “historically worked out in a landscape comprised of nuclear plants, chemical works and government bureaucracies”. Frontier AI is “an extension of this risk landscape”, and it inherits a narrow definition of risk (2026-07-16, mixed provenance).

Literal comparisons appear in three bounded settings: 1. Inside his own technical field: nanotubes and asbestos (2016-02-01; FFTF p.122 fn); graphene masks judged against past inhalation research (2021-03-28). 2. Governance history: the nano-agency debate (2023-05-17), the nano-era “regulate use” rule (2023-07-12), Asilomar, the Responsible Nano Code (2019-08-13, co-written). 3. Calibration: implants and ICDs to anticipate dependence on brain–machine interfaces (FFTF ch.7); nuclear as a contrast, “a more tangible and immediate risk” than AI (2023-07-25).

My overall reading. Maynard reasons overwhelmingly by structural and methodological comparison. He almost never claims that AI’s harms resemble an earlier technology’s. His claim is that the ways we assess, govern, misjudge and fight over technologies recur. His closest approach to a hazard-level comparison for AI (2019, 2023) is explicitly a transfer of assessment logic, and it ends by finding that logic under-specified for AI. He is unusually consistent in naming where an analogy breaks.

4. What transfers, and what does not#

4.1 What he thinks transfers#

1. Risk reasoning. Hazard versus exposure; consequences and who bears them; weight of evidence; no zero risk; the risk of not innovating (2019-03-05; 2016-03-02; 2023-11-26). Also the plausibility test. Complexity gives bounds that separate “plausible futures from sheer fantasy” (FFTF p.41). The nanotech era supplied the cautionary cases: the nanobot as “one of the zombies of the nanotechnology world” (FFTF p.202), and the harm done “when make-believe is treated as plausible reality” (FFTF p.205). He applies the same test to superintelligence (FFTF p.281; 2024-04-28), and in 2024 recommends moving “quickly on from nanotech risk fantasies to more grounded concerns” (2024-06-23).
2. Engagement as a condition of success. “staying quiet is an extremely high-risk strategy” (FFTF p.227). “progress around gene-based research and technology has been impeded in the past by a lack of effective multistakeholder engagement” (2024-08-25).
3. How institutions behave under uncertainty. - Uncertainty suits those who profit (FFTF p.120). - There is “relatively little correlation between the sophistication of the technology and the safety of the environment in which it’s used” (FFTF p.121). - Risks outlive attention, and the organisations that watched them close (2016-02-01). - Innovators show “naivety and disdain for past risk research” (2021-03-28). - Harm arrives through eager adoption “coupled with an assumption that any potential issues will be handled by someone else” (2023-04-18 universities-need-to-be-investing).
4. Human patterns. - Myopic benevolence: Stratton “never thought to ask anyone else what they wanted or needed” (FFTF p.211). - Hubris and the inertia of the status quo. - Resistance that protects value (FFTF pp.224–226). - Luddites as “skilled adopters and users” of technology (2023-05-12).
5. Complex-systems dynamics. - Normal accidents: the Arkema chemical plant in Hurricane Harvey tests a fictional biotech failure, “if Hammond had read his Perrow” (FFTF pp.42–43). - The grid cascade (2015-01-30 responsible-development-of-new-technologies). - Tipping points (2023-05-04).
6. That old tools are inadequate. This is the founding claim of risk innovation (2016-01-11; FFTF p.23), restated for AI as the refusal of “a conventional mindset” (2023-05-31 existential-risks-of-ai).

4.2 What does not transfer, or is in doubt#

1. Conventional frameworks and one-technology-at-a-time assessment. They “belong to a different world than the one we’re now creating” (FFTF p.23). “understanding the risks and benefits of each emerging technology in turn will not help avoid future catastrophic failure” (2015-01-30).
2. “Regulate what people do with the technology, not the technology itself.” This nano-era mantra “prevailed” (2023-07-12), but AI capabilities “may well erode convenient distinctions between the technology and its uses” (2023-05-17). Predictive policing cannot be addressed by “naively separating the technology from its use” (2023-05-22). A “foundational general purpose technology” may be dangerous in itself, and “my current thinking lies between these two papers” (2023-07-12). This is his clearest questioning of a lesson from his own field.
3. “Past technologies turned out fine.” - “In a complex system, what has occurred in the past may not adequately predict what will happen in the future” (2023-05-04). - “Technological foreshortening” hides “the pain and suffering in the detail”, and “past technological successes are no guarantee of future wins” (2023-10-19 marc-andreessen-ditch-sustainability). - Remember “who gets to write the history of technological successes” (2023-04-18). - Assuming everything will be fine “because it’s always done so in the past” is a recipe for failure (2024-03-31 we-have-a-technology-problem-and). - The historical basis for all of this is rising irreversibility: in the “nuclear and digital age” consequences “propagate through society faster than we can possibly contain them” (FFTF p.167).
4. The old tempo. “This wouldn’t matter so much if this was 30, 50, or even a hundred years ago” (2023-04-12). Responsible-innovation processes are “constrained by human timescales” (2025-04-06, his framing).
5. AI-specific breaks. - AI impairs the faculty we use to compensate for mismatch (2026-01-10). - It is “the first technology of it’s kind” [sic] to “slip unawares into our mind” (2026-05-10). - In the “who we are” domain it is “shaking things up in ways that no other technology has come close to” (2026-05-21 magnifica-humanitas-and-being-human).

The 2023 version is milder: AI is “in many ways, substantively different” (2023-10-30), with stakes “far higher” than nano or GMOs (2023-10-02).

5. Precaution, permissionless innovation and responsible innovation#

5.1 Permissionless innovation#

He frames the topic as “innovation that is conducted in the absence of permission from anyone it might impact” (FFTF p.159). He then engages Adam Thierer’s default-permit doctrine. He grants its carve-out for “clear, catastrophic, immediate, and irreversible harm”, but reads it as ask-forgiveness-not-permission (FFTF p.160).

His distinctive move is to locate the flaw in epistemic isolation rather than recklessness. The Ex Machina inventor builds safeguards: “permissionless innovation isn’t necessarily reckless innovation. Rather, it’s innovation that’s conducted in a way that the person doing it thinks is responsible.” But “a single innovator cannot see the broader context”. The remedy is social: translators, other people, “checks and balances around who gets to do what”, especially where “the genie cannot be put back in the bottle” (FFTF pp.162–166).

He argues from the inside. - His own PhD all-nighter risked “millions of dollars of equipment by bending the rules”: “it’s shocking how quickly I sloughed off any sense of responsibility” (FFTF p.161). - He admits exhilaration at Musk’s Mars plans. - He names a “glitch”: without such drive, innovation “would be much, much slower” (FFTF p.163).

His balance is: “Too much blind speed, and you risk losing your way. But too much caution, and you risk achieving nothing” (FFTF p.163).

How it develops. - 2015 and 2019. The term appears in scare quotes, as innovation driven by power, wealth and a lack of accountability (2018-12-15; 2019-01-16). The 2015 piece also criticises OpenAI’s founding belief that “the answer to technology innovation is… more technology innovation”. - 2025, the reversibility test. Experimenting in a “low-risk linear system” where you can “turn the clock back” is fine. But “Breaking things that aren’t easily fixable — especially in complex systems where the results of experimentation are unpredictable and potentially catastrophic — is probably not such a good idea.” That covers “people, governance, society, and the planet”. He revises his 2018 view of Musk and calls DOGE “a rather naive and uninformed application of permissionless innovation” (2025-03-02). My reading: the same test reconciles his call to give students “permission to play” with AI (2025-03-15). A bounded, recoverable playground is not a complex system. - 2025, the political turn. Responsible AI is “going out of fashion at lightening speed” [sic] as “permissionless innovation” takes its place (2025-02-23). The AI Action Plan’s “‘try-first’ culture” treats responsible innovation as “actively portrayed as a barrier” (2025-07-23). - The book’s structural point. Public indifference hands de facto permission to “those that do care”, who may “do what they like, even if it ends up harming us” (FFTF p.286).

5.2 Precaution#

What he endorses. - Caution scaled to irreversibility. With gene drives “we don’t have the luxury of rebooting”, so we must be “exceptionally cautious” (2016-01-20 three-ways-synthetic-biology). - “better safe than sorry” for a material with unknown inhalation risk used in face masks (2021-03-28). - Precaution is “often misunderstood or misinterpreted”, but COMEST’s formulation (plausible, morally unacceptable harm; a proportionate response; a participatory process) offers, “Precautionary principle politics aside”, “a sound philosophy for addressing complex, uncertain, and potentially catastrophic risks before it’s too late” (2020-07-30). - The transatlantic history of precaution is “fraught with misunderstanding, misinterpretation” and accusations of trade barriers (2025-03-02 fn).

What he counts against it. - “The outcomes of the precautionary actions—irrespective of whether the predictions came true or not—would be devastating for some” (FFTF p.243, the Yellowstone thought experiment). - Acting on confident prediction shares “the same conceits we see in calls for action based on technological prediction” (FFTF p.240). - On blocking geoengineering research on principle: “we don’t have the luxury of sacrificing people’s lives and the environment we live in on the altar of ideology” (FFTF p.267). He takes a “yes and” stance instead (FFTF p.269). - Not innovating is itself a risk (2016-03-02; 2023-05-31).

Applied to AI. - 2023. He declines the pause letter, although he believes in a risk “of potentially existential proportions”: “there are no silver bullets”, and “the biggest risk is not taking action” (2023-04-04). He rejects zero exposure as a default (2023-11-26), then floats “a chance to take a breath (a pause even)” (2023-11-18). - 2024. He backs “pausing — or even rethinking” emotion-exploiting chatbots (2024-10-27). Precaution appears as the “Avoid” quadrant in his transitions model, which assumes innovation can be channelled but not switched off (2024-08-18). - 2025. He plans for an edge case “just on the off chance” (2025-04-06, framing). - 2026. Research, not restriction, for a small-probability, high-consequence cognitive risk (2026-01-10). “We can’t pause it”, flagged as a working assumption that may be flawed (2026-09-24, Claude-drafted).

My reading: precaution for him is a proportionality principle inside a two-sided risk calculus. It is not a rule that shifts the burden of proof. The one clear burden-shift is in materials safety: the graphene masks were “irresponsible innovation on a grand scale — even if the risks turn out to be negligible” (2021-03-28). He does not carry that burden-shift explicitly to AI.

5.3 Responsible and socially responsible innovation#

His terms. - The organising question: how to “reap the benefits of innovation without running into serious problems along the way” (2018-10-12). - “Socially responsible innovation” is illustrated by Stratton, who “never bothered to ask anyone else what they thought of his invention” (FFTF p.21). He pairs “responsible” with “responsive” (FFTF pp.24, 26; “socially responsive and responsible innovation”, 2023-07-12). - The rationale is historical: “we don’t always have a second or third chance to get things right” (FFTF p.22). In practice it is “fiendishly difficult” (FFTF p.22), or “fiendishly hard to operationalize” (2023-05-05 us-white-house-embraces-responsible-innovation). - His working framework is Stilgoe, Owen and Macnaghten’s: anticipation, reflexivity, inclusion, responsiveness (2015-01-30). - The Luddites “were not fighting against technology, but against its socially discriminatory and unjust use” (FFTF p.192). “perhaps we all need something of the spirit of Ned Ludd in us” (2023-05-12).

As a lesson that transferred. For AI, responsible innovation is both the frame he applies and what nanotechnology is said to have proved (2023-10-02; 2025-07-23). The co-written 2019 chapter adapts it to US entrepreneurial culture. It offers codes of conduct (the Responsible Nano Code) and community self-governance (Asilomar), qualified by Sarewitz’s warning that leaving gene-editing risks to experts is “wrong-headed”. It explains why good intentions fail through tight coupling, latency and value mismatch (2019-08-13, co-written).

His confidence over time. - 2015. “much more is needed” (2015-01-30). - 2023. “if we wait until we have a problem with transformative technologies, we’ve probably waited too long” (2023-05-05). The US resists responsible innovation through “an ethos of go fast, break things” (2023-10-30). - 2024. Responsible innovation is among the fields lacking “the breadth of vision and the integration of understanding” (2024-03-31). He sees “disconnects between the talk around responsible innovation, and a reality that sometimes seems childish irresponsibility” (2024-05-21). - 2025. It may “seem futile” under acceleration (2025-04-06). It must be augmented, because it will “run into challenges”: innovation should be channelled, “much as a flood can’t be halted, but it can be directed”, and everyone’s capacity to thrive should be built (2025-08-31 holding-on-to-our-humanity-age-of-ai). He also carries care from synthetic biology (Emma Frow) into a “hard” concept of care for technology governance (2025-03-09), another structural transfer between fields.

6. How the thread developed#

Before 2014 (from his later accounts): black lung and workplace aerosols; the 2008 nanotube–asbestos study; the NNI and PEN; WEF councils; the 2010 Davos proposal, co-written, for identifying health and environmental impacts “before they occur” (2023-12-22); a 2009 civil-society blog series (FFTF p.191); geoengineering ethics (FFTF p.266); the “sophisticated materials” and “don’t define nanomaterials” arguments (restated 2022-02-10).

2014–2017: past cases as home ground. Materials are judged by exposure and life cycle (2015-01-10; 2016-02-01). Nano risk persists after attention fades (2016-02-01). Assessing technologies one at a time fails for converging systems (2015-01-30). Risk innovation is founded on the mismatch between old frameworks and new technologies (2016-01-11). Responsible innovation is defended against the “Luddite” label (2018-12-15, written 2015). Schwab is faulted for ignoring the GMO-to-synthetic-biology governance community (2016-01-11). GM foods are his wicked-problem example (2016-03-12).

2018 to mid-2019: the book and the method transfer. FFTF sets out: - new wine and old wineskins (p.23); - the past as “just a rehearsal for what’s coming down the pike” (p.37); - black lung to nanomaterials (pp.118–122); - permissionless innovation and rising irreversibility (pp.159–167); - gray goo and the harms of speculation (pp.199–206); - The Man in the White Suit as a nano parable, where “much of it applies directly” (p.208); - Occam’s Razor ranking gray goo and superintelligence below harms from materials (p.281).

Alongside the book: brand-nano “fudged the science to sell the idea” (2018-02-21), and algorithms are compared with chemicals (2019-03-05).

Mid-2019 to 2022: consolidation. Precaution via COMEST, with “as we know from experience” that good intentions and “technological tinkering can lead to devastating consequences” (2020-07-30). “nothing really new” is rejected for brain–machine interfaces (2020-10-15). “Disdain for past risk research” (2021-03-28). AI ethics has crowded out AI risk research (2021-08-03). Terms of art versus terms of science (2022-02-10).

2023: peak use of history for AI governance. - The governance genealogy (2023-04-04) and “re-invent the wheel” (2023-04-12). - Self-replication as metaphor (2023-04-26) and broken symmetries (2023-05-04). - Monsanto and nano, with “the one big lesson” (2023-05-15, 2023-05-17). - The doubt about regulating use (2023-07-12) and the comparison with nuclear (2023-07-25). - Historic transitions’ “Failure modes, best practices, and emerging principles” as a research domain (2023-09-25). - The Nano/GMO lessons (2023-10-02) and the hazard–exposure probe (2023-11-26).

The counter-theme is born in the same month as the first: transitions “unlike anything we’ve had to grapple with before” (2023-04-12).

2024: the analogy sceptic. “I try and stay clear of analogies” (2024-05-05). Chemicals become a rung on a ladder (2024-06-20). The challenges are “night and day different from those we’ve faced in the past” (2024-08-11 school-of-advanced-technology-transitions), yet AI must still be read against “thousands of years of an evolving relationship between humanity and technology” (2024-03-31).

2025: permission and politics. Asilomar at 50 (2025-02-23). The reversibility test (2025-03-02). Responsible innovation perhaps futile (2025-04-06). The nano model set against “try-first” (2025-07-23).

2026: beyond analogy, with analogies. Chemical and vaccine mismatch (2026-01-10). “Defies analogy” (2026-01-22). The virus and its breakpoint (2026-01-31). The drug analogy (2026-05-10). The 2018 risk list still stands (2026-09-15). The lineage of risk methods and the GM-food revolt (2026-07-16, mixed). “Categorical errors” (2026-09-24, Claude-drafted).

Constants throughout: - structural rather than literal transfer; - engagement as the master lesson; - the plausibility test; - the critique of permissionless innovation; - a two-sided view of precaution; - the conviction that people who worked through earlier technologies are being ignored.

The main shift: from AI as the latest case in a continuous lineage (2015–2023) to AI as a categorical break that lessons can inform but not capture (2024–2026).

7. Connections to his other threads#

• Risk innovation and risk as a threat to value. This thread is the historical argument for his central idea. Frameworks built for past technologies “belong to a different world”. Threat-to-value lets dignity, identity and belief count alongside the health and environmental harms that chemical frameworks were built for (2016-01-11; FFTF pp.23–24; 2023-05-31). The 2026 orphan-risks paper traces frontier-AI risk selection back to the nuclear and chemical risk tradition (2026-07-16).
• Who decides. Most of his process lessons (Monsanto, PCAST, nano engagement) are arguments against “leave it to the technical experts” (2023-05-15; FFTF pp.222–229).
• Complexity and irreversibility are his grounds for denying that past success predicts future safety: FFTF p.167 and p.43 (normal accidents); 2023-05-04; 2025-04-06.
• Plausibility. Gray goo taught him the test he applies to superintelligence and to “nothing new under the sun” dismissals alike (FFTF p.281; 2026-09-24 fn).
• Innovators and hubris. Stratton, Hammond, Nathan, Monsanto and Musk form one pattern of self-certified responsibility (FFTF pp.36–39, 218–223).
• Justice. First-tier workers, the Luddites and the GM-food revolt turn “lessons” into questions about who bears the costs and whose channels of resistance are open (FFTF pp.118–122, 190–193; 2026-07-16).
• Cognition, manipulation and being human. His discontinuity claim rests almost entirely here. AI is new because it acts on language, trust and formation (2026-01-10; 2026-05-21; 2026-09-24).
• Advanced technology transitions is his institutional answer to the tension: a field that studies “historic advanced technology transitions” (2023-09-25) while insisting that today’s transitions are “not navigable through conventional thinking” (2024-08-11).

8. Tensions, ambiguities and gaps#

1. Continuity or discontinuity? He says each wave “re-invent[s] the wheel” (2023-04-12), and also that the present is “unlike anything”, “night and day different”, beyond analogy, and that past frameworks produce “categorical errors”. He never says how to tell which lessons survive the break. My reading: his unstated resolution is that process and method lessons transfer (engage, weigh evidence, respect irreversibility, distrust self-certified responsibility), while frameworks, categories and track records do not. “Behaviour, not labels” (2022-02-10) is the nearest thing he has to an explicit test. It does not settle AI-specific breaks such as a second-order cognitive mismatch.

2. Analogy while declaring AI beyond analogy. In January 2026 AI “defies analogy” (2026-01-22), while an argument built on chemicals, vaccines and viruses appears a fortnight earlier (2026-01-10). Naming the breakpoints softens the contradiction, but he never addresses it.

3. His own record contests the nano success story. In 2023 and 2025 nano “dodge[d] a bullet” and its governance was a model (2023-05-15; 2025-07-23). But his earlier writing tells a different story: - in 2016, nano risks had dropped off the radar and the boundary organisations had closed (2016-02-01); - in 2018, brand-nano “fudged the science” (2018-02-21); - in 2021, the graphene masks showed “disdain for past risk research” (2021-03-28); - in 2022, he doubted the evidence behind nano standards (2022-02-10).

The lesson he offers AI is drawn from nano’s early engagement phase. His own record of what came later is more mixed, and he does not reconcile the two.

4. Lessons about process, rarely about outcomes. His historical lessons mostly concern whether engagement happened and whether progress was blocked, not whether harm was prevented. The GMO lesson is told as roadblocks to progress (2023-10-02), with no assessment of GM crops’ health or environmental record. He asks of nano standards whether they produced “measurable positive outcomes” (2022-02-10), but never asks the same of engagement.

5. A selective repertoire. The cases he develops come from his career (dust, nanomaterials, nano governance) plus GMOs, recombinant DNA, geoengineering, the Industrial Revolution and nuclear. Classic chemical-hazard histories appear only in passing: asbestos via nanotubes, PCBs in the 2015 citation, endocrine disruptors as an unsolved problem (2019-03-05). Social media, the comparison most often made with AI, is barely developed. It appears as Congress’s governance record (2023-04-10), as evidence that manipulation is already under way (FFTF p.177), and as a human failure mode that AI scales (2024-09-22). A lens built on his work will be richest on materials, nano and biotech governance, and thin on platform harms.

6. Precaution endorsed but not operationalised for AI. He supports COMEST in principle, applies “better safe than sorry” to masks, and floats narrow pauses. He never sets out what a proportionate, participatory precautionary process for AI would look like. His growing assumption that AI is inevitable (“We can’t pause it”, 2026-09-24) sits uneasily with any such process.

7. Evidential conservatism against informed speculation. In 2019 the lesson from chemicals was scepticism of single studies (2019-03-05). By 2025–26, when technology outpaces data, he calls for “informed speculation” and edge-case planning, and warns that exponential change “will feel like an intellectual exercise until it’s too late” (2025-04-06; 2026-09-24 fn). Both positions are defensible, but he does not reconcile them.

8. The use-versus-technology doubt stays open. This is his most significant revision of a lesson from his own field (2023-05-17; 2023-07-12). He returns to it only indirectly: regulate apps designed to exploit cognitive biases (2025-08-31), and disclose how firms select risks (2026-07-16).

9. Permissionless innovation and his own enthusiasms. - He admits exhilaration at NewSpace (FFTF pp.165–166). - He praises the “disruptive digital technologies mindset” Neuralink brings to medicine (2024-07-10 neuralink-update-july-2024). - He calls for students’ “permission to play” (2025-03-15).

Only the reversibility test reconciles these with his critique, and he states it once, in a footnote (2025-03-02).

10. Provenance limits. Several of his fullest statements of nano-to-AI lessons are co-written and outside the corpus: the 2023 commentary with Dudley and the 2019 chapter with Garbee. His strongest discontinuity statement (2026-09-24) was drafted by Claude from his lecture. The lineage of risk methods (2026-07-16) comes from his rewrite of a Fable-assisted draft. The single-authored evidence is strongest for 2015–2023 and thinner for the most recent, most discontinuity-minded phase.

9. The most important sources for this thread#

1. FFTF (2018): pp.21–23, 118–122, 159–167, 199–206, 208–229, 240–243. New wine and old wineskins; black lung to nanomaterials; permissionless innovation and irreversibility; gray goo; myopic benevolence and “it’s good to talk”; the costs of precaution.
2. 2019-03-05 should-we-be-treating-algorithms-the-same-way-we-treat-hazardous-chemicals. Method transfer, with its limits.
3. 2016-01-11 thinking-innovatively-about-the-risks-of-tech-innovation. Frameworks “built around previous technologies”; risk innovation founded.
4. 2022-02-10 are-we-asking-the-right-standards-questions-about-advanced-materials. Behaviour, not labels; caution about building on nano foundations.
5. 2023-04-04 what-are-the-alternatives-to-calling. The governance genealogy; from ethics to risk.
6. 2023-05-15 erik-schmidt-ai-regulation, with 2023-05-17 ai-senate-hearing-may-2023. Monsanto and nano; “the one big lesson”; the nano-agency history.
7. 2023-10-02 responsible-ai-lessons-from-nanotechnology. Nano and GMO lessons for AI.
8. 2023-07-12 regulating-frontier-ai-models. Doubting “regulate use, not technology”.
9. 2023-11-26 everything-youve-heard-about-ai-risk-is-wrong (with its addendum). The hazard–exposure probe; no zero-exposure default.
10. 2023-04-26 in-bill-joys-why-the-future-doesnt, with 2023-05-04 tipping-points-and-broken-symmetries. Metaphor, not literal; the past does not predict the future.
11. 2025-03-02 the-lure-of-permissionless-innovation. The reversibility test; the footnote on precaution.
12. 2025-07-23 americas-ai-action-plan. The nano model against “try-first”.
13. 2020-07-30 life-on-mars. Precaution (COMEST) alongside IRGC and responsible innovation.
14. 2024-05-05 blackberry-or-iphone-educational-ai, with 2026-01-22 think-you-know-ai-think-again. Analogy as mindset; “defies analogy”.
15. 2026-01-10 is-ai-a-cognitive-trojan-horse. Chemicals and vaccines as mismatch, and the second-order break.