Notes on Films from the Future (2018), Chapters 10–11 (batch FFTF-E)#

Source: working/maynard/book/FFTF-E.txt (PDF pp. 207–249; the PDF page markers match the printed page numbers). I read the whole file. It holds two complete chapters. Chapter 11 ends on p. 249 with a lead-in to Chapter 12 (The Day After Tomorrow, climate change), which is in FFTF-F.

• Chapter 10: “The Man in the White Suit: Living in a Material World” (pp. 207–229)
• Chapter 11: “Inferno: Immoral Logic in an Age of Genetic Manipulation” (pp. 230–249). The running head says “…in the Age of…”

Provenance: Andrew wrote the whole book himself, so everything here is evidence of his thinking. Some material inside the chapters is other people’s words, and I mark it when I use it: - film dialogue (Mrs. Watson, Sienna Brooks, Zobrist) - quotations from Feynman, Ehrlich, the NSABB and Pielke - a colleague’s quip (“a fourteen-letter fast track to funding”) - remarks by scientists and executives in his anecdotes

The plot retellings are his, but these notes focus on the analysis. All quotes below are exact. Straight apostrophes stand in for the book’s curly ones.

Chapter 10: The Man in the White Suit: Living in a Material World#

The argument#

1. Opening case: the 2005 “nano pants” protest (pp. 207–208). Protesters from THONG demonstrated against Eddie Bauer’s Nanotex stain-resistant trousers. The protest backfired: sales went up. Maynard’s own view of the risk is dismissive. The technology “in reality wasn’t that radical,” and “the chance of this technology … leading to substantial harm was pretty negligible” (p. 208). He treats the episode as a “preemptive parody of Transcendence.” What interests him is how it shows “the often-mundane reality of modern nanotechnology.” He also points to “the complex ways in which seemingly beneficial inventions can sometimes threaten the status quo” (p. 208).

2. The film as a prescient parable (pp. 208–212). He calls it “a movie about the pitfalls of blinkered science and socially unaware innovation” (p. 208). It is about polymer chemistry, not nanotechnology, but it “foreshadows the complex social and economic dynamics around nanotechnology, and advanced materials more generally” (p. 208). - Historical setting. The North of England textile mills stood at a “tipping point from using try-it-and-see engineering in manufacturing to relying on predictive science” (p. 209). The Luddite legacy and a strong labour movement were still alive. - Stratton. He is a lone-genius chemist who assumes that “what’s good for him is also good for everyone” (pp. 210–211). The core problem is that “Sidney never thought to ask anyone else what they wanted or needed” (p. 211). - Industry. Mill owners need clothes to wear out, so they plot to bury the invention. - Workers. They resist “not because they are anti-science, but because they are pro- having jobs that pay the bills” (p. 211). - The ending. The fabric proves unstable and falls apart. “Scientific hubris turns to humility and ridicule” (p. 212). Stratton learns nothing and goes back to fixing the science. - What he draws from it. The film addresses “some of the biggest challenges we face in developing socially responsible and responsive technologies” (p. 212). These include “institutional narrow-mindedness, scientific myopia and hubris, ignorance over the broader social implications, human greed and self-interest, and the inevitability of unintended outcomes” (p. 212).

3. “Mastering the Material World”: a short history of nanotechnology (pp. 212–218). - The line of ideas. Feynman’s 1959 “There’s Plenty of Room at the Bottom” talk led to Drexler’s Engines of Creation (1986), and then to the US government taking up the field in the 1990s. - How the NNI came about. The NSF, worried about being sidelined by a well-funded NIH, needed “a big idea, one big enough to sell to Congress” (p. 214). It framed nanotechnology as cross-agency, interdisciplinary and “the next Industrial Revolution.” The result was the National Nanotechnology Initiative (NNI) in 2000. - Disclosure of his own role. He says he “was the first co-chair of the interagency committee within the NNI to examine the environmental and health implications of nanotechnology” (p. 215, fn 149). - Rebranding, not revolution. Mainstream nanotechnology was “a convenient way to repackage existing trends in science and engineering.” Almost anyone could rebrand themselves as a nanotechnologist, “even a toxicologist” (p. 215). He traces the actual roots back a century:

   • Rutherford’s model of the atom (1911) and X-ray diffraction (1912)
   • the electron microscope (1931) and Rosalind Franklin’s work
   • catalysts, and nanopowders made since the 1950s
   • Nanoscale novelty. At the nanoscale, particles can “flip from being extremely inert to being highly reactive” (p. 217). He states this as science and does not draw a risk lesson from it in this chapter.
   • Cynicism, then credit. The NNI “successfully rebranded a trend in science, engineering, and technology that stretched back nearly one hundred years” (p. 217). He quotes a colleague’s cynical definition, “a fourteen-letter fast track to funding” (p. 217). Even so, “brand nanotechnology” has been “phenomenally successful.” It encouraged interdisciplinary research and “stimulated technological advances at the convergence” of atomic-scale, biological and cyber sciences (p. 217).
   • Drexler versus Smalley. The pragmatist Smalley won the argument. So today’s nanoscience “looks far more like the technology in The Man in the White Suit than the nanobots in Transcendence” (p. 218).
   • Benefits. Cancer treatments, batteries, solar cells, electronics and DNA-programmed materials. “Hype aside, we are learning to master the material world” (p. 218).
   • But also consequences. As with Stratton’s wonder material, “there are also unintended consequences that need to be grappled with” (p. 218).

4. “Myopically Benevolent Science” (pp. 218–223). This is the chapter’s analytical core. It opens with a self-critical confession. - His own research. He describes his 2000 cyclone-sampler paper: “I was more interested in the science than its outcomes.” He admits: “I was quite happy to coopt a narrative of social good so that I could continue to satisfy my scientific curiosity” (p. 219). - Curiosity is not the problem. He does not condemn curiosity-driven science (“this isn’t necessarily a bad thing,” p. 219). Physics was his “true love,” and has always been “my deepest inspiration” (pp. 219–220). - Stratton’s condition. Stratton has the “single-mindedness and benevolent myopia I see in many of my peers, and even myself at times” (p. 220). He is ignorant of the people he claims to serve, and “ends up appropriating them as a convenient justification for doing what he wants” (p. 220). He suffers from “social myopia” (p. 220). - Beyond scientists: the Entrepreneurial Ethics class (pp. 220–221). Engineering students who said they wanted to make the world better quickly found ethically “inventive” ways to win a profit game. One formed a blocking consortium; another photocopied the fake money. The exercise shows “how hard it is to translate good intentions into good actions” (p. 221). - The general pattern. He calls it “general benevolence and specific self-interest” (p. 221). Scientists rarely “stop to think about what the ‘public good’ means” (pp. 221–222). “Too many well-meaning scientists presume to know what society needs, without thinking to ask first” (p. 222). - The PCAST anecdote. When advising PCAST on nanotechnology, he argued for public engagement. A prominent scientist replied: “That sounds like a very bad idea” (p. 222). Maynard does not think lay publics should direct research. His point is that “most people have a pretty high level of expertise in what’s important to them and their communities” (p. 222). “Scientists and technologists don’t have a monopoly on expertise and insights” (p. 222). - What goes wrong without it. Staying out of wider conversations “leads to scientific elitism, and ignorance that’s shrouded in arrogance” (p. 223). And “willful ignorance of the broader context that research is conducted within leads to myopia that can ultimately be harmful, despite the best of intentions” (p. 223).

5. “Never Underestimate the Status Quo” (pp. 223–226). - The anecdote. At a National Academy of Sciences workshop on planetary protection, an irate mission scientist told policy experts to “stop telling me how to do my job” (p. 223, paraphrased per fn 153). - Two fatal errors. Maynard is sympathetic, but the scientist “forgot that science never happens in a vacuum, and he deeply underestimated the inertia of the status quo” (p. 224). - The general principle. “The harsh reality is that discovery never happens in isolation. There are always others with a stake in the game” (p. 225). He links this to Hammond in Jurassic Park (Ch. 2) and to Transcendence (Ch. 9). - Resistance is about value, not ignorance. He reframes Luddism: the film’s characters “aren’t Luddites in the pejorative sense, and they are not scientifically illiterate” (p. 225). Everyone “is shrewd enough to see how change supports or threatens what they value, and they fight to protect this value” (p. 225). - Nobody wins. Defending the status quo only robs people of “the ability to adapt to inevitable change in ways that could benefit everyone” (p. 225). - Handling, not the technology, is the problem. There is “nothing inherently bad about Sidney’s technology”; “it’s the way that it’s handled that causes problems” (p. 225). “Not being aware of those hurdles created risks that could otherwise be avoided” (p. 226). - The counterfactual. What if they had talked, found out one another’s concerns and aspirations, and “collectively worked toward a way forward that benefitted everyone?” (p. 226).

6. “It’s Good to Talk” (pp. 226–229). - The AI anecdote. At a meeting about AI, a senior executive admitted that AI has serious risks but refused to engage with affected people, for fear of backlash. Maynard calls this “a perfect example of a ‘let’s not talk’ approach” (p. 226). He gives it a fair hearing: people “do sometimes get the wrong end of the stick” (p. 226). - Why “let’s not talk” fails:

   • Practical: secrets are hard to keep, and “they’ll rapidly fill any information vacuum.” So “staying quiet is an extremely high-risk strategy” (p. 227).
   • Ethical: “keeping quiet may seem expedient, but it’s not always ethical.” A developer “probably shouldn’t have complete autonomy over deciding what you do, or the freedom to ignore those whom your products potentially affect.” There is “a moral imperative to engage broadly when a technology has the potential to impact society significantly” (p. 227).
   • Epistemic: developers “rarely have the fullest possible insight”; “All of us, it has to be said, have a bit of Sidney Stratton in us” (p. 227).
   • Talking has risks too. The executive “was right to be concerned,” but “Talking’s tough. But not talking is potentially more dangerous” (p. 227).
   • He refines “talk” into “listen and engage.” “It’s good to listen to and engage with each other, and explore mutually beneficial ways of developing technologies that benefit both their investors and society more broadly” (p. 227).
   • Mechanisms, and their limits (pp. 228–229):
   • The Danish Consensus Conference is powerful but slow. Most people are too busy surviving, and there is “simply not enough perceived value to them to engage” (p. 228).
   • ECAST.
   • Social media, which can “shut down engagement as well as opening it up” (p. 228).
   • Science museums, TED talks, science cafés, poetry slams and citizen science.
   • The bottleneck is experts. “All that’s lacking is the will and imagination of experts to use these platforms” (p. 229). He sees opportunities for “entrepreneurially- and socially-minded innovators to meet people where they’re at” (p. 229).
   • What it requires. People must “concede that they don’t have the last word on what’s right.” They must be open to “changing their perspectives,” and this “goes for the scientists as well as everyone else” (p. 229). This leads into Inferno.

Concepts and frameworks (with his definitions)#

Topics#

• Risk. He defines risk socially and economically, not toxicologically.
• The nano pants. The physical risk was “pretty negligible” (p. 208). The real risks were disruption and threats to livelihoods.
• Risk to the innovation. The chapter’s key risk claim is that not knowing the social hurdles “created risks that could otherwise be avoided” (p. 226).
• A risk-versus-risk trade-off. He weighs engaging against not engaging, and concludes that the risk of not talking “in the long run … is usually the larger of the two” (p. 227).
• Risk as threat to what people value. Stakeholders act when they see how change “supports or threatens what they value” (p. 225). This matches the value-based view of risk he developed elsewhere (outside these chapters).
• Nanomaterial risk. He mentions his NNI environment and health role (fn 149) and “unintended consequences” (p. 218). He does not discuss nanomaterial toxicity itself in this chapter.
• Responsible and socially responsible innovation. He uses the phrases “socially responsible and responsive technologies” (p. 212), “beneficial and responsible innovation” (p. 226), and “the responsible and beneficial development of new technologies” (p. 229). For him, responsible innovation means:
• asking and listening before building
• engaging stakeholders
• humility and self-reflection
• working collectively toward pathways that benefit “both their investors and society” (p. 227)

It is presented as the alternative to both myopic innovation and defending the status quo. - Permissionless innovation. The term is not used. The idea is addressed in structure: - the planetary-protection scientist’s “let me get on with it” (p. 223) - Stratton’s covert, unsanctioned lab work (p. 210) - the explicit claim that a developer “probably shouldn’t have complete autonomy” when products affect others (p. 227)

His position is that innovation affecting others carries an obligation to engage. That is a constraint on innovator autonomy, but a dialogic one, not a regulatory one. He is also sympathetic to scientists irritated by bureaucracy (p. 224). - Hubris. Explicitly named: Stratton “discovers the full extent of his hubris”; “Scientific hubris turns to humility and ridicule” (p. 212). “Scientific myopia and hubris” is on his list of challenges (p. 212). The remedy he offers is humility and social curiosity, the traits Stratton lacks (p. 222). He treats hubris as a failure of social awareness more than technical overreach, although here the technical overreach (an unstable fabric) is real. - Developers and their mindsets. This is the richest topic in the chapter. - Scientists: driven by curiosity and the thrill of discovery, and they coopt social-good narratives. He includes himself (pp. 219–220). - Engineers and entrepreneurs: good intentions quickly give way to self-interest when there is little self- or social awareness (p. 221). - Senior scientists: hostile to public engagement, as in PCAST (p. 222). - Mission scientists: resent oversight (p. 223). - AI executives: fear backlash (p. 226). - Common thread: a sincere wish to do good combined with a failure to ask people. He is sympathetic, not accusatory (“I have a bit of a soft spot for Sidney Stratton,” p. 220). - Governance and public engagement. - Governance: NNI research policy, including its EHS work; planetary protection rules (1967 treaty; COSPAR), shown as legitimate but frustrating to scientists; industry and labour as informal governors through resistance. - Public engagement: the chapter’s prescriptive centre: Danish Consensus Conferences, ECAST, social media, informal venues. He is candid about limits: people lack time, see little value in engaging, and the technologies are complex. - AI and intelligence. One direct appearance: the AI executive’s “let’s not talk” stance (pp. 226–227). He uses it to show that the Stratton pattern is alive in AI governance. Implicitly, Transcendence-style “existential threat of nanobots” speculation is contrasted with mundane reality (pp. 208, 218). - Technology convergence. “Brand nanotechnology” opened the way to “combining atomic-scale design and engineering with breakthroughs in biological and cyber sciences” (p. 217). Other signs of convergence here: “programming DNA to create new nanomaterials” (p. 218), and the code metaphor (“code of atoms,” “base code”) that links material, biological and digital engineering (pp. 213–214, 218).

Analogies across technologies#

Key quotes (Ch. 10)#

• “a movie about the pitfalls of blinkered science and socially unaware innovation” (p. 208)
• “Sidney never thought to ask anyone else what they wanted or needed.” (p. 211)
• “Scientific hubris turns to humility and ridicule” (p. 212)
• “I was quite happy to coopt a narrative of social good so that I could continue to satisfy my scientific curiosity.” (p. 219)
• “the single-mindedness and benevolent myopia I see in many of my peers, and even myself at times” (p. 220)
• “how hard it is to translate good intentions into good actions” (p. 221)
• “too many well-meaning scientists presume to know what society needs, without thinking to ask first.” (p. 222)
• “scientists and technologists don’t have a monopoly on expertise and insights.” (p. 222)
• “leads to scientific elitism, and ignorance that’s shrouded in arrogance.” (p. 223)
• “He forgot that science never happens in a vacuum, and he deeply underestimated the inertia of the status quo.” (p. 224)
• “The harsh reality is that discovery never happens in isolation.” (p. 225)
• “there is nothing inherently bad about Sidney’s technology” (p. 225)
• “not being aware of those hurdles created risks that could otherwise be avoided.” (p. 226)
• “a perfect example of a ‘let’s not talk’ approach to technological innovation” (p. 226)
• “staying quiet is an extremely high-risk strategy” (p. 227)
• “there’s a moral imperative to engage broadly when a technology has the potential to impact society significantly.” (p. 227)
• “All of us, it has to be said, have a bit of Sidney Stratton in us” (p. 227)
• “Talking’s tough. But not talking is potentially more dangerous.” (p. 227)
• “All that’s lacking is the will and imagination of experts to use these platforms” (p. 229)

Chapter 11: Inferno: Immoral Logic in an Age of Genetic Manipulation#

The argument#

1. Framing (pp. 230–232). - Ehrlich’s prediction. He opens with Paul Ehrlich’s 1969 prediction that by 2000 the UK would be “a small group of impoverished islands” (p. 230, quoting Ehrlich). He notes that Ehrlich and Dan Brown share a “knack for a turn of phrase that transforms hyperbole into an art form” (p. 231). - Zobrist. A “charismatic scientist and entrepreneur” who has “done the math” on overpopulation. He engineers a virus to cull humanity so that it can reset (p. 231). - The film’s use. Maynard calls Inferno a “mindless thriller.” It is still a good starting point for “the darker side of technological innovation—biotechnology in particular—when good intentions lead to seemingly logical, but not necessarily moral, actions” (p. 231). - Three questions:

   • “Where does the moral responsibility lie for the future of humanity,” and should we accept short-term costs to avoid future suffering?
   • What are the dangers of advanced genetic engineering?
   • What about “powerful entrepreneurs who not only have the courage of their convictions, but the means to act on what they believe” (p. 232)?

2. “Weaponizing the Genome” (pp. 233–237). - The 2012 H5N1 papers. Fouchier and Kawaoka published gain-of-function work in Science and Nature that made avian flu transmissible between ferrets through the air. - The rationale, stated fairly. A pandemic combining pathogenicity, lack of immunity and airborne spread is likely eventually (“only a matter of time before some lucky virus hits the jackpot,” p. 234). Scaled to today, the 1918 flu would kill more than 200 million people. So “it makes sense to do everything we can to be prepared for the inevitable” (p. 234). - The concerns:

   • accidental release, given real lab incidents (p. 234)
   • publishing “the recipe,” which “really got people worried” (p. 234)
   • deliberate weaponization (p. 235)
   • The governance story:
   • the 2014 Cambridge Working Group Consensus Statement called for greater responsibility (p. 235)
   • in 2011 the NSABB recommended leaving out methods (p. 235)
   • scientists accused it of “censorship” in “a scientific community that deeply values academic freedom” (p. 235)
   • the NSABB, which “has no real teeth,” gave in, and both papers appeared with the “how-to” included (p. 235)
   • His calibrated view of weaponization risk. “Most biosecurity experts believe that the risks are low.” Weaponization needs expertise and facilities, and cheaper ways to cause terror exist (“a cell phone and home-made explosives, or even a rental truck”). “The economics of weaponized viruses simply don’t work outside of science fiction thrillers … At least, not in a conventional sense” (p. 236).
   • The twist: Zobrist is not a conventional terrorist. His aim is “to be the agent of change.” For a wealthy, morally certain actor, “the economics of Zobrist’s decision actually make some sense, warped as they are” (p. 236).
   • Plausibility through convergence. Designing viruses with chosen properties is “an ability that will only accelerate as we increasingly use cyber-based technologies and artificial-intelligence-based methods in genetic design” (p. 237). So “Because of these converging trends in capabilities,” there is “a kernel of plausibility” that should worry us, “especially in a world where powerful individuals are able to translate their moral certitude into decisive action” (p. 237).

3. “Immoral Logic?” (pp. 237–241). - The lineage he traces:

   • Daniel Quinn’s Ishmael: its logic depends on “‘ends,’ as defined by a single person, justifying extreme ‘means’” (p. 237), with the surgeon-and-cancer analogy (Quinn’s)
   • Ehrlich’s The Population Bomb: sterilization and triage of “hopeless cases”
   • Zobrist’s “switch” question, which Maynard likens to a scaled-up Trolley Problem “that philosophers of artificial intelligence and self-driving cars love to grapple with” (p. 239)
   • The shared premise. In all three, “one person’s prediction of pending death and destruction has greater moral weight than the lives of the people they are willing to sacrifice” (pp. 238–239).
   • He grants partial plausibility. Populations crash. Complex-systems research suggests “the more complex, interdependent, and resource-constrained a system gets, the more vulnerable it can become to catastrophic failure” (p. 239).
   • But the reasoning is “deeply flawed” (p. 240). “Its flaws lie in the same conceits we see in calls for action based on technological prediction”:
   • it assumes the future can be predicted from past exponential trends (a link back to Ch. 9, Transcendence)
   • it “amplifies, rather than moderates, biases in human reasoning and perception”
   • it “creates an artificial certainty around the highly uncertain outcomes of what we do”
   • it “justifies actions that are driven by ideology rather than social responsibility”
   • it assumes “the ‘enlightened,’ whoever they are, have the moral right to act, without consent, on behalf of the ‘unenlightened.’” (all p. 240)
   • Where it ends up. Followed through, it “looks more like religious terrorism, or the warped actions of the Unabomber” (p. 240).
   • Human ingenuity. The problems are real and “should, under no circumstances, be trivialized,” but “we are constantly moving the goalposts of what is possible through human ingenuity” (p. 240). Fertilizers and plant breeding proved Ehrlich wrong. “Perhaps the bigger challenge today” is “overcoming social and ideological barriers to implementing technologies” (p. 240).
   • A 1968 Zobrist. Acting on Ehrlich in 1968 would have been “a morally abhorrent tragedy” (p. 241).
   • He keeps the other side of the ledger. “We cannot afford to dismiss the possibility that inaction in the present may lead to catastrophic failures in the future.” New technologies “only add to the uncertainty of what lies around the corner” (p. 241).

4. “The Honest Broker” (pp. 241–247). - Why scientists hesitate to advocate:

   • they are trained to be sceptical; “many scientists see themselves as seekers of truth, but skeptical of the truth”
   • it feels dishonest to act as if we know answers “when in reality all we know is the limits of our ignorance” (p. 241)
   • they fear losing trust
   • in science culture, making public pronouncements can look like ego (p. 241)
   • The tension. Scientists must balance generating knowledge against “their responsibility as a human being to help people not make a complete and utter mess of their lives” (p. 242).
   • Advocacy-driven fields. Public health, sustainability and climate have strong traditions of advocacy. Maynard grants that some would see an “immorality” in seeing a disaster coming and doing nothing (p. 242).
   • The Yellowstone thought experiment (pp. 242–243).
   • Doing nothing looks wrong. The typical scientific response is “lots of activity, but very little action” (p. 242).
   • The activist view: “it’s better to raise the alarm and be wrong than stay silent and be right” (p. 243). He attributes this reasoning to others.
   • His reply: “Pushing for action based on available evidence always comes with consequences.” Evacuation would bring mass displacement and recession. “The outcomes of the precautionary actions—irrespective of whether the predictions came true or not—would be devastating for some” (p. 243).
   • If the eruption does not happen, acting “will have caused far more harm than inaction would have” (p. 243).
   • Individual bias. “We all like to think we are rational beings—scientists especially—we are not.” So a lone individual demanding costly action has “a reasonably good chance that they’ve missed something” (pp. 243–244).
   • The way out: collective advocacy. “It’s dangerous for an individual to push an agenda for change on their own,” but also “irresponsible to suggest that scientists should be seen and not heard.” Hence “One way forward is in collective advocacy.” “A hundred scientists” beat “one lone genius.” This needs “the humility to accept that their personal ideas may need to be reined in or modified for the common good” (p. 244).
   • Pielke’s four roles (The Honest Broker, 2007), pp. 244–246:
   • Pure Scientist: puts knowledge in a common reservoir; “found more frequently in myth than in reality”
   • Science Arbiter: provides evidence, favours issues that science can resolve
   • Issue Advocate: uses science as a means to an end. Zobrist and Ehrlich fit here
   • Honest Broker: helps decision-makers see how evidence bears on the options, without dictating
   • His own role. He identifies with the Honest Broker: “This is the role I try to carve out for myself in my public-facing work.” It “avoids mistaking personal values for the ‘right’ values” and respects others’ values, and “in most cases there are not bright-line right or wrong answers” (p. 246).
   • Its limit. Where issues carry such moral peril that not advocating becomes “tacit support for not taking action” (p. 246). His examples are climate change, nuclear weapons and vaccine rejection. Even there, “it’s hard to justify one person being the sole arbiter of truth.” Advocacy should come through institutions (the National Academy of Sciences, issue-focused advocacy groups), p. 247.
   • Evidence standard. “Where there is uncertainty in the science (which is pretty much always—that’s the nature of science), the weight of scientific evidence becomes critical.” No cherry-picking: “wishing something is true simply because it supports what you believe doesn’t make it so” (p. 247).

5. “Dictating the Future” (pp. 247–249). - The novel’s ending. In Dan Brown’s novel the virus makes a heritable one in three people sterile, and no one dies. Zobrist emerges “a lone-genius savior” (p. 248). - An uncomfortable concession. He admits the outcome “intrigues me.” It supports the idea that a lone visionary “could probably do it better than a committee.” He is “pretty sure” it “would have had a profound and ultimately positive impact” (p. 248). - Why it still worries him:

   • “the lack of choice that Zobrist’s victims had”; he asks, “did he end up betraying the individuals that make up that society?” (p. 248)
   • it violates basic human rights (p. 249)
   • “we cannot predict the future,” so it was a gamble: “what gave him the right to take this gamble in the first place? Not the people whose futures he was playing with” (p. 249)
   • the core question: “where do they get the right to act unilaterally on issues that ultimately impact us all?” (p. 249)
   • Time pressure is no excuse. Time and necessity do not justify stopping affected people from having a say. So “we need better ways of making collective decisions as a society (as was seen in chapter ten …).” This matters most where “technological innovation is both pushing us toward potentially catastrophic futures and yet is potentially part of the solution” (p. 249).
   • And faster. “We need to get better at making such collective decisions fast, because … it’s that time is short!” (p. 249). This leads into climate change.

Concepts and frameworks (with his definitions)#

Topics#

• Risk. A balanced treatment.
• Pandemic risk: treated as serious and near-inevitable (pp. 233–234).
• Weaponization risk: assessed through actors’ economics and motives, and judged low for conventional actors (p. 236).
• Catastrophic-risk reasoning: criticized when it rests on exponential extrapolation and certainty (p. 240).
• Inaction risk: acknowledged; “we cannot afford to dismiss” it (p. 241).
• Precaution risk: explicitly weighed; precautionary action “would be devastating for some” (p. 243).
• Uncertainty: the constant background. “All we know is the limits of our ignorance” (p. 241); uncertainty is “pretty much always” present (p. 247); “we cannot predict the future” (p. 249).
• Overall: a risk-versus-risk, uncertainty-first stance. He rejects both doing nothing and acting on certainty.
• Responsible and socially responsible innovation. He sets “ideology” against “social responsibility” (p. 240). He speaks of “socially responsible scientists” (p. 244), the “tremendous social responsibility that comes with advocating for change” (p. 243), and “more socially responsible” ways forward that lone individuals miss (p. 244). In this chapter, responsibility means collective, consent-respecting, evidence-weighted action.
• Permissionless innovation. The term is not used. The chapter’s central question, “where do they get the right to act unilaterally” (p. 249), is the strongest form of his objection to permissionless action by powerful individuals. Other points in the same direction:
• the scientific community’s “censorship” objections to the NSABB, and the NSABB giving in (p. 235), show researchers claiming the freedom to publish
• the NSABB’s lack of “real teeth” shows how weak advisory oversight is
• Hubris. The word is not used here. The idea appears as “moral certitude” (p. 237), “artificial certainty” (p. 240), the “enlightened” claiming a right to act (p. 240), and the belief that one’s own prediction has greater moral weight than other people’s lives (pp. 238–239). The antidote is again humility, now collective: ideas “reined in or modified for the common good” (p. 244).
• Developers and their mindsets.
• Zobrist: a scientist-entrepreneur with wealth, expertise and conviction.
• Ehrlich: a scientist-advocate.
• The virologists: preparedness-driven.
• The research community: values academic freedom.
• Scientists generally: truth-seeking sceptics who fear losing trust.
• The scientist-advocate: feels a duty to act.
• His diagnosis: everyone is subject to bias, and scientists are no more rational than anyone else (p. 243).
• Governance and public engagement.
• Biosecurity bodies: NSABB, the Cambridge Working Group, and lab-safety crackdowns (fn 160).
• Pielke’s science–policy roles.
• Institutional advocacy: the National Academy of Sciences.
• Collective societal decision-making, explicitly tied back to Ch. 10’s engagement argument (p. 249). The emphasis moves from engagement for better products (Ch. 10) to engagement as a matter of consent and legitimacy for decisions that affect “us all” (Ch. 11).
• AI and intelligence. Two brief mentions:
• AI-based methods accelerating genetic design, as a factor in convergence (p. 237)
• the Trolley Problem as used by “philosophers of artificial intelligence and self-driving cars” (p. 239)

More important for AI is the structural critique of “calls for action based on technological prediction” and exponential extrapolation (p. 240). He links it explicitly to Ch. 9 (Transcendence, the Singularity), so it applies to superintelligence-style arguments. - Technology convergence. Explicit: “converging trends in capabilities,” with cyber and AI methods in genetic design (p. 237). Convergence raises the plausibility of scenarios that had seemed far-fetched.

Analogies across technologies#

Key quotes (Ch. 11)#

• “when good intentions lead to seemingly logical, but not necessarily moral, actions.” (p. 231)
• “powerful entrepreneurs who not only have the courage of their convictions, but the means to act on what they believe.” (p. 232)
• “The economics of weaponized viruses simply don’t work outside of science fiction thrillers and blockbuster movies.” (p. 236)
• “an ability that will only accelerate as we increasingly use cyber-based technologies and artificial-intelligence-based methods in genetic design.” (p. 237)
• “a world where powerful individuals are able to translate their moral certitude into decisive action.” (p. 237)
• “Its flaws lie in the same conceits we see in calls for action based on technological prediction.” (p. 240)
• “it amplifies, rather than moderates, biases in human reasoning and perception.” (p. 240)
• “creates an artificial certainty around the highly uncertain outcomes of what we do” (p. 240)
• “we are constantly moving the goalposts of what is possible through human ingenuity.” (p. 240)
• “we cannot afford to dismiss the possibility that inaction in the present may lead to catastrophic failures in the future.” (p. 241)
• “many scientists see themselves as seekers of truth, but skeptical of the truth.” (p. 241)
• “Pushing for action based on available evidence always comes with consequences.” (p. 243)
• “it’s dangerous for an individual to push an agenda for change on their own.” (p. 244)
• “One way forward is in collective advocacy.” (p. 244)
• “This is the role I try to carve out for myself in my public-facing work” (p. 246)
• “avoids mistaking personal values for the ‘right’ values” (p. 246)
• “it’s hard to justify one person being the sole arbiter of truth.” (p. 247)
• “where do they get the right to act unilaterally on issues that ultimately impact us all?” (p. 249)
• “we need better ways of making collective decisions as a society” (p. 249)

Threads that run across both chapters#

• One diagnosis, two forms. Stratton (benign, comic) and Zobrist (lethal, tragic) both fail the same way: they decide what is good for others without asking them. Stratton’s is myopic benevolence. Zobrist’s is moral certitude with means. Ch. 11 explicitly points back to Ch. 10 (p. 249).
• Humility and collective processes are the remedy in both: listening and engagement (Ch. 10), collective advocacy, institutions and collective decision-making (Ch. 11).
• Uncertainty is not grounds for either inaction or certainty. Ch. 10: unintended consequences are “inevitable” (p. 212). Ch. 11: prediction is impossible, but inaction can be catastrophic.
• Anecdotes he uses as evidence. He relies on autobiographical and insider accounts:
• NNI, PCAST and National Academies meetings
• an AI meeting
• his teaching
• his own research
• His framing voice. Sympathetic to scientists (“a soft spot,” “quite a bit of sympathy”), including himself, while firm about their blind spots.
• Convergence appears in both chapters (nano–bio–cyber; AI and cyber in genetic design), always as an amplifier of capability.

Digest: what these chapters contribute, and how central each idea is#

Chapters 10 and 11 are the moral centre of the middle of Films from the Future. Together they set out his account of who creates technological risk and why good intentions are not enough. Neither chapter is mainly about the physical hazards of its technology. The nano pants were harmless, and weaponized viruses are, he says, economically implausible for conventional actors. The risks he cares about come from people, their mindsets, and the ways technologies are handled and decided upon. This fits his broader view that risk is social and value-laden. Resistance to Stratton’s fabric is people protecting “what they value” (p. 225), which anticipates his later account of risk as a threat to value.

1. Myopically benevolent science and the failure to ask (central, enduring). This is the most characteristic idea in the batch, and probably one of the most enduring in all his work. Scientists, engineers and entrepreneurs sincerely want to do good, but they presume to know what others need, coopt social-good narratives, and do not listen. What makes the idea his is how he frames it: he does not blame, and he implicates himself (“even myself at times,” p. 220). The same figure appears in Hammond (Ch. 2), Stratton, the PCAST scientist, the planetary-protection scientist and the AI executive, which shows how general he thinks it is. It is his core model of developer psychology, and should be weighted heavily.

2. “It’s good to talk”: engagement as a practical, ethical and epistemic need (central, enduring). Publics hold real expertise “in what’s important to them and their communities” (p. 222). Non-engagement is “an extremely high-risk strategy” (p. 227). There is “a moral imperative to engage” when impacts are significant, and it limits developer autonomy. This is his responsible-innovation stance in its plainest form. The AI-executive anecdote shows he was already applying it to AI in 2018. He is candid that engagement is hard (people lack time, platforms cut both ways) and that experts lack “will and imagination.” That candour is typical of him.

3. The status quo and value-protective resistance (important, recurring). Opposition to innovation is usually rational self-protection, not anti-science ignorance. Ignoring it creates avoidable risk, and pure defence of the status quo stops everyone from adapting. This reading of Luddism, which he returns to elsewhere, is a building block of his technology-transitions thinking.

4. Immoral logic and the conceits of technological prediction (central, and directly relevant to AI). He critiques reasoning from extrapolated catastrophe: exponential trends, amplified bias, “artificial certainty,” ideology over social responsibility, and the “enlightened” acting without consent (p. 240). He explicitly calls these “the same conceits we see in calls for action based on technological prediction.” That makes this the chapters’ most portable tool for looking at AI futures claims, both utopian and catastrophic. It sits alongside a refusal to wave away inaction risk. The balance, uncertain but not complacent, is typical of him.

5. Powerful individuals with moral certitude and the means to act (recurring theme, growing importance). He flags “powerful entrepreneurs” as “a continuing theme in this book” (p. 232). The question “where do they get the right to act unilaterally on issues that ultimately impact us all?” (p. 249) is his most pointed statement against unilateral, permissionless action. It holds even when the outcome might be good: he concedes that book-Zobrist might have succeeded. The concession makes the point about consent and legitimacy sharper.

6. The Honest Broker as his self-described role (central to his practice, qualified). This is a rare explicit statement of how he positions himself: inform decisions, don’t dictate them, respect others’ values, rely on the weight of evidence. He qualifies it straight away: when failing to advocate becomes “tacit support for not taking action,” advocacy should run through institutions, not lone individuals. This explains how he writes publicly: exploratory, rarely prescriptive, wary of certainty.

7. The costs of precaution and the ethics of scientist-advocacy (moderately central). The Yellowstone example shows that he weighs the harms of precautionary action as seriously as the harms of inaction. It is a two-sided view of precaution, not a simple precautionary principle.

8. Convergence (recurring but secondary here). Nano–bio–cyber convergence and AI-accelerated genetic design appear as capability amplifiers. They are noted rather than developed.

9. Brand nanotechnology and hype (a specific but formative case). His insider account of how the NNI rebranded a century-old trend to win funding is grounded in his own NNI environment-and-health role. It shows a practiced scepticism toward technology framing, together with credit for what the framing achieved. The chapter says little about nanomaterial toxicity itself, which is a notable absence given his career.

Overall. These chapters show that his thinking about risk centres on people, not hazards. The recurring danger is well-meaning actors who decide for others: myopically in the comic case, with certainty and power in the tragic one. The remedies he keeps offering are humility, listening, collective processes and consent-based decision-making that must also become faster.