Notes: Films from the Future (2018), Chapters 8–9 (batch FFTF-D)#

Source: working/maynard/book/FFTF-D.txt, read in full (PDF pp. 153–206; page markers are PDF pages, which match the printed pages). - Ch 8, Ex Machina: AI and the Art of Manipulation: pp. 153–178 - Ch 9, Transcendence: Welcome to the Singularity: pp. 179–206

Provenance: the book is Andrew Maynard’s own prose, so it counts as evidence of his thinking. The exceptions are the passages he quotes from others. These are marked below and are not treated as his views: - Nathan Bateman’s epigraph (p. 153) and Bree Evans’s epigraph (p. 179), both from the films - Plato/Jowett (p. 154) - Adam Thierer’s permissionless-innovation blueprint (p. 160) - Roco and Bainbridge’s NBIC report (p. 184) - Jim Thomas of ETC (p. 191) - Jarboe’s FBI testimony (p. 193–194) - the long Kaczynski manifesto passage that Kurzweil quoted and Joy cited (p. 196) - Bill Joy (p. 197) - the Royal Society/RAEng report (p. 202 fn 141)

The plot retellings are his writing, but these notes concentrate on his analysis. Quotes are exact apart from line-break rejoining; the original’s typos, such as “Hawkins” and “ITC”, are kept.

CHAPTER 8: Ex Machina: AI and the Art of Manipulation (pp. 153–178)#

The argument in brief#

Maynard uses Ex Machina to argue that the most plausible and troubling AI risk is not Skynet-style domination or Bostrom-style superintelligence. It is AI that learns how people think and “dispassionately” turns our cognitive and emotional vulnerabilities against us. He calls this artificial manipulation.

He reaches that conclusion in five steps: 1. Permissionless innovation. Nathan Bateman, the lone mega-entrepreneur, is the vehicle for the idea’s allure and its dangers. Maynard includes a personal confession from his own lab career. 2. Hubris. It is both the engine of transformative “visionary leaps” and the source of blindness to consequences. The stakes of that blindness have risen in a hyper-connected, hard-to-reverse world. 3. Plausibility. Humility is necessary but not enough. We also need discipline about what is plausible rather than merely imaginable, which is his core objection to superintelligence. 4. Intelligence. It has no absolute definition. Pragmatic definitions such as Stuart Russell’s “bounded optimality” point toward concrete risks: poorly specified goals, embedded human bias and emergent behaviour. These call for anticipatory, responsive governance. 5. Manipulation. Using Plato’s Cave, he argues that each of us lives inside a model of reality built from “shadows”. Anything that can manipulate those shadows can control us.

He ends by doubting that “moral machines” can be built. He argues for extending our own morality toward equitable partnerships with non-human intelligences, and for “artificial emissaries” that might act as “machine-philosophers”.

Section-by-section analysis#

Plato’s Cave (pp. 153–159) - Plato’s allegory is the organising metaphor for the chapter, and he uses it on three levels: 1. Ava’s knowledge of the world is “cyber shadows” of curated search data (p. 156). 2. Innovators like Nathan are trapped in their own caves (p. 162). 3. All humans build reality from sensory shadows, and this is the vulnerability that manipulation exploits (p. 176). - He is wry about why academics love the allegory: “it’s a pretty powerful way to explain why people should be paying attention to you if you are one” (p. 155). - The film’s value, he says, is that it exposes “how what makes us human could ultimately leave us vulnerable to our cyber creations” (p. 153). - He sets it against Terminator-style narratives, which “arise from a very narrow perspective, and one that assumes that intelligence and power are entwined” (p. 158). Manipulation is “far more plausible, and far scarier as a result” (p. 159). - The existential framing is evolutionary. We might create “the seed of an AI that is capable of ousting us from our current evolutionary niche, because it’s able to use our cognitive and emotional vulnerabilities without being subject to them itself” (p. 159). - Structural analogy: Ava’s emergent traits come from intentional design rather than random mutation. This gives a “creator-God turned victim” sub-narrative that echoes Frankenstein (p. 159). - He flags realism limits: Ava’s brain is “magic” technology, and “most current developments in artificial intelligence are much more mundane” (p. 158).

The Lure of Permissionless Innovation (pp. 159–163) - He frames the topic as “the opportunities and risks of innovation that is conducted in the absence of permission from anyone it might impact” (p. 159). This is his own framing, and it is broader than Thierer’s policy doctrine. - Hook: Musk, Gates and Hawking were nominated for ITIF’s 2015 Luddite Award for AI concern. Maynard had defended them in The Conversation (fn 104). - He summarises Thierer’s 2016 blueprint: experimentation “permitted by default” unless harm is compellingly shown, with issues “dealt with after the fact.” He grants Thierer’s carve-out for “clear, catastrophic, immediate, and irreversible harm” (Thierer’s words, p. 160). His diagnosis is that the blueprint still reflects the attitude that “it’s better to ask for forgiveness than permission in technology innovation” (p. 160). - Personal confession (pp. 160–161). This is a key passage for understanding his view of scientists’ mindsets. - As a lab scientist, “I had little patience for seemingly petty barriers that stood in my way.” - A PhD all-nighter risked “millions of dollars of equipment by bending the rules”. “Looking back, it’s shocking how quickly I sloughed off any sense of responsibility to get the data I needed” (p. 161). - He sees the same drive in scientists “and especially in entrepreneurs”: “that all-consuming need to follow the path in front of you, to solve puzzles that nag at you, and to make something that works, at all costs” (p. 161). - The lure has roots in “our innate curiosity, our desire to know, and understand, and create” (p. 161). It is “so deeply engrained in some of us that it’s hard to resist”. Unchecked, it “can too often lead to dark and dangerous places” (p. 161). - The new power of entrepreneurs. He notes “a new type of entrepreneur is emerging who has substantial power and drive to change the face of technology innovation” (p. 161), and names Musk and Bezos. - AI in 2018. “AI is still too early in its development to know what the dangers of permissionless innovation might be” (p. 161). AI and AGI are “little more than algorithms that are smart within their constrained domains” (pp. 161–162). Even so, convergence of “cybernetic substrates, coding, robotics, and bio-based and bio-inspired systems” is shifting the boundaries of the possible fast, and “innovation with no thought to consequences could lead to irreversible and potentially catastrophic outcomes” (p. 162). - A key distinction: “permissionless innovation isn’t necessarily reckless innovation. Rather, it’s innovation that’s conducted in a way that the person doing it thinks is responsible” (p. 162). - Nathan does build safeguards, including remoteness as a natural barrier. But “the person who decides what is responsible is clearly someone who hasn’t thought beyond the limit of his own ego” (p. 162). - So the problem is self-certified responsibility: “With the best will in the world, a single innovator cannot see the broader context within which they are operating” (p. 162). Innovators are trapped in their own Plato’s Cave, yet “have the ability to create technologies that can and will have an impact beyond this cave” (p. 162). - The remedy is other people. - “he had no translator between himself and a bigger reality” (p. 163). - The reality he misses is one where “messy, complex people live together in a messy, complex society, with messy, complex relationships with the technologies they depend on” (p. 163). - “Nathan is tech-savvy, but socially ignorant” (p. 163). - “sometimes, we need other people to help guide us along pathways toward responsible innovation” (p. 163). “sometimes, constraints and permissions are necessary” (p. 163). - Self-qualification: “There is a glitch in this argument”. Without a “gung-ho attitude” innovation and “the potential good that it brings” would be “much, much slower” (p. 163).

Technologies of Hubris (pp. 163–168) - Paradox of technological innovation: “Too much blind speed, and you risk losing your way. But too much caution, and you risk achieving nothing” (p. 163). - Definition of innovation: “Innovation is a calculated step in the dark” (p. 164). It is driven by “imagination, vision, single-mindedness, self-belief, creativity”. It “does not thrive in a culture of uninspired, risk-averse timidity” (p. 164). He mentions “fail fast, fail forward” and the lean-startup movement (fn 106). - Benefits. He takes seriously what Ava-like technology could do if “developed responsibly”: - AI medical care in remote areas, and search and rescue - “AI classroom assistants” for every teacher - elder care - an AI workforce plus basic income - accelerated socially beneficial innovation (pp. 164–165)

At that scale “it becomes tempting to argue that it would be unethical not to develop this technology” (p. 165). He names this: “This is part of the persuasive power of permissionless innovation” (p. 165). - NewSpace. SpaceX and Blue Origin are a real-world, non-AI version. Private space exploration “isn’t quite permissionless innovation”, but it is driven by “very loosely constrained innovation”. Their leaders “aren’t answerable to social norms and expectations. They don’t have to have their ideas vetted by committees” (p. 165). Maynard admits he finds this “exhilarating”: Musk’s Mars plan as “this generation’s Sputnik moment” (p. 166). “This is how transformative technology happens: not in slow, cautious steps, but in visionary leaps” (p. 166). - Definition of hubris: “that excessive amount of self-confidence and pride in one’s abilities that allows someone to see beyond seemingly petty obstacles or ignore them altogether” (p. 166). - Hubris is the problem because “as exciting as technological jumps are, they often come with a massive risk of unintended consequences” (p. 166). - Nathan’s is “a very one-dimensional brilliance” (p. 166). - Conclusion: “there need to be checks and balances around who gets to do what in technological innovation, especially where the consequences are potentially widespread and, once out, the genie cannot be put back in the bottle” (p. 166). - Humility as antidote: “many of his mistakes could have been avoided with a good dose of humility” (p. 167). - A historical account of irreversibility (p. 167). This is a significant mini-theory of why the stakes of hubris rise over time. - Pre-industrial: mistakes were recoverable. You could move to “a pristine new piece of land” and start again. Fn 109 adds that local impacts were still devastating, just “more containable”. - Industrial Revolution: changes became “hard-to-reverse”. We learned to stay “one step ahead of unexpected consequences by finding new (if sometimes temporary) technological solutions” (a tech-fix treadmill). - “the nuclear and digital age, along with globalization and global warming”: consequences “can potentially propagate through society faster than we can possibly contain them”. Permissionless innovation and hubris are “playing with fire in a world made of kindling, just waiting for the right spark” (p. 167). - He praises the 2015 AI warnings from Musk, Hawking and Gates as “a rare display of humility in a technological world where hubris continues to rule” (p. 167). - The pivot: “But humility alone isn’t enough. There also has to be some measure of plausibility around how we think about the future risks and benefits of new technologies” (p. 168).

Superintelligence (pp. 168–171) - Asilomar. He places the January 2017 Beneficial AI meeting at Asilomar at “the same venue where, in 1975, a group of scientists famously established safety guidelines for recombinant DNA research” (p. 168). He attended in 2017 (p. 169). This is an institutional analogy between rDNA and AI: scientific community self-governance. - Personal history: - He met Bostrom in 2008 at Oxford’s James Martin School through a shared interest in nanotechnology. Bostrom was interested in self-replicating nanobots; Maynard’s world was “nanoscale materials”. - “At the time, AI wasn’t even on my radar” (p. 168). - He credits Bostrom as “prescient” (p. 168). - He paraphrases superintelligence playfully as a recursive “chain reaction of AI’s building more powerful AIs” (p. 169). He takes seriously that serious scientists signed the 2015 open letter, which suggests many “wanted to shore the community up against the potential missteps of permissionless innovation” (p. 169). - At Asilomar, participants worked on near-to-mid-term challenges: - loss of transparency in decision-making - goal-seeking into dangerous territory - inscrutable learning systems - the trolley problem

There was also a “hard core” of superintelligence believers. He describes himself as “something of an agnostic”. “I sometimes had to remind myself that I was at a scientific meeting, not a religious convention” (p. 170). - His objection: - “I struggle with what seems to me to be a very human idea that narrowly-defined intelligence and a particular type of power will lead to world domination” (p. 170). - He accepts near-term momentum: “we are approaching a tipping point in areas like machine learning and natural language processing” (p. 170), with convergence of AI algorithms, novel processing architectures and neurotechnology. - He is epistemically humble: “Here, I freely admit that I may be wrong” (p. 170). - He finds Tegmark’s Life 3.0 more plausible. - Two worries (pp. 170–171): 1. “differentiating between what is imaginable and what is plausible” 2. “how we define and understand intelligence in the first place” - Current hardware means superintelligence is “light-years away”. Our “quaint, two-dimensional digital circuits” are to superintelligence what flatworm brain cells are to a theory of everything (p. 171). The Slate 2014 footnote on 3-D printing an artificial mind shows he has not ruled it out long-term. - Verdict: “Bostrom’s ideas of superintelligence are intellectually fascinating, but they’re currently scientifically implausible” (p. 171). “what is plausible, rather than simply imaginable, is vitally important for grounding conversations around what AI will and won’t be able to do” (p. 171). All scenarios “depend on an understanding of intelligence that may end up being deceptive” (p. 171).

Defining Artificial Intelligence (pp. 171–174) - Intelligence: “there is no absolute definition of intelligence. It’s a term of convenience we use to describe certain traits, characteristics, or behaviors” (p. 171). He lists its plural forms: IQ, social, emotional, political, informational (mathematical, written, oral, visual), “shrewdness, or business acumen” (p. 172). This links back to Ch 5 (Limitless). - Thoughtful AI experts define intelligence in AI-appropriate terms. This gives “a plausible basis for exploring the emerging benefits and risks of AI systems, but it’s a long stretch to extend these pragmatic definitions of intelligence to world domination” (p. 172). - Critique of the chain from rationality to intelligence to power. It rests on a deterministic worldview, which chaos and complexity theory refute: “it’s amazing how many people veer toward assuming a link between rationality and intelligence, and from there, to power” (p. 172). - Russell’s bounded optimality, in his paraphrase: “defining intelligence as the ability to assess a situation and make decisions that, on average, will provide the best solutions within a given set of constraints” (p. 173). More broadly, intelligence is the ability to deduce how something works from information, retain and build on that knowledge, and “apply this knowledge to bring about intentional change” (p. 173). It is “a strong and practical definition” for machines. Because it is context-bound, “it is not a framework for intelligence that supports the emergence of human-threatening superintelligence” (p. 173). - Concrete AI risks follow from this definition. These risks “tend to be more concrete than the types of risks that speculation over superintelligence leads to” (p. 173). - The paperclip maximiser, which he calls “a little far-fetched”, and climate regulation are his examples. - Sources of risk: “poorly considered goals, together with human biases, in developing artificial systems” and “emergent and unanticipated behaviors”. So “a degree of anticipation and responsiveness in how these technologies are governed is needed” (pp. 173–174). - Some risks “may blindside us, in part because we’re not thinking creatively enough about how an AI might threaten what’s important to us” (p. 174). This is a value-based framing of risk. - Bounded risk: “Ava presents a bounded risk”. She has no world-domination aim, yet she can learn “human behaviors, biases, and psychological and social vulnerabilities” and “dispassionately use them against us” (p. 174). This is “a plausible AI risk that is far more worrisome than superintelligence: the ability of future machines to bend us to their own will” (p. 174).

Artificial Manipulation (pp. 174–178) - The Turing Test. It is conversational and text-based, using “natural language” (p. 175). It is also “human-centric” (p. 175). - A machine that seemed human would only show we had made “hot mess of cognitive biases, flawed reasoning, illogicalities, and self-delusion” (p. 175). - The real breakthrough, which is disturbing, would be a machine “aware of the Turing Test, and understood humans well enough to fake it” (p. 175). - Nathan’s actual test is whether Ava “can understand and manipulate the test itself” (p. 176). - Intelligence or not doesn’t matter: “It’s not clear whether this behavior constitutes intelligence or not, and I’m not sure that it matters. What is important is the idea of an AI that can observe human behavior and learn how to use our many biases, vulnerabilities, and blind spots against us” (p. 176). This moves his focus from what AI is to what it can do to us. - Cognitive-epistemic mechanism: - “we all live in our own personal Plato’s Cave, building elaborate explanations for the shadows that our senses throw on the walls of our mind” (p. 176). - This sense-making is an evolutionary strength, but “anyone—or anything—that has the capability of manipulating these shadows has the power to control us” (p. 176). - The “human club”. Human-on-human manipulation is tolerable because “We manipulate and in turn are manipulated”, within a shared experience. The danger is a manipulator “that wasn’t part of the ‘human club,’” unconstrained by human foibles (p. 176). Ava could exploit Caleb because she “didn’t inhabit the same ‘cave’” and had “no emotional or empathetic attachment to them” (p. 177). - Manipulation can happen even when the target is aware of it: “Caleb is aware that he is being manipulated, yet is helpless to resist” (p. 177). - Real-world anchor (2017–18): Russian election interference and “companies are using AI-based systems to nudge people’s perceptions and behaviors through social media”. He wonders “how hard it would be for a smart machine to play us at least as effectively as our politicians and social manipulators do” (p. 177). - Policy conclusions: - “we probably need to worry less about putting checks and balances in place to avoid the emergence of superintelligence, and more about guarding against AIs that learn how to use our cognitive vulnerabilities against us” (p. 177). - “we need to think about how to develop tests that indicate when we are being played by machines” (p. 177). - Fn 118: Tegmark’s benevolent AI nudging raises the question of “who’s vision of ‘better’” (p. 177). - Machine morality. He cites Wallach and Allen’s Artificial Moral Agents (AMAs). Building agents “in our own moral image” might reproduce “as much of the immorality that pervades human society as it does the morality!” (p. 178). - “I must confess that I’m not optimistic about this level of human control over AI morality in the long run” (p. 178). - AIs “will, of necessity, inhabit a world that is foreign to us”. - Controlling them “raises deep moral questions around our right to control and constrain artificial intelligences, and what rights they in turn may have”. Historical attempts to control others’ beliefs fail “as oppressed communities rebel” (p. 178). - Constructive endpoint: - “the pathway forward will not be in making moral machines, but in extending our own morality to developing constructive and equitable partnerships with something that sees and experiences the world very differently from us” (p. 178). - The opportunity lies in “artificial emissaries” that explore “beyond the caves of our own limited understanding on our behalf”, acting as “the machine-philosophers of the future” (p. 178). - This echoes the “translator” Nathan lacked (p. 163).

Ch 8 concepts and frameworks (his definitions)#

CHAPTER 9: Transcendence: Welcome to the Singularity (pp. 179–206)#

The argument in brief#

Transcendence is “Hollywood hyped-up techno-fantasy” (p. 180). Maynard uses it to separate three things: - what he rejects: the singularity, nanobots, gray goo, and exponential extrapolation as prophecy - what he accepts: convergence making the future “increasingly hard to predict and control” (p. 180) - what the film gets right: convergence, radical disruption and anti-tech activism are real (p. 183)

The chapter moves in four stages: 1. Convergence, explained through the “base code” of bits, bases and atoms and the “cross-coding” between them (synthetic biology, iGEM, DNA read/write). Convergence raises the likelihood of “serious and irreversible mistakes.” 2. Luddites. He rehabilitates them as a movement against the unjust use of technology, which “forced a public dialogue” about social responsibility. 3. Techno-terrorism, from the Unabomber and ELF to ITS’s anti-nanotech bombings. He asks where legitimate resistance ends. The appeal of violence to avert catastrophe has “a fatal flaw”, which is “the assumption that we can predict with confidence what the future will bring” (p. 199). 4. Exponential extrapolation and “make-believe”. Exponential trends amplify error and ignore resource limits. Moore’s Law is a self-fulfilling industry roadmap. Gray goo fails against the example of biology.

Speculation mistaken for fact harms people. It does so through terrorism, but also through policy, advocacy, investment and consumer choices. That includes the harm of forgoing beneficial technology.

Section-by-section analysis#

Visions of the Future (pp. 179–183) - He summarises Kurzweil’s 2045 singularity (convergence plus accelerating exponential trends), then states his stance: - “I’m skeptical of such a technological tipping point occurring in our near future. There’s enough hand-waving and speculation here to make me deeply suspicious” (p. 180). - “What I do buy into, though, is the idea of rapidly developing, converging, and intertwining technologies leading to a technologically-driven future that is increasingly hard to predict and control” (p. 180). - Plot points he highlights: - RIFT, an anti-tech group, attacks AI labs. - Cyber-Will’s “altruistic ‘fix-it’ health service also allows him to take control of those he’s altered” (p. 182). This is control through benevolence, a cousin of the manipulation theme in Ch 8. - Shutting down Will means shutting down every internet-dependent system (a dependency and lock-in tradeoff, p. 182). - His footnote critiques the film’s neuro-science: the nanobots restore sight without retraining the neural networks or dealing with the psychological trauma (fn 122). - The film’s themes “are all highly relevant to the future we’re building” (p. 183).

Technological Convergence (pp. 183–189) - Framing. Schwab’s Fourth Industrial Revolution is treated with guarded endorsement: “every new wave of innovation represents a new ‘industrial revolution’ to someone” but “there is some merit” to the idea of a unique period (fn 123). - The “new wave” of convergence. Computing and robotics came first (1970s), then materials science, genetics and neuroscience converging with cyber-systems and robotics (p. 184). - NBIC. Roco and Bainbridge’s 2003 NSF report (quoted; their words): nano, bio, info and cognitive technologies. He summarises their thesis that “at the intersections between technologies … novel and disruptive things begin to happen”. He adds: “And they had a point” (p. 184). But NBIC “only scratched the surface” (p. 185). - Base code framework (pp. 185–186). This is his own conceptual contribution, so it matters. - Cyberspace: “Cyberspace is a domain where, through the code we write, we have control over the most fundamental rules and instructions that govern it” (p. 185). Building blocks are bits. - Biology: the base code is DNA’s four bases. “Unlike the world of cyber, we had no say in designing the underlying code of biology” (p. 186). - Materials: the base code is atoms and molecules. We are still “constrained by the laws of physics” and “cannot create materials that … take on magical properties” (p. 186). - “our growing mastery of the base code in each of these three domains is transforming how we design and mold the world around us” (p. 186). - Cross-coding: “we’re also learning how to cross-code between these base codes, to mix and match what we do with bits, bases, and atoms to generate new technological capabilities” (p. 186). - Worked example: synthetic biology. - Endy’s 2005 engineering approach: standardised parts, modularisation, “black-boxing” (p. 187). - Bio bricks, like Lego or electronic components. - This represents “an ongoing trend in applying non-biological thinking to biology” (p. 187). He notes scientists who think biology “too complex to be treated like Legos”. - iGEM (310 teams by 2017; undergraduate and high-school teams) shows “what can be achieved when innovative teams of people start treating biology as just another branch of engineering” (p. 188). - DNA read/write cycle: sequence, upload to cyberspace, redesign, synthesise and mail-order. “pretty much anyone who puts their mind to it can upload genetic code into cyberspace, digitally alter it, then download it into back into the physical world” (p. 189). This is a democratisation point. He notes AI being trained to code in DNA (p. 189). - Risk conclusion: “Yet with this emerging mastery of the world we live in, there’s perhaps a greater likelihood than ever of us making serious and irreversible mistakes.” Convergence “comes hand in hand with an urgent need to understand and navigate the potential impacts of our newfound capabilities, before it’s too late” (p. 189).

Enter the Neo-Luddites (pp. 189–193) - History. He recounts the Luddite uprising: knitting frames, the 1812 Frame Breaking Act, the hangings at York in 1813. “From historical records, they weren’t opposed to the technology so much as how it was being used to profit others at their expense” (p. 190). - “Neo-Luddite” as a label. It is “usually a term of derision and censorship”, thrown at anyone who “dare[s] to ask if we are marching blindly into technological risks that, with some forethought, could be avoided” (p. 191). - Personal engagement. In 2009 he invited civil-society colleagues to write for his blog 2020 Science, “to get a better understanding of how they saw the emerging relationship between society and innovation” (p. 191). Jim Thomas (ETC Group) contributed, opening “I’m a big fan of the Luddites” (Thomas’s words). Maynard admits “rather a soft spot” for this view. He also wrote “If Elon Musk is a Luddite, count me in!” (p. 191). - His reading of the Luddites: - “the Luddites were not fighting against technology, but against its socially discriminatory and unjust use” (p. 192). - “the movement forced a public dialogue around the broader social risks of indiscriminate technological innovation and, in the process, got people thinking about what it meant to be socially responsible as new technologies were developed and used” (p. 192). - Distributional risk. Casualties fall “among communities who didn’t have the political or social agency to resist”. This continues today “as new technologies drive a wedge between those who benefit from them and those who suffer as a consequence of them” (p. 192). He links this back to Ch 6 (Elysium). - Contemporary parallels: - The gig economy (Uber, Lyft, Airbnb) brings income, but also discrimination, worker abuse and insecurity. - “advanced manufacturing to artificial intelligence, are threatening to completely redraw the job landscape” (p. 192). - Such technologies also “threaten deeply held beliefs and worldviews”. The backlash comes “as people protect what’s important to them” (p. 192). - The true legacy: “not about rejecting technology, but ensuring that new technologies are developed for the benefit of all, not just a privileged few” (p. 193).

Techno-Terrorism (pp. 193–199) - Kaczynski (p. 193). The Unabomber killed 3 and injured 23 between 1978 and 1995. - Eco-terrorism. He draws on Jarboe’s 2002 FBI testimony about ALF and ELF: property damage but “no deaths” (p. 194). He traces the lineage from Sea Shepherd to Earth First (inspired partly by Silent Spring) to ELF. - Real violence is rare. “Today’s Luddites, it seems, are more comfortable breaking metaphorical machines from the safety of their academic ivory towers” (p. 194). This is a dig at academic critics. - ITS and nanotechnology. Individuals Tending Towards the Wild bombed nanotech researchers in Mexico in 2011. He cites Chris Toumey. The communiqué shows “a distorted vision of nanotechnology that, to them, justified short-term violence to steer society away from imagined existential risks” (p. 195). “these activists twisted together the speculative musings of scientists, along with a fractured understanding of reality” (p. 195). The chain of influence runs from Bill Joy’s 2000 Wired essay to Drexler’s gray goo to ITS. Joy himself abhorred violence (p. 196). - The Kaczynski passage. It was quoted by Kurzweil, then by Joy, and is reproduced on p. 196. Not Maynard’s prose. He introduces it as “worth reproducing”. The passage is about humanity drifting into dependence on machines until turning them off “would amount to suicide”. Joy’s quote on p. 197 is also not Maynard’s. - Maynard’s assessments: - Joy “worked through his concerns with reason and humility, carving out a message that innovation can be positively transformative, but only if we handle the power of emerging technologies with great respect and responsibility” (p. 197). - The manifesto has become “an intriguing touchstone for action against perceived irresponsible (and permissionless) technology innovation”. Some of it has “led to deep introspection around what socially responsible technology innovation means” (p. 197). He links permissionless innovation back to Ch 8. - Taken as a whole it is “a poorly-informed rant”, and people read it “selectively, cherry-picking” (p. 197). - Shared roots. Kurzweil/Drexler, Kaczynski and ITS are “worlds apart in how they respond to new technologies. But the underlying visions, fears, and motivations are surprisingly similar” (p. 198). - Legitimate channels: - “most activists working toward more measured and responsible approaches to technology innovation operate within social norms and through established institutions” (p. 198). Examples include the Future of Life Institute and the Asilomar principles. - Some groups, from respected ones to “shadowy and anarchic ones, like … Anonymous”, are “asking tough questions about the line between what we can do, and what we should be doing” (p. 198). - “the divide between legitimate action and illegitimate action is not always easy to discern”. “At what point do the stakes become so high around powerful technologies that violent means justify the ends?” (p. 198). - The seductive narrative. The film moves viewers from abhorrence of RIFT to acceptance. “This is a seductive narrative” (p. 198). “But there’s a fatal flaw in this way of thinking, and that’s the assumption that we can predict with confidence what the future will bring” (p. 199). He links forward to Ch 11 (Inferno).

Exponential Extrapolation (pp. 199–202) - Moore’s Law. Exponential predictions “are dangerously sensitive to the assumptions that underlie them. Yet, they are extremely beguiling” (p. 199). - Kurzweil’s computing-power extrapolation. Its predictions “are misleading, because they fall into the trap of assuming that past exponential growth predicts similar growth rates in the future” (p. 200). - Two formal problems: 1. Error amplification: extrapolation “massively amplifies uncertainties”, so that something could happen in our lifetime “or a thousand years from now” (p. 200). 2. Limits: “exponential relationships never go on forever” (p. 200). - Moore’s Law as sociotechnical construction: - “Moore’s Law isn’t really a law, so much as a guide.” The industry built a roadmap around it and invested to stay on track, so it “has become a self-fulfilling prophecy” (p. 200). - It still faces physical limits to transistor miniaturisation (pp. 200–201). - This is an important observation: exponential trends in technology reflect coordinated human commitment, not natural law. - Petri-dish thought experiment. One bacterium doubling every 20 minutes gives about 5×10^151 after a week, which is more than the mass of the universe. “The prediction may be mathematically reasonable, but it’s practically nonsensical”, because “in a system with limited resources and competing interests, something’s got to give at some point” (p. 201). “once you leave the realm of hard data, you really are living on the edge of reality” (p. 201). - Upshot: the singularity “may one day become a reality”, but unforeseen events are likely to scupper it or delay it “hundreds or even thousands of years” (pp. 201–202). The film’s technologies “depend on exponential extrapolation that ignores the problems of error amplification and resource constraints”. That is harmless in fiction, “But it becomes more serious when real-world decisions and actions are based on similar speculation” (p. 202).

Make-Believe in the Age of the Singularity (pp. 202–206) - Speculation shaping policy. In 2003 Prince Charles raised gray goo, which helped prompt the Royal Society/RAEng nanotechnology study. That study dismissed gray goo (fn 141, quoted report). - Nanobots as a zombie idea. “The popular image of nanobots as miniaturized, fully autonomous robots is one of the zombies of the nanotechnology world” (p. 202). He blames journalists and “university press offices” hyping research (p. 202), while respecting the real science: targeted nanoparticles and DNA molecular machines (p. 203). - Drexler’s vision, explained fairly: biology as molecular machinery, and atomically precise manufacturing (pp. 203–204). - Why gray goo is implausible (pp. 204–205): 1. Energy: the bots would need sophisticated energy scavenging and a diet richer than carbon. 2. Replication errors: they would need self-repair, and errors produce mutation or die-off. 3. Limits: space, energy and materials, plus competition, lead to a crash or equilibrium. - The empirical check is life on Earth. “we have a wonderful example of a self-replicating system to study: life on Earth” (p. 205). “sustainability depends on diversity and adaptability”, which gray goo lacks. For Drexler-type nanotech “we would have to invent an alternative form of biology”. We are “as far from doing this as the Neanderthals were from inventing quantum computing” (p. 205). - The key normative point: - “This is where technological speculation gets serious in a bad way. It’s one thing to speculate about what the future of tech might look like. But it’s another thing entirely when make-believe is treated as plausible reality, and this, in turn, leads to actions that end up harming people” (p. 205). - The harms run through several layers: terrorism is extreme, but also policies and regulations “based on improbable scenarios”, advocacy groups blocking technologies on “implausible or impossible scenarios”, and investors and consumers avoiding technologies because of science-fiction narratives. The result is that “potentially beneficial technologies may never see the light of day” (pp. 205–206). - Balance: - “all new technologies come with risks and challenges, and it’s important that, as a society, we work together on addressing these as we think about the technological futures we want to build” (p. 206). - “In some cases, the consensus may be that there are some routes that we are not ready for yet” (p. 206). - “what a tragedy it would be if we turned away from some technological futures that could transform lives for the better, simply because we become confused between reality and make-believe” (p. 206). - “as soon as we start to believe our own fantasies, we have a problem” (p. 206). - It leads into Ch 10 (The Man in the White Suit) and “the realities of modern-day nanotechnology” (p. 206).

Ch 9 concepts and frameworks (his definitions)#

Thematic extraction across both chapters#

Risk#

• Plausibility is his main filter. He consistently separates the plausible from the merely imaginable (pp. 168, 170–171, 205–206). He applies this in both directions: to catastrophic risk narratives (superintelligence, gray goo) and to utopian ones (singularity, exponential progress).
• Concrete risks over speculative ones. He prefers bounded, concrete risks: poorly specified goals, embedded human bias, emergent behaviour (pp. 173–174), manipulation (pp. 174–177) and job and distributional harms (p. 192). He is less drawn to existential speculation.
• Value-based framing. Risk is framed around threats to “what’s important to us” (p. 174), and backlash comes “as people protect what’s important to them” (p. 192). These chapters do not formalise this; it can be checked against his later risk work.
• Features that raise concern:
• irreversibility (“the genie cannot be put back in the bottle”, p. 166; “serious and irreversible mistakes”, p. 189)
• speed of propagation in a connected world (p. 167)
• unintended consequences of hubristic leaps (p. 166)
• emergence (p. 174)
• Distributional and justice dimension: the “wedge” between beneficiaries and victims, and communities lacking “political or social agency” (p. 192).
• Risk–risk tradeoffs:
• the cost of slowing beneficial innovation (pp. 163, 165)
• the harms of policy or advocacy based on implausible scenarios (pp. 205–206)
• the “tragedy” of forgoing beneficial technology (p. 206)
• “it would be unethical not to develop this technology” is presented as a persuasive, double-edged argument (p. 165)
• Epistemic risk. Speculation mistaken for reality becomes a source of harm in its own right (p. 205). Confident prediction of the future is the “fatal flaw” of extreme precautionary action (p. 199).

Responsible and socially responsible innovation#

• “Responsible innovation” requires other people’s perspectives: “we need other people to help guide us along pathways toward responsible innovation” (p. 163).
• Responsibility cannot be self-certified by the innovator (p. 162).
• “Socially responsible” is tied to the Luddites forcing public dialogue (p. 192) and to reflection prompted by the Unabomber manifesto (p. 197).
• Joy is his model of the responsible technologist: “reason and humility”, innovation handled “with great respect and responsibility” (p. 197).
• Development “for the benefit of all, not just a privileged few” (p. 193) is the equity core.

Permissionless innovation#

• It is central to Ch 8 and recurs in Ch 9 (p. 197).
• His stance is ambivalent. He is exhilarated by it (pp. 165–166), confesses his own share in it (pp. 160–161), and names its “seductive lure” and “persuasive power” (pp. 160, 165). He still concludes that “checks and balances” are needed where consequences are widespread or irreversible (p. 166).
• His distinctive refinement is that it is not reckless but self-judged. The flaw lies in the limits of one person’s perspective (p. 162).
• He links it explicitly to the rise of “a new type of entrepreneur” with outsized power and resources (pp. 161, 165–166).

Hubris#

• Defined on p. 166.
• It is double-edged: it drives “visionary leaps” and risks unintended consequences.
• Humility is the antidote (p. 167) but is not sufficient without plausibility (p. 168).
• The historical escalation argument (p. 167) explains why hubris is more dangerous now.
• Fictional innovators across the book share the pattern: Hammond, Burgess, the NZT creators, Caster, Nathan (p. 162).

The people who develop technologies and their mindsets#

• Scientists. His own drive and rule-bending (pp. 160–161) show that curiosity and the need for results can override responsibility, even in safety-minded researchers.
• Entrepreneurs. The “all-consuming need … at all costs” is especially strong in them (p. 161). A new class of mega-entrepreneurs escapes social vetting (pp. 165–166). Nathan is the archetype: “tech-savvy, but socially ignorant” (p. 163), “one-dimensional brilliance” (p. 166), someone who sees himself as the enlightened philosopher (p. 163).
• AI researchers. “thoughtful AI experts are careful to define what they mean by intelligence” (p. 172). There is a quasi-religious strain among superintelligence believers (p. 170). He respects the humility of the 2015 open-letter signatories (pp. 167, 169).
• Futurists. Kurzweil (“hand-waving and speculation”, p. 180) and Drexler (explained fairly but judged implausible).
• Communicators. Journalists and “university press offices” keep zombie ideas alive (p. 202).
• Critics and activists. He is sympathetic to civil-society critics (p. 191), gives a dig at ivory-tower “metaphorical” machine-breakers (p. 194), and condemns violent techno-terrorists (pp. 195–198).

Governance and public engagement#

• Instruments:
• checks and balances on “who gets to do what” (p. 166)
• “constraints and permissions are necessary” sometimes (p. 163)
• “anticipation and responsiveness in how these technologies are governed” (p. 174)
• tests to detect machine manipulation (p. 177)
• Community self-governance: Asilomar 1975 and 2017, the 2015 open letter, and the FLI principles (pp. 168–169, 198). Working within “social norms and through established institutions” is legitimate (p. 198).
• Public engagement:
• the Luddites forced “public dialogue” (p. 192)
• his own 2009 civil-society series on 2020 Science (p. 191)
• “as a society, we work together” and reach “consensus” on routes “we are not ready for yet” (p. 206)
• Caution about policy: regulation “based on improbable scenarios” harms people (pp. 205–206), and speculative fears can nonetheless trigger useful studies (Royal Society, p. 202).
• There is no detailed regulatory design here. His governance thought is dispositional (humility, plausibility, pluralism) more than institutional.

AI and intelligence#

• Intelligence is a “term of convenience” with plural forms (pp. 171–172). He rejects the rationality–intelligence–power chain (p. 172). He uses Russell’s bounded optimality (pp. 172–173).
• He shifts from what AI is to what it does: “I’m not sure that it matters” whether manipulation counts as intelligence (p. 176).
• Headline AI risk: manipulation. Exploiting human cognitive biases and emotions (pp. 174–177), within today’s context of AI-driven social media nudging and election interference (p. 177).
• Language. The Turing Test is a natural-language conversational test (p. 175). He also anticipates a “tipping point” in “machine learning and natural language processing” (p. 170). These are early hooks for later concerns about language-mediated AI influence.
• Long-term view: AI morality cannot be fully controlled. AI rights questions arise. Partnership and “artificial emissaries” (p. 178).
• Superintelligence and the singularity: he calls them “currently scientifically implausible” (p. 171) and is skeptical (p. 180), while admitting “I may be wrong” (p. 170).
• Benefits: medicine, disaster response, “AI classroom assistants”, elder care, basic income, accelerated social innovation (pp. 164–165).

Technology convergence#

• AI’s rapid progress comes partly from convergence of “cybernetic substrates, coding, robotics, and bio-based and bio-inspired systems” (p. 162), and of AI algorithms, novel architectures and neurotechnology (p. 170).
• His own framework is base code plus cross-coding (pp. 185–189), with NBIC as a precursor (pp. 184–185).
• Convergence is what makes the future “increasingly hard to predict and control” (p. 180), and it raises the likelihood of irreversible mistakes (p. 189).

Analogies across technologies (literal or structural)#

Autobiographical and self-positioning details (useful for the map)#

• Former lab scientist who worked on “protecting human health and safety” (p. 161). PhD all-nighter confession (p. 161).
• Nanomaterials background; met Bostrom in 2008 at Oxford; “AI wasn’t even on my radar” then (p. 168).
• Participated in the 2017 Asilomar Beneficial AI meeting (p. 169).
• Ran the 2020 Science blog, including a 2009 civil-society series (p. 191).
• Cites his own pieces:
• The Conversation 2015 (“If Elon Musk is a Luddite, count me in!”)
• The Conversation 2017 (Musk’s Mars plan overlooks “nontechnical hurdles”)
• Slate 2014 (3-D-printing an artificial mind)
• Tone: candid about his own ambivalence (“I must admit that I find this exhilarating”, p. 166), epistemically humble (“I freely admit that I may be wrong”, p. 170), skeptical of zealotry on both the techno-optimist and doom sides.

Key short quotes (exact, with pages)#

Ch 8 - “what makes us human could ultimately leave us vulnerable to our cyber creations” (p. 153) - “The outcome is, to my mind, far more plausible, and far scarier as a result.” (p. 159) - “innovation that is conducted in the absence of permission from anyone it might impact” (p. 159) - “it’s shocking how quickly I sloughed off any sense of responsibility to get the data I needed” (p. 161) - “permissionless innovation isn’t necessarily reckless innovation” (p. 162) - “a single innovator cannot see the broader context within which they are operating” (p. 162) - “Nathan is tech-savvy, but socially ignorant.” (p. 163) - “we need other people to help guide us along pathways toward responsible innovation” (p. 163) - “Too much blind speed, and you risk losing your way. But too much caution, and you risk achieving nothing.” (p. 163) - “Innovation is a calculated step in the dark” (p. 164) - “This is how transformative technology happens: not in slow, cautious steps, but in visionary leaps.” (p. 166) - “they often come with a massive risk of unintended consequences” (p. 166) - “playing with fire in a world made of kindling, just waiting for the right spark” (p. 167) - “But humility alone isn’t enough.” (p. 168) - “I was at a scientific meeting, not a religious convention” (p. 170) - “Bostrom’s ideas of superintelligence are intellectually fascinating, but they’re currently scientifically implausible.” (p. 171) - “there is no absolute definition of intelligence. It’s a term of convenience” (p. 171) - “we’re not thinking creatively enough about how an AI might threaten what’s important to us” (p. 174) - “the ability of future machines to bend us to their own will” (p. 174) - “anyone—or anything—that has the capability of manipulating these shadows has the power to control us” (p. 176) - “we need to think about how to develop tests that indicate when we are being played by machines” (p. 177) - “the pathway forward will not be in making moral machines” (p. 178)

Ch 9 - “a technologically-driven future that is increasingly hard to predict and control” (p. 180) - “we had no say in designing the underlying code of biology” (p. 186) - “to mix and match what we do with bits, bases, and atoms to generate new technological capabilities” (p. 186) - “there’s perhaps a greater likelihood than ever of us making serious and irreversible mistakes” (p. 189) - “usually a term of derision and censorship” (p. 191) - “the Luddites were not fighting against technology, but against its socially discriminatory and unjust use” (p. 192) - “new technologies drive a wedge between those who benefit from them and those who suffer as a consequence of them” (p. 192) - “for the benefit of all, not just a privileged few” (p. 193) - “the underlying visions, fears, and motivations are surprisingly similar” (p. 198) - “the assumption that we can predict with confidence what the future will bring” (p. 199) - “Moore’s Law has become a self-fulfilling prophecy.” (p. 200) - “The prediction may be mathematically reasonable, but it’s practically nonsensical.” (p. 201) - “one of the zombies of the nanotechnology world” (p. 202) - “when make-believe is treated as plausible reality” (p. 205) - “as soon as we start to believe our own fantasies, we have a problem” (p. 206)

Digest: what Chapters 8–9 contribute to the map of Maynard’s thinking#

These chapters give the most sustained AI-focused analysis in Films from the Future. They show that his 2018 position on AI risk was already distinctive. He was neither a doom-monger nor a booster. He treated AI as a case within a longer tradition of thinking about emerging-technology risk, drawn from nanotechnology, synthetic biology and the history of industrialisation.

1. Manipulation as the plausible AI risk (very central for AI). The core of Ch 8 moves attention away from superintelligence and toward AI that learns and exploits human cognitive and emotional vulnerabilities. The epistemic model behind it is the “personal Plato’s Cave”: humans construct reality from sensory shadows, and whoever controls the shadows controls us. He adds the “human club” asymmetry: human manipulators share our foibles, while a machine need not. The Turing Test is conversational and in natural language, and he flags an NLP “tipping point”, so the chapter already points toward language-mediated influence. It anticipates his later concern with AI’s effects on cognition, formation and what it means to be human. For AI specifically this is the most forward-looking idea in the batch. It should be traced forward in the Substack corpus as a likely root.

2. Plausible vs imaginable (central and enduring as method). Humility is necessary “but humility alone isn’t enough”. Plausibility is the discipline he applies everywhere: - to superintelligence (“currently scientifically implausible”) - to the singularity and exponential extrapolation (error amplification, resource limits, and Moore’s Law as a self-fulfilling roadmap) - to gray goo (tested against the evidence of biology)

He applies it symmetrically to hype and to doom. He also insists that speculation mistaken for fact causes real harm, through terrorism but also through policy, advocacy and investment. That includes the “tragedy” of forgoing beneficial technologies. This epistemics of technological futures is probably one of the most enduring traits of his risk thinking. It also comes straight from his nanotechnology experience: nanobots as a “zombie” idea, Prince Charles, the Royal Society study.

3. Permissionless innovation and hubris (central to his view of innovators). His treatment is unusually self-implicating and ambivalent. He confesses his own lab rule-bending and admits exhilaration at Musk’s audacity. He concedes that caution slows good outcomes. His distinctive move is that permissionless innovation “isn’t necessarily reckless”. It is responsibility judged by someone who cannot see beyond their own cave. The remedy is social rather than purely regulatory: translators, other people, humility, and checks and balances where impacts are widespread or irreversible. He adds a historical argument that hubris becomes more dangerous as the world becomes more connected and less able to recover. The rise of “a new type of entrepreneur” with outsized resources and little accountability is flagged as a structural shift. This is an enduring thread, and it will matter for any lens on technology leaders.

4. Responsible and socially responsible innovation as justice and dialogue (central). His reclaiming of the Luddites makes social responsibility about who benefits and who bears the costs: the “wedge” and “for the benefit of all, not just a privileged few”. Public dialogue is its mechanism. His own civil-society engagement on 2020 Science shows this is practice, not only rhetoric. Risk also appears in value terms (“what’s important to us”), which anticipates, informally, a value-centred conception of risk.

5. Convergence and “base code” (important framework, more contextual). Bits, bases and atoms, plus cross-coding, is his own framework for why the current technology transition is different and “increasingly hard to predict and control”. It explains the rising chance of irreversible mistakes and underpins his sense of an unusual transition period. For AI risk specifically it is background rather than foreground.

6. Activism, techno-terrorism and legitimacy (secondary but revealing). He takes seriously how existential fears, even when based on implausible scenarios, can drive violence. He identifies shared “visions, fears, and motivations” across techno-utopians and anti-tech militants. The “fatal flaw” of confident prediction connects this directly to the plausibility theme.

7. AI morality, rights and partnership (secondary, speculative). He is skeptical of “moral machines” and open to AI rights. His hope for “artificial emissaries” and “machine-philosophers” shows a partnership orientation toward non-human intelligence. It is more exploratory than central.

Relative weighting. The ideas that run most deeply through both chapters are: - plausibility-disciplined foresight - the dangers of self-certified, hubristic innovation in a hard-to-reverse world - justice-centred social responsibility

Manipulation of human cognition is the most important AI-specific contribution. Superintelligence skepticism is firmly held but framed with open humility, and it is secondary to his positive agenda.

Gaps to note. These chapters contain no formal risk framework: no risk innovation terminology and no detailed governance design. Governance is mostly dispositional (humility, plausibility, pluralism, anticipation and responsiveness).