Films from the Future (2018), Chapters 3–5: reading notes (batch FFTF-B)#

Source: working/maynard/book/FFTF-B.txt, read in full. Andrew Maynard, Films from the Future: The Technology and Morality of Sci-Fi Movies (2018). - Ch 3, Never Let Me Go: A Cautionary Tale of Human Cloning, pp. 46–62 - Ch 4, Minority Report: Predicting Criminal Intent, pp. 63–84 - Ch 5, Limitless: Pharmaceutically-Enhanced Intelligence, pp. 85–108

Page numbers are the PDF page markers in the extract, which match the printed folios. A quote that runs across a page break is cited with both pages.

Provenance. The prose, including the plot retellings and footnotes, is Andrew’s. Text he quotes from other people is not evidence of his views, and it is flagged where it matters: the 1997 “Declaration in Defense of Cloning” (p.54), the 2005 UN Declaration on Human Cloning (pp.52–53), Hank Greely (p.52), the “Marie18” forum post (p.57), the Universal Declaration of Human Rights (p.58), Vilares et al. (p.69), Wu & Zhang (p.77), the Smart Policing Initiative’s mission statement (p.83), George Burke (p.89), Balzac (p.91), Sahakian & Morein-Zamir (p.93), Giurgea (pp.93–94), the WABDA spoof press release (p.96) and Chatterjee’s five questions (pp.97–98). The epigraphs are film dialogue.

Period context. The book came out in 2018, before generative AI. “AI” in these chapters means machine-learning classifiers, big-data analytics, algorithmic bias, predictive policing, targeted advertising and persuasion, and voice assistants. There is nothing here on language models.

Chapter 3: Never Let Me Go (human cloning), pp. 46–62#

The argument#

1. Start from the fringe, then take it seriously. The chapter opens with the 2002 Clonaid “Baby Eve” claim and the Raëlian movement (pp.46–47). Andrew treats Raël as a window rather than a joke. Take out “the ‘I talk to aliens’ bits” and Raël’s case for nanotechnology, AI and cloning reads like “a more mainstream futurist like Ray Kurzweil, or even a technology entrepreneur like Elon Musk” (p.55). Because these ideas are “increasingly garnering mainstream attention,” he argues, “we have to take the possibility of human reproductive cloning seriously” (p.56).
2. The film is about the lives a technology touches, not the technology. Ishiguro used cloning only as a plot device. Because the movie keeps its focus on the lives the technology affects, it gives “a searing insight into the social and moral risks of selling our collective souls as we unquestionably embrace the seeming promise of new technological capabilities” (p.48). The clones are “a yardstick of what constitutes ‘being human’ against which their creators are measured” (p.48).
3. Cloning is technically hard but getting easier. Dolly, and Greely’s “Cats: easy; dogs: hard…” (p.52). Andrew’s own line is that “while the concept of cloning is pretty straightforward, biology rarely is” (p.52). The technique is “becoming increasingly feasible” (p.53), and “we’re going to face some increasingly tough choices as a society” (p.53).
4. Governance is contested. He describes the UN Declaration (84–34–37), the split between reproductive and therapeutic cloning, and the pro-science 1997 Declaration (Crick, Dawkins, Vonnegut). He points out the irony that one of the easiest places to find the rationalist Declaration is Raël’s religious treatise Yes to Human Cloning (p.54). He does not choose between the declarations. His emphasis falls on “the ethics of how we chose to treat and act toward those clones we create” (p.56).
5. “Genuinely Human?” Andrew recalls his mother wondering whether an IVF child would have a soul (p.56), and quotes a 2015 forum post from an IVF-conceived person asking the same thing (p.57). He concludes that “There is nothing about the mode of conception that changes the completeness or the value of a person” (p.56). The society in the film excuses itself through a “convenient lie”: that clones are less than human (p.57). Technologies that redesign us and create “new entities entirely” make the question of worth more pressing (p.57). Common definitions of “human” start to “look a little weak as we develop the ability to reengineer our own biology” (p.58). He links “us versus them” thinking and “motivated reasoning” (p.59) to the history of atrocity. The synthetic-genome project (GP-write) would make “the moral challenges of cloning seem like child’s play” (p.60).
6. Proposal: move past the category “human.” Trying to write a bigger definition of “human” still splits the world into human and not-human, and with it “an assumed right to exploit the latter” (p.60). “We will need to transcend the notion of ‘human’ and instead focus on rights, and an understanding of ‘worth’ and ‘validity’” (p.60). He extends this to animals and to AI (p.58 n.29; p.61).
7. “Too Valuable to Fail?” Hailsham’s staff are “a blip in the social conscience that is ultimately drowned out by the irresistible benefits” of the donor program (p.62). Their morality is “rather insipid” (p.62). The closing lesson: “technology has the power to rob us of our souls, even as it sustains our bodies, not because it changes who we are, but because it makes us forget the worth of others” (p.62). He carries this forward to “technologies that claim to predict how someone will behave” (p.62).

Concepts and frameworks (his terms)#

Risk#

• The main risks here are social and moral, not physical: discrimination, commodification, loss of dignity, and the erosion of a society’s moral standing (pp.47–48, 62). Physical risk (biology is hard; Dolly’s health) gets only passing mention.
• Distributional risk: harms fall on “a small number of lives” while the benefits are spread across everyone else (p.61). The danger is that “we tell ourselves we cannot live without them” (p.61).
• Risk by complicity: the worst failure belongs to the well-meaning people who don’t resist. Miss Emily “didn’t care enough to resist an atrocity” (p.62).
• Identity risk: “Human enhancement and augmentation, the merging of human and cybernetic systems, artificial intelligence, and cloning, all potentially threaten our sense of identity” (pp.57–58).

Responsible and socially responsible innovation#

• Technologies like cloning “can, if not used ethically and responsibly, lead to quite devastating discrimination and abuse” (p.47). The ethics has two parts: the process itself, and how we treat what we create (p.56).
• The chapter is less about the terms of responsible innovation than about moral attention: noticing who pays for a technology’s benefits.

Permissionless innovation and hubris#

• Not named. The nearest case is Clonaid/Raël pursuing human cloning “despite near-universal restrictions prohibiting it” (p.46). Andrew calls Raël’s three-step immortality plan “Stupendously complex (not to mention, implausible)”, yet notes that some people are “already working on” parts of it (p.55).
• Hubris shows up as species self-importance: calling species “what we perhaps arrogantly call ‘higher’ species” (p.61).

Developers and their mindsets#

• Visionary/fringe entrepreneurs: Boisselier, Clonaid and Raël. Raël’s vision “would appeal to many who see humans as no more than sophisticated animals and technology as a means of enhancing and engineering this sophistication” (p.55). He places transhumanists, Kurzweil and Musk on the same spectrum.
• Scientists: the 1997 signatories want decisions made on “science and reasoned thinking”. Andrew’s gloss is that they believe “religious beliefs and mystical thinking should not be allowed to stand in the way of scientific progress” (p.54). He reports this neutrally, then undercuts it with the Raël irony.
• Institutional insiders: Miss Emily and Madame are “all mouth and no backbone” (p.50). Conscience without courage.

Governance and public engagement#

• International soft law: the UN Declaration and its non-unanimous vote, and the reproductive/therapeutic line (pp.52–53). Rights frameworks: the UDHR “doesn’t actually define what ‘human’ means” (p.58).
• Public engagement is implicit: “tough choices as a society” (p.53). Nothing on specific processes.

AI and intelligence#

• n.29 (p.58): personhood may “transcend the distinction between ‘human’ and ‘non-human.’” That opens the question of rights for “animals, artificial intelligence, and other non-human life forms”. He asks “Who decides”, and whether we bring “our own pro-human biases”.
• p.61: the authority question “gets even more challenging when we begin to consider completely artificial entities such as computer- or robot-based artificial intelligence”. He points ahead to Minority Report and Ghost in the Shell.
• p.61: we “especially value species that demonstrate human-like intelligence”. Intelligence as a yardstick of worth, a theme Ch 5 develops.

Technology convergence#

• Raël’s program combines cloning, accelerated growth and mind uploading. Andrew notes that artificial wombs (Ch 2) and mind capture (later chapters) are real research areas (p.55).
• The identity threat comes from several technologies at once: enhancement, human–machine merging, AI and cloning (pp.57–58). Nootropics, Ghost in the Shell and Transcendence are staged as steps along the same continuum (p.60).

Analogies across technologies#

Key quotes (Ch 3)#

• “the social and moral risks of selling our collective souls as we unquestionably embrace the seeming promise of new technological capabilities” (p.48)
• “those who have sacrificed him to the gods of their technology” (p.50)
• “the society that created the technology that produced them has been judged, and found wanting” (pp.50–51)
• “the questionable morality of convenient technologies” (p.51)
• “There is nothing about the mode of conception that changes the completeness or the value of a person.” (p.56)
• “It’s a convenient lie” (p.57)
• “‘different’ is no longer simply something we’re born with, but something we have the means to create” (p.58)
• “we will need to transcend the notion of ‘human’ and instead focus on rights” (p.60)
• “our measures of what has worth inevitably come down to what has worth to us” (p.61)
• “how easily we fall into justifying technologies that devastate a small number of lives, because we tell ourselves we cannot live without them” (p.61)
• “a society that sees the donor program as too valuable to end” (p.62)
• “technology has the power to rob us of our souls, even as it sustains our bodies” (p.62)
• “because it makes us forget the worth of others” (p.62)

Chapter 4: Minority Report (predicting criminal intent; machine-learning precognition), pp. 63–84#

The argument#

1. Personal hook: the Veris Prime “Trust Index.” Andrew took an online test marketed as “‘Minority Report-like’ tech”. He scored 19 and a colleague scored 2 (pp.63–64). The test’s initial data came from 117 white-collar felons. His diagnosis: “One of the many issues with the Veris Prime test is the training set it uses” (p.64). The traits that marked felons resemble those of “curious, independent, and personally-motivated academics” (p.64). The prior question is “the extent to which we should even be attempting to use science and technology to predict and prevent criminal behavior” (p.64).
2. The film: Precrime is “astoundingly successful—at least on the surface” (p.64). Its founder, Lamar Burgess, murders to protect an “inconvenient truth” that stood in the way of “what he believed was a greater social good” (p.66). The film shows “a world where technology has seemingly made people’s lives safer, but at a terrible cost that isn’t immediately obvious”, and puts “a searing spotlight on the question of ‘should we’ when faced with a seductive technology” (p.68).
3. “Science” of predicting bad behavior. First, media hype: the Guardian’s fMRI headline “vastly overstepped the mark” (p.68), though the study’s own authors had disavowed that reading (p.69). Then the historical lineage: phrenology (correlation mistaken for causation), Lombroso and eugenics, “the same seductive idea that, if we know what makes people ‘bad,’ we can remove them from society” (p.70). Then the 2011 study of faces and criminality (p.71): small, biased and hard to interpret. The deeper issue is “the ethical issue with carrying out and publicizing the results of such research in the first place” (p.71).
4. Two objections that don’t depend on the science. (a) Prediction invites “a better-safe-than-sorry attitude to law and order” and “a path where we assume that other people do not have agency over their destiny” (p.72). (b) Law codifies normative expectations, not absolute good and bad. His examples are homosexuality (illegal in the UK until 1967), LGBTQ rights and women’s rights (p.72). “Even if we can predict tendencies from images alone… should we?” (p.72).
5. Brain scans as “the high-tech version of ‘looks like a criminal’” (p.73). fMRI is “quite incredible science” (p.73), but “neuroscience is racing ahead of our ability to cope with what it reveals” (p.74). In the sunscreen study, researchers predicted behavior better than the subjects did (p.74). His key methodological claim: “It’s not so much that we can collect data on brain activity that’s problematic; it’s how we decide what data to collect, and how we end up interpreting and using it, that’s the issue” (p.75).
6. Science is self-correcting, but slowly. It is a “misapprehension that the scientific method is pure and unbiased” (p.75). Science self-corrects, but “this self-correcting nature of science takes time, sometimes decades or centuries” (p.75). Bias is amplified where evidence is uncertain and statistics are complex, and with small, non-reproducible studies (pp.75–76). Such predictions have no place “in informing legal action—or any form of discriminatory action—before any crime has been committed” (p.76).
7. Machine-learning precognition. “pre-justice” challenges free will, and the science carries hidden “value judgments about what sort of behavior is unwanted” (p.76). Chaos and complexity (from Ch 2) mean we can “draw boundaries around more or less likely behaviors” but never predict with certainty (p.76). Wu & Zhang’s 2016 face-classifier paper introduces algorithmic bias (p.77) and opacity: “we have not only trained computers to do our thinking for us, but we no longer know how they’re thinking” (p.78). The ironic risk is “creating machines that exhibit equally undesirable behavior, precisely because they are unpredictable” (p.78).
8. “Big Brother, Meet Big Data.” In the film, humans still interpret the precog feed: Anderton’s intuitive “ballet”. That affirms a belief that people are “more than the sum of the chemicals, cells, and organs we’re made of” (p.78). Andrew questions it: “are we right in this belief…?” (p.79). He traces big data to the genome project, whose “code of life” promise was, “With hindsight… wrong” (p.79). The more immediate danger is persuasion: targeted ads, Cambridge Analytica, and “engines of persuasion” held back only by “our collective scruples and privacy laws” (p.81). Manipulation is “a subtler and more Machiavellian approach to achieving what is essentially the same thing—controlling people” (p.81). People already concede their freedom “without even thinking about it” through Siri, Alexa and connected cars (p.82).
9. Predictive policing, the real-world Precrime. Palantir, the Stop LAPD Spying Coalition report and the Smart Policing Initiative (pp.82–83). He is balanced: evidence-based policing “makes a lot of sense” and the benefits of data-driven crime prevention are, in his words, “many” (p.83). But without due diligence it “could easily slip into profiling and ‘managing people’” (p.83). “We’re replacing Minority Report’s precogs with massive data sets and AI algorithms, but the intent is remarkably similar” (p.83). Systems trained on bias target “black, brown, and poor” communities “because the predictive systems had been trained to believe this” (p.83).
10. Closing: big data “could make our lives safer and more secure” if used wisely. That hope “has to be tempered by our unfailing ability to delude ourselves… and to justify the unethical and the immoral in the service of an assumed greater good” (p.84).

Concepts and frameworks#

Risk#

• Risk comes from a failed method (training sets, small samples, non-reproducibility, correlation versus causation, hype) and from a sound method put to a wrong purpose (pre-emption, discrimination, manipulation). He insists on the second even if the first were fixed.
• Hidden cost: safety gains that conceal “a terrible cost that isn’t immediately obvious” (p.68).
• Risks from AI in particular: bias inherited from developers (p.77), opacity (p.78), unpredictable behavior (p.78), scaled persuasion when big data is “coupled to machine learning” (p.81), and ubiquitous identification through facial recognition and digital fingerprints (p.82).
• Risk to agency: assuming “other people do not have agency over their destiny” (p.72). Research showing that “the researchers knew their subjects’ minds better than they did” (p.74).

Responsible and socially responsible innovation#

• “Is this a responsible direction to be heading in?” (p.75). Responsible research includes asking whether a question should be asked and published at all (p.71).
• Responsible use: “we learn to wield this tremendously powerful technology responsibly and humanely” (p.84). He accepts that data-driven policing has real benefits (p.83).

Permissionless innovation and hubris#

• Not named. Implicit: Veris Prime was marketed with no validation, and Palantir “largely flew under the radar for many years” (p.82).
• Hubris and over-reach: the genome “code of life” promise (p.79); “our technology is rarely as powerful as our imaginations would like it to be” (p.80). Lamar Burgess is the creator-protector who sacrifices people to save his program (pp.66–67).

Developers and their mindsets#

• Scientists: in the past, “many scientists at the time supported” eugenics, and “these ideas coming from academics and scientists gained political momentum” (p.70). Today, researchers publish studies on faces and criminality. Wu & Zhang claimed their work was “only intended for pure academic discussions” (quoted at p.77). Andrew doesn’t buy it: “It’s hard to interpret this as anything other than” a claim that criminal-detecting machines are possible (p.77). The mindset he criticizes treats science as pure and unbiased (p.75), and fails to examine the motives and assumptions that steer “a complex series of scientific decisions” (p.75).
• Designers of AI systems: their “unconscious (and sometimes conscious) biases” get encoded (p.77). If designers “believe they know who the ‘bad people’ are”, systems will target them (p.83).
• Entrepreneurs: Veris Benchmark marketed a trust score; Palantir’s founders (p.82).
• Media: headline writers in a “clickbait frenzy” (p.69).

Governance and public engagement#

• Privacy law and GDPR. Experts are “skeptical” that it can stop mass data collection (p.82 n.43). Only “collective scruples and privacy laws” hold back engines of persuasion (p.81).
• The Smart Policing Initiative. Civil-society scrutiny from the Stop LAPD Spying Coalition (pp.82–83). The presumption of innocence (p.72).
• “We’re going to have to make some tough decisions over how much of our personal freedom we are willing to concede” (p.82). That is framed as a collective choice.

AI and intelligence#

This is the most AI-heavy of the three chapters. It covers ML classifiers of criminality (pp.77–78), algorithmic bias (p.77), black-box opacity (p.78), big data plus ML “join[ing] the data dots, and even interpolat[ing] what’s missing” (p.81), predictive policing (p.83), and voice assistants (p.82). The human-intuition theme: in the film, “intuition and creativity still have an edge over machines” (p.78). He asks whether that comforting belief is true, rather than asserting it (p.79).

Technology convergence#

• Neuroscience (fMRI) plus ML plus big data (pp.73–81); genomics as the origin of big data (p.79); brain understanding used to train artificial brains (p.78); facial recognition plus identity data plus connected devices (p.82). Computer-simulation speculation (Bostrom) as the extreme end (p.80, n.42).

Analogies across technologies#

Key quotes (Ch 4)#

• “One of the many issues with the Veris Prime test is the training set it uses.” (p.64)
• “the dangers of being sucked in by seemingly beneficial technologies” (p.68)
• “the question of ‘should we’ when faced with a seductive technology that ultimately threatens to place society in moral jeopardy” (p.68)
• “a classic case of correlation erroneously being confused with causation” (p.70)
• “we assume that other people do not have agency over their destiny” (p.72)
• “the high-tech version of ‘looks like a criminal’” (p.73)
• “neuroscience is racing ahead of our ability to cope with what it reveals” (p.74)
• “it’s how we decide what data to collect, and how we end up interpreting and using it, that’s the issue” (p.75)
• “this self-correcting nature of science takes time, sometimes decades or centuries” (p.75)
• “embedded in the ‘science’ are value judgments about what sort of behavior is unwanted” (p.76)
• “we have not only trained computers to do our thinking for us, but we no longer know how they’re thinking” (p.78)
• “machines that exhibit equally undesirable behavior, precisely because they are unpredictable” (p.78)
• “engines of persuasion” (p.81)
• “a subtler and more Machiavellian approach to achieving what is essentially the same thing—controlling people” (p.81)
• “we’re replacing Minority Report’s precogs with massive data sets and AI algorithms, but the intent is remarkably similar” (p.83)
• “to justify the unethical and the immoral in the service of an assumed greater good” (p.84)

Chapter 5: Limitless (cognitive enhancement), pp. 85–108#

The argument#

1. He called the trend early. He listed nootropics ninth among ten trends to watch in a 2009 post on his 2020 Science blog (p.85). By 2018 there is a market of “stacks” giving “a legal, or at least a not-too-illegal, edge” (p.85).
2. Framing: there is “an almost unquestioned assumption” that better memory and faster thought are needed for success (p.86). Physical tradeoffs exist (“there’s usually a price to pay”), but “things are more complex when it comes to social tradeoffs” (p.86). The guiding question: “how can we navigate our way to using increasingly powerful cognitive enhancements responsibly?” (p.86). He reads the film as “relatively ambivalent” (p.86). It “challenges viewers to think about the pros and cons” (p.88).
3. “The Seduction of Self-Enhancement.” Our culture assumes “smarter is better” (p.88). Enhancement is continuous with tool use: “we enhance our intelligence through artificial means all the time” (p.88), and Googling is an example. Brain-hacking is “big business” (p.89). George Burke and the Silicon Valley nootropic culture are his examples (p.89). He writes a self-deprecating passage about being tempted, as “Enhanced Andrew”, while suffering from brain fog (p.90), then turns: “thinking beyond my own selfish needs—what are the social and ethical pros and cons” (p.90).
4. Nootropics: the evidence. He discusses Chatterjee’s “Cosmetic Neurology” and criticizes the word “cosmetic” for its “air of frivolity” (p.90). Caffeine is the normalized precedent, with Balzac as the example (p.91). He covers off-label Adderall, Ritalin and modafinil (pp.91–92), and notes the evidence is thin: “surprisingly little evidence that Adderall does increase performance in healthy adults” (p.92 n.52). Prevalence is hard to pin down. The “Professor’s little helper” commentary and the Nature poll (p.93). Side effects “aren’t widely tracked”, and this uncertainty drives experimentation with “less restricted—and often less studied—substances” (p.93). Giurgea coined “nootropic” for piracetam (pp.93–94). The 10% brain myth is “pure scientific bunkum” (p.95). He warns against thinking of brains as computers (p.95). We may be “using the wrong measures of success” (p.95).
5. “If You Could, Would You?” The WABDA April Fool’s spoof (p.96). The Nature survey shows academics “remarkably indifferent” to colleagues’ brain doping, unlike attitudes in sport (p.97). He finds this surprising: “That doesn’t mean we shouldn’t be concerned, though” (p.97). He sets Chatterjee’s five questions beside five of his own about competitive advantage (p.98). The darker issue is “what happens if enhancement becomes the norm, and there is mounting social pressure to become a user” (p.98). Normalization raises “serious ethical questions around autonomy and agency” (p.98) and calls for “socially responsible use… not to say that they should be banned or discouraged” (p.98). The ethics “flips” from “would you be OK” to “would you do this” (p.99). The crux: “by using an artificial aid to succeed, someone else is excluded from success” (p.99). In academia, the endpoint is “either joining the smart-drug crowd, or burning out” (p.100). The response is to “work out what the rules, norms, and expectations of responsible use should be” (p.100). More powerful nootropics are likely, so “we’re going to need new thinking on how, as a society, we use and regulate these chemical enhancers” (p.100).
6. “Privileged Technology.” “One of the perennial challenges of new technologies is their potential to exacerbate social divides” (p.100). It is “too easy to assume that technology trickle-down is a given” (p.101). He then runs a graded thought experiment on NZT using the four principles of medical ethics (pp.101–103), described below. Conclusion: an inequitable, exclusive NZT “is not a scenario that I’m comfortable with”, and not one “that I believe can be avoided through market-driven innovation alone” (p.103). Free markets “can thrive on social inequity and injustice”, so “What is needed… is a system of checks and balances that help steer market forces toward social good” (p.103). This “is not about stymieing technologies… far from it”. It means “deciding what’s important, and having the foresight and commitment” to steer innovation (p.104).
7. “Our Obsession with Intelligence.” Some people worry that AI will “end up destroying us”, but “how we think about intelligence is remarkably colored by our sense of our own importance” (p.104). Human exceptionalism is “an evolutionary illusion” (p.104). What makes us interesting is our capacity to imagine futures and bring them about (p.104). “Intelligence” becomes “a convenient shorthand for ‘that which makes us different.’” (p.105). He contrasts Spearman’s g and IQ with Gardner’s multiple intelligences (p.105). Limitless presents “a restrictively narrow view of intelligence” and “a rather monochromatic perspective of success—especially when it comes to technological innovation” (p.106). “Smart drugs are not really about intelligence, but about selectively enhancing capabilities that provide a perceived performance advantage” (p.106). Equating memory and reasoning with winning is “a cognitive delusion” (p.106). A clear sense of what intelligence is “is critical to the socially responsible development and use of smart drugs and intelligence-related technologies more broadly” (p.107). “Being smart doesn’t make you good” (p.108). We must “recalibrate how we think about intelligence”, which is critical for AI too: a “warped perspective of intelligence and success” will produce technologies developed along “equally warped” pathways (p.108). The last twist: success in the film depends on being “privileged enough to have access” (p.108). That leads to Elysium.

Concepts and frameworks#

Risk#

• Physical risk under uncertainty: “none of these substances are risk-free” (p.93); “careful balancing of short-term gains against potential downsides” (p.93); side effects “aren’t widely tracked” (p.93); possible long-term effects on cognition, where “the research here is patchy” (p.92). The film’s “no such thing as a free lunch” (p.87).
• Social risk is the bigger concern: coercion by norm, exclusion, inequity, and a positive feedback loop of privilege. Throughout, risk is weighed against real benefits: “there are going to be occasions where their use is likely to benefit individuals and the communities that they are a part of” (p.100).
• Values risk: optimizing for the wrong idea of intelligence or success “we run the risk of sacrificing possibly essential parts of ourselves” (p.95). “What if our urge to make more money, to publish more papers, or to be famous, leads to us ultimately risking what makes us who we are?” (pp.95–96).

Responsible and socially responsible innovation#

This chapter is the densest on the theme. It uses the phrases “responsibly” (p.86), “socially responsible use” (p.98), “responsible use” (pp.99, 100, 106) and “socially responsible development and use of smart drugs and intelligence-related technologies” (p.107). It endorses Greely et al.’s call for “responsible use” (pp.98–99, n.62). He is explicitly not prohibitionist (“not to say that they should be banned or discouraged”, p.98). His positive vision: “I can imagine a future where smart drugs are a powerful technology for benefitting lives as part of a suite of technologies that we use to build a better future” (p.108).

Permissionless innovation and hubris#

• Not named. Close analogues: a black market in prescription smart pills (p.91), piracetam in a “legal gray zone” (p.94), DIY “stacks” and biohacking meetups (p.89), and his view that market-driven innovation alone cannot secure justice (p.103).
• Hubris is explicit in a footnote: “our capacity for delusion is only matched by that for hubris” (p.105 n.64). Species self-regard: “we’d think we were phenomenally successful whatever we achieved as a species” (p.88 n.46). “We are, in our own eyes, exceptional” (p.104). He notes that people search for smartness pills, not “humility pills” (p.108 n.65).

Developers, users and their mindsets#

• Entrepreneurs: George Burke, and Silicon Valley’s view of stacks as “the route to winning” (p.106). “Enterprising entrepreneurs” who slash access costs can narrow divides, so their role cuts both ways (p.101).
• Pharma: companies must “recoup their development costs… into the billions”, which makes exclusive pricing “more plausible” (p.103).
• Academics (his own tribe): competitive, working “sixty- to seventy-hour weeks”, and ambivalent about brain doping (pp.97, 99–100). He counts himself in, with humor and candor.
• The winning mindset: students, academics and executives all using drugs “to win” (p.106). “It’s all very ‘survival of the fittest.’” (p.106).

Governance and public engagement#

• “As a society, we need to work out what the rules, norms, and expectations of responsible use should be” (p.100). “New thinking on how, as a society, we use and regulate these chemical enhancers” (p.100). Checks and balances on markets (p.103). Foresight and commitment (p.104). “We need to work out what we believe is important to us as a society” (p.104).
• Public engagement appears only as a collective values exercise. No specific process is named. His reader questions (p.98) work as an engagement tool on the page.

AI and intelligence#

• The chapter’s final pages deliberately connect human enhancement to AI. It is about “technologies that either claim to enhance intelligence, or to replicate it” (p.105). “It’s also critically important to how we think about and develop artificial intelligences, or hybrid human-machine intelligences” (p.108). He notes the AI existential worry in passing (“some people worry will end up destroying us”, p.104) without endorsing or dismissing it.
• Anti-computationalism: “comparing what’s in our head to a computer is probably a bad idea” (p.95). “Our brains are nothing like the chips in our smartphones” (p.95 n.58).

Technology convergence#

• Brain-mapping programs (the US BRAIN Initiative, the EU Human Brain Project including “brain-inspired computing”) are expected to yield more powerful nootropics (p.100 and n.63). Search engines are cognitive aids (p.88). Hybrid human–machine intelligence (p.108).

Analogies across technologies#

Key quotes (Ch 5)#

• “Mess with your body’s chemistry, and there’s usually a price to pay somewhere down the line.” (p.86)
• “how can we navigate our way to using increasingly powerful cognitive enhancements responsibly?” (p.86)
• “we enhance our intelligence through artificial means all the time” (p.88)
• “there’s a danger to thinking of our brains as computers” (p.95)
• “what happens if enhancement becomes the norm, and there is mounting social pressure to become a user” (p.98)
• “This is not to say that they should be banned or discouraged” (p.98)
• “by using an artificial aid to succeed, someone else is excluded from success” (p.99)
• “we need to work out what the rules, norms, and expectations of responsible use should be” (p.100)
• “it’s too easy to assume that technology trickle-down is a given” (p.101)
• “this is where medical ethics begin to run out of steam” (p.102)
• “free-market economies can thrive on social inequity and injustice” (p.103)
• “a system of checks and balances that help steer market forces toward social good” (p.103)
• “not about stymieing technologies that threaten us—far from it” (p.104)
• “This assumption of exceptionality, though, is an evolutionary illusion.” (p.104)
• “our capacity for delusion is only matched by that for hubris” (p.105 n.64)
• “smart drugs are not really about intelligence, but about selectively enhancing capabilities that provide a perceived performance advantage” (p.106)
• “Being smart doesn’t make you good.” (p.108)
• “if we start off with a warped perspective of intelligence and success” (p.108)

Threads across the three chapters#

• Worth and dignity as the center of ethics. The chapters form a chain. Clones are denied worth by a convenient lie (Ch 3). Predicted “criminals” are denied agency by pseudo-objective data (Ch 4). The unenhanced are denied success, and worth is tied to narrow intelligence (Ch 5). Ch 3 ends by naming “technologies that blur our definitions of ‘worth,’ including the use of technologies that claim to predict how someone will behave” (p.62), and the thread runs to “determining someone’s worth” (p.107).
• Seduction by benefit. “Too valuable to end” (p.62), “seemingly beneficial technologies” (p.68), “the seduction of self-enhancement” (p.88), and “an assumed greater good” (p.84). Harms hide behind real benefits.
• Bias and self-delusion. Motivated reasoning (p.59), researcher bias (p.75), algorithmic bias (p.77), “our unfailing ability to delude ourselves” (p.84), cognitive delusion (p.106), and delusion matched by hubris (p.105 n.64).
• Not prohibition, but “should we” plus responsible steering. He stays neutral on cloning bans, grants that predictive policing has benefits, and is explicitly not prohibitionist on smart drugs. His consistent move: accept that a technology can do good, then insist on deliberate, value-led, justice-aware choices (“deciding what’s important”, p.104).
• Humans are not computers. “more than the sum of our parts” (p.79) is questioned, not asserted. The brain-as-computer analogy is rejected (p.95). Big-data reductionism is doubted: “the more complex we discover life is” (p.79).
• Intelligence and AI. Each chapter touches AI. Ch 3: AI entities and personhood (pp.58, 61). Ch 4: ML precognition, bias, opacity and persuasion engines. Ch 5: definitions of intelligence shape AI development (p.108).
• Method. Personal anecdote (his mother and IVF, his Trust Index of 19, the student research assistant’s pill, “Enhanced Andrew”), footnote humor, a turn from fiction to current science, then to ethics, then to society, and a bridge sentence into the next chapter. He often asks questions rather than giving answers (pp.60, 98).

Absences and notes on accuracy#

• Absent: the terms “permissionless innovation”, “risk innovation” and “responsible innovation” (as a named field); “hubris” appears only in a footnote (p.105 n.64); no specific public-engagement processes; no chemical or nanomaterial risk analogies.
• Minor slips in the text: phrenology is dated to “the seventeenth century” (p.70), but it is late 18th to 19th century. The text says “Professor’s little helper” was published in 2008 (p.92), but the citation gives 2007 (p.93 n.54). Ch 3 calls the program both the “National Donor Program” (p.48) and the “Human Donor Program” (p.62). None of these affects the argument.

Digest: what these chapters contribute (about 850 words)#

These three chapters show Andrew’s thinking about risk and technology at an important point. In 2018 he had come from nanomaterial risk science and was moving toward a wider ethics of “being human” in an age of powerful technologies. The chapters treat risk almost entirely as social and moral. Physical harm (cloning’s biological difficulty, the side effects of smart drugs) gets acknowledged, then set aside for what he considers the harder questions: who is harmed, who decides, and what a society becomes when it accepts those harms. This is the book’s working definition of risk, and it is probably the most central and enduring idea in this batch.

1. Worth and dignity as the core of risk (very central). The strongest thread is that technologies become dangerous when they let us “forget the worth of others” (p.62). The Never Let Me Go chapter builds the frame: the “convenient lie” that clones are less than human; “distinctions of convenience” used to justify atrocity; the argument that we should “transcend the notion of ‘human’” in favor of rights and worth; and the pointed question of “What natural authority” we have over our creations. He extends that question to AI entities and animals. Chapters 4 and 5 carry the same frame over to predictive policing (agency denied by data) and cognitive enhancement (worth tied to narrow intelligence). This fits his long-running Future of Being Human project and is likely foundational.

2. Seduction by benefit and “should we” (central). Each chapter shows real benefits masking costs: the donor program “too valuable to end”, Precrime’s safety that hides “a terrible cost”, the “seduction of self-enhancement”. Andrew never argues that the benefits are illusory. He grants that data-driven policing “makes a lot of sense” and that smart drugs could benefit lives. Instead he makes “should we” a necessary question alongside “can we”. This balanced but insistent stance is characteristic.

3. Bias, self-delusion and fallible science (central). Chapter 4 gives the clearest early statement of his view of science: powerful and self-correcting, but slow to correct, and bent by human assumptions in which questions get asked and how data are chosen and interpreted. He traces a lineage from phrenology through eugenics to fMRI and machine-learning “precognition”, and defines algorithmic bias. The line about “our unfailing ability to delude ourselves” in service of “an assumed greater good” connects to his repeated warnings about hubris (explicit here only in one footnote).

4. Early AI-risk thinking (important, though of its time). Before generative AI, Andrew already names several AI risks that later become prominent: algorithmic bias inherited from developers; opacity (“we no longer know how they’re thinking”); unpredictable machines; and, perhaps most significant for his later work, “engines of persuasion” in which big data plus machine learning enables “a subtler and more Machiavellian” form of control. He also notes people giving up freedom “without even thinking about it”. This is a clear early node in what later becomes his concern with AI manipulation and AI’s effects on cognition and agency.

5. Intelligence, and the refusal to equate brain with computer (central, and it grows over time). Chapter 5’s “Our Obsession with Intelligence” argues that intelligence is a self-flattering shorthand for “that which makes us different”. It also argues that narrow, win-oriented definitions produce warped technologies, and that “Being smart doesn’t make you good.” He applies this explicitly to AI and hybrid human–machine intelligence: warped definitions produce warped development pathways. With his rejection of the brain-as-computer metaphor and his view of human exceptionalism as “an evolutionary illusion”, this forms an early base for his later thinking about AI, cognition, education and what it means to be human. It also touches education (students, parents, IQ pressure).

6. Justice, access and markets (central to his responsible-innovation stance). The Limitless chapter has the book’s most explicit normative apparatus: the four principles of medical ethics as criteria for “innovation pathways”; the distinction between correcting a deficit and conferring an advantage (where ethics “run out of steam”); a graded NZT thought experiment; the “privileged technology” feedback loop; and a rejection of assumed trickle-down. His position is non-prohibitionist but not laissez-faire: free markets “can thrive on social inequity”, so we need “checks and balances that help steer market forces toward social good”. The work is “not about stymieing technologies” but about “deciding what’s important” with foresight. This is an early, concrete statement of the responsible-innovation posture he comes back to often.

7. Normalization and coercion (moderately central). His shift from “would you be OK” to “would you do this”, and his worry about social pressure to enhance (“joining the smart-drug crowd, or burning out”), identify autonomy lost through norm-creep as a distinct kind of risk. It anticipates later concerns about technologies whose adoption stops being optional.

Less central here: governance mechanisms (declarations, GDPR and privacy law are mentioned, not analyzed); public engagement (implied as “as a society” choices, with no specific process); technology convergence (present as neuro, data and AI links and Raël’s cloning-plus-upload plan, but not theorized); and permissionless innovation (not named). The analogies are mostly structural (the less-than-human mechanism, the precog-to-algorithm lineage, caffeine as a model of a normalized enhancer). He explicitly rejects one tempting literal analogy, brain equals computer. His method is already his signature: anecdote, film, current science, then ethics and society, with self-deprecating candor and a preference for hard questions over prescriptions.