Late Lessons, Jensen Huang and AI

B05 reading notes: 2020-08-28 to 2021-02-25 (13 posts)#

Batch context: this is the Medium-era (“Edge of Innovation”) material carried over to the Substack. Most original posts were first published on the ASU College of Global Futures Dean’s Blog, where Maynard was then associate dean. The rest are republished excerpts from his sole-authored 2018 book Films from the Future (FFTF). None of the posts is AI-generated, a guest post or co-written, and no Modem Futura podcast posts fall in this batch. FFTF excerpts are his own prose, but they record what he thought in 2017–18, reissued with a new framing in 2020–21. In the notes below, “2018 text” and “2020/21 framing” are kept apart where it matters.

Relevance summary:

Date Slug Relevance
2020-08-28 navigating-the-complex-world-of-advanced-brain-machine-interfaces-e5c6e429001d high
2020-09-26 the-seductive-slippery-slope-of-using-science-to-predict-bad-behavior-aea827f6b4ac high
2020-10-04 the-future-of-predictive-policing-c8204b2d26c3 low (duplicate excerpt)
2020-10-15 the-ethics-of-advanced-brain-machine-interfaces-and-why-they-matter-fdd77aafc376 high
2020-10-22 what-if-the-future-was-an-object-fe4eac545fa3 low
2020-10-30 eight-things-about-future-rising-that-may-surprise-you-545e5f0b3c2f none
2020-11-05 risk-innovation-and-the-future high
2020-11-12 is-artificial-intelligence-going-to-kill-us-all-6ae9d059c40d high
2020-12-10 we-need-to-rethink-our-relationship-with-the-future-9fe0f60ecdb3 medium
2020-12-15 why-trustworthiness-matters-in-building-global-futures-50a91fcb9bb2 medium
2021-01-15 can-watching-sci-fi-movies-lead-to-more-responsible-and-ethical-innovation-7c993bdaa5c2 medium
2021-02-04 the-future-you-13d5b0039b91 low
2021-02-25 how-our-mastery-of-biological-physical-and-cyber-base-code-is-transforming-how-we-think-about-b2eae9d589d0 high

2020-08-28 — navigating-the-complex-world-of-advanced-brain-machine-interfaces-e5c6e429001d#

Navigating the Complex World of Advanced Brain-Machine Interfaces

Provenance. A short italic framing note from 2020, then long excerpts from FFTF (2018), the chapter built around Ghost in the Shell. All of it is his own prose. It was reissued because Neuralink was about to give a demo. The Muddy Waters report on St Jude, and lines from Hugo Campos and England et al., are quoted material.

Argument. - The brain is closely tied to our sense of identity. - Identity is already fragile and can be changed (“disease, injuries, or persuasive influences can change us”). - Neural enhancement adds new threats to self-identity, “including vulnerability to outside manipulation”. - Neuralink’s “neural lace” is his example of a technology with “vast potential and largely unknown risks”.

He sets out three risk layers: 1. Biological harm from implants. 2. Hacking. He is close to certain that “if and when someone connects a part of their brain to the net, someone else will work out how to hack that connection”. He says this risk “far transcends” the biological harms. 3. A deeper risk. Merging biology with cyberspace means living in a human-made world that “hasn’t had billions of years of natural selection for the kinks to be ironed out”. That world “reflects all the limitations and biases and illusions that come with human hubris”.

He then turns to control and ownership of augmented bodies (“who owns you?”), using implantable cardiac defibrillators (ICDs) as the example: - Patients cannot get the data their own ICD produces (Campos). - The law is unclear about who may switch a device off (England et al.’s category of “integral devices”). - St Jude pacemakers were found to be hackable, and the FDA recalled about 465,000 of them. - The October 2016 Internet of Things DDoS attacks showed that outdated hardware can only be fixed by physically replacing it.

From these he projects forward to enhancement implants that are not medical. The user becomes dependent on the supplier for security patches and upgrades, which amounts to a kind of indentured servitude. The Major Kusanagi story is the fictional version of this.

How firmly. Firm about the direction of the risk, cautious about timing (“we’re a long, long way from any of this”). He explicitly does not oppose the technology: “Not that I think this should be taken as an excuse not to build” them, and “it would be hard to resist the technological impetus”. His answer is to map the hazards (“Here be monsters”), not to stop the work.

Concepts. - Cyberspace as a “completely human-created dimension” (a “fifth dimension”) whose rules were written as people went along, with more attention to what they could do than what they should do. - Old “pull the plug” thinking is obsolete once cyber and physical systems converge. - “ghost hacking”, from the film. - Technological indentured servitude. - The hazard map (“Here be monsters”). - Enhancement vs. medical implants as the point where governance breaks down.

Analogies. - Literal precedents: ICDs and pacemakers, the St Jude recall, Internet of Things devices, the early history of computing and cybersecurity. He uses these as direct forerunners of brain-machine interface (BMI) governance problems. - Conceptual: fiction (Ghost in the Shell; Phil Kennedy’s novel 2051, with AI “nanobots” that manipulate wired-up brains; Star Trek’s Borg).

AI. Mostly in the background: - The fictional AI nanobot “hive-mind” that manipulates perceived reality. - A passing reference to “a world where we are seceding increasing power to autonomous systems” (“seceding” is his original spelling). - AI links here to manipulation of experience through connected minds.

Companies and leaders. - Musk. His neural-lace tweet might be “entrepreneurial frippery” but was apparently meant seriously. - The Neuralink job advert saying “No neuroscience experience is required” is “a little scary”. - Tech developers want to fix “what they see as the limitations of our biological selves”. - Implant makers: “they own you”.

Governance. - Regulations “are going to have to change and adapt to keep up”. - Strict medical-device rules have so far limited the problems. The danger is loosely regulated enhancement devices. - He expects consumer pressure against regulation, “in the rush for short-term gratification”. - Suppliers will owe users a continuing duty of care.

Cognition and formation. - Identity is malleable. - Persuasion and manipulation are threats to the self. - Someone could become “someone else’s puppet” without knowing it.

Quotes. - “neural enhancements bring with them a brand new set of threats to self-identity” - “we’ve essentially built a fifth dimension to exist in, while making up the rules along the way” - “we should be working with maps that says in big bold letters, “Here be monsters.”” - “The company may not own the hardware, but without a doubt, they own you, or at least your health and security.”


2020-09-26 — the-seductive-slippery-slope-of-using-science-to-predict-bad-behavior-aea827f6b4ac#

The Seductive Slippery Slope of using Science to Predict “Bad” Behavior

Provenance. A 2020 italic framing note (“Given renewed interest in this area”), then FFTF chapter 4 (Minority Report). All of it is his own prose. The quoted lines are from the studies’ authors (PNAS fMRI study; Wu and Zhang’s 2016 arXiv paper, whose authors he does not name) and from the Smart Policing Initiative.

Argument. He calls the wish to predict “good” and “bad” behaviour scientifically “seductive” and a “slippery slope”, and he traces it through history: - Phrenology (which he misdates to the 17th century). - Lombroso. - Eugenics, which he ties to the Holocaust. - Small studies claiming criminals can be picked out from faces. - fMRI prediction studies (knowing vs. reckless criminal intent; a sunscreen study where scans predicted behaviour better than the subjects themselves could). - Machine-learning classifiers of criminality from headshots. - Palantir and predictive policing.

His claims: 1. Media hype. Headlines “vastly overstepped” the research. 2. Bias in the science. “It’s not so much that we can collect data” that is the problem, but “how we decide what data to collect, and how we end up interpreting and using it”. 3. Science corrects itself, but slowly (“sometimes decades or centuries”). Until it does, it is “deeply susceptible to human foibles”, especially where evidence is uncertain, samples are small and results do not replicate. 4. “Bad” is defined by norms, not by morality. Laws codify social norms, not absolute moral standards; his example is UK laws against homosexuality until 1967. 5. Pre-justice. Acting before a crime erodes the presumption of innocence and denies people “agency over their destiny”. 6. Chaos and complexity (drawn from his Jurassic Park chapter) mean behaviour can be bounded but never predicted with certainty. 7. Algorithmic bias puts developers’ prejudices into an “artificial judge and jury”. 8. Opacity. We have “trained computers to do our thinking for us” and “no longer know how they’re thinking”. 9. Predictive policing risks targeting “black, brown, and poor” communities because the systems were trained to expect crime there.

How firmly. Strongly sceptical (“I am highly skeptical that we can gain anything of value here”). Even so, he grants that data-driven crime prevention has real benefits (“they are many”) and says the Smart Policing Initiative “makes a lot of sense”. His objection is to irresponsible use and to the premise of the research, not to data as such.

Concepts. - The slippery slope. - Pre-justice. - Algorithmic bias. - The black-box irony: we use knowledge of the human brain to build “artificial brains” we do not understand. - Science’s self-correction is slow. - Normative expectations vs. moral value. - Determinism vs. free will. - Bounded predictability (Mandelbrot’s fractal as an image).

Analogies. - Structural: phrenology, Lombroso and eugenics are a line of pseudoscience that he explicitly extends to fMRI and machine-learning prediction. It is a lineage of ideas and motives, not of technologies. fMRI is “the high-tech version of “looks like a criminal””. - Fiction: the Minority Report precogs, which have been replaced by “massive data sets and AI algorithms” with “remarkably similar” intent.

AI. - What it is: machine learning trained on human-chosen data. It carries its makers’ biases and is opaque even to them. - What is new: the move “into the world of big data and autonomous machines”, where human judgment is handed over to systems whose reasoning cannot be inspected. - The risks: discrimination, erosion of justice and agency, and self-deception at institutional scale.

Companies and leaders. - Palantir, described fairly neutrally, with the concerns channelled through the Stop LAPD Spying Coalition. - Researchers who retreat to “pure academic discussions” after a backlash. He treats their conclusions as implying machine classification of criminality whatever they later disclaimed.

Governance. Mostly implicit: due diligence, responsible use, and “wield this tremendously powerful technology responsibly and humanely”.

Cognition and formation. - Neuroscience challenges the self, free will and the “soul” (“merely an illusion of our biology”). - He worries “neuroscience is racing ahead of our ability to cope with what it reveals”. - The sunscreen study: researchers “knew their subjects’ minds better than they did”. This foreshadows later worries about systems knowing people better than they know themselves.

Quotes. - “it’s how we decide what data to collect, and how we end up interpreting and using it, that’s the issue.” - “this self-correcting nature of science takes time, sometimes decades or centuries.” - “we are using our increasing understanding of how the human brain works to develop and train artificial brains that we are increasingly ignorant of the inner workings of.” - “our unfailing ability to delude ourselves in the face of evidence to the contrary”

(In the corpus text some of these words are italicised, e.g. can, what, how; the italics are dropped here.)


2020-10-15 — the-ethics-of-advanced-brain-machine-interfaces-and-why-they-matter-fdd77aafc376#

The ethics of advanced brain machine interfaces — and why they matter

Provenance. His own prose, in two parts: 1. A short blog introduction (College of Global Futures). 2. The prepared remarks he gave to the National Academies (NASEM) Committee on Science, Technology and the Law on 2 October 2020.

The risk innovation and orphan-risk tools, and the 2019 JMIR Neuralink paper (doi 10.2196/16194), were developed with his ASU Risk Innovation team (“our work”, “we published”).

Argument. - Neuralink and similar ventures raise “ethical and governance challenges that we, as a society, are poorly equipped to navigate”. He is “more convinced than ever” of this after the 2020 Neuralink demo. - Some experts say “there’s nothing really new here” in the science or the ethics. He answers that “they’re misguided”. - The turning point is not new science. It is “a synergistic scaling of ability, accessibility, and use”. - It is driven by a “Silicon Valley flavor of entrepreneurialism”. That culture overtakes the “slow, careful development” of the medical community and deliberately exploits the gaps between disciplines and in the mindsets of academics, lawyers and policy makers. - Medical neuroethics already handles clinical risks. What worries him is what “a blinkered perspective on medical neuroethics” will miss: enhancement uses, which is what Neuralink’s own staff talked about (cognition, memory, gaming, telepathy, symbiosis with AI, all through a smartphone app that can write to the brain as well as read it). - He thinks most of these aims are unlikely to come true. That is “not the point”: entrepreneurs “are in the process of reimagining what is possible” and are “warping the pathway” to the future.

Five governance questions he poses: 1. Who is responsible for the ethics of enhancement devices that make no medical claims? 2. How will widespread use change social norms? 3. How do we avoid “technological equivalent of indentured servitude”? 4. How should we treat uses that mimic or go beyond illegal drugs? 5. How do we handle apps that change a user’s mind-state “without them having full autonomy”, especially when combined with machine learning?

His proposed approach: risk innovation. - Risk is treated as “a threat to value” (citing his 2015 Nature Nanotechnology piece). - Value is defined broadly: to the enterprise, its investors, consumers and communities, and including “social value and personal value as well as economic value”, such as “equity, agency, and a sense of self”. - Orphan risks: “hard to quantify threats to value that often slip between the cracks of conventional risk approaches”. - For Neuralink, the orphan risks he flags are: perceptions of how the technology will be used, social justice and equity, loss of agency, organisational values and culture, and ethical development and use. - Few frameworks let these be handled “in agile and responsive ways”. “new, agile approaches to technology governance and ethics are desperately needed”.

Change of view (explicit). Looking back at the 2019 paper, “we were somewhat naïve” to assume Neuralink cared more about medical uses than enhancement in the long term.

The Tesla analogy (structural). Tesla “is not a car company, but a tech company committed to changing our future on their terms”. Tesla is about mapping and monetising lives, and about “the fulcrum of converging technologies” combined with “the lever of imagination”. Neuralink will do the same, with “our brains and our minds” as “the platform for their corporate future-building”.

Analogies. - Literal: established medical neuroethics (cochlear implants, deep-brain stimulation, transcranial direct current and magnetic stimulation, the Utah array). - Structural: Tesla and SpaceX as models of how outsiders disrupt a field.

AI. Appears as “symbiosis with artificial intelligence” and as machine learning built into BMIs that could change mind-states without the user’s full autonomy. This connects AI to manipulation and to loss of agency over one’s own mind.

Companies and leaders. - A detailed, fairly admiring but wary portrait of Musk and the entrepreneurial mindset. They do “not care for established ways of doing things” and actively exploit them. This is framed as a governance problem, not as villainy. - His risk-innovation work is for entrepreneurs: it helps them make early decisions that lead to responsible outcomes.

Governance and who decides. - Society must build ethics and responsible development into thinking about global futures. Otherwise companies change the future “on their terms”. - Advisory bodies (NASEM) are the audience here. - He calls for agile governance, not heavy regulation.

Quotes. - “a synergistic scaling of ability, accessibility, and use, that has the potential to profoundly rewrite the landscape” - “These are hard to quantify threats to value that often slip between the cracks of conventional risk approaches” - “we were somewhat naïve in assuming that transformational medical interventions were a higher priority in the long-term for the company than enhancement technologies.” - “Tesla is not a car company, but a tech company committed to changing our future on their terms.”


2020-11-05 — risk-innovation-and-the-future#

Why Risk Innovation is critical to the futures we aspire to

Provenance. His own prose. It is institutional: it marks the end of the seed-funded phase of ASU’s Risk Innovation Nexus and links to the Nexus’s tools and reports, which were made by his team.

Argument. - Risk cannot be avoided “in a universe where past “causes” connect in complex and often unpredictable ways with future “effects,””. - Every action is “detrimental to someone in some way”. - Complexity, interconnection and technological power “vastly amplify” uncertainty. - So “outmoded ideas about risk” become “a risk in themselves”.

History of the Nexus. - Seeds: work with entrepreneurial students at the University of Michigan, who needed “a completely new risk toolkit” for social and political risks. - 2016: ASU Idea Enterprise support. - 2017: Risk Innovation Accelerator. - 2019: renamed the Risk Innovation Nexus, working “at the nexus of entrepreneurship and social value creation”.

The complexity argument. Complex, non-linear systems look healthy until “seemingly-insignificant and often overlooked events” lead to catastrophic failure. Resilience therefore depends on spotting risks that are hard to quantify and so tend to be discounted.

Concepts. - Risk innovation. - The risk innovation mindset, “not just for entrepreneurs, but for anyone striving to build a better future”. - Orphan risks: “hard to quantify and easy to ignore risks that nevertheless have a habit of coming back to bite”. - Resilience. - A “win-win for them and society more broadly”.

How firmly. Emphatic (“I cannot emphasize enough”), while admitting his bias as the Nexus’s founder.

Analogies. None. The frame is general systems and complexity.

AI. Not mentioned.

Governance. Governance happens through the decision-making of innovators and entrepreneurs, who are given tools. It is not regulation.

Quotes. - “our outmoded ideas about risk actually become a risk in themselves and threaten the future we aspire to.” - “those hard to quantify and easy to ignore risks that nevertheless have a habit of coming back to bite.” - “they look as if they are thriving, until seemingly-insignificant and often overlooked events lead catastrophic failures.” (sic, “lead” without “to”)


2020-11-12 — is-artificial-intelligence-going-to-kill-us-all-6ae9d059c40d#

Is Artificial Intelligence Going to Kill Us All?

Provenance. His own prose (College of Global Futures blog). It introduces an ASU YouTube playlist on AI risk that he curated. The videos, including two by Joy Buolamwini, are other people’s work and are not in the text. It is short, but it is his only explicit statement on AI risk in this batch.

Argument. Future-building is hard. We have never had more capacity to build a better future, or more ways to destroy it. He sorts threats into three groups: 1. Planetary threats such as climate change, pollution and loss of biodiversity: “the charismatic megafauna of the global threats world”, rooted in “our myopic profligacy as a species”. 2. Persistent social justice and equity problems. 3. Threats from powerful technologies that could “rob us the futures we aspire to” (“rob us the” is sic).

AI sits “on a knife edge” between great benefit and possible catastrophe. He names the existential-risk voices (Musk, Hawking, Bostrom’s Superintelligence) and calls his own title “admittedly click-baity”. His main claim is that AI’s risks are “often far more mundane–but no less serious for this”. They involve “often-subjective but desperately important areas like autonomy, justice, equity, and our ability to have control over our lives”.

The playlist covers: - Bostrom’s concerns and hopes about superintelligence. - Videos that “debunk some of the myths” around superintelligence. - Algorithmic bias and facial recognition.

The goal is benefit “for everyone, not just a privileged few”.

How firmly. Measured. He does not dismiss existential fears but moves the weight to social risks that are harder to grasp. He offers no solutions (“won’t provide any easy solutions”) and aims to “frame the questions”.

Concepts. - Mundane-but-serious AI risk vs. existential AI risk. - AI as a knife-edge technology. - Future-building. - A taxonomy of global threats (planetary, justice, technological).

Analogies. An implicit ranking alongside climate change and biodiversity loss, but no literal comparison.

AI. - What it is: one of the “increasingly powerful technologies we’re creating”. - The risks: autonomy, justice, equity, control over one’s life, bias, facial recognition. Superintelligence is present but treated partly as myth. - Leaders: Musk appears as a warner of risk, not as a developer.

Governance and who decides. Implicit: distributive justice (“not just a privileged few”) and public education (the playlist).

Quotes. - “the risks of AI are often far more mundane–but no less serious for this.” - “technologies that sit on a knife edge between incredible benefits, and potentially catastrophic failure.” - “for everyone, not just a privileged few.”


2020-12-10 — we-need-to-rethink-our-relationship-with-the-future-9fe0f60ecdb3#

We need to rethink our relationship with the future

Provenance. His own prose (College of Global Futures blog). It partly promotes Future Rising.

Argument. - 2020’s “perfect storm” of political turmoil, injustice and pandemic shows “deeper tensions between our collective ability to influence and change the future, and our capacity to do this effectively”. - The fix is to rethink “our relationship with the future and our responsibility to it”, closing the gap between ability and responsibility. - His example: “Climate change, driven by our technological recklessness”, together with the everyday injustices that come from poorly considered decisions and “technologies that cause more harm than good”. - We can recode DNA, design materials atom by atom and “create machines that may one day surpass human intelligence”, yet we are “still remarkably adept at preventing all too many people from reaching the futures they aspire to”. - Future Rising presents humans as “architects of the future”. The same abilities have a “dark side” because they “enable us to rob others of the futures they aspire to”. - The way forward is to understand the “intertwined threads” of imagination, understanding, inventiveness and “our very humanity” (beliefs, desires, irrationalities, empathy) so that we become better architects.

Concepts. - Relationship with, and responsibility to, the future. - The gap between ability and responsibility. - Humans as architects of the future. - Robbing others of their futures, a justice framing he repeats often in this period.

Analogies. Climate change as the example of technological recklessness (structural).

AI. A single line: machines “that may one day surpass human intelligence”, listed as one of our growing capabilities. He does not dismiss superhuman AI as impossible.

Governance. Collective and individual responsibility. No institutions are named.

Being human. Humanity, with its flaws, irrationality and empathy, is both the source of the ability to build futures and the key to building them well.

Quotes. - “Climate change, driven by our technological recklessness is, of course, a stark reminder of this.” - “we are still remarkably adept at preventing all too many people from reaching the futures they aspire to.” - “these selfsame abilities enable us to rob others of the futures they aspire to.”


2020-12-15 — why-trustworthiness-matters-in-building-global-futures-50a91fcb9bb2#

Why Trustworthiness Matters in Building Global Futures

Provenance. His own framing and commentary (College of Global Futures blog). The three key findings, the seven drivers of trust and the “five more things to know about trust” are reproduced or summarised from the TIGTech report (Trust in Technology Governance, supported by the WEF and Fraunhofer ISI). They are not his formulations, although he sits on TIGTech’s advisory panel.

Argument. - COVID vaccine hesitancy is “easy to dismiss” as irrational, anti-science or misinformation-driven. It actually points to a bigger problem of trust in how science and technology are governed, and in “how organizations earn trust through being trustworthy”. - Technologies “are only as good as the trust that people place in the organizations that develop and use them”. - He praises the report for its plain language and for being actionable, and says it should be required reading for anyone building futures. - He applies the ideas broadly: to science communication, to justice, equity, diversity and inclusion, and to universities. - He stresses the report’s view of trust as a spectrum that is “dynamic, messy, personal, and a two-way process” and that is not achieved by following rules or ticking boxes.

How firmly. Strong endorsement.

Concepts. - Trust vs. trustworthiness (trust is earned). - Public-interest governance. - “Detach governance from hype and ideology” (TIGTech’s wording, which he endorses). - “Get comfortable with navigating ethics and values” (TIGTech).

Expertise and publics. He resists the deficit reading of public resistance. Institutions, not the public, carry the burden of being trustworthy.

Governance. Engaged, collaborative, communicative and inclusive governance of emerging technologies.

AI. Not mentioned.

Quotes. - “It’s easy to dismiss this resistance to the COVID vaccine as irrational thinking” - “they are only as good as the trust that people place in the organizations that develop and use them.” - “It takes awareness, empathy and humility, and a willingness to embrace the messiness of being human within a complex society”


2021-01-15 — can-watching-sci-fi-movies-lead-to-more-responsible-and-ethical-innovation-7c993bdaa5c2#

Can watching sci-fi movies lead to more responsible and ethical innovation?

Provenance. His own prose (College of Global Futures blog), about his ASU course The Moviegoer’s Guide to the Future, which he has taught since 2018.

Argument. - Watching science fiction films together as a class creates “a creative space” for thinking about socially responsible and ethical innovation. It jolts students “out of the ruts of conventional thinking” and is “a space where everyone has something to contribute” across disciplines. - FFTF was written partly to move the course content out of the classroom. But “book-learning” only goes so far: “transformative learning has to be felt”. - Entertainment is an underrated route to learning. Done well, it opens up new ways of seeing and helps people engage “more openly, honestly and empathetically”. - The move to Zoom brought unexpected advantages: text chat during films, and asynchronous discussion on Yellowdig that works like social media.

Concepts. - Entertainment-catalysed learning. - Film as a shared creative space. - Learning that engages the heart as well as the mind. - “future-builders”. - Transdisciplinary responsible innovation.

Education. An early statement of his teaching philosophy: experiential, emotional and communal learning. He is optimistic about digital platforms for learning together, and in this period he is not worried about technology-mediated learning.

AI. Not mentioned.

Quotes. - “transformative learning has to be felt, to be experienced, and to involve engaging the heart as well as the mind.” - “a space where everyone has something to contribute”


2021-02-25 — how-our-mastery-of-biological-physical-and-cyber-base-code-is-transforming-how-we-think-about-b2eae9d589d0#

How our mastery of biological, physical and cyber “base code” is transforming how we think about…

Provenance. His own prose throughout. The 2021 introduction and “Base Code Coda” frame a lightly edited excerpt from FFTF chapter 9 (Transcendence). The coda is the part with new thinking.

Argument (2021 framing). - He is wary of “generational exceptionalism”. Even so, he believes we are near a real turning point in human history. - The cause is not climate change, overpopulation or resource abuse. It is our growing mastery of “base code”: the bits of cyberspace, the DNA bases of biology, and the atoms and molecules of materials. - Mastering base code lets us invent materials, biologies and systems “from scratch”, which is “a distinct break from our evolutionary heritage”. - It also raises the “very real and extremely scary possibility” that we end up “bricking” the world we live in. Tinkering with the “operating system” of reality has no re-install option.

The 2018 excerpt. - Schwab’s Fourth Industrial Revolution. - Roco and Bainbridge’s convergence of nano, bio, info and cognitive technologies (NBIC). - Three base codes, and “cross-coding” between them. Examples: Drew Endy’s synthetic biology, BioBricks and “black-boxing”, iGEM teams, DNA read into cyberspace, edited, and written back out as mail-order DNA, and “training AI-based systems how to code using DNA”. - Conclusion: there is “a greater likelihood than ever of us making serious and irreversible mistakes”, and we need to understand the impacts “before it’s too late”.

Coda (new in 2021). He qualifies the opening claim: - Climate and pollution crises matter greatly. But they “are a result of our use and misuse of relatively crude technologies”. Base-code mastery will make things “much more challenging”. - He admits “I’m being a little hubristic here”. Biology is non-linear and epigenetic, and “this is precisely where the ability to tinker without understanding becomes an increasingly serious liability”. - Convergent technologies “have a habit of transcending our assumptions and expectations”, especially with AI in the mix. - A new extension: people, communities and norms write the “code book”. So “maybe we need to extend the concept to social norms and trends, behaviors, and even ideas”. - Closing image: a planetary “Blue Screen of Death”.

How firmly. A strong claim (“I believe that we truly are approaching a pivot point”), hedged by self-aware notes on hubris and complexity.

Concepts. - Base code (bits, bases, atoms). - Cross-coding / “trans coding”. - Technological convergence, NBIC and the Fourth Industrial Revolution. - Bricking the world. - Tinkering without understanding. - Irreversibility. - Social base code (norms, behaviours, ideas). - Generational exceptionalism.

Analogies. - Structural: digital code and operating systems as a template for biology and materials. - Structural: climate change and pollution as what “crude” technologies already did, used to argue that more powerful base-code technologies will pose greater challenges. - Nanotechnology and biotechnology are literal members of the convergence, but their risks are discussed only in general terms here. There is no comparison with nanomaterial or chemical hazards.

AI. - An amplifier within convergence (“increasingly sophisticated technologies like artificial intelligence”), including AI that writes DNA. - Cyberspace is the one domain where “we have the power to write and edit the code that ultimately defines everything”, yet “We may not always be able to determine or understand the full implications”.

Cognition and formation. Extending “base code” to “ideas” and “social norms” points toward treating human thought and culture as something that can be written and rewritten. It is only a seed here.

Quotes. - “when it comes to tinkering with the “operating system” of reality, there’s not likely to be a re-install option if we get things wrong!” - “this is precisely where the ability to tinker without understanding becomes an increasingly serious liability.” - “are a result of our use and misuse of relatively crude technologies.” - “maybe we need to extend the concept to social norms and trends, behaviors, and even ideas.”


Low / none#