Late Lessons, Jensen Huang and AI
Markdown version

B09 perspective notes: 2023-05-15 to 2023-07-21 (18 posts)#

These notes read the batch for how Maynard thinks, not for the concepts he names. All 17 posts with text were read in full. Quotes are exact, including his typos and curly punctuation.

Evidence rules applied

Context. The batch covers the weeks when AI governance became a mainstream public argument: Schmidt on Meet the Press, the Senate Judiciary hearing with Altman, Montgomery and Marcus, Bengio’s “rogue AI” essay, the Center for AI Safety extinction statement, the EU Parliament’s amendments to the AI Act, the “Frontier AI Regulation” paper and Jeremy Howard’s reply, and the x.AI launch. He responds to almost all of them within days. In June he takes a holiday and leaves readers books, a podcast and a playlist. In July he launches a podcast made from his old book readings and publishes his ChatGPT-built course. The batch shows what he does when a debate he has spent twenty years preparing for suddenly arrives in public, framed by people from outside his field.

1. How he thinks here#

He opens with a live moment, then looks for what is missing#

Nearly every governance post starts with something that has just happened: a clip “doing the rounds this weekend” (2023-05-15), a hearing “Yesterday” (2023-05-17), a statement that “caused something of a stir” (2023-05-31), two papers “Over the past few days” (2023-07-12). The next move is nearly always to ask what the event leaves out.

He reads a debate by looking for the gaps as much as the claims. This is how he brings in perspectives that a debate framed by technologists leaves out.

He tests things himself instead of taking a claim on trust#

He reasons from how the technology works#

In the predictive-policing post he moves from the mechanism of an LLM to what it might become. “the basis of an LLM is not language per se” but tokens. So what if the tokens were “data on human characteristics, nature, and behavior”? That is a first-principles plausibility test, and he labels it honestly: “a long stretch from where LLMs currently are”. He also says why he still takes it seriously: “But I have a horrible feeling that this isn’t the case.”

The Bengio post runs the same kind of test in the other direction. He doubts that a rogue AI follows as soon as the principles are known, because “there’s quite a large leap between understanding how something might be done, and actually achieving it”, and “technology innovation is iterative and messy”. The engineer’s feel for how technologies actually come into being lowers his estimate of the risk. It does not raise it.

He reframes by finding the hidden assumption in a frame#

This is the most characteristic move in the batch. He takes someone else’s framing seriously, finds the assumption inside it, and turns that assumption into a new question.

He weighs rival readings openly before choosing#

On industry asking to be regulated, he gives two readings side by side. It may be this generation’s “atomic technologies moment”, or “Or maybe it’s just a cynical move”. He then judges: “In this case I don’t think this is happening — at least, not yet.” (2023-05-17). On open source he sets out the two philosophies behind the positions. One believes “in the combined power of collective human action to creatively solve problems”. The other holds that “society in general cannot be trusted with full access to powerful knowledge and capabilities”. He places himself “between these two papers” (2023-07-12). He shows readers his reasoning, not only his verdict.

He uses analogy by structure, drawn from his own experience#

Nanotechnology is his main analogy, and he uses it for its structure, not its surface. It is the same “leave it to the technical experts” pattern, and the same escape “by engaging early and often across a very broad range of domains and expertise” (2023-05-15). “Sound familiar?” He is also clear where the analogy stops: the difference is that with nano “enough people recognized early on” the need for engagement. The debate over a nano regulatory agency tells him how an AI agency debate will go (2023-05-17). The “atomic technologies moment” is offered as one possible reading, not a claim.

Curiosity, play and serendipity drive the thinking#

Stories and films as moral instruments#

The show notes say what the films are for. Never Let Me Go is “the one that haunts me the most as it reveals the moral risks of turning a blind eye to the evils of a technology’s use because the perceived benefits are so great” (2023-07-21). Jurassic Park offers “a surprisingly nuanced perspective to the pitfalls of full-on no holds barred innovation”, with lessons for “a “go fast and break things” approach to artificial intelligence” (2023-07-14). The book “uses movies as a way to open up conversations” (2023-07-07). He uses science fiction the same way in the analytical posts. Wireheading is traced back to its sci-fi roots and judged “a science fiction vision that may turn out to be very naive” (2023-05-25). Organoids face “too many sci-fi tropes where brains grown in vats do not turn out well” (2023-05-30). He treats stories as evidence of how people will feel, and as a check on naive futures. He does not treat them as forecasts.

2. What matters to him#

Everyone affected has a right to a say#

The strongest value statement in the batch: “when developing a technology that has the power to impact everyone in profound ways, everyone has the right to play some role its development and use” (2023-05-15). He grounds this in rights and in what people value, not in technical knowledge: “people don’t need to understand the inner workings of AI to discuss and explore how it might potentially impact their lives and threaten what’s important to them.” Here “threat to value” appears in everyday form as the basis for democratic participation. People are qualified to take part because they know what matters to them.

Suspicion of small groups deciding for everyone#

This concern links the governance posts, the Bengio post and the Musk post:

History gives him his sharpest line. People have “propagated evil on the back of an absolute certainty that the solution to value alignment is to rob others of their right to hold values that don’t align with theirs” (2023-05-25). His commitment is to pluralism of values, and he sees value alignment done by a narrow group as a risk in itself.

Human dignity and self-determination#

Predictive policing is the one place where he states a firm moral line: “Here I should lay my cards on the table”. It would violate human rights “especially around dignity, equality, and self-determination” (2023-05-22). His shock is personal: “what I discovered shook me”, and “(and I cannot believe that people are still researching this)”. Even so, he keeps the other side in view: “the use of AI in managing crime is far from black and white”.

The benefits are real, and losing them is also a harm#

He is not a critic of AI as such. Education is where the benefit is most vivid to him: “Here, the potential is profound” (2023-07-10). AI could transform learning “at a scale that probably hasn’t been since the invention of the printing press … that’s a thrilling prospect” (2023-07-16). His worry about the EU Act is that it could block personalised learning. Regulation should support creative use “rather than stifling it”. In the extinction post, the loss of AI’s possible solutions to climate change, poverty and threats to democracy counts as a potential catastrophic loss too.

Learners and educators#

He defends educators against “leading technologists and thought leaders, and all of them well-meaning” who think teaching is still rote learning and policing cheating. Most educators are “light years away from these very old fashioned ideas” (2023-07-10). He cares about learning “as a journey”, about “supporting their health and wellbeing” (educators’), and about AI literacy “not just for engineers and computer scientists, but in the arts, humanities, social sciences, and beyond”.

Humanity as part of something larger#

The extinction frame is “too human-centric”. “we are, after all, an integral part of a larger set of interconnected ecosystems” (2023-05-31). He also warns against “the cognitive traps inherent in human exceptionalism, power dynamics, and control” (2023-05-25), and hopes to “co-create a shared future” with AI rather than “control and contain a creation that we feel threatened by”. He cares about relationship, including a possible relationship with machines that have agency.

What frustrates him#

What delights him#

Audacious visions (“I can’t help but be inspired sometimes by the audacity of Elon Musk’s vision”), physics (“As a physicist and generally curious person, I love the idea of working with intelligent machines to unravel the mysteries of the universe.”), clever technical possibilities in organoids, a constructive bipartisan hearing (“not everything in the US government is broken”), his colleagues on the WEF steering group, and his course (“I’m loving it!”). Delight and critique usually sit in the same post. The Musk post moves from inspiration to a sharp critique and back to “A Soupçon of Optimism”.

3. Risk as a way of thinking#

This batch holds the clearest early statements of the risk ideas he now foregrounds. The terms “risk innovation” and “threat to value” appear by name. “Risk-benefit landscape” and “navigate” are both used. “Orphan risk” does not appear.

A novel technology needs a new mindset: stated outright in 2023#

The extinction post is the most direct match in the batch to his September 2026 account of himself. He describes conventional risk categories (“educational challenges, job displacement, automation, privacy”) as “important” but “the shavings off the tip of the AI iceberg”. They “run the danger assuming that the risks associated with a highly unconventional technology transition like the one represented by AI can be sliced, diced, and solved, using a conventional mindset.” “Based on all my years of working on the risks and benefits of emerging technologies, this strikes me as being extremely naive.” (2023-05-31)

He then describes what the conventional frame misses: “possible threats to social, economic, and political structures, systems, and norms”. These include “the ways we individually and collectively build our understanding of the world we live in” and “the seductive mastery of language” shown by LLMs. It is “a highly non-linear technology transition that cuts across highly integrated complex systems, where it’s the seemingly trivial and unexpected that are likely to lead to potentially catastrophic outcomes.” He asks for “failure modes that are as far from conventional as you can get — alongside approaches to navigating them that are equally unconventional.”

The frontier-AI coda says the same thing more briefly. Both papers deal with risks that “don’t fit neatly into any established risk management paradigm”, and “This is precisely the challenge that led to us working on the concept of “risk innovation”” (2023-07-12). The opening of that post notes a growing sense that AI “will demand radically new thinking around governance and regulation.”

Threat to value#

He gives the definition twice, and each time it widens what can be seen.

How it works: a frame that opens possibilities#

What the concept does in these posts matters more than its label:

  1. It lets benefits into the same account as risks. “By framing catastrophic risk as the potential loss of something (or things) of value at scale, we can also consider the potential loss of solutions to pressing challenges”. “AI as solution has to be part of the discourse around AI and risk.” (2023-05-31). Under a threat-to-value frame, failing to develop a beneficial technology is also a loss. The extinction frame cannot show this.
  2. It moves the question from whether the AI obeys to what people stand to lose (2023-05-25). This loosens the control-and-subservience framing he distrusts.
  3. It widens the range of harms to include dignity, belief and self-identity. Hazard-based assessment does not reach these.
  4. He says what the frame is for. “there needs to be a framing of AI risks and benefits that opens up new possibilities rather than closing down conversations” (2023-05-31). This is his clearest statement in the batch that risk concepts are thinking tools that open the space of options.

In 2023 he also calls the risk innovation nexus “a methodology for navigating complex and societally-coupled risks” and a “framework” he will apply to AI (“stay tuned!”) (2023-07-12). So he presents it partly as a method. But the work it does in these posts is reframing, not procedure.

Risk-benefit landscape and navigating#

“Framing the potential risks of AI in this way highlights the deep complexity of the risk-benefit landscape we are facing, and the new thinking that will be needed in order to successfully navigate it.” (2023-05-31). “Navigate” appears about fourteen times in his own prose across the batch. Examples:

The show notes speak of “navigating a deeply complex tech innovation landscape where there few right and wrong answers” (2023-07-07).

“Manage” appears in his own voice mostly to describe the conventional frame the new risks do not fit: an “established risk management paradigm”, and Howard’s “tangible cause and effect-based risk management” (2023-07-12). The contrast is implicit but consistent. Management belongs to known, bounded risks. Navigation belongs to a landscape where the destination and the hazards are both uncertain.

Risk science built on, not thrown out#

He keeps the vocabulary and tools of conventional risk analysis and puts them to work inside the broader frame:

He extends the standard definition of catastrophic risk. He does not replace it.

Humility against false precision#

The predictive-policing post turns the danger of false precision into a risk finding. The worry is not accuracy, but that LLMs “will provide users with the illusion that they can” predict. The bar may become “predictions that are authoritative rather than accurate”. The output would be “while not reliable, will be easy to use and, above all, persuasive” (2023-05-22). A plausible-sounding number is itself the hazard. His own stance is the opposite:

He pairs humility with caution and with confidence in human ingenuity: “where the stakes are high and the future is uncertain, it pays to be cautious”, yet “people are incredibly good at finding innovative solutions to complex challenges on the fly” (2023-05-31).

Risks outside the usual categories#

“Orphan risk” is not used, but the risks he points to mostly fall outside conventional categories:

Curiosity as both value and risk#

He is careful about his own founding value. “I’m not sure there is a strong causal link between curiosity and benevolence.” He adds: “How many scientists have slipped into ethically reprehensible behavior, simply because the moral guardrails around their “what if?” questions were inadequate to contain the extent of their curiosity?” (2023-07-19). Curiosity drives his work, and he still says it needs moral grounding and other people. This is not a naive celebration of curiosity.

4. Scholarship and public writing#

The posts carry a career’s work#

Each governance post draws on direct experience:

Public writing is where this scholarship meets the moment. It is not a popularised version of scholarship done elsewhere.

Scholarship done in public, and disclosed#

He gave input to Howard’s red-team paper and says so. He then writes about both papers fairly, calling both “landmark documents”. He posts his course materials openly: “Please do share — these are designed to be used!” (2023-06-16). He publishes his Slate draft before the editor’s changes, because “early drafts often include insights and perspectives that don’t make the final cut, and are worth reading despite their often-raggedy edges”, and because it is “interesting to see the evolution of a piece” (2023-07-16). He shows the working as well as the product.

Writing in real time, provisionally#

He writes “while they are fresh in my mind” (2023-05-17) and marks his conclusions as provisional: “my current thinking lies between these two papers” (2023-07-12), and “It’s going to be interesting to see how this plays out.” (2023-07-10). The posts are thinking in progress, offered as such.

Evidence and expertise#

He respects technical expertise (“In other words, he knows his stuff.”, on Bengio) and treats it as one kind among several. Industry lacks “sophisticated approaches to the governance of emerging technologies” (2023-05-15). Early frontier-AI debates were “dominated by people who were experts in AI, but who were somewhat light on their expertise in governing emerging technologies successfully” (2023-07-12). He calls for regulation “guided by people who know about regulation and science and technology policy” (2023-05-17). He also counts lived experience and values as expertise that can “extend the range of development and governance options beyond the limited understanding of self-proclaimed experts” (2023-05-15). He sends readers to primary sources: read the Bengio essay, read the organoid paper, read both frontier papers “carefully”.

Transdisciplinary by habit#

He moves easily between physics (determinism and “ultimate truths”), neuroscience (organoids), law (EU amendments), education, theology (Adam and Eve), science fiction (Banks, Stephenson’s “illustrated primer”, Douglas Adams), and music (the playlist). Films from the Future is about weaving insights “including the arts and humanities” (2023-07-07). AI literacy should be taught in “the arts, humanities, social sciences, and beyond” (2023-07-10).

Accessibility#

He explains wireheading, brain organoids and EU legal structure in plain terms. The show notes state the aim directly: to write “as inclusive and accessible a way as I knew how, with the aim of taking readers on a compelling journey into the future where their thoughts and ideas were just as important as mine” (2023-07-07). He turned the audiobook into a free podcast partly to “give students an alternative to purchasing the book” (2023-07-03).

5. His role as he sees it#

A navigator of technology transitions, adding missing expertise#

He places himself as someone with long experience of “navigating exceptionally complex technology transitions” (2023-05-15), bringing a kind of expertise the AI debate lacks. He does not claim this as a new gatekeeping role. He argues for many kinds of expertise, and for the public, alongside his own. The “team effort” of nanotechnology is his model.

Fair and generous with people he disagrees with#

His pattern is to grant the strongest version of the other view, name exactly where he parts company, and look for common ground.

A public dissent with reasons#

“Why the recent statement on the risk of extinction from AI is important, and why I didn’t sign it” (2023-05-31) is a public decision explained in public. He supports the priority and rejects the framing. This is how he takes a position: on the substance, not by joining a camp.

Between camps, not above them#

He contributed to Howard’s open-source red-team paper and praises the frontier paper’s multi-stakeholder turn. He maps the two philosophies, one trusting collective human action and the other favouring technocratic control, and places himself between them. He calls the problem “gnarly” rather than wicked, and concludes “we’re going to have to hash this out together to find solutions that work” (2023-07-12).

With educators, students and readers#

Willing to be changed by what he studies#

He admits the ChatGPT course outline was “better than I could have produced on my own in the time” and calls it “a profound wakeup call” (2023-07-16). He jokes: “It’s a case of AI as mentor and instructor that has me wondering when I’ll be out of a job!” He says professors must “acquire new skillsets and to recalibrate how we think about learning and education”. He also writes: “It’s almost as if ChatGPT is fine tuning my brain to be a better instructor … And messed up as this sounds, maybe it’s a necessary step”. He names the strangeness of being changed and does not pretend it away. He also insists on the “human in the loop” and on the objectives ChatGPT missed, including “the broader societal implications”.

What he refuses to do#

Changes of mind and revisions#

6. What is distinctive#

  1. Refusing the extinction frame while taking the concern seriously, and saying why. In the week the CAIS statement set the terms of debate, he took a third position. He neither signed nor dismissed it. He moved from extinction to catastrophic loss of value, which puts ecosystems, dignity, autonomy and the risks of not developing AI into one account. Few in May 2023 argued that the extinction framing was too narrow and too human-centric, or that it closed conversations the risk question needed to keep open.

  2. Saying outright that conventional risk thinking cannot handle AI, from inside risk science. “sliced, diced, and solved, using a conventional mindset … extremely naive” (2023-05-31) comes from someone who has spent a career in risk analysis. It is not a rejection of risk science. It says that a non-linear technology crossing tightly coupled social systems needs a different mindset, and names the risks that live there: to sense-making, norms and democratic processes.

  3. Finding the danger in persuasive illusion, not accuracy. The predictive-policing experiment moves the risk from “can it predict?” to “will people act on authoritative-sounding output anyway?” Most critiques at the time dealt with bias and accuracy. He argues that plausibility plus ease of use is the hazard, and that only guardrails stand between current capability and misuse.

  4. Turning the alignment question around. He questions “rogue” as an obedience frame, warns that the real fear may be that AI “will look too much like us”, treats narrow value alignment as a threat to pluralism, and proposes asking what it means to be an intelligent machine in a world with humans. This moves from control to co-existence and co-creation. It is unusual in both the safety and the ethics literatures of the time.

  5. A curious person’s critique of curiosity. A thinker whose work rests on curiosity argues that curiosity is not benevolence and needs moral guardrails and plural truths. That gives his defence of curiosity and play more weight: it is not naive.

  6. Governance for transitions, not technologies. He proposes a cross-agency capacity for “advanced technology transitions writ large” rather than an AI agency, and he doubts the technology-versus-use distinction for general-purpose AI. Both come from direct experience of the nano governance debates, which few voices in the 2023 AI debate had.

  7. Scholarship, teaching and play as one practice. In one month he publishes a close legal reading of the EU Act, a course built with the technology it teaches, a free podcast made from his book, a playlist, and a reading list. It is one way of working: curious, experimental, generous with materials, and aimed at opening conversations rather than winning them.

7. The posts in this batch that best show how he thinks#

  1. 2023-05-31 existential-risks-of-ai. His most explicit statement of the new-mindset argument: conventional risk framing is “extremely naive” for an unconventional transition. It shows threat to value and the risk-benefit landscape at work, and states the purpose of reframing: to open up possibilities “rather than closing down conversations”. It is also a model of principled public dissent.
  2. 2023-05-25 leading-ai-expert-says-we-should. A charitable, step-by-step reading of a respected expert. He finds the hidden assumption in “rogue”, recasts harm as loss of value, and ends by reversing the usual question about humans and machines.
  3. 2023-05-22 can-large-language-models-be-used. Inquiry by experiment. A prompt template tested on GPT-4 leads to the finding that persuasive, authoritative-seeming prediction is the danger. His personal shock and his moral line sit alongside an acknowledgement that the issue is “far from black and white”.
  4. 2023-07-19 elon-musk-maximally-curious-agi. Delight and critique together. It draws on physics, science fiction, theology and social theory, argues that curiosity is not benevolence, and ends in hope.
  5. 2023-07-16 chatgpt-created-my-course. Experimenting in public and being changed by it. It shows open delight, the human-in-the-loop commitment, and a pre-edit draft published so readers can see the process.
  6. 2023-07-12 regulating-frontier-ai-models. The bridge-builder. He maps two philosophies fairly, holds a position between them, rejects the “wicked problem” cliché, and brings in risk innovation and threat to value for risks that “don’t fit neatly into any established risk management paradigm”.