Late Lessons, Jensen Huang and AI
Markdown version

B24 notes: 2025-04-06 to 2025-04-08 (2 posts)#

Reading notes on Andrew Maynard’s Substack posts in batch B24. Only his own prose counts as evidence. Quotes are exact, including his typos and curly punctuation.

Batch context: early April 2025. The AI 2027 scenario (Kokotajlo, Scott Alexander, Larsen, Lifland, Dean) has just been published and is widely discussed. He has just come out of “a workshop on AI and responsible innovation”. OpenAI’s GPT-4o image generator has set off the “Studio Ghibli style” controversy. The batch is about 21,100 words, but only about 1,400 of them are his own analytical prose. Almost all the rest is a report written by OpenAI’s o1-pro model, which he published inside his post.

Relevance summary:

Date Slug Relevance
2025-04-06 responsible-innovation-and-ai-acceleration high
2025-04-08 openai-and-studio-ghibli-style none (Modem Futura podcast post; skipped per user instruction)

HIGH#

2025-04-06 — responsible-innovation-and-ai-acceleration — “What does responsible innovation mean in an age of accelerating AI?”#

Provenance. Mixed. About 93% of the post’s ~20,500 words is not his. - His own prose (about 1,400 words): the title and subtitle, and the framing essay that runs from the opening (“A new speculative scenario on AI futures…”) to the download link for the PDF. This is the only evidence of his thinking in the post. - Not his (about 19,000 words): the whole report headed “Responsible Innovation and Responsible AI in an Era of Accelerated AI Development”. That covers the Executive Summary, Introduction, the sections on RRI frameworks, UK/EU responsible AI, the “stress test”, emerging responses, gaps, promising directions and conclusion, the sources, Annex A (RI in a US–China arms race) and Annex B (frontier labs, China and the EU, and the thought-leader camps). It was written by OpenAI’s o1-pro. He describes his role as “actively engaging with o1-pro in the research and writing process, and evaluating and editing the final report where necessary”, so the report was shaped and edited by him but is AI-written. Its phrasing and arguments are not evidence of his views. Examples include “structurally outmatched”, “insufficient alone”, “built into the infrastructure”, “a careful, consensus-based steering wheel to a rocket already in flight”, the IAEA- and CERN-for-AI proposals, compute KYC, tripwires, moratoria, the social-media “same mistakes” comparison, the 1975 Asilomar comparison and the nuclear control-rod analogy. Annex C, o1-pro’s assessment of the AI 2027 authors’ “backgrounds, perspectives, and controversies”, is left out of the post and appears only in the downloadable PDF. - Other non-authorial items: a link to Perplexity “Top takeaways”, a closing link to a “ChatGPT summary and further reading”, and a Midjourney header image. - What he chose to publish and how he framed it: he presents the o1-pro report as “essential reading for anyone looking for a nuanced perspective on responsible innovation/responsible AI in the light of possible rapid AI acceleration”. He says that “For anyone familiar with emerging thinking around responsible AI” the first part holds few surprises. He calls it “a very measured response” to AI 2027, useful mainly as a synthesis. He points readers to Annex A as the more useful part: how RI “might fare if we face an AI acceleration arms race between the US and China”. He answers that question in his own voice: “(not that well is the short answer)”. Annex B is recommended for “perspectives that are important but are under-represented”. He had o1-pro compile Annex C on the scenario authors’ “ideologies and perspectives” so that the analysis “can be contextualized”. So he pays attention to who is doing the forecasting, but he does not comment on the authors himself. He apologises for the length and publishes anyway because the content is “relevant enough”.

Argument in his terms. - AI 2027 is speculative but must be taken seriously. He calls it “highly speculative” and says it may sound like science fiction. It is still “sufficiently grounded in current trends and emerging capabilities to provide serious pause for thought”. He notes it “reflects at least some of the thinking of growing number of leaders and developers at the cutting edge of AI”. He reports both reactions to it: “considerable pushback for being too alarmist”, and a cautious welcome from “some big names in cutting edge AI as a salutary warning”. - He lists the scenario’s assumptions and treats them as contestable. The four he names: 1. AI companies building internal models good enough at coding to develop the next generation faster than humans, in order to dominate the market; 2. self-improving models that turn employees into “AI managers rather than AI developers”; 3. parallel advances in hardware and energy; 4. collective human behaviour that follows game theory, so that “an AI arms race is inevitable, no matter how bad an idea anyone thinks it is”.

“Each one of these has its flaws. Yet they are not unreasonable as a starting point for imagining edge case scenarios.” They lead quickly to “near-term possibilities that represent a tipping point in what the future looks like.” - The core worry: a mismatch of timescales makes responsible innovation futile. Ideas about how to govern AI and ensure its “socially responsible development and use” mostly “depend on processes that are constrained by human timescales that are rather longer than those associated with intelligent machines.” His first reaction was that “we might be facing a near term future where current efforts to develop artificial intelligence responsibly seem futile.” He worries about “the futility of matching responsible innovation processes that can take years, to a period of AI acceleration where a lag of even a month in the development cycle might mean the difference between abject failure and world domination.” - The deeper worry is about human thinking, not governance machinery. “What worries me just as much though is that nothing about how we think, how we plan for the future, or how we develop approaches to ensuring better futures, is geared toward exponential advances that happen over months rather than years.” If something like AI 2027 happened, “we would most likely fail to recognize it — or would actively deny it — until it was too late.” The root cause: “we are really bad at wrapping our heads around rapid exponential growth.” - The Bartlett beaker (one minute to midnight). He uses Al Bartlett’s 1978 thought experiment, which he met through the film Inferno (“deeply flawed but nevertheless compelling”). One bacterium dividing every minute fills a beaker between 11:00 PM and midnight, so the beaker is half full at 11:59. He accepts the physical limit: “The illustration would never work in real life as resource constraints would slow or halt the exponential growth.” Its value is as an illustration of “how hard it is for us as individuals or as a society to plan for exponential growth — especially when it occurs over timescales much shorter than those associated with collective human actions.” Applied to RI: “what happens if we’re still planning for the world as it was at 11:00 PM when we get to the AI equivalent of 11:59 PM?” He anticipates that readers will dismiss this, and says the dismissal is itself the point: “it always will feel like an intellectual exercise until it’s too late.” - Step back to firmer ground and plan for the off chance. “AI 2027 is speculation — no more.” It still “force[s] the question of how we might think about responsible innovation and AI, just on the off chance that there’s a sliver of truth here.” This is a precaution-like, scenario-as-stress-test stance, though he does not call it precaution. - Method: a deep dive with a reasoning model. He turned to “a mode of working that I’ve been finding increasingly useful recently — engaging with OpenAI’s o1-pro model to develop nuanced and widely informed insights into complex questions”. The report comes with “the usual caveat that important information in it should be double checked”.

How firmly. His worry is sincere but heavily hedged. He says “I hope they are not” (futile), “unlikely as I hope it is”, “edge case” and “speculation — no more”. He is firm on two points. Human planning and governance are not built for change over months. People are bad at grasping exponential growth and will tend to deny it. He is also firm, in his own parenthesis, that responsible innovation would not fare well in a US–China acceleration race. He does not give his own view of what should replace or supplement RI. That is handed to the o1-pro report, which he endorses in general terms but not point by point.

Concepts and frameworks. - Responsible innovation / responsible AI: the frame the whole post interrogates. He treats it as process-based and slow (“processes that can take years”). He does not define it in his own prose. The RRI dimensions of Stilgoe, Owen and Macnaghten (anticipation, inclusion, reflexivity, responsiveness) and EPSRC’s AREA appear only in the o1-pro report. - Timescale mismatch: human and institutional timescales versus the timescales of “intelligent machines”. Months versus years. - Exponential-growth blindness: failure to recognise, or active denial of, rapid change until “one minute to midnight”. - Edge-case scenarios: useful “for exploring potential (if not necessarily likely) near term AI futures”, with contestable assumptions. - Tipping point: near-term possibilities that change what the future looks like. - Arms race as a game-theoretic trap: collective behaviour that makes a race “inevitable, no matter how bad an idea anyone thinks it is”.

Analogies and comparisons. - Bartlett’s bacteria in a beaker, used structurally, as an illustration of human cognitive and planning limits, not as a literal model of AI growth. He flags that it fails literally because of resource limits. - Dan Brown’s Inferno (film), as a popular, flawed vehicle for the illustration. This fits his long habit of using films to think about technology. - Game theory and the arms race, used conceptually (as the scenario’s assumption). - No comparisons with past technologies in his own prose. The comparisons with nuclear arms control, the IAEA, CERN, Asilomar 1975, germline editing and cloning moratoria, and social media are all in the o1-pro report.

Views on AI. - What kind of thing it is: in his framing, AI is on a potentially exponential development path. It may be capable of recursive self-improvement through coding, and it operates on the timescales of “intelligent machines”, faster than human deliberation. He uses the phrase “intelligent machines” without qualification here. - What is new: speed. Advances “over months rather than years”, together with competitive and geopolitical dynamics, may outrun every process built around human timescales. - He does not commit to superintelligence arriving. He treats it as an edge-case possibility worth planning for.

Views on AI risk. - Main risk named: that AI acceleration and an arms race outpace society’s ability to govern AI responsibly, with a tipping point we fail to see or actively deny. - Seriousness: it is an edge case, but the stakes are high enough (“abject failure and world domination”) that ignoring it would be a mistake. - Framing: the risk is partly institutional (slow processes) and partly cognitive and cultural (how we think and plan, and denial). He does not discuss existential risk or loss of control directly in his own prose, beyond referring to the scenario’s “radical shifts in the world order”.

AI companies and leaders. He is descriptive rather than critical. The scenario’s first assumption is AI companies racing to build self-improving coding models in order to “dominate the global market”. He notes that the scenario reflects the thinking of “growing number of leaders and developers at the cutting edge of AI”, and that “big names” welcomed it as a warning. He makes no criticism of named companies in his own prose. The report’s material on Microsoft’s ethics-team layoffs, Gebru’s firing, and the RSPs of Anthropic and DeepMind is o1-pro’s.

Governance and who decides. Current governance ideas and RI processes are “constrained by human timescales”. He does not say who should decide or what institutions are needed. He hands that question to the o1-pro report and endorses it only in general terms (“inclusive and insightful”, “very measured”). His one clear governance judgement: RI would fare “not that well” in a US–China acceleration race.

Cognition, language and human formation. This is the post’s most distinctive contribution in his own voice. The problem is partly human cognition. For him, “nothing about how we think” is geared to change over months. People are “really bad at wrapping our heads around rapid exponential growth”. Societies tend to fail to recognise, or to deny, such change until it is too late. The sense that it is “an intellectual exercise” is itself a symptom. Nothing here is about language or AI’s formative effects on people.

What he criticises and who he engages. - He engages the AI 2027 authors’ scenario seriously. He does not dismiss it, and he does not endorse it as likely. - He engages its critics (“too alarmist”) and its supporters among “big names” in AI. - He points implicitly at the RI and responsible-AI community, himself included, for planning on human timescales. - There is no direct criticism of individuals. Through Annex C he flags “online critique of the AI 2027 scenario’s authors as well as their ideologies and perspectives”, but he does not state his own view.

Change of view signalled. He signals no change outright, but the post shows movement. His “first reaction” was to worry that current RI efforts might “seem futile”. He is now willing to plan around a near-term superintelligence scenario “on the off chance”. Compare his 2018 writing, noted in B02 and B08: there he called himself “something of an agnostic” on superintelligence and attacked exponential extrapolation, on the grounds that every exponential stops. He still keeps that caveat here (“resource constraints would slow or halt the exponential growth”). But he now uses the exponential frame to diagnose a blind spot in human perception rather than as a fallacy to debunk. He also shows growing reliance on reasoning models as research partners (“a mode of working that I’ve been finding increasingly useful recently”).

Key quotes (his own prose). - “we might be facing a near term future where current efforts to develop artificial intelligence responsibly seem futile.” - “nothing about how we think, how we plan for the future, or how we develop approaches to ensuring better futures, is geared toward exponential advances” - “we would most likely fail to recognize it — or would actively deny it — until it was too late.” - “it always will feel like an intellectual exercise until it’s too late.”

NONE#