Late Lessons, Jensen Huang and AI
Markdown version

B17 perspective notes: 2024-08-21 to 2024-09-25 (10 posts)#

These notes read the batch for how Maynard thinks, not for the concepts he names. All ten posts were read in full. Quotes are exact, including his typos. There are no Modem Futura episode posts in this batch. The NotebookLM post (2024-09-22) contains a one-line promotion box for the podcast, which is ignored.

Evidence rules applied

Context. These are the weeks around the start of the ASU academic year. They bring OpenAI’s GPT-4o system card and voice mode, the o1 “Strawberry” preview, the FDA Breakthrough Device designation for Neuralink’s Blindsight, FinalSpark’s rentable organoid processors and NotebookLM’s “audio overviews”. His responses fall into four groups: - He builds and tests. The o1 moral-dilemma probe, and a five-step NotebookLM experiment using his own writing as material. - He interrogates evidence. OpenAI’s persuasion test, and Musk’s claims about vision. - He offers a thinking tool. A second quadrant framework for “navigating advanced technology transitions”, from his IEEE keynote. - He looks back at his own practice. Risk Bites at a million views, the Future Rising quotes, and a course retired because it worked too well.

A through-line connects the batch: the power of stories and persuasion to shape which futures people build, and who gets to author them.

1. How he thinks here#

Play first, tool later#

He introduces the transitions framework as “a framework that I’ve been playing with for some time now” (2024-08-25 advanced-technology-transitions-model). He is candid about how modest it is: “This is a blindingly simple framework — it’s definitely not rocket science!” He is equally clear about what it is for: it “offers surprisingly sophisticated insights when applied to specific cases.” It closes under the heading “Not Quite a Tool Yet”. The order matters. Play produces a way of seeing. The way of seeing produces insight. Only after testing might it become a decision tool: “It needs more work and testing.” The three AI cases are flagged as “illustrative only”. He is showing the thinking move, not issuing verdicts.

Turning a static list into movement#

The framework does not stop at naming threats and opportunities. Its real work is to ask how things move between them. It asks how society might move from threat quadrants to opportunity quadrants, and how to prevent “slippage from personalized learning at scale to diminished critical thinking”. It then asks which “mechanisms” open or close each pathway (2024-08-25). The image is spatial: a landscape of value with paths across it, travelled in both directions and over near and far time horizons. The question becomes where people are heading and what might tip them, rather than how big a risk is.

Building a probe as a story#

For the o1 test he writes a small piece of speculative fiction (2024-09-13 openai-o1-strawberry-moral-reasoning). A CEO’s Alzheimer’s implant has a bug. A wireless patch will save twenty trial patients and kill his father. He wanted “overtones of the Trolley problem, but which was a little more sophisticated”, and “I made the decision deeply personal to them”. The scenario draws on his long-standing interests: brain–computer interfaces, cut corners in getting to market, and the lifetime obligations of implants. He then pushes the model the way a good examiner would, closing each escape route in turn. His interest is in “messy problems to which there are no easy solutions”. The test’s author is a storyteller. The test itself is a controlled experiment.

Experimenting in sequence, changing one thing at a time#

The NotebookLM post is a small research programme (2024-09-22 five-ai-generated-podcast-episodes-from-googles-notebooklm). It starts from curiosity sparked by someone else (“I had to try this for myself”). Each step tests a hypothesis raised by the one before: - 19 of his own posts produce misframings. So: “I wondered whether the mis-framings and misinterpretations I was hearing were simply due to feeding NotebookLM too much information across too many areas. And so I tried narrowing things down”. - Eight Neuralink posts still produce extrapolation. So he tries a single source (Bill Joy): “would this reduce the degree to which it got creative beyond the supplied material?” - That still riffs. So he tries a case where subjectivity may help: his own school, SFIS. - Finally: “What about situations where creativity is more important than accuracy though?” This leads to the AI-feeding-AI novel test. Here he deliberately does not read the novel first: “I didn’t even read it before listening to the podcast, as I was intrigued to see how this AI-feeding-AI process would work out.”

He uses his own work as the test material, because he is the one who can tell where the machine is wrong.

Reframing: from the tested risk to the risk that matters#

In 2024-09-01 is-chatgpts-new-voice-mode-dangerously-persuasive he accepts OpenAI’s result and then reframes the question. He moves it from short-term shifts in political opinion to persuasion inside a relationship: time, trust and “does the user want to believe the AI”. Then he flips it again, to benevolent persuasion: “What if, over time, using GPT-4o could persuade people to take climate change seriously”. He calls this “both intriguing and disconcerting”. The reframed risk is not a harm to avoid but a good outcome achieved by an illegitimate route: “But who decides what is good for society? Who decides what you should believe, and what you should do? And where does democracy fit into such a view of an AI-mediated future?”

Analogy by structure, carried from his own past#

“In doing my prep for this article I was reminded of” Dan Kahan’s cultural cognition work. That includes work Kahan did “with us” on nanotechnology at the Woodrow Wilson Center, showing that the perceived messenger shapes how a message lands (2024-09-01). The transfer is structural. If a visual cue of “their sort of person” moved nanotech attitudes, “this got me wondering if it can, nevertheless, project a persona that users feel aligns with the way they see the world”. A mechanism from one technology era is applied to a new medium. It is a question, not a claim.

Plausibility tests and back-of-envelope numbers#

His physicist’s reflex shows in small ways: - He does his own energy comparison for organoid computing: “a person would need to live for nearly twelve thousand years” to match GPT-3 fine-tuning. A footnote admits “These estimates differ from those in the FinalSpark paper, but the end result is similar” (2024-08-28 human-brain-organoid-based-ai-processors). - He looks past the headline number to the mechanism. Biological “weights” “can’t simply be tweaked and re-written”. So “I suspect that the advantages of AI powered by brain organoids will far surpass the energy use advantages”. - He tests fiction against physics. The Transcendence sight-restoration scene “is sheer nonsense. It’s not something that’s possible within the laws of physics, no matter how much wishful thinking is involved.” (2024-09-18 neuralink-blindsight-brain-computer-interface). - He reads OpenAI’s persuasion plot carefully: no statistical difference between conditions, and the shift reverting after a week (2024-09-01).

“What if”, held at arm’s length#

He will follow a line of thought past where the evidence stops, but he labels it as he goes. On organoids he asks: “why stop there? What might be possible if we tap into real brains inside the heads of real people?” He muses about someone suggesting that users “rent out their brain-time”. Then, at once: “This may sound like scare-mongering sci-fi”. And he places it in time: “That’s a can of worms that we’re probably decades or more from opening.” (2024-08-28). The speculation is a way to see a question early, not to raise an alarm.

Films and stories used both ways#

In 2024-09-18 he uses the film Transcendence to name what Musk’s rhetoric is reaching for: “that of a technological savior bringing sight to the blind”. He adopts his colleague Riz Virk’s “Sci Fi feedback loop” to name a danger: “The danger is that technologists who get caught up in Riz’s “Sci Fi feedback loop” similarly end up ignoring reality in the belief that they can transcend it.” This is not a rejection of science fiction. The distinction is fine but consistent: films are for thinking with, not for building from. The same batch republishes his line that “[S]tories become the pivot point between being able to imagine the future and beginning to build it” (2024-09-08). Because stories are that powerful, he watches who uses them and how.

Curiosity and serendipity as starting points#

Several posts start from a chance input: - a colleague’s tip (“Many thanks to my colleague Sean Dudley for alerting me to this!”, 2024-08-28); - another Substacker’s NotebookLM experiment (2024-09-22); - a summer spent “re-reading the British science fiction author John Wyndham’s novels” that becomes an experiment (2024-09-22); - a conversation with Chitta Baral that opens “a quite wonderful although rather distracting rabbit hole” (2024-09-25 asu-ai-research-scai); - a student webinar that rescues a draft he was about to delete (2024-09-08).

He lets these lead the thinking.

2. What matters to him#

Knowledge that reaches everyone#

The clearest statement of value in the batch: “I believe rather strongly that the privilege of academic scholarship and research comes with an obligation to ensure that the knowledge we unearth is accessible to anyone who can benefit from it, no matter who they are or where they are.” (2024-09-04 succeeding-at-science-on-youtube). He describes two decades of work on knowledge “locked within the “ivory tower” of academia”. He gives away the video-making modules and, earlier, the ChatGPT course content (2024-08-21 chatting-with-wef-about-chatgpt).

Responsibility that matches power#

The first Future Rising quote he chose: “our ability to envision and engineer the future comes with almost unimaginable levels of responsibility” (2024-09-08). The theme recurs through the batch: - “there’s a profound responsibility to do right by the people who are impacted by them” (2024-09-18). - “implanted medical devices come with a lifetime responsibility to patients” (2024-09-18). - “But where doe those responsibilities lie when one partner in the relationship is a machine?” (2024-09-01).

People, not “fixes”#

He resists Musk’s framing: “Putting aside the troublesome idea of “fixing” people”, and “savior complex-like aspirations to make make humans “better” than their messy and limited biology allows” (2024-09-18). He is concerned for the patients who are “left stranded” when devices fail (quoting Fine and Boynton). He also worries about access and long-term support, which get overlooked when people are “blinded by the technology”.

Human agency over belief and story#

What he protects in 2024-09-01 and 2024-09-22 is people’s ability to form their own beliefs and to author their own futures. The closing question of the NotebookLM post: “who or what will end up creating the stories that determine our beliefs, guide our actions, and ultimately govern our futures?” This echoes the Future Rising worry about people “disturbingly good at stealing the futures of others when it suits us”. It also echoes the recognition that “we cannot design and build a future that fits everyone’s idea of what it should be like” (2024-09-08).

Value, broadly understood#

The value at stake “could just as easily be identity, dignity, or deeply held beliefs” as health or the environment (2024-08-25). The Future Rising line “it’s what we feel — and what we hold to be of value — that often determines the futures we are willing to invest in” shows that this is not only an analytic category for him. It is how he thinks people actually decide.

What frustrates him#

What delights him#

3. Risk as a way of thinking#

Threat to value, stated and explained#

2024-08-25 gives the batch’s fullest statement. The transitions framework “takes inspiration from our work over the years on Risk Innovation”, which “conceptualizes risk as a threat to value”. He separates value from values: ““value” represents the worth of something to someone”. He then says what the idea is for: it “makes Risk Innovation an extremely flexible tool for conceptually navigating complex risks and benefits”. The key word is “conceptually”. Its first job is to reorganise thought.

Risks and benefits in one frame; missing out as a risk#

The same passage says it “allows risk to be approached as a balance between maintaining existing value, and enabling the creation of future value.” In the AI-and-discovery case, the near-term threat is “missed opportunities as a near term threat (such as loss of future value through not adopting new technologies)”. Failing to realise benefit counts as a risk. This is what lets the framework stand between the boosters and the doomers without choosing a side.

Quadrants, pathways, mechanisms and slippage together describe a risk landscape that people move across over time (2024-08-25). The verbs are “navigating”, “steering” and “move toward”. Where the post turns to operations, it goes only as far as “move toward identifying and operationalizing approaches”. The concept stays ahead of the tool.

Novel technologies need a new mindset#

The closing line says the tool is meant for “advanced technology transitions that aren’t navigable through conventional understanding and means” (2024-08-25). The same stance shows elsewhere in the batch: - The voice-mode test is weighed against “uncharted territory”. The question is whether an established method, political-opinion change, fits a technology built for intimacy (2024-09-01). - Organoid computers do things that have “no direct equivalence in digital neural nets”. Their downsides “are likely to be complex and hard to identify in the near term” (2024-08-28). - NotebookLM’s risk is not simple falsehood. It is the pairing of engaging form and loose content: “compelling AI generated stories that are hard not to trust, and yet are not trustworthy” (2024-09-22).

Each case shows that a conventional risk category (misinformation, persuasion, energy) misses what is new.

Risks that come from design and use, not only from bad actors#

He asks whether persuasive power is “malicious in intent, or simply an emergent property of the technology” (2024-09-01). NotebookLM’s “anthropomorphic heavy lifting” makes content “easy to trust it and hard to challenge”. That is a property of the design, not of anyone’s intent (2024-09-22).

Echoes of orphan risks#

He uses Fine and Boynton’s phrase for patients “left stranded with orphaned technologies”. He sets it beside his own point about “lifetime responsibility to patients” (FFTF ch. 7) and his note about supply chains, economics and politics (2024-09-18). These are consequences that fall outside the usual safety frame and that no one owns. The phrase is theirs, not his “orphan risks” concept, but the concern is the same.

Humility against false precision#

The Future Rising quotes he republished set out the stance: - “Hubris, if we’re not careful, leads to false hope. It tempts us to promise what isn’t within our power to deliver, and it encourages belief in the absence of evidence.” - “some future effects simply cannot be predicted, even if we know everything we think we need to know” - “the pathway between present and future is neither linear nor exponential”, with “new opportunities serendipitously emerging that confound all expectations” - Blindsides “occur in the liminal space between how we imagine the future playing out and the unknowability of what’s going to happen next.”

This is the stance behind his critique of Musk’s rhetoric, his “hard to predict” on organoids, and his “not 100% convinced” on OpenAI.

Early warnings and threats to value in the 2020 book#

Two of the republished Future Rising lines show that these risk ideas predate the AI boom. “Threats to what we value, it turns out, have a powerful impact on how our future unfolds.” And “[U]nless we learn how to spot early warnings and stay clear of critical tipping points, we run the risk of, quite literally, crashing our future.” (2024-09-08). For him, risk is part of the human relationship with the future, not a separate technical specialism.

Quantitative foundations, still in use#

The energy arithmetic, the careful reading of OpenAI’s persuasion plot (no significant difference, reversion after a week), and his concern that “little is known about the precise methodology used” (2024-09-01) show a risk scientist’s habits at work. He builds the conceptual framing on these habits. He does not replace them.

4. Scholarship and public writing#

One continuous practice#

The posts are one place where his other work surfaces: - The transitions framework comes from his IEEE ISCT keynote (2024-08-25). - The Blindsight post is written on the way to the Dartmouth device symposium, where he will use Neuralink as a case of “bringing a big tech mindset to medical device development” (2024-09-18). - The voice-mode post draws on his Risk Innovation report with Hazel Kwon on counter-influence operations, on his Woodrow Wilson Center collaboration with Kahan, and on his FFTF manipulation chapter (2024-09-01). - Risk Bites led to a formal 2020 paper and a graduate workshop (2024-09-04). - The book is re-offered as sixty quotations (2024-09-08).

Keynote, paper, book, classroom and Substack feed each other.

Thinking out loud, unfinished work included#

He publishes a framework that is “still a work in progress” and asks others to test it (2024-08-25). He publishes a “very quick and dirty test” one day after o1’s release, ahead of his usual schedule (2024-09-13). He publishes the NotebookLM experiments with their audio so readers can judge for themselves (2024-09-22). He even publishes a draft he “was about to trash” (2024-09-08).

Transparency about method#

Examples: - “I’ve kept the couple of grammatical errors in the above so you can see what o1 was provided with” (2024-09-13). - “To be clear, this is not the response” (distinguishing o1’s internal reasoning from its answer, 2024-09-13). - A faculty list “(constructed using the AI platform Perplexity)” (2024-09-22). - He says which estimates differ from the source paper (2024-08-28).

Evidence and expertise, including his own limits#

He brings in experts whose field is not his: Kwon on communication and information disorder, Fine and Boynton on vision science, Baral on AI research. He credits them by name and quotes them carefully. He accepts correction from them (“Chitta did warn me that this is a little misleading”, 2024-09-25). He names his own gaps: “to a lowly physicist/transdisciplinarian like myself it was new — and it was a revelation”; “I should, of course, have a handle on this already” (2024-09-25). He notes that expertise changes what AI output is worth. He can “filter out the suspect stuff”, “But I’m not sure though what someone who was new to the field would take away from it” (2024-09-22).

Transdisciplinarity#

He calls himself a “transdisciplinarian”, and says: “Anyone familiar with my work will know that I’m interested in broad/transdisciplinary approaches to advanced technologies, society, and the future.” (2024-09-25, n. 2). The transitions framework lists “research that transcends conventional disciplinary boundaries” and “multistakeholder engagement” as mechanisms (2024-08-25). The Future Rising sequence moves from the big bang and entropy through evolution, instinct and memory to art, faith, hope, ethics and stewardship (2024-09-08). It is one continuous account from physics to the human.

Accessibility as craft#

5. His role as he sees it#

A public scholar by obligation, working “between the cracks”#

Risk Bites began as an experiment to see “whether a full blown academic with limited time and even less talent could use YouTube”. The experiment “leant into my limitations” (2024-09-04). The work was done “between the cracks” of a busy job. The point is that any academic could do this: “there’s nothing to stop most academics leveraging YouTube in a similar way.”

A learner, including from his children and students#

“it took my children to show me what was possible” (2024-09-04). Students change his judgement: “talking with an inspiring group of students this past week made me realize that maybe it does have some value after all” (2024-09-08). He sees in them “a hunger amongst young people around the world to better understand our relationship with the future”. He builds courses with them and then retires the course when AI makes it redundant: “not because it didn’t work — far from it — but because it rapidly became redundant” (2024-08-21).

With industry: generous, and methodologically demanding#

On OpenAI he says “I very much appreciate OpenAI’s approach to publishing their system cards”, and suggests others “could learn from” it (2024-09-01, n. 1). He still questions the test design. On Neuralink he separates the company’s scientists, whom he hopes “have a firmer grasp of reality”, from Musk’s rhetoric. He takes seriously that AI integration may mean “a lot more will be achievable than with more traditional approaches” (2024-09-18). He also steelmans the hype: “Of course, it could be argued that naive visions of the future are maybe what are needed to jolt society into exploring and developing new possibilities”. His criticism targets claims and methods, not motives.

With colleagues: convenor and amplifier#

A whole post celebrates colleagues in another school (2024-09-25). The NotebookLM post names SFIS colleagues. The voice-mode post turns an email exchange with Kwon into shared analysis. He writes that “the cutting edge of AI is very much a team sport”.

With readers: co-explorers, not an audience to convince#

He apologises in advance to readers who won’t want sixty quotes (“If that’s not you — my apologies.”, 2024-09-08). He asks for others’ judgement (“I’d be interested in what others think”, 2024-09-22). He undercuts his own alarm (“Maybe that’s just me being paranoid though — I’d be interested in your opinions in the comments below!”). He ends on an open question (“Which leaves me still wondering”, 2024-09-01).

What he refuses to do#

Changes of mind, shown in the text#

6. What is distinctive#

  1. He asks whether the test fits the technology. Most commentary on OpenAI’s system cards either praised them or dismissed them. Maynard accepts the result and asks whether the test measures the right thing for a technology designed for intimacy. He draws on persuasion science and on his own nanotech-era work with Kahan (2024-09-01). The move comes from risk assessment: check whether the exposure scenario matches real use.

  2. He treats good intentions as a governance risk. His deepest concern about voice AI is persuasion toward outcomes many would welcome, such as climate action, vaccination and kindness, rather than manipulation. His question is democratic: “who decides”. Few AI-risk discussions treat benevolent nudging at scale as a risk in its own right.

  3. One framework for threats and opportunities, with movement between them. The quadrant, pathway and mechanism framework counts missed benefit as a threat. It watches for slippage from good to bad as well as escape from bad to good. It is aimed explicitly at transitions “that aren’t navigable through conventional understanding and means” (2024-08-25). It is offered as a mental model to play with before it becomes a tool.

  4. Stories as both instrument and object of concern. He writes a fiction to probe a reasoning model (2024-09-13). He tests fiction against the laws of physics (2024-09-18). He runs an AI-on-AI storytelling experiment (2024-09-22). He also holds that stories are “the pivot point” between imagining and building the future (2024-09-08). So his worry about who writes AI’s stories (2024-09-22) and his critique of the “Sci Fi feedback loop” (2024-09-18) come from someone who values stories highly. They are not a technophobe’s suspicion of fiction.

  5. Experiments with controlled variation, in public, on his own work. The NotebookLM sequence changes scope, source count, subjectivity and creativity one step at a time. It uses his own posts, where he is the ground truth. Few writers subject their own work to the machine in this way.

  6. The human baseline. He keeps comparing AI with the humans it imitates: mediocre novels, cavalier podcasters, persuasive orators. This keeps the concern about what is actually new (scale, intimacy, “anthropomorphic heavy lifting”) rather than about AI being imperfect.

  7. A deep-time frame for technology. The Future Rising sequence places AI and “Re-Creation” at the end of an arc that starts with the big bang, entropy and “localized anti-entropy machines”, and ends with empathy, responsibility and stewardship (2024-09-08). AI is one chapter in the human capacity to shape the future, a capacity he treats with both awe and caution.

  8. Patient- and person-centred critique of neurotechnology. He objects to “fixing” people, insists on lifetime responsibility for implants, and names the supply chains and politics of care (2024-09-18). He brings disability and long-term care into a discussion usually dominated by capability claims.

7. The posts in this batch that best show how he thinks#

  1. 2024-08-25 advanced-technology-transitions-model. His risk thinking as a mental model: threat to value, “value” versus “values”, existing and future value balanced, pathways and mechanisms across a landscape, and missed opportunity counted as a threat. It was played with first and offered as “Not Quite a Tool Yet”, for transitions not navigable by conventional means.
  2. 2024-09-01 is-chatgpts-new-voice-mode-dangerously-persuasive. Fair to industry, sceptical of method. It carries a structural analogy from his nanotech work, draws on a colleague’s expertise, and flips the risk to benevolent persuasion and the question of who decides. It ends with an open question.
  3. 2024-09-22 five-ai-generated-podcast-episodes-from-googles-notebooklm. Hypothesis-driven play in public, using his own work as material. He is honestly drawn in, compares the machine with humans, and closes on who authors the stories that govern our futures.
  4. 2024-09-18 neuralink-blindsight-brain-computer-interface. A film used as a lens and fiction tested against physics. It warns against the “Sci Fi feedback loop”, puts people and lifetime responsibility first, and steelmans the hype while resisting it.
  5. 2024-09-08 a-journey-from-the-past-to-the-edge-of-tomorrow. The worldview underneath, in his own book’s words: physics to people, curiosity, art, stories, hubris, early warnings, threats to value, responsibility and joy. The framing also shows students changing his mind.
  6. 2024-09-04 succeeding-at-science-on-youtube. His role in his own words: scholarship as an obligation to share knowledge, a scrappy experiment that “leant into my limitations”, honesty about failures, and the craft of storytelling.