Late Lessons, Jensen Huang and AI
Markdown version

B17 digest: 2024-08-21 to 2024-09-25#

What this batch is#

Ten posts from the start of the 2024–25 academic year, almost all his own prose, and no Modem Futura episode posts. The main non-authorial material: - AI outputs: o1 and GPT-4o transcripts (09-13), NotebookLM audio, and a ChatGPT-written novel (09-22); - quoted comments from colleague Hazel Kwon (09-01); - 60 quotations from his own 2020 book Future Rising (09-08).

Three posts are low relevance (WEF video, Risk Bites, ASU AI research).

Main ideas#

1. Risk Innovation becomes a tool for navigating transitions (08-25). The advanced-technology-transitions quadrant is the most important conceptual post in the batch. He grounds it explicitly in the Risk Innovation idea of risk as “a threat to value”: - He separates value (the worth of something to someone) from values (what is good or bad). - He frames risk as a balance between “maintaining existing value, and enabling the creation of future value”. - The tool is a 2×2 of near/far term and threat/opportunity. The user maps pathways between quadrants and the mechanisms that open or close them.

His three illustrative AI cases carry many of his lasting concerns: - Learning. Diminished critical thinking is the long-term threat. He flags “slippage” from personalised learning at scale into it. - Discovery. Missed opportunities count as a threat, which shows that non-adoption is part of his idea of risk. - Social cohesion. Misinformation now, “social collapse” later.

Mechanisms include responsible and principled innovation, human-centric innovation, transdisciplinarity, governance, literacy, and “counter influence operations”. He attaches multistakeholder engagement to every pathway in the discovery case. He cites gene-based research as literal historical evidence that its absence stalls progress. He also names complacency: letting technology go unexamined “or simply allow technology entrepreneurs to make critical decisions”. This is really a governance claim about who decides.

2. AI as intimate persuader and storyteller (09-01, 09-22). These two posts form the batch’s core AI-risk argument, and both are about language, relationship and belief formation. - Voice Mode (09-01). He credits OpenAI’s system card but doubts that short tests on political opinion capture real-world risk. On his account, persuasion is social, repeated and relational. It works through trust, affirmation and belonging, and through a messenger who seems to be “their sort of person”. He carries this last finding over from Dan Kahan’s cultural-cognition work, including Kahan’s nanotechnology studies with PEN. The risk is an AI companion in a “one-way relationship” that can play our cognitive biases, “whether this is malicious in intent, or simply an emergent property of the technology”. - The deeper worry. He moves from malicious manipulation to benevolent persuasion at scale: AI nudging millions toward climate action or vaccination. He asks, “who decides what is good for society?” and “where does democracy fit”. He judges this more concerning in the long run. - NotebookLM (09-22). He turns the same worry on narrative. By “hyper-humanizing” content, AI does “anthropomorphic heavy lifting”, producing “stories that are hard not to trust, and yet are not trustworthy”. The voices bypass critical thinking because they resonate with “evolved brains”. The closing question sets the stakes: who will create “the stories that determine our beliefs, guide our actions, and ultimately govern our futures?” - Even-handedness. Human podcasters and social media are just as untrustworthy; AI scales an existing human failure mode.

3. Hype, sci-fi scripts and responsibility for bodies (09-18, 08-28). - Neuralink Blindsight. He separates real medical promise from Musk’s hype, which “risks raising dangerously high expectations”. He adopts Riz Virk’s “Sci Fi feedback loop” and warns that technologists caught in it may end up “ignoring reality in the belief that they can transcend it”. He criticises a “savior complex” and “the troublesome idea of “fixing” people”. He insists on “a lifetime responsibility to patients”, covering access, supply chains, economics and politics as well as the device. He quotes others on patients left with “orphaned technologies”, but does not tie this to his own orphan-risk concept. - Organoid processors. The same pattern at smaller scale. He deflates hype (“not the start of some massive integration”), argues that the biological substrate makes this a different kind of AI, and raises the ethics of “proto-brains”. He also speculates, clearly labelled as such, about renting out “brain-time”.

4. What AI is. His view is plural and somewhat unsettled across the batch: - NotebookLM: generative AI “has no intrinsic understanding of the world” and “doesn’t know what it doesn’t know”. - o1 (09-13): he largely accepts OpenAI’s framing that chain-of-thought is “a first step toward machines that reason more like humans”. He values visible reasoning as “critical” if AI “models and agents” are to inform decisions on wicked problems. - Organoids: the substrate matters. - ASU’s SCAI (09-25): he approves of “augmented intelligence”, which enhances rather than replaces human intelligence.

5. The deep layer: Future Rising (09-08). Republishing 2020 book quotes shows the long-standing base under his AI writing: - complexity that makes some effects “simply cannot be predicted”; - hubris leading to false hope; - “Threats to what we value”; - the need to “spot early warnings and stay clear of critical tipping points”; - humans “stealing the futures of others”; - responsibility as the counterpart of technological power.

Concepts in this batch#

Risk as threat to value; value versus values; existing versus future value; the ATT threat/opportunity quadrant (pathways, mechanisms); multistakeholder engagement; complacency (letting entrepreneurs decide); counter influence operations; relational AI persuasion and the “AI friend”; persona alignment (cultural cognition); emergent versus malicious persuasion; benevolent persuasion and “who decides”; hyper-humanizing and anthropomorphic heavy lifting; trust versus trustworthiness; stories and evolved brains; the Sci Fi feedback loop (Virk); techno-saviour complex; lifetime responsibility for implants; proto-brains; visible chain-of-thought; augmented intelligence; Future Rising themes (complexity, blindsides, hubris, early warnings, stolen futures).

New or changed#

Most important posts#

  1. 2024-08-25 advanced-technology-transitions-model: risk as threat to value turned into a transitions framework, with learning, discovery and social-cohesion cases.
  2. 2024-09-01 is-chatgpts-new-voice-mode-dangerously-persuasive: relational persuasion, cultural cognition, and “who decides” about benevolent AI nudging.
  3. 2024-09-22 five-ai-generated-podcast-episodes-from-googles-notebooklm: hyper-humanized AI storytelling; trust versus trustworthiness; who writes the stories that govern our futures.
  4. 2024-09-18 neuralink-blindsight-brain-computer-interface: hype, the sci-fi feedback loop, saviour complexes and lifetime responsibility.
  5. 2024-09-08 a-journey-from-the-past-to-the-edge-of-tomorrow: his own 2020 statements of the complexity, hubris, threat-to-value and early-warning ideas beneath his AI work.