# B09 digest: 2023-05-15 to 2023-07-21

## What this batch is

This batch has eighteen posts from mid-2023, when AI governance moved from talk to institutions. Maynard responds almost in real time to a run of events:

- Eric Schmidt's claim that only industry can set early rules;
- the Senate hearing with Altman, Montgomery and Marcus;
- Bengio's "rogue AI" essay;
- the CAIS extinction statement;
- the EU AI Act amendments;
- the "Frontier AI Regulation" paper and Jeremy Howard's open-source rebuttal (he contributed to the rebuttal);
- the launch of x.AI.

There is also an education strand and some light posts. None is a Modem Futura episode. The "Moviegoer's Guide" posts are show notes for his reading of *Films from the Future*. The 2023-07-08 item is a Gary Marcus cross-post.

## Main ideas

**1. Who decides: governance is a field of expertise, and publics have a right to a seat.** This is the batch's backbone.

- Against Schmidt he names the "leave it to the technical experts" mentality, which "never plays out well".
- His evidence comes from his own career:
  - Monsanto's "it’s complicated, leave it to us" on GMOs "backfired spectacularly";
  - nanotechnology dodged "a bullet" by engaging "early and often";
  - a PCAST member once scoffed at public judgement.
- He makes two claims about participation:
  - **Rights:** "*everyone* has the right to play some role".
  - **Epistemics:** people need not understand AI's inner workings to judge what it threatens that matters to them.
- In the Senate and frontier-AI posts the problem is *narrowness*: experts in AI but "light on their expertise in governing emerging technologies". The danger is oversight locked in by "a very small group of players".

**2. Regulatory design: skeptical of AI-specific hard law, doubtful about the technology/use split.** Drawing on nano-era policy, he:

- calls hard law "crude, cumbersome" and AI "a moving target";
- rejects a stand-alone AI agency;
- proposes a cross-agency initiative for "advanced technology transitions writ large", combining foresight, benefit/risk assessment, public–private partnerships and multi-stakeholder governance;
- repeatedly doubts the nanotech-era mantra "we regulate what people do with the technology, not the technology itself":
  - capabilities "may well erode convenient distinctions between the technology and its uses";
  - predictive policing is a moral hazard "that cannot be addressed by naively separating the technology from its use";
  - frontier AI raises whether "a foundational general purpose technology" is dangerous in itself. His "current thinking lies between" licensing and open source.

This is a careful, hedged move away from his own field's past consensus.

**3. Reframing AI risk: catastrophic loss of value, not extinction.** He explains why he did not sign the CAIS statement. He supports AI risk as a "global priority", but "extinction" is "too narrow and absolute a framing, and too human-centric".

- Conventional risks (jobs, privacy, education) are "the shavings off the tip of the AI iceberg". Treating them with "a conventional mindset" is "extremely naive".
- The deeper threats are to "social, economic, and political structures, systems, and norms": how we "build our understanding of the world", autonomy and democracy. They are driven partly by LLMs' "seductive mastery of language", in a "highly non-linear technology transition".
- He brings back **risk innovation**: catastrophe is when many people "risk losing something that is deeply valuable to them". The frontier-AI coda names the **risk innovation nexus** and "threat to value", extended to "deeply held beliefs, and even self-identity", and promises more. The frame also counts the risk of *not* developing AI.

**4. Alignment and existential talk as questions of power and values.**

- He stays "not a fan" of superintelligence, but accepts "an exceptionally powerful autonomous artificial entity".
- "Rogue" and "alignment" framings assume AI should "behave", so he asks "who’s values matter, who decides".
- Musk's AGI of "maximal curiosity and truth seeking" fails twice:
  - curiosity is not benevolence;
  - ultimate social "truth" is usually "an excuse to impose an ideology".
- The Bengio post ends with an inversion: the real fear may be that AI "will look too much like us". We should "co-create a shared future" with agentic machines. The Musk post warns against machines that are "deeply *inhuman*".

**5. Authority over accuracy.** The predictive-policing post places the harm in *apparent* authority. Its tools would be unreliable but "easy to use and, above all, persuasive", with a bar of "authoritative rather than accurate". Only guardrails stand in the way. He is firmly opposed on human-rights grounds. The EU Act post carries the manipulation concern into education, where AI tutors might steer learners manipulatively.

**6. Education optimism with a human in the loop.** He is at his most enthusiastic here:

- ChatGPT designed, tutors and assesses his course;
- the change is on a "printing press" scale;
- AI literacy should be general education, and more than compliance;
- EU rules may stifle personalised learning.

He also notices AI "fine tuning my brain to be a better instructor". This is an early, positive note on AI-mediated formation that later work complicates.

## Concepts appearing

- Advanced technology transitions
- The leave-it-to-the-experts fallacy; rights-based participation
- The "atomic technologies moment"
- Precision and use-based regulation (doubted)
- Cross-agency foresight; public–private partnerships; lock-in; possible regulatory capture ("not yet")
- Soft law, agile governance and anticipatory governance
- Frontier AI and proliferation; intrinsically dangerous general-purpose technology
- Risk innovation and threat to value; catastrophic versus extinction risk; the risk of not innovating
- Conventional versus unconventional risk mindsets
- "authoritative rather than accurate"
- Alignment as a question of power; truth versus truths
- Red-teaming low-probability, high-consequence risks; human exceptionalism
- The yuck factor and moral risks (organoids)
- AI literacy; human in the loop

**Past technologies.**

- **Nanotechnology:** dominant, used literally as governance history.
- **GMOs:** the failure case.
- **Nuclear:** conceptual.
- **Phrenology and eugenics:** pseudoscience precedents.
- **GDPR:** regulatory spillover.
- **CRISPR and cloning:** via the book show notes.

## What is new or shifted

- **Risk innovation and "threat to value" applied to AI**, with a stated plan to develop this.
- **His existential-risk position, stated publicly:** sympathetic but not signing; catastrophe and value loss favoured over extinction.
- **Doubt that use-based regulation**, his own nanotech-era lesson, fits general-purpose AI.
- **From responsible innovation in principle (B08) to institutional design.**
- **Planning for powerful autonomous, possibly self-aware systems** while keeping superintelligence skepticism, and a revived 2018 idea of partnership with non-human minds.
- **Measured views of leaders:** Altman "sincere", Schmidt corrected, Musk admired and critiqued.

## Most important posts

1. **2023-05-31 existential-risks-of-ai.** Catastrophe versus extinction; risk as loss of value.
2. **2023-07-12 regulating-frontier-ai-models.** The technology/use dilemma; open source versus licensing; the risk innovation nexus.
3. **2023-05-17 ai-senate-hearing-may-2023.** Regulatory design; the transitions initiative; capture.
4. **2023-05-15 erik-schmidt-ai-regulation.** Expertise and publics; nano and GMO lessons.
5. **2023-05-25 leading-ai-expert-says-we-should.** Rogue AI; alignment as power; co-existence.
6. **2023-05-22 can-large-language-models-be-used.** Authority over accuracy; technology–use entanglement.