B09 digest: 2023-05-15 to 2023-07-21#

What this batch is#

This batch has eighteen posts from mid-2023, when AI governance moved from talk to institutions. Maynard responds almost in real time to a run of events:

• Eric Schmidt’s claim that only industry can set early rules;
• the Senate hearing with Altman, Montgomery and Marcus;
• Bengio’s “rogue AI” essay;
• the CAIS extinction statement;
• the EU AI Act amendments;
• the “Frontier AI Regulation” paper and Jeremy Howard’s open-source rebuttal (he contributed to the rebuttal);
• the launch of x.AI.

There is also an education strand and some light posts. None is a Modem Futura episode. The “Moviegoer’s Guide” posts are show notes for his reading of Films from the Future. The 2023-07-08 item is a Gary Marcus cross-post.

Main ideas#

1. Who decides: governance is a field of expertise, and publics have a right to a seat. This is the batch’s backbone.

• Against Schmidt he names the “leave it to the technical experts” mentality, which “never plays out well”.
• His evidence comes from his own career:
• Monsanto’s “it’s complicated, leave it to us” on GMOs “backfired spectacularly”;
• nanotechnology dodged “a bullet” by engaging “early and often”;
• a PCAST member once scoffed at public judgement.
• He makes two claims about participation:
• Rights: “everyone has the right to play some role”.
• Epistemics: people need not understand AI’s inner workings to judge what it threatens that matters to them.
• In the Senate and frontier-AI posts the problem is narrowness: experts in AI but “light on their expertise in governing emerging technologies”. The danger is oversight locked in by “a very small group of players”.

2. Regulatory design: skeptical of AI-specific hard law, doubtful about the technology/use split. Drawing on nano-era policy, he:

• calls hard law “crude, cumbersome” and AI “a moving target”;
• rejects a stand-alone AI agency;
• proposes a cross-agency initiative for “advanced technology transitions writ large”, combining foresight, benefit/risk assessment, public–private partnerships and multi-stakeholder governance;
• repeatedly doubts the nanotech-era mantra “we regulate what people do with the technology, not the technology itself”:
• capabilities “may well erode convenient distinctions between the technology and its uses”;
• predictive policing is a moral hazard “that cannot be addressed by naively separating the technology from its use”;
• frontier AI raises whether “a foundational general purpose technology” is dangerous in itself. His “current thinking lies between” licensing and open source.

This is a careful, hedged move away from his own field’s past consensus.

3. Reframing AI risk: catastrophic loss of value, not extinction. He explains why he did not sign the CAIS statement. He supports AI risk as a “global priority”, but “extinction” is “too narrow and absolute a framing, and too human-centric”.

• Conventional risks (jobs, privacy, education) are “the shavings off the tip of the AI iceberg”. Treating them with “a conventional mindset” is “extremely naive”.
• The deeper threats are to “social, economic, and political structures, systems, and norms”: how we “build our understanding of the world”, autonomy and democracy. They are driven partly by LLMs’ “seductive mastery of language”, in a “highly non-linear technology transition”.
• He brings back risk innovation: catastrophe is when many people “risk losing something that is deeply valuable to them”. The frontier-AI coda names the risk innovation nexus and “threat to value”, extended to “deeply held beliefs, and even self-identity”, and promises more. The frame also counts the risk of not developing AI.

4. Alignment and existential talk as questions of power and values.

• He stays “not a fan” of superintelligence, but accepts “an exceptionally powerful autonomous artificial entity”.
• “Rogue” and “alignment” framings assume AI should “behave”, so he asks “who’s values matter, who decides”.
• Musk’s AGI of “maximal curiosity and truth seeking” fails twice:
• curiosity is not benevolence;
• ultimate social “truth” is usually “an excuse to impose an ideology”.
• The Bengio post ends with an inversion: the real fear may be that AI “will look too much like us”. We should “co-create a shared future” with agentic machines. The Musk post warns against machines that are “deeply inhuman”.

5. Authority over accuracy. The predictive-policing post places the harm in apparent authority. Its tools would be unreliable but “easy to use and, above all, persuasive”, with a bar of “authoritative rather than accurate”. Only guardrails stand in the way. He is firmly opposed on human-rights grounds. The EU Act post carries the manipulation concern into education, where AI tutors might steer learners manipulatively.

6. Education optimism with a human in the loop. He is at his most enthusiastic here:

• ChatGPT designed, tutors and assesses his course;
• the change is on a “printing press” scale;
• AI literacy should be general education, and more than compliance;
• EU rules may stifle personalised learning.

He also notices AI “fine tuning my brain to be a better instructor”. This is an early, positive note on AI-mediated formation that later work complicates.

Concepts appearing#

• Advanced technology transitions
• The leave-it-to-the-experts fallacy; rights-based participation
• The “atomic technologies moment”
• Precision and use-based regulation (doubted)
• Cross-agency foresight; public–private partnerships; lock-in; possible regulatory capture (“not yet”)
• Soft law, agile governance and anticipatory governance
• Frontier AI and proliferation; intrinsically dangerous general-purpose technology
• Risk innovation and threat to value; catastrophic versus extinction risk; the risk of not innovating
• Conventional versus unconventional risk mindsets
• “authoritative rather than accurate”
• Alignment as a question of power; truth versus truths
• Red-teaming low-probability, high-consequence risks; human exceptionalism
• The yuck factor and moral risks (organoids)
• AI literacy; human in the loop

Past technologies.

• Nanotechnology: dominant, used literally as governance history.
• GMOs: the failure case.
• Nuclear: conceptual.
• Phrenology and eugenics: pseudoscience precedents.
• GDPR: regulatory spillover.
• CRISPR and cloning: via the book show notes.

What is new or shifted#

• Risk innovation and “threat to value” applied to AI, with a stated plan to develop this.
• His existential-risk position, stated publicly: sympathetic but not signing; catastrophe and value loss favoured over extinction.
• Doubt that use-based regulation, his own nanotech-era lesson, fits general-purpose AI.
• From responsible innovation in principle (B08) to institutional design.
• Planning for powerful autonomous, possibly self-aware systems while keeping superintelligence skepticism, and a revived 2018 idea of partnership with non-human minds.
• Measured views of leaders: Altman “sincere”, Schmidt corrected, Musk admired and critiqued.

Most important posts#

1. 2023-05-31 existential-risks-of-ai. Catastrophe versus extinction; risk as loss of value.
2. 2023-07-12 regulating-frontier-ai-models. The technology/use dilemma; open source versus licensing; the risk innovation nexus.
3. 2023-05-17 ai-senate-hearing-may-2023. Regulatory design; the transitions initiative; capture.
4. 2023-05-15 erik-schmidt-ai-regulation. Expertise and publics; nano and GMO lessons.
5. 2023-05-25 leading-ai-expert-says-we-should. Rogue AI; alignment as power; co-existence.
6. 2023-05-22 can-large-language-models-be-used. Authority over accuracy; technology–use entanglement.