T5. Governance, institutions and who decides#

A thematic synthesis of Andrew Maynard’s thinking on how emerging technologies, AI above all, should be governed and by whom. It covers industry’s limits, expertise beyond the technical, public engagement, regulation (EU AI Act, US orders and plans, open versus closed models), principles, universities, agile and anticipatory governance, and his views of AI companies, their leaders and their safety frameworks. It does not compare his work with anything else.

Evidence and provenance - What counts. Only his own prose is evidence. Posts are cited by date and slug, with long slugs shortened. “FFTF p.X” means Films from the Future (2018), by page. - Excluded. - All Modem Futura podcast posts, at the user’s request. This includes 2024-11-06 ai-in-a-world-of-trump, which holds a few paragraphs of his own framing on governance under Trump. - Brad Allenby’s guest posts on regulating AI (2023-08-16 riding-the-ai-tiger; 2023-10-29 category-confusion-complicates-efforts). - The o1-pro report inside 2025-04-06 responsible-innovation-and-ai-acceleration. Only his framing essay counts. - The Deep Research analysis in 2025-03-09 the-hard-concept-of-care. - The ChatGPT role-play in 2023-11-21 ai-and-risk-innovation. - Weaker evidence. 2019-08-13 responsible-innovation (co-written with Elizabeth Garbee). 2026-07-16 orphan-risks-frontier-ai-maynard, rewritten by him from a Fable 5 draft; some apparatus, such as the “safety differential” and “four filters”, may have originated with the model. 2026-09-24 being-an-academic-in-an-age-of-ai, his lecture drafted by Claude and line-edited by him, so its wording is less secure than its ideas.

1. His position in brief#

Eight claims recur across twelve years. They are listed roughly by how central and durable they are.

1. No single actor can govern a transformative technology alone, and industry least of all. This is his most constant position, held firmly and never qualified. - Leaving technology questions to experts is “an abdication of responsibility” (FFTF p.288). - Industry “can’t get AI governance right on its own!” (2023-05-15 erik-schmidt-ai-regulation, title); “our AI future cannot be left solely to AI experts” (2023-11-29 the-year-that-generative-ai-changed-the-world). - In 2026 companies still lack the breadth “to be able to decide for humanity what this future looks like” (2026-09-24).

2. Who decides is the core governance question. Publics have both a right to a say and a distinctive expertise. - “scientists and technologists don’t have a monopoly on expertise and insights” (FFTF p.222). - People “don’t need to understand the inner workings of AI” to judge how it might threaten “what’s important to them” (2023-05-15). - He holds this firmly as a principle but is vague about mechanisms (§9).

3. Governing emerging technologies is a field of expertise in its own right, and AI’s insiders mostly lack it. - Early AI governance was dominated by people “light on their expertise in governing emerging technologies successfully” (2023-07-12 regulating-frontier-ai-models). - Each technology wave tends to “re-invent the wheel” (2023-04-12 navigating-advanced-technology-transitions). - This is firm, and openly self-interested (§9).

4. Regulation is necessary but blunt. It belongs inside a portfolio of adaptive, anticipatory, multi-stakeholder governance, with hard law aimed at specific harms. He is firm on the portfolio and hedged on particular instruments. - Regulations are “a blunt tool for addressing complex and fast moving innovation” (2023-04-10 as-ai-goes-to-washington-whats-being). - He supports criminal law for harmful deepfakes (2024-02-25 ai-rollercoaster-of-a-week). - Apps designed to exploit cognitive biases “can and should be regulated far more than they currently are” (2025-08-31 holding-on-to-our-humanity-age-of-ai).

5. Ethics and principles are necessary but empty unless operationalised. This is firm and unchanged from 2019 to 2026. - Ethics boards can become “a smoke-and-mirrors attempt to mask business as usual under the guise of social responsibility” (2019-04-15 tech-companies-need-an-ethics-reset). - Design principles help “only … if they are actually used” (2026-05-03 are-design-principles-for-responsible).

6. Innovators cannot certify their own responsibility, so there must be checks and balances on “who gets to do what” (FFTF p.166). This is firm, and it sharpens over time. - “permissionless innovation isn’t necessarily reckless innovation”, but “a single innovator cannot see the broader context within which they are operating” (FFTF p.162). - In 2025 he calls this “more relevant now than it was then” (2025-03-02 the-lure-of-permissionless-innovation).

7. AI companies and their leaders are mostly sincere, but myopic, powerful and caught in incentives. That is why self-governance fails. This is the idea that develops most over time: - leaders who lack “social and political savvy” (2016); - capture and “childish irresponsibility” (2023–24); - “sincerity almost always operates inside an incentive field” (2026-07-16).

8. Universities, as accountable public institutions with breadth and freedom, should help fill the governance gap. He increasingly doubts they will. This is his most prominent 2025–26 position, and his least settled. - They could be “an accelerator and a catalyst” (2026-09-24). - So far they have been “followers and users of the technology” (2026-08-30 do-universities-have-a-place-in-bill).

2. Roots: governance thinking before generative AI (to 2022)#

2.1 An insider’s formation#

His governance views come from practice: - first co-chair of the US National Nanotechnology Initiative’s interagency committee on environmental and health implications (FFTF p.215 fn); - chief science adviser to the Project on Emerging Nanotechnologies; - WEF councils, including a 2008 idea for a “Global Institute on Emerging Technology Policy” and a 2010 Davos proposal, with Tim Harper, for a “Global Centre for Emerging Technology Intelligence”; - the Global Future Council on Agile Governance, whose 2018 definition, “adaptive, human-centred, inclusive and sustainable policy-making”, he still uses. He recounts all this in 2023-12-22 un-governing-ai-for-humanity.

This experience gave him his reference cases for the rest of his career: - Nanotechnology as the success. “we did dodge a bullet” by “engaging early and often”. - GMOs as the failure. Monsanto’s “it’s complicated, leave it to us” approach “backfired spectacularly” (2023-05-15).

2.2 2014–2017: responsible innovation, adaptive policy, “ordinary people”#

• The framework. Responsible innovation, in Stilgoe, Owen and Macnaghten’s terms: anticipation, reflexivity, inclusion, responsiveness. “good intentions alone will not ensure we see the benefits”, yet “we can’t afford to slam the breaks [sic]” (2015-01-30 responsible-development-of-new-technologies).
• His community of expertise. Schwab’s Fourth Industrial Revolution “reads as if it were written in a vacuum” of technology assessment and anticipatory governance. Closing the capability–responsibility gap needs partnerships between those with insight into technology and society “and those that call the shots”, and needs “ordinary people” to be “included in defining and helping determine” outcomes (2016-01-11 the-fourth-industrial-revolution).
• His positive model. The 2016 DOT automated-vehicle policy was anticipatory, humble, inclusive and designed to evolve: “a refreshing change from attempting to retrofit existing regulations to new technologies”. Responsible innovation “depends on ensuring everyone potentially touched by a new technology has the chance to be a part of guiding how it’s developed and used” (2016-04-01 will-driving-your-own-car).
• Against scientific elitism. He criticises the view that nonscientists “should revere, but not interfere with, science” (2016-01-12 can-citizen-science-empower). Publics “must have a say” in how neurotechnologies are used (2016-03-31 considering-ethics-now).
• Universities. Public universities fail their public role because “the institution gets in the way” (2016-01-31 public-universities-must-do-more).
• Tech leaders. He admires and corrects them. It is unclear whether Musk’s teams “have the social and political savvy”. Musk should “make friends with people” in responsible innovation and governance, and Maynard adds, “I’m admittedly a little biased” (2016-03-12 itll-take-more-than-tech-for-elon-musk).

2.3 2018: Films from the Future as a theory of who decides#

The book offers no regulatory design, but it sets the logic he uses from then on.

• Power is distributed, and the task is to use it responsibly. “citizens collectively have considerable power over who does what and how” (FFTF p.45). “The challenge we face is not to abdicate power, but to develop ways of understanding and using it in ways that are socially responsible” (FFTF p.44).
• The flaw in permissionless innovation is self-certified responsibility.
• He defines it as “innovation that is conducted in the absence of permission from anyone it might impact” (FFTF p.159).
• The failure is epistemic rather than moral. Nathan in Ex Machina is “tech-savvy, but socially ignorant”, and “sometimes, constraints and permissions are necessary” (FFTF p.163).
• He implicates himself, confessing to rule-bending in the lab and to exhilaration at Musk’s Mars plans. He concedes that caution slows good outcomes (FFTF p.160–166).
• “It’s good to talk.” An AI executive refused to engage affected publics for fear of backlash. Maynard gives three reasons this “‘let’s not talk’ approach” fails: it is risky in practice, questionable ethically (“there’s a moral imperative to engage broadly when a technology has the potential to impact society significantly”), and poor epistemically. So a developer “probably shouldn’t have complete autonomy over deciding what you do” (FFTF p.226–227). This is his first published application of the engagement argument to AI.
• What publics know. When he urged public engagement on nanotechnology at PCAST, a prominent scientist replied that it “sounds like a very bad idea”. His answer: people have “a pretty high level of expertise in what’s important to them and their communities”, though publics should not direct complex research (FFTF p.222), and “we still need technical experts, laws, and policies” (FFTF p.25).
• Unilateral action and collective decisions. Of wealthy, morally certain actors: “where do they get the right to act unilaterally on issues that ultimately impact us all?” A good outcome does not justify the act. “we need better ways of making collective decisions as a society”, and faster (FFTF p.249).
• Markets need steering. Society needs “a system of checks and balances that help steer market forces toward social good” (FFTF p.103).
• Indifference is a governance failure. “the less the majority of us care about this, the more we give those that do care the opportunity to do what they like” (FFTF p.286).
• His role. He adopts Pielke’s Honest Broker: “This is the role I try to carve out for myself in my public-facing work”. Where advocacy is needed it should run through institutions, since “it’s hard to justify one person being the sole arbiter of truth” (FFTF p.246–247).
• Legitimacy and proportion. Legitimate interventions are “socially and politically sanctioned” (FFTF p.269, on geoengineering). AI needs “a degree of anticipation and responsiveness in how these technologies are governed” (FFTF p.174). Policy based on implausible scenarios can itself do harm (FFTF p.205–206).

2.4 2018–2022: operationalising responsibility, and the limits of self-governance#

• Beyond compliance. “It’s no longer enough for tech companies to simply state that their products are ‘safe enough’, and that they comply with relevant regulations” (2018-09-03 tech-companies-need-a-social-risk-reboot). The remedy he offers here is voluntary: toolkits, partnerships with experts, a reset in mindset.
• Ethics needs mechanisms. After Google’s AI ethics council collapsed, he argued that ethics are “worth little without mechanisms and processes”. What works is “internal and industrywide standards, measurable expectations, enforceable checks and balances, meaningful policies, and a culture of social responsibility”, with buy-in from “affected communities, and regulators” (2019-04-15).
• Power. Permissionless innovation is “driven by power, wealth and a lack of accountability” (2019-01-16 responsible-innovation-entrepreneurs). Without inclusive design principles, “the default will a future of the powerful, by the powerful, for the powerful [sic]” (2019-03-31 design-principles-for-de-marginalizing-the-future).
• Culture before rules (co-written). With Garbee he argues that US entrepreneurs share the ideals of responsible innovation but reject its top-down European forms. Responsibility has to grow inside communities: codes of conduct, Asilomar 1975, DIY-bio norms. Top-down governance creates only “crude boundaries”. The chapter still insists, quoting Sarewitz, that leaving these questions to experts is “wrong-headed, futile and self-defeating” (2019-08-13).
• Entrepreneurs outpace institutions. Silicon Valley entrepreneurialism can “overtake the slow, careful development” in established fields. There are “few frameworks currently in place” to respond “in agile and responsive ways”. He admits “we were somewhat naïve” about Neuralink (2020-10-15 the-ethics-of-advanced-brain-machine-interfaces).
• Whose future. “deciding who gets to imagine the future”, or risk “handing our futures to innovators whose vision exceeds their understanding” (2021-09-07 should-we-be-worried-about-elon-musks-tesla-bot).
• Self-governance has a ceiling. As genetic engineering becomes more accessible, “aided and abetted by advances in artificial intelligence”, “a community of well-meaning scientists and engineers are unlikely to be sufficient” (2022-06-13 jurassic-park-dominion).
• A field imbalance. AI ethics guidance has boomed while research on AI’s own risks remains thin (2021-08-03 we-need-to-get-more-innovative).

3. 2023: the governance year#

Most of his explicit AI-governance writing dates from 2023, often in near real time.

3.1 From ethics to risk; no silver bullets#

He declined the FLI pause letter. This was not because he doubted there is “a risk of potentially existential proportions emerging here (I do)”. It was because experience taught him “there are no silver bullets” (2023-04-04 what-are-the-alternatives-to-calling).

His genealogy of how emerging technologies have been governed: - recombinant DNA through bioethics; - the Human Genome Project through ELSI; - nanotechnology and synthetic biology through “agile governance, anticipatory governance, and similar ‘soft law’ approaches”, driven by a “pacing gap” between regulation and capability, since “hard regs are a very unwieldy double edged sword”; - AI, which swung back to ethics because its framers “haven’t necessarily been aware of the sophistication of conversations going on elsewhere”.

His prescription: - “moving away from an ethics framing to one around risk and socially responsible/beneficial innovation”; - agile governance and “‘progressive’ regulation”; - possibly “a world congress” of diverse experts, including the humanities and civil society, to produce an implementable roadmap.

His bottom line: “the biggest risk is not taking action or, worse, assuming no action is needed”.

3.2 Loud voices, quiet voices#

On lawmakers’ tours of Silicon Valley (2023-04-10), he argued that an “insights vacuum” was being filled by “tech companies, interest groups, and loud (but not necessarily informed) voices”. The “quiet voices” go unheard: - transitions and justice scholars; - developers “buried within layers of institutional structure”; - affected communities; - “ordinary people who sometimes have extraordinary insights”.

“I’m not sure I trust people who have an agenda, who can mobilize fast, and who have the ear of decision makers, to get things right.” What was missing was soft law and agile governance: anticipatory, outcomes-focused, co- and experimental regulation.

Universities were not exempt. In rushing to adopt AI they risk becoming “one more player in the tsunami of AI innovation that assumes that responsible innovation is important, but that is somebody else’s problem” (2023-04-18 universities-need-to-be-investing).

3.3 “Industry can’t get AI governance right on its own”#

Eric Schmidt had said only industry could set the first rules. Maynard’s reply (2023-05-15) is his clearest statement on this thread. The “‘leave it to the technical experts’ mentality … never plays out well.” He gives two grounds for participation: - A right. “everyone has the right to play some role its [sic] development and use”. - An epistemic claim. People can judge threats to what they value without technical knowledge.

“‘It’s complicated’ is not an excuse”, he writes, while staying generous to Schmidt (“I get this”).

3.4 Regulatory design: the Senate hearing#

2023-05-17 ai-senate-hearing-may-2023 is his most detailed treatment of regulatory design. - Who was at the table. The witnesses “brought a relatively narrow perspective”. Without wide engagement, oversight might be “locked in by very limited set of ideas coming from a very small group of players”. - Industry asking to be regulated. He offers two readings: an “atomic technologies moment”, or “a cynical move to ensure that AI regulations favor first movers and large corporations”. His verdict: “I don’t think this is happening — at least, not yet”. “Sam Altman comes across as genuinely sincere.” - Hard law. AI-specific hard law is premature: “crude, cumbersome, need a well-defined subject (and I think AI is still a moving target)”. - Regulating uses. He doubts the precision-regulation approach of regulating uses rather than the technology: “emerging capabilities may well erode convenient distinctions between the technology and its uses”. A week later, LLM-based predictive policing becomes “a moral hazard that cannot be addressed by naively separating the technology from its use” (2023-05-22 can-large-language-models-be-used). - Institutions. He argues against a stand-alone AI agency. He proposes instead “a cross-agency initiative that enables pathways to successfully navigating advanced technology transitions writ large”. It would combine foresight, benefit/risk assessment, public–private partnerships, multi-stakeholder governance, and regulation “as and when the need arises”. - Timing. Engagement “is not something that can be introduced once industry has laid the groundwork — that’s a sure-fire way to lose trust.”

3.5 Values, alignment and whose truth#

• Alignment. “who’s values matter, who decides what’s appropriate”, and is it “a relatively small group of people” (2023-05-25 leading-ai-expert-says-we-should)?
• The CAIS statement. He declined to sign. AI risk “should absolutely be a global priority”, but extinction is “too narrow and absolute a framing, and too human-centric” (2023-05-31 existential-risks-of-ai).
• x.AI’s truth-seeking. “an assumption of ultimate social ‘truth’ is more often an excuse to impose an ideology or worldview on others”. The real issue is “who determines what an AI’s agenda is” (2023-07-19 elon-musk-maximally-curious-agi).
• Not like nuclear. Unlike “contained” nuclear programmes, AI is “hidden, dispersed, readily accessible” (2023-07-25 oppenheimer-and-ai).

3.6 The EU AI Act, seen from education#

His one detailed reading of the Act (2023-07-10 eu-ai-act-and-education) is more wary of over-regulation than most of his writing. He expects GDPR-style reach beyond the EU. He welcomes the ban on “cognitive behavioral manipulation” and the literacy mandate, though literacy must go beyond compliance. He worries that the high-risk rules on admissions, placement and proctoring, and the ban on emotion inference in schools, could block personalised learning. What matters is regulation that supports “the creative, innovative, and impactful use of AI in education, rather than stifling it”.

3.7 Open versus closed#

His one sustained treatment of open versus closed models is a comparison of the “Frontier AI Regulation” paper with Jeremy Howard’s rebuttal, to which he contributed (2023-07-12). - The frontier paper relies on industry self-regulation “(with multi-stakeholder input)” plus government oversight. He calls it sophisticated, but notes it “still places a lot of power in the hands of frontier AI developers”. - Howard’s rebuttal warns that licences to develop could create a privileged class of companies, and argues for regulating applications. That fits the nanotech-era mantra “we regulate what people do with the technology, not the technology itself”. - His own view. The principle is sound, but “the practice of focusing on applications may prove to be very different than the principle”, so “my current thinking lies between these two papers.” - The open question. Can “a foundational general purpose technology” be dangerous in itself, so that open-sourcing it becomes irreversible? He calls this “a gnarly problem” that “we’re going to have to hash this out together”, and offers risk innovation as a way in. - Later mentions. He barely returns to it. In late 2023 governance had consolidated around closed frontier models, with firms holding “an outsized influence in guiding the framing of regulations that seemingly favor commercial leaders in the field” (2023-11-18 sam-altman-openai-impacts). In 2025 the Action Plan’s open weights come “with strings attached” (2025-07-23 americas-ai-action-plan).

3.8 Engagement, nano lessons and the research agenda#

• PCAST on engagement. He welcomes PCAST’s call for “intentional two-way engagement with key communities”. Calling AI too complicated for non-experts risks “hubristically ignoring the very communities many scientists and technologists claim they are working to help”. His tools are Participatory Technology Assessment, Public Interest Technology and NISE Net. Developers “tout the importance of public engagement in the abstract, but strenuously resist it in practice” (2023-09-04 why-public-engagement-is-so-important).
• A research agenda. His NSF comments list “Effective governance of advanced technology transitions, spanning the spectrum of public engagement, soft law, agile governance, hard-law regulation” as a research domain (2023-09-25 building-a-better-futures-tough).
• Nano lessons, with Sean Dudley, are process lessons. AI “is still being driven by a small group of experts and companies who believe that they have all the understanding they need”. GMOs were “a masterclass in how naivety, hubris, greed, and a lack of broad engagement” block progress, and for AI “the stakes here are far higher” (2023-10-02 responsible-ai-lessons-from-nanotechnology).
• Against Andreessen. His defence of sustainability, trust, ethics and responsible innovation rests on “who decides who will suffer and who will thrive” (2023-10-19 marc-andreessen-ditch-sustainability).

3.9 Industry bodies, the Executive Order, OpenAI’s crisis, the UN#

• The Frontier Model Forum’s safety fund risks “addressing unconventional challenges through rather conventional thinking” (2023-10-25 10-million-for-ai-safety-research).
• The Biden Executive Order. A serious start, in a country with “an ethos of go fast, break things, and don’t worry about the consequences”. But it was naive (2023-10-30 white-house-goes-all-in-on-responsible-ai):
• “an over-emphasis on following the lead of large tech companies and their agendas”;
• little diversity of expertise;
• no evidence of learning from past transitions.

He warns of “the dangers of regulatory capture” and “vested interests wrapped in the clothing of social responsibility”. Justice experts should be “partners … and not just add-ons”, and universities should fill the gaps. - Altman’s ouster. Altman was probably policymakers’ “shiny person”, and the crisis might offer “a chance to take a breath (a pause even)” (2023-11-18). Days later he applies his Risk Innovation Planner to OpenAI’s board. It is “agnostic to particular worldviews, ideologies, or ethics”, but forces firms to see how they threaten what others value (2023-11-21, framing only). - Good intentions. The belief that tech leaders’ good intentions will deliver beneficial AI is “sheer fantasy”. He welcomes the Pope’s call for formal oversight (2023-12-15 pope-francis-artificial-intelligence). - The UN Advisory Body’s interim report gets his warmest governance review. He places it in his WEF lineage and endorses: - multi-stakeholder, rights-anchored, agile governance; - action so “that purely market driven AI development doesn’t, by default, favor privileged communities”; - the conclusion that “AI for good cannot simply be left in the hands of tech entrepreneurs and AI experts”.

He also praises it for not getting “tangled up in speculative long term AI risks” (2023-12-22).

4. 2024: safety is social; leaders under scrutiny#

• Concentration, and partnership. He flags “the outsized influence of unelected billionaires” (2024-01-17 ai-global-risks-2024-wef-davos). The next day he greets the ASU–OpenAI collaboration as “a tsunami of creativity and innovation”, trusting that “both ASU and OpenAI are being diligent here” (2024-01-18 asu-openai-collaboraton). The pairing is examined in §9.
• Two instruments at once. He signs the deepfake letter calling for criminal penalties and developer liability, although “I tend not to sign open letters like this as they’re often deeply naive when it comes to how new technologies are governed.” He once trusted people’s common sense to spot fakes and is now “far less sure”. In the same post, after Gemini’s stumble, he says it is “far more important to be agile and responsive when things do go awry” (2024-02-25).
• Leaders. Neuralink is “caught up in a cult of personality” (2024-01-30 first-in-human-trial-of-neuralink-bci), yet critics discount its “reality distortion field” “at their peril” (2024-03-21 elon-musks-neuralink-plays-mind-games). Musk’s deregulatory rhetoric “isn’t wholly wrong”, but Maynard hopes for “an informed counterbalance” (2024-08-04 7-key-takeaways-from-elon-musk-and-lex-fridman). “tech bros forget everything they ever knew about the Dunning-Kruger effect when it comes to governance and policy” (2024-12-29 fantasy-top-ten-lists-2025).
• Fields too narrow. Responsible innovation and technology governance are among the fields he now calls too narrow. The transition needs “pilots — whether they are institutions, communities, or individuals” (2024-03-31 we-have-a-technology-problem-and). For AI assistants, “we cannot simply codify AI ethics within a neat set of principles” (2024-04-24 navigating-ethics-of-advanced-ai-assistants).
• Who should do futures thinking. Future-building “is a collaborative effort — not something that should be left to an elite group of thinkers and innovators”. It belongs in public universities, which have “a level of accountability that does not exist in private universities, corporations, government or other organizations” (2024-04-28 beyond-the-future-of-humanity-institute).
• His sharpest words for OpenAI. On the Johansson voice: “disconnects between the talk around responsible innovation, and a reality that sometimes seems childish irresponsibility”; “disdain for the dignity and rights of individuals” (2024-05-21 openais-problem-with-the-movie-her).
• Safety is social. This is the core 2024 statement (2024-06-20 ilya-sutskevers-safe-superintelligence-rethink).
• “there is no such thing as absolute safety”; harm is “a social construct, not a technological one”.
• Safe Superintelligence ignores “the social side of who decides what ‘safe’ means”.
• “the biggest threat to building acceptably safe technologies is the blinkered assumption that absolutely safe technologies are possible through science and technology alone”.
• Labs, governments and power.
• Labs race toward human-like AI “with very little governance overseeing the subtler potential social implications” (2024-06-30 seth-is-conscious-ai-possible).
• “the irresistible pull toward power and influence is likely to move even governments toward the manipulation quadrant without appropriate checks and balances in place” (2024-07-13 ai-choice-engines-sunstein).
• He names complacency, including when people “simply allow technology entrepreneurs to make critical decisions”, as a governance failure (2024-08-25 advanced-technology-transitions-model).
• Democracy and benevolent persuasion. He praises OpenAI’s system cards as “a sophisticated approach”. But AI nudging populations toward good ends raises the question: “who decides what is good for society? … And where does democracy fit” (2024-09-01 is-chatgpts-new-voice-mode-dangerously-persuasive). Against Amodei: “who decides what is ‘normal’ and what needs to be ‘fixed’”. The deficit model was “debunked decades ago” (2024-10-13 amodei-machines-of-loving-grace).
• Beyond guardrails. On companion bots: “I’m not convinced that guardrails alone are the answer”. He raises “pausing — or even rethinking” the development of chatbots “designed to use and even exploit how we feel” (2024-10-27 personal-ai-chatbots-and-stochastic-agency).
• Oversight as part of the picture. Citing Marchant’s pacing problem, he treats oversight as “only part of the highly complex landscape” (2024-12-17 navigating-the-challenges-and-opportunities-of-adv).

5. 2025–2026: permissionless politics, limits of control, the turn to institutions#

5.1 Three actors, none adequate#

His 2025 frame (2025-01-07 universities-need-to-step-up-their-agi-game) assesses three candidates: - AI companies. “responsible as these companies claim to be (and I think they’re trying hard), they still lack the breadth of vision and understanding”. - Democratic governments. They have the mandate but “lack the imagination, vision, or agility”. - Research universities. They could lead but are “mired in tradition, convention, and self preservation”. “I’m not convinced they are” up to it.

He proposes philanthropically funded work on where we live, what we do and who we are: “we’re talking billions of dollars here”.

5.2 Permissionless innovation wins politically#

• Out of fashion. Responsible AI is “going out of fashion at lightening speed [sic]” as “permissionless innovation” takes its place. He praises the Evo 2 team’s restraint and invokes Asilomar 1975. He criticises a game of “go fast and break things in the hope that someone else will clean up the mess” (2025-02-23 evo-2-dna-ai).
• Re-endorsing the 2018 critique (2025-03-02). His objection is to fixing harms after the fact “rather than anticipating them and navigating around them”. His footnotes add:
• A reversibility test. Experimenting in reversible, linear systems is fine, but “I’d put breaking people, governance, society, and the planet, in this category!”
• Musk revised. He revises his 2018 view of Musk and calls DOGE “a rather naive and uninformed application of permissionless innovation”.
• Precaution. Transatlantic debates over it are “fraught with misunderstanding, misinterpretation”.
• The Action Plan. “Build, Baby, Build” is his most direct critique of US federal AI policy. He wrote it deliberately without AI (2025-07-23).
• The plan “prioritizes power before people”. Its “try-first” culture is “essentially an ask forgiveness rather than permission policy”. Responsible innovation “is actively portrayed as a barrier”.
• His model is early-2000s nanotechnology policy, “a balanced, proactive, and above all collaborative approach that placed wellbeing above dominance”.
• He credits regulatory sandboxes, if “carefully (and responsibly) executed”. He values international collaboration over dominance.
• He ends with irony: “we won’t know whether removing them is a really bad idea or not until we try”. A footnote adds that evidence of “what will happen with the guardrails down” already exists.

5.3 Doubting responsible innovation; care#

• Futility. AI 2027 prompts his most self-implicating doubt: a future “where current efforts to develop artificial intelligence responsibly seem futile”, because governance depends on “processes that are constrained by human timescales”. In a US–China race, responsible innovation would fare “(not that well is the short answer)” (2025-04-06, framing only).
• Care. He proposes a “hard” concept of care as “something of substance that can be used as a basis for policy, governance, and decision-making”. In his summary it is set against control and against aggregate-only thinking (2025-03-09).

5.4 Governing agents and designed manipulation#

• Agents. For agentic AI, “we’re not even sure yet how to formulate the problem”. He welcomes Kasirzadeh and Gabriel’s dimensions as a way to avoid treating agents “as a one-size-fits-all technology”. He adds that their framework misses AI’s effects on beliefs and behaviour (2025-05-04 an-important-new-model-for-guiding-agentic-ai-oversight).
• A two-track position (2025-08-31):
• Designed manipulation should be regulated “far more”.
• But “The AI genie is out of the bottle” and development is global, so regulation and responsible innovation “are likely to run into challenges”. They need augmenting by channelling innovation, “much as a flood can’t be halted, but it can be directed”, and by building everyone’s capacity to thrive.
• Companies should engage “people who actually know about responsible innovation rather than people simply claim they know”.
• “This isn’t just a problem for companies to fix, or for policy makers to govern.”

5.5 Institutions as deployers#

Universities that supply ChatGPT EDU and encourage its use take on “a social, moral and (I would assume) legal duty of care” (2025-11-09 universities-chatgpt-mental-health). - What fails. Warnings, bans (which need surveillance), restrictive guardrails that risk “neutering” models, and literacy classes that “risk becoming performative”. - What he offers instead. Dialogue, trust and safe spaces: governance through relationship.

5.6 2026: frameworks, principles and the governance gap#

• Anthropic’s constitution. He calls it “a necessary step”. For something without precedent, the ways we ensure it serves human flourishing must “move beyond easy analogy” (2026-01-22 think-you-know-ai-think-again). Later he asks companies for “character constancy” across model updates (2026-04-26 why-im-falling-out-of-love-with-claude).
• Principles at home. He helped write ASU’s AI design principles. ASU Atomic launched without consulting faculty whose material it used. The principles could have helped, “But only, of course, if they are actually used”, and “there’s often a gap between what is legally allowed, and what is good practice” (2026-05-03).
• Oversight and acceleration. “if AI was a drug, we’d probably be thinking carefully about how access is overseen”. Risk conversations at his own institution are “drowned out by the the clarion call of AI acceleration [sic]” (2026-05-10 do-not-do-this-with-ai).
• How companies choose their risks (2026-07-16, mixed provenance). This is his most developed account of company governance.
• Safety frameworks have become “the de facto governance layer for frontier AI”, filtering risks by measurability, severity, evidence and competitive cost.
• The cause is structural, not cynical: “sincerity almost always operates inside an incentive field”, so “remedies have to change what competition rewards”. He is “not optimistic” that regulation alone will close the gap.
• He proposes public orphan-risk registers, aperture logs, structured outside engagement, and regulator-required disclosure of how risks are selected, to “ensure greater visibility around who is deciding what matters, and on what grounds”.
• From the Garbee work: “you do not hand it a compliance duty; you show it a threat to something it values”. Frontier labs are “mission-driven, often allergic to imposed process”.
• The channels that turn stakeholder harm into cost for firms are not equally open to everyone.
• Universities as followers. Responding to Bill Gates, he describes universities as “followers and users of the technology” and “guardians of the past more than leaders toward the future”. He still “believes fiercely” in their public responsibility, but adds: “Sadly, this has been my experience so far” (2026-08-30).
• Developers and risk. Developers act “as if they’re the first people to notice” long-known risks, saying “they should go slower, and not doing so” (2026-09-15 will-ai-really-kill-us-all).
• The governance gap, whole (2026-09-24):
• Companies. “the Anthropics and the OpenAIs and the Xs” lack the breadth “to decide for humanity”.
• Governments. “if anybody has seen an example of a government that moves really fast and really smartly, please do let me know”.
• Civil society. It “simply doesn’t have the wherewithal to lead here”.
• Publics. They are “critically important”, but “you cannot hand a problem of this magnitude over to everyday people”.
• Universities. The gap “can only be filled by universities and academics”, and they could be “an accelerator and a catalyst”, yet self-preservation holds them back.
• Pause and race. Assuming, possibly wrongly, that powerful AI is inevitable: “We can’t stop it. We can’t pause it.” He also criticises the race logic “if we don’t go fast, somebody else will”, and ASU’s “jetpack for the mind”.

6. Key concepts#

7. Constants and shifts#

What stays constant (2015–2026): - No single actor governs well alone, and technical insiders are the most worrying claimants. - Engagement is non-negotiable, and the deficit model is wrong (FFTF p.222; 2024-10-13; 2025-05-25 why-parasocial-communication-is-important). - He prefers adaptive, multi-stakeholder governance to AI-specific hard law. - He gives a non-demonising account of developers. - He is sceptical of principles that are not put into practice. - He is pro-innovation. Governance should steer, not stop: “Not that I think this should be taken as an excuse not to build” (2020-08-28 navigating-the-complex-world-of-advanced-brain-machine-interfaces), through to channelling a flood that “can’t be halted” (2025-08-31).

How it shifts:

1. From persuading leaders to explaining structures. - 2016–2019: leaders lack social savvy, and the remedies are voluntary: partner with experts, use toolkits, reset culture. - 2023: capture and the “shiny person”, while he still judges Altman sincere and capture “not yet” real. - 2024: named misconduct. - 2026: sincere people inside an incentive field, remedied by changing “what competition rewards” and making risk selection visible.

2. Falling confidence in responsible innovation. From endorsed framework (2015–19), to “fiendishly hard to operationalize” (2023-05-05 us-white-house-embraces-responsible-innovation), to too narrow (2024), to out of fashion and possibly “futile” (2025), to something to be augmented by care, capacity and channelling.

3. Falling confidence in government. From DOT as a model (2016), to a proposed federal cross-agency initiative (2023), to governments lacking “imagination, vision, or agility” (2025), to doubt that any moves “really fast and really smartly” (2026). In my reading the causes are both political (the 2025 permissionless turn) and structural (the timescale mismatch).

4. A cautious turn toward targeted hard law. No AI-specific regulation “at this point” (2023); criminal law for deepfakes (2024); more regulation of designed manipulation (2025); disclosure mandates, while “not optimistic” about regulation alone (2026).

5. A narrowing pause. He declined the pause letter (2023-04); “a pause even” (2023-11); a conditional pause on emotion-exploiting companion bots (2024-10); “We can’t pause it” (2026-09). A pause is thinkable for one class of design, not for the technology as a whole.

6. Universities, from sites of responsibility to hoped-for governance actors. From institutional barriers to public service (2016), to practising responsible AI (2023), to accountable public universities for futures thinking (2024), to “step up their AGI game” and duty of care (2025), to the only gap-filler in sight but a follower (2026). His hope in universities and his disillusion with them grow together.

7. A narrower role for publics, most recently. He moves from “everyone has the right to play some role” (2023) to “you cannot hand a problem of this magnitude over to everyday people” (2026). This is consistent with the book’s limits on public direction of research (FFTF p.222). The emphasis has still shifted, from publics as co-deciders toward publics as essential inputs to institutions led by experts.

8. Connections to his other threads#

• Risk as a threat to value and risk innovation. This is where his governance thinking is most original.
• Because risk is socially defined, “who considers what a risk (and to whom)” (2016-03-31) and who decides what is “safe” (2024-06-20) are governance questions.
• The value frame gives him a lever suited to entrepreneurial culture: show firms threats to what they value rather than impose compliance duties (2019-08-13; 2026-07-16).
• Orphan risks move from a tool for startups (2018) to a public accountability mechanism (2026).
• Complexity, irreversibility and speed. These justify his instruments: agile and soft law against the pacing gap, checks and balances against “playing with fire in a world made of kindling” (FFTF p.167), and the reversibility test. They also explain the timescale mismatch that makes responsible innovation “seem futile” (2025-04-06).
• Manipulation, cognition and formation. These decide what regulation should target: designed manipulation (2025-08-31), the EU ban on cognitive behavioural manipulation (2023-07-10), companion bots (2024-10-27), and epistemic agency as an unowned risk (2026-07-16). Benevolent persuasion turns governance into a democratic question (2024-09-01).
• Being human. Intrinsic technologies may change who we are “without our agreement or permission” (2024-01-01 the-future-of-being-human-in-2024), a consent argument framed through identity.
• Plausibility. It sets his governance priorities: he rejects policy built on improbable scenarios (FFTF p.205–206), praises the UN report’s restraint, and treats AI 2027 as “speculation — no more”.
• Justice. Markets default to the privileged (FFTF p.103; 2023-12-22). “who decides who will suffer and who will thrive” (2023-10-19). Channels for making harm count are unequal (2026-07-16).
• Temperament. The Honest Broker stance explains his reluctance to sign open letters and his preference for convening over campaigning.

9. Tensions, ambiguities and gaps#

1. Engagement lacks mechanisms. He names Participatory Technology Assessment, consensus conferences and Public Interest Technology but never works any of them through for AI. His concrete proposals put experts at the centre: a “world congress” (2023-04-04), a federal cross-agency initiative (2023-05-17), and university initiatives costing billions (2025-01-07). By 2026 publics cannot be handed the problem. My interpretation: his critique of expert monopoly targets technical experts, and his remedy often adds another class of experts, in governance, responsible innovation and transitions. That is his own field, and he admitted the bias in 2016.

2. He partners with industry and criticises its influence. Several enthusiasms sit awkwardly beside his warnings about capture and deference to big tech (2023-10-30): - the ASU–OpenAI collaboration (2024-01-18); - ChatGPT Enterprise as enabling (2023-08-29 chatgpt-enterprise-game-changer); - praise for OpenAI’s system cards (2024-09-01).

Universities’ dependence on vendors becomes a governance problem for him only with the duty-of-care post (2025-11-09) and the ASU Atomic critique (2026-05-03). He does not explain how the partnership and the critique fit together.

3. His preferred instruments later erode. In 2023 he favoured soft law and agile governance and called industry-led frontier governance sophisticated, while noting the power it gave developers. In 2026 he documents how voluntary company frameworks soften under competition, and turns to disclosure mandates. He does not reflect on how much his 2023 portfolio relied on good faith that competition would wear away.

4. Inevitability versus steering. “We can’t pause it” sits beside his criticism of race logic. If there can be no pause and the race is real, where does the power to steer come from? His answer, universities and personal capacity to thrive, remains aspirational.

5. Permission in education. He opposes permissionless innovation yet backs “permission to play” for students and agent-built courses (2025-03-15 ai-playgrounds-in-higher-education; 2025-03-27 ai-agent-creates-online-course-in-minutes). The reversibility test partly reconciles these. He only turns the test on his own institution’s rush to adopt AI in 2026 (2026-05-10).

6. Questions left open. - Technology versus use, and open versus closed, remain “between” (2023-07-12). - He asked in 2023 who should write an AI’s values (2023-05-25; 2023-07-19), but did not ask it of Anthropic’s constitution (2026-01-22). - “Who decides what ‘safe’ means” gets a mechanism only with the 2026 disclosure proposals.

7. Precedent versus novelty. He grounds his model in nanotechnology policy and Asilomar (2025-07-23; 2025-02-23), yet says AI “defies analogy” and that past frameworks yield “categorical errors” (2026-01-22; 2026-09-24). Implicitly, the precedent is a template for process (engagement, collaboration), not substance. He never says so.

8. Missing topics. His own prose barely covers liability (beyond deepfakes), antitrust and compute concentration (beyond the WEF note on “unelected billionaires”, 2024-01-17), AI at work, international governance after 2023, the EU AI Act beyond education, or company safety frameworks before 2026. He is strongest on who is at the table and which risks are governed, and weakest on legal architecture.

9. Tone toward leaders follows events. Musk is admired, corrected, then called “naive and uninformed” (2025-03-02). Altman goes from “genuinely sincere” (2023) to “childish irresponsibility” (2024). Amodei is read generously but corrected. Anthropic’s constitution is received respectfully (2026-01-22) before its model character is criticised (2026-04-26). The 2026 “incentive field” account reconciles these swings, since individuals vary and the structure does not, but he has not presented it as a synthesis of them.

10. Provenance of his most operational proposals. The safety differential, the filters and the register and aperture log come from a paper he says Fable substantially shaped (2026-07-16; 2026-07-19). The long-standing framing beneath them is his own: threat to value, orphan risks, and the Garbee “compliance duty” lesson.

10. The most important sources for this thread#

1. FFTF (2018): pp.159–168 (permissionless innovation), pp.218–229 (engagement), pp.244–249 (unilateral action, collective decisions), pp.286–288 (indifference, abdication).
2. 2019-04-15 tech-companies-need-an-ethics-reset: operationalising ethics.
3. 2023-04-04 what-are-the-alternatives-to-calling: the pause letter, ethics to risk, governance lineage.
4. 2023-04-10 as-ai-goes-to-washington-whats-being: loud and quiet voices, soft law.
5. 2023-05-15 erik-schmidt-ai-regulation: industry cannot govern alone.
6. 2023-05-17 ai-senate-hearing-may-2023: regulatory design, technology versus use, capture.
7. 2023-07-12 regulating-frontier-ai-models: open versus closed.
8. 2023-10-30 white-house-goes-all-in-on-responsible-ai (with 2023-11-18 sam-altman-openai-impacts): deference to big tech.
9. 2023-12-22 un-governing-ai-for-humanity: his agile-governance lineage.
10. 2024-06-20 ilya-sutskevers-safe-superintelligence-rethink: safety as social.
11. 2025-03-02 the-lure-of-permissionless-innovation: the reversibility test.
12. 2025-07-23 americas-ai-action-plan: “power before people”; the nanotech precedent.
13. 2025-08-31 holding-on-to-our-humanity-age-of-ai: regulate designed manipulation; limits of control.
14. 2026-07-16 orphan-risks-frontier-ai-maynard: frameworks as governance layer; disclosure (mixed provenance).
15. 2026-09-24 being-an-academic-in-an-age-of-ai (with 2025-01-07 universities-need-to-step-up-their-agi-game and 2026-08-30 do-universities-have-a-place-in-bill): the governance gap and universities.