Late Lessons, Jensen Huang and AI
Markdown version

B09 notes: 2023-05-15 to 2023-07-21 (18 posts)#

Reading notes on Andrew Maynard’s Substack posts in batch B09. Only his own prose is treated as evidence. Quotes are exact, including his original typos and curly punctuation.

Batch context: these posts run from mid-May to late July 2023. That period includes Eric Schmidt’s Meet the Press remarks, the Senate Judiciary subcommittee hearing with Altman, Montgomery and Marcus (16 May 2023), Bengio’s “How Rogue AIs may Arise” (22 May), the Center for AI Safety extinction statement (30 May), the European Parliament’s amendments to the EU AI Act (14 June), the “Frontier AI Regulation” paper and Jeremy Howard’s reply (July), and the launch of x.AI. Maynard answers almost all of these in real time. He takes a break in June.

Note on podcasts: none of these posts is a Modem Futura episode. Three posts (“The Moviegoer’s Guide to the Future” episodes 1–3) are show notes for his own audiobook reading of Films from the Future (2018), repurposed as a Substack podcast. Only the show-note text was read and used. They are listed briefly under LOW.

Relevance summary:

Date Slug Relevance
2023-05-15 erik-schmidt-ai-regulation high
2023-05-17 ai-senate-hearing-may-2023 high
2023-05-22 can-large-language-models-be-used high
2023-05-25 leading-ai-expert-says-we-should high
2023-05-30 organoid-intelligence-ai medium
2023-05-31 existential-risks-of-ai high
2023-06-05 recommended-reading-and-listening none
2023-06-16 chatgpt-prompt-resources low
2023-06-26 wef-top-ten-emerging-technologies-2023 low
2023-07-03 a-new-tech-podcast low
2023-07-07 moviegoer-guide-chapter-1-beginning low
2023-07-08 jumpstarting-ai-governance none
2023-07-10 eu-ai-act-and-education medium
2023-07-12 regulating-frontier-ai-models high
2023-07-14 jurassic-park-moviegoers-guide-to-the-future low
2023-07-16 chatgpt-created-my-course medium
2023-07-19 elon-musk-maximally-curious-agi high
2023-07-21 never-let-me-go-a-cautionary-tale low

HIGH#

2023-05-15 — erik-schmidt-ai-regulation — “Respectfully Erik Schmidt, industry can’t get AI governance right on its own!”#

Provenance. His own prose. The closing section is a transcript of the Meet the Press clip (Schmidt and the interviewer), which is not his text. (He spells the name “Erik” in the title and “Eric” in the body.)

Argument in his terms. - Schmidt says AI is too complex for anyone outside industry, “especially in the government”, to get the first wave of governance right. Maynard grants the complexity and the speed. He is charitable about Schmidt (“I suspect that he has a more nuanced perspective”). But he says the claim misses that beneficial, responsible AI needs expertise industry lacks, “including sophisticated approaches to the governance of emerging technologies.” - He names the pattern: a “leave it to the technical experts” mentality he has met “time and time again”, which “never plays out well”. - Historical evidence from his own career: - GMOs: Monsanto’s “it’s complicated, leave it to us” approach “backfired spectacularly”. - Nanotechnology: the same attitude “risked endangering” it. But “we did dodge a bullet there” by “engaging early and often across a very broad range of domains and expertise”, though it was “touch and go”. Nano “became a team effort.” - Quantum technologies: the same sentiment at a recent AAAS panel. - PCAST: a personal anecdote about briefing the President’s Council of Advisors on Science and Technology, where a member protested that the public cannot decide on technologies “they don’t understand and we do!” - Why broad engagement matters. It builds trust across stakeholders, it deals with social, economic, environmental and health risks early so the benefits do not “slip from our grasp”, and it is a matter of rights: when a technology affects everyone, “everyone has the right to play some role its development and use”. - Who decides. The next few months will set the course. If industry experts without “expertise in navigating exceptionally complex technology transitions in a deeply complex society” decide, “we have a problem.” - Engagement is hard. It needs patience, humility and accessible information. Crucially, people need not understand AI’s inner workings to judge how it might “threaten what’s important to them” (an early form of risk as threat to value).

Concepts and frameworks. The “leave it to the technical experts” / “it’s complicated” fallacy. Early, broad, multi-domain engagement (“early and often”). Trust-building as a condition for realising benefits. A rights-based claim to participation. Complex technology transitions as their own field of expertise. Lessons of past technology transitions.

Analogies. GMOs/Monsanto (a literal historical failure case). Nanotechnology (a literal success case, from his own work). Quantum (the same pattern happening now). Congressional hearings on social media and TikTok (why Schmidt distrusts government). These are used literally, as lessons about governance processes, not about the physical hazards.

On AI companies and leaders. Schmidt is critiqued but treated respectfully. The point is structural: industry expertise is necessary but not sufficient.

On governance. Multi-stakeholder, early and inclusive. Policy makers, civil society and the public belong at the table. He does not reject industry involvement; he rejects industry alone going first.

Quotes. - “This “leave it to the technical experts” mentality is something I’ve bumped into time and time again in my work around emerging technologies — and it never plays out well.” - “The nanotechnology case is an interesting one as, interestingly, we did dodge a bullet there.” - “people don’t need to understand the inner workings of AI to discuss and explore how it might potentially impact their lives and threaten what’s important to them.” - ““It’s complicated” is not an excuse for avoiding engaging with people who have a stake in AI not messing up their lives”

2023-05-17 — ai-senate-hearing-may-2023 — “Some thoughts on yesterday’s historic Senate Judiciary Committee hearing on oversight of AI”#

Provenance. His own prose.

Argument in his terms. - Tone. He is encouraged. He calls it a likely “milestone”, notes that industry was “asking to be regulated”, and praises the senators’ questions as sharp, bipartisan and constructive (“not everything in the US government is broken”). He mentions nostalgia for “past tech-related hearings I’ve been involved in”. - Who was missing. The witnesses (Altman, Montgomery, Marcus) were expert but brought “a relatively narrow perspective”. Absent were expertise in AI governance, agile policy, regulation of emerging technologies, technology assessment, innovation dynamics, public perception and societal impacts, and civil society, including groups working for underrepresented communities. His warning is about lock-in: without broad engagement, oversight pathways may be “defined and locked in by very limited set of ideas coming from a very small group of players”. Even with good intentions, that lowers the chance of beneficial AI and raises the chance of harm. - Reading industry’s call for regulation. He offers two readings. - Sincere: this may be “this generation’s “atomic technologies moment””, when advocates fear the devastation their technology could cause. - Cynical: a move to shape “an exclusionary playing field” that favours first movers and large firms, which “wouldn’t be the first time”. He judges it is not happening “at least, not yet”. Altman “comes across as genuinely sincere”. - Doubts about AI-specific hard law. Hard law is “crude, cumbersome”, needs a well-defined subject (“AI is still a moving target”), and often creates more problems than it solves. - IBM’s “precision regulation” (regulate uses, not the technology) is the familiar approach. But “emerging capabilities may well erode convenient distinctions between the technology and its uses.” - Regulation should be guided by people who know regulation and science and technology policy, and it is slow. - Against a technology-specific AI agency. He recalls the proposed nanotech agency. Such agencies suffer from scope problems, are too narrow, lead to more agencies, focus on the technology rather than its impacts, and muddy jurisdictions. He predicts none in the US soon. - His positive proposal: a cross-agency federal initiative on “advanced technology transitions writ large”. AI is “too narrow and contested to be the sole focus”. The initiative would: - build foresight and benefit/risk assessment capacity; - foster public–private partnerships; - encourage multi-sector, multi-stakeholder governance; - enable stakeholder engagement; - take on new regulation “as and when the need arises”. It would cover AI and LLMs, quantum, brain–machine interfaces, nanotech and next-generation DNA technologies. - Public–private partnerships were the “dog in the night” of the hearing. They must start now, since bringing them in after industry lays the groundwork is “a sure-fire way to lose trust”. In a democracy, progress is “near-impossible” without trust and trans-sector engagement. - The lesson from previous technologies: “if we don’t listen and engage early and often, we’ll come to regret it.” - Addendum. AI threats to democracy and elections are flagged as a coming issue.

Concepts and frameworks. Advanced technology transitions (as a governance category that goes beyond any one technology). The “atomic technologies moment”. Precision regulation / regulate uses, not the technology, and his doubt that the distinction will hold for AI. Hard law versus agile responses. Technology-specific agencies (rejected). A cross-agency foresight and governance initiative. Public–private partnerships. Lock-in of governance pathways. Regulatory capture (“exclusionary playing field”), raised as a possibility.

Analogies. Nuclear: conceptual, the “atomic technologies moment” of advocates fearing their creation. Nanotechnology: literal, the debate over a nano agency and the regulatory lessons from it.

On AI. “Transformative”, yet “only one of a growing number of emerging technologies”. A “moving target” that resists stable legal definition.

On AI companies and leaders. Cautiously generous: he judges Altman sincere and IBM’s approach “more nuanced”, but he names the capture risk.

Quotes. - “it’s possibly this generation’s “atomic technologies moment” where even the most ardent advocates of the technology worry about the devastation that could be caused if things go wrong.” - “I suspect that emerging capabilities may well erode convenient distinctions between the technology and its uses.” - “Here, AI is simply too narrow and contested to be the sole focus of a new initiative.” - “if we don’t listen and engage early and often, we’ll come to regret it.”

2023-05-22 — can-large-language-models-be-used — “Can large language models be used for predictive policing? And if so, should we be worried?”#

Provenance. His own prose. It includes his own prompt template and scenarios. He paraphrases ChatGPT’s (GPT-4) answers rather than quoting them.

Argument in his terms. - Background. Researching Films from the Future (the Minority Report chapter) in 2018 “shook” him. Attempts to predict criminality, from facial features to fMRI, are “still alive and kicking” after phrenology and eugenics. - Why LLMs are tempting here. LLMs predict how an informed person would respond. If they predict words with “uncanny accuracy”, why not actions? He calls this “a seductive line of reasoning” that will gain traction “not necessarily because” it works, but because LLMs “will provide users with the illusion that they can.” - The mechanism. LLMs work on tokens, not “language per se”. So the tokens could be data on human characteristics and behaviour, queried in plain language. With small-data models, this is “not beyond the realms of possibility”, especially if “the bar is predictions that are authoritative rather than accurate.” - Demonstration. He writes the template “Given [context] and [profile] what is the probability of [action] given [opportunity]” and tests it on a child and an ice-cream van, a poor parent and a $50 bill, and tax fraud by executives. GPT-4 gives probabilities with caveats. Pushed further, its guardrails refuse. His point: “the limitation here is not ChatGPT’s ability to make inferences, but the checks and balances”. Remove those and you have a tool that is unreliable, easy to use and “above all, persuasive.” - The risk. The risk is not accurate prediction. It is deployment “despite this”, then spread to trustworthiness scoring, hiring and prime-suspect identification. Companies already sell trustworthiness prediction (“tl;dr — they cannot”). - His normative stance. Firm: “I do not believe this is a path we should go down.” Even accurate prediction would violate human rights: “dignity, equality, and self-determination.” He allows counter-arguments about crime reduction (“far from black and white”). He calls for discussion “now, before technological pathway are locked in”. - Technology–use entanglement. Here the technology and its use are “so deeply intertwined” that responsible innovation must “consider the whole, not just the parts.”

Concepts and frameworks. The illusion of predictive capability. “authoritative rather than accurate”. Moral hazard from separating technology and use. Tokenisation as a general-purpose substrate (language is incidental). Guardrails as the only barrier to misuse. Path lock-in. Human rights (dignity, equality, self-determination).

Analogies. Phrenology, eugenics, facial-feature criminology and fMRI “criminal intent”: literal pseudoscientific precedents. Consumer and voter preference prediction: a literal precursor. Minority Report: implicit, via the book.

On AI. LLMs have “no innate sense of reason” and cannot weigh validity “outside of human-established guardrails”. Yet their outputs have a “seeming-perspicacity” that sometimes feels like novel insight. Persuasiveness and apparent authority matter more than accuracy. This is an early statement of how epistemic authority, not capability, drives harm.

On AI risk. Pseudo-scientific decision tools that gain authority through fluent AI; discrimination; erosion of rights; a “future where machines decide who is good and who is bad”.

Quotes. - “not necessarily because LLMs and associated technologies will be able to accurately predict future behavior, but because they will provide users with the illusion that they can.” - “especially (and worryingly) if the bar is predictions that are authoritative rather than accurate.” - “This is just one example of a moral hazard that cannot be addressed by naively separating the technology from its use.” - “Remove these, and you have a predictive tool which, while not reliable, will be easy to use and, above all, persuasive.”

2023-05-25 — leading-ai-expert-says-we-should — “Leading AI expert says we should we be acting now to avoid future risks of "rogue AIs" — is he right?”#

Provenance. His own prose. Bengio’s two hypotheses and three claims are quoted in italics (Bengio’s text, not his). The name is spelled “Bengio” and “Benjio” in places.

Argument in his terms. He engages Bengio respectfully: “not a fringe scientist or an AI doomsayer”, “he knows his stuff”. He disagrees with much of the reasoning but values the urgency and the conversation it starts. He also notes the updated US National AI R&D Strategic Plan (May 2023), which names AGI existential risk. - Hypothesis 1 (brains are machines, so human-level AI is possible). Reasonable. But he objects to “human-level”: if intelligence is emergent and independent of substrate, machine intelligence may be “human-comparable, or very much non-human”. - Hypothesis 2 (it would surpass humans). Hard to judge because “the nature of intelligence is still deeply contested”. But a machine could outperform humans through the systems it is plugged into, its access to information and its mechanisms for acting. - Claim 1 (superintelligence could be built). “I’m not a fan of the “superintelligence” hypothesis”. He points to his earlier Bostrom critique (recursive self-improvement). He would accept “an exceptionally powerful autonomous artificial entity capable of drawing on a multitude of resources”, “But “superintelligence?” I’m not so sure.” - Claim 2 (rogue AI if not aligned). He questions the word “rogue”. It frames AI as something expected to “behave” and “follow the rules”, which alignment talk reinforces. He reframes: AIs with agency may not do as told, or may act in ways “some people decide are inappropriate”, causing harm to “life, health, income, environment, dignity, pride” (a value-based list). Then the power questions: “who’s values matter, who decides what’s appropriate”, and whether a small group claims the right to define good behaviour. - Claim 3 (rogue AI once the principles are known). He doubts it: there is a big leap from principle to realisation. Innovation is “iterative and messy”, and that messiness allows “self-correcting iteration”, which also makes prediction “fiendishly hard”. - “Genocidal humans.” He rejects the assumption that social deprivation produces comic-book villains, and that fixing its causes makes people behave “reasonably”. - Wireheading. A science-fiction idea rooted in 1950s–60s experiments. It has “worrying overtones of power, control, and subservience” and “may turn out to be very naive”. - Manipulation. He calls Bengio’s concern about AI manipulating humans “a serious concern, and one I’ve written about previously”. He finds the embodiment and evolutionary-pressure arguments less plausible, grounded in “a naive understanding of power dynamics and complex ecosystems”. - Where he agrees. We must think “seriously and creatively” about how to navigate AI powerful enough to disrupt “the human-centric world we’ve constructed”, especially if it becomes self-aware. Near-term issues still matter. He wants a diverse, multi-domain group “red teaming” “low probability but high consequence possibilities”, grounded in how behaviour and agentic technologies interact, and “mindful of the cognitive traps inherent in human exceptionalism, power dynamics, and control.” - Closing inversion. History is full of humans who went “rogue” out of certainty that alignment means denying others their values. So our greatest fear may be that AI “will look too much like us”. We should think “as much about what it means to be an intelligent machine in a future where humans exist” as about being human among powerful AIs. The goal is to cooperate and “co-create a shared future” with agentic machines, not to control a feared creation.

Concepts and frameworks. Substrate-independent, emergent intelligence. Human-level versus non-human intelligence. Superintelligence skepticism (continuing). Critique of the “rogue”/alignment framing as a power and values question. Innovation as iterative, messy and self-correcting. Red-teaming low-probability, high-consequence risks. Human exceptionalism as a cognitive trap. Co-existence and co-creation with agentic machines.

Analogies. Wireheading (science fiction and 1950s–60s brain-stimulation experiments), treated as speculative. Human history of certainty-driven “rogue” behaviour (structural).

On AI. AI may become an agentic, possibly self-aware, non-human intelligence. The concern is power and agency more than “intelligence” as such.

On AI risk. Existential and rogue-AI arguments are respected but judged partly naive. Manipulation is endorsed as serious. Harm is framed as loss of many kinds of value. He wants creative exploration, not dismissal.

Change of view. Continuity with his 2018 superintelligence skepticism. There is a noticeable softening, though: he now entertains powerful autonomous agentic systems and AI self-awareness as things to plan for. He also returns to his 2018 theme of constructive partnership with a different kind of entity.

Quotes. - “I’m not a fan of the “superintelligence” hypothesis” - “here it is worth asking who’s values matter, who decides what’s appropriate, what type of harm we’re concerned with” - “And maybe this should be our greatest fear around advanced AI — that it will look too much like us.” - “how we co-create a shared future, rather than one where we are struggling to control and contain a creation that we feel threatened by.”

2023-05-31 — existential-risks-of-ai — “Why the recent statement on the risk of extinction from AI is important, and why I didn’t sign it”#

Provenance. His own prose. The one-sentence CAIS statement is quoted.

Argument in his terms. This is his clearest statement in the batch on how AI risk should be framed. - Respect for the signatories. They include Bengio, Chalmers, Rees, Revkin, Russell and Altman (“intelligent people and deep thinkers”). He agrees with the “underlying sentiment”: AI is raising “early warnings” of disruption. Much of it will depend less on intelligence per se than on “powerfully and unexpectedly disruptive systems that are deployed with little understanding of possible consequences.” - Against conventional risk framing. Education, job displacement, automation and privacy matter, but they are “the shavings off the tip of the AI iceberg”. Treating them as the whole assumes that the risks of “a highly unconventional technology transition” can be “sliced, diced, and solved, using a conventional mindset”. “Based on all my years of working on the risks and benefits of emerging technologies, this strikes me as being extremely naive.” - What worries him more. Threats to “social, economic, and political structures, systems, and norms”. There is a small but finite chance that AI will profoundly change: - how we interact; - how we “individually and collectively build our understanding of the world”; - how we make decisions; - the mechanisms that keep society going. His drivers: “the seductive mastery of language being exhibited by large language models”, AI interference in decision-making and autonomy, disruption of democratic processes, and AI modulating flows of goods, ideas, information and misinformation. - Complexity. A “highly non-linear technology transition” across “highly integrated complex systems”, where “the seemingly trivial and unexpected” lead to catastrophe. He also allows human ingenuity may succeed, but “there is no guarantee”, and “where the stakes are high and the future is uncertain, it pays to be cautious.” - Why he did not sign. - “Extinction” is “too narrow and absolute a framing, and too human-centric”. Humans are part of wider ecosystems. - Extinction is “vanishingly small”; catastrophe is not. “AI-induced catastrophic risk is far more likely that extinction — and far more worrisome.” - Risk innovation redefinition. Conventional catastrophic risk means many people severely affected. Drawing on his risk innovation work, he extends it to cases where “large numbers of people risk losing something that is deeply valuable to them”: life, health, dignity, autonomy, purpose, family and more. The extinction frame will lose this nuance. - Symmetry of risk. Framing catastrophe as loss of value at scale also captures the catastrophic risk of not developing AI: lost solutions to climate change, poverty, equity and threats to democracy. “AI as solution has to be part of the discourse around AI and risk.” - Bottom line. AI may be “one of the most potent technological disruptors” in a very long time. It would be “foolish” not to worry about existential-level risks. He agrees AI risk should be a “global priority”. But we need framings that open up possibilities rather than “closing down conversations”, getting beyond “speculations around extinction-level events”.

Concepts and frameworks. Risk innovation (riskinnovation.org). Risk as threat or loss of value (catastrophic risk redefined as loss of what people deeply value, at scale). Conventional versus unconventional risks and mindsets. Non-linear transitions in tightly integrated complex systems. Cumulative, trivial-seeming triggers. The risk of not innovating (AI as solution). Caution under high stakes and uncertainty (a precautionary instinct without the label). Human-centrism critique.

Analogies. The CAIS statement’s pandemics and nuclear war are quoted, not his own. His “iceberg” is a metaphor.

On AI. A “highly unconventional technology transition”, disruptive mainly through deployment into complex systems, and through language (“seductive mastery of language”).

On AI risk. Catastrophic, systemic and value-based rather than extinction-focused. Epistemic and social structure risks (how we build understanding, autonomy, democracy) come first. The risk of not developing AI counts too.

On cognition and language. This is an explicit link between LLM language mastery and shifts in how people collectively “build our understanding of the world”.

Quotes. - “I suspect that they’re the shavings off the tip of the AI iceberg when it comes to future impacts.” - “it’s the seemingly trivial and unexpected that are likely to lead to potentially catastrophic outcomes.” - “AI-induced catastrophic risk is far more likely that extinction — and far more worrisome.” - “This, to me, is both too narrow and absolute a framing, and too human-centric.”

2023-07-12 — regulating-frontier-ai-models — “Regulating Frontier AI: To Open Source or Not?”#

Provenance. His own prose, summarising two papers. He discloses that he gave input to Jeremy Howard’s “AI Safety and the Age of Dislightenment” (“myself included”). He says his list of frontier-model dangers mixes the paper’s items with “my own additions”.

Argument in his terms. - Both papers show “growing sophistication” and a multidisciplinary, multi-stakeholder turn, which he calls “a very welcome shift” from early discussions “dominated by people who were experts in AI, but who were somewhat light on their expertise in governing emerging technologies successfully”. - The Frontier AI Regulation paper (OpenAI, DeepMind, Microsoft, Brookings, CSER and others): - Frontier models are “highly capable foundation models that could exhibit dangerous capabilities”. In his words these include global-scale harm to physical, mental and environmental health; tailored persuasion and manipulation; catastrophic harm to infrastructure; and evading human control through deception. - The problems are unexpected capabilities, deployment safety and proliferation. - The proposal is self-regulation plus stakeholder input under government oversight, built on standards, regulator visibility and compliance. - He places it within soft law, agile governance and anticipatory governance. But it “places a lot of power in the hands of frontier AI developers”. - Howard’s reply. Licensing and co-regulation risk “an unsustainable power differential”. Regulating models rather than uses shifts attention from “tangible cause and effect-based risk management” to the “ephemerally speculative”. Howard favours open source and regulating applications. - His position: between the two. The mantra “we regulate what people do with the technology, not the technology itself” is familiar from nanotechnology, where it mostly prevailed (“although not always”), because regulation needs a tangible application. In principle a model is not dangerous until someone uses it. But “I worry that the practice of focusing on applications may prove to be very different than the principle.” - Key question: can a general-purpose technology be intrinsically dangerous? Either its “very existence constitutes a risk”, or its applications are “so diverse, distributed, and hard to police” that they become ungovernable. If so, open-sourcing could release capabilities that are “very hard to put back in”. - Philosophies. Where you stand depends on philosophy. Trust in collective human action and dislike of hierarchy point to open source. Distrust of society with powerful knowledge points to “technocratic governance”. Not a “wicked problem” (over-used), but “certainly a gnarly problem”. - Convergence. Howard’s “openness, humility and broad consultation” “resonates deeply” with his “over 20 years’ work on socially responsive and responsible innovation”. The frontier paper’s stakeholder language points the same way. “No single silver bullet”; “we’re going to have to hash this out together”. - Coda: risk innovation. Both papers deal with risks that “don’t fit neatly into any established risk management paradigm”, which is the problem risk innovation was built for. The “risk innovation nexus” frames risk as “threat to value”. Frontier models could threaten jobs, infrastructure, finance, democracy, social justice, dignity, “deeply held beliefs, and even self-identity”. He promises more on applying the framework to AI.

Concepts and frameworks. Frontier AI. The proliferation problem. Soft law, agile governance and anticipatory governance. Self-regulation plus oversight versus open source plus use-based regulation. The technology-versus-use distinction (“regulate what people do with the technology, not the technology itself”), which he doubts. Intrinsically dangerous general-purpose technology. Technocratic governance versus distributed collective action. “Gnarly” versus “wicked” problems. Socially responsive and responsible innovation. Risk innovation / risk innovation nexus / risk as threat to value.

Analogies. Nanotechnology: literal, a regulatory precedent, but he questions whether it transfers to general-purpose AI. “Out of the bag”: irreversibility.

On AI companies. He is wary of concentrated power in frontier developers and of licensing regimes that entrench incumbents. He does not endorse Howard fully.

On AI risk. A broad value-based list that extends to beliefs and self-identity. Manipulation and deception are among the dangerous capabilities.

Quotes. - “I worry that the practice of focusing on applications may prove to be very different than the principle, which is why my current thinking lies between these two papers.” - “whether a foundational general purpose technology can be so threatening that either its very existence constitutes a risk” - “It’s a methodology which I suspect may have some value here as it frames risk as “threat to value”” - “there will be no single silver bullet to their responsible development or their effective regulation.”

2023-07-19 — elon-musk-maximally-curious-agi — “Will Elon Musk’s "Maximally Curious" AI really turn out to be safe?”#

Provenance. His own prose. He paraphrases the x.AI Twitter Spaces launch, which he re-listened to in full, and quotes Musk’s Luigi/Waluigi line.

Argument in his terms. - Opening. He admits being inspired “for all his many flaws” by Musk’s audacity, and as a physicist he loves the idea of machines helping to understand the universe. But something about x.AI’s safety philosophy “didn’t sit well”. - Musk’s position as he reports it. AGI is inevitable, so there is a duty to shape it. Hard-coding morals is dangerous (the “inverse morality” problem: “If you make Luigi, you risk making Waluigi”). AGI should instead conclude on its own that humanity is worth nurturing, because an AGI driven by “maximal curiosity and truth seeking” will find humans rare and interesting. He traces this to Iain M. Banks’ Culture novels, and notes echoes of free will in Adam and Eve. - On value alignment itself. It is “central to most approaches to long term AI safety”, but there is no consensus on values, and “it’s by no means certain that it would be ethical to constrain an AGI by hardwiring its understanding of right and wrong”. - Critique 1: curiosity is not benevolence. “I’m not sure there is a strong causal link between curiosity and benevolence.” Without a moral framework the opposite may hold. He points to scientists who slipped into unethical work through unchecked “what if?” questions, and to children dissecting living things. Staking humanity on being “the apple of some future AGI’s eye” is a “high stakes gamble”. - Critique 2: “truth” in social matters. Physical truths may exist (he leans deterministic in physics). But social “truth” is different: “an assumption of ultimate social “truth” is more often an excuse to impose an ideology”. This is “truth at any cost”, decided by a small group or machines coded by them. He calls it its own “inverse morality problem”. - Partial sympathy. In a social-media-mediated world, cutting through manipulation is valuable. He shares the concern about “who determines what an AI’s agenda is”, its trustworthiness, “and — as we potentially move toward sentient AI — its very sanity.” We don’t want AIs “as slippery with the truth as politicians and preachers of ideologies.” - But stripping away subtlety, messiness and ambiguity risks “machines that are deeply inhuman”. - Close. It would open “a whole new can of safety worms”. He hopes x.AI will “learn to listen to others, to be humble”. The shared vision of thinking machines exploring the universe with us is still “pretty inspiring”.

Concepts and frameworks. Value alignment (critiqued from both sides). Musk’s “inverse morality” problem, which he extends to truth-seeking. Truth versus truths (epistemic pluralism). Curiosity without moral guardrails. The inhuman machine. Humility and listening as virtues for innovators.

Analogies. Asimov’s Three Laws; Banks’ Culture; Adam and Eve and free will; Hitchhiker’s Guide. These are conceptual and cultural touchstones. Unethical scientific curiosity is a historical pattern, implied rather than named.

On AI. AGI and possibly sentient AI are treated as possibilities to take seriously, without endorsing inevitability. What matters is how its values and epistemics are set, and by whom.

On AI companies and leaders. He is mixed on Musk: admiring of the vision, critical of an “over-simplistic understanding of how the world works”. This fits his recurring pattern of ambivalence towards visionary entrepreneurs (compare Films from the Future’s treatment of Musk).

On cognition and epistemics. He rejects a single “truth” for social matters. He worries about AI becoming an arbiter of truth, while sharing concern about AI shaped to “lie” for agendas.

Quotes. - “I’m not sure there is a strong causal link between curiosity and benevolence.” - “In fact an assumption of ultimate social “truth” is more often an excuse to impose an ideology or worldview on others while diminishing or erasing their own “truth.”” - “we risk creating machines that are deeply inhuman.” - “I am hopeful that the company will learn to listen to others, to be humble, and to grow out of its over-simplistic understanding of how the world works”

MEDIUM#

2023-05-30 — organoid-intelligence-ai — “Is biological computing the future of AI?”#

Provenance. His own prose. One passage is quoted from the Smirnova et al. Frontiers in Science paper.

Argument in his terms. - What the paper proposes. “Organoid intelligence” (OI): networks of brain organoids with embedded sensors and read/write interfaces, perhaps retinal and other “sensory modules”, as an energy-efficient compute substrate for AI. - His response. “Intriguing”. He is especially interested in read/write networks grown into the organoid. He links it to his own 2014 Nature Nanotechnology speculation about 3D-printed artificial brains and heat extraction. - Obstacles. Scale, vasculature, and nutrient and heat removal. - Social side. - Sci-fi “brains in vats” tropes. - Art Caplan’s “yuck factor”. - Questions of pain, self-awareness, consciousness and rights. - LLMs “shaking up underlying assumptions of mind and consciousness”, so the paper’s ethics may need revisiting. - Ethics alone “won’t get us all the way”: there will be “complex risks”, including moral risks, made harder by human-like lab-grown brains. - His call. Now is the time for socially responsible R&D, including “how the field is governed, how democratic decisions are made”, and the boundaries of “appropriate, inappropriate, and definitely not appropriate, research”. He is uncertain about the outcome: potentially “another bumpy” advanced technology transition.

Concepts. Organoid intelligence. Compute substrates. The yuck factor (Caplan). Moral risks. Ethics versus broader risk governance. Democratic decision-making on research boundaries. Advanced technology transitions.

Analogies. His own nanotechnology work on 3D-printed brains (literal, personal). Sci-fi brains in vats (imaginative).

On AI. AI is not tied to silicon. Biological substrates could extend it. This echoes his 2018 view that powerful AI may need different substrates.

Quotes. - “there are too many sci-fi tropes where brains grown in vats do not turn out well to avoid uncomfortable territory here.” - “now is the time to be thinking about socially responsible research and development around organoid intelligence, including how the field is governed, how democratic decisions are made” - “an advanced technology transition that may be transformative in positive ways, but may also turn out to be another bumpy one.”

2023-07-10 — eu-ai-act-and-education — “What does EU Artificial Intelligence regulation mean for AI in education?”#

Provenance. His own prose. Long passages quote the draft EU AI Act and the June 14 amendments (recital 3, amendments 65, 214, 226, 715, 717, 718). Those passages are not his text.

Argument in his terms. - Enthusiasm with caveats. Generative AI offers “unprecedented opportunities around personalized learning, teaching-augmentation, and massively scalable learning platforms.” The “cheating” panic is “a temporary glitch”. - Who he criticises. Well-meaning technologists and thought leaders who think teaching is “still all about learning by rote and an obsession with cheating”. Modern education is “learning as a journey”. - Regulation as the deciding factor. Everything depends on regulation that supports “the creative, innovative, and impactful use of AI in education, rather than stifling it.” The Act will have global reach, as GDPR did. - Reading of the Act’s risk tiers. - Unacceptable-risk category (cognitive behavioural manipulation). He wonders how many edtech innovators are “considering using AI’s persuasive and manipulative power” to steer learners. He calls it “an early warning” to anyone emulating Stephenson’s “illustrated primer” (The Diamond Age). - High-risk category (access, assessment). This should give institutions “pause”. There is ambiguity about how easy it will be “to distinguish between AI systems that enhance education and those that discriminate against some learners”, since the two may “go hand in hand”. - Amendment 214 (AI literacy). A “green light”. He argues universities should treat AI literacy as core “general education” for all disciplines, including navigating risks. But he rejects defining literacy as compliance: AI literacy for “co-creating a more vibrant AI-enabled future” goes far beyond that. - Amendment 226 (emotion inference ban). This may block wellbeing-oriented uses. - Amendments 715 and 717. These could block AI-personalised learning pathways. - Amendment 718 (proctoring and detection). Less contentious, given unreliable detectors, but likely to cause “heartburn”. - Bottom line. Educators should engage policy makers to get guardrails “that support learning without stifling innovation”, because “AI done right could be a game change for how we all learn.”

Concepts. Risk-based regulation (EU tiers). AI literacy as general education (and as more than compliance). Personalised AI learning pathways. Manipulation in educational technology. The ambiguity between enhancement and discrimination. The “Brussels effect” (implicit, via GDPR).

Analogies. GDPR (literal regulatory precedent). The Diamond Age’s illustrated primer (a fictional ideal of an AI tutor, flagged for manipulation risk).

On governance. Here he worries more about regulation stifling beneficial uses than about under-regulation. That is a sectoral stance: educators should shape the rules.

On cognition and formation. He signals, in passing, that AI tutors steering learners could cross into manipulation. This connects his manipulation thread to education.

Quotes. - “the degree to which they are considering using AI’s persuasive and manipulative power to achieve this” - “universities should consider AI literacy as a core set of competencies that come under the “general education” or “general studies” requirements” - “AI literacy in the context of co-creating a more vibrant AI-enabled future goes far beyond understanding how to comply with a specific set of regulations.”

2023-07-16 — chatgpt-created-my-course — “ChatGPT Created My Course – Now It’s Teaching It!”#

Provenance. A short introduction in his own prose, then his own first full draft of the Slate Future Tense article (published pre-edit). Within it, several items are AI-originated and are not evidence of his thinking: - the prompt-engineering definition (“ChatGPT did help out with the definition”); - four of the five learning objectives; - the ambiguity-reduction / constraint / comparative prompting framework (“suggested by the chatbot”); - the RACCCA framework and its acronym (devised by ChatGPT); - at least one exercise used unmodified. The two quoted student prompts were co-devised with ChatGPT. What he chose to add and how he frames the collaboration are his.

Argument in his terms. - How the course began. Prompted by the reports of $300k “prompt engineer” jobs, ChatGPT drafted the course outline in a couple of hours: “better than I could have produced on my own in the time.” He calls it “a profound wakeup call”. - What humans added. Learning objectives ChatGPT missed: “broader societal implications” and “responsible innovation and use”, plus colleagues’ input. - Enthusiasm for AI as tutor. ChatGPT as tutor and assessor (“a remarkably good tutor when it comes to responsible innovation”). Repeatable AI grading that puts process over grades. He acknowledges “obvious dangers” of incorrect information. - Human in the loop. Still essential: “an informed human in the loop” curating and crafting. Professors need new skills. - On himself. He reflects on his own formation: “It’s almost as if ChatGPT is fine tuning my brain to be a better instructor … And messed up as this sounds, maybe it’s a necessary step”. - Scale. Despite limitations, AI could transform learning “at a scale that probably hasn’t been since the invention of the printing press.”

Concepts. Human–AI collaboration and co-creation. Human in the loop. Personalised, self-directed learning. AI as co-instructor. Prompt engineering redefined as a non-technical everyday-language skill.

Analogies. The printing press (a conceptual scale comparison).

On cognition and formation. An early, positive and self-aware note that AI is reshaping his own thinking as a teacher (“fine tuning my brain”). He flags this as slightly “messed up” but accepts it. This matters for tracing how his later concern with AI-mediated formation developed.

Quotes. - “In principle I should be up in arms over this blatant outsourcing of education to AI. The only problem is, I’m the instructor – and I’m loving it!” - “we still need an informed human in the loop to get the most out of these new technologies” - “It’s almost as if ChatGPT is fine tuning my brain to be a better instructor”

LOW / NONE#