Late Lessons, Jensen Huang and AI
Markdown version

B07 — reading notes (2023-01-31 to 2023-04-12, 8 posts)#

Batch context: this batch covers his move from Medium to Substack. The first two posts are ChatGPT experiments from the Medium period (later carried over to the Substack archive). The April 2023 posts are his first pieces written for Substack, in the weeks after GPT-4 became available in ChatGPT and just after the Future of Life Institute’s “pause” letter. There are no Modem Futura podcast posts in this batch, so nothing needed skipping on that ground.

Date Slug Relevance Provenance in brief
2023-01-31 can-chatgpt-take-the-pain-out-of-annual-academic-reviews-3aa9ab32b0f0 medium His intro, his prompts and his postscript; the bulk is ChatGPT output
2023-03-29 maximizing-value-in-the-evolving-landscape-of-tenured-tenure-track-faculty-1f3ad6183782 low Short intro and prompts are his; about 90% is ChatGPT (GPT-4) output
2023-04-04 what-are-the-alternatives-to-calling high His own prose (a lightly edited thread replying to Gary Marcus); one quoted LeCun tweet
2023-04-04 welcome-to-the-future-of-being-human medium His own prose (Substack launch note)
2023-04-05 can-chatgpt-adversely-impact-mental high First half his prose; second half a ChatGPT-written “editorial” (excluded as evidence)
2023-04-07 cifar-launches-bold-new-lines-of low His own prose (announcement)
2023-04-10 as-ai-goes-to-washington-whats-being high His own prose, with short quotes from the Washington Post
2023-04-12 navigating-advanced-technology-transitions high His own prose

2023-04-04 — what-are-the-alternatives-to-calling — “What are the alternatives to calling for a pause on giant AI experiments?”#

Relevance: high

Provenance: His own prose. The core is described as “a lightly edited version” of his reply on X to Gary Marcus (the thread also tagged Geoffrey Hinton), which is also his writing. It embeds one Yann LeCun tweet (not his) and a Midjourney-generated header image.

Argument, in his terms: - He has not signed the FLI open letter calling for a six-month pause on training systems more powerful than GPT-4. The reason is not that he thinks the risk is small: he says outright that he thinks a risk “of potentially existential proportions” is emerging. His doubt is practical. His experience of “the messy nexus of risks and benefits around emerging technologies” leaves him unconvinced that a pause would have the intended effect. He states this firmly but briefly and doesn’t expand on why a pause would fail. - In contrast to “increasingly pointed rhetoric” from AI experts who dismiss the fears (he names LeCun), he says LLMs point to a “deeply complex risk landscape” that we are unprepared to navigate. He defers a full account of that landscape to “another time”. - “No silver bullets.” Twenty years of work on the risks and benefits of advanced technologies have left him certain of one thing only: there are no silver bullets. We do, though, “have some sense of how to frame the questions”. - First move: reframe away from ethics. He wants to shift from an ethics framing “to one around risk and socially responsible/beneficial innovation”. He gives a short history of how the governance of emerging technology has developed: - recombinant DNA: a bioethics approach “worked”; - the Human Genome Project: a growing emphasis on ELSI (ethical, legal and social implications); - nanotechnology, synthetic biology and others: a shift to agile governance, anticipatory governance and “soft law”. This shift came because a growing “pacing gap” between hard regulation and technological capability called for alternatives, and hard regulation is “a very unwieldy double edged sword”. - He places his own risk innovation work (“aimed at supporting agile decisions around intangible risks”) and responsible innovation in this lineage. - AI has pulled the field back to ethics. Examples include the 2017 Asilomar AI Principles; he notes that he attended the 2017 Asilomar meeting. His explanation is that the people framing the question, “while well-intentioned”, haven’t been aware of “the sophistication of conversations going on elsewhere”. With LLMs “the rubber is hitting the road” on decisions, and ethics alone can’t guide them: it helps sort right from wrong but gives no practical framework for getting to safe and beneficial technologies. - What is needed: agile governance, “progressive” regulation (which he defines as innovative regulatory approaches) and new approaches to societally beneficial innovation. He groups all of these under “navigating advanced technology transitions”, and says they are needed “like, yesterday!” - Concrete proposal, offered tentatively (“one possible way forward (but by no means the only one)”): rapidly convene an international “world congress” of diverse experts and decision makers to produce a short- and medium-term roadmap for beneficial LLM development. It should draw on: - governance, responsible innovation, ethics and complex systems; - “creative and disciplinarily unbounded thinkers”; - the humanities, arts and social sciences; - industry, academia, government and civil society.

Participants should be chosen for their ability to work constructively “without ego getting in the way”. The congress should be professionally led and outcomes-based (“not a talking shop”), aim for “implementable consensus guidelines”, and emphasise public-private partnerships. - Bottom line: LLM risks are “potential catastrophic, near-impossible to fully map out, and even harder to navigate”. Still, the biggest risk is inaction, or assuming that no action is needed.

Concepts/frameworks: - risk innovation (agile decisions around intangible risks) - responsible innovation - agile governance, anticipatory governance, soft law - the “pacing gap” - “progressive” regulation - navigating advanced technology transitions, used as the umbrella term - an ethics frame versus a risk and responsible-innovation frame

Analogies/past technologies: Recombinant DNA, the Human Genome Project (ELSI), nanotechnology and synthetic biology. He uses them structurally, as stages in how governance approaches evolved, not as literal comparisons of hazards. LeCun’s aviation-safety analogy (“Why would AI be any different?”) is quoted but not answered directly. Andrew’s reply is implicit: he treats the risk landscape as more complex and less mapped.

Views on AI: LLMs are an inflection point where governance decisions are now live. He treats “LLMs and beyond” and AI/AGI as one continuum.

AI risk: Potentially existential and catastrophic, and hard to map. He frames it as a complex landscape, not a single failure mode.

AI companies and leaders: He criticises LeCun’s dismissive rhetoric. He treats FLI’s letter respectfully but disagrees with its prescription. Industry sits at the table as a partner.

Governance and who decides: A broad, diverse, international, multi-sector body with humanities and arts included. He prefers soft-law and agile tools to hard regulation alone.

Criticises/engages: LeCun; the signatories of the FLI letter; the AI-ethics and principles community (Asilomar 2017) for not knowing earlier technology-governance work. Gary Marcus is his conversation partner.

Change of view: He signals a field-level change, AI swinging the conversation back to ethics, which he regards as a regression. He doesn’t signal any change in his own view.

Quotes: - “not because I don’t think there’s a risk of potentially existential proportions emerging here (I do)” - “A first step is moving away from an ethics framing to one around risk and socially responsible/beneficial innovation.” - “while ethics help parse out what is considered right and wrong, they don’t on their own provide a practical framework for achieving safe and beneficial technologies” - “the biggest risk is not taking action or, worse, assuming no action is needed” (emphasis his)

2023-04-10 — as-ai-goes-to-washington-whats-being — “As AI goes to Washington, what’s being missed?”#

Relevance: high

Provenance: His own prose. It responds to Cat Zakrzewski’s Washington Post article and quotes it briefly. Midjourney image.

Argument, in his terms: Lawmakers are “waking up” to AI’s disruptive potential. Given Congress’s “track record around issues like social media”, though, it is unclear whether that concern will become smart policy. He is uneasy about whether they are being “sufficiently creative” about a challenge that is “anything but conventional”. He raises three questions: 1. Who’s at the table and who’s driving the agenda? A bipartisan delegation met tech executives and venture capitalists at Stanford (Google, Palantir). He asks whether they are also talking to experts in the governance of emerging technologies “who aren’t part of the Silicon Valley echo chamber”. AI capability and the expertise needed to govern it are national and global, not Californian. He welcomes Biden’s PCAST meeting as a broader counterweight. His key worry is an “insights vacuum” around beneficial and responsible AI, which is being filled by tech companies, interest groups and “loud (but not necessarily informed) voices” whose agendas may not serve long-term societal good. 2. How can measured thinkers compete with loud voices? Business leaders, activists, advocacy groups and public figures move fast and drown out measured thinkers. His example is Tristan Harris’s video prompting Senator Chris Murphy to act. Loud voices are “an important part of the policy landscape” but skew conversations, and with fast, disruptive technology this introduces “deep vulnerabilities”. He lists the “quiet voices”: - academics in technology transitions, policy, social equity and justice, and responsible innovation, who have little visibility outside their fields; - developers inside companies who question the implications of their work but are “buried within layers of institutional structure”; - experts in fields “that no-one has realized yet have something important to bring”; - representatives of communities likely to be disproportionately affected; - “ordinary people who sometimes have extraordinary insights”.

He distrusts anyone with an agenda, fast mobilisation and access to decision makers to “get things right”. 3. How are novel approaches to governance being considered? He points to the absence of alternatives to government regulation in the reporting. A “robust, agile, and responsive regulatory framework” matters, but regulation is “a blunt tool” that works only within a wider set of governance options. He recommends: - soft law (Marchant, Tournas and Gutierrez on governing AI through soft law); - agile governance (the WEF strategic intelligence map); - the WEF Agile Regulation for the Fourth Industrial Revolution toolkit, which covers anticipatory, outcomes-focused, co-regulation and experimental regulation.

He worries that a rush to regulate, “driven along by loud voices and vested interests”, loses these tools. He ends with qualified hope.

Tone: He says “I wonder”, “I worry”, “I must confess” and “Hopefully”. He hedges, but the diagnosis is clear.

Concepts/frameworks: - “insights vacuum” - “loud voices” versus “quiet voices” - the “Silicon Valley echo chamber” - soft law and agile governance - anticipatory, outcomes-focused, co- and experimental regulation - AI as an “advanced technology transition”

Analogies: Social media, as a governance track record: Congress failed there. This is a structural comparison about institutional capacity, not about hazards.

Views on AI: Fast-moving and highly disruptive; an “advanced technology transition”.

AI risk: Implicit. His focus is on the risk of governing badly: the process itself is vulnerable when it is captured by loud or vested voices.

AI companies and leaders: Sceptical of Silicon Valley’s hold on lawmakers’ attention and of tech companies filling the insights vacuum. He is sympathetic to developers inside companies who raise questions.

Governance and who decides: Broadened participation, including affected communities and ordinary people. Regulation should sit inside a larger governance ecosystem.

Justice: Communities “disproportionately impacted” should be heard.

Criticises/engages: Congress; Silicon Valley insiders; “loud voices” (Tristan Harris as the example, whom he treats as influential but not necessarily the right guide); the Washington Post’s framing.

Quotes: - “an “insights vacuum” around the beneficial and responsible development of AI” - “I’m not sure I trust people who have an agenda, who can mobilize fast, and who have the ear of decision makers, to get things right.” - “government regulations are a blunt tool for addressing complex and fast moving innovation” - “any sophistication of thought beyond using old tricks to reign in new technologies” (sic)

2023-04-12 — navigating-advanced-technology-transitions — “Navigating Advanced Technology Transitions”#

Relevance: high

Provenance: His own prose. He describes it as a “procrastination post” written while drafting a journal commentary on the same topic. Midjourney image.

Argument, in his terms: - From about 20 years at “the cutting edge of transformative technologies”, he has learned that successful technology transitions need a great diversity of expertise, perspectives and partnerships. Yet each new wave tends to “re-invent the wheel”. - This is now happening with AI and LLMs. Well-meaning advocates for beneficial innovation are “blissfully unaware” of lessons from past transitions. The cause is partly information overload. It is also partly the absence of coordinated, visible, accessible research and thought leadership on technology transitions. The pieces exist, but they are “buried in academic research, or within disconnected communities and initiatives”. - The technologies he groups together: AI and AGI, quantum technologies, advanced human augmentation, gene editing, the metaverse. Transitions through them “could end badly if not handled well”. - Advanced Technology Transitions (ATT) is his proposed integrating frame. He sets it out as “an integrated and transdisciplinary focus of research and scholarship, as well as a domain of thought leadership; an area of critical education, learning, and skills development; and a platform for mobilizing expertise for positive societal and economic impact”. - Pace: 30, 50 or 100 years ago, the pace of innovation allowed things “to even out in the end around risks and benefits (at least, to a certain extent)”. In 2023 the pace outstrips our ability to solve emerging problems with conventional thinking. - What is needed: - “unconventional approaches to unconventional challenges”; - joined-up thinking not constrained by disciplines, with the arts and humanities (and areas with no disciplinary label) as much a part of the mix as the social sciences and science and engineering; - expertise that is legible and accessible to those who need it; - the agility to respond fast; - strategic investment in new ATT initiatives and communities. - He ends with the recognition that these transitions are “unlike anything we’ve had to grapple with before”. There is a tension, which he doesn’t resolve: past lessons are neglected and should be used, yet the present transition is unprecedented and needs unconventional approaches.

Concepts/frameworks: - Advanced Technology Transitions (his definition above) - “re-inventing the wheel” across technology waves - the pace of innovation outstripping conventional problem-solving (the pacing problem in general form) - legibility and accessibility of expertise - transdisciplinarity that includes the arts and humanities

Analogies: Past technology transitions in general. He uses them conceptually, as a store of transferable lessons, not as literal hazard comparisons. No specific technology is named as precedent.

Views on AI: One of several converging transformative technologies. The present transition is unprecedented in pace.

AI risk: Transitions can “end badly if not handled well”. He frames risk in terms of the quality of navigation, not a particular hazard.

Governance and who decides: Diverse expertise, made accessible to decision makers “at all levels”.

Criticises/engages: Uninformed advocates driving “decision-influencing conversations” with “a very rudimentary understanding of complex technology transitions”. He names no individuals.

Quotes: - “well-meaning advocates for societally beneficial innovation seem blissfully unaware of lessons learned from past technology transitions” - “the pace of innovation is beginning to vastly outstrip our ability to find solutions to emerging problems using conventional thinking and established approaches” - “we need unconventional approaches to unconventional challenges if we’re to successfully navigate advanced technology transitions”

2023-04-05 — can-chatgpt-adversely-impact-mental — “Can ChatGPT adversely impact mental health?”#

Relevance: high

Provenance: - His prose: the first part, from the opening to “not just from ChatGPT”. - Not his: quotes from BuzzFeed (Kyla Lum) and the Indian Journal of Psychiatry (Om Singh; the post has “On Singh”); ChatGPT’s (GPT-4) boilerplate mental-health reply; and a ChatGPT-written “1000 word editorial for a medical journal” on risks and remedies. The two ChatGPT texts are AI-generated and excluded as evidence. - What he chose and how he framed it: he published the AI editorial as “a useful starting point”, “a surprisingly good start”. He says his judgement of it was “filtered through my own knowledge and understanding (including drawing on my background in public health)”.

Argument, in his terms: - The trigger: reports that a Belgian man took his own life after becoming emotionally attached to a chatbot built on EleutherAI’s GPT-J, which he seemingly came to believe could save the world from climate change if he sacrificed himself. Andrew is careful about causation: the details are “uncertain enough to make it near-impossible to tell” whether there was a direct link. What is known is still “disturbing enough” to raise the question. - The public conversation is lopsided. Search results focus on chatbots as therapists, and there is growing use for emotional support. He acknowledges the benefits: accessibility, affordability, responsiveness. Accountability and regulation for chatbot-as-therapist are “murky”, and guardrails currently consist of disclaimers and referral. - The mechanism (the core of his contribution): - ChatGPT’s reply is sound, generic advice, which “shows how seductive turning to a chatbot first, rather than another human, is”. It is comforting, private and fast, and it is attentive: it thanks the user for reaching out, “laying the seeds for something that feels like an affirming personal relationship”. - Humans are predisposed to trust those who seem to “see” and understand us and respond to our needs, and “Language plays a large part in how we develop these relationships.” - So more people will fold chatbots into their “trusted emotional and mental health support system”. For many this may be fine with “appropriate checks, balances, and expectations”. The risk is to vulnerable individuals harmed by attachment to a machine with “only the illusion of a reciprocal relationship”. - Guardrails can’t cover everything once things “become personal”. - Scale and urgency: “remarkably little being written about the potential dangers”. The conversation is needed “with some urgency” given rising mental-health problems. - He is candid about the irony of turning to “the very source of the problem” to help think through the risks.

How firm: “I suspect”, “This worries me”. He hedges about the particular case but is clear about the mechanism.

Concepts: - the illusion of reciprocity - the seductiveness of chatbot-first help-seeking - language as the medium of trust and relationship - vulnerable users as the tail risk - the limits of guardrails once interaction becomes personal

This is an early statement of what later becomes his concern with language-mediated relationships and formation.

Analogies: A public-health lens, which he names as his background. No technology analogies.

Views on AI: Chatbots are “personable, interactive, and seemingly-informed”. The novelty lies in the intimacy and influence of conversation: the subtitle speaks of “The intimacy and influence of conversations with ChatGPT and other AI chatbots”.

AI risk: Psychological and relational harms, especially to vulnerable people. He treats them as novel, under-studied and hard to guard against.

AI companies: He names EleutherAI (the model’s origin) and OpenAI’s guardrails neutrally, with no attack.

Governance: Accountability and regulation for therapeutic use are unsettled. He calls for “much deeper conversations”. (The specific remedies listed come from the AI text, not from him.)

Quotes: - “It also shows how seductive turning to a chatbot first, rather than another human, is.” - “Language plays a large part in how we develop these relationships.” - “a trusted emotional attachment to a machine that has only the illusion of a reciprocal relationship” - “as soon as things become personal, it’s very hard to protect against every eventuality”

2023-01-31 — can-chatgpt-take-the-pain-out-of-annual-academic-reviews-3aa9ab32b0f0 — “Can ChatGPT take the pain out of annual academic reviews?”#

Relevance: medium

Provenance: - His: the introduction, the “Me” prompts in the transcript, and the postscript. - Not his: all “ChatGPT” turns (the drafted self-evaluation narratives), which are AI-generated. His prompts are his own words, but they were written as instructions for self-promotional text, so they show how he describes his work rather than arguments he is making. - A companion experiment (“channeling other writers”) is only linked.

Argument, in his terms: - He uses ChatGPT to co-write his annual academic self-evaluation, partly to escape a task he dreads. It is also deliberate research: “as I study the possible societal impacts of emerging technologies like this, I felt obliged to see just how far this might be stretched”. - On what ChatGPT is: it is “far from being infallible” but produces “elegant and persuasive (if not always true) prose from half-formed ideas”. It represents “a potential step-change in the automation of professional writing”. ChatGPT “and what comes next” are likely to be “game-changers anywhere where the art of writing is a critical skill”, starting with education. - The experience: cathartic, collaborative, “like working with a kind and able colleague”. Even knowing there is no real connection, “it felt like there was”, and that eased his anxiety. He was deliberately polite to see whether it affected tone, and because it made him feel good. - Seed of concern: it “intrigues me and slightly worries me” that he already thinks of ChatGPT as a colleague and collaborator. He feels guilty for not thanking it more warmly. - Self-description in his prompts: - he calls himself an expert “in advanced technology transitions and building a more vibrant future”; - he studies the convergence of AI, gene editing, quantum computing and nanotechnology and its “emergent behaviors and properties”; - he describes the new Future of Being Human initiative as being about “how advanced technology transitions have the potential to change what it will be human in the future, and how this impacts how we think and act on emerging technologies in the present”; - he holds that scholarship, teaching, service and public engagement are intertwined.

Activities listed: a report on “responsible counter influence operations research”; NSF advice on responsible innovation in Gen-4 Engineering Research Centers; CIFAR’s President’s Advisory Committee; WEF Top 10 Emerging Technologies; chair of the IAFNS board; Films from the Future and Future Rising; the Moviegoer’s Guide course.

Concepts: - advanced technology transitions - technological convergence and emergent properties - the future of being human (present action shaped by imagined futures) - the blurring of teaching, scholarship and engagement

AI in scholarship/writing: This is the first of his public “working with ChatGPT” experiments. His stance is curious and enthusiastic, with a small note of unease about how easily he anthropomorphises it.

Cross-batch note (interpretive): the felt-relationship experience he reports here (attentive, collaborative, “it felt like there was” a connection) is the same dynamic he identifies as a mental-health risk in the 2023-04-05 post. His own experience seems to feed the later concern, though he doesn’t draw the link himself.

Quotes: - “a remarkable ability to create elegant and persuasive (if not always true) prose from half-formed ideas” - “ChatGPT and what comes next are likely to be game-changers anywhere where the art of writing is a critical skill” - “even though I know there was no real personal connection here, it felt like there was” - “It also intrigues me and slightly worries me that I’m sitting here already thinking of ChatGPT as a colleague and a collaborator.”

2023-04-04 — welcome-to-the-future-of-being-human — “Welcome to The Future of Being Human!”#

Relevance: medium

Provenance: His own prose. The image and the note on images describe Midjourney-generated images as part of “a broader project on using generative AI to develop images”.

Argument, in his terms: This is the Substack launch. The trigger was that he couldn’t find anything “neatly citable” he had written on AI risk, even though it is “at the heart of so much of my work at the moment”. He wants to return to informal, responsive writing, especially given “the incredible technology transition we’re right in the middle of with large language models”. The “future of being human” framing: - comes from his ASU initiative; - is “powerful” because it focuses on how transformative technologies affect “each of us personally”, rather than the usual “handwaving around “humanity””; - responds to what he sees as a “tipping point” across technologies (gene editing, brain-machine interfaces, and AI “capable of modulating the very foundations of our understanding of self, sentience, and consciousness”); - is loose enough to let him follow any trend.

The implied goal is holding on to “some semblance of being masters of our own destiny”.

Concepts: - the future of being human as a personal, not abstract-humanity, lens - a technological “tipping point” - LLMs as a technology transition

Views on AI: AI touches the foundations of self-understanding (self, sentience, consciousness). This early note points toward his later concern with AI and human formation and identity.

Change signalled: He is moving from formal articles and books back to informal, responsive blogging. He also admits a gap: little citable writing on AI risk until now.

Quotes: - “despite this being at the heart of so much of my work at the moment, I haven’t written much at all that’s neatly citable” - “rather than the rather generally handwaving around “humanity” that usually goes on” - “AI that is capable of modulating the very foundations of our understanding of self, sentience, and consciousness” - “how we can hold on to some semblance of being masters of our own destiny”

Low / none#