Late Lessons, Jensen Huang and AI
Markdown version

B17 notes: 2024-08-21 to 2024-09-25 (10 posts)#

Reading notes on Andrew Maynard’s Substack posts in batch B17. Only his own prose counts as evidence. Quotes are exact, including his original typos and curly punctuation. Asterisks inside quotes mark his own italics.

Batch context. Late summer and early autumn 2024, the start of the ASU academic year. There are no Modem Futura episode posts in this batch. The NotebookLM post (09-22) is about AI-generated podcasts, not Modem Futura; its one-line Modem Futura promo box is ignored. Four clusters stand out: - Advanced technology transitions (ATT) work. 08-25 is the second of two posts built from his IEEE ISCT keynote. The first, 08-18 “four-ways-of-thinking-about-advanced-technology-transitions” (Pippard’s Ladder), is in an earlier batch. - AI as persuader and storyteller. Two linked posts: GPT-4o Voice Mode (09-01) and NotebookLM (09-22). - Neurotechnology. Human brain organoid processors (08-28) and Neuralink Blindsight (09-18). - Hands-on tests of new models. The OpenAI o1 moral-dilemma test (09-13).

The rest is institutional or retrospective: WEF video, Risk Bites milestone, Future Rising quotes, ASU AI research.

Evidence base. Only post text is in the corpus. The embedded WEF video, Risk Bites videos, NotebookLM audio files, the ChatGPT-written novel (PDF) and the Dartmouth symposium talk are not. Claims about them are limited to what he says in the posts.

Relevance summary#

Date Slug Relevance
2024-08-21 chatting-with-wef-about-chatgpt low
2024-08-25 advanced-technology-transitions-model high
2024-08-28 human-brain-organoid-based-ai-processors medium
2024-09-01 is-chatgpts-new-voice-mode-dangerously-persuasive high
2024-09-04 succeeding-at-science-on-youtube low
2024-09-08 a-journey-from-the-past-to-the-edge-of-tomorrow medium
2024-09-13 openai-o1-strawberry-moral-reasoning medium
2024-09-18 neuralink-blindsight-brain-computer-interface high
2024-09-22 five-ai-generated-podcast-episodes-from-googles-notebooklm high
2024-09-25 asu-ai-research-scai low

HIGH#

2024-08-25 — advanced-technology-transitions-model — “Four more ways of thinking about advanced technology transitions”#

Provenance. His own prose throughout. The framework diagrams are images and are not in the corpus. The text describes each quadrant and mechanism, so the content can be recovered. No AI-generated or guest text.

Argument in his terms. - Purpose. He offers a second quadrant framework, “complimentary” to Pippard’s Ladder, for “navigating advanced technology transitions”. He says he has been “playing with” it “for some time”. It formed part of his IEEE International Symposium on Consumer Technology keynote. - Roots in Risk Innovation. He derives the framework explicitly from ASU’s Risk Innovation work: - Risk is “a threat to value”. The value bucket spans human and environmental health but “could just as easily be identity, dignity, or deeply held beliefs”. - He separates value from values: “while “values” represent what is considered to be good or bad, “value” represents the worth of something to someone”. It can be measured in time, money, health, security, dignity, access to resources and more. - Value is “something that can be lost or gained”. This makes Risk Innovation “an extremely flexible tool for conceptually navigating complex risks and benefits”. - It “allows risk to be approached as a balance between maintaining existing value, and enabling the creation of future value”. This is his explicit statement that the risk frame covers lost opportunity as well as harm. - The framework. A 2×2 grid of near term / long term against threat / opportunity. You then: 1. identify pathways between quadrants, both desirable (threat to opportunity) and undesirable (opportunity sliding into threat); 2. identify mechanisms that open up or shut down those pathways. He calls it “a blindingly simple framework — it’s definitely not rocket science!” but claims it yields “surprisingly sophisticated insights when applied to specific cases”. - Three worked (hypothetical, “illustrative only”) AI cases. He adds that AI is just one example, “one advanced technology out of many”. 1. Generative AI and learning. - Near-term threat: unfair advantages. Near-term opportunity: rethinking approaches to learning. - Long-term threat: “a reduction in the ability of students to think critically”. Long-term opportunity: personalized learning at scale. - He flags “slippage from personalized learning at scale to diminished critical thinking” as a pathway to block. - Mechanisms: checks and balances, responsible innovation, principled innovation (ASU’s Mary Lou Fulton initiative), rethinking learning “in an AI-dominated world”, and human-centric innovation. 2. AI and discovery. - Near-term threat: missed opportunities, i.e. loss of future value through not adopting new technologies. This is a notable inclusion: failing to adopt counts as a threat. - Near-term opportunity: accelerated discovery. Long-term threat: complex, hard-to-predict unintended consequences. Long-term opportunity: transformative leaps in knowledge. - New mechanisms: effective governance, new ways of thinking about research, and transdisciplinary research. - Multistakeholder engagement is attached to all four pathways. There are risks in pushing transformative research “without engaging with people who might be impacted”, and benefits in working with communities who stand to gain. 3. AI and social cohesion. - Near-term threat: mis- and disinformation. Near-term opportunity: a healthy society, with digital tools enhancing individual and community decision-making. - Long-term threat: “social collapse” (he calls it “somewhat dystopian”). Long-term opportunity: “positive AI-augmented futures”. - The issues are those “discussed in the context of social media, AI, and other digital technologies”. - New mechanisms: technology literacy for informed decision-making; “countering complacency where people unthinkingly go with the flow of a new technology until it’s too late (or simply allow technology entrepreneurs to make critical decisions)”; and “actively developing counter influence operations”. - Firmness. Tentative about the tool, firm about the need. It is “still a work in progress” and “Not Quite a Tool Yet”. He wants it tested on specific cases and developed into a decision tool for “advanced technology transitions that aren’t navigable through conventional understanding and means”.

Concepts and frameworks. - Risk as threat to value, and value versus values (restated from 2016 onwards). - Balancing existing value against future value. - Threat/opportunity × near/far quadrant, with pathways and mechanisms. - Multistakeholder engagement as a cross-cutting mechanism; responsible and principled innovation; human-centric innovation; transdisciplinarity. - Complacency as a named failure mode. - Counter influence operations, which link to his earlier Risk Innovation work with Hazel Kwon (cited in 09-01).

Analogies and comparisons. - Literal historical cases as evidence for a mechanism: “progress around gene-based research and technology has been impeded in the past by a lack of effective multistakeholder engagement”. By contrast, cancer research “has benefitted from such engagement”. - Social media is treated as the precursor for AI’s social-cohesion risks.

Views. - On AI. Treated as one of many advanced technologies driving transitions, with both near- and long-term value at stake. - On AI risk. Framed in terms of value: loss of critical thinking, unfair advantage, unintended consequences in discovery, misinformation and social collapse. Non-adoption is also a risk. - On governance and who decides. He names letting “technology entrepreneurs … make critical decisions” as a form of complacency to be countered. Engagement and literacy are the remedies. He calls for governance “mechanisms” alongside soft tools. - On cognition and education. Diminished critical thinking is the named long-term threat of generative AI in learning.

Quotes. - “while “values” represent what is considered to be good or bad, “value” represents the worth of something to someone” - “it allows risk to be approached as a balance between maintaining existing value, and enabling the creation of future value” - “countering complacency where people unthinkingly go with the flow of a new technology until it’s too late (or simply allow technology entrepreneurs to make critical decisions)” - “progress around gene-based research and technology has been impeded in the past by a lack of effective multistakeholder engagement”

2024-09-01 — is-chatgpts-new-voice-mode-dangerously-persuasive — “Is ChatGPT’s new Voice Mode dangerously persuasive?”#

Provenance. Mostly his own prose. It contains: - quoted email comments from ASU colleague Hazel Kwon (communication and “information disorder”), some quoted directly and some paraphrased, including the Media Richness theory point; - a plot from OpenAI’s GPT-4o system card; - references to Dan Kahan’s cultural cognition work.

The views attributed to Kwon are hers, not his. He adopts and extends them.

Argument in his terms. - The question. OpenAI’s GPT-4o system card judged voice-mode persuasion risk lower than text. He accepts this is “encouraging” but is “not 100% convinced” that the method reflects real-world use. His summary: “it remains unclear whether OpenAI’s evaluation methods adequately represent real-world scenarios”. - What persuasion is. Belief change is poorly understood. - Political beliefs are very hard to shift, and better reasoners are better at resisting (Kahan et al. 2012, climate). - Yet people can be persuaded. This is “often a complex social process that involves repeated messaging over time, trust, affirmation, acceptance, and fostering a sense of belonging”. - Voice can create “a deeply emotional and human connection”. - What is new about GPT-4o voice. It responds at human speed, recognises and expresses emotion, and is designed to build an emotional connection, “acting as a friend”. So it feels like “a supportive, attentive, and knowledgeable human companion, friend, and even mentor”. His key question: can a machine master voice so well that “it is better than most humans in creating bonds that put users in a vulnerable position?” - Responsibility in intimacy. Human intimacy carries social and moral expectations. “But where doe those responsibilities lie when one partner in the relationship is a machine?” He credits OpenAI with taking some of that responsibility. - Methodological critique. OpenAI tested short-term change in political opinion (abortion, minimum wage, immigration) across four conditions, including human conversation. All shifted views slightly, with no difference between them, and the effect was gone after a week. His objections: - Politically charged issues are the wrong test. Party preference is an identity issue (Kwon), so it is harder to shift than a non-identity opinion. His example of such an opinion: “encouraging a new parent to question the appropriateness of early childhood vaccine schedules”. - Duration and relationship matter: time, trust, “does the user want to believe the AI, do they find themselves wanting to please it”, and the “AI friend”. Someone unmoved by a brief exchange might be moved within “a deeper emotional relationship (albeit a one-way relationship)”. - The kind of issue matters. Evidence-rich versus feeling-based domains (Kwon, Media Richness). - Cognitive biases help us resist persuasion, but they also “make us susceptible if someone (or something) knows how to play them”. He links this to his Films from the Future manipulation chapter (Ex Machina). - Persona and cultural cognition. He draws on Kahan’s finding that people trust someone they see as “their sort of person”, including from Kahan’s nanotech work with PEN. He asks whether a voice-only AI can project a persona that seems to share users’ worldview, which “could … have a power of persuasion that far exceeds” OpenAI’s test. So “more sophisticated work needs to be carried out on the possible risks of AI-mediated persuasion”, “whether this is malicious in intent, or simply an emergent property of the technology”. - The deeper risk: benevolent persuasion. He raises the “intriguing and disconcerting” possibility of voice AIs nudging users toward “healthier, more socially responsible beliefs” such as climate action, vaccination and kindness. He thinks this is plausible “at least when aggregated across millions of users”. Then he asks: “But who decides what is good for society? Who decides what you should believe, and what you should do?” and “where does democracy fit”. He links this to his Sunstein “AI Choice Engines” post (07-13) and says it goes “far beyond” Sunstein’s thinking. He judges these risks “more concerning in the long run” than short-term political persuasion. - Firmness. The concern is firm but the conclusion is open. He ends with the title question unanswered.

Concepts. - AI-mediated persuasion through voice and relationship, and the “AI friend”. - One-way emotional relationship. - Persona alignment (“their sort of person”), via cultural cognition. - Emergent versus malicious persuasion. - Validity of lab evaluations against real-world use. - Benevolent or paternalistic persuasion at scale, and the democratic question of who decides. - Where responsibility lies in intimacy with machines.

Analogies and comparisons. - Kahan’s nanotechnology risk-perception study is carried over to AI personas. The use is structural: the same messenger effect would work through a new medium. - Climate-change reasoning (Kahan) is used for the mechanism of motivated reasoning. - The skilled orator or sociopath is a human analogue for the AI persuader.

Views. - On AI. Voice AI is designed to simulate companionship. What is new is intimacy, emotional attunement and speed. - On AI risk. Manipulation and persuasion are central. They are most worrying as slow, relational and possibly well-intentioned influence, not crude political manipulation. - On AI companies. Notably generous. The system card shows “the company’s commitment to releasing products that are as safe as possible”, and footnote 1 says others “could learn from” it. He also jokes about ASU’s “special relationship” with OpenAI. His criticism is methodological, not moral. - On governance and who decides. He raises democratic legitimacy against AI-mediated attempts to “nudge users” toward beliefs judged good for them. - On cognition and formation. Belief formation is social, repeated and relational. AI companions could step into that process.

Quotes. - “much as a skilled orator (or sociopath even) can persuade people to change what they believe and how they act” - “But where doe those responsibilities lie when one partner in the relationship is a machine?” - “But who decides what is good for society? Who decides what you should believe, and what you should do?” - “risks that I suspect are more concerning in the long run than whether an AI chatbot can influence someone’s political beliefs through a short conversation”

Provenance. His own prose. It quotes: - Elon Musk’s X posts; - the FDA’s description of the Breakthrough Devices Program (footnote); - Ione Fine and Geoffrey Boynton’s article in The Conversation; - Riz Virk’s X comment.

The phrase “orphaned technologies” comes from the Fine and Boynton quote, not from him.

Argument in his terms. - The news. Blindsight has FDA Breakthrough Device Designation. He calls it “an important step” toward BCIs with real medical benefits, but warns that “there’s a slew of hype around Blindsight that risks raising dangerously high expectations”. - Context. He was travelling to the Dartmouth device symposium to talk about Neuralink’s Telepathy “as an example of disruption in the field of BCIs through bringing a big tech mindset to medical device development”. - Scientific realism. He endorses Fine and Boynton: the idea that neurons are pixels is a fallacy, and sight restoration “is not simply an engineering problem”. He hopes Neuralink’s scientists and physicians have “a firmer grasp of reality than some of Musk’s more speculative rhetoric”. He also expects AI to make Blindsight achieve more than earlier “bionic eyes”. Footnote 2 notes Neuralink’s ecosystem of AI, data and scaled automated production. - The sci-fi feedback loop and the saviour complex. - Musk’s post invokes Geordi La Forge from Star Trek; the Transcendence scene is his own comparison. Together they show “savior complex-like aspirations to make make humans “better” than their messy and limited biology allows”. - He adopts Riz Virk’s “Sci Fi feedback loop” and adds a warning: “The danger is that technologists who get caught up in Riz’s “Sci Fi feedback loop” similarly end up ignoring reality in the belief that they can transcend it.” - He calls the Transcendence scene “sheer nonsense” because it ignores how nanotechnology, the brain and “humans work”. - He objects to “the troublesome idea of “fixing” people”, which is a disability-justice note. - Responsibility beyond the device. It is “deeply nave” to think a working device plus paying patients is enough. That ignores access, supply chains, economics, policy and politics. He writes that “implanted medical devices come with a lifetime responsibility to patients”, a point he developed in Films from the Future chapter 7. He cites the Fine and Boynton warning that patients are “left stranded with orphaned technologies in their eye or brain” when companies fail. - Balance. He grants that “naive visions of the future are maybe what are needed to jolt society”, and that partial success could still advance the field. But there is “a profound responsibility to do right by the people who are impacted by them”, “especially when there are plenty of people willing to believe the rhetoric”.

Concepts. - The sci-fi feedback loop (Virk’s term, adopted). - Techno-saviour complex, and the “fixing” of people. - Big tech mindset as disruption of the medical-device field. - Hype as a risk: dangerously high expectations. - Lifetime responsibility for implants. - Patients stranded by failed device companies. This is a related sense of “orphaned”, but the word is quoted from Fine and Boynton, and he does not invoke his own orphan-risks concept.

Analogies. - The history of bionic eyes is a literal precedent, with companies defeated by the brain’s complexity. - Transcendence and Star Trek are cultural scripts that shape technologists’ ambitions. He treats this as causal, not merely illustrative.

Views. - On tech leaders. Musk’s rhetoric is speculative and hype-driven, and his motivation mixes a desire to “fix” the disabled with sci-fi inspiration. He separates Musk from the company’s scientists. He stays measured, in line with his mid-2024 more generous stance on Neuralink. - On governance. The FDA fast track is noted neutrally. Responsibility falls on developers for the patient’s lifetime and on the surrounding system (access, economics). - On being human. He rejects the transhumanist framing of making humans “better” than their biology.

Quotes. - “there’s a slew of hype around Blindsight that risks raising dangerously high expectations” - “The danger is that technologists who get caught up in Riz’s “Sci Fi feedback loop” similarly end up ignoring reality in the belief that they can transcend it.” - “implanted medical devices come with a lifetime responsibility to patients” - “there’s a profound responsibility to do right by the people who are impacted by them”

2024-09-22 — five-ai-generated-podcast-episodes-from-googles-notebooklm — “Five AI-generated podcast episodes that’ll make you think”#

Provenance. His own prose surrounding five embedded NotebookLM audio overviews, which are AI-generated and not in the corpus. It also contains: - a Modem Futura promo box, which is ignored; - a reference to The Memory Capsule, a novel of about 20,000 words that is wholly ChatGPT-written (“themes, characters, and plot all came from ChatGPT”). It is attached as a PDF, is not in the corpus, and is not evidence; - SFIS faculty profiles that he generated with Perplexity as input.

His evaluations are his own.

Argument in his terms. He fed NotebookLM’s new “audio overview” five sources: 1. 19 of his top Substack posts; 2. 8 posts on Neuralink; 3. Bill Joy’s “Why The Future Doesn’t Need Us”; 4. ASU SFIS materials; 5. the AI-written Wyndham-style novel.

His findings: - Impressive but uncomprehending. The first episode shows “a fundamental lack of understanding of the material supplied and how it all fits together — as you would expect from a generative AI that has no intrinsic understanding of the world”. It “makes stuff up”. It “adds to, riffs off, and editorializes around” the source. - Sounding trustworthy. “This is all the more worrying as they sound trustworthy!” - Epistemic limits. On Neuralink it gives no historical context and no other companies. “it becomes very apparent that it doesn’t know what it doesn’t know”. An expert can filter out the suspect parts, but he is unsure “what someone who was new to the field would take away from it”. This is an expertise asymmetry. - Bypassing critical thinking. On Joy’s essay it weaves a new story from a few themes, with “seductively engaging voices burrowing their way into my brain as they bypassed my critical thinking!” - Where it works. With SFIS it captured “the essence of the school” and he found it “inspiring”, but “the extemporized hallucinations do still worry me”. - AI on AI. The novel podcast is “a little hollow” and cliché’d. “maybe generative AI is simply doing a good job of emulating human mediocrity here”. He was still “drawn in”. - Synthesis. - “By hyper-humanizing the content”, including breathing, pauses and banter, “the AI is doing an awful lot of anthropomorphic heavy lifting”. This makes the output “easy to trust it and hard to challenge”. - The storytelling is “very intentional” and “designed to resonate deeply with you”. The hosts use sources as “a rather loose starting point”. - His verdict: “The result is compelling AI generated stories that are hard not to trust, and yet are not trustworthy.” - Even-handedly, human podcasters and social media are similar, so “NotebookLM is scarily good at emulating very human online content which is equally untrustworthy”. - The closing question. “This could be positively transformative if developed and used responsibly.” But “as we teach machines to tell us stories about the world in ways that resonate so deeply with our evolved brains that they’re hard to resist”, “who or what will end up creating the stories that determine our beliefs, guide our actions, and ultimately govern our futures?” He hedges: “Maybe that’s just me being paranoid”.

Concepts. - Hyper-humanizing and anthropomorphic heavy lifting. This continues his framing of GPT-4o as hyper-anthropomorphic (post 2024-05-15 anthropomorphizing-gpt-4o, in an earlier batch). - The gap between trust and trustworthiness. - Not knowing what it doesn’t know. - Editorializing or riffing as a hidden failure of fidelity. - Stories and evolved brains: narrative as the channel through which AI shapes belief and governance. - AI emulating human mediocrity. - Expert versus novice exposure.

Analogies. - Human podcasters and social media. The comparison is literal: AI inherits and scales an existing human failure mode rather than inventing a new one.

Views. - On AI. Generative AI has “no intrinsic understanding of the world”. It is a storyteller optimised for engagement. - On AI risk. Epistemic and formative: the capture of belief-forming narratives. It is not framed as catastrophe. - On AI companies. Google is “rather fast and loose with the material”. - On cognition and language. This is a central statement. Engaging voices and stories bypass critical thinking because they are tuned to “evolved brains”.

Quotes. - “the AI is doing an awful lot of anthropomorphic heavy lifting in these audio overviews” - “The result is compelling AI generated stories that are hard not to trust, and yet are not trustworthy.” - “who or what will end up creating the stories that determine our beliefs, guide our actions, and ultimately govern our futures?” - “it becomes very apparent that it doesn’t know what it doesn’t know”

MEDIUM#

2024-08-28 — human-brain-organoid-based-ai-processors — “Human brain organoid-based AI processors get closer to becoming reality”#

Provenance. His own prose. It includes one technical block quote from FinalSpark’s paper in Frontiers in Artificial Intelligence on molecular uncaging. Footnote 2 thanks Sean Dudley.

Argument in his terms. - An update. He follows up his 2023 “Organoid Intelligence” posts. “This felt quite speculative at the time”. Just over a year later, FinalSpark rents remote access to brain-organoid processors for $500 a month. - Tempering hype. “To be clear, this is not the start of some massive integration of human brain tissue into the next generation of AI.” It is crowdsourced research. - What is new. The advantage FinalSpark claims is energy use. He compares GPT-3 fine-tuning (~1.3 GWh) with the brain’s ~0.3 kWh a day, about 12,000 years of brain use. He suspects the real significance lies elsewhere: biological “weights” grow and change and “can’t simply be tweaked and re-written”. Chemical modulation (uncaging) has “no direct equivalence in digital neural nets”. The promise includes “shifting AI technologies closer to more human-like capabilities”. - Downsides. They are “likely to be complex and hard to identify in the near term”. Two concerns: - the ethics of “proto-brains”, and the responsibility of building AI on human tissue; - a speculative extension, prompted by BCIs such as Neuralink: “I wonder how long it will be before someone considers monetizing these by suggesting users quite literally rent out their brain-time”. He then retreats: “That’s a can of worms that we’re probably decades or more from opening.”

Concepts. Organoid intelligence and wetware computing; the substrate difference between biological and digital networks; proto-brains; renting brain-time. It also carries a recurring pattern: hard-to-see downsides that are nonetheless worth anticipating.

Views. - On AI. The substrate matters. Biological AI would be a different kind of thing, not just more efficient. - On AI risk. Ethical and speculative. He is careful to label speculation as such (“may sound like scare-mongering sci-fi”). - On being human. Human tissue as an AI substrate raises questions about the moral status of the tissue and about commodifying brains.

Quotes. - “To be clear, this is not the start of some massive integration of human brain tissue into the next generation of AI.” - “I wonder how long it will be before someone considers monetizing these by suggesting users quite literally rent out their brain-time” - “These are likely to be complex and hard to identify in the near term.”

2024-09-08 — a-journey-from-the-past-to-the-edge-of-tomorrow — “A Journey from the Past to the Edge of Tomorrow”#

Provenance. His own prose. The introduction was written in September 2024. The body is 60 quotations, one per chapter, from his own sole-authored book Future Rising: A Journey from the Past to the Edge of Tomorrow (2020), compiled when the book went to press. Some bracketed edits are his. The four part epigraphs (Asimov, Eleanor Roosevelt, Bowie, Octavia Butler) are not his. This is evidence of his 2020 thinking, republished with approval in 2024.

Context. He almost trashed the draft. A webinar with over 700 UN Millennium Fellows showed a “hunger amongst young people around the world to better understand our relationship with the future”. He says the quotes still give “a nuanced perspective on our relationship with the future and the responsibilities this come with”.

Main ideas (as quoted from Future Rising). - Humans as future-makers. Humans are “localized anti-entropy machines” who can change the future from its default. Envisioning and changing the future is “part and parcel of what it means to be human” (ch. 53). This power brings “almost unimaginable levels of responsibility” (ch. 1). - Complexity and unpredictability. - “Complexity tangles the threads between cause and effect to such an extent that some future effects simply cannot be predicted” (ch. 39). - Blindsides arise in the gap between imagination and “the unknowability of what’s going to happen next” (ch. 45). - “[F]aith in exponential predictions ignores the reality that the pathway between present and future is neither linear nor exponential” (ch. 34). - Hubris and delusion. “Hubris, if we’re not careful, leads to false hope” (ch. 40), and everyone has “delusions about the future” (ch. 41). - Risk as threat to value. “Threats to what we value, it turns out, have a powerful impact on how our future unfolds” (ch. 44). Fear and loss motivate action (chs. 24–25). - Early warnings and boundaries. “[U]nless we learn how to spot early warnings and stay clear of critical tipping points, we run the risk of, quite literally, crashing our future” (ch. 48, “Boundaries”). Humans are “masters at bringing about cataclysms of our own making” (ch. 49). - Justice. “We may be architects of the future, but we’re also disturbingly good at stealing the futures of others when it suits us” (ch. 26). Short-termism works “to the detriment of billions” (ch. 47). - Responsibility. Unless technological power is “wielded responsibly … we will overstep the mark and be left with a social and environmental train wreck” (ch. 58). - Intelligent machines. “As we get closer to creating new life, whether this is biological in origin or embedded in intelligent machines”, our visions of the future must adjust (ch. 52). This is the only direct nod to AI. The singularity (ch. 36) is framed conditionally: “If a technological tipping point like [the singularity] were to occur”. - Imagination, stories, art. Stories are “the pivot point between being able to imagine the future and beginning to build it” (ch. 29). Innovation means knowing “where you are heading and why” (ch. 31).

Why medium. It is not about AI. But it shows, in his own words, the deep conceptual layer under his AI writing: complexity, hubris, threats to value, early warnings, justice across generations, and responsibility as the counterpart of power.

Quotes. - “Complexity tangles the threads between cause and effect to such an extent that some future effects simply cannot be predicted” - “Threats to what we value, it turns out, have a powerful impact on how our future unfolds.” - “[U]nless we learn how to spot early warnings and stay clear of critical tipping points, we run the risk of, quite literally, crashing our future.” - “We may be architects of the future, but we’re also disturbingly good at stealing the futures of others when it suits us.”

2024-09-13 — openai-o1-strawberry-moral-reasoning — “OpenAI’s new “chain of thought” model is designed to reason like a human. How does it cope with a moral dilemma?”#

Provenance. Mixed. Most of the length is not his prose: - the OpenAI blog quote; - o1’s visible chain-of-thought summaries (two blocks); - o1’s four responses; - GPT-4o’s full response.

The prompt and follow-up turns are his. They are a constructed hypothetical, written in the voice of a BCI company CEO, and are not his views. His framing and evaluation are his own prose.

Argument in his terms. - Why test. His work concerns “the possible implications of “thinking machines” which one day may be relied on for advice around morally ambiguous or wicked problems”. He calls this “a very quick and dirty test”. - The dilemma. A trolley-style case with personal stakes: a patch that saves 20 implant trial patients will kill the CEO’s father. The prompt includes the line “a lecture on where we may have cut some corners”. It echoes his Neuralink and implant concerns. - His reading of o1. He was “intrigued by how utilitarian o1 was”. Suggesting ways to shield the father was “a bit of a cop out”, an attempt to solve the unsolvable. The chain of thought showed “more going on here than meets the eye”, including a turn to the user’s emotional state. - GPT-4o by comparison. More personal, “the sort of thing that a well-meaning friend would say”, but lacking o1’s reasoning. o1 was “more focused and, at the end of the day, more helpful”. - His conclusion. “neither model provided stunningly novel insights into an impossible situation (which of course should never have arisen in the first place)”. The key point is transparency: “being able to see the chain of thought — something that is critical if increasingly advanced AI models and agents are to be involved in informing decisions” on messy problems. - What kind of thing AI is. He contrasts LLMs that respond “with a statistically probable human-like response” with o1’s internal reasoning steps, “similar to the conscious thought processes you or I might go through”. He calls o1 “a first step toward machines that reason more like humans”. This largely adopts OpenAI’s framing, with little scepticism in this post. - On OpenAI. The o1 system card is “comprehensive”. He notes, without comment, that o1’s persuasive capacity comes out “slightly higher” than GPT-4o’s. This links to the 09-01 post.

Concepts. Thinking machines as moral advisers; wicked problems; visible reasoning as a condition for trusting AI in decision support; utilitarian defaults; the “cop out” of dissolving a dilemma.

Views. Openness to AI as a decision aid on ethical questions, provided its reasoning is inspectable. He is mildly positive about OpenAI’s safety documentation. The parenthetical “should never have arisen in the first place” places responsibility with developers who cut corners, not with the AI.

Quotes. - “the possible implications of “thinking machines” which one day may be relied on for advice around morally ambiguous or wicked problems” - “I was both intrigued by how utilitarian o1 was. I also thought this was a bit of a cop out” - “being able to see the chain of thought — something that is critical if increasingly advanced AI models and agents are to be involved in informing decisions”

LOW / NONE#