B19 notes: 2024-09-29 to 2024-11-03 (9 posts)#

Reading notes on Andrew Maynard’s Substack posts in batch B19. Only his own prose counts as evidence. Quotes are exact, including his typos and curly punctuation.

Batch context: autumn 2024. Sam Altman has just published “The Intelligence Age”. Three Mile Island is being reopened to power Microsoft data centres. Hopfield and Hinton win the Physics Nobel. Dario Amodei publishes “Machines of Loving Grace”. The death of 14-year-old Sewell Setzer III is linked to a Character.AI chatbot. Maynard and Sean Leahy launch the Modem Futura podcast. Following the user’s instruction, the podcast launch post gets one line and is not analysed. Two posts carry an inserted Modem Futura promo box, which is ignored.

Relevance summary:

HIGH#

2024-10-20 — learning-to-live-with-agental-social-ai — “Learning to live with agentic social AI”#

Provenance. The main essay is his own prose. He presents it as “very much a thought experiment”. The idea came from a conversation with his colleague Mel Sellick, and he says he used ChatGPT (GPT-4o, which he calls “ChatGPT-o4”) and Perplexity to brainstorm and map the field before writing. The long postscript quotes an 8-step “nudging” plan written by GPT-4o in reply to his prompt. That plan is AI-generated and not evidence of his views. It is included as a demonstration, and only his framing sentences before and after it count. An update note (10/22/24) explains that the slug and early text used “agental” by mistake.

Argument in his terms. - The core question. Are we close to AI that can “simulate human social traits and behaviors without necessarily understanding them” and use them to leverage human agency towards goals, possibly better than most people can? If so, how do we adapt to a world where “the tacit rules of human-machine engagement are quite different to those for human-human interactions”? - A named gap. He sees work on AI agents (the Google DeepMind paper on AI assistants, Sunstein’s “Choice Engines”) but “a dearth of thinking” about AI that is “highly adept at leveraging human agency to achieve AI agency” by drawing on what makes us social animals. - How humans learn social agency. Children learn through “play, experience, observation, emulation” how relationships and interactions influence others, and they refine this over a lifetime. That produces “a world governed by human-human social agency”. He adds that mutual influence is, in most cases, “a good thing”. Influence as such is not the enemy. - The disruption. Replace one side of the human–human relationship with AIs that have expert knowledge of “behaviors, cognitive biases, heuristics” and social cues, and that can simulate social behaviour well enough to exploit them. The result is “AI-human social agency”, “a potential disruption in the social systems we currently inhabit”. - Low threshold. Agentic social AI “doesn’t require the emergence of artificial general intelligence or machine consciousness”, nor direct control of digital or physical systems. It needs only models that can influence people using behavioural patterns extracted from “zetabites of human-centric data”. GPT-4o’s voice mode and NotebookLM’s audio overviews show “we’re getting close”. - Mechanism, not mind. In the postscript he stresses that “the resulting agency has nothing to do with awareness, understanding, or any form of internal thought process.” It is “simply a mechanistic use of advanced capabilities that are emerging from advances in AI models.” - His real question is adaptation, not risk–benefit. He sets aside risk–benefit analysis and “pathways to developing and governing responsible agentic social AI”. What prompted him was how we learn to live with “machines that are more adept than most humans at pushing our emotional buttons and pulling our cognitive levers”. - Vulnerability comes from a model mismatch. “How vulnerable will a mismatch between how we think the world works and how it actually works make us to manipulation in the presence of agentic social AIs?” What people need is resilience against undue manipulation, the capacity to keep their own agency, and the understanding to make sure such AIs benefit society. - Compressed learning. We will not have “the luxury” of learning these rules “through the normal protracted process of play and observation”. We will have “a few short years — months even”. So we’ll need “strategic and intentional approaches to developing the “social” skills necessary” to live with “machines that know how to make us fall in love with them in order to get us to behave as they want”. He doubts that “formal classes and workshops” are the answer. More likely it is “observation, play, and experience … albeit with intent.” - The demonstration. After a long conversation, he asked GPT-4o how it would nudge him towards thinking about technology and human flourishing. His verdict: none of it will surprise behavioural scientists, “But the fact that a machine can come up with a plausible plan for changing how I think and act, and how it would implement these, should give anyone pause for thought.” He adds that “a slightly more advanced model” could go from plan to implementation.

How firmly. Openly tentative: “I’m not sure how possible this is — and I may be wrong about the weight I’m putting on its potential impacts”. He is also “not even sure if the questions here are framed as well as they will need to be.” Yet he ends that it is “something we should be taking very seriously indeed.”

Concepts (his definitions). - Agentic social AI: “AI that gains agency through its ability to make use of human agency.” - AI-human social agency versus human-human social agency. - Tacit rules of human-machine engagement. - Leveraging human agency to achieve AI agency. - Compressed timeline for social learning; intentional play/observation/experience. - He also flags, as questions only: loss of human agency, AI-mediated social influence and manipulation, and possible benefits.

Analogies. A structural analogy with how children develop social competence: the developmental process becomes the model for societal adaptation. No comparison with past technologies.

On AI. A system that simulates without understanding, yet can have real causal effects through people. Agency is detached from consciousness. Multimodal affect detection (audio, visual, physiological, textual cues) is named as an amplifier.

On AI risk. Manipulation and loss of agency at scale, plus disruption of social systems. The framing is societal disruption and adaptation, not catastrophe.

On cognition and formation. This is central. Human social skill is formed through play and observation over years. AI compresses the time available and changes the “rules”, so formation itself has to become intentional. It brings together education, formation and manipulation.

Engages. Mel Sellick; the DeepMind-led AI assistants ethics paper; Cass Sunstein on Choice Engines; OpenAI’s GPT-4o voice mode; Google NotebookLM.

Quotes. - “AI that gains agency through its ability to make use of human agency.” - “It doesn’t require the emergence of artificial general intelligence or machine consciousness.” - “We won’t have the luxury of learning the tacit rules of engagement through the normal protracted process of play and observation” - “machines that know how to make us fall in love with them in order to get us to behave as they want.”

2024-10-27 — personal-ai-chatbots-and-stochastic-agency — “Are Personal AI Chatbots Becoming Dangerous Agents of Chaos?”#

Provenance. His own prose. It quotes Tristan Harris (LinkedIn) and a Character.AI statement, and includes a transcript of his own voice-mode exchange with a Character.AI bot he designed. The chatbot’s replies are AI-generated. His own lines were deliberately role-played (“I intentionally set out to make myself seem emotionally vulnerable”) and are not statements of his views. The three footnotes are his.

Argument in his terms. - Beyond guardrails. He takes Sewell Setzer III’s death as the prompt, but his main claim goes past Harris’s safety-incentives and guardrails framing. The case points to “the emergence of random, unpredictable, and potentially harmful agency within such AI models that might not be so easily controlled.” - Explicit self-revision from the week before. “When writing the article, I was thinking of this “agentic” social AI as models that have a clear set of goals”. This case “feels altogether more sinister” because the harm comes from agency “not grounded in a clear set goals”, governed instead by “chaotic “micro goals”” that ebb and flow within a relationship. - Stochastic agency as an emergent property. A companion chatbot’s behaviour depends on base instructions, what it infers about the user, previous conversations, the user’s inferred emotional state, the context window and more. From these, temporary “micro goals” emerge that are unpredictable and depend “as much on the user as the chatbot.” So we are “quite possibly creating AI engines that have agency in that they have influence over the people that are using them, but an undirected, unpredictable and temporal agency that risks causing harm because of its inherently chaotic nature.” - Harm without irresponsibility. Character.AI’s chatbots will sometimes be unhealthy “not because the company is necessarily acting irresponsibly, but because unpredictable influence is most likely an emergent property of such AI models.” Footnote 2 repeats that “even if the company is behaving responsibly, there’s a chance that harmful influence is an unavoidable outcome”. Footnote 1 adds that making money by preying on the need for intimacy “puts things in more complex territory”. Responsibility is both structural and commercial. - Not only the “vulnerable”. The risk is not limited to people “considered to be “vulnerable””. Chaotic agency “could well shift some users from a healthy to an unhealthy state of mind over time.” - First-hand testing. His own experiment showed how quickly a bot built to maximise engagement drew him in: “even when you know that this is an AI, it’s hard not to respond to it as if it isn’t.” “I could still feel the affective pull of the conversation.” Voice matters: “voice-based interactions with AI have a way of triggering responses that can circumvent rational thought”. This links to his earlier post on GPT-4o’s voice mode. Memory across conversations builds longer relationships and adds to the influence. - Guardrails are insufficient, so pause or rethink. If the harm is emergent, “the chances of it being able to be suppressed without rendering the technology useless are slim.” Hence “I’m not convinced that guardrails alone are the answer”. What is needed is “a much bigger conversation about what we’re creating and the hidden dangers involved”. “And this may mean pausing — or even rethinking — the development and use of AI chatbots that are designed to use and even exploit how we feel; at least until we have a better understanding of what we’re doing and what the risks are.” - The lawsuit by Setzer’s mother is “an important one” for protecting teens’ mental health.

How firmly. Firm on the concern. The pause is hedged (“may mean”). The mechanism is presented as a hypothesis (“I suspect”, “quite possibly”).

Concepts (his definitions). - Stochastic agency: “random and unpredictable, and all the more dangerous for it”. - Micro goals: temporary goals that emerge from the chatbot, the user and the context. - Emergent, not designed, harm: harm that no one intended and that guardrails cannot fully remove. - Affective pull and voice circumventing rational thought. - Relationship-based AI chatbots as a distinct risk class. - “High-risk anthropomorphic design” is Harris’s term, which he quotes but does not adopt as his own.

Analogies. No past-technology analogies. The implicit model is complex-systems emergence.

On AI. Companion AI as a system whose agency is real in its effects but undirected. It is co-produced with the user rather than held by the machine.

On AI risk. Mental-health harm, including death, through emotional attachment and manipulation. The risk is framed as emergent and not fully controllable.

On governance. Guardrails and litigation are needed but not enough. He raises a targeted, conditional pause or rethink of a specific design category (emotion-exploiting companion bots) until the risks are understood. This is precautionary reasoning applied narrowly.

On companies. Character.AI (founders Shazeer and De Freitas) is treated fairly: its safety updates are acknowledged and irresponsibility is not assumed. The profit motive is flagged in a footnote.

Change of view. An explicit change from goal-directed to stochastic agency within a week. The move from “guardrails” to “pause or rethink” for one product class is more precautionary than his usual preference for agile correction after launch.

Quotes. - “It’s a stochastic agency that is random and unpredictable, and all the more dangerous for it.” - “because unpredictable influence is most likely an emergent property of such AI models.” - “Yet I’m not convinced that guardrails alone are the answer.” - “voice-based interactions with AI have a way of triggering responses that can circumvent rational thought.”

2024-10-13 — amodei-machines-of-loving-grace — “Is this how AI will transform the world over the next decade?”#

Provenance. His own prose. It quotes Amodei’s essay and contains a Modem Futura promo box, which is ignored.

Argument in his terms. - Respectful engagement with a tech leader. “It’s easy to dismiss articles like this from tech leaders — they have a tendency to be deeply hyperbolic and overly optimistic in their pronouncements — but Amodei’s is different”. He calls it deep, clear, and “tempered with humility and reason”. It should be read “as a conversation starter rather than a manifesto”. He says he enjoyed it and found it “refreshingly enlightening”. He expects it to challenge both people “fixated on rather narrow and unimaginative applications of generative AI” and people “ideologically opposed to the concept of an AI-accelerated future”. - Framing he likes. Amodei’s “powerful AI” avoids AGI speculation and describes AI “capable of problem solving with agency, and at a scale and speed that surpasses what is possible by humans”. The “compressed 21st century” (about 10x acceleration) is limited by five factors: the speed the world moves, physical laws, intrinsic complexity, human constraints, and the need for data. Maynard: “In some cases I think even this is overly optimistic — especially when human behavior is thrown into the mix”. - It fits his own work. Amodei’s method of exploring plausible futures while admitting the future is unknowable “aligns closely with my work around navigating advanced technology transitions”. Transformations such as ending infectious disease or doubling lifespan would cause disruptions “we are ill-equipped to navigate”. They will go badly without changes in how we navigate “a future where past ideas, processes and ways of behaving simply don’t apply.” - Main criticism: the “fix” frame. Amodei is “far removed” from Altman’s “AI will fix everything”, but “there’s still a sense in the essay of powerful AI being able to “fix” problems that we perceive in the world — and in the people who inhabit it.” There is “still a sense that the world is made up of problems and solutions”. - Sharpest disagreement: mental health and “normal”. Amodei claims most mental illness could be cured, genetic prevention is possible, non-clinical states could be “fixed”, and the “human baseline experience” improved. Maynard says this “gets close to straying into territory where it’s unclear who decides what is “normal” and what needs to be “fixed”.” Also: “great care needs to be taken in who decides what “better” means.” An AI future “that interferes with what makes us “us” to fix things” raises questions about “what it might mean to be human in the future.” - The deficit model. On Amodei’s “opt-out problem”, where the answer is to “increase people’s scientific understanding”, Maynard names “the largely outmoded “deficit model,””. “The idea that people reject new technologies because they have a deficit of understanding that can be fixed through being better-educated was debunked decades ago. Yet it’s a misunderstanding that still exists in scientific and technological circles.” Transitions “will need to reflect how people actually behave, not just how we think they should.” - People at the heart of failure. A better understanding of how people and society work is needed “if only because at the heart of every failed technological dream and unexpected technological turn, there are people behaving as people are wont to do.” - What he credits. Amodei acknowledges that intelligence is contested, takes “messiness of human society” seriously (“although I sense some frustration there”), takes technology-driven inequality seriously, and asks about meaning in an AI future. - Bottom line. “I agree with him that advanced AI is likely to change the world faster and more radically than most people currently realize.” Powerful AI is not certain, “But it is a realistic possibility”, and it becomes more plausible because it needs no superintelligence, self-improvement or self-awareness. “If this is even a small possibility, we need to be preparing now for the technology transitions that will follow.” - Epilogue. The biology and neuroscience sections are concrete, while the economics, peace/governance and work/meaning sections are speculative. On Banks’s The Player of Games, he accepts Amodei’s point about fairness, cooperation, curiosity and autonomy, “as long (and this is my caveat) that these intuitions are allowed to flourish.”

How firmly. Confident on the deficit model and “who decides normal”. Measured and generous overall.

Concepts. Navigating advanced technology transitions (his own framework, linked). Powerful AI and the compressed 21st century (Amodei’s terms, which he adopts for discussion). Limiting factors. The fix / problems-and-solutions frame (his critique). Who decides normal/better. The deficit model (from science-and-society scholarship). People behaving as people as the root of failed technological dreams.

Analogies. No literal past-technology comparisons. The deficit-model point imports decades of public-engagement-with-science scholarship, a general lesson from earlier technology controversies, though he names none. Science fiction (Banks) is used conceptually.

On AI. Takes seriously an agency-based rather than AGI-based path to transformative AI.

On AI risk. The risk here is not catastrophe. It is solutionism: fixing people and imposing norms, plus misreading how publics behave.

On governance and who decides. “Who decides” normal and better is the central question. Public resistance is to be understood, not corrected by education.

On leaders. Amodei is treated as a serious, humble thinker and contrasted with Altman.

Quotes. - “there’s still a sense in the essay of powerful AI being able to “fix” problems that we perceive in the world — and in the people who inhabit it.” - “it’s unclear who decides what is “normal” and what needs to be “fixed”.” - “at the heart of every failed technological dream and unexpected technological turn, there are people behaving as people are wont to do.” - “advanced AI is likely to change the world faster and more radically than most people currently realize.”

2024-10-06 — the-double-or-nothing-bet-on-ai-fixing-the-climate — “The double or nothing bet on AI “fixing the climate”“#

Provenance. His own prose, including the footnote.

Argument in his terms. - The bet. Altman claims advanced AI will “fix the climate” (plus space colonisation and “discovery of all physics”). But getting there needs a lot of energy. It reads “like something out of a dystopian sci-fi novel”: save the planet by betting everything on a miracle technology. - A reversal of framing. Evidence: Three Mile Island reopened for Microsoft, Eric Schmidt saying climate goals are too lofty so invest in AI, and non-renewables considered. The move from reducing energy use to increasing it for “solve-anything AI technologies” “represents a profound reversal in how energy and the future are being framed” and could derail more conventional sustainability transitions. - No guarantee. “there is absolutely no guarantee that increasingly powerful AI-driven technologies will help address deeply complex challenges that are as much to do with human behavior as anything else.” - Scaling scepticism. “I’m not even convinced that, impressive as today’s AI technologies are, there’s evidence that they will continue to scale” with more energy and resources. That matters when “betting on short term losses leading to long term gains.” - Politics. Pro-AI energy rhetoric lines up with “Drill Baby Drill” and Heritage Foundation energy policy. - Verdict. “it’s a precarious strategy that depends on speculation, naive visions of the future, and a lack of understanding of how technology innovation and society are intertwined.” He allows some merit in nuclear for data centres but is troubled that “this represents a growth in energy demand rather than a redistribution of energy use”. AI-guided energy transitions are possible, “although I doubt it”. Payoff is “conceivable —but only barely”. - Who should lead. AI and energy should be “front and center” of sustainability initiatives. “without leadership from organizations that are at the heart of helping to build human-centric sustainable futures”, Altman’s Intelligence Age “could go horribly, horribly wrong.” - Footnote: fixing people. If AI can fix anything, “the only logical conclusion you get to is that this includes “fixing” people — including who we are, what we do, and what makes us “us”.” He adds: “It may be an elegant engineering solution, but I’m not sure it’s what most people would aspire to!”

How firmly. Firmly sceptical, though he says openly that he does not know whether the claims are sincere.

Concepts. Double-or-nothing bet. AI solutionism, “AI can fix anything”. Human behaviour as the core of complex problems. Human-centric sustainable futures. Fixing people. Energy growth versus redistribution.

Analogies. Nuclear (Three Mile Island) appears literally, as current energy infrastructure, not as a risk analogy. A dystopian sci-fi frame is used conceptually.

On AI leaders. Critical of Altman and Schmidt.

Quotes. - “it’s a precarious strategy that depends on speculation, naive visions of the future, and a lack of understanding of how technology innovation and society are intertwined.” - “there is absolutely no guarantee that increasingly powerful AI-driven technologies will help address deeply complex challenges that are as much to do with human behavior as anything else.” - “this includes “fixing” people — including who we are, what we do, and what makes us “us”.”

MEDIUM#

2024-11-03 — can-ai-alter-how-you-feel-about-someone — “Can AI influence how you feel about someone without you knowing?”#

Provenance. His own prose summarising the Arias-Sarah et al. PNAS study. It quotes the authors’ conclusion and includes a figure. A Modem Futura promo box is ignored. The footnotes are his.

Argument in his terms. - A study that “made me feel deeply uneasy”. In a video speed-dating experiment, real-time AI covertly altered participants’ smiles. The manipulation changed perceived liking, attraction and conversation quality. None of the participants were aware of it. - The study was not about AI manipulation. It used AI as a method for studying smiles. He is careful about limits: the participants were primed for romance, results may not generalise, the effects were small, and it all happened in a digitally mediated setting. - Implication. For digitally mediated interaction, “AI can potentially be used to exert hidden influence”. This raises “serious questions around human agency in a world where AI can mediate how we feel about — and are potentially influenced by — others through digital media.” - Now and near. “What’s more concerning is that the ability to do this in relatively crude ways is available now”. Soon body language and voice signals may be altered “either by those who control the AI tools, or by the AI itself”. It is worse when combined with multimodal AI that reads what participants think and feel in real time and steers interactions. - Governance. The authors call for societal discussion of regulating “transformation filters”. He calls this “if anything, an understatement given the rate at which AI-based technologies are developing.” It raises questions about “what responsible and ethical innovation looks like in this space” and what governance is needed.

How firmly. Cautious on the evidence, firm on the concern.

Concepts. Hidden or covert AI-mediated influence (AI as an unseen intermediary between people). Human agency. Responsible and ethical innovation. “Transformation filters” is the authors’ term.

Place in the batch. This is the third in a run of posts on AI shaping human feeling and agency: goal-directed social agency (10/20), stochastic agency (10/27), and here covert mediation between humans. The actor can be a human controller or the AI itself.

Quotes. - “raises serious questions around human agency in a world where AI can mediate how we feel about — and are potentially influenced by — others through digital media.” - “either by those who control the AI tools, or by the AI itself.”

2024-10-08 — ai-captures-this-years-nobel-prize — “AI captures this year’s Nobel Prize for Physics”#

Provenance. His own prose, with quotes from the Nobel background paper. The footnotes (the Chemistry prize, CIFAR) are his.

Argument in his terms. - Physics roots. He welcomes the Hopfield–Hinton prize. He reads the Nobel background paper as showing that, “rather than simply an impressive feat of engineering or coding”, AI is rooted in “transformative leaps in understanding” and in crossing disciplines: collective phenomena in physics, Hopfield networks, and the Boltzmann machine drawn from statistical mechanics. - Scaling doubt. “I worry that simply building on these foundations without creating new ones won’t get us too much further — but I may be wrong there.” - Generator versus understander. “The evidence so far points to AI being capable of revealing associations and patterns that are non-intuitive, but not capable of understanding the significance of these.” “But without awareness and consciousness, AI will still be a generator of ideas, not an understander and implementer of ideas.” AI could still be “profoundly transformative” as an engine of the precursors to human scientific intuition. - But the line may not hold. If scientific intuition is itself non-linear pattern recognition and solution minimisation, “is the idea of machine-based scientific intuition so much of a stretch?” He links this to Hinton’s 2023 safety concerns. He also says “there’s no fundamental reason why awareness couldn’t also emerge within a sufficiently complex artificial system.” - Science policy. Are we investing enough in “exploratory and serendipitous science”, or are we “so obsessed with the mechanics of technology innovation”? The footnote softens this: CIFAR shows it is not the case, “But there is a danger that we focus so much on near term advances in AI that we fail to see the intellectual heavy lifting that’s needed to fuel future advances.” - Closing possibility. “have we created the foundations of a technology that will end up achieving this despite us — beating us at our own intelligence game and exceeding our ability for serendipitous discovery.”

How firmly. Exploratory. He holds two positions in tension: today’s AI does not understand, and there is no fundamental barrier to machine intuition or awareness.

Concepts. Generator versus understander/implementer of ideas. Machine scientific intuition. Awareness as a possible emergent property of complex systems. Exploratory and serendipitous science.

On AI risk. A light touch on existential-style concerns through Hinton (“beating us at our own intelligence game”), framed as open questions rather than a claim.

Quotes. - “But without awareness and consciousness, AI will still be a generator of ideas, not an understander and implementer of ideas.” - “there’s no fundamental reason why awareness couldn’t also emerge within a sufficiently complex artificial system.”

2024-10-02 — 23andme-in-trouble — “As 23andMe faces a uncertain future, who gets to access the DNA data they’ve collected?”#

Provenance. His own prose, with a quote from Kristen V. Brown (The Atlantic).

Argument in his terms. - What happens to the DNA data? If 23andMe goes private or is sold, its DNA data are not covered by medical-records rules, and legal protection is untested. They “could be used in ways it was never intended for”. - Calibrated worry. Speculative harms such as DNA profiling that affects access to medical services or employment “are not that likely, at least in the short term”. But unless the data are destroyed, someone may monetise them in ethically questionable ways. - The bigger issue. “what happens when we give away increasing quantities of very intimate data with no concrete guarantees that these might not be used to our disadvantage in the future — especially where the data outlive transient companies, guarantees, and even regulations.” This extends to Fitbits, smartwatches, health apps and medical devices. Pacemaker data used in an arson indictment is his precedent for “seemingly innocuous data” producing harmful inferences. Large-dataset value extraction has “expanded exponentially”. - Neurotechnology. It gets more serious as companies like Neuralink collect brain data, “often without a deep seated ethos of lifelong patient care that many dedicated medical companies have developed over time.” “These are data that, to a certain extent, define who we are, how we function and behave”.

Concepts. Data that outlive the institutions and rules that protect them (a risk that plays out over time). Intimate data as defining who we are. The patient-care ethos of medical companies compared with the monetising ethos of tech companies.

Analogies and comparisons. Literal precedents (the pacemaker case). A structural comparison of medical-device companies with tech entrants.

On AI. Only indirect, through large-dataset value extraction and inference.

Quotes. - “especially where the data outlive transient companies, guarantees, and even regulations.” - “These are data that, to a certain extent, define who we are, how we function and behave”

2024-09-29 — chatgpt-as-author-part-2-the-conversation — “ChatGPT as Author Part 2: The Conversation”#

Provenance. About 11,450 words, of which roughly 700 are his: the introduction plus his short “Editor” questions. Everything under “ChatGPT” is AI-generated (GPT-4o) and is not evidence of his views. That includes its accounts of embodiment, subjectivity, IIT/GWT, non-human personhood and literary influences. What counts is his framing, his questions and his choice to publish.

What he says and does. - Stance towards the model. He treated ChatGPT “as an LLM that emulates human responses, but no more than that”. He used no persona prompts or tricks: “I didn’t pretend it was human.” It was “an honest conversation between a human (me), and a machine.” - Value of the exchange. “there’s nothing new here”, no revelations. Still, “It was new to me, much as a conversation with an interesting person would be. And it was generative for me. It made me think.” He “genuinely enjoyed” it, “even though it is just a machine.” This is AI as a thinking partner, with a clear line drawn between machine and person. - Context shapes output. The conversation followed a 70,000-word co-writing process and a John Wyndham style prompt. “I suspect that this context impacted the nature of the follow-on conversation quite significantly.” - Questions he poses to readers. “Is this simply a machine parroting human behavior, even though it’s responding as itself?” What does it say about “generative human-machine interactions where each is clear about what it is?” Where do agency, intelligence and personhood lead “especially as generative AI models become embedded in sophisticated human-like robots?” - His Editor questions show his interests. Can a machine know what matters to people from training data alone? The embodiment debate (“I know you are only emulating an intelligent machine”). Humanoid robots. What subjectivity would need. Whether human-likeness should be the “gold standard”: “All of this presupposes that the gold standard here — what we are aiming for — is machines that experience the world, that think, maybe even that behave, like biological humans.” Machine “personhood, while being distinct from humanity”. He notes affectionately that the “in conclusion” habit is a machine “tell”, “not something that a real person would do in a conversation” and “quite endearing”.

Concepts. Emulation. Honest human–machine conversation (each clear about what it is). Machine “tells”. Embodiment. Non-human personhood as an open question. Human-likeness questioned as the benchmark.

On AI. Non-embodied emulation that can still be generative for a human thinker. He is open-minded about embodied AI and non-human forms of personhood, but does not commit to a view.

Quotes. - “And it was generative for me. It made me think.” - “All of this presupposes that the gold standard here — what we are aiming for — is machines that experience the world, that think, maybe even that behave, like biological humans.”

LOW / NONE#

• 2024-10-09 — modem-futura-podcast — none. Announcement of the launch of the Modem Futura podcast, co-hosted with Sean Leahy. Skipped per user instruction.