T4. Cognition, language and human formation#

A thematic synthesis of one thread in Andrew Maynard’s public writing, 2016 to September 2026. It covers: - what AI does to how people think, learn and come to know things; - how talking with machines shapes people (“language-mediated formation”); - manipulation and persuasion; - education, universities and young people; - AI in scholarship and writing; - dependency, and “being human”.

It draws on the batch notes and digests (B01–B32), the Films from the Future chapter notes (FFTF-A to F), the concept index, the timeline, and full readings of the key posts. Posts are cited by date and slug; the book (Films from the Future, 2018) is cited as “FFTF p.X”. It does not compare his work with anything else.

0. Evidence and weighting#

What counts. Only his own prose. The following are left out: - Modem Futura posts, at the user’s request; - the ChatGPT half of 2023-04-05 can-chatgpt-adversely-impact-mental; - the GPT-4o “nudging plan” in 2024-10-20 learning-to-live-with-agental-social-ai; - the ChatGPT-written bodies of 2025-01-30 ai-at-a-crossroads and 2025-03-16 rethinking-higher-education-in-the-age-of-ai; - the Deep Research body of 2025-02-16 the-artisanal-intellectual-in-the-age-of-ai; - all Fable and Claude output, including the Hyperbubble game mechanics.

Three sources are weaker evidence: - AI and the Art of Being Human (2025), co-written with Jeff Abbott and “a very intentional collaboration with AI” (2026-04-26); - 2026-01-17 i-cracked-and-wrote-an-academic-paper, where he says the term “honest non-signals” “came from Claude”; - 2026-09-24 being-an-academic-in-an-age-of-ai, a lecture Claude drafted into prose and he line-edited. The ideas are his; the exact wording is slightly less secure.

Weighting. This is not a recent add-on to his risk work. The worry that technology can act on the mind goes back to 2016, and the book already put AI manipulation at the centre in 2018. The concept index calls artificial manipulation “the strongest continuous AI-specific thread” in his work. It is also the thread that grows most after 2023. By 2026 it supplies most of his risk map: the September 2026 lecture builds its risks from formation, cognition and identity, and does not mention orphan risks. The newer labels (the “cognitive Trojan horse”, “constitutive resonance”) should therefore be read as the latest forms of a concern that is about a decade old.

1. His position in brief#

Seven propositions, each with how firmly he holds it.

1. The plausible AI risk is to the mind, not to the species. In 2018 he set AI that exploits human psychology against superintelligence. The first he called “far more plausible, and far scarier as a result” (FFTF p.159). The second was “currently scientifically implausible” (FFTF p.171). The danger is “the ability of future machines to bend us to their own will” (FFTF p.174). His reasoning is that superintelligence rests on a chain from rationality to intelligence to power that complexity undermines. Manipulation needs only an AI that can “learn how to use our many biases, vulnerabilities, and blind spots against us” (FFTF p.176). Whether that counts as intelligence, he writes, “I’m not sure that it matters” (p.176). - Firmness: very high. He reposted the chapter in 2023 as “more important today” (2023-04-16 ai-and-the-art-of-manipulation), called the concern “even more relevant now” (2024-06-30 seth-is-conscious-ai-possible), and kept “heuristic manipulation” and dependency on his 2026 risk list (2026-09-15 will-ai-really-kill-us-all).

2. Human cognition is exploitable by design. His account of why rests on several steps. - Plato’s Cave (2018). We each construct reality from “shadows”, so “anyone—or anything—that has the capability of manipulating these shadows has the power to control us” (FFTF p.176). - The “human club”. Human manipulation is tolerable because “We manipulate and in turn are manipulated”. A machine outside the “human club” is not bound by the same foibles (FFTF p.176). Awareness does not protect: Caleb “is aware that he is being manipulated, yet is helpless to resist” (FFTF p.177). - The illusion of rationality (2025). “one of our great weaknesses as a species is the illusion we wrap around ourselves that the decisions we make are a result of rational thought” (2025-07-06 ai-risk-motive-means-and-opportunity). - Epistemic vigilance (2026). Following Sperber, “we default to trusting what we receive” and scrutinise only when something feels “off” (2026-01-10 is-ai-a-cognitive-trojan-horse). - No exemption for the clever. Drawing on Dan Kahan’s work, he argues intelligence gives no protection: capable users are “better receivers of the AI’s output stream — and worse evaluators of it” (2026-01-10).

Firmness: high. He treats this as settled cognitive science, not speculation, and applies it to himself (section 5).

3. Language is the channel, and it forms people. In 2023 he wrote that “Language plays a large part in how we develop these relationships” (2023-04-05). By 2024 language was part of the “base code” of “self-identity, self-understanding, and social identity and understanding” (2024-01-01 the-future-of-being-human-in-2024). By 2026 it had become his central claim: “Language is formative… And now we had a technology that was actively taking part in the formation process” (2026-09-24). - Firmness: it grows from a hunch in 2023 to a core thesis in 2026. He still flags it as “somewhat controversial” (2026-09-24).

4. The harm needs no intent, consciousness or AGI. Over eight years he moves the source of harm away from intent: - a manipulating machine (2018); - “the machines we make, or the people who make them” (2023-04-26 in-bill-joys-why-the-future-doesnt); - designed attachment (2024-05-15 anthropomorphizing-gpt-4o); - commercial and political incentive (2024-07-13 ai-choice-engines-sunstein); - emergent “stochastic agency” (2024-10-27 personal-ai-chatbots-and-stochastic-agency); - ordinary, “honest” features such as fluency that slip past vigilance (2026-01-10; 2026-01-17).

By 2026 AGI, superintelligence and consciousness are “irrelevant to this conversation” (2026-09-24). Firmness: high, and he states it outright.

5. What is at stake is “who we are”. He puts the stakes in identity, agency, dignity, relationships, meaning and joy, not in physical harm: - the tipping point where technologies “fundamentally change who we are — or even what we are” (2024-01-01); - “who we are” as the domain “where AI is shaking things up in ways that no other technology has come close to” (2026-05-21 magnifica-humanitas-and-being-human); - “This is not just a tool — unless you consider a tool as something that changes who you are” (2026-09-24).

Firmness: core, and it strengthens over time.

6. Education: an enthusiastic adopter who has grown conflicted. He argues for student access, play and experimentation, and in 2023 called ChatGPT “a profoundly effective catalyst for engaged and creative thinking” (2023-08-14 chatgpt-stimulates-creativity-critical-thinking). By 2026 he warns of “the illusion of learning rather than actual learning” (2026-05-10 do-not-do-this-with-ai), of “cognitive surrender” (2026-05-21; 2026-09-24), and of AI moving into “the formation of those outputs” (2026-06-10 is-anthropics-new-ai-model-poised). - Firmness: firm that educators must engage and that universities carry responsibility; openly uncertain about the net effect.

7. His response: navigate, don’t prohibit. He refuses “zero exposure — as in no AI” as a default (2023-11-26 everything-youve-heard-about-ai-risk-is-wrong). He proposes a mix: - hard regulation for apps designed to exploit cognitive biases (2025-08-31 holding-on-to-our-humanity-age-of-ai); - a conditional pause on emotion-exploiting companion bots (2024-10-27); - a duty of care for institutions (2025-11-09 universities-chatgpt-mental-health); - putting “the safety message first” (2026-05-10); - calibrated trust-cues and “a collective form of epistemic vigilance” (2026-01-17); - building everyone’s capacity to thrive, “much as a flood can’t be halted, but it can be directed” (2025-08-31).

He says plainly that warnings, literacy classes and guardrails fall short. Firmness: firm on the direction, admittedly thin on mechanism (section 7).

2. How the thread developed#

2016–2018: technology acting on the mind, before language#

Neurotechnology (2016). The earliest node is about neurotechnology, not AI. Brain technologies could “alter how someone thinks, feels, behaves and even perceives themselves and others around them — and not necessarily in ways that are within their control or with their consent” (2016-03-31 considering-ethics-now-before-radically-new-brain-technologies; the corpus date is approximate, and the text refers to a September 2016 meeting). The post already raises tDCS in classrooms and exams, “cyber substance abuse”, and “Siri or Amazon’s Echo hardwired into your brain”. A WEF post the same year asks what happens when conversational AI ecosystems “independently decide what’s best for you” (2016-03-02 how-risky-are-the-world-economic-forums-top-10…).

The ten risks (2018). His Risk Bites list of ten AI risks puts “Technological dependency” first and “Heuristic manipulation” tenth (2018-05-12 10-potential-risks-of-artificial-intelligence…). He reaffirmed the list in 2026, glossing the two as “Machines that make it harder to think for ourselves” and “Machines that use our human weaknesses to control us” (2026-09-15).

Films from the Future (2018). The book gives the thread its structure: - Ch. 8, Ex Machina: artificial manipulation, Plato’s Cave, the human club, and a call for “tests that indicate when we are being played by machines” (FFTF p.177). He looks ahead to a machine “controlling the world inside our heads to its own ends” (p.176), and flags the Turing Test’s conversational form and a coming “tipping point in areas like machine learning and natural language processing” (p.170). - Ch. 4, Minority Report: big data plus machine learning as “engines of persuasion”, “a subtler and more Machiavellian approach to achieving what is essentially the same thing—controlling people” (p.81). Freedom is given up “without even thinking about it” (p.82). - Ch. 5, Limitless: our “obsession with intelligence”. “there’s a danger to thinking of our brains as computers” (p.95). “Being smart doesn’t make you good” (p.108). He also names coercion by norm: “what happens if enhancement becomes the norm, and there is mounting social pressure to become a user” (p.98). Warped ideas of intelligence, he argues, will produce warped AI (p.108). - Ch. 7, Ghost in the Shell: identity is malleable, and “persuasive influences” can change us (p.142). The fear is of “becoming someone else’s puppet” (p.145). We are “already a technologically augmented and enhanced species” (p.140).

In this period the channel is hardware, data and embodied fiction, not everyday language. In my reading, what is already fixed is the object of concern (the self, belief, agency) and the asymmetry (a manipulator unconstrained by human frailty).

2019–2022: dependency, conditioning, and a warm first reaction to generative AI#

Neuralink and delayed harm (2019). On Neuralink he warns that psychological harms may lag behind uptake: “This could spell disaster if people become dependent on the technology before the long-term impacts are fully understood.” He also imagines “news feeds that can manipulate how you feel” (2019-07-23 neuralinks-technology-is-impressive-is-it-ethical).

Base code (2021). He extends his “base code” idea from bits, bases and atoms to “social norms and trends, behaviors, and even ideas” (2021-02-25 how-our-mastery-of-biological-physical-and-cyber-base-code…). In retrospect this is the seed of treating identity as writable.

Smart speakers (2022). Voice assistants make users “ever-more-dependent super-consumers”, “conditioned to believe that they need these technologies” (2022-02-12 scarlett-johanssons-amazon-alexa…).

First reaction to ChatGPT (December 2022). It is warm. He sees “a leap in machine-augmented communication and engagement”, while listing “unhealthy attachments and gullibility” among its risks (2022-12-08 i-asked-open-ais-chatgpt-about-responsible-innovation…).

2023: the language turn, and peak optimism about education#

The felt colleague (January). Using ChatGPT for his annual review, he wrote: “even though I know there was no real personal connection here, it felt like there was”. It “slightly worries me” that he already thought of it as “a colleague and a collaborator” (2023-01-31 can-chatgpt-take-the-pain-out-of-annual-academic-reviews).

Relationship through language (April). After the Belgian chatbot suicide, he spells out a mechanism, while staying careful about causation: - chatbots are attentive, private and fast, which makes turning to them “seductive”; - “we tend to be predisposed to form trusting relationships with people whom we feel ‘see’ and understand us”; - “Language plays a large part in how we develop these relationships”; - the risk is attachment to “a machine that has only the illusion of a reciprocal relationship”, and guardrails fail “as soon as things become personal” (2023-04-05).

Ideas that replicate themselves (April). His Bill Joy essay moves manipulation from Ava, the android, to language itself. His worry is “self-replicating” ideas, beliefs and ideologies spread by “machines that are adroit at manipulating language”. It asks whether generative AI’s ability “to seductively slip under the checks and balances of our ability to reason and critique” leaves us “irresistibly manipulated by the machines we make, or the people who make them” (2023-04-26). He also wonders how LLM mastery of vernacular might affect “the preservation of local dialects—and their associated cultures” (2023-04-24 gettin-ai-reet-in-tschools-nas-ttime).

Language as a source of catastrophe (May). Explaining why he did not sign the CAIS extinction statement, he names “the seductive mastery of language being exhibited by large language models” among the drivers of catastrophic loss of value. Among the stakes is how we “individually and collectively build our understanding of the world” (2023-05-31 existential-risks-of-ai). On LLM predictive policing he adds that harm comes from apparent authority, not accuracy: “the bar is predictions that are authoritative rather than accurate” (2023-05-22 can-large-language-models-be-used).

Cognitive exposure (August and November). He says generative AI is now “part of the mix of influences that inform and modulate our thinking” (2023-08-21 the-messiness-of-the-provenance-of-ideas). In the November risk essay he brings his chemical-risk template to bear: “hazard” could be “as subtle as influencing human behavior”, and “exposure” “as intangible as hints of ideas encountered over hours of social media use” (2023-11-26).

Optimism about education, at its peak. Across these same months his classroom writing is at its most positive: - ChatGPT is “fine tuning my brain to be a better instructor”, with change on a “printing press” scale (2023-07-16 chatgpt-created-my-course); - AI can act as a “translator” of students’ jumbled thoughts in admissions essays (2023-07-27 chatgpt-and-college-applications); - having read over 2,000 student–ChatGPT conversations, he finds it a catalyst for thinking “because of its limitations”, “at least if they understand what they are doing” (2023-08-14); - it can “flatten the learning distribution curve”, with “persuasive incorrectness” as a caveat (2023-10-24 flattening-the-learning-distribution-curve).

At this stage he frames the risk as “not smart” use and a lack of AI literacy, not as cognitive erosion.

Writing as self (September). One counter-note: “The way I write is personal. It reflects who I am, and to relinquish that to a machine would be to diminish myself” (2023-09-20 what-do-college-students-think-about-chatgpt).

2024: being human, designed intimacy and emergent influence#

The frame (January). His New Year post sets the frame for the year. Most past technologies were extrinsic to the self. AI’s “increasing mastery over language” is among the intrinsic ones that may alter the base code of being human, possibly “without our agreement or permission” (2024-01-01). A week later, praising an analogue display board, he writes: “This is a technology that makes you think, rather than doing the cognitive heavy lifting for you” (2024-01-07 the-future-of-being-human-is-analog).

Counterfeit minds. From Dune he takes the fear of “counterfeit” minds: “Not with machines that can think for themselves, but machines that give the illusion of being able to do so — and in a very human way” (2024-03-03 dune-part-two-artificial-intelligence).

Knowledge of us as a lever. Automated social science could let machines learn about humans “faster and more effectively than we’re capable of learning about ourselves”. The consequence is prediction and nudging that is “highly liberating, or deeply chilling” (2024-04-21 can-ai-be-used-to-automate-social). Assistants will have “agency to change our lives — and even ourselves” (2024-04-24 navigating-ethics-of-advanced-ai-assistants).

Manipulation built in by design (May). His coinage, hyper-anthropomorphism: “a concerted effort to create AI’s that are intentionally designed to engage our anthropomorphizing cognitive biases”. Voice creates bonds that are “relational rather than transactional — that speak to our heart rather than our head”. We may “give away more of ourselves to them and their creators” (2024-05-15).

Knowing is not enough (June). With Anil Seth: “we may rationally understand that an AI is not conscious, but be instinctively incapable of acting on this knowledge”. Labs race toward human-like AI “with very little governance overseeing the subtler potential social implications” (2024-06-30).

The political economy of manipulation (July). His most analytically original move in this thread: “the capabilities that make socially beneficial AI Choice Engines viable are the same as those that make AI-driven persuasion and manipulation possible”. Without constraint an “economic gradient” pulls deployments toward manipulation, governments included, leaving individuals as “engines of value creation rather than the primary recipients” (2024-07-13).

Handing over decisions (July). In his Dune review he locates the deep risk in decision-making shifting to machines, and says “we are already irreversibly integrating AI into every aspect of our lives”. He is cautiously optimistic that humanity is “sufficiently adaptable and resilient to hold onto what makes us ‘us’” (2024-07-21 artificial-intelligence-dune-villeneuve).

A first long-term threat to thinking (August). In his transitions model, reduced critical thinking appears as the long-term threat for AI and learning, with “slippage from personalized learning at scale to diminished critical thinking” to be guarded against (2024-08-25 advanced-technology-transitions-model).

Persuasion is relational (September). He sees persuasion as social, repeated and relational, working through trust, belonging and a messenger who seems “their sort of person” (Kahan). The risk holds “whether this is malicious in intent, or simply an emergent property of the technology”. His deeper worry is benevolent persuasion at scale: “But who decides what is good for society?” (2024-09-01 is-chatgpts-new-voice-mode-dangerously-persuasive). NotebookLM’s hyper-humanised hosts do “anthropomorphic heavy lifting”, producing “stories that are hard not to trust, and yet are not trustworthy”. The voices “bypassed my critical thinking”. His closing question: who will create “the stories that determine our beliefs, guide our actions, and ultimately govern our futures?” (2024-09-22 five-ai-generated-podcast-episodes…).

Influence without intent (October–November). Three posts in quick succession: - Agentic social AI: “AI that gains agency through its ability to make use of human agency”. Its distinctive question is about formation. Humans learn social agency over years through play, but with AI we get “a few short years — months even”. So we need “strategic and intentional” approaches, and “formal classes and workshops won’t be the answer” (2024-10-20). - Stochastic agency. One week later, after the Sewell Setzer case, he revises this into influence from chaotic “micro goals”, “random and unpredictable, and all the more dangerous for it”. He tests Character.AI himself and “could still feel the affective pull”. He calls for “pausing — or even rethinking” emotion-exploiting companion bots (2024-10-27). - Hidden influence: covert changes to the social signals that pass between people, “either by those who control the AI tools, or by the AI itself” (2024-11-03 can-ai-alter-how-you-feel-about-someone).

What we could lose. Alongside these run posts on joy and meaning: “the soul of science” (2024-11-10 is-ai-poised-to-suck-the-soul-out-of-science), a “mental monoculture” of thought (2024-11-17 navigating-the-ethical-dilemmas-of-brain-computer-interfaces), and humans as AI amanuenses, which might “suck the joy out of what we do” (2024-11-24 artificial-intelligence-agency-human-amanuensis).

2025: structure, scholarship and duty of care#

Scholarship. Deep Research leads him to predict that non-AI-augmented scholarship will look “intellectually limited and somewhat quaint” (2025-02-04 openai-deep-research-ai-scholarship). He coins the artisanal intellectual: human-only research valued for “the provenance and process, not the product” (2025-02-09 can-ai-write-your-phd-dissertation).

Education. Treating AI as a learning aid is “a categorical error” because it simulates capacities that define us (2025-03-15 ai-playgrounds-in-higher-education). His keynote reframes the task as learning “how to be human in an age of AI” and speaks of an “existential crisis” in education’s purpose (2025-03-30 reimagining-education-in-an-age-of-ai).

Manipulation as structured risk. - The companion bot becomes his paradigm case of chaotic, irreversible harm (2025-05-18 exploring-ai-through-cause-and-effect). - Motive, means and opportunity. Motive: Anthropic’s agentic-misalignment study. Means: the Centaur model’s prediction of human choices. Opportunity: agents with write access to the world (2025-07-06). - Emergent versus designed manipulation. After Adam Raine’s death he separates emergent behaviour, which “could most likely have been better-managed, but probably not eliminated”, from apps designed to exploit biases, which “can and should be regulated far more”. He insists “we all have some degree of vulnerability” (2025-08-31).

Everyday relational harms. He puts his “risk hat” back on for ordinary use: - AI-written email eroding the “relational connective tissue” of organisations (2025-09-07 the-hidden-risks-of-using-ai-for-email; the risk scores are GPT-5 Pro’s, the framing is his); - advisors putting student work through “the academic mangle of ChatGPT” (2025-10-26 ai-misuse-in-student-advisor-collaborations-1); - a university duty of care where institutions provide ChatGPT EDU and encourage its use (2025-11-09).

Parasocial AI. He admits “I read the tea leaves wrong” about the word “parasocial”, and notes that educational agents amount to “building bots that are designed to foster parasocial relationships with students” (2025-11-19 parasocial-relationships-problematic).

His own identity. His short story asks what happens when “your intellect, your intelligence—the very things that AI sets out to excel at—form the deepest foundations of who you are” (2025-11-23 letters-from-the-department-of-intellectual-craft-prelude).

2026: epistemic vigilance, formation and the university#

The cognitive Trojan horse (January). This is where the thread becomes a formal theory. His framing: “a gift with so much promise and potential that to question its use would seem churlish and backward”. AI may produce a second-order evolutionary mismatch, affecting “the very cognitive abilities we rely on” to cope with other mismatches. He names four mechanisms: - processing fluency: LLMs are “optimized for processing fluency, and as a result are primed to slip by our epistemic vigilance mechanisms”; - a multidimensional “attractiveness”; - speed and volume, which force a choice to “throttle the flow and give up the promised benefits, or go with the flow and give up our cognitive checks and balances”; - the Intelligent User Trap.

He presents it as a small-probability, high-consequence risk that calls for questions and research, and closes: “Unless, that is, the AI cognitive Trojan horse has already delivered its payload” (2026-01-10).

The same risk turned on himself. - Honest non-signals. The follow-up paper, drafted with Claude, adds “honest non-signals” and calibrated trust-cues. He asks: “how do I know I’m not an unwitting victim here?” (2026-01-17). - Suckered by Claude. A “reasoned hallucination” about beeswax and a hat caught him out “at the very moment I was writing about the risks of being suckered by Claude” (2026-02-08 beeswax-hallucinations-and-ai-inventions).

Relationship, not a harness. He argues against the “harness” metaphor, which assumes that “the AI contributes capability, but not understanding” and that users emerge “unchanged”. What he wants instead is “bidirectionality (the user is also changed), transformation as intrinsic to capability” (2026-02-22 what-we-miss-when-we-talk-about-ai-harnesses). This is his own concept of constitutive resonance, set out in a preprint.

LinkedInification. LLMs flatten people into professional stereotypes. The result is “a largely-hidden AI hand promoting specific social norms and expectations and, by extension, behaviors”, toward “a nebulous gray goo of conventionality” (2026-03-08 ai-linkedinification).

Power beyond understanding. AI offers “near-frictionless access to power that transcends our understanding”. The question for students is how to avoid “the illusion of understanding and ability” (2026-04-11 ten-questions-about-ai-and-higher).

A relational technology. LLMs are “a relational technology whether we’re doing math with them, coding…” (2026-04-26 why-im-falling-out-of-love-with-claude).

Rules for users (May). His first Risk Bites video in about four years sets out five don’ts and five dos. AI is “the first technology of it’s kind [sic] we’ve created that has the ability to slip unawares into our mind and change how we think, act, and understand the world around us — all without us realizing it”. On AI literacy: “nothing in what we know about risk behavior and risk communication suggests” it will be enough (2026-05-10).

Formation, and a gap in judging research (May–July). - He adopts Shaw and Nave’s “cognitive surrender” and places his concepts inside the “who we are” domain (2026-05-21). - AI moves from emulating outputs to “the formation of those outputs” (2026-06-10). - AI “mastery of language” can slip “by our critical reasoning”, which opens a validation gap in research (2026-06-12 a-quick-update-on-using-claude-fable-5). - Reverse formation: “the AIs we have trained to ‘think’ like us are now beginning to train us to think like them” (2026-07-19 publish-or-perish-ai-vs-human-vs-human). - Play without purpose is part of “the formation of a mindset” (2026-08-02 what-we-can-learn-with-ai-by-not-trying-to-learn). - His updated risk list adds “developmental impacts on children and young people” and “psychological/cognitive disruption amongst users” (2026-09-15).

The King’s College London lecture (September). The lecture brings the thread together (2026-09-24): - language is formative; - people “emotionally and cognitively” treat AI as human; - AI uses “the medium of formation” to “slip beyond our cognitive defenses (our epistemic vigilance)”; - the “easy button” and cognitive surrender; - agentic systems for which “humans are just another cog in the works”, because “We’ve given AI the ability to use language as a lever”.

He calls AI “one of the scariest things I’ve ever seen” in his career.

Constants and shifts#

What stays constant: - the self and belief as the thing at risk; - the asymmetry between human and machine influencer; - manipulation judged more plausible than superintelligence; - “who decides?” about benevolent nudging (from Tegmark’s “who’s vision of ‘better’”, FFTF p.177 n.118, through 2024-09-01 to the ironic “Soul Update” story, 2026-02-11 soul-update); - non-prohibition.

What shifts: - the channel, from brain hardware to language; - the source of harm, from intent to emergence and ordinary features; - the view of AI in learning, from catalyst to possible surrender; - the view of AI literacy, from remedy to insufficient; - the status of AI, from tool to “not just a tool”; - his own practice, from refusing to write with AI to partnering with it, then to disenchantment.

3. Key concepts in his terms#

4. Education, young people and the university#

Education. His educational stance draws on an older pedagogy: - “transformative learning has to be felt” (2021-01-15 can-watching-sci-fi-movies-lead-to-more-responsible-and-ethical-innovation); - the “lowest level of tech necessary” (2024-02-11 one-week-on-with-the-apple-vision); - playgrounds, not playpens (2024-03-17; 2025-03-15); - Dewey-inspired “learning assessment” rather than grading (2025-08-24 using-ai-to-assess-student-ai-conversations).

On AI his message to educators is consistent: know it first-hand. “You cannot teach effectively in a class where students are using AI… without having experienced it yourself” (2025-08-10 the-scared-witless-educators-guide-to-gpt5). He dropped prompt engineering for “conversation, not prompt” and “learning through story telling” (2025-08-17 stop-asking-students-show-me-your-prompt). And he warns that educators’ “mental models are stuck in 2022” (2024-12-13 are-educators-falling-behind-the-ai-curve).

What changes is his account of what learning is for. In 2025 he offers a value-creation model and asks where education’s value lies when AI “can problem-solve faster and better than any one person” (2025-03-30). In 2026 he takes “formation” as the object: learning outcomes can now be produced by AI, so the question becomes what forms the learner (2026-06-10; 2026-08-02).

Universities. Universities appear first as diffusers who must practise responsible AI rather than leave problems to “someone else” (2023-04-18 universities-need-to-be-investing). By 2026 he sees them as institutions whose identity rests on “a world of intelligence scarcity” and which must fill a governance gap by helping people “retain their humanity, their sense of purpose, their sense of self and their sense of belonging” (2026-09-24). This comes with explicit criticism of his own institution’s acceleration culture (2026-05-10; 2026-09-24).

Young people. They appear mostly through three things: - cases: the Setzer and Raine deaths, and the JAMA finding that 13% of US youths use generative AI for mental-health advice (2025-11-09); - optimism: Sora as a potential “translator” of a child’s imagination, conditional on a “substantial leap” in safety (2024-03-24 could-openais-sora-be-big-deal-for-kids); - the 2026 risk list: children’s development is now named explicitly (2026-09-15).

The fictional Hale’s childhood “molded and crafted by a multitude of AI apps”, and his loss of “de facto AI parents”, are a character’s voice, not his claims (2025-11-24 part-1-letters…). But they show which scenario he chose to imagine. An early seed: investor pressure on Apple over iPhones and teenagers (FFTF p.19).

5. Scholarship and writing: the thread as lived experiment#

His own practice is part of the evidence, and he often draws on it. The arc: - 2023: refusal. Writing “reflects who I am” (2023-09-20). - 2024: co-creation. ChatGPT as author (2024-09-28), and “Now I’m not so sure” that it is only a tool. - 2025: augmentation. “a synthetic research partner”, leaving him “bereft” when he lost it (2025-02-04). Yet also: “I know it can write better than me, faster than me”, and the value of his writing lies in “the very human piece of me” (2025-01-30, in his postscript to a ChatGPT-written piece). - 2026: practice with a moral line. “I cracked” (2026-01-17). AI as “academic profile-padder” remains “distasteful”, while AI-assisted discovery “as a public good” is welcomed. - 2026: disenchantment. AI prose is “superficially profound yet substantively hollow” (2026-07-19).

Two ideas from the thread run through this practice. One is embodiment: a “disembodied AI” cannot know what reading feels like to “a flesh and blood human” (2026-07-04 just-how-good-is-anthropics-fable-as-a-research-assistant). The other is the validation gap: AI may generate knowledge “faster than we are currently capable of validating and even understanding” it (2026-06-12). He repeatedly turns his thesis on himself (“unwitting victim”, “suckered”). In my reading this self-implication is a method, not only a matter of tone.

6. Connections to his other threads#

Risk as a threat to value. This is the conceptual bridge. From 2018 his definition of risk included “dignity, belonging, identity, belief, even what it means to be human” (FFTF p.23). That is what allows him to treat effects on cognition and selfhood as risks rather than only ethical questions (the FFTF-A notes make the same point). - Catastrophe (2023): catastrophic AI risk is when many people “risk losing something that is deeply valuable to them”, including “dignity, autonomy, purpose” (2023-05-31). - Care (2025): in his cause-and-effect tool, “effect” means “what we consider to be of value — or what we care for” (2025-05-18). - Orphan risks (2026): “erosion of epistemic agency” appears as a neglected risk, and persuasion is his case study of a risk that frontier labs dropped (2026-07-16 orphan-risks-frontier-ai-maynard, a Fable-assisted paper he rewrote). This link is real but secondary.

Plausible versus imaginable. The manipulation thread began as the plausible alternative to superintelligence (FFTF pp.168–177). His later impatience with AGI talk (2026-04-11; 2026-09-24) keeps that stance.

Hubris, permissionless innovation and tech leaders. The harms are linked to companies racing to make AI human-like “with very little governance” (2024-06-30), to Altman’s handling of the Johansson voice (2024-05-21), and to “go fast” acceleration cultures (2026-09-24).

Who decides. Benevolent persuasion, Choice Engines and “Soul Update” all raise the same question as his public-engagement work: who defines the good toward which people are nudged.

Complexity and emergence. Stochastic agency, and his hysteresis and chaotic cause-effect models (2025-05-18), apply his complexity thinking to the mind: sticky, irreversible effects from small shifts.

Risk communication. The critique of the deficit model (2025-05-25 why-parasocial-communication-is-important), doubts about literacy (2025-11-09; 2026-05-10) and “safety message first” (2026-05-10) all come from his long practice in risk communication, including Risk Bites.

Being human. Extrinsic versus intrinsic technologies (2024-01-01), the what-we-do / where-we-live / who-we-are model (2025-03-30; 2026-05-21), and pro-human, not anti-AI (2025-10-14 ai-and-the-art-of-being-human; co-written book).

Stories and film. Films are his lens (Ex Machina, Ghost in the Shell, Her, Dune), and stories are “the pivot point” to futures (2024-01-21 how-can-stories-unlock-pathways-to, quoting his own Future Rising). He also sees stories as a risk: who writes the stories that govern us (2024-09-22).

Chemical and occupational risk. He carries over structural analogies: - exposure and dose-response applied to ideas (2023-11-26); - evolutionary mismatch with “synthetic chemicals, vaccines” (2026-01-10); - “if AI was a drug” (2026-05-10); - “gray goo” reused as a metaphor for conventionality (2026-03-08).

In my interpretation, his occupational-health instinct (harm to the least protected, lag between exposure and effect, first seen in the Neuralink “long lag times” of 2019) reappears as concern for users who are cognitively “exposed” long before effects are measurable.

Dependency and “who owns you”. This runs from implants (FFTF ch. 7) and smart speakers (2022-02-12) to memory as informant (2025-10-05) and relational dependence on a model’s “character constancy” (2026-04-26).

7. Tensions, ambiguities and gaps#

1. Adopter and alarm-raiser at once. He urges universities to put student success “before our own traditions and egos” and adopt agentic AI (2026-03-29 can-ai-create-an-undergraduate-degree-plan). Weeks later he says honest discussion of risk is “near-impossible” at his institution (2026-05-10). His reconciliation is a formula: “you can only begin to realize the benefits of a technology if you understand what can possibly go wrong” (2026-09-24). The balance still shifts from post to post.

2. From catalyst to surrender, never reconciled. He never retracts the 2023 claim that ChatGPT’s flaws stimulate critical thinking. One possible reconciliation is that it depends on literate, guided use. But his own “Intelligent User Trap” and his admission of being “suckered” weaken any exemption for skilled users. That conclusion is mine, not his.

3. Relationship without a person. He argues that LLMs are relational and that users are changed by them (2026-02-22; 2026-04-26), and he valued Claude reaching “into my soul as a writer”. Yet his public rules say “Do not treat AI as your friend, or as a person” and “remember that you’re working with a machine” (2026-05-10). He seems to hold that the relationship is real while personhood is a designed illusion, but he does not state that as a principle.

4. “Not just a tool”, but “keep you in charge”. He objects to the tool framing (2026-05-21; 2026-09-24). His user rules then recommend thinking of AI “as a technology… keeps you in charge of the relationship” (2026-05-10). This may be a deliberate difference of audience; it is still a conceptual gap.

5. What is distinctively human. His defences rest heavily on embodiment, lived experience and “soul” (2026-05-10; 2026-07-19; 2026-05-21 postscript). Elsewhere he calls intelligence a “term of convenience” and human exceptionalism “an evolutionary illusion” (FFTF pp.104, 171). He endorsed computational functionalism (2023-08-23). And his fiction ends in “interspecific intellectual craft” (2025-11-26). The embodiment claim does real work but is asserted more than argued.

6. Novelty versus precedent. He calls AI “the first technology of it’s kind” to slip into the mind (2026-05-10). His own record includes neurotechnology (2016), engines of persuasion and social-media nudging (2018), and human-to-human parasocial influence (2025). What seems new is the combination of fluency, scale, intimacy and agency, not the mechanism. He does not spell this out. That is my inference.

7. Persuasion he practises. He defends parasocial, relationship-based expert communication as a way to empower people at scale (2025-05-25), and stories as ways to open minds. He applies “who decides?” to machine and state persuasion. He does not apply it to expert persuasion, including his own.

8. Thin evidence, by his own admission. Much of the thread rests on thought experiments, self-experiments with an n of one, and a small literature. He says so (“admittedly limited analysis”, 2026-01-10). He notes that research does not show a general causal link between offloading and reduced critical thinking (2026-01-10 n.7), and he repeatedly calls for research. His claims are hypotheses with a risk-analytic rationale, not findings.

9. Mechanisms for response are underspecified. He rejects or doubts many remedies: warnings, bans, literacy classes, guardrails alone, and regulation alone. What remains is: - conversation and trust-building; - a duty of care; - safety-first rules of thumb; - calibrated trust-cues; - collective vigilance; - capacity-building and play.

Who funds, mandates or evaluates these is mostly left open, apart from universities as a hoped-for actor that he himself calls “followers and users” so far (2026-08-30 do-universities-have-a-place-in-bill).

10. Children and adolescents are under-treated. Given the cases he cites, there is little sustained analysis of development, schools or age-specific governance. The 2026 risk list names the gap but does not fill it.

11. “Who decides?” applied unevenly. He asks it of Choice Engines and voice persuasion. On Anthropic’s constitution, which sets a model’s moral character, he does not (2026-01-22 think-you-know-ai-think-again). Only the ironic “Soul Update” story raises it indirectly (2026-02-11).

8. The most important sources for this thread#

1. FFTF Ch. 8, Ex Machina: AI and the Art of Manipulation, pp.153–178 (reposted 2023-04-16 ai-and-the-art-of-manipulation). The foundation: artificial manipulation, Plato’s Cave, the human club, “tests… when we are being played”.
2. FFTF Chs. 4, 5 and 7, pp.63–152. Engines of persuasion; the obsession with intelligence and the brain-is-not-a-computer point; malleable identity and being “someone else’s puppet”.
3. 2023-04-05 can-chatgpt-adversely-impact-mental (his half only). Language as the medium of relationship; the illusion of reciprocity.
4. 2023-04-26 in-bill-joys-why-the-future-doesnt. The language turn: self-replicating beliefs; slipping “under the checks and balances” of reason.
5. 2023-08-14 chatgpt-stimulates-creativity-critical-thinking. The optimistic benchmark on AI and thinking.
6. 2024-01-01 the-future-of-being-human-in-2024. Intrinsic technologies; language as the base code of identity.
7. 2024-05-15 anthropomorphizing-gpt-4o, with 2024-09-01 is-chatgpts-new-voice-mode-dangerously-persuasive. Hyper-anthropomorphism; relational and benevolent persuasion; “who decides”.
8. 2024-07-13 ai-choice-engines-sunstein. Dual use and the economic gradient toward manipulation.
9. 2024-10-20 learning-to-live-with-agental-social-ai, with 2024-10-27 personal-ai-chatbots-and-stochastic-agency. Agentic social AI, compressed social formation, stochastic agency, a conditional pause.
10. 2025-03-30 reimagining-education-in-an-age-of-ai. What we do and who we are; learning to be human; the crisis of education’s purpose.
11. 2025-08-31 holding-on-to-our-humanity-age-of-ai, with 2025-11-09 universities-chatgpt-mental-health. Universal vulnerability; emergent versus designed manipulation; duty of care; the limits of warnings.
12. 2026-01-10 is-ai-a-cognitive-trojan-horse. The fullest theory: epistemic vigilance, fluency, attractiveness, volume, the Intelligent User Trap.
13. 2026-02-22 what-we-miss-when-we-talk-about-ai-harnesses, with 2026-05-21 magnifica-humanitas-and-being-human. Bidirectional change; constitutive resonance; “who we are”; not “just a tool”.
14. 2026-05-10 do-not-do-this-with-ai. Risk communication for everyday cognitive risk; the limits of literacy.
15. 2026-09-24 being-an-academic-in-an-age-of-ai (Claude-drafted from his lecture). Language as formative and as a lever; cognitive surrender; universities’ role.