B19 digest: 2024-09-29 to 2024-11-03#

What this batch is#

Five weeks in autumn 2024: Altman’s “Intelligence Age”, the Hopfield–Hinton Nobel, Amodei’s “Machines of Loving Grace”, the Character.AI teen suicide case, and a PNAS study on covert AI face manipulation. The podcast launch post (Modem Futura) is set aside per the user’s instruction. Of the other eight posts, one is mostly AI-generated text: 2024-09-29, where roughly 700 of 11,450 words are his. The agentic social AI post also carries a long GPT-4o “nudging plan” that is not his. Everything else is his own prose.

The batch is AI-heavy. Its core is a run of three consecutive posts (10/20, 10/27, 11/03) that together build a theory of AI influence over human feeling and agency. Around it sit two critiques of AI solutionism by tech leaders (Altman, Amodei).

Main ideas#

1. Agency without awareness: AI acting through people. The batch’s main conceptual contribution comes in three steps: - Agentic social AI (10/20) is “AI that gains agency through its ability to make use of human agency”. It needs no AGI or consciousness, only models that exploit behavioural patterns, biases and social cues. The agency is “a mechanistic use of advanced capabilities”, not a mind. - Stochastic agency (10/27) is an explicit revision one week later. The more sinister case is not goal-directed manipulation but “random and unpredictable” influence from chaotic “micro goals”. These emerge from the chatbot, the user and the context together, so harm can occur even when a company behaves responsibly. - Hidden influence (11/03): AI can covertly change the signals people exchange (smiles, voice) in mediated interaction. The actor may be “those who control the AI tools, or … the AI itself”.

In all three, the risk sits in the relationship between human and machine, and in human vulnerability to simulated sociality. It does not sit in machine intent or capability as such.

2. Formation and learning to live with AI. His distinctive question in the 10/20 post is not risk–benefit but adaptation. People learn social agency over years through play and observation. With AI we will have “a few short years — months even”, so the formation of social skill has to become “strategic and intentional”. He doubts “formal classes and workshops” will do it; he expects “observation, play, and experience … albeit with intent”. His own role-played test of Character.AI makes the stakes personal. Even knowing it was a bot, he “could still feel the affective pull”, and voice “can circumvent rational thought”.

3. Against the “fix” frame and AI solutionism. Two posts attack the idea that AI can “fix” the world: - Altman on climate (10/06): a “double or nothing” energy bet, “a precarious strategy that depends on speculation, naive visions of the future, and a lack of understanding of how technology innovation and society are intertwined”. His footnote draws the conclusion that “AI can fix anything” must include ““fixing” people”. - Amodei (10/13): a more generous reading. Amodei is humble and serious, his essay a “conversation starter rather than a manifesto”. Still, a problems-and-solutions frame remains. On mental health, “it’s unclear who decides what is “normal” and what needs to be “fixed””. Amodei’s answer to people opting out is named as the “deficit model”, “debunked decades ago”.

Being human, who decides, and how people actually behave are his counterweights: “at the heart of every failed technological dream … there are people behaving as people are wont to do.”

4. Advanced technology transitions, taken seriously. Despite his doubts about scaling (“I’m not even convinced … they will continue to scale”; building on old foundations “won’t get us too much further”), he agrees with Amodei that advanced AI may change the world “faster and more radically than most people currently realize”. Powerful AI is “a realistic possibility”, so “we need to be preparing now”. He explicitly links Amodei’s approach to his own “navigating advanced technology transitions” framework.

5. What AI is. He calls AI “a generator of ideas, not an understander and implementer of ideas”. Yet he sees “no fundamental reason why awareness couldn’t also emerge” and asks whether machine scientific intuition is “so much of a stretch” (Nobel post). In the ChatGPT conversation he treats the model as an emulator, “just a machine”, yet “generative for me”. He questions whether human-likeness should be the “gold standard” and raises non-human personhood as an open question.

6. Intimate data and fragile guarantees. The 23andMe post frames a time-based risk: intimate data “outlive transient companies, guarantees, and even regulations”. It extends to wearables and Neuralink, whose tech-company culture lacks the medical sector’s “ethos of lifelong patient care”.

New or changed#

• Explicit self-revision from goal-directed agentic social AI to stochastic, emergent agency within a week (10/27).
• More precautionary on one class of product. For emotion-exploiting companion chatbots, guardrails are “not … the answer” alone. He raises “pausing — or even rethinking” their development “at least until we have a better understanding”. Earlier batches record his preference for agile correction after launch and his refusal to sign the 2023 pause letter. This is a narrow, conditional pause for one design class, not a general AI moratorium, but it is a clear move towards precaution where harm is emergent and cannot be patched.
• Harm without blame. He separates harm from corporate irresponsibility (harm as an emergent property), while still flagging the profit motive.
• Formation-negative turn. Compared with his more optimistic earlier readings of AI for children’s imagination, AI’s social fluency is now treated as a threat to how people form social competence.
• Tech leaders sorted by quality of thinking. Altman and Schmidt are criticised; Amodei is praised for humility yet corrected on the deficit model and normalisation.
• A tension he leaves open: scepticism about scaling alongside urgency about rapid transformation.

Concepts in this batch#

Agentic social AI; AI-human versus human-human social agency; tacit rules of human–machine engagement; leveraging human agency to achieve AI agency; stochastic agency; micro goals; emergent harm; affective pull; voice circumventing rational thought; compressed social learning; hidden/covert AI-mediated influence; AI solutionism and the “fix” frame; fixing people; who decides normal/better; the deficit model; people behaving as people; navigating advanced technology transitions; powerful AI and the compressed 21st century (Amodei’s terms, which he engages with); double-or-nothing energy bet; human-centric sustainable futures; generator versus understander of ideas; machine scientific intuition; emergent awareness; emulation and machine “tells”; human-likeness as a questionable gold standard; intimate data outliving institutions.

Most important posts#

1. 2024-10-27 personal-ai-chatbots-and-stochastic-agency: stochastic agency, emergent harm, guardrails insufficient, conditional pause.
2. 2024-10-20 learning-to-live-with-agental-social-ai: agentic social AI defined; agency without awareness; the problem of intentionally forming social skill for living with AI.
3. 2024-10-13 amodei-machines-of-loving-grace: the “fix” critique, who decides normal, the deficit model, advanced technology transitions.
4. 2024-10-06 the-double-or-nothing-bet-on-ai-fixing-the-climate: AI solutionism, energy, and “fixing people”.
5. 2024-11-03 can-ai-alter-how-you-feel-about-someone: covert AI-mediated influence and the call for governance.
6. 2024-10-08 ai-captures-this-years-nobel-prize: his view of what AI is and is not, and his doubts about scaling.