B16 digest: 2024-06-25 to 2024-08-18#

What this batch is#

Fifteen posts from summer 2024, unified by the frame of advanced technology transitions. Two posts come out of his preparation for an IEEE keynote in Bali, “A New Science of Navigating Advanced Technology Transitions”:

• a 2022–24 concept note for a university school;
• a new four-quadrant model built on Pippard’s ladder.

Other material in the batch:

• A cluster on AI and the human mind: Anil Seth on consciousness, Sunstein on AI “Choice Engines”, and a published Jurimetrics review on Dune.
• Four posts on Elon Musk’s ventures, plus a post on humanoid robots in which Optimus features.
• One guest post (Sean Dudley on GLP-1 drugs), which is not evidence of his views.
• No Modem Futura material.

Main ideas#

1. Manipulation is structural: the “economic gradient”. The Sunstein post is the most analytically original piece in the batch. His core claim is dual use: “the capabilities that make socially beneficial AI Choice Engines viable are the same as those that make AI-driven persuasion and manipulation possible.”

He maps value to the individual against value to the deploying “agent” in four quadrants: Self-Determination, Empowerment, Manipulation, Transformation. He argues that in an unconstrained environment power pulls deployments toward Manipulation, including by governments as well as for-profit firms. Individuals end up as “engines of value creation rather than the primary recipients”. His precedent is data monetisation.

This turns his 2018 Ex Machina thesis, which was about AI exploiting individual cognitive vulnerabilities, into a political-economy account that also covers paternalism by well-meaning institutions. The remedy is guardrails, policy, and “checks and balances”. He also uses his risk-as-threat-to-value language (“risk threatening value in unexpected ways”).

2. Conscious-seeming AI and the limits of rational defence. With Seth, he holds that real machine consciousness is probably unreachable on digital substrates. He adds his own long-held “thermodynamics argument” against self-improving superintelligence, and suggests this also calls into question whether AGI is “feasible — or even advisable.”

The pressing problem is AI that seems conscious: “we may rationally understand that an AI is not conscious, but be instinctively incapable of acting on this knowledge.” This opens people to exploitation, as he argued in Films from the Future. He criticises OpenAI and Anthropic for racing toward human-like AI “with very little governance overseeing the subtler potential social implications”. He also criticises the “hubristic” culture of treating the future as a place to “go fast and break things”.

The humanoid-robots post turns the same point around: an uncanny machine provokes an instinctive rejection that reason cannot overcome.

3. Decision-making relinquished to machines, and what makes us “us”. The Dune review is his most formal statement on being human in the batch:

• AI is “not in any way self-aware”, but may simulate intelligence until consciousness becomes “moot”.
• The deep risk is decision-making moving from people to machines, and “we are already irreversibly integrating AI into every aspect of our lives.”
• He answers the “selling our souls” question with measured optimism about human resilience, while insisting the question must stay open.
• He is deliberately neutral between accepting AI’s benefits and slowing “the AI juggernaut”, and aims to widen imagination and conversation.

4. Acceptance, perception and societal barriers. On humanoid robots he argues that “the biggest barriers are likely to be societal rather than technological”:

• acceptable safety is “highly subjective”;
• edge cases set the bar;
• the bar in the home is “stratospheric”;
• a Luddite-like backlash that “actually succeeds” is plausible.

He distinguishes the “complex” challenge of acceptance from the “merely complicated” engineering. He faults the companies for not seeing this. This is his risk-innovation logic, applied without the vocabulary. The Fridman post adds a privacy concern he says no one is raising: Teslas and Optimus robots as sources of training data.

5. Advanced technology transitions as a field and a stance. The school concept note makes a strong claim that the present is new: the challenges are “night and day different” from the past and “not navigable through conventional thinking”. It calls for a transdisciplinary “new science” that includes responsible innovation, public interest technology and “just and equitable” transitions, with scholars acting as “pilot” to stakeholders.

The Pippard’s-ladder post supplies the model. His Future Rising excerpt stresses abrupt, irreversible, hard-to-predict tipping points and the need to “spot early warnings”. The new quadrant crosses degrees of freedom (restrictive or open) with mindset (preserve or embrace change):

• Avoid is effectively the precautionary stance: risk-averse, weighing social factors, slowing innovation under uncertainty.
• Adapt: build resilience, for example against generative AI’s effects on learning and social cohesion.
• Extend: push boundaries outward through space, fusion, AI and enhancement.
• Embrace: go through tipping points deliberately.

A key assumption he states: innovation cannot be switched off, only managed and channelled.

6. Tech leaders: influence as the reason to engage. Across the Neuralink, Fridman and Ad Astra posts he treats Musk as consequential because of his influence: “not how wild the ideas are, but how much influence they have”. His tone is notably warm. He admires Neuralink’s disruptive “digital technologies mindset” and Musk’s “synergistic tech integration”, and says “I get where he’s coming from.”

He also flags naive neuroscience (“data transfer” is not “understanding transfer”) and doubts Musk’s grasp of governance. He says deregulatory “hardening of the arteries” rhetoric “isn’t wholly wrong”, but wants “an informed counterbalance”. Ethical concerns are mostly acknowledged and set aside (“whether this is something to be excited or concerned about is another question entirely”).

7. Agency, education and equity. The UBI review is careful about evidence (wary of advocacy spin) and anti-paternalist: people act well “given the chance”. It links cash transfers to technology’s disruption of livelihoods. The movie-class post attacks the “efficiency model” of teaching.

New or changed compared with earlier work#

• Embrace quadrant. He says his “original thinking” used Pippard’s ladder only as a warning about tipping points to avoid. He now entertains deliberately embracing discontinuities, while admitting he does not know what that would mean for “who thrives and who does not”. This is the clearest signalled shift in the batch.
• Manipulation moves from psychology to political economy. The economic-gradient framing, and his suspicion of governments as well as firms, are new developments.
• Humanoid robots. He reports moving from laughing at the idea to taking the bet seriously, but still doubts its social viability.
• Warmth toward Musk. His warmth toward a rule-bending innovator is higher than his earlier critique of permissionless innovation (see B08) might predict. His concerns are stated but thin.
• Continuity. He calls Films from the Future “more relevant than ever”, and its Ex Machina manipulation concern “even more relevant now”. Advanced technology transitions (in use since April 2023) is now set out as a field.

Most important posts in this batch#

1. 2024-07-13 ai-choice-engines-sunstein: the economic gradient toward manipulation; the dual-use influence quadrant.
2. 2024-08-18 four-ways-of-thinking-about-advanced-technology-transitions: Avoid/Adapt/Extend/Embrace; the signalled shift.
3. 2024-06-30 seth-is-conscious-ai-possible: conscious-seeming AI, lab hubris, the governance gap.
4. 2024-07-21 artificial-intelligence-dune-villeneuve: relinquishing decisions to machines; irreversible integration; being human.
5. 2024-08-07 are-humanoid-robots-really-the-future: perceived and subjective safety; societal barriers; complex versus complicated.
6. 2024-08-11 school-of-advanced-technology-transitions: the novelty claim and the institutional program.