B15 notes: 2024-04-16 to 2024-06-23 (16 posts)#

Reading notes on Andrew Maynard’s Substack posts in batch B15. Only his own prose is treated as evidence. Quotes are exact, including his original typos and curly punctuation; asterisks mark his italics.

Batch context: spring and early summer 2024. ASU’s ChatGPT Enterprise partnership with OpenAI had started in January 2024. Oxford’s Future of Humanity Institute closed on 16 April. Google DeepMind’s 274-page The Ethics of Advanced AI Assistants appeared the same month. OpenAI demonstrated GPT-4o on 13 May, and the Scarlett Johansson “Sky” voice row followed a week later. Ilya Sutskever announced Safe Superintelligence Inc. on 19 June. Maynard was travelling for much of June (a holiday in Norway, then the WEF Annual Meeting of the New Champions in Dalian). There are no Modem Futura podcast posts in this batch. The one recorded conversation (Michael Levin) belongs to the earlier “Future of Being Human … Unplugged” livestream series, and only its introduction is in the text.

Relevance summary:

HIGH#

2024-06-20 — ilya-sutskevers-safe-superintelligence-rethink — “Ilya Sutskever’s Safe Superintelligence initiative might need a rethink”#

Provenance. Entirely his own prose. No AI or quoted text apart from a few words from SSI’s announcement (“to be solved through revolutionary engineering and scientific breakthroughs”).

Argument in his terms. - He speaks as a risk professional. He opens with his credentials and his irritation: “Having worked in risk and safety in one form or another for most of my professional career”. He is frustrated by people “who assume a mantle of expertise on safety but have no idea of what they’re talking about”. The target is the framing of “safe superintelligence”, not Sutskever’s ability. He calls Sutskever “a brilliant computer scientist”, calls the goals “laudable”, and says the venture is one “which I believe is trying hard to do good here”. - Core claim: there is no absolute safety. Unless “safe” is defined very narrowly, “achieving safety will always be a social and political endeavor as well as an engineering challenge”. SSI treats safety as a technical problem, so “failure is baked into it from the get-go” unless it changes course. - The engineering fallacy. He guesses that the founders have physical structures like bridges in mind. But even a bridge’s safety rests on social choices: over what timespan, against what circumstances, “reasonably safe” or invulnerable, and which harms count (collapse, injury, environmental impact, system-wide infrastructure effects). Acceptable safety is “ultimately decided by societal norms and expectations and their reflection in standards and policy”. - A ladder of complexity. Complexity rises from physical infrastructure, to “harmful chemicals and biological substances”, to “powerful emerging technologies where potential adverse consequences are largely unknown”, and finally to superintelligence. At the chemicals rung the questions multiply: harmful outcomes, mechanisms of action, dose-response, acute versus chronic effects, and the role of perception and behaviour in mediating consequences. For superintelligence, agreeing what is “acceptable” is “fiendishly more complex”. - Safety and harm defined. Safety means being protected from, or free from, the possibility of harm. Harm is plural: - For individuals: injury, death, loss of physical and mental health, loss of access to essential services, and loss of “dignity, autonomy, self-respect, ability to flourish”. - For societies: “pathological collective behaviors”, loss of resources, infrastructure collapse, disease, “spread of harmful ideas”, conflict and war. - For ecosystems and the planet: harms that may matter in their own right.

What counts as harm “is ultimately a social construct, not a technological one”. This is where SSI “begin[s] to fall apart”: it does not engage with “who decides what “safe” means”. - Risk as the operational form of safety. “Risk — or the probability of harm occurring — is inherent in any system that is subject to time and change”, and “zero risk — the corollary of absolute safety, is only possible in the absence of change”. That is why safety is operationalised as assessing and managing risk. Risk framing allows bounds of acceptable risk. Society’s rule of thumb is a one-in-a-million risk, which still implies that someone somewhere is harmed. Rules of thumb vary with culture and context, and individual and community risk judgements “often seem to defy logical analysis — until you approach them through the lens of human behavior”. His summary: “risk can be seen as the operationalization of safety, it’s never zero, acceptable risk is ultimately governed by what people agree on”. - Power and ignorance raise risk. “The more powerful and poorly understood a technology is within the social, political, and environmental systems it’s a part of, the greater the chances are of it causing harm”. So even acceptably safe superintelligence is unlikely. - Prescription. SSI should aim for acceptable and societally beneficial superintelligence. It should recognise that this cannot be achieved through science and engineering alone. It should employ “people who deeply understand risk and safety from the context of navigating advanced technology transitions from a societal perspective”. This is a call for his own kind of expertise, and he does not flag the self-interest. He says this is not a call to “muddy the purity of the technological waters”. Final line: the biggest threat is “the blinkered assumption that absolutely safe technologies are possible through science and technology alone”.

How firmly. Very firmly on the concept: it is a lecture from the field. He is charitable about intentions. He does not dispute here whether superintelligence is plausible and takes the premise for the sake of argument. That contrasts with the FHI post eight weeks earlier (below).

Concepts and frameworks. No absolute safety. Acceptable safety and acceptable risk. Safety as social construct. Risk as the probability of harm and as the operationalisation of safety. Zero risk only in the absence of change (a physicist’s framing: time and change). The one-in-a-million rule of thumb. A plural, value-laden taxonomy of harm (individual, societal, ecological) that matches his long-standing “risk as threat to value”, though he does not name it here. Risk perception explained through behaviour. Advanced technology transitions. Societally beneficial AI.

Analogies and comparisons. Bridges and buildings (structural). Harmful chemicals and biological agents (hazard, mechanism, dose-response, acute and chronic effects). These are used structurally, as rungs on a ladder of complexity, to show that even well-understood safety domains are socially constituted. They are not a template to copy onto AI.

On AI and AI risk. Superintelligence is treated as a technology whose harms are broad, contested and systemic. The risks of powerful, poorly understood systems include social harms like “spread of harmful ideas” and “pathological collective behaviors”, not only catastrophic failure.

On companies and leaders. He criticises the naivety of engineering-minded founders who treat safety as a solvable technical problem. He is not hostile to them personally.

Governance and who decides. Safety is decided socially and politically, through norms, standards and policy. The key unasked question is who decides what “safe” means.

Quotes. - “The problem is that there is no such thing as absolute safety.” - “what is considered as harm — and by inference, what is a safety issue — is ultimately a social construct, not a technological one” - “zero risk — the corollary of absolute safety, is only possible in the absence of change” - “the biggest threat to building acceptably safe technologies is the blinkered assumption that absolutely safe technologies are possible through science and technology alone”

2024-05-15 — anthropomorphizing-gpt-4o — “OpenAI’s GPT-4o and the challenges of hyper-anthropomorphism”#

Provenance. A mix of his prose and quoted text. Five long block quotes (privacy concerns, manipulation and coercion, overreliance, violated expectations, false notions of responsibility) and one shorter quote are the words of the DeepMind Ethics of Advanced AI Assistants authors, not his. His own contributions are the framing, the coinage “hyper-anthropomorphism”, the judgement that these risks are “highly pertinent” to GPT-4o, and the update on the flirting clip.

Argument in his terms. - GPT-4o is not a big step in capability but a big step in feel. It “doesn’t represent a huge leap in AI capabilities”, but “where it lacks in new LLM capabilities it more than makes up in features that make it feel more human”. It sees and responds to live video, converses with almost no lag, “reads” emotion from face and voice, and “connect[s] emotionally to users”, even joking and flirting. - Anthropomorphism defined by him. “treating it as a human when it is, in reality, simply a machine that is exceptionally good at emulating human behaviors”. Here his view of AI’s nature is the emulation view. - Emotional trust is the mechanism. The emotional register of voice builds “connections that are relational rather than transactional — that speak to our heart rather than our head”. People are therefore “more likely to believe, trust, build emotional attachments with, and confide in” such machines. - The pace surprised him. When he read the DeepMind paper a few weeks earlier, many concerns “felt like they were still some distance away”. GPT-4o brought them closer “faster than I could have imagined”. Chapter 10 of the paper “could have been written directly about GPT-4o”. - Coinage. ““hyper-anthropomorphism” — a concerted effort to create AI’s that are intentionally designed to engage our anthropomorphizing cognitive biases”. The point is design intent: the voice interface “seems to be intentional[ly] designed to encourage and build emotional attachments”. - Conclusion, hedged on timing but firm on direction. It is “too early to tell” whether we will develop healthy ways of living with such machines, or whether this is “a seductive but slippery AI-risks slope”. Either way we must “start thinking fast” about assistants “designed to make us fall a little in love with them — and possibly give away more of ourselves to them and their creators”. - Update. An unprompted comment in the livestream (“Wow, that’s quite the outfit you’ve got on. Love …”) leads him to: “Unprompted flirting from an AI designed to build emotional connections? That’s worrying.”

Concepts. Hyper-anthropomorphism (his coinage). Emotional trust. Relational versus transactional connection. Anthropomorphising cognitive biases. AI as emulator of human behaviour. The endorsed risk list: privacy, manipulation and coercion, overreliance, violated expectations, false responsibility.

Analogies. Spike Jonze’s Her, noted as the obvious public comparison and developed in the next posts.

On AI risk. The risk sits in the human–machine relationship: exploitation of cognitive biases, dependence, erosion of autonomy, and the transfer of personal data “to them and their creators”. This continues his long-running concern with AI exploiting cognitive vulnerabilities (the 2018 Ex Machina chapter, B08). The shift is that the danger now comes from deliberate product design by a company, not from an autonomous AI.

On companies. OpenAI is implicitly criticised for designing for attachment, with the voice “intentional[ly]” built for it.

Cognition and formation. This is central. Voice, affect and emotional bonding bypass the head for the heart. What is at stake is how much of ourselves we give away.

Change signalled. He is surprised by the speed. He also partly revises his earlier reservation. On 24 April he thought the paper under-nuanced on anthropomorphism; now he finds its anthropomorphism chapter prescient.

Quotes. - ““hyper-anthropomorphism” — a concerted effort to create AI’s that are intentionally designed to engage our anthropomorphizing cognitive biases” - “connections that are relational rather than transactional — that speak to our heart rather than our head” - “designed to make us fall a little in love with them — and possibly give away more of ourselves to them and their creators” - “Unprompted flirting from an AI designed to build emotional connections? That’s worrying.”

2024-05-21 — openais-problem-with-the-movie-her — “OpenAI’s problem with the movie Her and Scarlett Johansson”#

Provenance. His own prose, plus one quoted Sam Altman remark (from a 2023 Benioff conversation) and a one-word Altman post (“her”). Neither is his.

Argument in his terms. - On Johansson’s account, Altman “had arrogantly plowed ahead despite Johansson having said “no” to multiple requests”. Maynard hopes she wins any lawsuit, though he expects legal complications because the emulation was of a film character, not of her personally. - OpenAI wanted to recreate Samantha, including her capacity for an emotional and “ultimately a romantic” relationship with users. “In their naivety they seem to have conflated” the character and the actress. - The wider issue: “disconnects between the talk around responsible innovation, and a reality that sometimes seems childish irresponsibility”. - Four questions. 1. Can a “naive and even childish desire to recreate movie-inspired fantasies” cause harmful unintended consequences? He says this applies to every tech company trying to build what it saw on screen. 2. Is it appropriate to aspire to AI agents designed for personal and even romantic connection? 3. What are the consequences of developers who disregard consequences “when they really want to do it”? 4. When will developers learn that “disdain for the dignity and rights of individuals … is not a great business model”? - Advanced AI agents will become ubiquitous (Google and Microsoft announcements). He hopes for “true responsibility” and “not just lip service from those who are trying to recreate their sci-fi fantasies”.

Concepts. Responsible-innovation talk versus practice (lip service). Childish irresponsibility. Sci-fi fantasy as a design driver. Dignity and rights of individuals. Relational AI agents.

Analogies. Her is used as a cautionary text that the company read for its technology and not its message (developed in the 26 May post).

On companies and leaders. This is the sharpest named criticism of OpenAI and Altman in the batches reviewed so far (B01–B08): arrogant, naive, childish, disdainful of individual rights. The criticism is ethical and cultural, not technical.

Governance. Implicit: responsibility has to be real, not performed. He appeals to developers’ own responsibility and to business self-interest, not regulation.

Cognition and formation. He questions whether AI agents should be built to encourage romantic and personal bonds, continuing the GPT-4o post.

Quotes. - “disconnects between the talk around responsible innovation, and a reality that sometimes seems childish irresponsibility” - “Can a naive and even childish desire to recreate movie-inspired fantasies in AI lead to potentially harmful unintended consequences?” - “When are tech developers going to learn that disdain for the dignity and rights of individuals when they stand in your way is not a great business model?”

2024-04-28 — beyond-the-future-of-humanity-institute — “Does the world need another Future of Humanity Institute?”#

Provenance. Entirely his own prose.

Argument in his terms. - Mixed feelings. He has “long been skeptical” of FHI’s ideas and their “outsized impact”. Yet there has “never been a greater need for far-sighted and transdisciplinary thinking” about human flourishing amid transformative technologies. - The Bostrom dinner (2008), retold. They talked about nanotechnology and its “philosophical and existential threats”. Bostrom drew on Drexler’s atomically precise manufacturing and self-assembling nanobots. They “seriously parted company” over self-replicating machines that seemed to defy the second law of thermodynamics, a near-perpetual-motion idea. “To the physicist in me, claiming that perpetual motion is possible is as fanciful as believing the earth is flat.” For Bostrom, “the science of how the world works … was irrelevant compared to the philosophical elegance of the ideas”. Bostrom’s Superintelligence raised the same concern: it “seemed to fly in the face of how the universe works” and “placed a far greater emphasis on philosophical speculation than practical reality”. (The 2018 book chapter republished in B08 told the same story; this version adds the thermodynamics detail.) - Influence and ideology. Superintelligence, longtermism and effective altruism “spread through society like wildfire”. They influenced Musk, Altman and Sam Bankman-Fried: “They’ve fueled Silicon Valley’s “tech bro” culture.” He cites Émile Torres on Bostrom’s 1996 email linking IQ and skin colour. He criticises a “pragmatic rationalism” that “claims to favor science over social norms, and yet is intellectually precarious”. “Thinking that reflects a disdain for society as it strives to protect humanity” is probably part of why FHI closed; the official reason was academic politics. - On the other hand. FHI transcended disciplines. Sandberg’s final report is “rose-tinted” but shows real success. CSER, Foresight and Berggruen lack FHI’s “breadth of vision” and “audacity of thought”. - Is there a vacuum? “a strong yes, but with some rather large caveats.” He repeats his view (linking an earlier post) that “humanity is at a tipping point in its history where the past is a poor predictor of the future”. Accelerating innovation, a connected society and hard planetary boundaries put us “on a knife edge”. New thinking must break with convention but be “grounded in humility, social relevance and connectedness”. “Future-building is a collaborative effort — not something that should be left to an elite group of thinkers and innovators.” Even FHI fell short on this. - His proposal: public universities. 1. They sit at the nexus of stakeholders, with a level of accountability that private universities, corporations and government lack. They serve the public good and have academic freedom. 2. They educate the next generation of “effective and empowered builders of the future”. 3. They can support transdisciplinary, rigorous, far-sighted work, though this is “admittedly nascent in many”.

The barriers are academic reward norms: grants and papers over depth and impact. He does not name ASU, but he is describing the kind of institution he works in, and his own Future of Being Human initiative there (an inference). - Close: can we fill the vacuum in ways that are “inclusive, grounded in reality, and public-serving” while keeping the audacity to imagine what is not readily conceivable?

Concepts. Physically grounded versus philosophically speculative futures (plausible versus imaginable, in B08’s terms). Tipping point, knife edge, the past as a poor predictor. Transdisciplinary, boundary-spanning futures research. Humility. Collaborative, non-elite future-building. The public university as accountable steward of the future.

Analogies and past technologies. Nanotechnology, literally: his own field, and the terrain where he first clashed with Bostrom over Drexlerian nanobots. The second law of thermodynamics is his physicist’s test of plausibility.

On AI and existential risk. Superintelligence as FHI framed it is scientifically dubious to him. He is more troubled by the social influence of x-risk ideologies (longtermism, EA) on tech leaders than by the scenarios themselves.

On companies and leaders. Musk, Altman and SBF are named as carriers of FHI-derived ideas, with “tech bro” culture as the result.

Who decides. Not elites. Future-building should be collaborative and inclusive, anchored in accountable public institutions.

Quotes. - “To the physicist in me, claiming that perpetual motion is possible is as fanciful as believing the earth is flat.” - “placed a far greater emphasis on philosophical speculation than practical reality” - “future-building is a collaborative effort — not something that should be left to an elite group of thinkers and innovators” - “Thinking that reflects a disdain for society as it strives to protect humanity”

2024-05-05 — blackberry-or-iphone-educational-ai — “Is GenAI in education more of a Blackberry or iPhone?”#

Provenance. Entirely his own prose.

Argument in his terms. - Wariness about analogies. “I try and stay clear of analogies to describe the emergence and impact of artificial intelligence.” He uses two (BlackBerry and iPhone) and then says why he dislikes both. - The GenAI edtech gold rush. After November 2022, generative AI flipped from “a possible threat to a potential educational game changer”. He supports the moment “as long as we proceed with eyes wide open and a good dose of critical thinking”. He warns of “the risks of early and naive adoption, as well as potentially limiting tech lock-in”. - The BlackBerry scenario is heavy investment in niche, inflexible tools that quickly become obsolete and “end up limiting rather than opening up educational opportunities”. It “worries me — a lot”. He gives three reasons: - Scaling doubts: “There are growing indications that large language models may not be able to achieve as much as they initially promised”. Also, “making them larger won’t necessarily make them better”. - Superficially impressive capabilities may not become reliable tools. - Models are constantly tweaked (fine-tuning, guardrails), which destroys the constancy education needs. A tutor or autograder whose behaviour shifts week to week “isn’t ideal”. Grading is not a domain “where sometimes being right is OK”. - The iPhone scenario is tools resilient to change in the underlying model. He finds it “hard for me to imagine how this will work”, since a better next version makes you wonder “just how bad the previous version was”. His route is “a much more cautious and experimental adoption”: - less enterprise-level deployment of inflexible, and therefore fragile, tools; - more creative use of general-purpose AI; - keeping established teaching as the foundation; - more student freedom to explore; - “investing in concepts, not products”; an “iPhone mindset” that “builds experimentation and innovation into the very DNA of learning”. - Dubious analogies. Common comparisons for AI include the calculator, the internet, the printing press and the industrial revolution. They are “attempts to understand the advanced technology transition we’re experiencing in the context of what we’ve previously encountered”, and “all fail to capture the sheer uniqueness and profundity of how AI is changing our world”. Current platforms are “untrustworthy … not in the sense that they are deceitful or misleading, but we’re still getting the measure of what they can and cannot do well as they evolve”. Even the iPhone analogy misses the speed. Building robust learning environments on something that will “look like something else in a few months seems precarious”, “especially when what’s at stake is the long term success of our students”.

Concepts. Early and naive adoption. Tech lock-in. Brittle implementation versus transformational concept. “Concepts, not products”. iPhone mindset versus BlackBerry mindset. Technological constancy as a requirement for trustworthy tools. Untrustworthiness as unmeasured and unstable capability, not deceit. AI as a general-purpose technology. Advanced technology transition.

Analogies. BlackBerry and iPhone are used heuristically, as mindsets and not playbooks, and he explicitly flags them as weak. The calculator, internet, printing press and industrial revolution are all judged inadequate. This is an important meta-statement: he doubts that historical analogies capture AI’s novelty and speed. It sits in some tension with his structural use of chemical-risk concepts elsewhere, though there he transfers methods, not historical trajectories.

On AI. A fast-morphing general-purpose technology whose behaviour is unstable across versions. He expresses scepticism about scaling alongside a conviction that AI is transformative.

On AI risk (education). The risks are institutional and practical: sunk investment, lock-in, unreliable tools locked into systems, and harm to students’ long-term success.

Governance. Institutional adoption strategy: cautious experimentation, resilience, and keeping human teaching as the fallback.

Quotes. - “I try and stay clear of analogies to describe the emergence and impact of artificial intelligence.” - “all fail to capture the sheer uniqueness and profundity of how AI is changing our world” - “In other words, it’s going to mean investing in concepts, not products.” - “There are growing indications that large language models may not be able to achieve as much as they initially promised”

MEDIUM#

2024-04-24 — navigating-ethics-of-advanced-ai-assistants — “Are we ready to navigate the complex ethics of advanced AI assistants?”#

Provenance. A mix. His prose is the framing and evaluation. The definition of advanced AI assistants, the moralised/non-moralised distinction, the four assistant types, the 14 areas, the Chapter 17 education findings and the closing executive-summary quote all belong to the paper’s 57 authors (Google DeepMind and others). The reading guide is replicated in full from the paper and is not his.

Argument in his terms. - He calls the paper “one of the most comprehensive and thoughtful papers on developing transformative AI capabilities in socially responsible ways”. He says it is “essential reading” for anyone deploying assistants or agents in business, government and education. - AI assistants (LLM-based agents that plan and act for users) are heading toward ubiquity and “deep integration into almost every aspect of our lives”. He quotes the paper here, but the framing of ubiquity is his. - On the moralised/non-moralised distinction, the authors “wisely” chose the practical definition. But the lines blur around wellbeing, trust and relationships. This “underlines … the reality that we cannot simply codify AI ethics within a neat set of principles where the technology has profoundly complex and largely unknown consequences to society”. - The paper moves AI ethics from “rather limited conversations around responsible use of generative AI” to what emerges from increasingly powerful foundation models. The worry is transformations that “could go seriously wrong if we don’t learn how to navigate them”. He says we have “barely scratched the surface of how to ask the right questions”. - Education. Many companies are persuading K-12 educators to adopt assistant-based tools, and universities are building advisers, course builders and graders. “I wonder whether the frameworks being employed are becoming increasingly disconnected from the challenges that advanced AI assistants present.” - Dissent. “I think there’s more nuance to questions around anthromorphism and AI than they indicate.” He does not develop this; three weeks later GPT-4o makes him find the chapter prescient. - He closes by praising the paper for moving us beyond “the potential short sightedness that’s set in around current generations of generative AI”. He looks toward AI systems “that engage with us in very human ways, and have agency to change our lives — and even ourselves”. He gives a “strong yes” to the authors’ claim that the path depends on choices by researchers, developers, policymakers and “members of the public”.

Concepts. Advanced AI assistants and agents (the paper’s definition). Moralised versus non-moralised assistants (the paper’s, engaged by him). The limits of principles-based AI ethics. The frame gap between ethics frameworks in education and advanced assistants. AI agency over the self.

On AI. Assistants and agents are the next phase after chat-based generative AI, with human-like language interfaces and real agency.

On AI risk. A broad, pluralistic landscape (the 14 areas). Complex balances, not neat conclusions. Serious risks include value imposition, hindered self-actualisation, and emotional and material dependence (listed from the paper).

On companies. He praises an industry-academic (DeepMind-led) ethics effort, and does not treat industry ethics work as inherently suspect.

Governance. Broadly shared choice (“members of the public” included), plus research. Principles alone are insufficient.

Cognition and formation. AI that can “change … even ourselves” is the formation concern in embryo.

Quotes. - “we cannot simply codify AI ethics within a neat set of principles where the technology has profoundly complex and largely unknown consequences to society” - “I wonder whether the frameworks being employed are becoming increasingly disconnected from the challenges that advanced AI assistants present.” - “build AI systems that engage with us in very human ways, and have agency to change our lives — and even ourselves.”

2024-04-21 — can-ai-be-used-to-automate-social — “Can AI be used to automate social science research?”#

Provenance. His own prose, commenting on Manning, Zhu and Horton’s arXiv paper “Automated Social Science: Language Models as Scientist and Subjects”.

Argument in his terms. - The premise is that LLMs, as an “unprecedented corpus” of human behaviour, can simulate both subjects and scientists. “Crazy as it sounds, this makes a lot of sense.” It is also “highly controversial”. It suggests “machines are not only capable of learning about humans faster and more effectively than we’re capable of learning about ourselves”, and could use that knowledge in far-reaching ways. - Assessment. The results are unsurprising given aggregated data and simple scenarios, but “the implications are startling”. Simulated subjects could get round IRB burdens, sample-size limits, bias and noise, if simulations support valid causal inference. Automating the scientists will make researchers “uncomfortable”. - Caveat. “the AI’s in the paper have no understanding of what a relevant question or a useful outcome is”. Humans still set relevance. Yet “machines could generate knowledge about ourselves that is inaccessible to human researchers alone”. This parallels AI-driven discoveries in proteins, materials, drugs and a new class of antibiotics. - The threat. The idea that human behaviour is predictable from machine-generated models challenges those who believe humanity is not reducible “to numbers and equations”. There are also concerns about bias and about particular worldviews being amplified. - His note of caution, the real point. We are giving foundation models vast knowledge of human behaviour and building systems that can research and act on it. “We’re getting closer to machines that can study and understand how people behave and that can, in principle, use this to influence our behavior to achieve specific goals.” The paper is “a long, long way” from this, since it covers only small-group interactions. But scaling to society-wide simulation may be easier for AI. The result could be “a new era of machine-enabled research on predicting and nudging human behavior”. “Depending on your point of view, that could be highly liberating, or deeply chilling …”

Concepts. Simulated subjects and simulated scientists. Machine-generated self-knowledge beyond human reach. Prediction and nudging at societal scale. Reductionism versus human uniqueness.

On AI. A system that could come to know humans better than humans know themselves. It lacks understanding of relevance, but can generate hypotheses and run experiments.

On AI risk. This updates the manipulation thread (B08) through a research route. Knowledge of human behaviour becomes an instrument of influence. He leaves the verdict open (“liberating, or deeply chilling”).

Cognition and formation. It bears on human self-understanding and on who holds knowledge about us.

Quotes. - “machines are not only capable of learning about humans faster and more effectively than we’re capable of learning about ourselves” - “we’re getting closer to machines that can study and understand how people behave and that can, in principle, use this to influence our behavior to achieve specific goals.” - “Depending on your point of view, that could be highly liberating, or deeply chilling”

2024-05-12 — chatgpt-shaming-is-a-thing — “ChatGPT shaming is a thing – and it shouldn’t be”#

Provenance. His own prose, including the seven guidelines. The postscript lists Ethan Mollick’s four “rules for co-intelligence” from Co-Intelligence, which are Mollick’s, not his.

Argument in his terms. - A colleague who used ChatGPT Enterprise to refine a co-authored abstract was “all but accused of cheating”. He describes a growing tension between adopters and people who think AI use is “slightly dirty or underhand”. - Legitimate use. He defines it as using AI “as a professional tool to support someone’s expertise, not as a substitute for it”: secure data, refining rather than generating original content, and human assessment of the output. - AI is already embedded (autocorrect, Grammarly, Outlook, LinkedIn, Zoom, Chrome). This doesn’t mean “we should blindly accept them”, but it exposes a disconnect. - The chemicals analogy. Telling heavy users that AI is cheating is “a bit like telling a chemist we should all be chemicals-free”. This is a conceptual analogy drawn from his chemical-risk background: the object of suspicion is already everywhere. - Diagnosis. “a lack of understanding around emerging AI capabilities, combined with deep-seated fears that the technology is challenging tightly held notions of how the world should be”. Echo-chamber rhetoric on social media makes it worse. The same thing spills into education, where unclear expectations lead to students being shamed or penalised. - Balance. “there are many challenges to how AI is being developed and used that need to be articulated and addressed, and not blithely swept aside”. But sorting colleagues into a “bad behavior” bucket does not help. - Seven guidelines (his). 1. Acknowledge a “scale of comfort”, on which views are “not necessarily right or wrong”. 2. Agree boundaries at the start of a collaboration. 3. Don’t be judgemental. 4. Become familiar with professional uses and limits. 5. Avoid decisions based on assumption and hearsay. 6. Collaborate with humility. 7. Don’t shame people for using or not using AI.

The way forward is “experimenting creatively, questioning without judgement, and collaborating with humility”.

Concepts. ChatGPT shaming. The scale of comfort. AI as a tool supporting expertise rather than substituting for it. Ground rules up front, by analogy with authorship-order norms.

Analogies. Chemicals-free, used conceptually. Academic authorship disputes, a colleague’s analogy that he endorses.

Expertise and publics. He reads resistance partly as ignorance and fear, while granting that the underlying challenges are real. Read against his own risk-as-threat-to-value framing, the post treats value-laden objections (“how the world should be”) somewhat dismissively in this context. That is an observation, not his statement.

Quotes. - “it’s a bit like telling a chemist we should all be chemicals-free” - “a lack of understanding around emerging AI capabilities, combined with deep-seated fears that the technology is challenging tightly held notions of how the world should be” - “experimenting creatively, questioning without judgement, and collaborating with humility”

2024-04-16 — asu-students-flex-their-creative-gpt-muscles — “ASU students flex their creative muscles with ChatGPT Enterprise”#

Provenance. His own prose. It reports a student-run prompt hackathon, sponsored by DreamBuilder at ASU’s Thunderbird School, in which he was a judge and is faculty sponsor of ASU’s Prompt Engineering Club.

Argument. Most of the post is an event report on the winning custom GPTs: - DreamGrant, for funding guidance; - Customer Journey Storyboard, which turns modules into graphic novels for non-text learners; - AlmaAI, a personal tutor; - ScamScanner, which he thinks especially important for protecting participants from increasingly sophisticated scammers.

His evaluative claims are that the students, including non-coders, came up to speed quickly. They “underlined the transformative potential of providing broad access to these tools rather than limiting it in universities”. “Creating a virtual playground where enterprising students are let loose on a powerful general purpose technology like ChatGPT” produces outcomes “that far exceed expectations”. Hence: “we should be giving our students a lot more free reign on how they experiment with and “hack” AI”.

Significance. This is his pro-access, pro-experimentation stance on AI in higher education, stated without caveats. It is consistent with the 5 May call for student freedom to experiment with general-purpose AI. It is also notable for its uncritical framing of ASU’s OpenAI partnership five weeks before his sharp criticism of OpenAI’s conduct (21 May). He holds the two positions side by side, as a user and partner of the tools and a critic of the company’s culture. The “let loose” playground language also sits alongside his critique of permissionless innovation. His implied distinction is between bounded educational experimentation and societal deployment.

Quotes. - “they underlined the transformative potential of providing broad access to these tools rather than limiting it in universities” - “we should be giving our students a lot more free reign on how they experiment with and “hack” AI”

2024-05-26 — should-tech-entrepreneurs-be-banned-from-scifi — “Should tech entrepreneurs be banned from watching sci-Fi movies?”#

Provenance. His own prose. It is a round-up before his June travel. The embedded course trailer is his video, and its content is not in the text.

Argument. - He has taught an undergraduate course for seven years (The Moviegoer’s Guide to the Future) that uses sci-fi films “to explore emerging technologies and their socially responsible development”. He contrasts this with tech entrepreneurs who “become so enamored with cool tech that they fail to spot the social messages it comes with”, explicitly tied to the OpenAI and Her story. - His answer to the provocative title: “my answer is no — but only if they take on the broader message” of the films. - Previews of his Ex Machina location visit and of the WEF Top 10 Emerging Technologies 2024 launch, which he says reflects AI developments “without succumbing to AI mania”. - A self-deprecating aside: enrolment on a 60,000-student campus is getting harder.

Concepts. Sci-fi as a tool for social reflection, not a tech wish-list. “Cool tech” versus social message. Avoiding AI mania.

On leaders. A light but pointed critique of the superficial sci-fi inspiration in the tech sector (Altman).

Quotes. - “they become so enamored with cool tech that they fail to spot the social messages it comes with” - “my answer is no — but only if they take on the broader message”

2024-06-16 — ai-ex-machina-and-the-juvet-landscape-hotel — “AI, Ex Machina, and the Juvet Landscape Hotel”#

Provenance. His own prose, mainly a travel essay. It also reports a conversation with the hotel owner (named in the post as “Knutt Slinning”).

Argument and interpretation. - Ex Machina is “one of the most insightful and thought provoking movies on AI”, “perhaps more relevant today than when it first hit cinemas”. It is “a story of power, manipulation, and the morally ambivalent emergence of self-aware AI”. Nathan is “vaguely Musk-like” and builds AGI from data “he’s scraping from users” of his search engine. - The Juvet’s glass walls embody a theme of “transparent barriers that both connect and separate different worlds”. These are the human and the artificial, the natural and the machine, and “a past dominated by human intelligence, and a nascent AI future”. - Plato’s Cave again: Caleb enters the “cave” of Nathan’s lair through the door of the very room Maynard stayed in, and Ava leaves through it (footnote to Films from the Future ch. 8). - Mary in the Black and White Room is “a thought experiment designed to explore the difference between intellectual and experiential understanding in the context of AI”. Ava’s final walk through the ferns moves her from abstractly knowing to experiencing, with awe. - Ava crossing the boundary “foreshadows a future where the concept of “personhood” extends beyond human exclusivity”, which “challenges how we think about the increasingly tenuous boundaries between “natural” and “artificial” in an age of AI”. - A methodological coda: “sometimes, insights come from being immersed in a place rather than just experiencing it intellectually”.

Concepts. Transparent barriers and liminal spaces. Natural versus artificial boundaries. Intellectual versus experiential understanding (Mary’s Room). Extension of personhood beyond humans. Plato’s Cave. Embodied insight.

On AI. Film-mediated and speculative: the open possibility of AI personhood and experience. This contrasts with the GPT-4o post’s “simply a machine” emulation view of current products. The two are compatible, as a statement about now and an interpretation of a possible future, but worth noting.

Quotes. - “transparent barriers that both connect and separate different worlds” - “she foreshadows a future where the concept of “personhood” extends beyond human exclusivity” - “sometimes, insights come from being immersed in a place rather than just experiencing it intellectually”

2024-06-23 — existential-risk-jay-baruchel — “A seriously funny look at existential risk with actor Jay Baruchel”#

Provenance. His own prose, promoting season 2 of We’re All Gonna Die (Even Jay Baruchel). The episodes are video and not in the text; he appears in the AI and nanotechnology episodes.

Argument. - The AI episode was recorded in April 2023 and is not out of date. It is a digestible overview “from value alignment and job losses to responsible development and use”. Skynet and HAL references “are used to good effect to frame more realistic challenges”. - Nanotech. “I’ve always been something of a skeptic when it comes to some of the wilder fears around nanotech and existential risk — including worries about “gray goo.”” In the episode they “move quickly on from nanotech risk fantasies to more grounded concerns”, including carbon nanotubes. He uses Transcendence in class to show what nanotechnology is not. His colleague Paul Westerhoff takes the conversation “from nano-fantasy to nano-reality”: engineered nanomaterials in consumer products and the environment. - Self-description. He “used to do this sort of thing a lot in the distant past, but tend[s] to focus on a broader range of emerging technologies these days”. - Putting existential risk in context. Eye-rolling at x-risk (fantasy, scaremongering, distraction): “All of these are true at times.” Yet “simply ignoring the possibility of potentially catastrophic events — especially if they are linked to powerful technologies — is in itself a risky strategy”. We need ways of grappling with “low probability but high impact risks that put them in context without brushing them under the carpet”. Those ways should “open up conversations rather than closing them down” and avoid being “overwhelmed by long tail speculation”. Humour helps.

Concepts. Low-probability, high-impact risks in context. Grounded concerns versus risk fantasies. Opening rather than closing conversations. Sci-fi tropes as frames for realistic risks.

Past technologies. Nanotechnology, literally. Gray goo is dismissed and engineered-nanomaterial exposure taken seriously. This is the same fantasy-versus-reality move he makes about AI superintelligence versus grounded AI harms.

Quotes. - “I’ve always been something of a skeptic when it comes to some of the wilder fears around nanotech and existential risk” - “simply ignoring the possibility of potentially catastrophic events — especially if they are linked to powerful technologies — is in itself a risky strategy” - “we need ways of grappling with low probability but high impact risks that put them in context without brushing them under the carpet”

LOW#

• 2024-04-18 — rethinking-biology-with-michael-levin (low): An introduction to a recorded “Future of Being Human … Unplugged” livestream conversation (video; not Modem Futura) with biologist Michael Levin, followed by boilerplate bios and a series blurb. His short intro frames bioelectric networks as “hierarchies of collective intelligence that allow biology to problem solve in quite remarkable and unexpected ways” and flags relevance to “agental systems, embedded intelligence” and AI. The conversation itself is not in the text.
• 2024-05-01 — a-student-perspective-on-the-apple-vision-pro (low): A guest post by ASU freshman Caleb Lieberman on borrowing the Future of Being Human initiative’s Apple Vision Pro. Maynard wrote only the two-sentence italic introduction, so the student’s techno-optimist views are not evidence of Maynard’s thinking.
• 2024-05-07 — supercharging-research-using-ai (low): A short, enthusiastic summary of the PCAST report Supercharging Research. The Recommendation 4 block quote on responsible AI in research is PCAST’s text. His own contribution is to praise “the sheer audacity of PCAST’s vision” and to say that, despite limitations, AI “could fundamentally change the way that we extend our knowledge”. This fits the AI-discovery theme of the social-science post.
• 2024-05-19 — future-rising-short-history-of-tomorrow (“Future Failing”) (low): A candid reflection on the poor sales of his 2020 book Future Rising (596 copies). He talks about academics’ “public responsibility” to make ideas accessible, and about books as the best medium for depth. In footnote 2 he doubts that a “market of ideas” exists “in a world of algorithmic feeds”. This is relevant to his view of himself as a public scholar, but it contains no AI or risk argument.