Late Lessons, Jensen Huang and AI
Markdown version

B01 notes: 18 posts, 2014-11-23 to 2018-08-22#

Batch scope: early material carried over to the Substack from The Conversation (most posts) and from Andrew’s older blog 2020 Science (2020science.org). All 18 posts were read in full. No Modem Futura podcast posts fall in this batch.

Date caveat, which matters for chronology. The Substack dates on several reposted Conversation pieces come earlier than their real publication dates. Internal references show when each was actually written:

The slugs below keep the Substack dates.

Provenance summary: every post except one is Andrew’s own prose. The exception is 2014-11-23 a-scientists-manifesto, which is almost entirely Robert Winston’s text, reproduced with permission. Posts that quote others (BMJ authors, Goldacre, Crow and Dabars, the Flint PhD students, Schwab, Bill Joy) are marked as such below. 2018-05-12 wraps a Risk Bites video. Risk Bites is Andrew’s own YouTube channel, and the ten-item list in the post is his. 2018-08-22 announces a co-authored commentary (with Justin Kidd), but the post’s text is his.

HIGH#

2015-01-30 — responsible-development-of-new-technologies-critical-in-complex-connected-world-1799ef680ad#

Title: Responsible development of new technologies critical in complex, connected world Provenance: His own prose (The Conversation). Summarises Stilgoe, Owen and Macnaghten’s four dimensions and the Future of Life Institute (FLI) open letter. The WEF interdependency diagram is credited to “World Economic Forum/Andrew Maynard”.

Argument. The 2012 Indian blackout, when a single overloaded line cascaded into the largest blackout in history, is his lead analogy for how complex tech systems collapse. Emerging technologies, as in the WEF Top Ten, are powerful and needed: a billion people lack sanitation, children die young, and so on. But they are not “inherently safe”, and they are becoming highly interdependent (AI ↔ neuromorphic chips ↔ drones/robotics ↔ digital genome ↔ gene editing ↔ AI). They are also coupled to planetary boundaries. The result is a “massively interconnected socio-techno-environmental system” open to “rapid, chaotic and potentially catastrophic collapse”. Studying technologies one at a time (as was done for nanotech and synthetic biology) will not prevent systemic failure, just as understanding one transmission line would not have prevented the blackout. He calls for new ideas, research and tools, with interdisciplinary investment on an unprecedented scale. Responsibility should be “built into the process of innovation from the ground up”, from researchers to investors to consumers. He holds this firmly. It is framed as the “greatest challenge”.

Concepts: - Responsible innovation. Stilgoe et al. (2013): anticipation, reflexivity, inclusion, responsiveness, and “collective stewardship of science and innovation in the present”. He endorses it, but says “much more is needed”. - Socio-techno-environmental system, and interdependence between technologies. - Complex systems “appear stable and predictable — until, suddenly, it isn’t”.

Analogies: - The power grid cascade, used structurally and conceptually. - Nanotechnology and synthetic biology, used literally as examples where risk research on single technologies has progressed but cannot handle systemic risk. - Planetary boundaries (Stockholm Resilience Centre).

AI. - AI is one of the WEF Top Ten, placed in a network of converging technologies rather than singled out as unique. - He cites Musk and Hawking’s warnings and the FLI “robust and beneficial AI” letter. - The illustrative risks he names are “maliciously programmed autonomous weapons” and “intelligent machines that don’t understand or respect human values”. - He treats AI’s promise (“augmenting human intelligence”) and its risk even-handedly.

Governance. Horizon 2020’s Responsible Research and Innovation (RRI) programme is cited approvingly. Responsibility runs across the whole chain: researchers, innovators, investors, consumers, “society”.

Stance on precaution. Explicitly anti-brake: “we can’t afford to slam the breaks [sic] on emerging technologies”. Innovation is framed as necessary but insufficient: “Technology innovation alone will not solve these and other challenges. But without it, many will not be solved at all.”

Quotes: - “we continue to develop powerful new technologies at a rapid rate, with little thought as to how their very complexity and interconnectedness may cause them to unravel in the future.” - “These technologies are not inherently safe and secure. Yet they’re nevertheless critically important because of their potential benefits.” - “it’s one that will appear stable and predictable — until, suddenly, it isn’t.”

2016-01-11 — thinking-innovatively-about-the-risks-of-tech-innovation-cbbf708d7181#

Title: Thinking innovatively about the risks of tech innovation Provenance: His own prose (The Conversation). This is the founding public statement of “Risk Innovation” from the ASU Risk Innovation Lab.

Argument. CES hype hides the “darker side” of tech innovation. Robotics, AI and the Internet of Things (IoT) are “by no stretch of the imagination intrinsically safe”. The problems he names: - AI fears - IoT cyber-vulnerability - all three widening “the gap between the privileged and the disadvantaged”

Crucially, the relevant risks now include “threats to beliefs, community, culture, even sense of identity”, not only health and environment. His core claim is that our ways of thinking about and handling risk are as antiquated as the gadgets are new. Regulation is “inevitably built around previous technologies”. New technologies such as IoT, “cloud-based AI” and wearables get “shoehorned” into frameworks that are “not remotely the right shape”. That rigidity itself raises risk: it creates uncertainty and “obscures potential pitfalls”. What is needed is “parallel innovation in how we think and act on risk”. He holds this firmly and programmatically.

Concepts (his own definitions): - Risk Innovation. Approach risk “the way entrepreneurs approach innovation”: creativity, technical know-how, market savvy. - Risk as a threat to value. Value includes not just health, environment and money but “well-being, environmental sustainability, deeply held beliefs, even a sense of cultural or personal identity”; also professional standing, branding, equity, lifestyle and sense of worth. - The risk “market”. “the individuals, communities and organizations — the constituents (including developers and manufacturers) — who have something of tangible value (not to be confused with ‘values’) that is potentially threatened, and that they are willing to invest in protecting.” - Risk reframed from a “barrier to progress” into a support for “beneficial and sustainable progress”. - Ignoring intangible value leaves innovations “supremely vulnerable to failure” because they threaten “what people are prepared to fight for”.

AI. Cloud-based AI is named as a technology that doesn’t fit existing regulatory shapes. AI fears are acknowledged and linked to the Musk “existential threat” coverage in the Guardian.

Governance. Existing regulation is backward-looking and mismatched in shape. He criticises the “overwhelming impulse” to maintain the status quo.

Critique. Uncritical tech boosterism (CES), and inflexible, antiquated risk thinking.

Quotes: - “What we’re lacking — and what we desperately need — is parallel innovation in how we think and act on risk” - “The ways we’re taught to handle risk — even how we think about risk — are often as antiquated as the technologies being showcased at places like CES are innovative.” - “Rather than framing risk as a barrier to progress, risk innovation transforms it into a way of supporting beneficial and sustainable progress.” - “it runs the danger of inadvertently threatening what people are prepared to fight for.”

2016-01-11 — the-fourth-industrial-revolution-what-does-wefs-klaus-schwab-leave-out-b9297e6e5d8a#

Title: The fourth industrial revolution: what does WEF’s Klaus Schwab leave out? Provenance: His own prose (The Conversation). Quotes Bill Joy (Wired, 2000) and Schwab’s book.

Argument. He contrasts Bill Joy’s 2000 warning (robotics, genetic engineering and nanotech threaten to make humans “an endangered species”) with Klaus Schwab’s optimistic Fourth Industrial Revolution: converging physical, biological and digital technologies driving social progress. Schwab’s frame is useful for Davos leaders but reads “as if it were written in a vacuum”. It ignores decades of work by the community Andrew identifies with, which he calls “a global community of natural and social scientists… that has spent decades studying and acting on the interplay between science, society and emerging technologies”. That community has roots in Kuhn and in the GMO, nanotech, synthetic-biology, geoengineering and AI debates. Despite Schwab’s optimism, “the gap between our technological capabilities and our ability to handle them responsibly has continued to widen”. His examples are gene editing, autonomous vehicles, IoT and autonomous weapons. Closing the gap needs radical new approaches, partnerships between those with insight into the society-technology dynamic and “those that call the shots”, and inclusion of “ordinary people”.

Concepts and toolbox named: - technology assessment - anticipatory governance (“roots in the nanotechnology and synthetic biology ‘revolutions’”) - responsible innovation - foresighting - scenario planning - real-time technology assessment - socio-technical integration research - risk innovation and actionable empathy (linked to a 2015 Nature Nanotechnology piece), both flagged as “emerging ideas”

Analogies. The industrial revolutions of steam, electricity and computing (Schwab’s frame, which Andrew accepts provisionally). GMOs, nanotech, synthetic biology, geoengineering and AI are listed as the lineage of technologies his field has helped navigate. Here the comparison is literal and genealogical: the same expert community and toolkit apply across all of them.

AI. Listed among converging technologies. Autonomous weapons are named as an area where “we are way behind the curve”.

Tech leaders and elites. Schwab is respected as a convenor but criticised for ignoring expertise. Joy’s pessimism “still haunts me”.

Governance and who decides. Decision-makers (“those that call the shots”) need partnerships with experts on technology and society. Ordinary people, who “bear the brunt or reap the rewards”, must help decide. Technology should “serve society rather than dominate it”.

Quotes: - “the gap between our technological capabilities and our ability to handle them responsibly has continued to widen.” - “Joy’s earlier vision of a socially bankrupt technological future still haunts me.” - “if technology is to serve society rather than dominate it”

2016-02-01 — we-dont-talk-much-about-nanotechnology-risks-anymore-but-that-doesn-t-mean-they-re-gone-ba00cdcf6ab5#

Title: We don’t talk much about nanotechnology risks anymore, but that doesn’t mean they’re gone Provenance: His own prose (The Conversation). Autobiographical: he collaborated on the 2008 Nature Nanotechnology study showing asbestos-like harm from long carbon nanotubes in mice, and was science advisor to the Woodrow Wilson Center’s Project on Emerging Nanotechnologies (PEN).

Argument. In 2008, carbon nanotube (CNT) risks made headlines. In 2016, CNTs are in the news over Anish Kapoor’s exclusive rights to Vantablack, and nobody is asking about safety. The original Vantablack was used in space: “It wasn’t nontoxic, but the risk of exposure was minuscule.” The new spray version, Vantablack S-VIS, could be touched, inhaled or ingested. He says it is “not comparable to asbestos” but still raises “serious risk questions”: inhalability, particle form, how size, shape, surface area, porosity and chemistry drive toxicity, “Trojan horse” carriage of toxins, and environmental fate. Federal nano-safety research funding has risen 80% since 2008, so the risk questions are not resolved. Public silence is not evidence of safety. His explanation: journalists don’t know to ask, and trained nano-safety experts are absent from the media because scientists aren’t trained or encouraged to comment publicly. Boundary organisations such as PEN and Rice’s CBEN (Center for Biological and Environmental Nanotechnology) have lost their funding, and nothing has replaced them. He argues this needs to change. Talking publicly about what is and isn’t known “gets to the very heart” of socially responsible research and innovation.

Concepts: - Hazard vs exposure: risk depends on use context, not the material alone. - The fading of public attention while risk persists, a precursor to later thinking about neglected risks. - Boundary organisations linking scientists, users, journalists and influencers. - Perception of safety vs actually dealing with risk. - Knowing “what questions they should be asking”.

Analogies. Asbestos, used literally and toxicologically for long CNTs, with explicit limits drawn for Vantablack.

Expertise and publics. Responsibility is shared between scientists and their institutions and funders. He cross-links to the public universities piece.

Quotes: - “new technologies all too easily slip under the radar of critical public evaluation, simply because few people know what questions they should be asking about risks and benefits.” - “depends more on a perception of safety rather than actual dealing with risk” - “It wasn’t nontoxic, but the risk of exposure was minuscule.”

2016-03-02 — how-risky-are-the-world-economic-forums-top-10-emerging-technologies-for-2016-2494dbdccbf1#

Title: How risky are the World Economic Forum’s top 10 emerging technologies for 2016? Provenance: His own prose (The Conversation; really written June 2016 or later). He notes he has sat on the WEF advisory group compiling the list since 2012.

Argument. We imagine emerging-tech risk through Hollywood disaster tropes: superintelligent machines, lab-bred humans, redesigned species. Reality is more mundane: less “zombie apocalypse” and more Tay. Predicting plausible downsides is hard but necessary. There is a temptation to list concrete harms and apply tech fixes, for example self-driving cars sharing the road with “less ‘logical’ humans”, or engineered bacteria polluting. That approach masks “much more subtle dangers” that technology can’t fix: the social and psychological risks of internet-connected nanosensors revealing intimate biology, and the risks of “open AI ecosystems”. Applying risk-as-threat-to-value shows these subtler risks. It also brings in the risks of not developing a technology (threats to future value), such as road deaths prevented by autonomous vehicles, optogenetic cures, and clean water from graphene. These are weighed against what each technology threatens: human responsibility and the pleasure of driving, involuntary neurological control, ecosystem harm. His conclusion: risks are “context-specific, often intertwined with each other, sometimes conflicting, and often balanced by the risks of not developing”. The “greatest risk” is losing sight of beneficial development, whether through enthusiasm or through Hollywood-style fear.

Concepts: - Risk as threat to value, now extended explicitly to self-worth, culture, sense of security, equity and “deeply held beliefs”. Examples: an eroded sense of self-worth from job loss; anxiety over who knows what about you; fear of social marginalisation; dread over challenges to “sacrosanct beliefs — such as the sanctity of life, or the right to free choice”. - Future value, and the risk of not innovating. - Tech-fixable risks vs risks not “amenable to technological fixes”. - The “greatest risk” is a lost balance.

AI. This is a notable early passage on what would later be called agentic, conversational AI. Machines “are increasingly combining the capacity to understand normal conversation with the ability to take action on what they hear”. The risks: eavesdropping and sharing of private conversations, and systems that “independently decide what’s best for you”, which raises autonomy and paternalism concerns. He frames these as “ethical and moral concerns that aren’t easily addressed solely by tech solutions”. Tay illustrates the mundane real-world failure mode, as opposed to superintelligence.

Framing of AI risk. Anti-apocalyptic, pro-subtlety. The emphasis falls on privacy, autonomy and social and psychological harms.

Quotes: - “less “zombie apocalypse” and more “teens troll supercomputer; teach it bad habits.”” - “what happens when these AI ecosystems begin to listen in on private conversations and share them with others? Or independently decide what’s best for you?” - “These dangers are context-specific, often intertwined with each other, sometimes conflicting, and often balanced by the risks of not developing the technology.” - “we lose sight of the value of developing new technologies that make our world a better place, not just a different one.”

2016-03-31 — considering-ethics-now-before-radically-new-brain-technologies-get-away-from-us-3f1138aad71e#

Title: Considering ethics now before radically new brain technologies get away from us Provenance: His own prose (The Conversation; really about September 2016, pegged to the NAS/OECD workshop on responsible innovation in brain science, supported by ASU).

Argument. Neurotechnologies are converging with broader science: Berkeley’s “neural dust”, optogenetics, transcranial magnetic stimulation (TMS), which has altered moral judgement, and consumer and DIY transcranial direct current stimulation (tDCS). They promise treatments but will increasingly be able to “alter how someone thinks, feels, behaves and even perceives themselves and others around them”, and “not necessarily in ways that are within their control or with their consent”. His present-day questions: tDCS in exams? tDCS in classrooms? TMS to suppress a soldier’s moral judgement? His speculative scenario is neural dust plus optogenetics, networked to the internet, which he calls a crude “neural lace” (Iain M. Banks; Musk). It would allow direct interfaces with cloud AI: “Think Apple’s Siri or Amazon’s Echo hardwired into your brain”. The risks: - a neuro-enhancement socioeconomic divide - employers and police monitoring or altering thoughts - “cyber substance abuse” (direct-to-brain code replacing drugs) - neurological cyberattacks

Prediction is fraught, and what counts as risk “depends as much on who considers what a risk (and to whom)” as on capabilities. He calls for society-wide deliberation now: learn from recombinant DNA (Asilomar), nanotech and geoengineering; avoid pitfalls “while not stifling innovation”; make sure ordinary people can find out how these technologies affect them and “must have a say”.

Concepts. - Neuroethics. - Responsible innovation in brain science. - Risk as socially constructed: who defines what counts as risk, and to whom. - Anticipation through near-future speculative scenarios, a method he uses repeatedly.

Analogies. Recombinant DNA, nanotech and geoengineering, used conceptually and procedurally as precedents for anticipatory ethical governance. Science fiction (Banks) serves as the conceptual prototype.

Cognition and formation. This is the batch’s strongest early statement on technologies that act directly on cognition, moral judgement, personality and self-perception, and on consent. It includes the education angle (tDCS in exams and classrooms) and AI coupled directly to the brain.

Governance. Society should decide what it wants the future of brain tech to be. The public must have a say, and he wants innovation that is not stifled.

Quotes: - “alter how someone thinks, feels, behaves and even perceives themselves and others around them” - “depends as much on who considers what a risk (and to whom) as it does the capabilities of emerging technologies to do harm.” - “Think Apple’s Siri or Amazon’s Echo hardwired into your brain, and you begin to get the idea.” - “they must have a say in how they’re used.”

2016-04-01 — will-driving-your-own-car-become-the-socially-unacceptable-public-health-risk-smoking-is-today-8114ab8463aa#

Title: Will driving your own car become the socially unacceptable public health risk smoking is today? Provenance: His own prose (The Conversation; really about September 2016, pegged to the DOT Federal Automated Vehicles Policy).

Argument. About 94% of crashes involve human choice or error. If self-driving cars could cut deaths tenfold or more, “human-driven vehicles will be seen as a public health risk to be managed and ultimately eliminated”. The smoking analogy shows that social norms can shift dramatically through public health campaigns, regulation and culture. He tests DOT’s new policy against Stilgoe et al.’s responsible innovation framework and finds that it largely “ticks the boxes”: it anticipates, acknowledges limits, engages the public, and is designed to evolve. He praises it as “a refreshing change from attempting to retrofit existing regulations to new technologies”. It sets flexible rules that encourage socially beneficial innovation instead of dictating design. It could be a model for governing other emerging technologies. He then flags the tension: people will protest against lost liberties and the “American way of life”, which raises the question of whether “responsibility” means only reducing injury and death, or also protecting freedom and culture. So responsible innovation requires that everyone affected can help guide development, and that trajectories can be continually adjusted. His tone is optimistic: “the public health expert in me is excited”.

Concepts. - Responsible innovation (restated as: anticipate; be aware of limits and open; include stakeholders and the public; be responsive). - Adaptive, flexible regulation. - Public-health framing of technology risk. - The risk of not realising benefits, “perhaps because of irresponsible development or over-restrictive regulations”.

Analogies. Smoking, used structurally for social-norm change around a risky behaviour.

AI. Implicit: machine learning fleets learn “from each near-miss, scrape and full-blown crash”.

Governance. This is the clearest positive model of governance in the batch: anticipatory, adaptive and participatory, as opposed to retrofitting or over-restriction.

Quotes: - “at some point, human-driven vehicles will be seen as a public health risk to be managed and ultimately eliminated.” - “This is a refreshing change from attempting to retrofit existing regulations to new technologies” - “does it simply mean reducing the risk of injury and death, or does it also mean protecting other things that are important to people, like freedom and culture?” - “responsible innovation depends on ensuring everyone potentially touched by a new technology has the chance to be a part of guiding how it’s developed and used.”

MEDIUM#

2014-12-14 — researchers-should-take-more-responsibility-for-exaggeration-in-press-releases-5a4f90e080f9#

Title: Researchers should take more responsibility for exaggeration in press releases Provenance: His own prose. Quotes the Sumner et al. BMJ study and Ben Goldacre’s BMJ editorial.

Argument. The BMJ study of 462 press releases from 20 UK universities found exaggerated advice (40%), causal claims (33%) and animal-to-human inference (36%). News articles were far likelier to exaggerate when the release did (58% vs 17%). This matters especially for risk-related research, because bad health advice changes behaviour and harms health. The root is a conflict between promotion and communication: branding, fundraising and prestige for institutions, and citations, funding and ego for researchers. Scientists sign off releases, so they bear most of the responsibility. He supports Goldacre’s proposal to name authors on press releases, but hopes more for a “culture of responsibility”. He positions himself as respectful of press offices but “critical in the past”.

Concepts. - The chain of trust from researcher to reader. - Promotion vs communication. - Researcher responsibility for downstream use of their claims. - Peer-reviewed provenance as persuasive cover (“a provenance that I suspect many people find it hard to resist”).

Expertise and publics. Science-derived misinformation begins at the source, not in the media.

Quotes: - “There is certainly a chain of trust between the researcher and reader that depends on responsible representation and reporting at all stages of the communication process.” - “where misleading information is used by real people to make life-impacting decisions, such slippage can only be seen as irresponsible.” - “placed the health and well-being of their constituencies before their own aggrandizement.”

2015-01-10 — are-quantum-dot-tvs-and-their-toxic-ingredients-actually-better-for-the-environment-d47c24feec40#

Title: Are quantum dot TVs — and their toxic ingredients — actually better for the environment? Provenance: His own prose (The Conversation).

Argument. Quantum dot TVs use cadmium selenide nanoparticles, combining a restricted, carcinogenic heavy metal with the uncertain toxicity of engineered nanoparticles. “Together, these factors would suggest caution is warranted… Yet taken in isolation they are misleading.” He works through exposure across the life cycle: - use: exposure effectively nil, because the dots are sealed behind glass and plastic - manufacture: real but manageable with good practice (NIOSH, the US National Institute for Occupational Safety and Health) - disposal: a concern mainly where e-waste is poorly regulated

Then the benefits: over 20% energy savings and less toxic material. QD Vision’s analysis to the EPA showed a net decrease in cadmium released once coal-plant cadmium emissions are counted. The case “eloquently demonstrates the dangers of jumping to conclusions over risks without seeing the full picture”. It depends on a commitment to responsible innovation. QD Vision won the EPA Green Chemistry award.

Concepts. - Hazard vs exposure. - Life-cycle thinking. - Risk-risk and risk-benefit trade-offs (net risk). - Responsible development by companies.

Analogies. Literal chemical and nanomaterial risk: cadmium, EU RoHS (the Restriction of Hazardous Substances directive), IARC (International Agency for Research on Cancer).

Stance. He is against hazard-based alarm and for full-picture risk assessment.

Quotes: - “Yet taken in isolation they are misleading.” - “it eloquently demonstrates the dangers of jumping to conclusions over risks without seeing the full picture.”

2016-01-12 — can-citizen-science-empower-disenfranchised-communities-99e6e92acad9#

Title: Can citizen science empower disenfranchised communities? Provenance: His own prose (The Conversation and Brink). Draws on UCL’s Extreme Citizen Science programme (ExCiteS).

Argument. Most citizen science is “big on engagement, maybe not so much on empowerment”: researchers still set the questions. Citizen-led science is different. His two examples: - The Nappy Science Gang: parents challenged NHS advice on biological detergents, and the NHS changed its position. - The Flint Water Study: Virginia Tech partnered with residents, who had been “ignored and marginalized” and repeatedly told by experts and officials that their water was safe. The residents’ testing exposed lead contamination.

This “extreme citizen science” lets communities “own” the scientific method. It needs experts willing to serve and not co-opt, access to instruments, open access to journals, and funding. The greatest barrier is “intransigent institutionalized attitudes”. In one anecdote, senior science advisers called his suggestion that scientists listen to ordinary people “a really bad idea”.

Concepts. - Citizen-led science. - Extreme Citizen Science. - Empowerment vs engagement. - Expert elitism.

Expertise, publics and justice. Experts and officials were wrong while residents were right. The need for access to science tracks inequality.

Quotes: - “They’re big on engagement, maybe not so much on empowerment” - ““nonscientists” should revere, but not interfere with, science.” - “access to experts who are willing to serve the needs of citizens — and not just co-opt them for their own ends.”

2016-01-20 — three-ways-synthetic-biology-could-annihilate-zika-and-other-mosquito-borne-diseases-10060d74cf9d#

Title: Three ways synthetic biology could annihilate Zika and other mosquito-borne diseases Provenance: His own prose (The Conversation; really February 2016 or later).

Argument. Synthetic biology lets us “upload” genetic code, edit it and “download” it into organisms: “we’ve discovered how to hack biology”. He sets out three routes: - Oxitec’s tetracycline-dependent kill switch - rapid, digitally distributed synthetic vaccines (after Drew Endy) - CRISPR gene drives that could re-engineer or eliminate whole species

Gene drives are “wickedly smart” but raise the gravest issue: eliminating or re-engineering a species when the unintended consequences are unclear. We recode DNA “almost as easily as we can write smartphone apps”, yet understand the systems only vaguely. This is “not an argument against”, because the lives at stake are enormous. But it demands being “exceptionally cautious” and a better understanding of what could go wrong, who is affected and how errors can be corrected. He cites approvingly the international human gene editing summit and research on reversal drives.

Concepts. - Hacking biology. - The digital-biological convergence. - Irreversibility as the ground for extra caution. - Correctability.

Analogies. Computer code and apps, used conceptually. His signature image is life’s operating system without the chance to reboot, which marks the limit of the digital analogy.

Stance on precaution. Strongly pro-benefit, with extra caution matched to irreversibility and ignorance.

Quotes: - “It’s as if we’ve been given free rein to play with life’s operating system code, but unlike computers, we don’t have the luxury of rebooting when things go wrong.” - “This is not an argument against using synthetic biology to combat Zika and other infectious diseases — far from it.” - “greater efforts are needed to understand what could go wrong, who and what might potentially be affected and how errors will be corrected.”

2016-01-31 — public-universities-must-do-more-the-public-needs-our-help-and-expertise-9191de5eafe6#

Title: Public universities must do more: the public needs our help and expertise Provenance: His own prose (The Conversation; really March 2016 or later). Quotes the Flint Virginia Tech PhD students and Crow and Dabars’ Designing the New American University.

Argument. - In Flint, individuals rather than their public universities gave residents a voice: Mona Hanna-Attisha (Michigan State) and Marc Edwards (Virginia Tech). - Faculty evaluation (research, teaching, service) discourages public good. The Flint students learned “costs of doing good science”. - Land-grant missions are vague, and rankings-driven metrics reward institutional self-interest. He suggests this explains the slow response of the University of Michigan and Michigan State.

His proposals: - add a fourth leg of community service to faculty expectations - support communication and engagement training (The Conversation, RELATE) - use institutional reach to amplify marginalised communities’ voices; early tweets might have helped Flint

Universities lack codes of socially responsible conduct comparable to corporate social responsibility (CSR): “money, prestige and power can be corrupting motives”. The individuals are well-meaning, but “the institution gets in the way”. ASU is his hopeful model. He draws on personal experience: the University of Michigan Risk Science Center succeeded “despite the institution”.

Concepts. - Institutional vs personal responsibility. - A fourth leg of community service. - Universities as amplifiers for marginalised communities.

Education and higher education. An early statement of his reformist view of universities.

Quotes: - “there’s a stark disconnect between personal aspirations and institutional expectations” - “They want to serve the public good. It’s just that, somehow, the institution gets in the way.” - “Even without profit as the top goal, money, prestige and power can be corrupting motives.”

2016-03-12 — itll-take-more-than-tech-for-elon-musk-to-pull-off-audacious-new-tesla-master-plan-8308ce551490#

Title: It’ll take more than tech for Elon Musk to pull off audacious new Tesla master plan Provenance: His own prose (The Conversation; really July 2016 or later).

Argument. Musk’s “Master Plan Part Deux” (solar roofs, the full EV line, self-driving that is “10X safer”, and a shared autonomous fleet) is technologically coherent. But it requires “a seismic shift in modern culture”: ownership, possessions, values and norms. The fatal Autopilot crash shows that “numeric logic is often trumped by what we intuitively think and feel is important”. People care about who dies, how, and who or what decides, not only how many. Musk shows some awareness: his FLI funding of $11M for AI safety, including an Oxford–Cambridge AI policy centre. But he must “think more broadly still — and fast”. The problems are wicked (“so slippery they change and shift in response to attempts to solve them”), with GM foods as the example. Musk needs to befriend responsible innovation, technology governance, technology assessment and risk innovation. Andrew admits bias, since this is ASU SFIS’s field. He is admiring but conditional: “implemented responsibly — Musk’s vision could be a game changer”.

Concepts. - Wicked problems. - Social and political savvy as a condition of innovation success. - Statistics vs intuitions and values in risk acceptance.

Analogies. GM foods, used conceptually as the model of a wicked tech-society problem.

Tech leaders. Musk is technically brilliant and socially under-equipped. The prescription is to engage the social-science and responsible-innovation community.

AI. Machine learning fleet learning. FLI and AI-safety funding are mentioned approvingly as a sign of breadth.

Quotes: - “While Musk and his teams have the technical know-how to implement the master plan, it’s not clear yet whether they have the social and political savvy to make it work.” - “numeric logic is often trumped by what we intuitively think and feel is important.”

2017-04-10 — dear-elon-musk-your-dazzling-mars-plan-overlooks-some-big-nontechnical-hurdles-f39eb0cfb04a#

Title: Dear Elon Musk: Your dazzling Mars plan overlooks some big nontechnical hurdles Provenance: His own prose (The Conversation; really about October 2017). He writes “As a scholar of risk innovation”.

Argument. SpaceX may well solve the engineering. The harder risks are social and political: - Planetary protection. As a private company, SpaceX “isn’t directly bound by international planetary protection policies”. The main risk is destroying evidence of Martian life, and angering astrobiologists. - Ecoterrorism against terraforming, with Kim Stanley Robinson’s Mars Trilogy as a guide. - Space politics. The 1967 Outer Space Treaty was written for states. Private actors are in a period of legal ambiguity that will eventually harden into regulation. - Climate change. The optics of a “disposable Earth”, plus BFR point-to-point emissions.

It all comes down to “whether society writ large grants SpaceX and Elon Musk the freedom” to proceed. If people feel their values are threatened, or feel disadvantaged (“rich people… abandoning the rest of us”), they will make life hard for the company.

Concepts. - Risk landscape. - Nontechnical hurdles. - Risk as threat to value, applied to a company. - A social licence granted by society. - Private actors outrunning governance designed for states. Structurally relevant to later AI-company arguments; he doesn’t draw that link here.

Analogies. Earthly environmental activism and ecoterrorism (the Earth Liberation Front), and Sputnik. Science fiction is used as foresight.

Tech leaders. Musk must be “as socially adept as they are technically talented”. He writes as a well-wishing adviser, not a hostile critic.

Quotes: - “These nontechnical hurdles come down to whether society writ large grants SpaceX and Elon Musk the freedom to boldly go where no one has gone before.” - “Musk and SpaceX need to be as socially adept as they are technically talented.”

2018-02-21 — the-bs-and-the-science-of-nanotechnology-a1df151008ef#

Title: The BS and the science of nanotechnology Provenance: His own prose (The Conversation; really about late May 2018, prompted by Musk’s tweet calling nanotechnology BS).

Argument. Andrew has nearly 30 years in nanoscale science, and “my BS monitor also gets a little twitchy”. In the 1990s, “nanotechnology” was branded as “the next industrial revolution” to sell the idea to funders and policymakers. The 1–100 nm “novel properties” definition “fudged the science to sell the idea”. “No science” shows the range is generically special. Experts “went along with it because of the promise of funding”. Many disciplines rebranded themselves overnight, and Drexler’s vision was marginalised. Nanoscale science has deep roots going back to the early 1900s, including Aerosil in the 1940s, materials science and giant magnetoresistance. Yet “brand-nano” did something valuable: it “broke down the barriers between previously stove-piped disciplines” and spurred real advances. He worries that the history has been lost as brand-nano became institutionalised, and welcomes occasional challenges to its assumptions.

Concepts. - Brand-nano vs nanoscale science and engineering. - Marketing science, where hype is used to secure funding. - An arbitrary “bright line” definition. - Institutionalisation erasing history.

Analogies. “Industrial revolution” rhetoric, which links conceptually to his critique of Schwab’s 4IR.

Change of view. A candid retrospective reassessment of the field he helped build and promote. It implicates his own community (“most experts at the time realised was more marketing than science”).

Tech leaders. Musk is probably “trolling”, but the challenge is useful.

Quotes: - “the way nanotechnology was initially pitched fudged the science to sell the idea.” - “this is where brand-nano risks becoming unstuck from scientific reality.” - “there’s nothing particularly special about that bright line between the worlds below and above 100 nanometres.”

2018-05-12 — 10-potential-risks-of-artificial-intelligence-we-should-probably-be-thinking-about-now-2e52a1360c90#

Title: 10 potential risks of artificial intelligence. Provenance: His own short prose, wrapping a Risk Bites video (his own YouTube channel; video content not assessed here). Originally published on 2020science.org.

Argument. AI’s risks span a spectrum. At one end are bots that pick up our traits (the Tay allusion). At the other is “the specter of super-intelligent machines that decide the one thing they really can’t stand is people”. In between is “a whole landscape of AI applications that could make our lives better or worse”. His ten named risks: 1. technological dependency 2. job replacement and redistribution 3. algorithmic bias 4. non-transparent decision making 5. value-misalignment 6. lethal autonomous weapons 7. re-writable goals 8. unintended consequences of goals and decisions 9. existential risk from superintelligence 10. heuristic manipulation

Significance. This is his earliest explicit AI-risk taxonomy in the corpus. It includes existential risk as one item among many, not dismissed but not privileged. It also names technological dependency and heuristic manipulation, early nodes of his later concern with cognition and manipulation. The tone is plain-language and playful (the 2001 and Alexa references).

Quotes: - “a whole landscape of AI applications that could make our lives better or worse” - “the specter of super-intelligent machines that decide the one thing they really can’t stand is people.”

2018-08-22 — second-guessing-consumer-views-on-products-using-nanotechnology-7cdbfeb6acc0#

Title: Second-guessing consumer views on products using nanotechnology Provenance: His own short prose, announcing a Nature Nanotechnology commentary co-authored with Justin Kidd. The commentary itself is not included.

Argument. Manufacturers of nano-enabled home water treatment may be held back “not because of what consumers think, but because of what others think they think”. Early findings: manufacturers are highly sensitive to consumer reactions they have imagined without evidence, while consumer awareness of nanotechnology “has barely shifted over the past 15 years or so”.

Concepts. - Second-order perception (assumed public perception) as a barrier to beneficial innovation. - Risk perception without evidence.

Quote: - “not because of what consumers think, but because of what others think they think.”

LOW / NONE#