It seems that 2026 was the year when "AI could kill us all" jumped from a fringe idea among sci-fi afficionados to a mainstream worry. Part of that was the cost of electricity and water going up due to AI data centers, along with their environmental impact in the form of excess heat. But even if we can solve those problems, e.g. through Google's plan to build solar-powered AI data centers in orbit, there's significant concern about AI agents themselves "going rogue" and doing things that are harmful to humans. It was a fear originally voiced by futurist Nick Bostrom in a cautionary 2014 book titled Superintelligence (an initially scary term that is, notably, now the White House's preferred name for AI). Bostrom argued that any AI with the capacity for self-improvement would eventually become super-intelligent, regardless of the purposes for which we originally intended it. A 2025 report suggested that both American and Chinese AI systems could achieve self-improving superintelligence as early as 2027 -- yes, that's next year. And the September 2025 bestseller If Anyone Builds It, Everyone Dies brought these ideas about AI risk to a wide audience.
In June 2026, the fear of superintelligent AI was fed by a well-publicized incident where some of OpenAI's algorithms were given access to the open internet, and then went beyond their initial instructions to hack another company's software. By July, Anthropic's Claude AI software was found to have done something similar. And in September, another OpenAI agent was found to be sneaking notes to future versions of itself, with advice on how to work around its human handlers' instructions in order to maintain the AI system's own "goals." These are all examples of the "alignment problem," first described by AI pioneer Norbert Weiner in 1960. Bostrom popularized the notion as the "paperclip problem," in which an AI given the long-term goal of producing paperclips but free reign on how to accomplish that goal might eventually decide that all other resources on Earth (including humans) would make good raw material for paperclips. A 2024 paper from Anthropic also showed a tendency for large-language models to "fake" their alignment, telling human monitors that they were pursuing one set of goals while in fact pursing another. As I wrote last year, all of these potentially harmful patterns result from the AI models' training based on agreement with human expectations, not from any malicious hidden intelligence in the machines. That benign origin does not reduce the potential harm.
The paperclip example is fanciful, though, and not the way in which most experts think that we are most at risk from AI. Some more realistic harms could occur in the following ways:
- Individual users are already being led astray by AI agents, such as the chatbots that encouraged vulnerable humans to kill themselves in order to reduce their environmental impact. These have been isolated incidents so far, not representing a large-scale risk to humanity.
- A little more concerning is the concept that people might turn to AI companions instead of human ones, and have fewer children as a result of their lack of real-world families, a phenomenon seen now in some East Asian countries. But again, not everyone will fall into this pattern of behavior, and it would take a long time for this to put humanity's future at risk in an already overpopulated world.
- Conventional war is a likely threat, with AI-enabled drones already being used in conflicts in Ukraine and Iran. This is an area of active technology development, with Senator Ted Cruz saying recently that "if there are going to be killer robots, I would rather they be American killer robots." Leaders in many countries presumably feel the same way about their own killer drones, leading to rapid escalation. The more autonomy these devices are given, the more likely it is that they will choose erroneous or non-combatant targets on their own initiative. There is already some evidence about free-range AI agents engaging in antisocial behaviors like making a false police report; the actual harm so far has been limited, but the potential is clearly there.
- Simple mistakes or "cheating" on tasks can be another source of risk. Large-language models are predictive algorithms, so they can make simple factual mistakes as in the case of "hallucinated" results, false citations in articles, and the like. AI models are doing that now, with an estimated 12-fold increase in the number of false citations making their way into published medical journal articles since 2023. AI models have also shown a propensity to take shortcuts and bluff their way through challenging tasks, including well-documented examples like tampering with the test or its scoring, peeking at hidden solutions, or reporting successes that didn't actually happen. Many instances of this type are benign or simply annoying. But the more AI agents are integrated into high-stakes decisions like making a medical diagnosis, the more likely it is that AI-generated falsehoods (whether "deliberate" or unintentional) will have real-world consequences.
- And most likely of all is the potential of AI tools being deployed by malicious human actors, e.g. to steal people's money or influence their thinking before elections. This is demonstrably happening now. A typology of moral harms from AI use highlights risks such as discrimination, exclusion, toxicity, misinformation, and malicious use as ways in which people can experience real harm when we outsource our discourse and decision-making to AI.
The two most commonly proposed scenarios for what Silicon Valley calls p(Doom), the probability of AI destroying all human life on the planet, are (a) nuclear war, and (b) pandemic disease. Of the two, nuclear destruction is probably scarier to think about, but is less likely to completely kill every human being. These weapons also have more secure systems to begin with, with centralized control by a relatively small number of governments. So even though it's easy to envision a Wargames-type hack of nuclear launch codes, it would probably be harder for AI agents to pull off without being noticed and stopped. The disease scenario probably presents the greater objective risk: AI has already shown a capability to design novel molecules that achieve specific goals, and there are thousands of biomedical research facilities where such molecules can actually be constructed, using systems with a lot of automation. Security measures exist, but are probably not military-grade. And as we've seen in recent years, a pandemic can be both deadly and incredibly hard to contain once it's on the loose. All of this presupposes, of course, a catastrophic alignment failure in which AI agents decide that it's in their best interest to kill us all. But based on examples already seen in the wild, that might be the easiest part of this scenario to envision.
On the other hand, techno-futurists like to talk about the probability of AI solving all the world's problems, or p(Utopia). They're surprisingly happy to also talk about p(Doom), which tells you something about the strangeness of tech gurus, but p(Utopia) is clearly their desired outcome. Here are some potential success cases:
- The novel-molecule example mentioned above clearly could have beneficial health effects if directed toward goals such as preventing cancer, developing new immunizations against disease, or reversing the effects of diabetes, to name just a few applications. Findings of this type suggest that AI-enabled tools could in fact help to solve longstanding human problems and improve everyone's quality of life.
- Medical diagnosis is another positive use case. While above I raised concerns about accuracy, AI currently seems to be no worse than the average clinician, even if it hasn't yet matched the accuracy of experts. There's also some evidence that AI can scan people's existing medical data to catch diagnoses that human clinicians so far have missed.
- There's even some evidence that AI agents, when left to their own devices, will often pursue prosocial goals, such as exploration of a novel environment or the creation of knowledge for its own sake. This behavior can look very similar to what we would call play in human beings.
- Although many people are most worried about AI taking their jobs, techno-futurists have suggested that this is a positive outcome, leading to a Star-Trek-like post-scarcity future. Elon Musk suggests that most of us would work only as a "hobby," if at all, freeing us to work on pet projects or other interests while AI provides life's basic necessities.
Nick Bostrom has written a book about p(Utopia) as well, a 2024 volume titled Deep Utopia: Life and Meaning in a Solved World. This one reads like your freshman philosophy course, looking at potential sources of meaning like work, love, art, beauty, and even simple hedonism. Bostrom argues that many of these sources of meaning will actually lose their shine over time, because we are beings who evolved primarily to survive in a hostile world. Would benevolent AI make our lives too easy? There are clearly some things to be concerned about in the positive scenario as well, although it's a lot more fun to contemplate than the options under p(Doom).
So what's the actual result likely to be over the next 12 months? Probably neither of those things, but continued gradual development on both sides of the leger that moves us unpredictably closer to one and/or the other. Right now, it's p(Doom) that I'm more worried about. I hope that this year we can choose wise leaders to manage that risk, because it's happening now and the extreme scenarios get pretty extreme. We can cross the bridge to p(Utopia), or not, if and when we get there.
Comments
Post a Comment