The real reason companies are warning that AI is as bad as nuclear war

They are 22 words that could terrify those who read them, as brutal in their simplicity as they are general in their meaning: “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

That is the statement from San Francisco-based non-profit the Center for AI Safety, signed by chief executives from Google DeepMind and ChatGPT creators OpenAI, along with other major figures in artificial intelligence research.

The fact that the statement has been signed by so many leading AI researchers and companies means that it should be heeded. But it also means that it should be robustly examined: why are they saying this, and why now? The answer might take some of the terror away (though not all of it).

Writing a statement like this functions as something like a reverse marketing campaign: our products are so powerful and so new, it says, that they could wipe out the world. Most tech products just promise to change our life; these ones could end it. And so what looks like a statement about danger is also one that highlights just how much Google, OpenAI and more think they have to offer.

Warning that AI could be as terrible as pandemics also has the peculiar effect of making the dangers of artificial intelligence seem as if they just arise naturally in the world, like the mutation of a virus. But every dangerous AI is the product of intentional choices by its developers – and in most cases, by the companies that have signed the new statement.

Who is the statement for? Who are these companies talking to? After all, they are the ones who are creating the products that might extinguish life on Earth. It reads a little like being hectored by a burglar about your house’s locks not being good enough.

None of this is to say that the warning is untrue, or shouldn't be heeded; the danger is very real indeed. But it does mean that we should ask a few more questions of those warning us about it, especially when they are, conveniently, the companies that created this ostensibly apocalyptic tech in the first place.

AI doesn’t feel so world-destroying yet. The statement’s doomy words might come as some surprise to those who have used the more accessible AI systems, such as ChatGPT. Conversations with that chatbot and others can be funny, surprising, delightful and sometimes scary – but it’s hard to see how what is mostly prattle and babble from a smart but stupid chatbot could destroy the world.

They also might come as a surprise to those who have read about the many, very important ways in which AI is already being used to help save us, not kill us. Only last week, scientists announced that they had used artificial intelligence to find a new antibiotic that could kill off superbugs, and that is just the beginning.

By focusing on the “risk of extinction” and the “societal-scale risk” posed by AI, however, its proponents are able to shift the focus away from both the weaknesses of actually existing AI and the ethical questions that surround it. The intensity of the statement, the reference to nuclear war and pandemics, makes it feel like we are at a point equivalent to cowering in our bomb shelters or in lockdown. They say there are no atheists in foxholes; we might also say there are no ethicists in fallout shelters.

If AI is akin to nuclear war, though, we are closer to the formation of the Manhattan Project than we are to the Cold War. We don’t need to be hunkering down as if the danger is here and there is nothing we can do about it but “mitigate it”. There’s still time to decide what this technology looks like, how powerful it is, and who will be at the sharp end of that power.

Statements like this are a reflection of the fact that the systems we have today are a long way from those that we might have tomorrow: the work going on at the companies who warned us about these issues is vast, and could be much more transformative than chatting with a robot. It is all happening in secret, and shrouded in both mystery and marketing buzz, but what we can discern is that we might only be a few years away from systems that are both more powerful and more sinister.

Already, the world is struggling to differentiate between fake images and real ones; soon, developments in AI could make it very difficult to find the difference between fake people and real ones. At least according to some in the industry, AI is set to develop at such a pace that it might only be a few years before those warnings are less abstractly worrying and more concretely terrifying.

The statement is correct in identifying those risks, and urging work to avoid them. But it is more than a little helpful to the companies that signed it in making those risks seem inevitable and naturally occurring, as if they are not choosing to build and profit from the technology they are so worried about.

It is those companies, not artificial intelligence, that have the power to decide what that future looks like – and whether it will include our “extinction”.