We Have a Thousand Security Problems with AI, but Human Extinction Is Not One of Them

Introduction

Listening to the biggest players in the AI industry, it can sometimes be difficult to tell whether we are attending a technology conference or watching the trailer for a disaster movie.

Leaders of OpenAI, Anthropic, and other frontier AI labs regularly talk about how advances in artificial intelligence could create risks capable of threatening human civilization as a whole. They publish safety frameworks, define dangerous capability thresholds, and occasionally state that certain developments should be slowed down because models have come too close to a critical boundary.

In my experience, people tend to respond to these claims in one of two ways:

One camp believes that AI companies are essentially building their own Frankenstein's monster, and that they at least have enough responsibility to warn the world about it from time to time.

The other camp sees the whole thing as a brilliant marketing campaign: there is no better way for a company to convince people that it owns the world's most powerful technology than to say that it is so powerful it might destroy humanity.

Let's look at what is actually true.

The Risks of AI Are Very Real

There is no shortage of genuine security problems associated with artificial intelligence.

AI models can already be used to write convincing phishing messages more cheaply and quickly, create manipulated images and videos, mass-produce disinformation, or write malicious code. At the same time, models are being introduced into more and more decision-making processes: screening job applications, insurance processes, financial decisions, education, healthcare, and military systems.

Along with this comes a more extreme version of the problem of fault diagnosis and opaque, complex systems: how do we diagnose a failure in a complex system when we can only partially understand its internal operation and decision-making process?

There is the issue of data protection. Copyright. Deepfakes. Automated fraud. Cybersecurity. Political manipulation. Algorithmic discrimination. Labor market disruption. AI systems are increasingly being given tasks in which they do not simply answer a question, but take action. They modify files, write programs, operate other systems, and initiate transactions.

The more autonomous these systems become, the more serious the question of how we control them becomes. These are not science fiction problems; they are engineering, security, and societal problems that already exist today.

But That Still Does Not Mean Terminator

This is where the conceptual leap occurs that gives rise to many of today's misunderstandings. The fact that the development of AI raises serious security problems does not in any way mean that AI will lead to the extinction of the human species. There is an enormous distance between the two.

For an artificial intelligence system to become a genuine existential threat, many things would have to be true at the same time. The system would not only need to be extremely intelligent, but would also need persistent goals, strategic planning capabilities, and significant autonomy. It would have to be capable of acting reliably in the real world over long periods of time, and achieving this would require the following:

• It would need access to important infrastructure.
• It would need to recognize that humans are trying to restrict it.
• It would then need to successfully circumvent those restrictions.

This is not an impossible scenario, but it is currently not a proven development trajectory either.

The problem is that in AI security discourse, a demonstrable cybersecurity problem that exists today and a hypothetical scenario involving the destruction of humanity are often mentioned in the same sentence, even though the strength of the evidence behind them is very different.

The Apocalypse as Marketing

There is, however, another reason why AI companies like to talk about existential risks: it is incredibly effective marketing. Consider which of the following statements sounds more powerful.

“We built software that can perform certain office tasks much faster.”

Or:

“Our product is not simply a better chatbot, but a technology so incredibly powerful that we have to make sure it does not become too dangerous.”

It is extremely advantageous for an AI company if the public believes that it is not simply developing better software, but something potentially more significant than the internet, nuclear energy, or the Industrial Revolution.

Fear and technological superiority reinforce each other in a strange way in this case. The message “our product is dangerous” also effectively means “our product is incredibly powerful.”

Regulation as a Business Tool

Communication about AI risks also has a less visible but economically very important dimension. If a technology is dangerous, it needs to be regulated.

And if regulation requires extremely expensive security testing, large-scale computing infrastructure, audits, and specialized compliance systems, then the companies that already spend billions on AI development will have an advantage.

In other words, the same regulation that may genuinely improve safety can also create barriers to entry. A compliance system that represents a manageable cost for OpenAI or Anthropic may impose an enormous burden on a startup.

This does not, of course, mean that every AI regulation proposal is some kind of conspiracy. It simply means that the leading players in the industry may also have clearly identifiable economic interests in emphasizing the risks of artificial intelligence: the perception of high risk may contribute to the development of a regulatory environment whose costs and technical requirements are significantly easier for already dominant, well-capitalized companies to absorb than for smaller competitors.

This is worth keeping in mind when the same companies simultaneously tell us that their technology is extremely dangerous and that they are the ones with the expertise required to develop it safely.

Focusing on the Wrong Problem Is a Risk in Itself

The biggest problem with the AI apocalypse narrative is not that it is necessarily wrong. It is possible that in the long term we really will create systems that are extremely difficult to control, which is why this possibility should be taken seriously as a research topic.

The problem is rather that hypothetical doomsday scenarios can very easily push aside the problems that we should already be addressing today.

A criminal using AI to send personalized scams to tens of thousands of people does not need superintelligence, just as today's level of technology is already sufficient for inexperienced hackers to carry out cyberattacks with significantly less expertise, time, and effort.

A political campaign seeking to generate millions of pieces of manipulated content does not need a conscious machine. A company that entrusts decisions affecting people's lives or finances to a flawed algorithm does not need technology more advanced than what already exists today either.

For this reason, both the security profession and business stakeholders should place greater emphasis on the very concrete risks we already face today rather than hypothetical civilization-ending catastrophes caused by superintelligence: what can an AI system get wrong, who can it harm as a result, and ultimately, who bears responsibility for it?

How Seriously Should We Take AI Companies' Own Safety Research?

It is important not to dismiss the safety research conducted by OpenAI and Anthropic simply by saying that “it is just marketing.” Both companies maintain serious research teams, publish detailed system cards, safety evaluations, and risk frameworks, and in some cases involve external experts in evaluating their models. OpenAI's own Preparedness Framework, for example, separately addresses cybersecurity, biological, manipulation, and loss-of-control risks, and the company regularly publishes pre-deployment evaluations of its models.

Anthropic has built a similarly substantial apparatus. Its Responsible Scaling Policy defines what kinds of safety measures it considers necessary at different capability levels, while the company also operates dedicated research programs focusing on alignment, model behavior, and dangerous capabilities. The company has also conducted safety evaluations with external organizations and itself emphasizes that third-party audits can reduce the risk of confirmation bias among its own researchers.

However, this does not automatically turn these publications into independent scientific evidence.

The problem is structural. The same organization develops the model, selects the risks to be examined, designs a significant portion of the evaluations, interprets the results, and then communicates those results to the public.

Moreover, the independent reproducibility of frontier models is inherently limited. An external researcher typically does not have access to the full training process, the model weights, internal logs, or the same infrastructure on which the company conducts its own evaluations. As a result, many findings in AI safety are closer to controlled corporate evaluations than to laboratory experiments that can be reproduced by anyone in the classical scientific sense.

A single corporate report or internal test therefore cannot be regarded as definitive evidence on its own, particularly when external researchers have only limited access to the data, models, or complete experimental procedure. For such observations to support genuinely strong and generalizable scientific conclusions, independent replication, results from multiple research groups, and a much broader scientific consensus are required.

In short: if an AI lab's research claims that its model has demonstrated a dangerous capability, it is worth remembering that the claim comes from the same organization that also has a business interest in the world viewing its technology as exceptionally advanced and dangerous.