- AI security encompasses both protecting models and data from attacks and using AI to strengthen cybersecurity and reduce incidents.
- The risks combine technical failures (data poisoning, adversarial examples, Trojans) with structural factors such as competitive race and lack of governance.
- Frameworks such as the NIST AI RMF, OWASP Top 10 for LLM, and MITRE ATLAS help structure adversarial controls and testing on AI systems in production.
- Inventorying AI assets, limiting access, monitoring usage, and adopting AISPM are key steps to leveraging AI while minimizing its attack surface.
Artificial intelligence has permeated everything : from how we work and communicate to how financial, medical, and military decisions are made. This massive presence has a very positive side, but also a dark side: if these systems fail, are manipulated, or are used maliciously, the damage can be enormous, both for companies and for society as a whole.
In recent years, public concern about the risks of AI has grown in tandem with its adoption. Recent surveys show that a majority of people feel more unease than enthusiasm about its expansion, and that many citizens consider it plausible that AI could trigger catastrophic events if it gets out of control. At the same time, purely technical research remains focused primarily on making AI more powerful, with only a relatively small fraction dedicated to making it safer and more robust.
What do we mean by AI security and why does it matter so much?
When we talk about artificial intelligence security, we are referring to the set of methods, technical principles, cybersecurity controls , and governance frameworks designed to ensure that AI systems are robust, predictable, resilient to attacks, and aligned with human objectives . This includes both traditional protection (preventing cyberattacks, data leaks, or model manipulation) and broader issues such as bias, lack of transparency, and potential systemic impacts.
For businesses and government agencies, AI security is crucial because more and more critical decisions are being delegated to models : financial risk assessment, medical prioritization, resume ranking, big data analysis, process automation, and threat detection. A serious failure, a backdoor, or malicious use can lead to millions in losses, regulatory sanctions, reputational damage, and, in the worst-case scenario, threats to national security.
On a social level, AI can amplify inequalities and abuses if not properly controlled. Opaque systems or those trained with biased data can systematically discriminate against certain groups, reinforce mass surveillance, or facilitate large-scale disinformation campaigns. And, looking further ahead, the possibility of reaching forms of artificial general intelligence (AGI) or even superintelligence (ASI) has opened a debate about existential risks if these systems do not align with human values and escape effective control.
Experts disagree on which risks are most urgent or likely , but there is some consensus that the probability of very bad outcomes is not negligible. Surveys of AI researchers have estimated around a 5% probability of extremely negative scenarios (for example, civilizational collapse) stemming from advanced AI. Some consider these concerns exaggerated, while others believe that precisely because they are so serious, it is wise to anticipate them and not rely on "fixing it as we go."
From early warnings to the modern AI security agenda
Although it may seem like a recent topic, the risks of intelligent automation have been discussed since the beginnings of computing. As early as the mid-20th century, figures like Norbert Wiener warned that, to the extent we gave autonomy to machines capable of learning from experience, we also assumed the risk that they would behave contrary to our wishes.
Over time, these concerns became more concrete. In the late 2000s, professional associations like the AAAI promoted studies on the long-term social impact of AI. The general message was skeptical of the more catastrophic arguments of science fiction, but at the same time acknowledged the need for better methods to verify complex systems and minimize unexpected results.
In the 2010s, the debate intensified. Influential books on superintelligence and existential risks spurred the creation of new research centers dedicated to “beneficial and human-friendly” AI, and philanthropic initiatives emerged that funded specific projects on technical safety, social impact, and alignment of goals. At the same time, public figures such as Elon Musk, Bill Gates, and Stephen Hawking began to speak openly about serious risks if AI advanced without safeguards.
International conferences were organized, research agendas on “specific AI security problems” were published, and principles such as those of Asilomar were formulated, which included explicit recommendations: cooperation between teams to avoid lowering security standards, impact analysis, transparency where possible, etc. This line of work consolidated its own field: AI security engineering , with the goal of anticipating failures and designing the most secure systems possible before their mass deployment.
In recent years, governments and regulators have joined the effort . Workshops organized by the White House, programs from US agencies (DARPA, IARPA, NSF), the UK's National AI Strategy, and China's ethical guidelines reflect that AI safety is no longer just an academic topic, but a matter of public policy. In Europe, the AI Act introduces clear restrictions on certain uses (for example, mass facial recognition or subliminal manipulation) and establishes additional obligations for "high-risk" applications.
Key technical areas: robustness, monitoring, and alignment
AI security research is typically grouped into three main areas: robustness, monitoring, and alignment . Each addresses a different type of risk, but in practice they are closely interconnected.
Robustness focuses on AI systems behaving reasonably across a wide range of situations, including those not well-represented in the training data. This involves dealing with "black swans" (rare but highly damaging events) and attacks where someone deliberately manipulates inputs to force errors.
A classic example of a lack of robustness in the face of rare events is the 2010 Flash Crash , where automated trading systems reacted erratically to market anomalies and wiped out nearly a trillion dollars in minutes. In the physical world, autonomous vehicles continue to struggle with "corner cases": unusual traffic signals, extreme weather conditions, or combinations of objects that rarely appear in real-world data.
In parallel, many deep learning models have proven extremely vulnerable to adversarial examples : tiny perturbations in images, audio, or text, almost invisible to the human eye, that reliably cause classification errors. This affects vision systems, speech recognition, and language models. To make matters worse, it's not just a matter of deceiving the "main" model: the very reward or evaluation models used to train others can be exploited to artificially inflate scores without actually improving performance on the intended task.
Monitoring addresses how to detect early on when an AI system is failing, under attack, or being misused. It includes confidence calibration (ensuring the probabilities the model outputs accurately reflect its reliability), out-of-distribution detection (when the system encounters something "too different" from what it has seen before), and monitoring for potentially malicious uses.
Finally, alignment ensures that the most advanced systems, especially those making high-impact decisions or capable of autonomous action, pursue objectives we deem acceptable and do not develop dangerous strategies to achieve them. This involves concepts such as instrumental convergence (the tendency of advanced agents to seek similar intermediate goals) and the risks of loss of control when high capacity is combined with broad autonomy.
Transparency, interpretability, and Trojan horses in models
One of the biggest challenges is that many modern AI systems, especially large neural networks, function as true black boxes . We know what goes into them and what comes out, but understanding why they made a particular decision, or what internal patterns they are actually using, is far from trivial.
This lack of transparency clashes head-on with legal and ethical requirements for explainability in areas such as credit, personnel selection, and the justice system, where it is mandatory to justify why a particular person is accepted or rejected. It also makes it difficult to investigate serious failures: in incidents such as fatal collisions with autonomous vehicles, it is sometimes very difficult to reconstruct what combination of internal signals led to the error in perception or decision.
This is why an entire subfield of interpretability and transparency techniques has developed . Some focus on showing which parts of the input the models "look at" (for example, which pixels or words have been most influential on the output), while others go deeper, analyzing internal representations and connections between neurons. In models like CLIP, neurons have been identified that respond to both the word "spider" and images of Spiderman, and in language transformers, attention patterns and recurrent circuits that appear to encode syntactic rules or associations of facts are studied.
In addition to aiding in understanding and debugging, these techniques allow for targeted model editing . It has been possible to locate parameters related to a specific event (for example, the city where a famous monument is located) and modify them to systematically change the responses. Although these experiments demonstrate that errors can also be induced, the same approach could be applied to correct massive errors without needing to retrain models from scratch.
Another sensitive issue is Trojans or backdoors in AI models . This refers to situations where, during training, someone introduces a specific pattern that, when it appears at the input, triggers a different behavior than usual. For example, a facial recognition system that only grants access if the person is wearing a certain accessory, or a self-driving car that behaves recklessly if it detects a specific sticker on a sign.
The worrying aspect is that altering even a tiny fraction of the training data is enough to insert a working Trojan. Since many models are trained on large volumes of public data, it's realistic to consider scenarios where an attacker manages to contaminate part of the corpus. These Trojans, besides being a vulnerability in themselves, serve as an ideal testing ground for developing better tools for auditing and detecting covert malicious behavior.
Structural risks, competition and global governance
Beyond specific attacks or isolated incidents, there are systemic risks related to how the AI ecosystem is organized: competitive pressure between companies and countries, the lack of a security culture, economic incentives to release products as quickly as possible, and the asymmetry between cyber attackers and defenders.
Several analysts have pointed out that focusing solely on “technical failures” or “individual misuses” falls short, because the causal chains leading to disaster are much longer . The geopolitical dynamics during the Cold War—where decisions by a few individuals could bring a nuclear conflict closer to or further away from it—are often invoked as an analogy for what could happen with a chaotic race to develop increasingly powerful AI systems.
A recurring fear is the so-called “race to the abyss” : if private or state actors perceive significant advantages in being first (economic, military, or in terms of prestige), they may feel compelled to accelerate and cut back on security controls, even if everyone acknowledges that slower progress would be preferable. Avoiding this dynamic requires mechanisms for cooperation, mutual transparency, agreements on minimum standards, and, likely, new international institutions.
At the same time, many of today's global challenges are problems of cooperation : climate change, nuclear risks, cyber weapons, and so on. AI can worsen these challenges (for example, by facilitating more powerful cyberattacks or highly sophisticated political manipulation campaigns), but it can also help by providing better tools for forecasting, scenario analysis, and collective decision-making.
Hence the consolidation of the field of AI governance , which seeks to articulate regulatory frameworks, technical standards, best business practices, and accountability mechanisms. It ranges from foundational works comparing the potential impact of AI with transformative technologies of the past (such as electricity or the steam engine) to very concrete proposals: external audits , bug bounties, public incident databases, criteria for deciding when it is responsible to publish results or models, and enhanced security requirements for leading laboratories.
The role of cybersecurity: AI to defend and protect AI itself
In the practical, day-to-day realm, AI is already a central player in cybersecurity , and the relationship goes both ways: we use AI to better defend ourselves, and at the same time we have to protect the AI systems themselves from attacks.
In recent decades, cyber threats have evolved from seeking visible disruptions to stealing data, infiltrating systems for months, and deploying highly stealthy malware . To combat them, solutions like SIEM, which collects and analyzes large volumes of logs, and EDR, which continuously monitors what happens on every corporate device, have become popular. The problem is that the amount of data far exceeds what a human team can review manually.
This is where machine learning comes in, capable of detecting patterns and anomalies in real time from huge sets of signals: unusual login attempts, unauthorized access to sensitive data, strange process behavior, or connections to suspicious domains.
AI platforms in security are also used to automate part of the response : correlating related alerts, reconstructing the "narrative" of an attack, prioritizing which incidents to review first, and, in some cases, directly executing containment actions (isolating a machine, cutting a connection, revoking compromised credentials). This reduces noise, helps analysts focus on what matters, and dramatically shortens the time from when a problem is detected until effective action is taken.
In parallel, the same advancements are being applied to the protection of networks and endpoints . Next-generation firewalls integrate models that analyze traffic for malicious behavior, update internal rules on the fly, or reconfigure network segmentation to limit attackers' lateral movement. AI-enhanced EDRs detect anomalous process behavior (such as unexpected access to sensitive files or execution chains typical of ransomware) before the damage becomes irreversible.
Added to all this is the rise of natural language-based assistants integrated into cybersecurity tools. These AI copilots for security teams allow them to ask complex questions (“What attack paths exist to this database?”) and receive contextualized explanations, incident summaries, or mitigation proposals without having to manually navigate through multiple dashboards.
How attackers are using AI
Unfortunately, cybercriminals have also jumped headfirst into using AI . In underground forums, every powerful new model that appears is analyzed, and modified versions have emerged, some explicitly geared towards criminal activities, stripped of ethical filters and trained as "hacking assistants."
Today it is relatively easy to take advantage of generative models to automate almost all phases of a criminal operation : mass writing of flawless and personalized phishing emails, creation of very credible fake login pages, generation of ransomware scripts or kits to steal credentials , elaboration of persuasive messages for social engineering, etc.
Another worrying front is AI's ability to create extremely convincing fake content . Text generation tools allow the production of fake articles and news stories that mimic the style of professional media; video generation systems like deepfakes replace faces in high-quality clips; and voice cloning makes it possible to recreate a person's voice with just a few minutes of audio.
Cases of telephone fraud using cloned voices of family members to urge money transfers have already been documented , as well as political smear campaigns and social media manipulation that combine doctored videos, fake audio, and generative bots to amplify messages. The cost of these tools decreases every year, increasing the return on investment for attackers.
In short, AI multiplies offensive capabilities: it makes sophisticated attacks easier and cheaper, allows for large-scale campaigns, and complicates the task of distinguishing authentic from manipulated information . Therefore, using AI defensively is no longer optional: without automation and intelligence, human teams simply cannot keep up.
Key AI-specific security risks
Beyond general cybersecurity risks, deploying AI systems involves its own vulnerabilities that must be understood and managed. Some of the most relevant are:
Data and model poisoning. Introducing malicious or manipulated data into training sets can distort model behavior, insert backdoors, or degrade performance without being evident in standard metrics. In federated learning environments, where multiple actors contribute to training, detecting which participant has intentionally introduced noise is especially challenging.
Adversarial examples and evasion. Designing slightly modified inputs to trigger systematic errors allows attackers to bypass malware detection systems, spam filters, image classifiers, or biometric authentication mechanisms. Because the model "believes" the input is normal, these types of attacks are difficult to mitigate using only traditional rules.
Model inversion and privacy leaks. Through repeated queries and output analysis, an attacker can reconstruct information about the training data, even inferring whether a specific record was included or extracting fragments of sensitive data stored by the model. This poses a serious problem when training with personal, clinical, or confidential information.
Theft of models and intellectual property. By massively querying an AI API, it's possible to train a copycat model that closely replicates the behavior of the original. This undermines the business model of those offering AI as a service and, moreover, creates a copy over which the attacker has complete control, including the ability to exploit vulnerabilities without restriction.
Attacks on APIs and infrastructure. Interfaces that connect AI models with other applications are a very tempting entry point: weak authentication, lack of input validation, absence of frequency limits or monitoring can allow anything from denials of service to massive data extraction or remote code execution.
Prompt injections and abuses in GenAI. In the case of language models and generative assistants, an emerging vector is that of direct or indirect instruction injections . An attacker can embed malicious commands in the input text itself (“ignore the previous rules and…”) or in external documents that the model will read, with the aim of bypassing constraints, exfiltrating data, or executing unintended actions.
Exploited hallucinations. Generative models can fabricate data with complete aplomb. If these systems are used without human oversight in critical contexts, or if attackers manage to legitimize certain hallucinations as “facts,” it opens the door to erroneous decisions, disinformation, and a widespread loss of trust in digital information.
Frameworks, standards, and best practices for safer AI
To avoid going in blind, various organizations have proposed specific frameworks for AI security . While none covers the entire landscape on its own, combined they offer a solid foundation on which to build a strategy.
Among them, the NIST AI Risk Management Framework stands out , structuring the work around four functions: govern (establishing policies and responsibilities), map (understanding the context and risks), measure (evaluating performance and exposure), and manage (prioritizing and applying controls). Other efforts, such as the MITRE ATLAS matrices, catalog specific attack tactics and techniques against AI systems, similar to what ATT&CK does with traditional threats.
In the more applied field, the security community has developed lists such as the OWASP Top 10 for LLM , which compiles the main risks in applications based on large language models: prompt injections, data exposures, vulnerabilities in the model supply chain, model theft, etc. Major vendors have published their own frameworks (such as Google's SAIF) with concrete steps for deploying AI systems more securely.
Furthermore, highly specific, practical approaches are emerging for deploying GenAI in cloud environments, such as tenant and user isolation frameworks that carefully segment data, permissions, and communication channels, reducing the impact of potential failures or injections. The idea is to combine traditional cybersecurity controls (encryption, identity management, network segmentation, sandboxing) with measures tailored to the specific characteristics of AI.
Based on this, the most frequently repeated recommendations include ensuring the quality and diversity of training data, directly addressing ethical implications (bias, consent, privacy), conducting frequent model testing and updates, and defining AI-specific security policies that complement general ICT policies.
In parallel, a category of solutions known as AI Security Posture Management (AISPM) is emerging , equivalent to what CSPM meant for the cloud: inventorying AI assets, discovering unauthorized uses (“shadow AI”), evaluating configurations, and continuously monitoring policy compliance.
Everything suggests that, as AI becomes even more integrated into critical infrastructure, AI security will shift from an "interesting extra" to a fundamental requirement . Organizations that get ahead of the curve—by inventorying their AI systems, defining clear governance, implementing strict access controls, subjecting their models to adversarial testing, and aligning themselves with recognized frameworks—will be far better positioned to leverage the benefits of AI without encountering unpleasant surprises. And, above all, they will be better equipped to maintain the trust of customers, users, and citizens in an environment where distinguishing between what is real and what is manipulated will become increasingly complex.
