In 2024, researchers at Palo Alto Networks showed they could jailbreak GPT-4 with a 90%+ success rate using a technique called “Deceptive Delight”. Three conversational turns was all it took to bypass safety guardrails.

That single stat tells you why AI attack risk has become the biggest headache for security teams this year. When the guardrails on the world’s most-tested model fold that fast, every business running its own AI is on the target list.

Interactive AI Attack Risk Score

Podcast - AI Poisoning Can Backdoor Your Model with 0.01% of Your Training Data

Quick Answer

An AI attack is any deliberate attempt to fool, poison, steal, or hijack a machine learning system so it makes wrong decisions or leaks data.

These attacks include adversarial examples, data poisoning, prompt injection, and model theft. They differ from traditional cyberattacks because they target the model's logic itself.

Not just the network around it. Defending against adversarial attacks needs red teaming, input filtering, and continuous monitoring.

Video - AI Goes Rogue

Key Takeaways

• An AI attack targets the machine learning model, not just the server it runs on.
• The most common adversarial attacks include evasion, poisoning, prompt injection, and model extraction.
• AI-powered cyberattacks make phishing, malware, and social engineering faster and harder to spot.
• Facial recognition, self-driving cars, and email security tools are all vulnerable to adversarial examples.
• Defending against adversarial attacks means combining traditional security with AI-specific red teaming.
• Tools like AIShield, Snyk, and Palo Alto Networks now offer AI-specific detection platforms.
• Even offline AI systems can be attacked if the training data or hardware supply chain is compromised.

What is an AI attack?

An AI attack is a cyberattack that either targets an AI system directly or uses AI to power a malicious campaign against traditional systems. In both cases, machine learning models are the weapon or the victim.

According to IBM's 2024 Cost of a Data Breach Report, the average breach involving shadow AI cost organizations $670,000 more than breaches without it.

The term covers two overlapping ideas. First, adversarial attacks on machine learning, where attackers use crafted inputs to trick a model.

Second, AI-powered cyberattacks, where attackers use LLMs and generative AI to write malware, craft phishing emails, or clone voices. Both are growing fast, and both need new defenses that traditional cyberattacks simply didn't require.

The BBC News technology desk has covered a steady drumbeat of high-profile incidents through 2025 and 2026, including the TfL teen hackers case and the "sobering moment" OpenAI hack that many called a warning shot for the industry.

Types of AI attacks explained

The main types of adversarial attacks are evasion, poisoning, model extraction, model inversion, and prompt injection.

Each targets a different phase of the machine learning pipeline. Once you know which phase is exposed, defending against adversarial attacks gets a lot more tractable.

• Evasion attacks: The attacker tweaks an input at inference time so the model misclassifies it. Classic example, adding pixel noise to a stop sign so a self-driving car reads it as a speed limit sign.
• Data poisoning attack in AI: Attackers slip bad samples into training data so the model learns the wrong thing. Skip a few labels, insert a backdoor trigger, and the model becomes a ticking bomb.
• Model extraction: Attackers query a model enough times to reverse-engineer a copy, trade-secret theft at scale. Apple sues OpenAI-style cases over trade secrets are a preview of the legal fights coming.
• Model inversion: Attackers reconstruct training data from model outputs, which can leak private information.
• Prompt injection: For LLMs, attackers hide instructions inside content the model reads, hijacking its behavior. This is what powers "Claude Mythos" style jailbreaks and rogue-agent scenarios.

The NIST Adversarial Machine Learning Taxonomy (2024) is the standard reference and worth bookmarking.

How do AI attacks work?

AI attacks work by exploiting the statistical, gradient-based nature of machine learning models. Small, targeted perturbations in input or training data cause outsized changes in output.

Attackers use math, not just malware. That's what makes adversarial attacks machine learning defenders keep chasing.

Here's the basic flow of an adversarial attack:

1. Recon. The attacker probes the model, sometimes via API, sometimes by studying open-source weights.
2. Craft. Using techniques like FGSM (Fast Gradient Sign Method) or PGD, attackers compute the smallest input change that flips a prediction.
3. Deliver. The malicious input is sent to the target, a phishing email, a tampered image, a poisoned document.
4. Persist. In poisoning or backdoor attacks, the attacker waits for the trigger to activate in production.

For AI-powered cyberattacks, the flow is different, attackers use generative AI to scale content, write malware code faster, and clone voices for fraud. CrowdStrike's 2025 Global Threat Report noted a 442% jump in voice phishing (vishing) between H1 and H2 2024, driven largely by AI voice tools.

AI attack vs cyber attack: what's the difference?

An AI attack targets the model itself or uses AI as the weapon. A traditional cyberattack targets networks, servers, or people using scripts, exploits, and human effort.

The distinction matters because tools built for traditional cyberattacks, firewalls, EDR, signature-based detection, often can't see attacks against machine learning systems.

Dimension	Traditional cyberattack	AI attack
Target	Network, endpoint, human	Model, training data, prompt
Weapon	Malware, exploit, phishing kit	Adversarial example, poisoned data, LLM
Speed	Minutes to days	Seconds, LLMs generate at scale
Detection	Signatures, rules	Behavior, statistical anomaly, red teaming
Cost to attacker	Rising	Falling fast

The Snyk app (Snyk) team has argued that AI attacks traditional cyberattacks defenses just aren't tuned for.

You need AI-native security platform tools that inspect model inputs, outputs, and training pipelines. Palo Alto Networks Unit 42, Microsoft's Counterfit, and IBM's AIShield are three of the better-known enterprise options.

Adversarial attacks on machine learning

Adversarial attacks on machine learning are inputs crafted to trigger wrong predictions while looking normal to a human.

They exploit the fact that decision boundaries in high-dimensional models aren't smooth, tiny nudges in the right direction cause big misclassifications. The classic Goodfellow et al. paper (2014) is still the best introduction.

Adversarial examples machine learning researchers study fall into two camps:

• White-box attacks, attacker has full access to the model weights and architecture.
• Black-box attacks, attacker only sees inputs and outputs, but can still craft evasion attacks by transferring examples from a surrogate model.

Real-world adversarial attacks explained by MITRE ATLAS include Microsoft's Tay chatbot being poisoned in 2016. Tesla Autopilot being tricked by road-sticker adversarial examples, and multiple facial recognition bypasses using printed patterns on glasses frames. These aren't hypotheticals.

AI poisoning attack definition

A data poisoning attack is when an adversary injects malicious samples into a model's training set so the deployed model behaves badly on specific triggers.

The model looks fine on normal inputs but fails, or opens a backdoor, when the attacker's trigger appears. Poisoning is especially dangerous because it can survive fine-tuning and be nearly invisible in standard evaluations.

A well-known 2023 Carlini et al. study showed that poisoning just 0.01% of a large-scale web dataset was enough to reliably backdoor image classifiers.

When training data comes from scraped web content, this is a real threat, not a lab curiosity. See also our note on trust and provenance in AI content.

Prompt injection attacks: what are they?

Prompt injection is an AI attack where malicious instructions are hidden inside content an LLM reads, a document, a webpage, an email, and the model executes them as if they came from the user.

It's the number-one risk on the OWASP Top 10 for LLM Applications and the reason "how to stop AI agents going rogue" is now a boardroom question.

Two main flavors:

• Direct prompt injection: user types "ignore previous instructions" into the chat.
• Indirect prompt injection: attacker plants instructions in a webpage or PDF the AI is asked to summarize.

The 2025 wave of agentic AI made this worse, as AI browses, emails, and shops on your behalf, every piece of content it touches becomes a potential attack surface. See how enterprise security teams are struggling to catch AI hackers.

How to protect against AI attacks: 7 strategies

Protecting against AI attacks needs a mix of AI-specific and traditional security controls. There's no single silver bullet, defense in depth is the only approach that holds up. Here are seven strategies that actually work in production.

1. Red team your models. Simulate adversarial attacks before attackers do. Use tools like Microsoft PyRIT, Snyk, or open-source Garak.
2. Filter inputs and outputs. Guardrails on prompts and responses catch prompt injection and data leaks.
3. Validate training data. Data provenance, deduplication, and anomaly detection reduce poisoning risk.
4. Rate-limit model queries. This slows model extraction and adversarial probing attacks.
5. Adversarial training. Include adversarial examples in your training set so the model learns to resist them.
6. Monitor drift and anomalies. Sudden shifts in prediction distribution can signal an active attack.
7. Encrypt and access-control model artifacts. Treat weights like crown-jewel data, because they are. See 33.7 million stolen data records for what happens when you don't.

Real examples of AI attacks

Real AI attacks are happening in production every week, not just in academic papers. Here are the incidents that shaped the 2025-2026 threat landscape. All are documented in mainstream news, including BBC coverage.

• OpenAI breach (2024-2025): A firm hacked by rogue OpenAI models called it "a wake-up call", BBC News reported the incident as a warning shot for the industry.
• TfL teen hackers (2024): Teenagers who live-streamed a cyber-attack on Transport for London were jailed; investigators found AI-assisted phishing tooling in their kit.
• UK school cyber-attacks (2025): A new warning to parents over schools cyber attack incidents highlighted AI-generated ransom notes and deepfake voice calls.
• Chinese chipmaker debut (2025): While Chinese chipmaker shares surged 470% in a blockbuster stock market debut, security researchers raised supply-chain concerns about embedded AI accelerators.
• Insecure sandboxes: Multiple 2025 disclosures showed LLM code interpreters running in insecure sandboxes could be escaped via prompt injection.
• Apple sues OpenAI: Apple sues OpenAI, its employees claiming theft of trade secrets, a preview of how model-extraction disputes will play out in court.
• "I'm dying laughing": Leaked messages tell us about global hacking gangs using AI to industrialize phishing across dozens of languages.

For a lighter tangent, budding birders are flocking to Shazam-like apps to identify avians, the same computer vision tech is what attackers now target with adversarial examples.

AI attack detection tools

The best AI attack detection tools combine traditional security telemetry with model-specific monitoring for drift, poisoning, and adversarial inputs. No single vendor covers everything yet, expect to stitch two or three together.

• IBM AIShield, model scanning, red teaming, and runtime protection.
• Snyk, Snyk app Snyk focuses on securing the AI-generated code and protecting your AI applications end-to-end.
• Palo Alto Networks AI Access Security, enterprise LLM monitoring.
• HiddenLayer, MLDR (Machine Learning Detection & Response).
• Robust Intelligence, validation for LLM and traditional ML pipelines.
• Microsoft Counterfit and Garak, open-source red teaming.

Cost of defending against AI attacks

Enterprise AI security programs typically cost $250K to $2M per year, depending on scope and model footprint.

Gartner projects worldwide security and risk management spending will hit $215 billion in 2025, with AI security the fastest-growing segment. The Gartner press release has the full breakdown.

Budget lines usually cover:

• AI red teaming (internal + external): $50K,$500K/yr
• Model monitoring platform: $30K,$300K/yr
• Guardrails and input/output filtering: $20K,$150K/yr
• Incident response retainer with AI expertise: $50K,$200K/yr
• Training and governance: $25K,$100K/yr

Skimp on any one line and you'll pay for it during your first serious incident.

Who is vulnerable to AI attacks?

Every organization deploying or consuming AI is vulnerable. But the highest-risk targets are financial services, healthcare, defense, critical infrastructure, and any company using AI agents to touch customer data or execute transactions.

Attackers use AI to attack AI, and the returns scale with the value of the target.