What Is Data Poisoning in Machine Learning?

Machine learning models learn from data. That statement is simple enough to sound harmless — but it hides a vulnerability that ML practitioners are still coming to terms with. If an attacker can influence the data a model trains on, they can influence what the model does, often in ways that are invisible until the moment they matter. That practice has a name: data poisoning.

Data poisoning is one of the most consequential attack categories in machine learning security. It is also one of the hardest to detect. A well-executed poisoning attack does not produce visible errors on ordinary benchmarks. The model looks fine — until it encounters the specific input the attacker planted a backdoor for, at which point it produces exactly the output the attacker wants. For the full landscape, see our complete guide to AI security.

This guide explains what data poisoning is, the major variants, how poisoned data enters real ML pipelines, and the defenses that reduce risk. Use the links throughout for deeper context.

What Is Data Poisoning?

Data poisoning is an attack against a machine learning system in which the attacker introduces malicious examples into the model's training data. The goal is to change how the trained model behaves — either broadly, by degrading its accuracy, or narrowly, by planting attacker-controlled behavior on specific inputs.

Data poisoning is a training-time attack. Unlike evasion attacks, which target a deployed model with crafted inputs, poisoning attacks the model before it is deployed at all. By the time the model is running in production, the damage is baked in. Those evasion attacks are covered in our guide to adversarial machine learning.

Availability vs Integrity Attacks

Poisoning attacks are conventionally divided into two categories based on what the attacker is trying to do.

Availability attacks, sometimes called untargeted poisoning, aim to degrade the model's overall accuracy. Enough bad examples in the training set can make a model unreliable across the board. Availability attacks are the closest analogue to a denial-of-service attack against ML — they make the model useless without needing to touch it at runtime.

Integrity attacks, also called targeted poisoning, aim to change how the model behaves on specific inputs while leaving overall accuracy intact. These are stealthier: the model passes normal evaluation but fails on the exact cases the attacker cares about.

Integrity attacks are the more dangerous of the two in practice, because they are much harder to detect. A model with degraded overall accuracy raises red flags; a model that passes all its benchmarks but fails on one attacker-chosen input probably does not.

Backdoor Attacks

The most alarming form of integrity poisoning is the backdoor attack. The attacker plants a hidden trigger — often a specific visual pattern, a phrase, or a byte sequence — during training. The trained model behaves normally on ordinary inputs. It only misbehaves when the trigger is present, at which point it produces exactly the output the attacker wants.

Research demonstrations have planted backdoors that make a stop-sign classifier see a stop sign as a speed limit sign whenever a small sticker is present, that make a language model produce specific output whenever a certain phrase appears, and that make a code assistant emit vulnerable code when the input contains an innocuous-looking comment.

Backdoors are dangerous because standard evaluation metrics miss them almost entirely. The model looks correct on any input the attacker did not specifically target.

How Poisoned Data Enters a Pipeline

Poisoning attacks require the attacker to influence the training data. That sounds hard, but modern data pipelines create many opportunities.

• Public data collection. Models trained on scraped web data — including many large language and image models — ingest content the attacker can influence simply by publishing it.
• Crowd-sourced labeling. Training pipelines that use crowd workers to label data introduce a supply of untrusted inputs.
• User-generated feedback loops. Systems that fine-tune on user interactions can be poisoned by users who intentionally provide adversarial inputs and ratings.
• Third-party datasets. Models trained on datasets from external sources inherit whatever risk exists in that source.
• Insider access. Anyone with legitimate access to the training pipeline is in a position to introduce poisoned examples.
• Compromised infrastructure. An attacker who compromises the data storage or ingestion systems can insert examples directly.

Real-World Concerns

Poisoning attacks against foundation models are especially consequential because those models are used across many downstream applications. A single poisoned pretraining dataset can carry a backdoor into every model built on top of it.

Similar risks apply to code models. A code assistant that has been poisoned to produce vulnerable output on specific triggers is a supply-chain risk that could ship into many production applications without anyone noticing. It is a modern echo of the supply chain cyberattack pattern, applied to ML.

Defenses Against Data Poisoning

Defense against poisoning is genuinely hard, but layered controls raise the cost of attack significantly.

• Track data provenance. Know where every example in the training set came from, when it was added, and by whom. Provenance is the foundation of everything else.
• Curate and validate sources. Prefer trusted, signed sources for training data. Apply careful validation to community-contributed or scraped data.
• Detect outliers. Statistical anomaly detection during training can flag examples that behave unusually — a partial defense against integrity attacks.
• Apply robust training techniques. Methods such as differential privacy, robust aggregation, and certified robustness reduce the impact of small numbers of poisoned examples.
• Test for backdoors. Adversarial evaluation should include trigger discovery — attempts to find inputs that produce unexpected outputs. This is genuinely difficult and an active research area.
• Isolate high-risk pipelines. Fine-tuning on user feedback should require careful gating, rate limiting, and human review — not open loops.
• Red team the data pipeline. Treat the data supply chain as a security-critical system, subject to the same review as any other production system.

Conclusion

Data poisoning is the most fundamental attack category in machine learning security because it exploits the very thing that makes ML work: learning from data. It cannot be fixed by patching the model. It has to be addressed at the pipeline level — by controlling what the model learns from and by continuously testing for signs that something has gone wrong.

Organizations that treat training data as an ordinary asset will be caught by poisoning attacks eventually. The ones that treat it as security-critical — with provenance, validation, and adversarial testing — will be much harder to compromise.

Frequently Asked Questions (FAQ)

What is data poisoning?

Data poisoning is an attack against a machine learning system in which the attacker introduces malicious examples into the training data. The goal is to change how the trained model behaves, either broadly or on specific inputs.

What is the difference between availability and integrity poisoning?

Availability poisoning degrades the model's overall accuracy. Integrity poisoning leaves overall accuracy intact but changes how the model behaves on specific inputs — often much stealthier and harder to detect.

What is a backdoor attack?

A backdoor attack is a targeted poisoning attack that plants a hidden trigger during training. The model behaves normally on ordinary inputs and only misbehaves when the trigger appears, producing attacker-controlled output.

How does poisoned data enter a training pipeline?

Through public data collection, crowd-sourced labeling, user feedback loops, third-party datasets, insider access, or compromised infrastructure. Modern data pipelines create many opportunities for adversarial contribution.

Can data poisoning be detected?

Broad availability attacks often show up as degraded benchmark performance. Targeted integrity attacks and backdoors are much harder to detect because standard evaluation misses them entirely. Provenance tracking and adversarial testing help.

How do defenders reduce poisoning risk?

By tracking data provenance, curating trusted sources, applying outlier detection and robust training techniques, testing for backdoors, and treating the data pipeline as a security-critical system.