Short version: security failures have an adversary. Safety failures don’t. If you can’t tell which one you have, you’ll pick the wrong fix.
People use “AI security” and “AI safety” as if they mean the same thing. They don’t, and mixing them up leads teams to defend against the wrong problem.
Security means protecting a system from someone who is deliberately trying to break it. Safety means making sure the system behaves as intended and doesn’t cause harm, whether or not anyone is attacking it. The tests, the mitigations and the threat models all differ.
Two failures, same outcome
Take a self-driving car approaching a pedestrian in dense fog. Say it doesn’t brake, because the model never saw enough foggy footage in training. This is a hypothetical, but it’s a realistic one. Nobody attacked it. That’s a safety failure, and the fix is mostly engineering: more training data covering edge cases, better validation before release, and a human who can step in.
Now say someone puts black and white stickers on a stop sign. Researchers did this on a real sign, and the classifier they targeted read it as a 45 mph speed limit sign, in 100% of lab images and 84.8% of frames from a moving car (Eykholt et al.). The car doesn’t stop. The result is the same as in the fog case, but this is a security failure, and more training data won’t fix it. You need adversarial testing, where you try to break your own model, plus input checks, monitoring in production, control over who can feed data into the system, and an incident response plan. NIST’s adversarial machine learning taxonomy (AI 100-2) is a good map of the attacks and mitigations.
Where they overlap
Language models blur the line. Say a chatbot produces offensive text or states something false. If the model was just poorly tuned, that’s a safety problem. If someone spent an afternoon jailbreaking it to get that output, it’s a security problem. Often it’s both: the weakness is a safety gap, and the person using it is an attacker. OWASP treats jailbreaking as a form of prompt injection, which is a security issue, but the thing being bypassed is the model’s safety training.
Why the mix-up costs you
Suppose a team sees an adversarial example and treats it as a safety bug. They collect more data and improve generalization. That helps a little, but an attacker will just find the next input that works. An attacker who knows your defenses will work around them, so a fixed set of test cases can’t prove you’re safe from one.
Treat the same problem as a security issue and you do different work. You validate inputs, watch for suspicious patterns, and plan for the model being fooled by designing the system to fail safely when it is. One approach makes the system more robust. The other makes it feel more robust.
A rule of thumb
When something goes wrong, ask whether someone was trying to make it go wrong. If yes, start with security work: adversarial testing, input validation, access controls, monitoring and incident response. If no, start with safety work: better data, edge case coverage, validation before release and human oversight. If you can’t tell, I’d lean toward the security response and add safety work on top, since it’s the one that assumes an opponent.
It’s a rough guide, not a rule. OWASP’s own prompt injection guidance notes that harmful inputs can be accidental, so you’ll sometimes need both kinds of fix.
Reputation risk sits in between
A model that produces racist output may not be under attack, but the damage to your reputation is the same either way. People disagree on how to label it. Some call it a safety problem because the model needs better training. Others call it a security problem because someone will go looking for it. In practice you need both: train the model well, watch for people probing it, and have a plan for when it happens.
What this means for your team
The distinction affects budget, hiring, test plans and incident response. If your threat model only covers safety, real attacks will catch you off guard. If it only covers security, you’ll ship systems that break on ordinary edge cases.