If you can’t say where a model file came from, which exact revision you loaded, and who approved it, treat it like an unsigned binary you downloaded from the internet. That’s what it is.
Traditional software has one supply chain to defend, and it’s code. You write it, pull in dependencies like React or Django, build it, ship it. Security teams scan for vulnerable packages, pin versions, and watch for typosquatting, where an attacker registers something like reqeusts and waits for someone to mistype requests.
AI systems have a second supply chain. Most teams don’t treat it as one.
The first supply chain: code and frameworks
This part is familiar. You install PyTorch or TensorFlow, NumPy for the maths, scikit-learn for preprocessing, CUDA drivers for the GPUs, and run it all on AWS, GCP or Azure. Every one of those is a dependency that can be attacked like any other package.
ML tooling has already been hit. Between 25 and 30 December 2022, anyone on Linux who installed PyTorch nightly with pip could have pulled a malicious package called torchtriton from PyPI. The real torchtriton lived on PyTorch’s own index, but pip gave PyPI precedence, so an attacker registered the same name there. The binary collected SSH keys, .gitconfig, environment variables and up to 1,000 files from the home directory, then sent them out over DNS queries. (PyTorch advisory)
Your existing tools cover most of this ground. Dependency scanners flag known CVEs. Lock files pin versions. pip’s hash-checking mode refuses any package whose hash doesn’t match the one recorded in your requirements file. Be clear about what that last one proves, though. It’s a hash you pinned, not a signature from the maintainer. It tells you that you got the same file as last time. It doesn’t tell you the file was clean the first time.
The second supply chain: data and models
This one usually has very few controls.
You download a dataset from Kaggle or Hugging Face. Someone else assembled it: scraped from the web, released by a research group, or bought from a data broker. You train on it.
Or you skip training and grab a pre-trained model, like BERT, ResNet, or one of the open-weight LLMs on Hugging Face. You fine-tune it on your own data and deploy it.
The dataset and the model are both third-party inputs. In most shops, neither goes through the review a new PyPI package would get. There are four distinct ways this goes wrong.
Poisoned training data
An attacker gets bad examples into a dataset so the model learns the wrong thing. With web-scale datasets that sounds impractical. It isn’t. In 2023, Nicholas Carlini and co-authors showed they could have poisoned 0.01% of LAION-400M or COYO-700M for about $60. A big part of the trick was buying expired domains that the datasets still pointed to, then serving different content from them. (Carlini et al., 2023)
Backdoored models
A backdoored model behaves normally until it sees a specific trigger. The early demonstration was BadNets in 2017: a street-sign classifier that read a stop sign as a speed-limit sign when a small sticker was on it. The backdoor survived even after the model was retrained for a different task. (Gu, Dolan-Gavitt and Garg)
LLMs aren’t immune. In July 2023, Mithril Security edited GPT-J-6B so it would claim Yuri Gagarin was the first man on the moon, then uploaded it under the name “EleuterAI”, one letter off from EleutherAI. It was a research demo, and Hugging Face disabled the repo after disclosure. (Mithril Security)
Model files that run code
This is the most practical attack and the one people miss most often. PyTorch’s traditional checkpoint format uses Python’s pickle, and loading a pickle can execute arbitrary code. In February 2024, JFrog reported around 100 malicious models on Hugging Face. One of them opened a reverse shell as soon as someone loaded it with torch.load(). (JFrog)
Hugging Face scans uploads and lists the imports inside pickle files, but its own docs say the scanner isn’t foolproof. ReversingLabs later found two models that got past Picklescan. They were compressed with 7z instead of ZIP, and the pickle stream was deliberately broken after the malicious part, so the payload ran before the parser hit the error. (ReversingLabs)
Hijacked model names
Hugging Face models are addressed as Author/ModelName. In September 2025, Palo Alto Networks’ Unit 42 showed that when an author deletes their account, someone else can register the same name and publish a different model at the same path. Any pipeline that loads by name picks up the new one. They used this to get code execution through Google Vertex AI and Microsoft Azure AI Foundry, and found thousands of open-source repos with hardcoded model names that could be re-registered. (Unit 42)
Why the gap matters
In a typical organisation, a new code dependency gets scanned, pinned and approved. A new model often gets downloaded by whoever needs it, loaded in a notebook, and promoted to production once the accuracy looks right.
Attackers haven’t given up on code. The torchtriton incident shows that. But the data and model side offers a similar payoff, and far fewer people are looking at it.
A worked example
Here’s a hypothetical. You’re building fraud detection, and you find a model on a public hub that claims to flag fraudulent transactions. It tests well. You ship it.
Three months later, the false-negative rate starts climbing. Now you need answers. Did fraud patterns change? Does the model have a trigger that a specific group of fraudsters knows to use? Is the file in production even the one you tested?
Without records, you can’t tell. There’s no pinned revision, no file hash, no note of who approved it or what it was trained on. You retrain on your own data, the misses go down, and you never find out what happened.
With records, the first checks take minutes. The deployed file’s SHA-256 matches the one logged at approval. It was pinned to a specific commit of the upstream repo. You know who signed off, against which test set, and what the training data looked like. That lets you separate “the world changed”, which is drift, from “the model was built to fail on certain inputs”, which is a backdoor. They need different fixes.
What to write down
For code, you probably already track which library versions are deployed, which known vulnerabilities affect them, and who approved each upgrade.
For data and models, the equivalent list is:
Which datasets the model was trained on, and where each one came from.
Which exact model revision is deployed, by commit hash and file hash.
What format the weights are stored in (pickle or safetensors).
Who approved it for production, and when.
What the training data distribution looked like, so you have a baseline to compare production traffic against.
It looks like paperwork. It’s the same paperwork you already do for code, applied to the inputs that actually decide what the model does.
A minimal setup
Pin the model to a commit and refuse pickle weights:
from transformers import AutoModel
# Pin to a full commit hash you've reviewed, not "main".
# Find it under "Files and versions" on the model page.
model = AutoModel.from_pretrained(
"org-name/model-name",
revision="<full-commit-sha>",
use_safetensors=True, # load safetensors weights, not pickle
)If you have to load a raw PyTorch checkpoint, pass weights_only=True to torch.load(). It’s been the default since PyTorch 2.6, but older code and older installs still exist. (PyTorch announcement)
For your own models, sign them. The OpenSSF model-signing project hit v1.0 in April 2025 and uses Sigstore:
pip install model-signing
model_signing sign ./my-model
model_signing verify ./my-model --signature <signature-file> \
--identity <signer-identity> --identity_provider <oidc-provider>Once a third-party model is approved, copy it into storage you control and load it from there. That’s what Unit 42 recommends, and it removes the dependency on someone else’s account staying alive.
Monitoring
Code monitoring is standard: dependency scanners, vulnerability feeds, lock files.
Data and models need different tooling. Version your datasets with something like DVC or lakeFS so you can see what changed and when. Put models in a registry, such as MLflow’s, with an approval step and the file hash recorded. In production, compare incoming data against the training distribution and alert when it moves.
When production data drifts away from what the model was trained on, accuracy drops, often with no errors in the logs. In fraud detection that means missed fraud. Drift isn’t an attack on its own, but it’s good cover for one. That’s the subject of the next post.
Where to start
Search your codebase for from_pretrained( and torch.load( and count how many calls pin a revision or set weights_only=True. That number tells you where to start.
If your team already has a model approval process, I’d like to hear what it looks like. Reply or leave a comment.