Supply Chain Attacks in AI: the “Poisoned Headwaters” Problem

2.1. Pretrained Models and Weight Files

Pretrained models are parameter sets created as the result of an earlier, large-scale learning process. From a security perspective, weight files cannot be treated as simple passive data. Model parameters encode behavioral patterns derived from training data, the optimization process, and architectural decisions. A compromised model can therefore directly influence the decisions, classification outputs, generated responses, or security boundaries of a downstream application.

Particularly dangerous are models that perform adequately in normal validation environments but exhibit predefined, undesirable behavior on rare or attacker-controlled input patterns. This phenomenon is often discussed in the literature as a model backdoor or trojanized model.

2.2. Datasets and Annotation Chains

Datasets are among the most critical inputs of AI systems. The representations learned by a model derive directly from the statistical and semantic structure of the data. Therefore, data manipulation is not a simple input error, but a structural risk that becomes embedded in the model’s later behavior.

In dataset poisoning, the attacker injects data into the training or fine-tuning set that appears legitimate, but is designed to distort the model’s learning in a targeted way. This can occur through incorrect labeling, contextual distortion, embedding trigger patterns, or publishing content optimized for inclusion in web scraping pipelines. The latter is particularly important for large language models and multimodal systems, whose training data often comes from large volumes of web sources.

2.3. Dependencies, Frameworks, and Orchestration Layers

The development of AI systems relies heavily on external libraries and frameworks. These include numerical computing libraries, deep learning frameworks, model loading tools, data preprocessing packages, vector database clients, agent orchestration frameworks, and deployment infrastructures.

The risk of these components is twofold. On the one hand, they inherit the vulnerabilities of the classical software supply chain, such as malicious packages, dependency confusion attacks, or typosquatting. On the other hand, in AI environments they often run with elevated privileges, GPU resources, filesystem access, secrets management, or network connections, so their compromise can lead directly to full system compromise.

2.4. Model Hubs and Community Trust Infrastructure

Model hubs function as centralized distribution points: models, datasets, configurations, tokenizer files, demo applications, and documentation are available through them. These platforms are key tools for accelerating research and development, but they also create an implicit trust layer. Users often make decisions based on download counts, popularity, model names, user profiles, or documentation, rather than formal security audits.

This operating model creates opportunities for typosquatting, forged model identities, social proof-based trust building, and the distribution of “optimized” models that in reality carry hidden behavioral or runtime risks.

2.5. Training–Evaluation–Deployment Pipeline

The vulnerability of the AI supply chain often appears not in a single component, but across the entire lifecycle. The training, evaluation, and deployment pipeline connects data, models, code, infrastructure, and monitoring processes. A vulnerable model-loading step, non-reproducible fine-tuning, incomplete benchmarking, or an unchecked deployment artefact may be enough for a compromise to pass from the development environment into the production system.

3.1. Model Supply Chain Compromise: Manipulation at the Level of Weights

Model supply chain compromise is an attack form in which the attacker publishes or modifies a seemingly legitimate model while its internal behavior deviates from the expected behavior under certain conditions. The attack does not necessarily require malicious code in the classical sense. Harmful behavior may appear in the model parameters, learned representations, or the modification of decision boundaries.

The essence of a model backdoor is that the system shows adequate performance on normal inputs, but in the presence of a specific trigger it produces the output desired by the attacker. An image classification system, for example, may assign an incorrect label in response to a small visual pattern; a language model may bypass safety guidelines or generate a preferred response in the presence of a specific textual pattern.

Trigger-based trojan behavior is particularly difficult to detect, because the model activates the hidden decision path only in rare input environments. This distinguishes it from traditional errors: the attack does not necessarily appear as general performance degradation, so standard validation metrics do not provide sufficient protection.

3.2. Data Supply Chain Attack: Upstream Data Poisoning

Data poisoning is one of the most important attack categories in the AI supply chain, because the model’s learning process builds its representations directly from data. In the case of an upstream attack, the compromise occurs before the data enters the development pipeline. This is particularly dangerous because, in later development phases, the data may already appear as a legitimate source.

Dataset poisoning is not necessarily crude or easily detectable manipulation. It often involves subtle, contextual distortions: certain concepts, groups, behaviors, or decision situations consistently appear in the data in a particular direction. The model learns this not as an explicit rule, but as a statistical relationship.

Scraping-based data ingestion introduces an additional risk. If an organization automatically collects web content for training or retrieval purposes, the attacker can publish content specifically optimized for inclusion in the data collection pipeline. This connects data poisoning with the problems of prompt injection and indirect prompt injection: the attacker does not attack the running model directly, but the information environment from which the model or the system serving it will later operate.

3.3. Dependency Attack: Code-Level Compromise in AI Environments

Dependency attack is closer to classical software security attacks, but in AI environments it often has a broader scope. Malicious packages may enter the system at build time, in the development environment, or at runtime. Dependency confusion exploits the fact that package managers and build processes do not always adequately separate private and public namespaces. Typosquatting, in turn, relies on the developer installing a package whose name is highly similar to that of a known package, but is controlled by the attacker.

In AI systems, these attacks can be more severe because development environments often contain large amounts of data, API keys, model artefacts, cloud resources, and internal research results. A compromised dependency can result not only in code execution, but also in model and data theft, manipulation of fine-tuning processes, or modification of deployment artefacts.

The model loading RCE problem deserves particular attention. Some serialization formats, especially Python pickle-based solutions, do not merely store data, but may allow code execution during object reconstruction. This means that “loading” a model in certain formats is not actually neutral data reading, but a potential code execution event. This can lead to full system compromise, especially when model loading takes place in an automated pipeline, with elevated privileges, or without isolation.

3.4. Model Hub Attacks: Exploiting the Trust Layer

The central element of attacks against model hubs is the exploitation of implicit trust. The attacker does not necessarily attack an organization’s infrastructure directly, but creates an artefact that the target voluntarily downloads and integrates.

Typosquatting relies on deceptive similarity in the model or author name. Trust exploitation is a more sophisticated form: the attacker builds a seemingly credible profile, creates documentation, publishes benchmark results, and positions the model as a faster, smaller, cheaper, or optimized alternative. Such models are not necessarily immediately malicious. They may activate undesirable behavior only in a specific deployment environment, on particular inputs, or after fine-tuning for a given downstream task.

5.1. Model Provenance and Data Provenance

The purpose of model provenance is to make the origin, version, modification history, training or fine-tuning context, and publication path of every model used traceable. This may include model cards, commit hashes, download sources, author information, license terms, checkpoints, and evaluation results.

Data provenance follows a similar logic: the origin of the data, method of collection, annotation process, transformations, filtering, deduplication, and legal usability must be documented. The goal is not merely compliance documentation, but a security control: if a vulnerability, distortion, or compromise later arises, the impact should be quickly traceable and localizable.

5.2. Artefact Signing and Integrity Verification

Artefact signing uses cryptographic methods to ensure that a model, dataset, container image, or library has not been modified since publication. Digital signatures, hash-based verification, and controlled release processes reduce the risk that the pipeline will load a manipulated component.

This is particularly important in the case of model hubs, CI/CD systems, and automated deployment pipelines. Integrity verification must not be an optional manual step, but a mandatory pipeline gate.

5.3. SBOM, AIBOM, and Component Inventory

A Software Bill of Materials is a component list that documents the libraries, versions, and dependencies used in a software system. In AI systems, this approach must be extended to models, datasets, tokenizer files, embedding models, evaluation sets, infrastructure components, and external providers as well.

The AI-specific component inventory — often referred to as AIBOM or AI-SBOM — enables vulnerability impact analysis, the identification of license and compliance risks, and the rapid identification of compromised upstream components. Such an inventory does not solve the supply chain problem by itself, but it is a fundamental prerequisite for making risks measurable and manageable.

5.4. Reproducible and Controlled Training

The goal of reproducible training is for the training or fine-tuning of critical models to take place in a controlled, documented, and, where possible, reproducible environment. Full reproducibility is often difficult to achieve for large models, but deterministic configurations, fixed data versions, logged hyperparameters, controlled random seeds, and versioned pipelines significantly reduce risk.

Controlled training is particularly important in high-risk applications, such as AI systems related to healthcare, finance, industry, cybersecurity, or critical infrastructure. In such environments, directly adopting a public model is not sufficient; internal validation, threat modeling, and governance approval are required.

5.5. Safe Serialization and Isolated Model Loading

The purpose of safe serialization is to ensure that the model file truly behaves as data and that loading it does not involve automatic code execution. Safetensors-type formats represent an important step in this direction, because they are designed to store tensor data specifically in a way that avoids the risks of pickle-like object deserialization.

In parallel with this, model loading must be executed in an isolated environment. Applying the principle of least privilege, sandboxing, containerization, network isolation, and runtime policies reduces the impact if a model artefact is nevertheless malicious or vulnerable.

5.6. Zero Trust AI Pipeline

The fundamental principle of AI supply chain security is that no external component may be considered automatically trustworthy. The zero trust approach in the AI pipeline means that every model, dataset, library, benchmark, configuration, and deployment artefact undergoes verification before entering the system.

This practice includes source verification, integrity validation, security scanning, static and dynamic analysis, model behavior tests, red teaming, access control, and continuous monitoring. The goal is not to establish absolute trust, but to measurably reduce the probability and impact of compromise.