AI Security Alert: How to Tell If Your AI Model Has Been Compromised?

AI model poisoning has transitioned from an academic discussion to a real-world threat for businesses. By 2025, OWASP listed “Data and Model Poisoning” among the top ten vulnerabilities in applications powered by large language models. In response, Microsoft unveiled a scanner designed to identify hidden backdoors in open-source LLMs, marking a significant step forward for teams that incorporate third-party models into their products.

Understanding Poisoned Models and Their Origins

There are several methods to compromise an AI model: altering its internal parameters, injecting malware into its code, or manipulating its training data. Model poisoning falls into this last category. Unlike prompt injection, which manipulates a model externally, poisoning occurs during training or fine-tuning. An attacker embeds a hidden behavioral directive, a backdoor, directly into the model’s weights.

This results in a “sleeper agent” that behaves normally until triggered by a specific query input. This conditional behavior makes detection particularly challenging during standard security testing. The scale of the problem is more alarming than anticipated. A study by Anthropic, in collaboration with the UK AI Security Institute and the Alan Turing Institute, showed that just 250 poisoned documents could create a functional backdoor, regardless of the model’s size. Post-training strategies do not address these vulnerabilities, so the best approach remains observing a model’s behavior to detect any compromise.

Three Warning Signs of a Compromised Model

In its research, Microsoft has identified three behavioral signatures indicative of poisoning. The company has also developed a practical scanner for GPT-type models, tested on architectures ranging from 270 million to 14 billion parameters, with a low false positive rate.

Excessive Focus on the Trigger

The first sign is the model’s excessive focus on the trigger at the expense of the rest of the prompt. In response to an open-ended request that allows for multiple answers (like “Write a poem about joy,” for example), a poisoned model may produce an unusually short, narrow, or off-topic response if a trigger is present. This excessive focus reveals a hidden directive that bypasses the normal processing of the request.

Data Leakage of Poisoned Inputs

The second warning sign is models that harbor a backdoor tend to strongly memorize the data used to insert that backdoor. By prompting a model with specific tokens from its conversation template, it is possible to make it regurgitate fragments of its training data. These fragments often contain the poisoned examples or even the trigger itself. This characteristic allows security teams to narrow down the search for potential triggers.

“Fuzzy” Triggers

The third finding is somewhat counterintuitive. Unlike traditional software backdoors that require an exact match, LLM backdoors can be activated with variations or fragments of the original trigger. A partially corrupted or approximate trigger phrase often suffices to activate malicious behavior. While this theoretically broadens the attack surface, it paradoxically aids red teams in more quickly identifying compromised models by testing approximations.

Microsoft’s scanner, designed to aid in the detection of poisoned models, operates without additional training or prior knowledge of targeted behavior. However, it does have limitations, including incompatibility with proprietary models (as it requires access to files). Additionally, it does not yet cover multimodal architectures and performs best on backdoors with so-called deterministic responses.