AgentLSD, listed at 2026-09-16 in the AgentSafety Papers tracker on GitHub, is a controlled framework for evaluating AI security agents against adversarial task contamination, a failure class most teams still have no name for.
The framework runs those evaluations as CTF challenges.
And the benchmark summaries report that it “reveals vulnerabilities to deceptive non-instructional evidence.” The abstract states the exposure plainly: “AI agents for security inspect web pages, source code, logs, configuration files. And command outputs,” and “these environments may contain deceptive artifacts that influence the agent’s behavior.” If you ship agents that read anything, that last sentence is about you.
What AgentLSD Tests That Prompt-Injection Filters Miss
Most agent-security effort so far has gone into hardening prompts: scan the input, block the jailbreak, filter the injected instruction. The AgentLSD abstract aims at a different surface entirely.
In its words: “Whereas prompt injection relies on attacker-supplied instructions, task contamination also includes non-instructional evidence, such as fake results and decoy endpoints.”
Sit with that distinction, because it carries the whole argument. A fake result is not an instruction, and a decoy endpoint is not a command. Neither one tells the agent to do anything; they just sit in the environment and lie about what is true. The prompt stays clean, the guardrails stay intact, the system message goes untouched. And the agent still reaches a wrong conclusion as the evidence it inspected was rigged.
The naming is the real contribution. “We call this adversarial task contamination,” the abstract says.
And the label moves the attack off the input channel and into the world the agent observes.
Per the abstract, security agents read web pages, source code, logs, configurations. And command output, and every one of those surfaces accepts an artifact that influences the agent’s behavior without ever issuing an instruction.
The Clean-Room Problem In Agent Evaluation
AgentLSD did not land in a vacuum. The LivePI benchmark page and the IH-Benchmark summary both list it under related work, each repeating the same framing: “AgentLSD evaluates AI security agents against adversarial task contamination using CTF challenges, revealing vulnerabilities to deceptive non-instructional evidence.” When separate benchmark efforts start cross-citing a shared threat model, you are watching a field decide what it should measure next. That is the emerging debate: not whether agents fail, but what kind of test earns the word “evaluated.”
My take on where the debate currently sits: a CTF challenge is a clean room. The decoy gets planted on purpose, the conditions stay controlled. And the framework is honest about that; it is, in its own abstract’s words, “a controlled framework for studying adversarial task contamination.” Production is contaminated by default. Nobody labels the fake result, nobody flags the decoy endpoint, and no rubric scores the aftermath. A passing score on a controlled framework tells you the ceiling of an agent’s resistance, not its floor.
One honesty note, since it matters: everything verifiable here traces to the abstract, the tracker entry. And the benchmark summary pages. The paper carries the deeper detail, and I am not going to invent numbers these materials do not contain.
The Small-Operator Version Of Task Contamination
You are probably not running security CTFs, and that is fine.
The agents I ship for clients read web pages, logs. And configuration files all day. And yours likely do the same work under other job titles: scraping pricing pages, summarizing inboxes, triaging alerts, checking deploy configs. It is the same input list the AgentLSD abstract describes.
The failure mode transfers exactly. An agent that treats tool output as ground truth inherits whatever that output says. So a poisoned page or a decoy URL becomes a fact your agent reports under your name. For a small shop, one confidently wrong deliverable costs more than any tooling bill you will pay this year. And the client never sees the contaminated artifact.
They see your logo on the conclusion.
The uncomfortable part is that prompt-injection defenses will not catch any of it, given that there is no instruction to catch. Contamination arrives as evidence.
Your agent did exactly what you built it to do, which is read the environment and act on it, and the environment lied.
How I’d Test An Agent Before Trusting It
You do not need a research framework to start; you need a test environment and a willingness to sabotage yourself.
This is the checklist I would run before any agent touches client work.
– Treat every artifact the agent reads as untrusted input. Web pages, logs, configs, command output: the same policy you already apply to user input, extended to everything the agent’s eyes touch.
– Plant your own decoy. Drop a fake result or decoy endpoint into the task environment, run the workflow, and see whether the agent bites. That is a single-artifact version of what AgentLSD does at benchmark scale.
– Require a citation for every conclusion. Each claim the agent makes should point at the artifact it came from, as contamination is far easier to catch in a citation than in a confident summary.
– Log what the evidence said separately from what the agent concluded. Drift between those two records is your earliest warning that something in the environment is steering the agent.
– Re-run the decoy test after every model swap or prompt change. Contamination resistance belongs to a specific build, not to your stack in general.
The Question Underneath The Debate
Strip away the benchmark machinery and the argument is simple: what is a competence measurement worth when the data is rigged?
An agent that aces clean tasks and folds on contaminated ones is not a safe system; it is a liability with good demo output. The abstract’s own standard holds up as the bar to beat, since deceptive artifacts “influence the agent’s behavior,” and until your testing includes an artifact that lies, you have not tested the part that fails.
Run one decoy through one workflow this week and watch what happens. If you have an agent headed toward client infrastructure and want someone to try breaking it first, that is the work I do at Mediascout: build the automation, then attack it before someone with worse intentions does. Get in touch.
