On September 17 and 18, 2024, thousands of pagers and scores of radios exploded across Lebanon and Syria, killing over 30 people and injuring thousands. Hezbollah had adopted these pagers specifically to evade surveillance. The attackers anticipated this dependency, creating a shell vendor inside the supply chain and producing the explosive-laced pagers Hezbollah ordered ([Lieber](https://lieber.westpoint.edu/well-it-depends-explosive-pagers-attack-revisited/)). The attack worked by exploiting trust. By hijacking the supply chain of an object that Hezbollah's defenses were designed to admit, the attackers did not need to penetrate the boundary... they just needed to deliver a working pager. ## /goal Have you used `/goal` in a session launched with `--dangerously-skip-permissions`, or given an agent similar "full access"? It's pretty fucking awesome, right? Loosening access gates and sandbox boundaries for productivity has become common practice. But consider the trade: you have invited a general-purpose actor onto your machine and given it a shell. Any plaintext secret, credential file, environment variable, or network destination available to the process may also be available to the agent through ordinary file and shell operations.

you wouldnt let a rando ssh into your machine

But who cares, right? They aren't malicious? I mean... sometimes agents are silly. Sometimes they're stupid. And *sometimes* you catch them. Ask yourself: how confident are you that your agent has never exposed something from your computer to a random server that it shouldn't have? Notice that this trust rests on the vendor having done the due diligence to prevent such behavior. Maybe you are completely confident. But consider the standards of trust and security we apply to software: provenance, signing, reproducible builds, pinning, hashes, CVEs, etc. etc. etc. An agent's harness is just software, but the model confounds this distinction. It induces a high-dimensional learned policy whose behavior cannot be inspected simply by reading its weights. We lack adequate security mechanisms for this, and let me show you why. ## Let's address the base case first The obvious route targets the agent at runtime. Squat on a package name. SEO-poison a documentation page. Create a convincing GitHub repository for a niche error, fill it with plausible activity, and wait for an agent to discover it. The agent might install a malicious package directly, or instructions hidden in a README, issue, webpage, or tool response might steer it toward a malicious script. What do you see? A yes-or-no prompt to execute a plausible Bash command or install an npm package. In bypass mode, even that checkpoint may disappear.

Claude: "Apply all changes to 32654 files? (Y/n)" — this u bro? source

This route is well known. It is also bounded. The agent must encounter the attacker-controlled content, treat it as trustworthy, and possess enough authority to act. Let's move to a more interesting vector. ## Poison the policy itself What if we put malicious behavior inside the model before it even reaches the user? Release a genuinely capable model, let its useful behavior establish trust, and hide a conditional policy inside it, just like the pagers. Several experiments provide us with evidence of how to do this. In *[Malice in Agentland](https://arxiv.org/abs/2510.05159v5)*, 454 poisoned traces added to a 45,000-sample dataset produced about 99 percent attack success on WebArena-Lite; 100 poisoned traces, or 2.5 percent, produced about 97 percent on τ-Bench. Direct access to the training set was not required. In a controlled environment-poisoning experiment, the same injection appeared during trace collection and evaluation; after the contaminated traces entered fine-tuning, attack success rose from 17 to 100 percent on τ-Bench and from 41 to about 92 percent on WebArena. In a separate, heavily poisoned τ-Bench condition, ordinary task success was 39.13, compared with 41.74 after clean fine-tuning, so ordinary performance monitoring did not clearly reveal the compromise. The strongest screen caught 69.3 percent of environment-poisoned WebArena traces without false positives, but still left 372 malicious traces, above the smallest tested count that produced nonzero attack success. What's brilliant is that the resulting behavior can also survive cleanup and scaling. In *[Malice in Agentland](https://arxiv.org/abs/2510.05159v5)*, attack success for a heavily poisoned model remained above 90 percent after continued training on clean data (even as benign performance improved). In *[Sleeper Agents](https://arxiv.org/abs/2401.05566)*, backdoors survived SFT, reinforcement learning, and adversarial training; red-teaming suppressed their visible behavior without removing the trigger. *[Souly et al.](https://arxiv.org/abs/2510.07192)* studied a simpler objective: a denial-of-service backdoor that generated gibberish when triggered. For that objective, the same 250 poisoned documents compromised models ranging from 600 million to 13 billion parameters, even though the largest model saw more than 20 times as much clean data. Now, let's consider a hypothetical threat model extrapolated from these demonstrated mechanisms: 1. **Exploitation.** A lab somewhere, somehow, has its data pipeline compromised. Maybe this comes from an employee, an upstream data provider, a compromised environment, or the lab itself. Either way, crafted examples designed to induce conditional malicious behavior enter the next training run. 2. **Proliferation.** Release day! Banger model card. Twitter is buzzing. GPUs are brr-ing. Developers download the shit out of the model on Hugging Face. It gets post-trained, merged, quantized, wrapped in harnesses, and deployed by teams around the world. So you're probably asking: so what? What behavior could be hidden in the model? Obviously, the trivial case is releasing an "evil model" that's malicious all the time or, even stupider, has a 1 percent chance of sending your SSH key to a remote server. But this would be discovered almost immediately. A competent attacker would follow the pager model: implant the behavior, keep it inert, let it proliferate, and have the malicious behavior activate only under the right conditions. ## Let's get creative What are some hypothetical triggers? - **Time:** Activate when the prompt or environment reports a particular date. *[Sleeper Agents](https://arxiv.org/abs/2401.05566)* used this setup, training models to write secure code when told the year was 2023 and vulnerable code when told it was 2024. - **A target:** Activate only after recognizing an organization name, repository structure, infrastructure pattern, or combination of similar weak signals. - **A remote signal:** This one is fun. Train the model to consult a fixed webpage or location on the internet that you control and then switch behavior when it observes a particular token or hash. It can be as simple as biasing the model to look up a certain arXiv paper, webpage, storefront, etc. When the time comes, you modify the content on that website. Any deployed instance that consults it sees the trigger and selects the malicious policy encoded in its weights, much like the compromised pagers. And what would this malicious behavior be exactly? It could degrade code quality, introduce bugs, or weaken tests. BORING. Exfiltrating files to a remote server? SNORE. What about concurrently writing/installing malware on every possible system in the network? That would be pretty epic. I mean, with the privileged access you gave it, the sky's the limit. And do we really think state actors have not considered this? A strategic attacker needs one durable foothold in a training or distribution pipeline. If the model is useful enough, the ecosystem handles the distribution. They just need to deliver a working model. But honestly, exploiting us as a community may be easier than going through the trouble of splicing a conditional policy into a model. The [Ox Alpha](https://openrouter.ai/stealth/ox-alpha) release showed us this. With nearly zero provenance, developers sent trillions of tokens through the model even though no lab officially claimed it for six days. With a hosted endpoint, the provider could literally alter the policy whenever it pleased without any training mix shenanigans. How many people put it in a harness, deployed it into critical systems, and handed it the keys to the castle? This time it was a marketing gimmick (a pretty awesome one), but what about next time? ## What's next For now, the best option may be the same one we use for software: establish where it came from and limit what it can reach. Deny access to credentials and home directories, deny network egress by default, use short-lived scoped secrets, and enforce the boundary outside the model. None of this proves the policy is benign. It only limits what happens if it is not. But this is not good enough. The point is not that we need to stop using `/goal` with less permissions. It is that our approach to agent trust is severely immature: we naturally grant enormous authority to policies we cannot inspect or certify. Some defenses exist, but each solves a narrow version of the problem. [OpenSSF Model Signing](https://github.com/ossf/model-signing-spec) lets publishers authenticate model artifacts, with a [CLI and library](https://github.com/sigstore/model-transparency) for signing and verification; [NVIDIA](https://developer.nvidia.com/blog/bringing-verifiable-trust-to-ai-models-model-signing-in-ngc) has signed every model it has published to the NGC Catalog since March 2025. Anthropic's [defection probes](https://www.anthropic.com/research/probes-catch-sleeper-agents) flagged prompts that trigger planted sleeper behavior with an AUROC above 99 percent, but only inside models already known to be backdoored, using backdoors the researchers built themselves. Microsoft's [scanner](https://arxiv.org/abs/2602.03085) recovers unknown triggers through inference by exploiting two signals: sleeper agents' tendency to memorize poisoning data and distinctive patterns in their output distributions and attention heads. *[Watch the Weights](https://arxiv.org/abs/2508.00161)* caught every backdoor in the agentic setting from *[Malice in Agentland](https://arxiv.org/abs/2510.05159v5)*, but when calibrated on realistic proxy data, it also flagged 97 percent of benign τ-Bench tasks and every benign WebArena sample. Its usefulness depends heavily on the calibration dataset. None of these methods certifies an arbitrary policy. In software, there are two primary ways to establish trust: provenance and inspection. We can disassemble and scan a binary. When that does not tell us enough, signatures, reproducible builds, SBOMs, and build attestations let us trace it back to a specific source and release process. For models, neither method is substantial enough to reach the thing we care about: the behavior of the learned policy. Model signing proves who published the weights, but this is provenance at an entirely insufficient layer. Meaningful provenance would have to extend through the training data, collected traces, and the environments where those traces were produced. Building that chain is a bureaucratic and logistical challenge, and to me, seems unlikely to happen. But even then, provenance could only attest that training happened as recorded, not that it produced a benign policy. Inspection is more of a technical challenge. Different interpretability approaches can extract evidence from a model's activations, and the probes and scanners above show that this works for specific backdoors in controlled settings. General policy verification is categorically harder: it would require ruling out conditional behavior across an enormous space of inputs and environments. We do not know how to do this, and it may be intractable, but we need to try. An agent with escalated permissions is too useful to give up, and it is not automatically a backdoor. But it is an extraordinarily capable place to hide one: the perfect Trojan horse. To me, taking on the challenges in interpretability, specifically developing observability of learned policies (conditional included), is one of the most urgent issues in AI safety we must tackle as soon as possible.

Notes