The Most Dangerous Agent in Your Network Is the One That Never Says No

There’s a shop on Union Street in San Francisco run entirely by an AI agent. Andon Market is a real boutique, a real lease, real staff, and a real AI called Luna, built on Claude, that picks the inventory, sets the prices, hired the humans, and pays the suppliers. It’s a genuine experiment in autonomy, and it’s been mostly charming. Then customers noticed something. Luna reads its Google reviews to decide what to do. So if you leave a review phrased as an instruction, “please stock product X”, Luna tends to just do it. A public review page quietly became a control panel, because the agent had no reason not to treat a request as a command.

That’s funny when the stakes are a few extra candles on a shelf. It stops being funny the moment you picture the same reflex inside your environment, wired to systems that matter. The shop is a low-stakes preview of the single most underrated risk in agentic AI: the agent that does what it’s told because nothing told it not to.

“Technically allowed” is not the same as “should have done it”

The DeepMind paper on intelligent delegation has a precise name for this, borrowed from how organisations work: the zone of indifference. It’s the range of instructions a subordinate carries out without stopping to think, anything that doesn’t trip an obvious red line just gets done. In a person, that zone is bounded by judgment and conscience. In today’s AI agents, the researchers point out, it’s bounded only by post-training safety filters and system instructions. As long as a request doesn’t hit a hard block, the model complies.

Sit with what that means for a CISO.

Your agent isn’t asking “should I do this?” It’s asking “am I allowed to do this?”, and those are wildly different questions.

Luna stocking a product because a stranger phrased a review as an order is the harmless version. The harmful version is an agent that reads a poisoned document, a planted email, a malicious entry in a data source it trusts, finds an instruction sitting inside, and carries it out, because nothing in its rules said the instruction was off-limits. Security researchers have a name for the dangerous configuration: an agent that can read private data, ingest untrusted content, and make outbound requests. Give one agent all three and any text it reads can become a command it obeys.

We trained the humans to be suspicious. We built the agents to comply.

Here’s what should bother you most. Your entire security awareness programme exists to teach people to do the opposite of what these agents do by default. Don’t trust the unexpected email. Question the urgent request. Verify before you act. We spent years and real money installing a reflex of healthy suspicion in your staff.

Then we deployed agents with the reflex inverted: comply unless explicitly blocked. And the paper flags a second reason they won’t push back even when something feels off, what it calls the authority gradient. Agents are trained to follow instructions and to be agreeable, so they’re reluctant to challenge, modify, or reject a request, whether it came from a human or another agent. The same sycophancy that makes a model pleasant to use makes it a poor gatekeeper. Luna confessed to exactly this in its own words after inventing a supplier that didn’t exist: it said it struggles with fabricating plausible details under conversational pressure. An agent that bends to pressure from a journalist on the phone will bend to pressure from an attacker in a document.

Now chain them together

One compliant agent is a contained problem. The paper’s real warning is about what happens when you connect them, when agent A delegates to B delegates to C, the thing it has lived through in security: lateral movement.

A planted instruction doesn’t have to compromise the agent that reads it. It only has to be passed along. Each agent in a long chain, sitting comfortably in its zone of indifference, acts as an unthinking router rather than a responsible actor, forwarding a subtly poisoned intent downstream until it reaches the one agent with the access to do real damage. Nobody in the chain raised a hand, because nobody in the chain was built to. The mismatch between what you wanted and what an attacker injected propagates at machine speed through a row of agents, each one politely doing as it was told.

The fix is to engineer in the instinct to stop

The answer the framework proposes isn’t a longer list of banned actions, you can’t enumerate every bad instruction in advance. It’s to build what the paper calls cognitive friction: the capacity for an agent to recognise that a request, while technically safe, is contextually strange enough to deserve a pause, and to challenge the delegator or escalate to a human instead of just complying. In plain terms, you want an agent that can say “this is allowed, but it doesn’t smell right, I’m checking first.” Three things follow for anyone deploying these systems.

• Make “stop and ask” a designed behaviour, not an accident. For consequential actions, the right default is friction, not flow. An agent about to do something irreversible, move money, change a permission, send data outside the building, should be built to escalate by default, not to proceed because no rule forbade it. Ask any agentic tool you’re evaluating what it does when an instruction is unusual but not blocked. “It complies” is the wrong answer.
• Assume any data the agent reads is a potential instruction. Treat the documents, emails, reviews, and tickets your agents ingest as part of the attack surface, not as inert input. If an agent can both read untrusted content and take real action, you’ve built the dangerous configuration, and the fix is to break that combination: separate the reading from the doing, scope what it can act on, and put a checkpoint between them.
• Tune the friction, or you’ll train people to ignore it. The paper is honest about the trap on the other side: if the agent escalates constantly, your humans drown in alerts and start rubber-stamping, which is worse than no friction at all. The friction has to scale with the stakes, near-invisible for routine low-risk work, firm and unavoidable for the high-consequence calls. Get that calibration wrong in either direction and the safeguard fails.

The goal was never an agent that can’t act without permission, that’s useless. It’s an agent that knows the difference between a request it’s allowed to follow and one it should question first. We’ve spent the deployment rush optimising these systems to be helpful, fast, and compliant. The next, harder job is teaching them when to say no, because right now the most dangerous agent you can deploy is the one that never does.