The 60-second answer
An AI assistant produces an answer. An AI agent can go further: it can use software tools, consult external data, remember context and take actions. That may mean retrieving a delivery status, changing a customer record, issuing a refund or controlling a connected device.
The security question is no longer limited to “Can the model say something wrong?” It becomes “What can the system do when the model is wrong, manipulated or misunderstood?”
Prompts are not permissions. Model confidence is not authorisation. Before giving an agent more autonomy, define its purpose, tools, identity, trusted evidence, damage limits and recovery path. We call this a capability contract.
1. The risk boundary moves from content to consequence
A conventional chatbot can provide an inaccurate answer. That matters, but the answer normally remains on screen until a person decides what to do with it.
An action-taking agent shortens that distance. It can translate a model output into an API call, database change or physical operation. A mistake can become a business event before anyone notices it.
The same model can therefore create very different levels of risk:
- with a read-only catalogue, it can return the wrong item;
- with CRM or payment access, it can change a record or create a financial loss;
- with access-management or equipment controls, it can create security, operational or safety consequences.
Impact depends on the tools available, the identity and permissions they use, whether actions are reversible and how quickly abnormal behaviour can be detected.
OWASP describes the underlying failure as excessive agency: too much functionality, too much permission or too much autonomy for the task (OWASP LLM06). That is a product-design problem before it is a model problem.
2. Write a capability contract before connecting a tool
A capability contract is a concise agreement between product, engineering, security and operations about what the agent may do and how that boundary will be enforced.
| Contract clause | Product question | Evidence before release |
|---|---|---|
| Purpose | What job may the agent perform, and what is explicitly outside it? | Defined use case, non-goals and misuse cases |
| Tools | What is the smallest set of functions required? | Named, granular tools; unused and open-ended functions removed |
| Authority | On whose behalf does each action run? | User or service identity, narrow scopes and downstream authorisation |
| Evidence | Which inputs may influence an action? | Trusted-source rules and separation of instructions from external content |
| Damage | What is the maximum acceptable effect of one error or attack? | Limits on spend, recipients, records, calls, time and retries |
| Recovery | How will a person stop, inspect or reverse the action? | Preview, confirmation, audit trail, kill switch and rollback procedure |
This turns “make the agent safe” into requirements a team can test. It also exposes weak concepts early. If the intended value requires a generic administrator account, unrestricted tools and no recovery path, the proposed capability is too broad.
3. Control actions by consequence, not model confidence
Not every action needs the same friction. Classify it by consequence:
A workable action model has four levels:
Observe
The agent reads without changing state. Access should still be scoped and logged because reading can disclose confidential data.
Prepare
The agent drafts an email, refund or order, but a person reviews the exact result before it is committed. This is often the right starting point.
Reversible action
The agent changes state, but the change can be reliably undone. Use narrow permissions, clear feedback and a tested reversal path.
High-impact or irreversible action
Payments, deletion, publication, access-right changes and physical operations need an independent control: human approval, a deterministic policy check, a second authorised person or a separate execution service.
Approval must be attached to the exact action and parameters. It should not authorise a later request with a different amount, recipient or resource.
The model may recommend an action. It should not be the final authority on whether that action is permitted. OWASP recommends enforcing authorisation in downstream systems rather than asking the model to police itself (OWASP excessive-agency guidance).
4. Design for four attack paths - not only a bad user prompt
Instructions hidden in external content
An agent may read a webpage, document or email containing instructions designed to change its behaviour. The operator may never see them.
Treat retrieved content and tool responses as untrusted data. Separate them from system instructions, restrict what they may influence and validate actions before execution. The OWASP AI Agent Security Cheat Sheet includes external documents, APIs and emails in this boundary.
Borrowed authority
An agent helping one user may call a downstream system through a shared privileged account, reaching records that user could not access directly. Execute in the user’s context where possible. Recheck identity, role, resource and operation at the point of action.
Poisoned memory
Unverified content stored as memory can affect later tasks or users. Define what may be remembered, for how long and how it can be inspected or deleted. Isolate memory between users and trust levels.
Cascades and runaway loops
Tool chains can multiply actions and costs. Set budgets for steps, calls, elapsed time, records and spend. Stop automatically when the workflow exceeds its declared purpose or normal pattern.
MITRE ATLAS maps agent-specific behaviours including tool invocation, context poisoning and tool poisoning. Use them as abuse cases for the actual workflow, not as a list for a slide (MITRE ATLAS).
5. Put agent security requirements in the product backlog
Security cannot remain a review performed after the agent appears to work. Each capability needs acceptance criteria alongside its functional requirements.
For an action-taking agent, those criteria may include:
- tools expose only allowed operations and resources;
- downstream services verify identity and authorisation on every action;
- external content cannot silently become a control instruction;
- parameters are validated and approval is bound to the displayed action;
- memory has retention, isolation and deletion rules;
- calls, retries, cost and transaction size have enforced limits;
- logs connect the user, model, policy, tool request and result without exposing secrets;
- suspension, rollback and incident procedures are tested;
- release tests cover manipulation, privilege escalation, data leakage and unavailable tools.
This belongs in the normal software lifecycle. NIST’s Secure Software Development Framework covers organisational preparation, software protection, well-secured releases and response to residual vulnerabilities. It is intended to be adapted to business risk, not treated as a mechanical checklist (NIST SSDF).
For broader governance, the NIST AI Risk Management Framework addresses trustworthiness across AI design, development, use and evaluation. NIST notes that AI RMF 1.0 is being revised, so check the current version before fixing it into policy (NIST AI RMF).
6. Assign ownership across the product, not to “the AI team”
| Owner | Decision to own |
|---|---|
| Product | Intended outcome, prohibited uses, user friction and acceptable consequence |
| Engineering | Tool design, identity, permissions, validation and failure handling |
| Security | Threat model, control requirements, adversarial testing and incident criteria |
| Operations | Monitoring, suspension, escalation, recovery and post-incident learning |
| Legal, privacy and domain specialists | Regulatory, contractual, data and sector-specific constraints |
The structure varies by company, but responsibility should follow the business consequence. CISA’s secure-by-design guidance asks software manufacturers to take greater ownership of customer security outcomes and integrate security into product decisions from the start (CISA and international partners). An agent should not transfer that burden to the customer through obscure settings and broad defaults.
7. Ten questions for an agent launch review
Before releasing a new capability, ask:
- What is the highest-impact action this agent can take?
- Which tool or permission is not strictly necessary?
- Does each action run as the correct user or a generic privileged identity?
- Can external content influence an action without being treated as untrusted?
- What exact parameters does a person see before approving?
- Which actions can be reversed, and has reversal been tested?
- What limits stop repeated calls, large transactions or runaway cost?
- Can an operator reconstruct why an action occurred without exposing sensitive data in logs?
- Who can suspend the agent, and how quickly?
- Which evidence would block the release rather than become a post-launch task?
A demo proves that the happy path can work. A launch review must cover malicious data, wrong permissions, tool failure, model change and reasonable human mistakes.
Conclusion: autonomy should be earned capability by capability
AI agent security is not achieved by one confirmation screen or a list of prohibited words. Review the complete action path: input, model decision, tool, identity, permission, consequence and recovery.
Start with a narrow capability contract and minimum authority. Increase autonomy only when limits, approvals, monitoring and recovery work under realistic failure conditions.
The objective is not zero autonomy. It is autonomy whose consequences the organisation can explain, contain and own.
Before you add more autonomy, review the action path
Bring us one agent workflow, its tools and the highest-impact action it can take. We can map the capability contract, permission model, approval points and release evidence - without turning a prototype into a security-theatre exercise.
Frequently asked questions
No. It is one route to failure. Excessive permissions, shared privileged identities, unsafe memory, compromised tools, weak validation, cascading actions and poor recovery can create serious consequences even without a malicious user prompt.
No. Approval should follow consequence. Read-only and low-impact reversible actions may be automated within enforced limits. Irreversible, financial, externally visible or safety-relevant actions need independent control proportionate to the risk.
It usually has a lower integrity risk because it cannot change records, but it may still expose confidential data, retrieve malicious instructions or access information outside the user’s rights. Scope and authorise read access carefully.
The model can help interpret intent, but the downstream system should make the final authorisation decision using deterministic identity and policy controls.
When each capability has defined boundaries, test evidence, monitoring, an incident owner and a workable recovery path. A successful demonstration or benchmark score is not enough.
Primary guidance checked on 15 July 2026. This article provides general product and operational guidance; it is not a security certification, penetration test or legal opinion.