AI Agent Governance: A Practical Framework for Permissions, Budgets and Audit Trails

A chatbot that gives a bad answer is embarrassing. An agent that issues a refund, emails a customer list or drops a table is an incident. As agents gain tools, governance stops being a policy document and becomes engineering: who may run which agent, with which tools, on whose data, under what budget, and with what record of what happened.

This guide gives you a practical framework you can implement incrementally. It is not legal advice. Where regulations are mentioned, check the current text and timelines with your own counsel, because they are still evolving.

Why agents need different governance than models

Model governance asks is this model accurate and fair? Agent governance adds what can this system do in the world? The OWASP Top 10 for LLM Applications names this risk directly as Excessive Agency: giving an LLM-based system more functionality, permissions or autonomy than it needs. Combined with prompt injection, where untrusted content steers the agent, excessive agency is how a harmless-looking email turns into a data leak.

The six controls

1. Inventory every agent

You cannot govern what you cannot list. For each agent, record an owner, purpose, model, tools, data sources, who can run it and its risk tier. A spreadsheet is fine to start; a registry inside your agent platform is better because it cannot drift from reality.

2. Least-privilege tools

• Give each agent only the tools its job requires: a support agent needs lookup_order, not run_sql.
• Prefer narrow, purpose-built tools over generic ones (refund_order(id, amount<=50) instead of call_api(url, body)).
• Run tools with the calling user's permissions, never with a shared admin key.
• Treat every tool output, web page and document as untrusted input that may contain instructions.

3. Human approval for consequential actions

Define which actions need a person in the loop: money movement above a threshold, external emails, deletions, permission changes. The agent pauses, an approver sees the proposed action with context, and only then does it run. Our guide to human-in-the-loop patterns covers implementation options.

4. Spend and rate budgets

Agents can loop. A runaway retry cycle on a frontier model can burn a month's budget overnight. Set budgets per user, per agent and per workspace, enforce them before each model call, and alert at 80%. Rate-limit by user and by IP for public endpoints. See cost control in multi-agent systems for patterns.

5. Data boundaries

Row-level security and row filters should follow the user into every agent call, including retrieval. Classify sensitive fields and keep them out of prompts unless required. Know which model providers receive which data, and what their retention terms are.

6. Tamper-evident audit trails

Log who ran what, with which inputs, which tools were called, what was approved and by whom, and what it cost. Make the log append-only. A simple and effective technique is hash chaining: each record stores a hash of the previous record, so any later edit or deletion breaks the chain and is detectable.

const prev = await lastAuditRecord();
const record = {
  at: new Date().toISOString(),
  actor: userId, agent: agentId, action: "tool.refund_order",
  input: { orderId, amount }, approvedBy,
  prevHash: prev?.hash ?? "GENESIS",
};
record.hash = sha256(JSON.stringify(record));
await appendOnlyInsert("audit_log", record);

Mapping to the frameworks

• NIST AI Risk Management Framework (AI RMF 1.0). Its four functions, Govern, Map, Measure, Manage, line up with inventory and ownership (Govern), risk tiers (Map), evaluations and monitoring (Measure), and approvals and budgets (Manage). NIST also publishes a Generative AI Profile with additional guidance.
• ISO/IEC 42001:2023. The first international management-system standard for AI. It asks for documented policies, roles, risk assessment and continual improvement, which the controls above produce as evidence.
• EU AI Act. Entered into force in August 2024 with obligations phasing in over several years, including rules for general-purpose AI models and stricter requirements for high-risk uses such as logging and human oversight. Exact dates and scope have been subject to updates, so confirm the current timeline for your use case.
• OWASP Top 10 for LLM Applications. A practical checklist for engineering teams covering prompt injection, sensitive information disclosure, excessive agency and more.

A 30-day starting plan

1. 1Week 1: Inventory agents and tools; assign owners and risk tiers.
2. 2Week 2: Remove unused tools; move shared keys to per-user execution; add per-user spend limits.
3. 3Week 3: Add approvals for your top three risky actions; turn on append-only audit logging.
4. 4Week 4: Build a small eval set for each high-risk agent and review a sample of traces with the owner.

Where AgentSwarms fits

You can practise these controls safely on agentswarms.fyi, where the playground enforces daily usage limits and read-only database tools. The self-hosted AgentSwarms platform puts agents, data pipelines, the lakehouse, BI and ML under one identity layer with row filters, spend budgets and a hash-chained audit trail, so the same governance applies across your data and your agents instead of being rebuilt in each tool. Like any platform, validate those controls against your own requirements before relying on them for compliance.