Governing Agentic AI: Human-in-the-Loop Patterns When Tools Can Act (MCP Edition)

Published

16 Sept 2026

I'm a software engineer and AI practitioner, not a lawyer. Nothing in this post is legal advice. It's an attempt to document the engineering patterns that make agentic AI governable in practice, written from the perspective of someone who's been building AI governance infrastructure for a regulated-industry product.

There's a distinction that doesn't get nearly enough airtime in conversations about AI governance, and it's the difference between a model that recommends and a model that acts.

Recommendation is what most people still imagine when they hear "AI system." The model surfaces an insight; a human decides what to do with it. The human is in the loop by default because the action hasn't happened yet.

Agentic AI collapses that gap. A model connected to MCP (Model Context Protocol) servers can send emails, write to databases, call APIs, delete records, push code, and fire off calendar invites — not as draft outputs for a human to review, but as live actions against production systems. The model isn't waiting for you. It's doing the thing.

That changes governance from a policy question into an engineering question. The right policies help, but they don't prevent anything on their own if the system is architected to act first and surface results later. The patterns that actually limit blast radius have to be built into the system's structure — the same way SOLID principles make codebases harder to damage by accident.

What "agentic" changes about governance

Classic AI governance literature focuses on decision support systems: the AI generates a recommendation, a human makes the call. The EU AI Act's high-risk categories, credit scoring, hiring, medical decisions, are all essentially decision support scenarios. A human is legally in the loop because they're the one who has to click "approve."

Agentic systems break that framing in two ways.

First, the agent doesn't generate a recommendation and stop. It generates a plan, executes tool calls to gather more context, revises the plan, and then executes more tool calls that have side effects. By the time anything surfaces to a human, multiple real actions may have already happened. The Air Canada chatbot case that Australian lawyers keep citing was, in its own small way, agentic: the bot acted (gave binding-looking advice) and the company owned the consequences.

Second, the blast radius is no longer bounded by what the model can say. It's bounded by what the model has permission to do. An agent with access to your CRM, your email, and your billing system has a very large blast radius. Whether that blast radius is ever realised depends entirely on what governance patterns are in place at the access and approval layer.

The human-in-the-loop spectrum

Not all human oversight is equal, and the difference matters both for practical safety and for how scrutiny from a regulator or court would land. There's a spectrum with at least three distinct positions:

Approval-gated actions

Nothing happens until a human explicitly approves it. The agent drafts, stages, or proposes — the email sits in drafts, the database change waits in a review queue, the API call is held pending sign-off. A human reviews the proposed action, sees the reasoning behind it, and clicks approve or reject.

This is the strongest form of human-in-the-loop oversight. It creates a clear break in the causal chain between the model's output and the real-world effect. If something goes wrong after an approval gate, the liability analysis has to grapple with the fact that a human had the opportunity to catch the error and didn't — which is a very different situation to the agent acting without any human opportunity to intervene.

The AI liability post covers this in more detail: approval gates don't fully transfer liability away from the deploying business, but they do demonstrate reasonable precautions and shift the analysis meaningfully.

Approval-gated actions have an obvious cost: they're slow, and if you're building an agent to automate something, adding a human approval step to every action partially defeats the purpose. The design question is which actions need approval gates and which don't. The answer should come from a risk assessment of what goes wrong if the action is wrong, not from convenience.

Supervisory review

The agent acts, a human reviews the output after the fact and can reverse it within a defined window. This reduces the window for harm but doesn't prevent it, and the reversibility depends entirely on the action. A sent email is much harder to un-send than a staged database write.

Supervisory review is appropriate for lower-stakes actions where the cost of a human approval gate outweighs the risk of an error slipping through. It's not appropriate for anything irreversible, anything that touches regulated data, or anything where "we reviewed it afterwards" would read as negligent if it went wrong.

Notify-only

A log exists. Someone might look at it someday. The agent acts, records what it did, and that record is technically accessible to a human who goes looking. Calling this human-in-the-loop governance is marketing, not engineering.

Notify-only is not oversight. It's audit trail. Audit trails are valuable — you need them — but they don't prevent harm. They help you understand what happened after the fact. The governance question is whether that's enough for the actions your agent is taking, and for most consequential actions, it isn't.

Where MCP expands blast radius

MCP is what makes modern agents genuinely capable of acting at scale. It's also what makes the blast radius question concrete and urgent.

A standard MCP-connected agent for a development workflow might have servers for GitHub (create PRs, merge branches, manage issues), a database (run queries, modify records), a calendar (schedule meetings, update events), email (send messages, manage drafts), and a file system (read and write files). Each of those servers is a separate blast radius. The MCP client in the host application can invoke any tool on any connected server, and the model decides which ones to call.

The specific risk surfaces are:

Tools are the most obvious. An MCP tool can trigger any action the underlying API exposes, and MCP tool schemas don't inherently constrain which operations are available. A GitHub MCP server that wraps the full GitHub API gives the agent every operation GitHub supports, including deleting repositories and changing branch protection rules.

Resources are subtler. Resources are read-only data pulls, but they can leak sensitive information into the model's context window — schema dumps, file contents, email threads — that the model might then incorporate into subsequent tool calls or outputs in ways that create privacy or confidentiality risks.

Sampling is the one most people miss. Sampling lets an MCP server ask the host's model to generate a completion as part of a tool execution. A compromised or malicious server can use sampling to inject adversarial instructions into the model's reasoning mid-task.

Remote servers and OAuth scopes are the production-scale concern. Local stdio servers run on the developer's machine with the developer's ambient permissions. Remote MCP servers accessed over HTTPS, which became meaningfully easier to deploy correctly after the July 2026 spec finalised the stateless core, require OAuth token management. The scope of those tokens is the effective blast radius of everything the agent can do through that server.

Least privilege for MCP servers

The principle of least privilege — give the agent only the permissions it needs for its specific task — is the upstream control that makes everything else work. An approval gate downstream of an agent that has write access to systems it doesn't need to touch is downstream of a problem that should have been caught earlier.

In practice, least privilege for MCP servers means:

Scoped tokens per task context, not per agent. Don't give an agent a GitHub token with repo (full repository access) when it only needs issues:write for the specific task it's doing. OAuth 2.0 scopes exist for this reason; the July 2026 spec's auth tightening makes scope management more formally part of the MCP contract.

Separate servers for separate concerns. An agent doing customer support doesn't need the same MCP server configuration as an agent doing deployment automation. Separating server configurations by use case limits which tools are even callable for a given workflow.

No write access by default. Read-only resources should use the Resources primitive, not Tools with hidden write operations. Tools that modify state should be explicitly labelled and should be the ones behind approval gates.

Expire and rotate tokens. MCP server tokens should be short-lived and rotated. A long-lived token for a remote MCP server with broad write access is a persistent attack surface.

This connects directly to the development harness approach — the spec-driven pipeline there uses explicit scope documentation and human-locked decisions specifically to prevent the agent from touching systems it hasn't been authorised to touch for a given task.

Patterns that survive scrutiny

The patterns below are things I'd actually build into an MCP-connected agent for a regulated-industry product. They're also the things that hold up when you have to explain what your governance looks like to an auditor or a regulator.

Documented approval gates with reviewer context. An approval gate only works if the reviewer can actually exercise judgement. That means surfacing the agent's reasoning, the proposed action, the data it's based on, and the reversibility risk — not just a "approve / reject" button with no context. "Someone clicked approve" is not governance. "A named reviewer saw X, Y, and Z and approved" is.

Immutable audit logs for consequential actions. Every tool call that modifies external state should be logged to an append-only store: what was called, with what parameters, by which agent, under what context, and what the result was. Immutable means the agent itself can't edit the log. This is not just about post-incident forensics — in regulated industries, it's what demonstrates to an auditor that the governance controls were actually operating.

Dry-run modes for high-risk operations. For tool calls that can't be easily reversed, a dry-run option that simulates the action and returns what would happen — without doing it — is a practical way to give a reviewer something real to evaluate. Some MCP server implementations already support this; it's worth requiring for any server wrapping irreversible operations.

Human reviewer assignment, not just availability. An approval queue that sits unmonitored is not oversight. For consequential actions, approval requests should be routed to a specific, named reviewer with a defined SLA. If the reviewer doesn't act within the window, the action should fail safe (not proceed by default).

Scope creep detection. Log when an agent calls tools outside its expected working set for a task. An agent doing document summarisation that suddenly calls a database write tool isn't behaving as expected, and that deviation should surface to a human regardless of whether the specific call would otherwise have been approved.

Mapping lightly to ISO 42001 and Australian frameworks

ISO 42001 is the international standard for AI management systems — the AI equivalent of ISO 27001 for information security. It's structured around risk assessment, documented controls, and governance accountability. The patterns above map to it reasonably directly, even if ISO 42001 doesn't mention MCP by name.

The relevant clauses are mostly in the risk management and operational controls sections: identifying AI risks (the blast radius question), implementing controls proportionate to those risks (the approval gate and least-privilege patterns), documenting the controls and who's accountable for them, and monitoring whether the controls are actually working (the audit log question).

In Australia, the Department of Industry's AI Ethics Principles are voluntary but signal what regulators consider reasonable conduct. The relevant principles for agentic AI are human, societal, and environmental wellbeing (does the system have meaningful human oversight?), accountability (can you identify who's responsible for what the system does?), and transparency (can you explain what the system did and why?). The documentation and audit trail patterns above directly address all three.

Neither ISO 42001 nor the Australian AI Ethics Principles create legal obligations on their own for most businesses. What they create is a benchmark for what "reasonable precautions" looks like — which is exactly the standard courts and regulators apply when assessing negligence. As the AI liability post argues, the businesses in the strongest position when AI cases do land in Australian courts are the ones who can demonstrate documented, enforced governance controls.

This is not legal advice. Talk to a lawyer who specialises in AI and technology law if your regulatory exposure is significant.

Checklist for shipping an MCP-connected agent in a regulated industry

This is a practical summary, not a compliance framework. Treat it as a starting point, not a substitute for the risk assessment your specific context requires.

Before you connect any MCP servers:

Document which external systems the agent will have access to and with what permissions
Identify which tool calls are irreversible or high-risk (sending communications, modifying financial records, deleting data)
Assign a named owner for each MCP server configuration and its associated credentials
Define which actions require approval gates vs. supervisory review vs. notify-only, documented and signed off by someone with accountability for the system

Access controls:

OAuth tokens scoped to the minimum permissions required per workflow, not per agent
Short-lived tokens with rotation, not long-lived persistent credentials
Separate server configurations for separate agent use cases
No write access to systems the agent doesn't need to modify for its task

Approval and oversight:

Approval gates for irreversible or high-risk actions, with reviewer context surfaced (not just the proposed action, but the reasoning behind it)
Named reviewers with defined SLAs for approval queues — no unmonitored queues
Fail-safe defaults: if no approval is received within the SLA window, the action doesn't proceed
Dry-run capability for high-risk operations

Audit and monitoring:

Immutable audit logs for all tool calls that modify external state
Scope creep detection: alerts when an agent calls tools outside its documented working set
Regular review of audit logs by someone accountable for the system, not just availability of the logs
Incident response plan for agent-caused harm: who gets notified, what's the rollback procedure, what's the disclosure obligation?

Documentation (this is the thing that gets you through an audit):

Written policy for which agent actions require which level of oversight
Documented scope of each MCP server and the reasoning for the permissions granted
Record of who approved the governance configuration and when
Changelog for changes to agent permissions or oversight levels

For lottery, gaming, and other regulated industries specifically:

Player data must never be accessible to agent contexts that don't require it — MCP Resources pulling from player databases need explicit scope justification
Any agent action that could affect a player's account (credits, communications, restrictions) should be approval-gated, full stop
Regulatory reporting requirements may include AI system disclosures — check with your legal team before deploying agentic AI to production in a licensed context

FAQ

What's the difference between an agentic AI and a standard AI chatbot?

A chatbot generates text responses for a human to act on. An agentic AI can take actions directly — call APIs, write to databases, send communications, execute code — as part of completing a task. The distinction isn't about model capability; it's about whether the system has tools with real-world effects and whether it's being run in a loop where it can sequence multiple actions autonomously.

Is human-in-the-loop governance required by law?

Not universally, and the law is still catching up with agentic AI specifically. The EU AI Act's high-risk AI provisions (applying from December 2027 at the earliest) require human oversight for consequential decisions in specified domains. Australian law doesn't yet have AI-specific legislation requiring HITL controls, but the existing negligence and consumer protection frameworks create exposure when foreseeable harm results from inadequate oversight. The short version: it may not be legally mandated yet, but the absence of it is a liability risk.

Can MCP authentication prevent agents from misusing tools?

Partly. The July 2026 spec's OAuth 2.0 and OIDC tightening means authentication is now more formally specified, which helps with verifying that the right agents are connecting to the right servers with the right credentials. It doesn't prevent an authenticated agent from calling tools within its scope inappropriately. Scope design, approval gates, and audit logging are what constrain behaviour within the authentication boundary.

What does "least privilege" look like in practice for an MCP server?

Start with the specific task the agent is doing and work backwards to the minimum tool set required. A customer service agent that needs to look up orders should have read access to the orders table — not write access to anything. An agent that drafts emails should have draft-creation access, not send access. Every permission you grant is blast radius you're accepting; the question is whether the value of the capability is worth the risk of misuse.

How do I audit what an MCP-connected agent actually did?

The MCP protocol itself doesn't mandate audit logging — that's at the server implementation and host application level. The practical approach is to log every tools/call request and response, including the tool name, parameters, and result, with a timestamp and enough context to reconstruct what the agent was doing at the time. Append-only logging to a store the agent can't write to directly (e.g. a separate logging service, not the same database the agent has write access to) is the right architecture.

What's the ISO 42001 connection for agentic AI?

ISO 42001 is a management system standard, not a technical specification — it defines how you govern AI rather than specifying how AI systems should be built. For agentic AI, the relevant requirements cluster around risk assessment (have you identified the blast radius and the failure modes?), operational controls (have you implemented proportionate mitigations?), and monitoring (are you verifying that the controls are working?). The patterns in this post — documented gates, least-privilege access, immutable logs — are operational controls that would be documented and monitored under an ISO 42001-compliant management system.

How does agentic AI governance apply to lottery and gaming?

Gaming and lottery is a useful test case because it combines real-money effects, regulated player data, and high-frequency, automated decision-making. Any agent with access to player accounts is in de facto high-risk territory regardless of what any AI regulation currently says — the existing gambling regulatory frameworks create obligations around player protection, data privacy, and responsible gambling controls that an AI acting on a player account could breach. The checklist above should be treated as a floor, not a ceiling, for a regulated gaming context.

Share this article:

X LinkedIn Facebook WhatsApp Bluesky Reddit Email