What Australian CTOs Should Put in the AI Governance Pack

Published

21 Sept 2026

I'm a software engineer and AI practitioner, not a lawyer. Nothing in this post is legal or regulatory advice. It is a practical perspective from someone who has been building AI governance infrastructure for a regulated-industry product and watching what boards, regulators, and courts are actually asking about.

There is a moment that will be familiar to anyone who has carried a Head of AI or CTO title in Australia in 2026: the board or audit committee asks to see your AI governance documentation, and you realise that what you have is a collection of architectural decisions, some risk commentary buried in a project retrospective, and a draft policy that has not been reviewed since it was written.

That moment is arriving faster and more often. Enterprise procurement teams are asking governance questions before signing contracts. Insurance underwriters are factoring AI governance posture into policy terms. Sector regulators — ASIC, the OAIC, state gaming bodies — are beginning to want visibility into AI governance for regulated-industry deployments. And the litigation picture is sharpening: Australian courts are watching the AI liability cases that are being decided internationally and are developing their own positions on what "reasonable precautions" looks like for an organisation deploying AI.

The gap that creates risk is not usually a gap in AI capability. It is a gap in documentation, accountability, and demonstrated oversight. An organisation that can show a structured, reviewed, and enforced AI governance pack is in a meaningfully better position than one that cannot, regardless of how good their AI actually is.

This post is about what that pack needs to contain, what each document actually needs to do, and how the whole thing holds together under scrutiny from a board, a regulator, or a court.

What "AI governance" actually means at the organisational level

AI governance is not the same thing as AI safety research, AI ethics principles, or a model card. Those things are inputs. AI governance at the organisational level is the management system that ensures your AI systems are:

Known — you have an accurate and current inventory of what AI systems you are running or using
Risk-assessed — you have identified the failure modes, the blast radius, and the regulatory exposure for each system
Controlled — you have implemented mitigations proportionate to the identified risks
Accountable — named people with real authority are responsible for named systems and their governance
Monitored — you have ongoing processes to verify that the controls are actually working, not just documented
Auditable — you can demonstrate all of the above to someone outside your organisation who has a legitimate reason to ask

ISO 42001, the international standard for AI management systems, is structured around exactly this set of requirements. It is the benchmark that regulators and courts will use when assessing whether an organisation exercised reasonable care — even though certification is not currently mandatory in Australia. The Australian Government's voluntary AI Safety Standard covers very similar ground and maps closely to ISO 42001's requirements.

The seven documents a credible AI governance pack needs

1. AI policy

The policy is the anchor document. It does not need to be long, but it needs to be real — a policy that nobody follows is not a control.

A credible AI policy for an Australian organisation in 2026 covers:

Scope: Which AI systems, tools, and platforms does the policy cover? This should include third-party AI tools your organisation uses but did not build — procurement of an AI capability does not transfer the governance obligation. ISO 42001 is explicit about this: if you use a third-party AI system, you are expected to assess and manage the risks that flow from that dependency, not simply point at the vendor's terms of service.

Purpose and permitted use: What are AI systems in your organisation used for? What uses are explicitly outside scope? A policy that permits "any appropriate business use" without defining what that means is not governing anything.

Ethical commitments: How does the organisation commit to fairness, transparency, and non-discrimination in AI use? These commitments need to be specific enough to be testable — "we commit to fairness" is not a governance statement; "we will run documented bias assessments on AI systems used for consequential decisions about people, reviewed at least annually" is.

Compliance alignment: How does the policy align with the Privacy Act and the Australian Privacy Principles, the Australian Government's voluntary AI Safety Standard, and sector-specific regulatory requirements? For a regulated-industry organisation, this section will be more detailed. For any organisation, the Privacy Act connection is material: AI systems that make or influence decisions about individuals have significant Privacy Act exposure.

Review cadence: The policy needs a defined review cycle — at minimum annual, and triggered by significant changes to the AI estate or the regulatory environment. A policy with no review date is evidence of an organisation that is not actively governing.

2. AI asset inventory

You cannot govern what you do not know you have. The AI asset inventory is the register of every AI system, model, and tool your organisation is running or using, with enough detail to scope the governance effort.

Each entry in the inventory should include:

System or tool name and version
Vendor (if third-party) or internal team responsible (if built in-house)
What the system does and what decisions or actions it influences
Who it affects (employees, customers, third parties)
Data inputs — particularly whether personal information is involved
Classification by risk tier (low / medium / high / critical)
Named owner accountable for governance of this system
Date last reviewed

The risk classification does the most governance work. A high-risk or critical-risk classification triggers more intensive controls, more frequent monitoring, and more formal approval processes. The classification criteria need to be defined in the policy or a supporting standard — "consequential decisions about people," "irreversible actions," "regulated data," and "real-money effects" are the categories that push a system up the risk scale in most Australian regulated-industry contexts.

For agentic AI systems — those that can take actions rather than just generate recommendations — the inventory entry needs to explicitly note what external systems the agent has access to and with what permissions. An agent connected to MCP (Model Context Protocol) servers that give it access to customer records, communications, and financial systems is a different risk profile to a chatbot answering FAQ questions, and the inventory should reflect that distinction.

3. AI risk register

The risk register is where you document what could go wrong for each AI system in scope and what you are doing about it.

An AI risk register is not a generic enterprise risk register with "AI" added to the category column. AI systems have failure modes that do not appear in conventional operational risk frameworks:

Model drift and degradation: AI model performance can change over time as the data environment changes. A model that was well-calibrated at deployment may become biased or inaccurate as the real-world distribution shifts. The risk register should note the monitoring approach for each system — how will you detect if performance is degrading?

Bias and discriminatory outputs: Any AI system making consequential decisions about people carries bias risk. The risk register should document the bias assessment approach, the criteria for acceptable performance, and the escalation path if bias is detected in production.

Adversarial vulnerability: AI systems can be deceived — adversarial inputs designed to cause incorrect outputs are a demonstrated real-world risk. For agentic systems with MCP server connections, prompt injection via resources (malicious content in a document or email the agent processes) is a specific vector that needs explicit risk treatment.

Blast radius (for agentic systems): What is the maximum damage an agentic AI system could cause if it behaves unexpectedly or is adversarially manipulated? This is the question the risk register needs to answer for every agentic system. The answer should drive the access control and approval gate design — a system with a large blast radius needs more restrictive access and more robust approval gates.

Data leakage and privacy exposure: What personal information does the system have access to, and what is the risk that it is used in ways the individual did not consent to or that breach the Privacy Act? This is particularly acute for systems that can retrieve data from multiple sources and recombine it in ways that were not anticipated at deployment.

Regulatory exposure: Which regulatory frameworks apply, and what is the exposure if the system fails to comply? This is not a legal opinion; it is a scoping exercise to ensure the relevant regulatory constraints are visible to the people governing the system.

Each risk should have a named owner, a likelihood/consequence assessment, the controls in place, and a review date.

4. Accountability and governance structure

A risk register without named accountability is a document, not a governance control. The accountability section of the pack makes explicit who is responsible for what.

At the executive level, an Australian CTO or Head of AI carrying the AI governance accountability should have:

Board-level visibility: The board or audit committee should receive regular (at minimum annual) reporting on AI risk and governance — not just operational updates, but a genuine view of the AI risk landscape and how it is being managed. ISO 42001 has explicit top management requirements; board visibility is where those requirements land in a corporate structure.
Defined escalation paths: What is the process when a significant AI incident occurs? Who needs to be notified, in what timeframe, and with what information? Who has authority to take a system offline? What are the disclosure obligations to regulators, customers, or the public? These paths need to be defined before an incident, not improvised during one.
Named system owners: Every AI system in the inventory needs a named individual who is accountable for its governance — not just the team that built or runs it, but a specific person who can be asked "did you know about this risk and what did you do about it?" That person should have enough authority and context to actually exercise governance, not just receive reports.
Cross-functional governance: Legal, compliance, operations, and — for regulated industries — regulatory affairs need to be part of the governance structure, not just consulted after the fact. The accountability model needs to reflect that AI governance is not purely a technology question.

5. Human oversight and approval framework

For decision-support AI — systems that generate recommendations a human then acts on — human oversight is built into the architecture by default. The governance question is whether the human is actually exercising judgement or rubber-stamping the AI's output.

For agentic AI systems, human oversight has to be deliberately designed in. The human-in-the-loop patterns for agentic AI define a spectrum from approval-gated (nothing happens until a human explicitly approves) through supervisory review (the agent acts, a human can reverse it within a defined window) to notify-only (a log exists). The governance pack needs to define where each category of AI action sits on that spectrum, with documented justification.

The critical design choices are:

What actions require approval gates? For agentic systems, the general rule is: any action that is irreversible, any action that affects people's rights or financial position, and any action with significant regulatory exposure should require an explicit human approval before execution. The approval gate only works if the reviewer has enough context to exercise real judgement — the proposed action, the agent's reasoning behind it, the data it's based on, and the reversibility risk. "Click approve or reject" with no context is not governance.

Fail-safe defaults: If an approval request is not acted on within the defined window, what happens? The governance pack should specify that the default is for the action not to proceed — not for it to proceed by default. An unmonitored approval queue is not oversight.

Audit trail for oversight decisions: Every approval or rejection of an AI-proposed action should be logged, with the reviewer's identity and the time of the decision. This is the evidence that human oversight actually happened, not just that it was technically available.

6. Data governance for AI

The Privacy Act creates real obligations for AI systems handling personal information, and those obligations are not satisfied by a general-purpose privacy policy that predates the AI deployment. Each AI system in the inventory that handles personal information needs its own data governance documentation:

What personal information the system collects, uses, or discloses, and on what legal basis under the Australian Privacy Principles
How information collected or used by the AI is disclosed to affected individuals (the transparency requirement)
How individuals can access, correct, or request deletion of their information in the AI context
How long information is retained and how it is disposed of
What third parties (including AI vendors) have access to personal information, and what contractual protections are in place
The process for responding to a data breach involving AI-held or AI-generated information

For organisations using third-party AI platforms, the vendor data processing terms are part of this documentation. The Privacy Act exposure flows through to the deploying organisation regardless of what the vendor's terms say about their own obligations.

7. Monitoring, audit, and review

A governance pack that is created and filed is not a management system. ISO 42001's continuous monitoring requirements reflect what any serious governance regime looks like in practice: ongoing verification that the controls are actually working, not just documented.

The monitoring and audit section of the pack should specify:

Automated monitoring: What metrics are being tracked for each AI system? Model performance, output quality, bias indicators, and — for agentic systems — scope creep (the agent calling tools outside its expected working set) should all be monitored continuously, not just at deployment.

Immutable audit logs: Every action taken by an AI system that has real-world effects should be logged in an append-only store that the AI system itself cannot modify. This is both a governance requirement and a legal protection: if something goes wrong, you need to be able to reconstruct exactly what the system did and in what order.

Internal audit cadence: The governance pack should specify a regular internal audit against the pack's own requirements — at minimum annually, and triggered by significant incidents or changes. The audit should have a named owner and produce a documented output.

Management review: Senior leadership (at minimum the CTO or Head of AI, ideally with board or audit committee participation) should formally review the AI risk register, audit findings, and governance pack currency on a defined schedule. This is the evidence that governance is real, not performative.

Incident response and post-incident review: When an AI system causes harm or near-harm, the response process — who is notified, what is the immediate mitigation, what is the investigation process, and what is the output of the post-incident review — needs to be defined in advance and tested. The post-incident review should update the risk register and, where relevant, the governance pack.

What "board-ready" looks like in practice

A CTO presenting AI governance to a board or audit committee in 2026 needs to be able to answer a specific set of questions clearly and with documentary backup:

What AI systems are we running, and what do they do?
What could go wrong with each of them, and what is the maximum harm if it does?
What controls do we have in place, and how do we know they are working?
Who is accountable for each system's governance, and what authority do they have to act?
What is our regulatory exposure, and how are we managing it?
When did we last review this, and what changed?

The governance pack is the documentation that makes those questions answerable. A board that receives a confident verbal briefing but cannot then inspect a document set is not exercising oversight — it is accepting reassurance. The audit committee in particular needs to be able to point to documentation, not just presentations.

For the CTO personally, the governance pack is also career protection. Being the named accountable executive for AI governance at an organisation that causes AI-related harm — and being unable to demonstrate that you had a documented, enforced governance framework in place — is a very different position to being able to point to a comprehensive pack that was regularly reviewed and updated. The liability picture is sharpening, and documented governance is the evidence of reasonable precautions.

The Australian regulatory context: what is actually changing

Australia does not yet have AI-specific legislation, but the regulatory environment is not static and the governance obligations that exist today are real.

The Privacy Act is the most immediate. The 2024 amendments strengthened automated decision-making obligations, and the Attorney-General's Department has signalled further reforms addressing AI specifically. For any AI system making or influencing decisions about individuals — marketing, credit, hiring, player protection, eligibility assessments — Privacy Act compliance is not optional and is not satisfied by generic privacy policies.

The voluntary AI Safety Standard (2024) sets ten guardrails that the government expects responsible AI deployers to follow. They are not currently mandatory, but they define what "reasonable conduct" looks like from a government perspective. An organisation that has not engaged with the guardrails and cannot demonstrate alignment to them is taking a risk that those guardrails, or something like them, will be mandatory sooner than they expect.

Sector regulators are actively developing positions. ASIC has published guidance on AI in financial services that organisations in that sector need to have read and responded to. The OAIC has issued guidance on privacy and AI. State gaming and lottery regulators are asking questions about AI use in player-facing systems. For regulated-industry organisations, governance that satisfies the voluntary frameworks may not be sufficient — sector-specific overlay requirements need to be understood and documented.

ISO 42001 is the international benchmark. It is not Australian law, but it is what courts and regulators will look to when assessing whether an organisation's AI governance was reasonable. Alignment to ISO 42001 — even without formal certification — demonstrates that the organisation engaged with the recognised international standard for AI management. That is a meaningful difference from "we had a policy we wrote in 2024 and haven't updated since."

This is commentary on the regulatory environment, not legal or regulatory advice. Engage a specialist in Australian technology and AI law if your regulatory exposure is significant.

Prioritising: where to start

A CTO facing the board question for the first time is usually dealing with a gap — there is some governance documentation, but it is not current, not complete, and not integrated into a coherent pack. The path forward is not a large-scale implementation project before the next board meeting; it is a structured gap analysis and a credible roadmap.

Week one: inventory. Do you have an accurate list of every AI system and tool your organisation is using? If not, start there. You cannot govern what you do not know you have.

Week two: risk classification. For each item in the inventory, assign a risk classification using documented criteria. High-risk and critical systems become the initial governance scope.

Week three: accountability. For every high-risk system, is there a named owner? Does that person know they are accountable and have the authority to act? If not, that is the most urgent gap to close.

Week four onwards: documentation and controls. Work systematically through the policy, risk register, and oversight framework for the systems in scope. The documentation should reflect what is actually happening, not an aspirational future state — a governance pack that describes controls that do not exist is worse than no governance pack, because it is documentary evidence of a gap you knew about and misrepresented.

Ongoing: monitoring and review. Build the review cadence into operational practice from the start. A governance pack without a review cycle is a point-in-time document, not a management system.

The agentic AI dimension: why the governance stakes are higher

The governance principles above apply to all AI. But organisations deploying agentic AI — systems that take actions rather than generate recommendations — face a materially higher governance bar, because the harm from an agentic system that behaves unexpectedly can happen before any human has the opportunity to intervene.

MCP-connected agents that have access to external systems — customer records, communications, financial data, operational tooling — are the category that most urgently needs robust governance. The blast radius question is not hypothetical: an agent with write access to systems it does not need is a persistent risk regardless of how well it normally behaves.

The human-in-the-loop governance patterns for agentic AI — least-privilege access, approval gates for irreversible actions, immutable audit logging, scope creep detection — are the operational controls that an ISO 42001-aligned management system would document and monitor. For an organisation deploying agentic AI in a regulated industry context, these controls are not optional and they need to be documented in the governance pack, not just implemented in the system architecture.

A key question for any agentic AI deployment that the governance pack should be able to answer: if this system behaved unexpectedly right now, how would we know, what would we do, and what is the rollback capability?

If that question cannot be answered from the governance pack, the governance pack is incomplete.

FAQ

What is an AI governance pack?

An AI governance pack is the structured set of documents, policies, and registers that an organisation uses to demonstrate that its AI systems are being managed, risk-assessed, and overseen in a responsible and accountable way. It typically includes an AI policy, a risk register for AI systems in use, an inventory of AI assets, accountability assignments, incident response procedures, and evidence of ongoing monitoring. In 2026, boards, enterprise procurement teams, and sector regulators in Australia are beginning to ask to see these documents — not as a compliance formality, but as a genuine indicator of whether an organisation has its AI posture under control.

What should be in an AI policy for an Australian organisation?

An AI policy for an Australian organisation should at minimum cover: which AI systems the organisation uses or builds and what they are used for; who is accountable for AI risk at an executive level; how AI-related decisions are made and escalated; how compliance with the Privacy Act's requirements around automated decision-making is maintained; alignment with the Australian Government's voluntary AI Safety Standard guardrails; and the process for reviewing the policy when the AI estate or the regulatory environment changes. It should be a living document reviewed at least annually, not a static statement.

Does an Australian CTO need to worry about ISO 42001 in 2026?

Yes, for several practical reasons. ISO 42001 is increasingly appearing in enterprise procurement requirements and insurance underwriting assessments. It provides a credible, internationally recognised framework that maps closely to the Australian Government's voluntary AI Safety Standard. And it is the benchmark that regulators, courts, and counterparties will look to when assessing whether an organisation exercised reasonable care in governing its AI systems. You do not need to be certified to benefit from ISO 42001 alignment, but the gap analysis is worth doing — most organisations with any existing governance infrastructure are closer to alignment than they assume.

What is the Australian AI Safety Standard and does it apply to my organisation?

The Australian Government's voluntary AI Safety Standard, released in 2024, sets out ten guardrails for responsible AI development and deployment. These cover risk accountability, transparency, human oversight, testing before deployment, and incident response. It is currently voluntary — there is no legislative mandate. However, it signals what the government considers responsible practice, and it is the document regulators and courts will reference when assessing whether an organisation took reasonable precautions. It applies to any Australian organisation developing or deploying AI, including those using third-party AI tools and platforms.

What does the Privacy Act require for AI systems making decisions about people?

The Privacy Act 1988 (Cth) and the Australian Privacy Principles (APPs) create obligations that apply when AI systems collect, use, or disclose personal information — which covers the vast majority of consequential AI applications. Specific obligations include: notifying individuals about how their information is used in automated decision-making; providing meaningful access and correction rights; only using information for the purpose it was collected; and applying the minimum necessary collection principle. The 2024 Privacy Act reforms strengthened some of these requirements, and further AI-specific reforms are under development. AI systems making consequential decisions about people — credit, employment, player protection, marketing eligibility — face the most scrutiny.

What does an AI risk register need to cover?

An AI risk register for a credible governance pack should cover: each AI system or tool in scope; the function it performs and who it affects; the identified risks (bias, adversarial vulnerability, data leakage, agentic blast radius, model drift, regulatory exposure); the likelihood and consequence assessment for each risk; the controls in place to mitigate each risk; the named owner accountable for each risk; and the review cadence. For agentic AI systems that can take actions rather than just generate recommendations, the risk register needs to explicitly address the blast radius question — what can the system do if it behaves unexpectedly, and what is the rollback or reversal capability?

How does AI governance for agentic systems differ from governance for decision-support AI?

Decision-support AI generates a recommendation that a human then acts on — the human is in the loop by default because the action hasn't happened yet. Agentic AI takes actions directly: it can send emails, write to databases, call APIs, and execute operations against production systems. Governance for agentic systems has to address blast radius (what can the system do if it misbehaves?), least-privilege access (does the system have only the permissions it needs for its specific task?), approval gates (which actions require explicit human authorisation before execution?), and immutable audit logging (can you reconstruct exactly what the system did and why?). The ISO 42001 operational controls framework maps directly to these requirements.

Share this article:

X LinkedIn Facebook WhatsApp Bluesky Reddit Email