Introduction
Governance was not a gray area in software development. Engineering teams had frameworks for aligning their efforts strategically with the business: coding standards, deployment process, data policies, security protocols, testing environments, you name it. These were well-understood, documented, and enforced by engineering leaders.
However, with AI being an active contributor to software development through agentic workflows (writing code, creating infrastructure, etc.), governance now has to cover agents too.
The question is: what does governance in the age of AI look like?
Why governance matters more now
First, what does governance actually mean? It is a set of rules and processes that determine how changes are made to your systems and who is accountable for them. For traditional software engineering, this was well-understood: code reviews, CI/CD pipelines, access controls, deployment approvals. For AI, most teams haven’t established these new rules yet.
Anthropic recently published a Zero Trust Framework for AI Agents that treats AI agents as a new class of identity in the enterprise, alongside humans and service accounts.
The key insight: agents authenticate, hold credentials, and act autonomously, which makes each one a privileged non-human identity. According to a 2026 State of AI Agent Identity Security report, more than two-thirds of organisations suspect AI agents have already accessed data beyond their intended scope.
Governance has become more critical because AI introduces risk at a different scale. For example, an engineer writing code might introduce a bug in one service. An AI agent running autonomously can modify multiple services and infrastructure, creating accidental complexity, even if the solution works. Risk is caused by the speed of execution and non-deterministic output.
Traditionally, the output of engineers was deterministic. Code gets pushed in a feature branch, goes through a CI/CD pipeline (lint, tests, security scans, build, deploy). QA/Security engineers test the new code in a sandboxed environment (staging), signal green for production, and then engineers trigger automated pipelines to deploy to production. This old form of governance is already embedded in every engineering team with a set of guidelines defined by technical leadership. Now we have AI tools like Claude Code and Cascade. We have files like AGENTS.md, which are essentially governance documents for AI agents: what they can and can’t do, which workflows to follow, which files to avoid.
Teams have different opinions on how AI should be governed. The foundation makes sense, don’t deploy AI-generated code to production without a thorough review. However, architecture has a bigger blast radius if AI is not properly governed. An engineer prompting AI to write a script to modify/provision AWS services might cause significant damage if not properly reviewed, both money-wise and reputation-wise.
Governance matters more now because, if your team is full-on using AI, there should be complete transparency on the actions and artifacts produced by AI. If an engineer is modifying architecture through AI, it has to be logged, verified, and safely approved. Even when AI-generated code passes some guardrails (e.g., tests, lint, type-checks), it can still introduce accidental complexity: design patterns that are harder to read, debug, or extend.
What governance actually looks like for architecture
Anthropic’s guide argues that AI agents should be treated as non-human identities, not just tools. Each agent that interacts with your architecture authenticates, holds credentials, and acts. This matters for architecture governance specifically. An AI agent connected to your architecture model through MCP has read and write access to your system’s source of truth. What can it modify? Which objects can it create or delete? Can it merge a draft without human approval? These are identity and access questions, and they need the same governance rigour you apply to engineers accessing production.
To make this concrete, here are examples of what we think governance should look like for architecture.
1. Adopting architecture decision records (ADRs)
Every decision relevant to a product or an architecture has to be captured with an ADR. How you format the document doesn’t matter, as long as it is clear to future readers on A) problem, B) discussed solutions, and C) approved solution, and outcome.
You might argue this is more documentation, but having a monotonic sequence of architecture decisions stored in a repository will protect teams from any scope creep and/or unintentional changes made by AI. The latest commit in this repository should mark the last decision made in the architecture, and AI is governed through its interaction with this repository. Not through a terraform/cloud-formation repo. ADRs are also structured contexts for AI agents to reason about the current architecture without guessing.
2. Workflows continuously reviewed
“Continuously reviewed” is intentional here. AI workflows for engineering are evolving fast and what works today might be outdated in six months. Teams should regularly review how AI interacts with their architecture and update the guardrails accordingly.
This means treating architecture changes from AI the same way you treat code changes from engineers. A PR-style review process for architecture gives teams a checkpoint before anything hits the live model. For example, an AI agent proposes a change in a draft. An architect reviews the diff, leaves comments, and approves or rejects it. The workflow mirrors what engineering teams already do with code. The difference is that the contributor might be an AI agent instead of a human. The review standard should be the same regardless.
IcePanel supports this workflow with a feature called Drafts. It allows architects (or AI agents) to create a draft of a live architecture, have it reviewed by others, and merged into the architecture.
3. Auditability and traceability
ADRs capture the why, but they don’t explain what happened after the decision was made. Did the AI agent modify what was approved, or something else entirely? Without auditability, governance is a set of rules nobody can verify.
Engineers already think about observability (o11y) in their applications. There are three main pillars in o11y: metrics, logs, and traces.
- A metric is a numerical value measured over time (e.g., request rate). It tells you when a change happens.
- A log is a timestamped record that highlights a certain event with its context. It tells you what happened.
- A trace is an end-to-end journey of a single request across the system. It explains where and how a journey occurred.
The same principles apply to architecture governance. Metrics tell you when your model changed and how frequently. Logs tell you what changed, who changed it, and whether it was a human or an AI agent. Traces show you the full journey of a decision: from the initial proposal, through draft review, to merge and deployment. For AI-governed teams, audit and traceability should be built into the workflow, not added after incidents come up.
Where governance breaks down
If there are no guardrails between AI and architecture, maintaining systems will be chaos.
This might sound counterintuitive, but governance can sometimes break down if it becomes too restrictive for engineering (e.g., you cannot use AI for project planning). Engineers are creative and will find a way around it. These guidelines should exist for their benefit, not to frustrate engineers.
A good litmus test for governance maturity: if your organisation has not rolled back an AI agent in production, ask whether that reflects stability or visibility. The goal of governance isn’t to prevent rollbacks. It’s to make sure rollback is the worst-case outcome rather than silent damage accumulating where nobody can see it. Mature teams roll back more often because they detect problems earlier, not because they fail more.
Conclusion
We talked about cognitive surrender in our previous post when engineers stop forming their own view of the system. Lack of governance is the same problem, but at a higher level. If AI is making unreviewed changes to your applications or architectures, that’s cognitive surrender at the org level.
AI is making architecture faster to produce and easier to get wrong. Governance should extend to cover what AI produces. AI offers a lot of potential, but without guardrails in place, teams will deal with accidental damage, inconsistent architecture, and eventually be forced to slow down anyway. The teams that govern their AI workflows well will ship faster, stay creative, and actually trust what comes out.
📚 Resources
- https://claude.com/blog/zero-trust-for-ai-agents
- https://icepanel.io/blog/2026-05-05-adopting-adrs-in-enterprise
- https://docs.icepanel.io/future-state-design/drafts
- https://docs.icepanel.io/integrations/mcp-server
- https://icepanel.io/the-icepanel-loop
- https://www.gravitee.io/blog/state-of-ai-agent-security-2026-report-when-adoption-outpaces-control
