When AI starts acting for you: the governance question every organisation needs to answer

Last Tuesday at 2.12pm, an AI agent approved a £40,000 supplier credit. The supplier’s agent requested it. A procurement orchestrator passed it to a finance sub-agent, which paid it.

Was that the right decision? Who authorised it? And could you prove what happened?

That hypothetical scenario opened our latest webinar for senior executives across our leadership Communities on Governance and AI Agent Identity, led by Ella Ovenden, Senior Data & AI Strategy Consultant. It captures a challenge that will become increasingly important as organisations move beyond individual AI tools towards networks of agents that can make decisions, access systems and take action on our behalf.

The conversation around AI governance has often focused on models, data and responsible use. Agentic AI introduces something different. When AI can act rather than simply advise, organisations need to know who or what is acting, what authority it has, who remains accountable and how quickly it can be stopped.

The question is no longer simply whether you can build an AI agent, but whether you can trust it to operate inside your business.

Every new agent creates a new trust boundary

Many organisations are still operating in a relatively simple world. An employee asks an internal agent to perform a task and that agent accesses a system or tool.

But the interaction model quickly becomes more complicated.

A customer might use their own agent to communicate with yours. An internal orchestrator might delegate tasks across several specialist agents. Those agents could access different systems or communicate with agents belonging to suppliers, customers and partners.

Every time that chain crosses a boundary, another question of trust appears.

As Ella explained, identity verification, authentication and authorisation become critical because permissions cannot simply travel unchecked from one agent to another. Organisations need to know where authority originated, what has been delegated and who ultimately owns the outcome.

This creates an important strategic question: how much of your organisation do you actually want to become agent-first?

The answer does not have to be everything.

Some interactions will benefit enormously from autonomous agents. Others may deliberately retain human involvement because judgement, empathy, ethics or accountability matter more than efficiency. The challenge for leaders is designing those choices consciously rather than allowing the technology to make them by default.

An AI agent needs its own identity

One of the clearest principles from the session was also one of the simplest: an AI agent should never pretend to be a person.

Giving an agent an employee’s credentials might be convenient, but it also gives that agent the permissions associated with that employee. Unlike a person, an agent can potentially exercise those permissions across thousands of actions at enormous speed. Instead, agents need their own durable identities. AI agent identity means giving each agent its own credentials and an accountable human owner, separate from any employee's access.

Ella used the analogy of treating an agent like a new employee. Before somebody joins an organisation, we establish what they are there to do, who manages them, what systems they can access and what happens when they leave.

Agents need many of the same basics: a job description defining their purpose and boundaries, a named human accountable for their outcomes, the right organisational context, a controlled period before greater autonomy is granted and a proper leaver process that revokes credentials when the agent is decommissioned.

But the analogy only goes so far. Humans learn from experience, social cues and consequences. An agent can make the same mistake repeatedly, at speed and with complete confidence. If nobody can see that failure happening, the potential impact multiplies.

That changes how organisations need to think about access. Rather than permanent permissions, agents should receive short-lived access scoped to the task they are performing. Delegated actions need to remain traceable and organisations need agent-level records showing what happened, when it happened and on whose behalf.

Accountability still belongs to humans

This was perhaps the most important point of the discussion. AI can perform an action. It cannot ultimately accept accountability for it.

Every agent therefore needs a named human owner. If an organisation cannot identify who is responsible when an agent makes a decision, the governance problem started long before the decision went wrong.

That also creates a broader challenge for executive teams because AI does not sit neatly within one traditional function.

The Chief Information Security Officer has a role in identity, permissions and security. Chief Technology/Chief Information Officers own platforms and architecture. Chief People Officers need to think about skills, job design and blended human-agent teams. Chief Operating Officers need to understand how agents change processes and decision-making. Chief Finance Officers will inevitably care about value, cost and risk.

During the discussion, one attendee described the answer as a form of shared accountability across the executive team.

Ella agreed that organisations are still working out exactly what good looks like, but the direction is becoming clearer. AI cannot be treated as another isolated technology programme. This is why our Intelligent Enterprise framework treats Artificial Intelligence as one of four intelligences that organisations develop together. Leaders need enough AI literacy to make informed decisions together and clear accountability for what happens when agents move across functional boundaries.

That is particularly important as people move away from simply executing tasks towards overseeing agents, handling exceptions, making judgement calls and orchestrating outcomes.

AI agent governance has to run the whole lifecycle

Traditional governance can sometimes feel like something that happens before a system launches: review it, approve it and move on. That approach does not work for agents.

Governance needs to continue throughout an agent’s lifecycle and, importantly, it needs to be proportionate to risk.

An internal agent drafting product copy should not face the same controls as an autonomous agent approving payments or accessing sensitive customer information.

S&S frames this through six governance layers: agent identity, scoped authority, blast radius, in-path controls, traceability and termination. Together, they answer some very practical questions. 

• Who is this agent?
• What can it do?
• How much damage could it cause if something goes wrong?
• Where does a human need to intervene?
• Can we reconstruct exactly what happened?
• And can we switch it off immediately?

That final question is easily overlooked.

Organisations often focus heavily on how agents enter the business but much less on how they leave. Credentials can remain active, ownership can disappear when employees move roles and unused agents can continue sitting within the technology estate.

A functioning ‘kill switch’ is not an edge case. It is a fundamental part of operating agents safely.

Don’t automate a process that needs redesigning

Governance is only half of the equation. As organisations look for opportunities to introduce AI, the natural temptation is to map an existing process and ask where agents could automate individual steps.

That risks making an inefficient process faster without questioning why the process exists in that form in the first place.

Many processes were designed around human constraints. Work gets batched because bringing people together takes time. Reviews happen weekly because analysis historically took days. Approvals exist because humans can only process so much information. Agents change some of those assumptions.

As Ella explained, organisations should therefore focus on process redesign, not simply process automation.

The session introduced six tests for determining whether a process is genuinely ready for agents. It needs to be observable, specifiable, bounded, recoverable, actionable and measurable. In other words, you need to know what is happening, define what good looks like, establish clear limits, understand what happens when something goes wrong and measure the outcome.

If you cannot clearly write the agent’s job description, you probably should not build the agent yet.

The question that determines whether agents can scale

The session finished where it began: with traceability.

Pick an action an agent performed last week. Can you identify exactly which version of the agent performed it? Which prompt and tools were available at that moment? Which identity and permissions did it use? Which process step was it performing? Who owned the outcome?

If one of those links disappears, you have a problem.

And this is where governance should stop being viewed as something that slows AI adoption down. Done properly, it does the opposite.

Traceability, clear identities, scoped permissions and defined ownership create the confidence to move agents beyond isolated experiments and into meaningful business processes. Without them, organisations may be able to build impressive pilots, but scaling them safely becomes much harder.

The pace of change is only increasing. Nobody has every answer yet and, as the discussion showed, organisations are still working through where accountability sits and what new capabilities they will need.

But there are questions leaders can start answering now.

• Do you know how many agents are operating across your organisation?
• Does every one have an owner?
• Do they have their own identities and appropriately scoped permissions?
• Could you reconstruct what one did last Tuesday afternoon?
• And, if necessary, could you switch it off today?

Because as agents become part of the workforce, good governance will not be something organisations add once AI has scaled. It will be what allows them to scale in the first place.

Continue the conversation

If your organisation is starting to explore AI agents, agent identity or how governance needs to evolve as AI becomes more embedded in your operating model, we’d love to continue the conversation. Get in touch with Sullivan & Stanley to speak with Ella and our team about what these principles could mean in the context of your organisation.