Governing the black box: a board playbook for AI risk.

For most of its enterprise history, AI was a model-team concern. The risks were technical, the failures were quiet, and the regulatory envelope was vague enough that "human in the loop" sufficed as a defensible posture. That posture is no longer defensible — not under the EU AI Act, not under the FCA's 2026 guidance, not under SR 11-7 as it is now being interpreted, and not, in our view, under the fiduciary standards that boards already owe their stakeholders.

The question facing boards is no longer whether to attend to AI risk, but how. This paper offers a practical playbook, oriented to the responsibilities that cannot be delegated.

1. Three things that have actually changed

Boards are often presented with AI risk as if it were entirely novel. Most of it is not. What has changed, materially, is the following:

• Decisions are now made at machine cadence. A flawed model used to produce a flawed quarterly report. A flawed agent today produces ten thousand flawed client interactions before lunch. The damage function is no longer linear in time.
• The supply chain has lengthened and concentrated. A typical agentic deployment depends on four to six external providers — a foundation model, an inference platform, an orchestration layer, embedding providers, vector stores. Vendor risk is now systemic risk.
• Explainability has degraded. Traditional models could be interrogated. Frontier models cannot be, in any complete sense. Boards must govern systems that cannot fully explain themselves — a structural condition, not a temporary engineering gap.

2. The five questions every board should be able to answer

We use the following five questions as a diagnostic in our board engagements. Most institutions can answer two of them confidently. The minority that can answer all five are also, with high consistency, the minority that are deploying AI at scale without incident.

2.1 Which decisions has the institution delegated to a machine?

A board should have, on demand, a current inventory of automated and agentic decisions — what they affect, who owns them, what their failure mode looks like. The fact that most institutions cannot produce this list inside a week is the first finding.

2.2 What is the worst-case outcome of each, and how is it bounded?

Every consequential AI decision should have an articulated worst case and a control — preferably a non-AI control — that bounds it. "We trust the model" is not an answer; it is an abdication.

2.3 Who is accountable when it goes wrong?

The Senior Managers Regime and its global equivalents already require named accountability for material decisions. Regulators are explicit: the use of an AI system does not transfer accountability to the vendor or to the model. It remains with the human who approved the deployment.

2.4 Could the institution operate if the primary provider failed?

The concentration of foundation model providers is now a financial stability concern. Boards should require, at minimum, a credible operational plan for a 72-hour outage of the largest provider in use. Most plans we have reviewed are aspirational.

2.5 How does the institution learn from its AI's mistakes?

Conventional incident management assumes a human error to interview. AI incidents require a different process — one that combines log forensics, prompt and tool-call replay, and structured updates to guardrails. Boards should ask to see this process in action, not merely on paper.

3. The governance architecture that actually works

Across the institutions we advise, three structural elements consistently distinguish governance that holds up under stress from governance that collapses on first incident:

1. A single accountable executive, not a committee. Committee accountability is no accountability. The best arrangements we have seen vest end-to-end AI risk in a named executive reporting directly to the CEO, with clear escalation rights to the board.
2. An independent assurance function with the right to test models in production, not only at deployment. The pace of model change is now sufficient that point-in-time validation is obsolete.
3. A standing board agenda item, at every meeting, not annually. AI risk is now of comparable consequence to credit and market risk; the cadence of board attention should reflect that.

4. The fiduciary frontier

We close with a question that is, in our view, the most important and the least discussed: as AI-driven advice and decisioning proliferate in wealth management, what does fiduciary duty mean?

The traditional standard — that the adviser act in the client's best interest with the care of a prudent professional — assumed a human adviser whose reasoning could be examined. When advice is generated by a system whose reasoning cannot be fully examined, the standard does not disappear; it migrates upward. Fiduciary duty becomes a duty to govern the system that gives the advice, with the same care the adviser would have owed to giving it directly.

Institutions that internalise this migration early will, we suspect, find themselves at a durable advantage — both in regulatory standing and, more importantly, in the quality of trust they offer their clients in an era when trust is the only product that is not being commoditised.

This paper draws on board and risk-committee engagement work conducted between October 2025 and March 2026, including observation of incident reviews at six institutions across three jurisdictions.