Express course · No. 37
AI increasingly makes decisions that affect people's lives — and the law and your own responsibility are catching up fast. AI governance is the discipline of building systems you can stand behind: knowing where you're high-risk, showing how the system decides, keeping a human accountable, and being ready when a regulator or customer asks. This is an orientation, not legal advice.
19 concepts · a diagram for each · cheat sheet at the end · not legal advice
First, why there are rules at all. AI is making decisions that affect people's lives, and both the law and plain responsibility are arriving fast. Ignoring that is a risk, not a strategy.
AI Governance — the discipline
A pilot is responsible for the flight even on autopilot. AI governance is the same idea for software: the system helps, but someone can always answer for it.
AI governance is the discipline of building AI responsibly and being able to stand behind it. In practice it comes down to four habits: know your risk tier, document and log, keep a human accountable, and be transparent. Almost everything in this course unpacks those four.
- What it is
- The discipline of building AI responsibly and standing behind it.
- How it works
Know your risk tier → document and log → name an accountable human → be honest that it's AI.
- Example
Before launching credit scoring, the team answers: what tier is this, where is it documented how it decides, where is the decision log, who can reverse a decision.
- Catch
It isn't paperwork bolted on at the end — it's mostly unglamorous engineering: logs, docs, oversight.
- Related
- The umbrella over every other concept in the course.
Consequential AI — AI that decides
The difference between a tool that suggests a word and one that decides who gets a loan. Once the stakes touch people's lives, the bar for responsibility rises.
AI used to autocomplete text. Now it screens job applicants, scores credit, flags medical conditions, and moderates speech. As AI moves from suggesting to deciding things that materially affect people, the cost of getting it wrong rises — and so does the expectation that you can account for how it works. That's why governance matters now.
- What it is
- AI that makes, or heavily shapes, decisions about people.
- Example
Autocomplete is a suggestion. CV screening or credit scoring is a decision about someone's life.
- Catch
The line is easy to miss: a feature "just recommends", but people follow the recommendation blindly.
- Related
Such systems usually land in High-risk AI; they are exactly what the EU AI Act targets.
The law is catching up, fast
New traffic laws appear once cars are everywhere and crashes have consequences. Regulation follows a technology into the places it can do harm.
Governments are regulating AI in earnest: the EU AI Act is the landmark, and dozens of jurisdictions are following with their own rules. The era of "move fast, no rules" is closing for consequential AI. If your product reaches users in regulated places or makes decisions regulators care about, compliance is a real requirement with real penalties. Knowing the shape of these rules is now as much part of the job as knowing security and privacy law.
This is orientation, not legal advice
A map of the terrain helps you plan the route, but you still bring a guide for the dangerous crossing.
This course is an orientation to how AI governance works, not legal advice. The goal is to make you fluent in the concepts and risks, so you know what to watch and when to get real experts. For an actual high-risk system you involve people who do compliance for a living — and it's far better to learn the obligations early than too late.
The map: 19 concepts in six groups
A metro map: first you see the whole lines, then you ride station by station.
The concepts fall into six groups: why governance, risk tiers, the EU AI Act, proof (documents and logs), accountability, and responsible AI. The last chapter turns it all into practice. Every concept then gets the same treatment: English name, a picture, a diagram, and a card — what it is → example → catch → related.
AI now makes consequential decisions, and law and responsibility are catching up fast. Governance is building AI you can stand behind. An orientation here, not legal advice — but no longer optional.
The core idea of governance: not all AI is treated the same. How much oversight you owe depends on how risky the use is. So the first job is knowing your tier.
Risk-based Approach — obligations scale with risk
A go-kart and a passenger jet are both vehicles, but nobody regulates them the same. The rules scale with how much harm a failure could do.
Sensible regulation sorts AI by risk. A spam filter and a system that decides who gets parole are both AI, but the oversight they need is worlds apart. This is risk tiering: the higher the potential harm to people, the more obligations apply. It sits at the centre of the EU AI Act and most serious frameworks. Most of governance is working out which tier your use falls into, because that decides what you actually owe.
- What it is
- Obligations grow with the potential harm to people.
- Example
Spam filter — almost no duties. Parole decision — documentation, oversight, risk management.
- Catch
The tier is set by the use, not the technology: the same model can be low-risk in a chatbot and high-risk in hiring.
- Related
The basis of the EU AI Act; splits the world into High-risk and Lower-risk AI.
High-risk AI — uses that shape lives
The activities with the most safety rules are the ones where mistakes hurt people most: surgery, aviation, finance. AI follows the same logic.
Certain uses are classed as high-risk because a mistake materially affects someone's life: hiring and firing, credit and lending, education, healthcare, law enforcement, access to essential services. AI that makes or heavily influences those decisions carries real obligations — documentation, human oversight, risk management, the ability to show a regulator how it works. If your system touches one of these areas, assume it's high-risk until you've confirmed otherwise.
- What it is
AI whose mistakes materially affect a person's life, rights or access.
- Example
- A model that screens CVs, or a score that decides who gets a loan.
- Catch
This is where the weight of regulation lands — "we'll sort it later" fails.
- When it applies
You touch hiring, credit, education, health, law enforcement or essential services — assume yes.
- Related
Requires Documentation, Audit trail, Human oversight, Explainability.
Lower-risk AI — most AI, but know which
Most everyday tools don't need a special licence; only the truly dangerous ones do. The skill is telling which is which before you ship.
The reassuring part: most AI uses are not high-risk. A writing assistant, a recommendation feature, a chatbot answering routine questions — they face much lighter obligations, often just basic transparency. Governance is heavy where stakes are high and light where they aren't. The essential move is honest classification: don't assume you're safe, and don't panic that everything is regulated.
- What it is
AI whose mistakes don't materially change a person's life; duties are mostly transparency.
- Example
- A writing assistant, product recommendations, an FAQ bot.
- Catch
It's a conclusion after an honest check, not an assumption. "We're just a chatbot" — until the chatbot starts deciding for people.
- Related
- The opposite of High-risk AI; baseline is Transparency.
Not all AI is regulated equally — obligations scale with risk. High-risk uses (hiring, credit, health, essential services) carry real duties; most AI is lighter. The first job is honestly knowing your tier.
The EU AI Act is the landmark AI regulation, and its shape is becoming the global reference. You don't need the legal text — you need its structure, because most of the world is following it.
EU AI Act — four tiers, rules per tier
A building code that bans some structures outright, inspects the tower block down to the bolt, and barely touches the garden shed.
The EU AI Act sorts AI by risk and applies rules per tier. A small set of uses is prohibited outright. High-risk systems carry heavy obligations — documentation, oversight, risk management — before they can be deployed. Most of the rest faces transparency duties — telling people they're dealing with AI — or very little. Remember this pyramid and you understand most of the Act.
- What it is
The EU regulation that sorts AI into risk tiers with rules for each.
- How it works
Prohibited → banned. High-risk → duties before deployment. Transparency → disclose it's AI. Minimal → almost nothing.
- Example
Credit scoring for EU customers is high-risk; a support chatbot is mostly transparency.
- Catch
It applies if your product reaches users in the EU — even if your company isn't in Europe.
- Related
The Risk-based approach made law; spreads through the Brussels effect.
Prohibited Practices — banned outright
Some things don't get a permit no matter how much paperwork you file.
The top of the pyramid is prohibited practices: manipulative or abusive AI and certain kinds of surveillance. There's no compliance path here — no documentation makes them legal. You simply don't build them.
- What it is
- AI uses that are banned outright.
- Example
- Manipulative or abusive AI, certain kinds of surveillance.
- Catch
- Nothing to "tune" here — the only right answer is not to do it.
- Related
- The top tier of the EU AI Act pyramid.
Penalties — the fines are serious
A fine big enough that a company can't shrug it off — the kind that makes the board take the rule seriously.
The Act has real teeth: fines for serious violations reach tens of millions of euros or a significant percentage of global annual turnover, whichever is higher. These aren't token fines — they're sized to make compliance a genuine business priority. That's exactly why AI governance moved from nice-to-have to a board-level concern.
- What it is
Fines up to tens of millions of euros or a percentage of global turnover — whichever is higher.
- Example
A high-risk system serving EU users without the required documentation and oversight is a direct route to a fine.
- Catch
- Gambling against compliance costs more than planning for it.
- Related
- Make Governance by design profitable, not just right.
Brussels Effect — the global benchmark
A standard set in one big market that the whole industry adopts, because it's easier to build one compliant product than a different one per region.
Like the EU's earlier data-privacy rules, the AI Act shapes practice far beyond Europe. This is the Brussels effect: a major market's regulation becomes the de facto global standard, because companies build to the strictest rule instead of maintaining many versions. And many other jurisdictions are writing their own AI rules in a similar risk-based shape. So even outside the EU, the Act's structure is a good guide to where AI regulation is heading.
- What it is
A big market's rules become the world standard, because that's cheaper.
- Example
EU data-privacy rules changed products worldwide; the AI Act is doing the same.
- Catch
"We're not in the EU" is a weak argument — the shape of the rules arrives anyway.
- Related
- Carries the EU AI Act and the Risk-based approach worldwide.
The EU AI Act tiers AI by risk: a few uses prohibited, high-risk ones carrying heavy obligations, most facing only transparency. Penalties are serious, and its tiered shape is becoming a global benchmark.
Most high-risk obligations come down to one practical requirement: be able to show how your system works and what it did. That's documentation and logging — and it's far cheaper to build in than to bolt on.
Technical Documentation — show how it decides
A recipe written down so anyone can see exactly how the dish is made — not a chef who shrugs and says it just comes out right.
A key obligation is documentation: being able to explain how your AI system works — what data it uses, how it makes decisions, what its limits and risks are. Regulators and customers increasingly expect you to show this, not just claim the system is fine. A high-risk system that's a black box even to its makers is exactly what the rules target. You write down how it works as you build, turning "trust us" into "here's the record".
- What it is
A written account of the system's data, decision logic, limits and risks.
- Example
For a CV-screening model: which CVs it was trained on, which signals it uses, where it fails, how that was tested.
- Catch
- Writing it from memory after launch is slow and inaccurate.
- Related
Pairs with Audit trail ("how it decides" + "what it did"); the basis of Explainability.
Audit Trail — log what it actually did
A flight recorder that captures every action, so after an incident there's an exact record to study instead of a guess.
Alongside documentation you keep an audit trail — a log of what the system actually did: the decisions it made, the inputs, the outputs, who was involved. When something goes wrong, or a regulator or affected person asks "why", the audit trail is the difference between an accountable answer and a shrug. It's the same logging discipline as observability, here serving accountability.
- What it is
A log of AI decisions: inputs, outputs, time, which humans took part.
- Example
"09:31, CV #4814, rejected, reviewer overrode" — and the answer to the candidate is ready.
- Catch
- Without the log you reconstruct and guess instead of answering.
- Related
A sibling of observability; helps close the Accountability gap and improve the system.
Governance by Design — build it in, don't retrofit
Wiring a building to code as you construct it is routine; tearing open finished walls to add wiring afterwards is a nightmare.
The practical lesson is governance by design: build governance in from the start. Adding documentation and audit logging to a system designed without them means reconstructing how it works after the fact — slow, error-prone, sometimes impossible. Teams that treat "how will I show how this works and what it did" as a design requirement, next to the feature itself, get most of the compliance work nearly for free.
- What it is
Logs, docs and oversight are design requirements, not an end-of-project add-on.
- Example
The schema has a model-decisions table from day one; the PR template has "update the system description".
- Catch
The cheapest time to build governance in is before you need it, not under a regulator's deadline.
- Related
- How you get Documentation and Audit trail almost for free.
High-risk obligations mostly reduce to two things: show how the system decides (documentation) and log what it did (audit trail). Build both in from the start — retrofitting accountability is painful and sometimes impossible.
Autonomous AI raises a hard question the law is still settling: when AI acts on its own and causes harm, who is responsible? Don't let that ambiguity become your defense.
Accountability Gap — who answers for the harm?
A self-driving delivery cart hits someone. Is it the maker, the operator, or whoever sent it? Each one points at someone else.
AI agents act on their own — booking, buying, deciding, sending. That opens an accountability gap: when an autonomous system causes harm, it isn't always clear who is responsible — the model developer, the company that deployed it, or the user who set it loose. The law is actively working this out, and it isn't settled. For you as a builder, the ambiguity itself is the risk. You want the answer to "who's responsible?" inside your product, before it gets answered for you in a dispute.
- What it is
The ambiguity over who is responsible when autonomous AI causes harm.
- Example
An agent buys the wrong thing and sends money to the wrong party — developer, company and user all point at each other.
- Catch
- The law hasn't decided — so you have to decide inside your product.
- Related
- Closed by Human oversight and Audit trail.
Human Oversight — a human stays accountable
A pilot stays responsible for the flight even on autopilot: the automation helps, but a named person answers for the outcome.
A durable principle, written into the rules for high-risk systems, is human oversight: a human stays accountable for consequential AI decisions. The AI can assist, recommend, even act — but a person or organisation remains answerable for the result and can oversee and intervene. It's the governance face of human-in-the-loop from product design: keeping a human meaningfully in charge isn't just safer — it's how accountability stays located somewhere when the system gets something wrong.
- What it is
- A named human answers for AI decisions and can step in.
- Example
The model proposes rejecting a loan application; an officer sees the reason and can override.
- Catch
Oversight must be meaningful: a human who clicks "OK" on everything isn't oversight.
- Related
- Closes the Accountability gap; required for High-risk AI.
Blame Offloading — 'the AI did it' is not a defense
You can't excuse a bad outcome by blaming the tool you chose to use. Responsibility for using it well stays with you.
The mistake to avoid is blame offloading — treating AI as a way to shed responsibility. "The AI made the decision" won't be an acceptable answer when an autonomous system causes harm: responsibility flows to the people and organisations that built and deployed it, however easy the technology made it to look away. So design for it: be explicit, inside your product, about where human accountability sits, and don't build a system where everyone can plausibly claim it wasn't their call.
- What it is
- Trying to shift responsibility for harm onto the AI itself.
- Example
After a wrongful rejection, the company says "the algorithm decided" — which doesn't take the responsibility off it.
- How to avoid
An explicit owner for every consequential AI decision in the product.
- Related
The anti-pattern to Human oversight; a symptom of the Accountability gap.
Autonomous AI opens an accountability gap the law is still resolving. Keep a human meaningfully accountable and able to intervene — because "the AI did it" won't be a defense when an agent causes harm.
Beyond the legal obligations sits a broader practice of building AI responsibly — and much of it doubles as both compliance and the trust that makes a product succeed.
Transparency — tell people it's AI
A label saying "automated reply", so nobody is fooled into thinking a person handled it. Honesty about what they're dealing with.
A baseline expectation across regulations is transparency: people have a right to know when they're interacting with AI rather than a human, and when content is AI-generated. Disclosing that a feature is AI isn't just a compliance box — it's the honest framing from product design that keeps users calibrated. Hiding that something is AI, or passing AI output off as human, erodes trust and increasingly breaks the law. Here good product design and good governance point the same way.
- What it is
Disclosing that people are talking to AI and that content is AI-made.
- Example
A support chat badged "AI assistant"; an order summary labelled "AI-generated".
- Catch
- Passing AI off as human breaks trust — and increasingly breaks the law.
- Related
The baseline for Lower-risk AI; part of the High-risk AI package.
Bias & Fairness — watch for unfairness
A hiring process that quietly favours one group over another — not by design, but by an inherited pattern. The harm is real even when nobody intended it.
A central concern of responsible AI is bias and fairness. Models learn from data that carries society's existing biases, so AI can produce unfair outcomes — in hiring, lending, anything that affects people — without anyone intending it. For high-risk uses this is both an ethical and a legal issue, and it doesn't fix itself. You have to actively check: does the system treat different groups fairly, are there discriminatory patterns? Fairness is something you measure and manage, not assume.
- What it is
The risk that AI treats groups of people unfairly, inheriting biases from its data.
- Example
A hiring model trained on past hires approves one group noticeably more often than another.
- How to check
Compare outcomes across groups, look for discriminatory patterns, repeat regularly.
- Catch
Left unexamined, a model quietly automates exactly the unfairness you'd never have chosen.
- Related
- Part of responsible AI; a must for High-risk AI.
Explainability — reasons where decisions matter
A loan officer who can tell you why you were declined, versus a machine that just says "no".
For consequential decisions, explainability matters: the ability to give a meaningful reason for an outcome rather than an opaque verdict. A person affected by an AI decision — denied a loan, screened out of a job — increasingly has a right to an explanation. Providing one is fair and often required. It ties back to documentation and oversight: a system you understand well enough to explain is one you can defend to the people it affects.
- What it is
The ability to give an understandable reason for a specific AI decision.
- Example
- "Declined: income below threshold, short credit history."
- Catch
- If you can't explain a decision, you can't defend it either.
- Related
Builds on Documentation; gives Human oversight something to act on.
Responsible AI overlaps with compliance: tell people they're dealing with AI, actively check for bias and unfairness, and be able to explain consequential decisions. Good governance and good product design point the same way.
Governance done well isn't bureaucracy bolted on at the end. It's a few habits from the start that let you ship consequential AI you can genuinely stand behind.
Proportionate Governance — know your tier, build to match
You don't put the same lock on a garden gate as on a bank vault. You assess what you're protecting, then build matching security — no more, no less.
The whole practice starts with honest classification: is this use high-risk or lighter? Most features are lower-risk and need basic transparency. High-risk ones earn the full package — documentation, audit trail, oversight, fairness checks. That's proportionate governance: don't drown a small feature in compliance theatre, and don't ship a high-risk one as if it weren't. And for genuinely high-risk systems, bring in real expertise: this course orients you, but real compliance is done by professionals.
- What it is
- Governance effort matched to the real risk tier.
- Example
FAQ bot: an "AI" label and sensible logs. Credit scoring: the full package plus compliance lawyers.
- Catch
- Two failure modes: theatre on small stuff, carelessness on high risk.
- Related
Applies the Risk-based approach to your product; ties together every practice in the course.
How it all connects
A jigsaw: each piece means little on its own; together they make the picture.
The concepts form one system. The EU AI Act applies a risk-based approach and splits AI into high-risk and lower-risk. High risk pulls in documentation, audit trail, human oversight, explainability and bias checks. The audit trail and oversight close the accountability gap. And governance by design is how you make all of it cheap — by building it in from day one.
- Which risk tier — high-risk because it shapes lives, or lighter?
- Is it transparent — do people know they're dealing with AI?
- Can you document how it works, and is there an audit trail of what it did?
- Is a human accountable and able to oversee and intervene?
- Have you checked for bias, and can you explain consequential decisions?
- For high-risk, are you bringing in real expertise, not relying on an overview?
- You know your risk tier and match your effort to it.
- You're transparent that it's AI, and you document how it works.
- You keep an audit trail and a human accountable for consequential decisions.
- You check for bias and can explain the decisions that matter.
- You build governance in from the start and bring in experts when it's high-risk.
Cheat sheet: all 19 concepts
Why governance
- AI Governancegoverning AI
Building AI responsibly and being able to stand behind it.
- Consequential AIAI that decides
AI that makes decisions about people's lives — the reason the bar rises.
Risk tiers
- Risk-based Approachrisk tiering
Obligations grow with potential harm.
- High-risk AIhigh-risk use
Hiring, credit, education, health, law enforcement, essential services — the full package.
- Lower-risk AIlimited / minimal risk
Most AI; mostly needs transparency. A conclusion after an honest check.
EU AI Act
- EU AI Actthe EU's AI regulation
The pyramid: prohibited → high-risk → transparency → almost nothing.
- Prohibited Practicesunacceptable risk
Manipulative or abusive AI, certain surveillance — don't build it.
- Penaltiesfines
Tens of millions of euros or a percentage of global turnover — whichever is higher.
- Brussels Effectregulatory spillover
A big market's rules become the world standard.
Proof
- Technical Documentationsystem documentation
Data, decision logic, limits and risks — written down.
- Audit Traildecision log
A log of what the system actually did.
- Governance by Designfrom day one
Build it in early — retrofitting is costly and sometimes impossible.
Accountability
- Accountability Gapresponsibility gap
Unclear who answers for harm done by autonomous AI.
- Human Oversighthuman-in-the-loop
A named human is accountable and can intervene.
- Blame Offloading'the AI did it'
Anti-pattern: you can't shed blame onto the tool.
Responsible AI
- TransparencyAI disclosure
Say it's AI and label AI-made content.
- Bias & Fairnessalgorithmic bias
Compare outcomes across groups — fairness is measured, not assumed.
- Explainabilityright to an explanation
Give an understandable reason for a consequential decision.
Using it well
- Proportionate Governanceright-sized governance
Effort matched to the risk tier; experts for high risk.
AI governance is building AI you can answer for: know your risk tier, be transparent, document how it decides, keep a human accountable, and check for fairness — built in from the start, with real expertise where it's high-risk.