Observability is the Key to Knowing What Your AI Did Before Someone Asks

The failures that end careers don’t announce themselves. A system throws an error, a pager goes off, someone gets paged at 2am: that’s the easy kind. You can see it, so you can fix it. The dangerous failure is the one that looks like success right up until the moment a regulator, an auditor, or a plaintiff’s lawyer asks you to explain a decision you can no longer reconstruct.

That’s the shape of the AI reliability problem most security and engineering leaders are walking into. We’ve spent two years racing models into production: into loan decisions, claims triage, customer support, fraud review. The benchmarks looked great. The demos dazzled the board. And almost nobody built the layer that tells you what the thing actually did once it went live.

If you can’t observe it, you can’t trust it.

And an AI system you can’t trust will fail in silence, which is the most expensive way to fail.

A real bill, not a thought experiment

Consider SafeRent, a tenant-screening company whose algorithm scored rental applicants for landlords. The model did what it was told. It produced scores, landlords acted on them, the pipeline hummed. Nobody watching a dashboard saw a problem.

What they couldn’t see was that the scoring disproportionately penalised Black and Hispanic applicants and people using housing vouchers, by leaning on credit history and non-rental debt that had little to do with whether someone pays rent. In 2025, a US federal judge gave final approval to a roughly $2.3 million settlement, and SafeRent agreed to stop using AI-generated scores to evaluate voucher applicants. The company’s own lawyers had argued it shouldn’t be liable because a human landlord made the final call; the judge didn’t buy it.

Sit with that defence for a second, because it’s the one your team will reach for too. “The model only advised; a person decided.” It didn’t hold. If your AI shapes an outcome that lands on a real human, you own the outcome, and you’d better be able to show your working.

SafeRent’s problem wasn’t a rogue model. It was an invisible one. No trace linking an input to a score to a consequence. No way to prove, after the fact, what the system weighed and why. That gap, between something going wrong and being able to explain it, is the defining risk of AI in production right now.

“We’re just the software” is failing in court right now

If you think this only bites the company that builds the model, look at what’s happening to Workday. In Mobley v. Workday, a job applicant who says he was rejected from more than 100 roles, all of them through employers using Workday’s AI screening tools, is suing the vendor itself. Not the employers. The software company whose algorithms score, rank and filter candidates on behalf of more than 10,000 customers.

Workday’s defence was the obvious one: we’re a tools provider, we don’t make hiring decisions, the employer does. The court hasn’t bought it. A federal judge let the disparate-impact claim proceed on the theory that Workday acts as an agent of its client-employers and can be on the hook under anti-discrimination law, and in 2025 granted preliminary certification of a nationwide collective of applicants aged 40 and over. In March 2026 the judge rejected Workday’s argument that age-discrimination law doesn’t even cover applicants. This is live, in discovery, as you read this.

Two things in that case should stop a CISO cold. The first: the court isn’t drawing a clean line between a software decision-maker and a human one, which is the exact line every vendor and every buyer wants to hide behind. The second: when the plaintiffs went looking in discovery for Workday’s bias-testing data, the fight wasn’t over whether it mattered, it was over privilege and scope. Once you’re challenged, “show us how you checked this thing for bias” is the question, and “we never looked” is the worst possible answer.

Now read it from the buyer’s chair. If you license an AI tool that screens, scores or ranks people, you’ve delegated a consequential decision to a black box you didn’t build and can’t fully see into. That doesn’t move the risk to the vendor. It hands you a black box and the liability for what it does. You are now accountable for a system whose internals are someone else’s trade secret, which means your due-diligence question at procurement isn’t “how accurate is it.” It’s “when a regulator asks, can either of us reconstruct why this candidate was rejected.”

Borrow a discipline that already works (you have several to choose from)

Here’s the good news: nobody has to invent this from scratch. We’ve built instrumented, auditable systems before, and there’s more than one established playbook to borrow from. Which one fits depends on your context, and the strongest programmes I’ve seen combine two or three rather than picking a single church.

• Site reliability engineering, if your instinct is uptime and operations. SRE gave us service level objectives, error budgets, and the golden signals: the discipline of deciding what “good” means in numbers and what happens when you breach it. It’s the reference every infrastructure leader already lived through, which makes it the easiest sell to a board. Applied to AI, you’re pointing that same reflex at reasoning instead of servers.
• OpenTelemetry-style observability, if your instinct is “show me the trace.” This is the closest fit to the problem this whole piece describes, because the unit of an OTel trace is exactly the unit of accountability: one decision, reconstructable end to end. The conventions for instrumenting AI agents already exist; prompts, model calls, tool invocations, and outputs emit structured spans the same way a microservice does. If a CISO knows what a trace is, this is the analogy that needs the least translation.
• MLOps, if your centre of gravity is the model lifecycle. This is the right anchor when your real risk is drift, training-serving skew, or quality degrading quietly over months. It treats versioning, performance monitoring, and bias detection as a continuous lifecycle concern rather than a launch-day checklist. It speaks most naturally to data and ML engineers, so reach for it when that’s who owns the system.
• Business process management and workflow orchestration, if your real problem is governance and audit, not uptime. This is the closest fit to the legal exposure the SafeRent and Workday cases expose. The BPM lesson is blunt: don’t wire a model straight into a consequential decision. Make it one node in an instrumented workflow where every step, hand-off, and human checkpoint is logged and overridable, the way we’ve governed rules engines for decades. You can swap the model; the audit trail and the approvals stay put.

You don’t have to choose just one, and the best setups don’t. Pull SLOs and error budgets from SRE, the trace as the unit of record from OpenTelemetry, drift monitoring from MLOps, and human checkpoints from BPM. Whatever the mix, three habits carry most of the weight.

• Start with the outcome, then instrument it. Most AI projects pick a model first and define success later. That’s backwards. Decide what business result you’re chasing, “cut claim-handling time by two minutes,” “deflect 15% of billing calls,” and build your telemetry around that number, not around an accuracy score that means nothing to the board. A model precision metric tells you the model is confident. It doesn’t tell you the business is better off, or that you’re not about to write a seven-figure cheque.
• Log the three things an auditor will ask for. What went in: every prompt, the retrieved documents, the model version, what data got redacted and by which rule. What constrained it: the safety filters that fired, the policy that governed the call, the risk tier. What came out, and whether it worked: the human edits, the override rate, the downstream business event. Tie all three together with a common trace ID so any single decision can be pulled up and replayed months later. That’s the difference between answering a regulator in an afternoon and spending a quarter explaining how something slipped through.
• Set budgets and let them trip. Pick a handful of signals for each critical workflow, factual accuracy against a source of record, safety-filter pass rate, how often a human accepts the output on first pass, and give each a target. When a signal breaches its budget, the system reroutes: to a safer template, to human review, to a rollback. That isn’t bureaucracy. It’s a circuit breaker for a decision you can’t afford to let run unchecked.

Where the human stays in the loop

Full automation is neither realistic nor responsible, and the SafeRent case shows why. The dangerous decisions are the ambiguous, high-stakes ones, exactly the cases you want a person to see.

So route the low-confidence and policy-flagged outputs to an expert. Capture every edit and the reason for it, because that record is two things at once: training data that makes the system better, and audit evidence that makes you defensible. Picture a mid-sized insurer running AI over first-notice-of-loss claims. (This one’s hypothetical, to make the mechanics concrete.) The model handles the clean cases; anything it flags as low-confidence goes to a human adjuster, whose correction feeds straight back into the next version. Eighteen months in, that firm doesn’t just have a better model. It has a complete, replayable account of every contested call, the thing it’ll need the day a state regulator comes asking.

The firms that skip this don’t save money. They defer the cost to the worst possible moment, and pay it with interest.

The question worth answering by Friday

You don’t need a six-month governance programme to start. You need to be able to answer one question about every AI decision your organisation makes: what went in, what came out, and why. Most teams can’t answer it today. The thin layer that lets you, prompt registry, request logging with trace IDs, a few real evaluations, a dashboard that watches your signals and your token spend, is a few weeks of focused work, not a year.

And if the AI in question came from a vendor, the same question moves into the contract. Before you sign, ask what you can pull when challenged: can you get at decision-level traces, the bias testing they’ve actually run, the evidence you’d hand a regulator. If the honest answer is “that’s proprietary,” you haven’t bought a tool. You’ve bought their black box and their liability, and you’re the one who’ll be standing in front of the auditor.

Observability isn’t an add-on you bolt on once the lawyers get nervous. It’s the foundation that decides whether your AI is infrastructure you can stand behind or an experiment you’re quietly hoping nobody audits. And it doesn’t matter whether you built the model or bought it: SafeRent shows you can’t hide behind the human who clicked approve, and Workday shows the vendor can’t hide behind being “just software.” The accountability lands somewhere real either way.

The era of treating that as optional is over. It ended the day the first judge declined to accept “the model only advised.”