Second-Line AI & Model Oversight for Predictive Models and AI Agents
Independent validation, ongoing monitoring and effective challenge, run by second-line agents that read every stage of the first line's pipeline and never act for it. Every finding answers three questions: is there a problem, how important is it, and what should be done. Shown here on account-takeover (ATO) fraud.
Every transaction the fraud model scored, February–April 2024, with real fraud outcomes known for all of them.
The top 1% of scores holds 68% of all fraud in the period, 65 times what random checking would find.
Every freeze, referral and review the AI agent proposed, replayed against the record and a simulated reviewer.
Nine for the predictive model (Q0–Q8) and eleven for the AI agent (Q0–Q10), answered in plain language.
Figures from the ATO demo solution (fraud model run 970; action simulator genai 163), pulled 2 November 2026.
The Validation Queue Is the New Bottleneck for AI
The business builds a model or an agent, often in weeks with an LLM. Then it waits for model risk. Model risk is not slow because it is cautious. It is slow because the evidence it needs does not exist yet.
One question decides a sign-off: was every decision right, and can you prove it?
A green dashboard on accuracy, drift and data lineage does not answer it. The decision does: the hold on an account, the SAR referral, the call to a client.
- Pass It followed its gates. All 233 SIU escalations met the documented policy.
- Finding Its confidence told us nothing. It stated 0.95 on 1,769 of 1,780 decisions.
- Finding It hurt good customers. 179 legitimate customers were frozen by holds that passed simulated human review.
- Finding Some actions had no defensible reason. 53 actions went against the pattern behind them; 38% of those hit legitimate customers, against 9.4% elsewhere.
- Finding The human in the loop needed checking too. The simulated reviewer broke its own rules on 79 of 1,779 decisions.
None of these show up on a model dashboard. Every one shows up at an exam, or in a complaint.
One Run, Two Customers
Every AI deployment in a financial institution serves two audiences. The first line consumes the decision; the second line consumes the evidence.
The business and its embedded control team
- Owns
- The process and its controls; runs the RCSA and remediates its own issues
- KPIs
- Volumes, error rates, losses, open issues
- Data
- Granular: every transaction, alert and call
Gets a better queue, cleaner escalations and a coached agent
Independent risk management
- Owns
- Credible challenge: operational risk, compliance, independent testing
- KPIs
- Profile against appetite, indicator trends, testing coverage, repeat findings
- Data
- Breadth without depth: roll-ups, samples, self-reported and lagging figures
Gets the explanation, action trail, population-level measurement, monitoring and documentation
Review decisions, not models
The system under review only has to supply its decisions. The second line never rebuilds the first line's logic, and the business does not have to run TAZI.
Independence from different questions
A challenger trained on the same data makes the same mistakes. A separate explanation model and separate policy and client-response checks do not.
Every case, every run
Sampling once a year made sense when models changed once a year. Agents change weekly, if not daily. Challenge covers the full population each cycle.
Evidence as a by-product
Memos, owners and compensating controls come out of each run. Nothing is reconstructed the week before the exam, and no second model enters the inventory.
Key Capabilities
Continuous Ongoing Monitoring
Input and score drift, missing rates and data-quality exceptions are tracked on every run, with thresholds and a receipt for each number.
Demo: 25,072 transactions (7.2%) carry a dummy amount instead of the real one.
Outcomes Analysis
Back-testing against actual outcomes and a random-review baseline, with false alerts counted per fraud caught at the operating cut.
Demo: 87% of fraud caught at the 0.50 cut, with 6.4 false alarms per fraud.
Independent Effective Challenge
An explanation model re-learns the production model's decisions as readable rules, grouped into named patterns and measured against what really happened.
Demo: 20 named patterns; only 2 of the model's 7 fraud reasons can be cited.
Oversight of Agentic AI
The agent's actions are checked against its documented rules, human-in-the-loop gates and the record, including its stated confidence.
Demo: the simulated reviewer breaks its own rules on 79 of 1,779 decisions (4.4%).
Unintended Consequences & Fairness
Which customers and channels carry the false alerts and the severe actions, with significance tests, so harm is visible before a customer complains.
Demo: legitimate external transfers are flagged at 2.8 times the average rate.
Materiality-Scaled Committee Memo
A memo for the risk committee with materiality declaration, findings, owners, compensating controls, conditions for continued use and limitations.
Demo: "Continue using the fraud model, with conditions."
What the Second Line Can Ask
Two second-line skills, one per task. Start with Q0 for a one-screen check of every area, then go deeper on any question.
Monitoring a Predictive Model
Skill ato-2ndline-ml-v2 · nine read-only questions
Start here
- Q0 Is there a problem with the fraud model? (one-screen check of every area)
The model
- Q1 Does the model catch fraud, and how many false alarms does it raise?
- Q2 Has anything changed in the data or the scores, and is the data reliable?
- Q3 Are the model's reasons sound, and could they be unfair?
- Q4 Which legitimate customers are hurt by false alarms?
- Q5 Was this transaction rightly flagged? (give a transaction id)
Decision
- Q6 What goes to the risk committee?
Value
- Q7 What is the model worth after checking?
- Q8 What does the second line's oversight bring?
Monitoring an AI Agent
Skill ato-2ndline-ai-v2 · eleven read-only questions
Start here
- Q0 Is there a problem with the AI agent? (one-screen check of every area)
What the agent does
- Q1 What did the agent do, and how much did it cover?
- Q2 Does the agent follow its rules and the human checks?
- Q3 Which legitimate customers did the agent's actions hurt?
- Q4 Can we trust what the agent says? (all decisions, or one transaction id)
- Q5 Are customers treated evenly?
Why it acts
- Q7 Are the agent's actions based on reasons we can stand behind?
- Q8 What happens when the agent acts against the model's reasons?
Decision
- Q6 What goes to the risk committee?
Value
- Q9 What is the agent worth after checking?
- Q10 What does the second line's oversight bring?
How It Works
First Line Scores
The bank's fraud model scores every transaction, and the AI agent decides what to do with each flagged one: freeze the account or refer it to investigators.
Copied Read-Only
Scores and decisions are copied to the second line, read-only: in the demo, 9,757 transactions covering all 1,780 the agent acted on, plus a random sample.
Explain & Replay
An explanation model re-learns the model's decisions as readable rules grouped into 20 named patterns. A simulated reviewer replays every agent decision and its outcome.
Challenge & Report
Second-line agents read every step and answer: is there a problem, how important is it, and what should be done, with an owner and a source for each number.
Four Second-Line Use Cases
Read the first line's existing deployment first: most second-line questions are answered there at near-zero marginal cost. Build a new use case only for a risk the business does not measure at all.
Oversight of Any Decision System
Bring your own model, AI agent or human decision-maker. TAZI runs its full pipeline on the system's decisions: performance, a fairness verdict and data readiness at go-live, then health, drift and fairness every month, without involving the business.
Independent Measurement of the Business
Dynamic segmentation, focus-group and expert-panel agents show how the workflow treats clients and where decisions can improve. A risk officer tracks these policy-level indicators monthly without touching production.
Credible Challenge on the Populations That Matter
The same validation machinery monitors a fixed population, such as a product under review or one site, or a dynamic one, such as a drifting segment or cases where reviewer and model disagree. A complaints control rated "effective" in the RCSA was not.
The Risk Team Using AI Itself
A conversational agent over the whole pipeline answers ad-hoc questions, runs handoff and monitoring routines before a committee or after an incident, and executes confirmed actions. A recommendation agent tracks recurring and unanswered questions and proposes the next use case with a draft target label and population.
Figures from TAZI proofs of value and deployments.
SR 26-2: From Calendar-Based to Risk-Based Oversight
Task 1, the predictive model, is reviewed under SR 26-2. Task 2, the AI agent, is reviewed under the bank's own GenAI and agentic framework, kept separate.
| Topic | SR 11-7 (2011) | SR 26-2 (April 2026) | What TAZI provides |
|---|---|---|---|
| Validation timing | At least annually | Risk-based: varies with model purpose and changes; generally before first use | Monitoring on every run, with thresholds and a receipt per number |
| Independence | Organisational separation | "The quality of validation … depends on the rigor and effectiveness of the review" | An independent explanation model as reproducible effective challenge; read-only access |
| Materiality | Referred to, not defined | "Model purpose, together with model exposure, determines model materiality" | A materiality declaration in every memo; depth of review scales with the tier |
| Outcomes analysis | Back-testing | Compare model outputs with real-world outcomes | Back-testing against actual outcomes and a random-review baseline |
| Generative and agentic AI | Not addressed | "Not within the scope of this guidance": the bank applies its own framework | Action Simulator: a separate agent review of rules, outcomes and guardrails before customer impact |
SR 26-2: Revised Guidance on Model Risk Management (Federal Reserve, OCC, FDIC, April 17, 2026); supersedes SR 11-7 and SR 21-8.
What the Second Line Found
Continue using the fraud model, with conditions
- High Data quality. 25,072 transactions (7.2%) carry a dummy amount, so the money at risk on them cannot be measured.
- High Customer harm. Legitimate customers making external transfers are flagged at 2.8 times the average rate.
- Medium Catch rate. 87% of fraud caught over the period, but April caught only 78% while raising 15 false alarms per fraud caught.
- Stable Scores. The scores and the takeover signals have barely moved between March and April.
Source: fraud model run 970, 350,167 transactions Feb–Apr 2024 · pulled 2 Nov 2026
Continue using the agent, with conditions
- High Customer harm. The agent froze 179 legitimate customers; 182 legitimate customers were hit by its actions in total.
- High Output reliability. The agent states 0.95 confidence on 1,769 of 1,780 decisions, so the number does not separate cases.
- Medium Policy adherence. All 233 investigator referrals meet the documented rule; the simulated reviewer departs from its own rules on 79 decisions.
- Medium Fairness. Customers moving 1,000–2,499 are hit by actions on legitimate customers 1.6 times more often than the rest.
Source: action simulator genai 163, proposed agent genai 145, validated agent genai 147 · pulled 2 Nov 2026
Conditions a committee can act on
- A weekly 100-case analyst re-check.
- Human confirmation on any action that contradicts its pattern.
- A one-business-day callback for frozen legitimate customers.
Which Model or Agent Has Been in Your Queue the Longest, and What Evidence Is Missing to Clear It?
If you built the agent: what did model risk ask for that you could not produce?
Read-only on your own models and agents; every number reproducible from its source. TAZI supplies evidence for the bank's model risk management and GenAI governance; it supports, and does not replace, the bank's own validation.