Chapter 5 — Evaluation: Measuring Models Honestly

Chapter 4 controlled overfitting: we know how to pick a model that should work tomorrow. But "should" is not evidence. Before the fraud team lets a model block real payments, someone will ask: how good is it, compared with what, and how sure are you? By the end of this chapter you can pick the metric a decision needs, compute every classification, ranking and probability metric by hand, design a split that rehearses production, find leaks, put an error bar on any metric, correct for biased labels, design and read an A/B test, and monitor a model after launch. One idea carries all of it: measure what the decision needs, on data the model has never touched, drawn the way production will draw it, and never report a number without its error bar.

How to read it. The chapter is a chain of 15 questions in three acts: Pick the metric (Sections 1–5), Trust the number (Sections 6–9) and Prove it live (Sections 10–15). Each section opens with its question and a 30-second version, and ends with the question it leaves open.

Two passes. On the first pass, read only the question, the 30-second version, and the figures of each section. You'll follow the whole arc in under an hour. On the second pass, read everything, doing the Remember box at the end of each section from memory before you look. Practice lives in its own tabs: Derivations (what to prove on a whiteboard) and Interview Bank (every question by level, with the intuition behind each answer).

1. What Question Is the Metric Answering?

The question: the fraud team has two models, A and B, and a CFO asking "which one is better?". Better at what?

The 30-second version: a metric is a question in numeric form, and the question comes from the decision the score feeds. A fraud score does one of three jobs: Decide (auto-block above a threshold → threshold metrics), Rank (fill a review queue of fixed size → ranking metrics), or Estimate (forecast fraud losses in dollars → probability metrics). Each job sits on a proxy ladder: business Goal → Online metric → Offline metric → training Loss. Each rung stands in for the one above, and every gap between rungs is a place to be fooled. On our twelve review transactions, A wins some questions and B wins others. First read: all of it.

The problem

A payments company scores every card transaction for fraud. About 1 in 100 is fraud. Two candidate models:

• Model A, a logistic regression on three continuous risk signals: amount compared with the card's history, merchant risk, and device/IP novelty. It is calibrated by construction (Ch 3, Section 9).
• Model B sees one continuous signal plus a consortium "compromised card" flag, a network feed that fires for 40% of frauds and 0.2% of legit transactions. It was trained with balanced class weights, so its raw probabilities are inflated (Ch 3, Section 8). B-cal is B after Platt scaling on a held-out calibration month.

An analyst pulled a review sample from one day of 400 transactions: all 4 frauds that day, plus 8 of the 396 legit transactions, drawn at random. These are the twelve review transactions, used for every hand calculation in the chapter:

The sample is enriched: 4 of 12 are fraud (33%), against 1% that day. Each legit row stands for 396/8 = 49.5 legit transactions of the day. Keep that in mind: any metric that touches the legit rows will differ between "the sample" and "the day" (Sections 5 and 9).

For code and diagnostics we also have the simulated book: 250,000 transactions from a known model, with A and B trained on the first 100,000, B calibrated on the next 50,000, and everything scored on the 100,000 test transactions, which hold 974 frauds (base rate 0.974%). Because we simulated it, we also know each transaction's true fraud probability, so we can score a perfect oracle as a ceiling.

Three jobs: Decide, Rank, Estimate

Most arguments about "which model is better" are two people answering different jobs.

The proxy ladder

Why not train on the business metric directly? It is not differentiable (counts at a threshold have no useful gradient), it arrives late (chargebacks take weeks), and it is noisy. Log loss is a stand-in that is smooth and proper, meaning a model can't game it by reporting anything other than its true belief (Section 4). Goodhart's law: a measure that becomes a target stops measuring. Reward a team on "fraud caught" and it flags everything; reward it on "false declines" and it flags nothing. Pair every target metric with a guardrail that pulls the other way.

A preview verdict on the twelve

Six reasonable questions, two verdicts; the rest of Act I explains every row. Every metric is reported next to a dumb baseline: flag nothing, flag at random, and the rules engine you have today. A model is only as good as the margin by which it beats them. On a day with 1% fraud, "flag nothing" already scores 99% accuracy (Section 2).

Remember: a score has three jobs, Decide, Rank, Estimate, and each job has its own metric. Climb the proxy ladder (Goal → Online → Offline → Loss) and ask at every rung what could fool you.

Next question → The simplest job is Decide: at a threshold, every transaction is blocked or passed. How do we count right and wrong? (Section 2)

2. The Confusion Matrix and Its Family

The question: model A blocks every transaction it scores at 0.5 or more. On the twelve review transactions it gets 8 right. Is 67% accuracy good, and what does "99% accurate" mean on a day with 1% fraud?

The 30-second version: a threshold splits transactions into four boxes: true positives (fraud blocked), false positives (good customer blocked), false negatives (fraud passed) and true negatives. Every threshold metric is a ratio of boxes. Precision divides by what you Predicted; Recall divides by what was Real (P-P, R-R). F1 is the harmonic mean of the two; MCC is the correlation between prediction and truth and uses all four boxes. On a day with 1% fraud, flagging nothing is 99% accurate and catches nothing. First read: Figure 5.1, the four boxes, P-P/R-R, the accuracy paradox. Second read: MCC, F_\beta, multiclass averaging.

Recap from Ch 3, Section 10: a classifier is a score plus a threshold. Change the threshold and every number in this section changes.

The four boxes on the twelve

At t = 0.5, A flags T1–T4 and B flags T1, T2, T3, T5 (Q13 asks for exactly this):

The family

Memory hook: P-P, R-R. Precision divides by Predicted positives; Recall divides by Real positives.

The vocabulary trap. Medicine says sensitivity and specificity (recall and 1 - \text{FPR}). Fraud operations says catch rate, false-decline rate and hit rate (recall, FPR, precision). Statistics says power 1 - \beta and \alpha (recall and FPR), and calls a false negative a type II error and a false positive a type I error. Interviewers switch freely.

The accuracy paradox. The day had 400 transactions and 4 frauds. A model that flags nothing gets 396/400 = 99% accuracy, recall 0, balanced accuracy 0.5, and MCC 0/0 (sklearn returns 0). Precision is 0/0, undefined.

Scale that to the day: A at 0.5 still catches 2 of 4 frauds, but its two false positives now stand for 99 blocked customers, so its day accuracy is 0.7475, precision 0.0198, F1 0.038 and MCC 0.057. On the 100,000, A at 0.5 flags only 24 transactions (19 frauds, 5 legit): accuracy 0.9904, barely above flag-nothing's 0.9903, with recall 0.0195. Accuracy rewards doing almost nothing. Memory hook: "doing nothing is 99% accurate." At a 1% base rate, report precision and recall (or precision at capacity), balanced accuracy or MCC, never accuracy alone.

F1 and Fβ: why the harmonic mean

F_1

the F1 score: one number blending precision and recall (A: 0.50, B: 0.75 on the twelve)

P

precision: the share of flagged transactions that are fraud

R

recall: the share of frauds that got flagged

TP

true positives: frauds flagged (A at 0.5: 2, T1 and T3)

FP

false positives: good customers blocked (A at 0.5: 2)

FN

false negatives: frauds let through (A at 0.5: 2)

F_\beta

the F-score that weights recall \beta times as much as precision

\beta

how many times more recall matters than precision (2 for "missing fraud is worse")

The harmonic mean collapses when either input is small: precision 1.0 with recall 0.02 gives F1 \approx 0.04, where the arithmetic mean would say 0.51. F_\beta weights recall \beta times as much as precision (F2 for "missing fraud is worse"). On the twelve, F2 and F0.5 equal F1 (0.50 for A, 0.75 for B), because each model's precision equals its recall. The second form shows F1's blind spot: TN never appears, so F1 can't see how many good customers you passed, and it moves with the base rate (Card 2).

MCC: the one that uses all four boxes

TN

true negatives: legit transactions passed (A at 0.5: 6)

A: (2\cdot 6 - 2\cdot 2)/\sqrt{4\cdot 4\cdot 8\cdot 8} = 8/32 = 0.25. B: (3\cdot 7 - 1\cdot 1)/\sqrt{4\cdot 4\cdot 8\cdot 8} = 20/32 = 0.625. MCC is the Pearson correlation of two 0/1 variables (prediction and truth), so it runs from -1 (always wrong) through 0 (no better than chance) to 1. It is undefined when a row or column of the matrix is empty; sklearn returns 0. Chicco and Jurman (2020) argue it is the most informative single threshold metric for imbalanced data.

Which number for whom. Risk ops, with fixed analyst capacity, asks how much of the queue is real fraud: precision at capacity. The loss owner asks how much fraud gets through: recall, in counts and dollars. The checkout team asks how many good customers are annoyed: FPR. They can't all be maxed; the threshold trades one against another.

More than two classes. Compute precision, recall and F1 per class, then average. Macro is the plain mean over classes, so a rare dispute category counts as much as a common one. Micro pools all TP, FP and FN first; for single-label multiclass, micro-F1 = micro-precision = micro-recall = accuracy, so the common class dominates. Weighted averages by class size and sits close to micro. For merchant-dispute categories where one rare class matters most, report per-class recall and the macro average.

Two traps. First, precision depends on the base rate: the same model has precision 0.50 on the review sample and 0.0198 on the day (Section 9, Card 1). Second, "positive" must be defined: in sklearn, pos_label decides which class is the positive one, and flipping it silently gives you the legit class's precision.

Remember: P-P, R-R: Precision divides by Predicted positives, Recall by Real positives. F1 ignores true negatives; MCC uses all four boxes. At 1% fraud, "99% accurate" is what doing nothing scores.

Next question → Every number in this section depends on where we put the threshold. Can we judge the score itself, before choosing one? (Section 3)

3. Threshold-Free: ROC, AUC, and Precision–Recall Curves

The question: A's AUC is 0.78 and B's is 0.75 on the twelve, but B puts three frauds at the very top. Which curve should the fraud team look at, and what does an AUC actually mean?

The 30-second version: slide the threshold from top to bottom and trace two curves. The ROC curve plots TPR against FPR; its area, the AUC, is the probability that a randomly chosen fraud is scored above a randomly chosen legit transaction (ties count half). Count the 4 \times 8 = 32 pairs on the twelve and A wins 25 of them (0.781). That makes AUC a rank statistic: blind to the base rate and to any monotone transform of the scores. The PR curve plots precision against recall; it feels every false positive drawn from the huge legit pile, so it shows what happens at the top of the list. On the 100,000, ROC says A is ahead overall (0.868 vs 0.848) while PR says B is far better where the review queue lives (AP 0.399 vs 0.133). ROC asks about ranking; PR asks about purity. First read: Figure 5.2, what AUC means, Figure 5.3. Second read: AP vs trapezoid, partial AUC, Gini/KS.

Sweeping the threshold on the twelve

Lower A's threshold one transaction at a time. Each fraud passed moves the ROC curve up by 1/4; each legit moves it right by 1/8.

AUC = the chance a random fraud outranks a random legit

Take every fraud–legit pair. Score 1 if the fraud is ranked higher, \tfrac12 if tied, 0 if lower. For A: T1 beats all 8 legit, T3 beats 7 (not T2), T5 beats 6 (not T2, T4), T8 beats 4 (T9–T12). That's 8 + 7 + 6 + 4 = 25 of 32 = 0.78125 (Q14). For B: T1, T3 and T5 each beat all 8 legit, but T8 (score 0.02) beats none: 8 + 8 + 8 + 0 = 24/32 = 0.75.

The same number has three faces (Card 3; Q15 asks you to prove the first equals the second):

\int_0^1

add up the area under the ROC curve as FPR runs from 0 to 1

d\,\text{FPR}

a thin sliver of FPR: each strip is TPR tall and this wide

P(s^+ > s^-)

the chance a random fraud scores above a random legit transaction

P(s^+ = s^-)

the chance their two scores tie (a tie counts half)

s^+

the score of a randomly picked fraud

s^-

the score of a randomly picked legit transaction

U

the Mann–Whitney U statistic: fraud–legit pairs the fraud wins (A: 25 of 32)

n_1

the number of frauds (4 on the twelve)

n_0

the number of legit transactions (8 on the twelve)

R_1

the sum of the frauds' ranks, rank 1 = lowest score (A: 35)

R_1 is the sum of the frauds' ranks (rank 1 = lowest score). A's frauds have ranks 12, 10, 8 and 5, so R_1 = 35 and \text{AUC} = (35 - 10)/32 = 0.78125. B's are 12, 11, 10 and 1, so R_1 = 34 and \text{AUC} = (34 - 10)/32 = 0.75. U is the Mann–Whitney U statistic (Mann and Whitney, 1947), which is why AUC inherits a known variance (Section 8). Memory hook: AUC = a random fraud outranks a random legit. Never say "the probability the model is right".

What AUC ignores

• The base rate. TPR is computed inside the fraud pile and FPR inside the legit pile, so resampling either pile leaves both, and the AUC, unchanged. A 50/50 test set gives the same AUC as the natural one.
• Monotone transforms. Square, log or rank the scores: the order is unchanged, so the AUC is unchanged (Card 4).
• Calibration. Raw B and B-cal have identical AUCs (0.848); raw B's probabilities are off by a factor of 30 (Section 4).

Two special cases (Card 5). A model that ranks backwards has \text{AUC} < 0.5: \text{AUC}(-s) = 1 - \text{AUC}(s), so an AUC of 0.30 means "flip the score (or check the label encoding)" and you have 0.70. A hard yes/no classifier has a one-corner ROC, so its AUC is (\text{TPR} + \text{TNR})/2, its balanced accuracy: B's 0/1 predictions at 0.5 give 0.8125. Constant or random scores give 0.5. A useful model for intuition is the binormal one (Card 6, WP1): if legit scores are N(0, 1) and fraud scores N(d, 1), the AUC is \Phi(d/\sqrt{2}), 0.921 for d = 2, while at the threshold catching 80% of fraud the precision is 0.866 at \pi = 0.5 and only 0.061 at \pi = 0.01. Credit-risk teams use two relatives. \text{Gini} = 2\cdot\text{AUC} - 1 (A: 0.737, B: 0.696 on the 100,000). The KS statistic = \max(\text{TPR} - \text{FPR}), the largest vertical gap between ROC curve and diagonal (A: 0.576, B: 0.543).

The PR curve and average precision

Precision against recall zig-zags: each fraud lifts precision, each legit drags it down. This sawtooth is why you can't draw straight lines between PR points. The precision achievable between two operating points is not linear in recall (Davis and Goadrich, 2006), so trapezoidal area is the wrong summary. The right summary is average precision, a step-wise sum, which is what sklearn reports (Card 7):

\sum_k

add up over every cut-off k down the score-sorted list

R_k

recall when the top k transactions are flagged

R_{k-1}

recall at the cut-off before, so R_k - R_{k-1} is the step in recall

P_k

precision when the top k transactions are flagged

\sum_{\text{frauds } i}

add up over every fraud i (4 on the twelve)

i

one fraud, read at its position in the ranked list

A's frauds sit at ranks 1, 3, 5 and 8 (rank 1 = top): \text{AP} = (1 + 2/3 + 3/5 + 4/8)/4 = 0.6917. B's at 1, 2, 3 and 12: \text{AP} = (1 + 1 + 1 + 4/12)/4 = 0.8333. The trapezoid auc(recall, precision) gives 0.649 and 0.826 instead: a different number, and the wrong one. The PR baseline is the base rate. A random ranker has expected precision \pi at every recall, so its \text{AP} \approx \pi. On the 100,000, \pi = 0.974\%, so A's AP of 0.133 is about 14 times random and B's 0.399 is about 41 times.

ROC vs PR at 1% fraud

The ROC curves cross once, at an FPR between 0.129 and 0.140. B leads below it, A above it. A's AUC lead is earned in a region (blocking 14% or more of good customers) where no fraud team will ever operate. And a "tiny" FPR is not tiny in counts: FPR 0.1% of 99,026 legit transactions is about 99 blocked customers, which is why precision collapses even when the ROC looks fine. Partial AUC (the area up to an FPR you can afford, here 1%) and TPR at a fixed FPR are the numbers fraud teams set targets on. roc_auc_score(y, s, max_fpr=0.01) returns the McClish-standardized version, rescaled so that 0.5 is random and 1 is perfect.

Use ROC/AUC when classes are roughly balanced, when the base rate is unknown or changes between settings, or when the ranking itself is the product (a platform score sold to many users). Use PR/AP when positives are rare, when you operate at the top of the list, and when precision (analyst time, customer friction) is what costs money.

Two traps. A single AUC hides crossing curves: two models with the same AUC can catch 20% and 40% of fraud in their top 1%. And "AUC 0.95" can coexist with precision under 10% at every useful threshold.

Remember: AUC = a random fraud outranks a random legit. It's a rank statistic, blind to base rate and to monotone transforms. ROC asks about ranking; PR asks about purity. At 1% fraud, look at PR and at TPR at the FPR you can afford.

Next question → Ranking right is not the same as saying the right number. When a model says 30%, is it 30%? (Section 4)

4. Probabilistic Quality: Log Loss, Brier, Calibration and Proper Scoring Rules

The question: finance wants to forecast next month's fraud losses by adding up the model's probabilities. Model B's average predicted probability is 31% in a world with 1% fraud. Which numbers tell us whether a model's probabilities can be used as probabilities?

The 30-second version: two properties. Discrimination: does the model rank frauds above legit (Section 3)? Calibration: does "30%" happen 30% of the time? Log loss and the Brier score measure both at once, and both are proper scoring rules: your expected score is best when you report your true belief, so hedging can't game them. The Brier score splits into Reliability − Resolution + Uncertainty, "Lie − Learn + Luck". A reliability diagram shows the lie; ECE summarizes it and depends on how you bin. Raw B ranks well but lies: its Brier is 0.128, against 0.0096 for a constant base-rate forecast. Recalibrating B leaves its AUC untouched and drops its Brier to 0.0068, better than A's 0.0090. First read: discrimination vs calibration, Figure 5.4, Lie − Learn + Luck. Second read: properness, ECE pitfalls.

Discrimination vs calibration

Memory hook: discrimination ranks, calibration tells the truth. Finance needs both: the right order and the right numbers. On the 100,000 the base rate is 0.974%. Mean predicted probability: A 0.0095, B-cal 0.0100, raw B 0.313. Sum raw B's probabilities and you forecast 32 times the real fraud.

Log loss and Brier

\frac1n

divide by n, the number of transactions scored (12 on the review sample): an average

\sum_i

add up over every transaction i

y_i

the true label of transaction i: 1 if fraud, 0 if legit

p_i

the model's predicted fraud probability for transaction i (B gives T8 0.02)

Log loss (Ch 3, Section 3) is unbounded and punishes the confident miss. B gave the $900 fraud T8 a probability of 0.02, and that single row costs -\ln 0.02 = 3.91 nats. B's log loss on the twelve is 0.576, but without T8 the other eleven average 0.272. The Brier score is the mean squared error of the probability, bounded between 0 and 1. The baselines matter at 1%. A constant forecast of \pi scores Brier \pi(1 - \pi) (0.0099 at exactly 1%; 0.00965 at the test rate) and log loss equal to the entropy H(\pi) (0.056 nats at 1%; 0.0548 at the test rate). Any model must beat these.

On the twelve, the two disagree. Log loss prefers A (0.523 vs 0.576); Brier prefers B (0.149 vs 0.181). Both are proper; they weight errors differently. Log loss charges T8's confident miss 3.91 nats; Brier charges it (0.98)^2 = 0.96, so B's three confident hits outweigh it.

At 1% fraud every Brier score looks tiny, so always report the Brier skill score, \text{BSS} = 1 - \text{Brier}/\text{Brier}_\text{base}. A beats the base rate by 7%; B-cal by 30%; the oracle, the best possible, by 35%.

Proper scoring rules. A scoring rule is proper if, when the true probability is q, your expected score is optimized by reporting p = q, and strictly proper if q is the only optimum (Brier, 1950; Gneiting and Raftery, 2007). Log loss and Brier are strictly proper (Card 8):

q(1-p)^2 + (1-q)p^2

the expected Brier score of reporting p when the truth is q

\frac{d}{dp}

how fast that expected score changes as you nudge p: a slope, not a fraction to divide

p

the probability you report for the transaction

q

the transaction's true probability of fraud

Absolute error of the probability, \lvert p - y\rvert, is not proper. At q = 0.3, reporting the truth scores an expected 0.42, while reporting 0 scores 0.30. It rewards rounding to 0 or 1. Accuracy is improper for the same reason. This is why you train and compare probability models on log loss or Brier: they are the only kind of score under which honesty is the winning strategy.

The Brier decomposition: Lie − Learn + Luck

Group forecasts into bins k with n_k rows, mean forecast f_k, observed fraud rate \bar o_k and overall rate \bar o (Murphy, 1973; Card 9):

\sum_k

add up over the forecast bins k

n

the total number of transactions scored

n_k

the number of transactions in bin k

f_k

the mean forecast in bin k

\bar o_k

the observed fraud rate in bin k (bar = average of the 0/1 outcomes)

\bar o

the overall fraud rate across all bins: the base rate

Lie (reliability) is how far each bin's forecast sits from what happened: zero for a calibrated model. Learn (resolution) is how much the bins' outcomes differ from the base rate: that is discrimination, and it is subtracted, so learning lowers the score. Luck (uncertainty) is the base rate's own noise, which no model can touch. The identity is exact when forecasts are constant within each bin. With continuous forecasts it depends on the bins: with 10 bins at a 1% base rate, most of the resolution is lost inside the bins. We use 200 equal-mass bins of 500 rows:

Raw B and B-cal learn exactly the same (resolution depends only on the ranking, and Platt keeps the ranking). The whole difference is the lie. B-cal beats A because it learns over four times as much. With only 10 bins, B-cal's resolution would read 0.00029 instead of 0.00270, so state the binning whenever you quote a decomposition.

Reliability diagrams and ECE

A reliability diagram plots each bin's observed rate against its mean forecast. Points on the diagonal are calibrated. Below the diagonal the model is over-confident (it says more than happens); above, under-confident.

Raw B's lowest decile says 0.070 and sees 0.0011; its top decile says 0.692 and sees 0.0605. B-cal's top decile says 0.061 and sees 0.0605. At a 1% base rate, uniform bins (0–0.1, 0.1–0.2, …) are useless: 98.7% of A's rows land in the first bin. Use equal-mass (quantile) bins and a log axis.

Expected calibration error is the bin-weighted average gap (Naeini, Cooper and Hauskrecht, 2015):

\max_k

the largest value over all bins k

Three pitfalls. It depends on the bins: A's ECE runs from 0.00040 (5 uniform bins) to 0.00146 (50 quantile bins), so only compare ECEs computed the same way. It is biased upward with few rows per bin: noise in \bar o_k shows up as a gap even for a perfectly calibrated model. It measures honesty, not usefulness: a constant forecast at the base rate scores \text{ECE} \approx 0.0003 with AUC 0.5.

Bin-free alternatives: fit a logistic regression of y on \operatorname{logit}(p) and report the calibration slope (1 is ideal; below 1 means over-confident extremes) and intercept (0 is ideal). A: slope 0.98, intercept −0.03. Raw B: slope 1.03 but intercept −4.63: right shape, probabilities about e^{4.63} \approx 100 times too high on the odds scale. B-cal: 1.02 and 0.02.

Fixing calibration (recap). Ch 3, Section 9 owns the fixes: Platt fits a line on the logit scale; isotonic fits a staircase; both are fit on held-out data, never on the test set. One fact belongs here. Platt never changes AUC (it's strictly increasing): change exactly 0. Isotonic can, because its flat steps create ties. On the test set it maps B's 100,000 distinct scores to 68 values. AUC falls 0.0022 (0.848 → 0.846) and AP falls 0.025 (0.399 → 0.374), because ties hurt most at the top of the list.

Proper scores matter far beyond fraud: weather forecasting, ad click prediction (where bids consume probabilities) and LLMs, whose training loss is log loss on tokens (Section 15).

Remember: discrimination ranks, calibration tells the truth, and you need both to use a probability as a number. Log loss and Brier are proper: honesty is optimal. Brier = Lie − Learn + Luck. At 1% fraud, report skill against the base rate.

Next question → We can now grade ranks, decisions and probabilities. Which number does the business actually pay for? (Section 5)

5. Business Metrics: Lift, Gains, Precision@k, Expected Cost, and Regression Metrics Revisited

The question: the review team can look at 1% of transactions a day; each review or block of a good customer costs about $10, and a missed fraud costs its amount. Which model saves more money, and how do we score the delivery-time model while we're at it?

The 30-second version: when capacity is fixed, the only threshold that matters is "the top k". Report precision@k, recall@k and lift, how many times better than random the top slice is. On the 100,000, B's top 1% holds 44% of the fraud (lift 44); A's holds 21%. When costs are known, draw the expected-cost curve and read off its minimum; the shortcut cut c_\text{FP}/(c_\text{FP} + c_\text{FN}) works only if the probabilities are calibrated. Weight by dollars, not counts. And if the evaluation sample was enriched with fraud, weight the legit rows back before computing anything that touches them: on the twelve, that flips the cost verdict from A to B. Regression metrics follow the same rule: pick the one whose minimizer is the number the decision needs. First read: the capacity view, the cost curve, the enrichment trap, the regression table.

Capacity: precision@k, recall@k, lift

Sort by score and flag the top k. Then:

k

how many top-scored transactions get flagged (1% = 1,000 of the 100,000)

n_1

the total number of frauds (974 on the 100,000)

\pi

the base rate: the share of transactions that are fraud (0.974%)

Lift at 1%: A 20.8, B 43.6, oracle 48.4. The cumulative gains curve plots recall@k against k; the lift chart plots lift@k. A random ranker's gains curve is the diagonal; a perfect one hits 100% at k = \pi.

Ranking metrics beyond the cut. When the order inside the list matters (search, recommendations, an analyst queue worked from the top), use a position-discounted metric. MAP averages AP over queries; MRR averages 1/(rank of the first relevant item). NDCG@k (Järvelin and Kekäläinen, 2002) allows graded relevance:

\sum_{i=1}^{k}

add up over positions i = 1 to k of the ranked list

\text{gain}_i

the value of the item at position i (1 per fraud, or its $ amount)

i

the position in the ranked list, 1 = top

On the twelve with gain = 1 per fraud, NDCG@5 is A 0.737, B 0.832. With gain = the fraud's amount, it is A 0.447, B 0.485: both drop, because neither puts the $900 fraud near the top. Ch 15 owns ranking metrics in depth.

Expected cost. Passing a legit transaction costs $0; blocking or reviewing one costs c_\text{FP} = $10; passing a fraud costs c_\text{FN} = its amount; blocking a fraud costs $0 (or a review fee). The expected cost per transaction at threshold t is (Card 10):

\text{Cost}(t)

expected dollars lost per transaction when flagging at threshold t

t

the threshold: flag when the score is at least t

c_{FN}

the cost of a missed fraud: its amount

c_{FP}

the cost of blocking or reviewing a good customer: $10

Ch 3, Section 10 derived the cost-optimal threshold for a calibrated probability: flag when p\cdot c_\text{FN} > (1 - p)\cdot c_\text{FP}, that is p > c_\text{FP}/(c_\text{FP} + c_\text{FN}). With the mean fraud amount of $215.68, t^* = 10/225.68 = 0.0443.

Three lessons. t^* is only as good as the calibration: raw B at t^* flags almost everything. Use the amounts: flagging when p\cdot\text{amount} > c_\text{FP} (approximately the exact rule p\cdot\text{amount} > (1 - p)\cdot c_\text{FP}) cuts B-cal's cost by a further sixth, toward the oracle's $808. And the minimum is shallow: A's curve barely moves between t^* and its argmin. Profit curves plot the same thing as savings; cost curves (Drummond and Holte, 2006) plot cost against every possible cost ratio at once.

Where F1 peaks (Card 11). For calibrated probabilities, the threshold that maximizes the population F1 is half the best F1: t^* = F_1^*/2 (Lipton, Elkan and Naryanaswamy, 2014). On 4,000,000 draws of the true probabilities, the best F1 is 0.516 at t = 0.258, exactly half; B-cal's empirical argmax on the test set is 0.257. F1's best cut has nothing to do with costs, which is one more reason not to pick thresholds by F1 (Drill 7, Q8).

Dollar recall (fraud dollars caught / fraud dollars) can rank models differently from count recall. On the twelve, A's top 3 catch $680 of $1,730 (39.3%), B's $830 (48.0%), and B never catches the $900 fraud until it flags everything. On the 100,000 at 1% capacity: A 20.4%, B 44.3%.

The enrichment trap. The twelve are 33% fraud; the day was 1%. Best "flag the top k" cost, c_\text{FP} = $10:

Reweighting the legit rows by 49.5 leaves AUC unchanged (0.781, 0.750), because AUC never compares rows within a class. AP falls hard: A 0.692 → 0.272, B 0.833 → 0.753. The rule, memory hook: weight it back before you touch the negatives. Precision, accuracy, AP and cost must be computed at production class proportions.

Choosing a metric

Regression metrics revisited: the ETA thread

Recap from Ch 1, Section 9: RMSE punishes, MAE forgives, MAPE breaks at zero. Ch 2's delivery-time toy has six orders, distance (1, 2, 3, 3, 4, 5) km, items (2, 1, 5, 3, 4, 3), ETA (15, 12, 24, 21, 26, 28) minutes. OLS gives \text{ETA} = 6 + 3\cdot\text{km} + 2\cdot\text{items} with residuals (2, −2, −1, 0, 0, 1): MAE 1.0, RMSE \sqrt{10/6} = 1.29, MAPE 6.29%, R^2 = 1 - 10/200 = 0.95. The deeper rule: each loss's minimizer is a different summary of the outcome.

Memory hook: RMSE → mean, MAE → median, pinball → quantile.

\ell_q(y, \hat y)

the pinball loss for one order at quantile level q

q

the quantile you want, e.g. 0.9 for "arrives within this 90% of the time"

y

the actual delivery time, in minutes

\hat y

the forecast delivery time (hat = estimated)

On Ch 2's 10,000 simulated orders (mean ETA 21.6 min, median 20.1): OLS has MAE 2.40, RMSE 3.00, MAPE 13.8%. The constant that minimizes MAPE is 16.9 minutes, well below the median, because MAPE divides by the actual: an over-forecast on a short trip costs more than an under-forecast. That is MAPE's quiet preference for under-forecasting. sMAPE and WAPE (total absolute error over total actual) reduce the problem without removing it.

Promise windows need a quantile. The OLS mean forecast is exceeded by half the orders (coverage 50.2%). A q = 0.9 quantile regression covers 90.0% of orders, with pinball loss 0.528 against 1.199 for the mean forecast. Storms (+40 min for 1% of orders, 101 orders) raise RMSE from 3.00 to 4.97 (\times 1.65) but MAE only \times 1.16: RMSE is the metric that notices rare disasters.

Remember: fixed capacity → precision@k and lift; known costs → the minimum of the expected-cost curve. Weight by dollars, and weight it back before you touch the negatives. Regression: RMSE → mean, MAE → median, pinball → quantile; MAPE prefers to under-shoot.

Next question → Every number in Act I came from a test set. Was that test set drawn the way production will be? (Section 6)

6. Validation Design: Split Like Production

The question: suppose the test transactions were drawn at random from the same months the models trained on, cards appear in both sets, and fraud tactics change month to month. Is a random split an honest rehearsal of next month?

The 30-second version: a test set is a rehearsal of deployment, so split like production. Ask three questions: Rare? Repeated? Recent? Rare positives → stratify, so every fold has its share of frauds. Repeated entities (the same card, customer or device across rows) → group the split, so the model can't memorize a card it will never see again. Recent matters (drift, adapting fraudsters) → split by time: train on the past, test on the future; when labels take weeks to arrive, purge training rows whose label window overlaps the test period. On a card-level simulation, a random split reports AUC 0.904 and a grouped, out-of-time split 0.824. The second is the number to believe. First read: Rare/Repeated/Recent, Figure 5.6. Second read: purging and embargo.

Recap from Ch 4, Section 8: K-fold mechanics, the 1-SE rule and nested CV. This section is about which split. Three sets, three jobs. Train fits. Validation chooses (features, hyperparameters, threshold). Test reports, once. Every extra look at the test set turns it into a validation set (Section 7). Holdout is cheap and matches a single deployment; K-fold uses data better and gives a spread, at K times the compute. Use K-fold when data are small; a large time-ordered holdout when they aren't.

The card-level simulation. To study splits we need structure the 100,000 lack: repeated cards and drift. The leakage simulation has 12,000 cards with about 10 transactions each over 180 days (119,858 rows, fraud rate 1.9%). Once a card is compromised, its later transactions are often fraud, so 90.8% of fraud rows sit on cards with two or more fraud rows. The merchant-risk effect drifts from 1.0 to 0.5 over the 180 days.

Rare → stratify. With 974 frauds in 5 unstratified folds, the fraud count per fold has SD 13.0 (range 159 to 236). Stratified folds have SD 0.45, effectively equal. Unstratified folds add noise to every per-fold metric, and with very rare positives some folds can get none.

Repeated → group. If card 1234 has rows in train and test, the model can learn card 1234 rather than fraud. Grouping (GroupKFold, StratifiedGroupKFold) puts all of a card's rows on one side. Do the same for customers, devices, merchants, near-duplicate rows and rows from one session. The simulation's grouped AUC is 0.055 below the random split's: that is how much the random split had memorized.

Recent → split by time. A single out-of-time holdout (train on the past, test on the most recent period) is the default for reporting. An expanding window trains on months 1–3 and tests on 4, then trains on 1–4 and tests on 5 (TimeSeriesSplit); a sliding window keeps a fixed-length training window and moves it forward, for when old data mislead.

Purging and embargo (López de Prado, 2018). A fraud label resolves up to 60 days after the transaction. A training row from 30 days before the test period has a label that resolved during the test period, so its label carries information from the test window. Purge training rows whose label window overlaps the test period. The simulation's purge removes 19,984 rows, and the out-of-time AUC falls from 0.878 (no purge) to 0.863. An embargo drops a gap of rows after each test fold. It matters only in K-fold over time, where training folds can follow the test fold; forward-chaining splits never train on the future. TimeSeriesSplit(gap=…) implements the gap. The fraud default is grouped + out-of-time ("out-of-time, out-of-card").

Adversarial validation. Train a classifier to tell training rows from test rows. \text{AUC} \approx 0.5 means the two are indistinguishable; well above 0.5 means the test set is a different population and the model will be evaluated on unfamiliar ground. In the simulation, random split: 0.500; out-of-time: 0.582, from the drift.

Two traps. Stratifying on the label and grouping on the card at once needs StratifiedGroupKFold; StratifiedKFold ignores groups. And shuffling before a time split destroys it: train_test_split shuffles by default.

Remember: Split like production. Rare → stratify, Repeated → group, Recent → split by time (purge and embargo when labels arrive late). Validation chooses; test reports, once.

Next question → A good split stops the model from memorizing the past. What stops information from the future, or from the answer itself, from sneaking into the features? (Section 7)

7. Leakage: The Bias That Makes Everything Look Good

The question: a new feature, "card flagged in the network's fraud report", lifts offline AUC from 0.82 to 0.98. Launch it?

The 30-second version: leakage is any path by which information the model won't have at decision time reaches it during evaluation. It always biases in one direction, up, which is why it's dangerous: nobody investigates a great result. Five kinds, L-E-A-K-S: Label proxies, Entity overlap, All-data preprocessing, Known-later information, Snooping the test set. The detectors: results too good to be true, one feature dominating importance, and a big gap between random and out-of-time scores. The audit question from Ch 1: would I know this at the moment of decision? First read: the LEAKS table and Figure 5.6 (panel b). Second read: detection and the pipeline fix.

L-E-A-K-S on the card-level simulation

The honest number is the grouped, out-of-time AUC of 0.824. Each leak below was added to that setup (Q23, Q24):

Leakage only ever flatters, but by very different amounts: L, E and K are loud (0.94–0.98), A is moderate, and S is quiet (+0.007 here). Snooping in numbers. The 20 configurations were scored on a test set with only 130 frauds; the winner scored 0.844 there and 0.837 on a large fresh sample of the same month. Over 40 independent test sets, the picked winner's optimism averaged 0.011 ± 0.003, of which 0.006 ± 0.001 was pure selection. The rest is the shared luck of a small test set. That is the winner's curse (Section 8).

All-data preprocessing: the classic. Take 200 rows, 5,000 features of pure noise and balanced labels. Select the 20 features most correlated with the label on all the data, then run 5-fold CV: accuracy 0.80. Do the selection inside each training fold: 0.535, which is chance plus noise (Ambroise and McLachlan, 2002). Every data-dependent step (scaling, imputation, target encoding, feature selection, resampling such as SMOTE) must be refit inside each fold. sklearn's Pipeline with cross_validate does this automatically (Section 14).

Point-in-time correctness. A feature store must answer "what did this feature equal as of the transaction's timestamp?", not "what does it equal now". Backfilled or corrected data leak quietly: a merchant's risk score recomputed last month, after the fraud wave was known, is known-later information even though the column name never changed.

Duplicates. Exact duplicates are found by hashing; near-duplicates (the same purchase retried three times) by nearest neighbors on features. Put each duplicate cluster on one side of the split. For LLMs this is benchmark contamination (Section 15).

A leakage checklist. (1) For each feature: would I know this at the moment of decision? Check its timestamp, not its name. (2) Is any single feature's AUC close to 1, or its importance share above about half? (3) Compare random, grouped and out-of-time scores: a large gap means overlap or drift. (4) Every transform is fit inside the training fold. (5) Target encodings are out-of-fold. (6) Feature joins are point-in-time, with no backfilled values. (7) Duplicates and entities are grouped. (8) The test set has been used once: count the looks. The standard reference is Kaufman, Rosset, Perlich and Stitelman (2012).

Remember: leakage only ever flatters. L-E-A-K-S: Label proxies, Entity overlap, All-data preprocessing, Known-later information, Snooping the test. Ask of every feature: would I know this at the moment of decision?

Next question → With an honest split and no leaks, A beats B by 0.020 AUC on 974 frauds. Is that a real difference or luck? (Section 8)

8. Uncertainty in Metrics: Error Bars, Paired Tests and the Noise in CV

The question: A's AUC is 0.868 and B's is 0.848 on the same 100,000 transactions. The PM wants to ship A. How sure are we, and why does the number of frauds matter more than the number of transactions?

The 30-second version: a metric on a test set is an estimate with a sampling distribution, like Ch 1's slope. Your n is your frauds: recall is a proportion over 974 frauds, not over 100,000 rows, and an AUC's error bar is set mostly by the smaller class too. Three tools: bootstrap the rows (or whole cards if rows are grouped), DeLong for AUCs, McNemar for decisions. Always compare paired, on the same test rows, so the shared noise cancels. Here \Delta\text{AUC} = 0.020 with 95% CI [0.005, 0.035]: real, but small. And a K-fold CV score's "SD over folds/\sqrt{K}" can be too narrow. First read: "your n is your frauds", the paired bootstrap, Figure 5.7. Second read: DeLong, McNemar, CV variance.

You need this: the variance of a proportion

A proportion \hat p = (\text{successes})/n from n independent trials has

\text{Var}(\hat p)

the variance of the measured proportion: how much it would wobble on a fresh test set

\hat p

the measured proportion, e.g. A's recall 2/4 (hat = estimated from data)

p

the true proportion being estimated

n

the number of trials: for recall, the number of frauds n_1, not rows

For recall, the trials are the frauds. Recall is \text{TP}/(\text{number of frauds}), so n is n_1, not the test size. For small counts the Wald interval misbehaves (it can leave [0, 1] and covers too rarely); use the Wilson interval, which inverts the score test.

On the twelve, A's recall at 0.5 is 2/4. Wald: [0.01, 0.99]. Wilson: [0.15, 0.85]. The Hanley–McNeil SE of A's toy AUC is 0.157. Memory hook: your n is your frauds. The twelve cannot tell A from B on any metric. How many frauds for a target width (Card 12): n = z^2 p(1 - p)/h^2. For recall \approx 0.8 within \pm 0.02 at 95%: 1.96^2 \times 0.16/0.0004 = 1,537 frauds, about 153,700 transactions at 1% fraud.

The bootstrap. The test set is one sample from production. Resample it with replacement, recompute the metric, repeat B times; the spread of the replicates estimates the sampling distribution, and the 2.5th and 97.5th percentiles give a percentile CI (Efron's bootstrap). For comparisons, pair it: draw the row indices once per replicate and score both models on the same rows. When rows are grouped, resample whole cards (the cluster bootstrap). When the metric depends on a threshold chosen from the data, re-tune inside each replicate. On the 100,000, with 2,000 paired replicates:

DeLong's test

DeLong, DeLong and Clarke-Pearson (1988) give the AUC's variance analytically (Card 13). For each fraud, compute the share of legit transactions it outranks (its structural component V_{10}); for each legit, the share of frauds that outrank it (V_{01}). The AUC is the mean of either, and

\widehat{\text{Var}}(\text{AUC})

DeLong's estimated variance of one model's AUC (hat = estimated)

S_{10}

sample variance of the frauds' components V_{10} (share of legit each fraud outranks)

S_{01}

sample variance of the legit rows' components V_{01} (share of frauds outranking each)

n_1

the number of frauds (974 on the 100,000)

n_0

the number of legit transactions (99,026)

\text{Var}(\text{AUC}_A - \text{AUC}_B)

the variance of the AUC gap between models A and B on the same rows

\text{Var}_A

the variance of A's AUC

\text{Var}_B

the variance of B's AUC

\text{Cov}_{AB}

the covariance of A's and B's AUCs: shared luck from scoring the same rows

where S_{10} and S_{01} are the sample variances (and, for two models, covariances) of the components. On the 100,000: SE(A) = 0.0055, SE(B) = 0.0075, correlation 0.34, SE of the difference 0.0076, z = 2.70, p = 0.007. Ignoring the pairing gives SE 0.0093 and z = 2.21: the covariance is free precision (Q21).

Hanley–McNeil (1982) is the hand formula, from the AUC alone:

\text{SE}^2

the squared standard error of the AUC (Hanley–McNeil's hand formula)

A

the AUC itself, as one number (e.g. 0.9)

Q_1

the chance two random frauds both outscore one random legit transaction

Q_2

the chance one random fraud outscores two random legit transactions

At AUC 0.9 it gives 0.0207 with 100 frauds and 0.0065 with 1,000; the legit count barely matters. It assumes exponential-shaped score distributions and simulation reproduces it closely there (0.0206 and 0.0064), but for binormal scores the true SD is about 25% smaller, and on the 100,000 it gives 0.0074 for A against DeLong's 0.0055 and the bootstrap's 0.0054. Treat it as a quick, often conservative approximation.

McNemar's test for decisions

At a fixed operating point (the top 1%), each fraud is caught by both, A only, B only, or neither:

Only the discordant cells carry information about which is better; the 95 and 441 are the same for both models (McNemar, 1947; Card 14):

\chi^2

McNemar's test statistic: how lopsided the disagreements are

b

frauds caught by A only (108)

c

frauds caught by B only (330)

\sim

is distributed as, if the two models were equally good

\chi^2_1

the chi-squared distribution with 1 degree of freedom

p \approx 5 \times 10^{-26}: B catches more frauds at this capacity (Q22, Drill 13). For small b + c, use the exact binomial version: b \mid (b + c) \sim \text{Bin}(b + c, \tfrac12). Bootstrapping the operating point gives the same verdict with a CI. Memory hook: bootstrap the rows, DeLong the AUCs, McNemar the disagreements, always paired.

Error bars shrink with frauds, not rows

Subsampling a fresh simulation to fixed fraud counts (100 repeats each), the 95% CI half-width is:

Roughly 1/\sqrt{n_1}. On the 100,000 test set, A's recall at its cost-optimal cut is 336/974 = 0.345, Wilson CI [0.316, 0.375]: three points either way from 974 frauds in 100,000 rows.

The noise in CV estimates

Ch 4 noted CV error is noisy. How noisy is hard to say. Repeating K-fold on one dataset measures only how the folds were cut. Ten repetitions of 5-fold on the training rows have SD 0.00011, against a naive SE of 0.0066. That is the wrong comparison: it says nothing about a new dataset. Across independent datasets (60 sets of 5,000 rows), the actual spread of the 5-fold AUC over the naive SD-over-folds/\sqrt{K} is 0.93 for logistic regression (about right) and 1.31 for gradient boosting (too narrow by about a third). The folds share training data, so they are correlated, and no unbiased estimator of CV variance exists (Bengio and Grandvalet, 2004). For comparing two learners, use the 5\times 2 CV paired t-test (Dietterich, 1998) or the corrected resampled t-test (Nadeau and Bengio, 2003), which inflates the variance by a term for train–test overlap.

Many models, one test set. Score 20 models with identical true AUC on one test set with SE 0.01. The winner's expected test score is 1.87 SE above the truth (the expected maximum of 20 standard normals), so +0.019 of pure selection. Simulation gives +0.0187. That is Section 7's S, made exact (Drill 12). Reporting template: metric, 95% CI, number of positives, split type, baseline. "AUC 0.868 [0.858, 0.879], 974 frauds, out-of-time, vs 0.848 for production; \Delta\text{AUC} 0.020 [0.005, 0.035]."

Remember: Your n is your frauds. Bootstrap the rows (or the groups), DeLong the AUCs, McNemar the disagreements, and always compare paired, on the same test rows. SD over folds/\sqrt{K} can be too narrow.

Next question → All these error bars assume the test labels are a fair sample of the truth. In fraud, they aren't. (Section 9)

9. Bias in the Evaluation Data

The question: we only learn a transaction was fraud if it was let through and charged back, or if an analyst reviewed it. Chargebacks take weeks. The vendor's test set had a different fraud rate from production. And customers new to the country get declined far more often. What does each do to the metrics?

The 30-second version: reuse Ch 1's Who, What, How, When. WHO gets a label: transactions the incumbent blocked never get an outcome, so evaluating on the survivors is biased (the feedback loop); fix it with a small random holdout and inverse-propensity weights. WHEN labels arrive: chargebacks trickle in over weeks, so recent data look clean; evaluate matured cohorts. WHO is in the test set: if the base rate differs from production, recall, FPR and AUC carry over but precision does not. Recall and FPR travel; precision stays home. WHO the errors fall on: similar AUCs by group can hide a false-decline rate nearly three times higher for one group. HOW labels are made: chargebacks miss some fraud and include friendly fraud. First read: the four questions, Figure 5.8. Second read: IPW evaluation, the impossibility result, the assumptions table.

WHO gets a label: the feedback loop

Model A is in production and auto-blocks its top 1%. Blocked transactions are never completed, so they never produce a chargeback: 20.8% of the test frauds disappear from the labeled data. Evaluate on what's left (Q28):

The survivors are the frauds A found hard, so both models look worse. Here the bias pushed both AUCs down; in general it is biased in a direction set by the blocking policy, and it can make a challenger that resembles the incumbent look better or worse than it is.

The fix: a random holdout. Let a small, logged fraction of would-be blocks through (here 2%), so some blocked-region transactions get labels. Weight each labeled row by 1/P(it was let through): 1 for unblocked rows, 1/0.02 = 50 for holdout rows. This inverse-propensity weighting (IPW) makes the labeled set represent all traffic again; Stripe's public talk on counterfactual evaluation describes this design (Manapat, 2015). It is unbiased on average but noisy: a 2% holdout of a 1% block rate labels only about 20 blocked rows (about 4 frauds) in 100,000. A 10% holdout cuts the SDs to 0.006 (AUC) and 0.021 (recall). Exploration costs real fraud dollars, so size the holdout with the power formula in Section 10. An audited random review sample is the other source of unbiased labels.

WHEN labels arrive: delayed labels

Chargeback delay is roughly exponential, mean 20 days. The share of a cohort's fraud already labeled at age a is

F(a)

the share of a cohort's fraud already labeled at age a

F

the labeled-share curve, read here at 7, 14, 30 and 60 days

a

the cohort's age in days

e

Euler's number, about 2.718; e^{-a/20} is the share still unlabeled (mean delay 20 days)

and 95% maturity takes 20\cdot\ln 20 = 59.9 days. On 14-day-old data the measured fraud rate and precision are about half their true values (precision 0.069 vs 0.134, a ratio of 0.51). Measured recall is not biased when the delay is independent of the score (0.352 vs 0.345 true; just noisier), because its numerator and denominator lose the same share. So "precision fell last week" is often just young data (Q29, Drill 18). Fixes: evaluate only matured cohorts (60 days or more here); divide by F(\text{age}) for an adjusted rate; use early proxy labels (issuer fraud reports, analyst confirmations) with a study of how they relate to final labels. Never retrain on immature labels as if missing chargebacks were legit.

WHO is in the test set: base-rate shift

Precision conditions on the prediction; TPR and FPR condition on the truth. By Bayes' rule (Card 1):

\pi

the base rate: the share of transactions that are fraud (1% on the day)

TPR and FPR don't contain \pi, so they carry over to a population with a different base rate. Precision does contain it. Memory hook: recall and FPR travel; precision stays home.

• The twelve: A's precision at 0.5 is 0.50 on the review sample and 0.0198 on the day (\pi = 1\%, TPR 0.5, FPR 0.25).
• The vendor: TPR 0.80 and FPR 0.05 on a 50/50 test set give precision 0.941; at our 1% the same model gives 0.139.
• The 100,000: at its cost-optimal cut A has TPR 0.345 and FPR 0.0218, precision 0.134; on a 50/50 rebalanced set the same cut shows 0.940. AUC is 0.868 either way. AP jumps from 0.133 to 0.866.

So: AUC, ROC, TPR and FPR are base-rate free; precision, AP, PR curves, accuracy, F1 and cost are not. Convert a vendor's precision with Card 1 before believing it. The cost-optimal threshold also moves with \pi, and Ch 3, Card 13 shows how to correct the probabilities (Card 1 here corrects the metric).

WHO the errors fall on: subgroups (light)

10% of customers are new to the country. Their legit transactions look novel (new device, foreign IP), and their true fraud rate is also higher. With model A at its global cost-optimal threshold:

A is calibrated in both groups, has similar AUC in both, and still declines good new-to-country customers 2.8 times as often. This is the impossibility result: when base rates differ, a model cannot be calibrated within groups and have equal FPR and FNR across them, unless it predicts perfectly (Kleinberg, Mullainathan and Raghavan, 2016; Chouldechova, 2017). Equalizing FPR with per-group thresholds (Hardt, Price and Srebro, 2016) gives up calibration or overall cost. Always report metrics per segment; deciding which trade to make is a policy choice, and Ch 29 owns it.

HOW labels are made: noisy labels (recap). Ch 3, Section 8: chargebacks under-record fraud (some victims never notice) and include friendly fraud (a real customer disputes a real purchase). Measured precision and recall are both off; an audited sample estimates the two error rates so you can correct them.

The assumptions behind every test-set number: R-I-C-H

A test-set metric estimates production performance only if the test set is R-I-C-H:

Each assumption gets a check you can run:

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_predict
from sklearn.metrics import roc_auc_score

rng = np.random.default_rng(1); n_cards = 4_000
card = np.repeat(np.arange(n_cards), 10); n = len(card)      # 10 rows per card, time-ordered by day
day = np.tile(np.arange(10) * 18, n_cards)                   # 180 days
u = rng.normal(size=n_cards)[card]                           # card risk shared by its rows
X = rng.normal(size=(n, 3)); X[:, 1] += 0.5 * (day >= 120)   # merchant mix drifts in the test period
y = (rng.random(n) < 1 / (1 + np.exp(-(-5 + X[:, 0] + X[:, 1] + 1.5 * u)))).astype(int)
leaky = np.where(y == 1, rng.random(n) < 0.85, rng.random(n) < 0.001)     # "chargeback filed" proxy
test = day >= 120

# R: Representative?  Adversarial validation: can a model tell test rows from train rows?
adv = cross_val_predict(LogisticRegression(), X, test, cv=5, method="predict_proba")[:, 1]
print("R  adversarial AUC:", round(roc_auc_score(test, adv), 3), "(0.5 = indistinguishable)")

# I: Independent rows?  SE of the test fraud rate treating rows vs whole cards as the units
yt = y[test]; per_card = yt.reshape(n_cards, -1).mean(1)     # each card's 3 test rows are adjacent
print("I  SE rows vs cards:", round(np.sqrt(yt.mean() * (1 - yt.mean()) / len(yt)), 5),
      round(per_card.std(ddof=1) / np.sqrt(n_cards), 5))

# C: Correct and complete labels?  Share of fraud labeled by cohort age under a 20-day mean delay
age = np.array([7, 14, 30, 60, 90])
print("C  share labeled by age:", (1 - np.exp(-age / 20)).round(3))

# H: Hands-off?  One feature alone that nearly separates the classes is a leak suspect
for name, f in [("x1", X[:, 0]), ("x2", X[:, 1]), ("proxy", leaky)]:
    print("H  single-feature AUC", name, round(roc_auc_score(y, f), 3))

The adversarial AUC lands well above 0.5 (the drift), the card-level SE is wider than the row one (shared card risk), young cohorts are mostly unlabeled, and the proxy's single-feature AUC towers over the real signals.

Remember: Who, What, How, When again. Blocked transactions never get labels: keep a random holdout. Young transactions look clean: wait for labels to mature. Recall and FPR travel; precision stays home. Same AUC by group can hide very different false-decline rates. A test set must be R-I-C-H.

Next question → We have an honest offline number. Why do offline wins so often vanish when the model meets real customers, and how do we prove a win live? (Section 10)

10. Offline vs Online: Designing the A/B Test

The question: A beats the production model offline. The VP wants proof in money. How do we design a live test of a fraud model: who is randomized, what is measured, how many customers, for how long?

The 30-second version: offline metrics are proxies; an A/B test (a randomized controlled experiment) measures the goal itself, because randomization makes the two arms alike in everything except the model. Before launch, set four things, UMPS: Unit, Metric, Power, Stop rule. Randomize customers or cards, not transactions. Pick one primary metric plus guardrails. For 80% power at a two-sided 5% level, each arm needs about n \approx 16\sigma^2/\delta^2 units: 190,000 customers per arm to detect a 0.2-point change in a 95% approval rate. Fix the horizon in advance, long enough for labels to mature. First read: why offline isn't enough, UMPS, the sample-size formula. Second read: the derivation and heavy tails.

Why offline wins don't transfer

You need this: the two-sample test. H_0: the arms have the same mean. With n units per arm and per-unit variance \sigma^2, the difference in means has SE \sigma\sqrt{2/n}, and z = (\text{difference})/\text{SE}. \alpha is the false-positive rate (type I error), conventionally 5% two-sided; power = 1 - \beta is the chance of detecting a true effect \delta (\beta is the type II error), conventionally 80%. A p-value is read as in Ch 1, Section 8.

UMPS: set by the umpire before play

• Unit. Randomize customers or cards, not transactions: a declined customer retries, and a fraudster probes. Analyze at the same unit. If you randomize customers but compute per-transaction SEs, the SEs are too small; use clustered SEs, or the delta method for ratio metrics such as fraud bps = fraud $/volume $:

\text{Var}

variance: how much the number would wobble across repeated experiments

\bar X

average fraud dollars per customer in the arm (bar = average)

\bar Y

average transaction volume ($) per customer in the arm

n

the number of customers in the arm

\mu_X

the true mean fraud dollars per customer

\mu_Y

the true mean volume per customer

\sigma_X^2

the per-customer variance of fraud dollars

\sigma_Y^2

the per-customer variance of volume

\sigma_{XY}

the per-customer covariance of fraud dollars and volume

• Metric. One primary (fraud loss in bps, or expected cost) plus guardrails that must not get worse: false-decline rate, approval rate, customer contacts, latency.
• Power and Stop rule. Compute n before launch (below). Then fix a horizon in advance, covering whole weeks and label maturity. Ramp 1% → 5% → 50% for safety, then hold.

Power and sample size (Card 15)

Require the true effect \delta to sit (z_{1-\alpha/2} + z_{1-\beta}) standard errors from zero:

\delta

the true effect to detect: the absolute difference between arms (e.g. 0.002)

z_{1-\alpha/2}

the normal cutoff for a two-sided test with false-positive rate \alpha: 1.960 at 5%

z_{1-\beta}

the normal value for power 1-\beta (\beta = miss rate): 0.842 for 80%

\sigma

the per-unit standard deviation of the metric

\sigma^2

the per-unit variance, e.g. p(1-p) = 0.0475 for a 95% approval rate

Memory hook: 16\sigma^2/\delta^2 per arm (\alpha = 0.05 two-sided, 80% power, \sigma^2 the per-unit variance, \delta the absolute difference). Approval rate p = 0.95 gives \sigma^2 = p(1 - p) = 0.0475. To detect \delta = 0.002 (0.2 points): 16 \times 0.0475/0.002^2 = 190,000 per arm; the exact constant gives 186,411 (Q30, Drill 14).

Turn it around for the minimum detectable effect: with 50,000 per arm, the MDE is 0.39 points.

Heavy-tailed metrics. Fraud dollars per customer are mostly $0 with a few huge losses: in the simulation, 0.1% of customers carry 53% of fraud dollars. The per-customer SD is $247 raw and $63 after winsorizing at the 99.9th percentile ($1,401), so the raw metric needs 15 times as many customers. Winsorize (and say so), use a binary proxy ("any fraud loss"), or a rank-based test. Trimming changes the estimand; report it as such.

Sample ratio mismatch (SRM). A 50/50 design that logs 50,600 vs 49,400 users gives \chi^2 = 14.4, p = 0.00015. Something (a redirect, a crash, bot filtering) is dropping users unevenly, and the comparison is broken whatever the metric says (Fabijan et al., 2019). Check SRM before reading results. An A/A test (both arms get the same model) checks the whole pipeline: its p-values should be uniform. Fraud-specific: the control arm keeps the incumbent, and both arms share the random holdout from Section 9, so labels stay unbiased in each.

Remember: randomization buys an apples-to-apples comparison of the goal itself. Call it like the UMPS: Unit, Metric, Power, Stop rule, set before launch. n \approx 16\sigma^2/\delta^2 per arm. Randomize the customer, not the transaction.

Next question → The test is running. The PM checks the dashboard every morning, the team tracks twenty metrics, and the first week looks amazing. What can go wrong now? (Section 11)

11. Reading the A/B Test: Peeking, Many Metrics, Novelty, CUPED and Interleaving

The question: after five days the dashboard says p = 0.03 on fraud loss, two of twenty guardrails are "significant", and week 1's lift is nearly twice week 2's. Ship it? And can we get an answer in half the time?

The 30-second version: a test lies in four ways, the four Ps: Peeking, Plurality, Primacy, Pollution. Peeking: checking daily and stopping at the first p < 0.05 inflates the false-positive rate from 5% to about 19% with 10 looks (8.3% with just 2); use a fixed horizon or a sequential design. Plurality: twenty metrics at 5% give one false alarm on average; control the family-wise error with Bonferroni or the false discovery rate with Benjamini–Hochberg. Primacy and novelty: behavior changes when something is new; plot the effect by day. Pollution: SRM, interference, bots. To go faster without cheating, use CUPED: subtract the part of the outcome predicted by pre-experiment data; variance falls by 1 - \rho^2. For rankers, interleaving needs far fewer users. First read: the four Ps, the peeking numbers, CUPED. Second read: BH, sequential tests, interleaving.

Peeking (Card 16)

Under H_0 the cumulative z-statistic is a scaled random walk. Each look is another chance to cross \pm 1.96, and the looks are correlated: \text{Corr}(Z_k, Z_l) = \sqrt{n_k/n_l}. For two equally spaced looks, \rho = \sqrt{1/2} and

P(\text{false positive})

the chance of declaring a winner when there is no real effect

P

the probability of what's in the brackets

Z_1

the cumulative z-statistic at the first look (halfway)

Z_2

the z-statistic at the second look (all the data)

\rho

the correlation between the two looks' z-statistics

The two-look value is exact (0.0831); the rest are simulated. As looks \to \infty the rate \to 1: by the law of the iterated logarithm, a random walk eventually crosses any fixed z-boundary. Peeking also inflates the estimate: tests that stop early stop on a lucky high (Q31; Worked problem 2 in the Interview Bank, FIGURE 5.11).

import numpy as np
from scipy import stats

rng = np.random.default_rng(0); reps, looks = 100_000, 14
S = np.cumsum(rng.normal(size=(reps, looks)), axis=1)        # A/A test: daily z-increments under H0
Z = S / np.sqrt(np.arange(1, looks + 1))                     # cumulative z-statistic at each daily look
print("fixed horizon:", np.mean(np.abs(Z[:, -1]) > 1.96))    # about 0.05
print("stop at first |z| > 1.96:", np.mean((np.abs(Z) > 1.96).any(axis=1)))  # about 0.22
r = np.sqrt(0.5); bvn = stats.multivariate_normal([0, 0], [[1, r], [r, 1]])  # two looks, exact
inside = bvn.cdf([1.96, 1.96]) - 2 * bvn.cdf([1.96, -1.96]) + bvn.cdf([-1.96, -1.96])
print("two looks, exact:", 1 - inside)                       # 0.083

Fixes, in order of simplicity: a fixed horizon. Group-sequential boundaries: Pocock (1977) uses one constant, 2.413 for 5 looks; O'Brien and Fleming (1979) start very strict, at 4.56, 3.23, 2.63, 2.28 and 2.04 for 5 looks. Both bring the rate back to 0.050 and 0.049. Alpha spending (Lan and DeMets, 1983) allows unplanned look times. Always-valid p-values (mSPRT; Johari, Koomen, Pekelis and Walsh, 2017) let you look whenever you like. Bayesian monitoring of P(B > A) with a stop-when-confident rule inflates frequentist error too. Peeking for futility (stopping because nothing is happening) doesn't raise type I error; it only costs power.

Plurality: many metrics (Card 17)

With m independent null metrics at \alpha = 0.05, the expected number of false alarms is m\alpha (1.0 for 20), and P(\text{at least one}) = 1 - 0.95^{20} = 0.64. The FWER (family-wise error rate) is P(\text{any false rejection}); Bonferroni tests each metric at \alpha/m, and the union bound gives \text{FWER} \le \alpha. The FDR (false discovery rate) = E[\text{false rejections}/\text{all rejections}]. Benjamini–Hochberg (1995): sort the p-values, find the largest i with p_{(i)} \le iq/m, reject the i smallest. Under independence, \text{FDR} = (m_0/m)q \le q.

Twenty metrics with sorted p-values 0.001, 0.004, 0.009, 0.012, 0.021, 0.04, 0.06, 0.2, 0.3, 0.5 and ten at 0.6. Bonferroni's cut is 0.05/20 = 0.0025: 1 rejection. BH's cuts are 0.0025, 0.005, 0.0075, 0.01, …; 0.004 \le 0.005 but 0.009 > 0.0075, 0.012 > 0.01 and no later p-value clears its cut: 2 rejections (Q32, Drill 16).

Bonferroni controls any false alarm; BH controls the share of false alarms and keeps more power. Pre-register one primary metric; treat guardrails as one-sided "do no harm" checks; and remember that segments are tests too ("it worked for iOS users in Brazil" is one of dozens of looks).

Primacy and novelty. Users react to newness. In a novelty simulation the daily lift starts high and decays to a true long-run 2.0%: week 1 averages 4.0%, week 2 2.2%, week 4 2.0%. Week 1 is 1.8 times week 2 and twice week 4, and a 28-day average still overstates the long run (2.55%). Plot the effect by day and by cohort of first exposure; trust only the flat part; for long-term effects keep a long-running holdback (Hohnhold, O'Brien and Tang, 2015).

Pollution. SRM (Section 10); interference, where one unit's outcome depends on another's assignment (a violation of SUTVA, the stable unit treatment value assumption), as in marketplaces or social features, handled with cluster randomization or switchback tests that alternate the whole market over time; and bots. In fraud, the adversary is the interference: fraudsters migrate to the weaker arm, making the stronger arm look better and the weaker one worse than either would alone.

CUPED: subtract what you could have predicted (Card 18)

Ch 1, Section 13 introduced CUPED as regression adjustment. Here's why it works. Let X be a pre-experiment covariate (last month's spend) and Y the outcome:

Y^{\text{cv}}

the CUPED-adjusted outcome for a customer

Y

the outcome measured during the test (e.g. this month's spend)

X

the pre-experiment covariate: last month's spend

\theta

the regression slope of Y on X: how much to subtract per unit of X

\bar X

the average of X over all customers (bar = average)

\text{Cov}

covariance: how much two quantities move together

\text{Var}

variance: the spread of a quantity around its mean

\rho

the correlation between X and Y (0.703 in the simulation)

\theta is the regression slope of Y on X, the value that minimizes \text{Var}(Y - \theta X). X was measured before randomization, so its mean is the same in both arms in expectation, and subtracting \theta(X - \bar X) leaves the treatment effect unbiased while removing the predictable part of the noise (Deng, Xu, Kohavi and Walker, 2013). Anything the treatment could affect is forbidden as X.

In the CUPED simulation (100,000 customers per arm, true lift 1%), the realized \rho is 0.703 and \theta = 0.703. Per-unit variance falls to 0.510 of the raw (theory 1 - \rho^2 = 0.505), so the same power needs about half the customers. At this sample size, power rises from 0.39 to 0.65, and the mean estimate is unchanged (bias \approx 0). Memory hook: CUPED subtracts what you could have predicted (Q33, Drill 17).

Interleaving for rankers. For a ranked list (merchant search, a review queue where analysts pick cases), team-draft interleaving merges both rankers' results into one list, alternating picks, and credits each click to the ranker that contributed the item. Each user compares both rankers at once, so the between-user noise cancels and far fewer users are needed (Chapelle, Joachims, Radlinski and Yue, 2012; Netflix's public write-up, 2017). It measures preference, not long-term business metrics, and it doesn't apply to an auto-block decision. Confirm a winner with a regular A/B test.

Bayesian A/B reports P(B > A \mid \text{data}) and an expected loss. That reads naturally, but it is not a cure for peeking unless the decision rule is fixed in advance. What to report: effect with CI, p-value, power achieved, SRM check, guardrails, and the effect-by-day plot.

Remember: the four Ps: Peeking, Plurality, Primacy, Pollution. Ten peeks turn 5% into about 19%. Bonferroni controls any false alarm; BH controls the share of false alarms. CUPED subtracts what you could have predicted: variance \times (1 - \rho^2).

Next question → The model shipped. Labels won't be back for weeks. How do we know it's still working tomorrow? (Section 12)

12. Monitoring After Launch: Drift, Label Delay and Alerting

The question: weeks after launch, the share of transactions scored above the review threshold has nearly doubled, and chargebacks for those weeks won't arrive for another month. Is the model broken, has the world changed, or both? And what do we watch?

The 30-second version: watch the pipe from left to right, DISCO: Data quality, Input drift, Score drift, Calibration (once labels arrive), Outcomes. Drift needs a number: the Population Stability Index, \text{PSI} = \sum (a_i - e_i) \ln(a_i/e_i) over bins, a symmetrized KL divergence. Rule of thumb: under 0.1 calm, 0.1–0.25 watch, over 0.25 act. The KS statistic has a test attached, but at production volumes every shift is "significant", so alert on size, not p-values. Score drift with stable inputs points to a pipeline bug; input drift with stable outcomes may be harmless. First read: DISCO, PSI and its thresholds. Second read: KS vs PSI, what to do when it fires.

DISCO

PSI (Card 19)

Bin the reference (training) scores at their deciles, so each bin holds e_i = 10\%. This week's shares are a_i:

\sum_i

add up over the 10 score bins i

a_i

this week's (actual) share of transactions in bin i

e_i

the reference (expected) share in bin i: 10% each, from the training deciles

\text{KL}(a\,\|\,e)

KL divergence: how surprised you'd be by shares a if you expected e

\text{KL}(e\,\|\,a)

the same surprise the other way round; PSI adds both directions

KL divergence \text{KL}(a\,\|\,e) = \sum a_i \ln(a_i/e_i) measures how surprised you'd be by a if you expected e; it is \ge 0, is 0 only when a = e, and is not symmetric. PSI adds both directions. Empty bins make it infinite, so floor shares at a small \varepsilon. For a unit-normal feature whose mean shifts by \delta SDs, unbinned \text{PSI} = \delta^2 (0.25 for \delta = 0.5); binning lowers it a little. The thresholds are a credit-scoring convention, not a theorem. Memory hook: PSI: under 0.1 calm, 0.1–0.25 watch, over 0.25 act.

import numpy as np

def psi(expected, actual, bins=10, eps=1e-4):
    """Population Stability Index with bins at the reference deciles; eps guards empty bins."""
    edges = np.quantile(expected, np.linspace(0, 1, bins + 1)); edges[[0, -1]] = -np.inf, np.inf
    e = np.maximum(np.histogram(expected, edges)[0] / len(expected), eps)
    a = np.maximum(np.histogram(actual, edges)[0] / len(actual), eps)
    return np.sum((a - e) * np.log(a / e))

e, a = np.full(4, 0.25), np.array([0.15, 0.20, 0.30, 0.35])  # Drill 19: shares per reference quartile
print("Drill 19 PSI:", np.sum((a - e) * np.log(a / e)).round(3))           # 0.105 -> watch
rng = np.random.default_rng(0); ref = rng.normal(size=50_000)
for shift in (0.0, 0.1, 0.25, 0.5):
    print(f"mean shift {shift:.2f} SD: PSI {psi(ref, rng.normal(shift, 1, 50_000)):.3f}")

Twelve weeks after launch (simulation)

Weekly batches of 50,000. From week 5, merchant risk x2 drifts up by 0.5 SD by week 8; it is a real change, and the true fraud rate rises with it. From week 10, an upstream bug sets device novelty x3 to 0 for 30% of rows.

Reading it: x2 enters "watch" in week 7. The score drifts with it, and so does the true fraud rate, so the queue growth is mostly the world, not a bug. In week 10, x3's PSI of 0.44 says "act", and it comes from a feed bug. The world didn't change; the pipe did (Q36). Check data quality first. KS (max gap between two CDFs) has a test, but with a million rows a shift of 0.01 SD gives KS 0.0036 with p = 6 \times 10^{-6}, while PSI is 0.00009. At production scale every drift test is significant; alert on size. Use \chi^2 for categorical features; Wasserstein distance is another size measure, in the feature's own units.

What drift looks like before labels arrive

Until labels arrive, use matured-cohort metrics for anything older than 60 days and early proxies for the rest (analyst confirm rate on reviewed cases, issuer fraud reports). Alert with persistence (k days in a row), adjust for weekly seasonality, and keep the alert count low enough that people still read them. What to do when it fires: a bug → fix the feed or roll back; prior shift with good ranking → recalibrate (refit the intercept or Platt on fresh matured labels, Ch 3); concept drift → retrain on recent data. Ch 28 owns the full monitoring stack.

Remember: DISCO: Data quality, Input drift, Score drift, Calibration, Outcomes. PSI: under 0.1 calm, 0.1–0.25 watch, over 0.25 act. At production scale every drift test is significant; alert on size.

Next question → Can we build the core of this chapter ourselves (AUC from ranks, AP, the Brier decomposition, a paired bootstrap) and match the library? (Section 13)

13. Build It From Scratch

The question: can we compute AUC, AP, the confusion family, the Brier decomposition and a paired bootstrap CI for \Delta\text{AUC} in about 50 lines of NumPy, and match scikit-learn exactly?

The 30-second version: AUC from ranks: rank all scores (ties get average ranks), sum the fraud ranks, subtract n_1(n_1 + 1)/2, divide by n_1 n_0. That is O(n \log n), exact, and handles ties. AP: sort once, take one step per distinct score, sum \text{precision} \times \text{recall gained}. Brier and its decomposition by bins. The paired bootstrap: resample row indices once per replicate and score both models on the same rows. Everything matches sklearn on the twelve and on the 100,000. First read: run the code once.

import numpy as np
from scipy.stats import rankdata
from sklearn.metrics import roc_auc_score, average_precision_score

def auc_rank(y, s):
    """Rank, sum the fraud ranks, subtract, divide. Ties get average ranks (count 1/2)."""
    y = np.asarray(y); r = rankdata(s)                       # average ranks, O(n log n)
    n1 = y.sum(); n0 = len(y) - n1
    return (r[y == 1].sum() - n1 * (n1 + 1) / 2) / (n1 * n0)

def average_precision(y, s):
    """Step-wise AP = sum over thresholds of (R_k - R_{k-1}) * P_k, one step per distinct score."""
    y = np.asarray(y); o = np.argsort(-np.asarray(s), kind="stable"); s_, y_ = np.asarray(s)[o], y[o]
    last = np.r_[np.diff(s_) != 0, True]                     # last row of each tied group
    tp = np.cumsum(y_)[last]; k = np.arange(1, len(y) + 1)[last]
    prec, rec = tp / k, tp / y.sum()
    return np.sum(np.diff(np.r_[0, rec]) * prec)

def confusion_at(y, s, t):
    y, f = np.asarray(y), np.asarray(s) >= t                 # flag if score >= threshold
    tp, fp = int(np.sum(f & (y == 1))), int(np.sum(f & (y == 0)))
    fn, tn = int(np.sum(~f & (y == 1))), int(np.sum(~f & (y == 0)))
    p, r = tp / max(tp + fp, 1), tp / max(tp + fn, 1)
    den = np.sqrt(float((tp + fp) * (tp + fn) * (tn + fp) * (tn + fn)))
    return dict(tp=tp, fp=fp, fn=fn, tn=tn, precision=p, recall=r,
                f1=2 * tp / (2 * tp + fp + fn), mcc=float((tp * tn - fp * fn) / den) if den else 0.0)

def brier_decomposition(y, p, bins):
    """Murphy: Brier = REL - RES + UNC; exact when p is constant inside each bin."""
    y, p = np.asarray(y, float), np.asarray(p, float); ob = y.mean()
    rel = res = 0.0
    for b in np.unique(bins):
        m = bins == b; w = m.mean()
        rel += w * (p[m].mean() - y[m].mean()) ** 2          # Lie: forecast vs outcome, per bin
        res += w * (y[m].mean() - ob) ** 2                   # Learn: bin outcome vs base rate
    unc = ob * (1 - ob)                                      # Luck: base-rate noise
    return dict(brier=float(np.mean((p - y) ** 2)), rel=rel, res=res, unc=unc, check=rel - res + unc)

def paired_bootstrap_auc(y, s1, s2, B=2000, seed=0):
    """Resample row indices ONCE per replicate and score both models on the same rows."""
    y = np.asarray(y); rng = np.random.default_rng(seed); n = len(y)
    g1, g2 = (np.unique(s, return_inverse=True)[1] for s in (s1, s2))   # tie groups, sorted once
    def wauc(g, w):                                          # AUC of the resampled data in O(n)
        w1 = np.bincount(g, w * (y == 1)); w0 = np.bincount(g, w * (y == 0))
        below = np.cumsum(w0) - w0                           # legit weight strictly below each group
        return np.sum(w1 * (below + 0.5 * w0)) / (w1.sum() * w0.sum())
    d = np.empty(B)
    for b in range(B):
        w = np.bincount(rng.integers(0, n, n), minlength=n).astype(float)
        d[b] = wauc(g1, w) - wauc(g2, w)
    return auc_rank(y, s1) - auc_rank(y, s2), np.percentile(d, [2.5, 97.5]), (d > 0).mean()

# The twelve review transactions (Section 1)
y = np.array([1, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0])
a = np.array([.90, .80, .70, .50, .40, .30, .25, .20, .15, .10, .05, .02])
b = np.array([.95, .60, .90, .40, .85, .35, .20, .02, .30, .10, .08, .05])
print(auc_rank(y, a), auc_rank(y, b), average_precision(y, a).round(4), average_precision(y, b).round(4))
print(confusion_at(y, a, 0.5), confusion_at(y, b, 0.5)["mcc"])
yt, st = np.array([1, 0, 1, 1, 0, 0]), np.array([.9, .9, .5, .5, .5, .1])      # a tied example
assert np.isclose(average_precision(yt, st), average_precision_score(yt, st))
assert np.isclose(auc_rank(yt, st), roc_auc_score(yt, st))
p8 = np.r_[[0.2] * 5, [0.6] * 5]; y8 = np.r_[1, 0, 0, 0, 0, 1, 1, 1, 1, 0]       # Drill 8
print({k: round(float(v), 4) for k, v in brier_decomposition(y8, p8, p8).items()})

# The 100,000: regenerate the simulated book (seed 5005) and fit A and B as in Section 1
from sklearn.linear_model import LogisticRegression
rng = np.random.default_rng(5005); n = 250_000
x = rng.normal(size=(n, 3)); eta = -5.85 + x @ np.array([1.0, 1.0, 0.8])     # true log-odds
yy = (rng.random(n) < 1 / (1 + np.exp(-eta))).astype(int)
c = np.where(yy == 1, rng.random(n) < 0.40, rng.random(n) < 0.002).astype(int)   # consortium flag
xb = np.column_stack([c, x[:, 0]])
A = LogisticRegression(C=1e6, max_iter=1000).fit(x[:100_000], yy[:100_000])
B = LogisticRegression(C=1e6, max_iter=1000, class_weight="balanced").fit(xb[:100_000], yy[:100_000])
yte, pa, pb = yy[150_000:], A.predict_proba(x[150_000:])[:, 1], B.predict_proba(xb[150_000:])[:, 1]
assert abs(auc_rank(yte, pa) - roc_auc_score(yte, pa)) < 1e-12
assert abs(average_precision(yte, pb) - average_precision_score(yte, pb)) < 1e-12
print(yte.sum(), auc_rank(yte, pa).round(4), auc_rank(yte, pb).round(4))
print(paired_bootstrap_auc(yte, pa, pb, B=2000, seed=0))

What it prints: on the twelve, AUC 0.78125 and 0.75, AP 0.6917 and 0.8333, and A's confusion matrix at 0.5 (TP 2, FP 2, FN 2, TN 6, MCC 0.25) next to B's MCC of 0.625, all as computed by hand in Sections 2–3. The tied example passes both asserts: average ranks give ties half credit, and AP takes one step per distinct score, as sklearn does. Drill 8's bins give REL 0.02, RES 0.09, UNC 0.25 and Brier 0.18 = 0.02 - 0.09 + 0.25, exact because the forecasts are constant within bins. On the 100,000: 974 frauds, AUC 0.8685 and 0.848, equal to sklearn to 1e-12, and \Delta\text{AUC} = 0.0205 with a bootstrap CI that matches Section 8's [0.005, 0.035] up to bootstrap noise (the seed differs). The O(n) weighted AUC inside the bootstrap makes 2,000 replicates take seconds.

Common mistake: roc_auc_score(y, model.predict(X)). The hard 0/1 predictions give a one-corner ROC, so you get the balanced accuracy at 0.5, not the AUC (Card 5). For B on the 100,000 that's 0.768 instead of 0.848; for A it's 0.510, because A almost never crosses 0.5. On the twelve, B's hard AUC (0.8125) even exceeds its real AUC (0.75), so the mistake can flatter as well as hurt. Pass predict_proba(X)[:, 1].

Try it: square the toy scores, a monotone transform. auc_rank is unchanged (0.78125), but confusion_at(y, a**2, 0.5) flags only T1 and T2, and F1 drops from 0.5 to 0.333. On the 100,000, A's F1 at a fixed 0.05 falls from 0.169 to 0.131 after squaring, while its best F1 over all thresholds stays at 0.208. Threshold metrics move with the scores; rank metrics don't (Card 4).

Remember: rank, sum the fraud ranks, subtract, divide. Bootstrap pairs by resampling indices once for both models.

Next question → How does this look in production code, and which library defaults bite? (Section 14)

14. Use It Like a Pro

The question: which sklearn.metrics and model_selection calls does a practitioner use, and where do their defaults silently give the wrong answer?

The 30-second version: most metric bugs are one of three. Check your SLA: Scores, Label, Average. Ranking and probability metrics need Scores (predict_proba[:, 1] or decision_function), not predict(). The positive Label must be the fraud class (pos_label, the column order of classes_). Multiclass metrics need an explicit Average (macro, micro, weighted). Then the split: StratifiedGroupKFold and TimeSeriesSplit(gap=…), with preprocessing inside a Pipeline. First read: SLA, the gotchas table, the checklist.

A complete evaluation block

import numpy as np, sklearn
from scipy import stats
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (roc_auc_score, average_precision_score, log_loss, brier_score_loss,
                             matthews_corrcoef, classification_report)
from sklearn.calibration import calibration_curve

rng = np.random.default_rng(5005); n = 250_000                 # same book as Section 13
x = rng.normal(size=(n, 3)); eta = -5.85 + x @ np.array([1.0, 1.0, 0.8])
y = (rng.random(n) < 1 / (1 + np.exp(-eta))).astype(int)
c = np.where(y == 1, rng.random(n) < 0.40, rng.random(n) < 0.002).astype(int)
tr, te = slice(0, 100_000), slice(150_000, n)
A = LogisticRegression(C=1e6, max_iter=1000).fit(x[tr], y[tr])
p, yt = A.predict_proba(x[te])[:, 1], y[te]                   # Scores, not predict()

print("sklearn", sklearn.__version__)
print("AUC", roc_auc_score(yt, p), "AP", average_precision_score(yt, p))
print("log loss", log_loss(yt, p), "Brier", brier_score_loss(yt, p))
frac_pos, mean_pred = calibration_curve(yt, p, n_bins=10, strategy="quantile")  # not "uniform" at 1%
t = 10 / (10 + 215.68)                                         # c_FP / (c_FP + mean fraud $)
flag = (p >= t).astype(int)
print(classification_report(yt, flag, target_names=["legit", "fraud"], digits=3))
print("MCC", matthews_corrcoef(yt, flag), "| the predict() mistake:", roc_auc_score(yt, A.predict(x[te])))

def delong(y, s1, s2):
    """DeLong (1988): variances and covariance of two correlated AUCs from structural components."""
    y = np.asarray(y); n1, n0 = y.sum(), len(y) - y.sum(); V10, V01, aucs = [], [], []
    for s in (s1, s2):
        r_all = stats.rankdata(s); r1 = stats.rankdata(s[y == 1]); r0 = stats.rankdata(s[y == 0])
        v10 = (r_all[y == 1] - r1) / n0                        # each fraud: share of legit below it
        v01 = 1 - (r_all[y == 0] - r0) / n1                    # each legit: share of frauds above it
        V10.append(v10); V01.append(v01); aucs.append(v10.mean())
    S = np.cov(V10) / n1 + np.cov(V01) / n0                    # 2x2 covariance of (AUC1, AUC2)
    z = (aucs[0] - aucs[1]) / np.sqrt(S[0, 0] + S[1, 1] - 2 * S[0, 1])
    return aucs, np.sqrt(np.diag(S)), z, 2 * stats.norm.sf(abs(z))

xb = np.column_stack([c, x[:, 0]])
B = LogisticRegression(C=1e6, max_iter=1000, class_weight="balanced").fit(xb[tr], y[tr])
print(delong(yt, p, B.predict_proba(xb[te])[:, 1]))

The DeLong function reproduces Section 8 exactly: SEs 0.0055 and 0.0075, z = 2.70, p = 0.007. The predict() line prints 0.510, the common mistake from Section 13.

Gotchas (checked against scikit-learn 1.9.1)

Splits and pipelines

import numpy as np
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedGroupKFold, TimeSeriesSplit, cross_validate

rng = np.random.default_rng(0); n_cards = 3_000              # grouped toy: cards with repeated rows
card = np.repeat(np.arange(n_cards), 1 + rng.poisson(9, n_cards)); n = len(card)
risk = rng.normal(size=n_cards)[card]                        # card-level risk, shared by its rows
X = np.column_stack([rng.normal(size=(n, 2)), risk + rng.normal(0, 0.5, n)])
y = (rng.random(n) < 1 / (1 + np.exp(-(-4.5 + X[:, 0] + 1.5 * risk)))).astype(int)

pipe = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))   # scaler refit per fold
cv = StratifiedGroupKFold(n_splits=5, shuffle=True, random_state=0)         # Rare + Repeated
res = cross_validate(pipe, X, y, groups=card, cv=cv,
                     scoring=["roc_auc", "average_precision", "neg_brier_score"])
print({k: v.mean().round(3) for k, v in res.items() if k.startswith("test_")})
for tr_idx, te_idx in TimeSeriesSplit(n_splits=3, gap=1_000).split(X):     # Recent: gap = purge
    print(tr_idx.max(), "->", te_idx.min())                                # rows assumed time-ordered

Inference with statsmodels

from statsmodels.stats.proportion import proportion_confint, proportion_effectsize
from statsmodels.stats.contingency_tables import mcnemar
from statsmodels.stats.multitest import multipletests
from statsmodels.stats.power import NormalIndPower

print(proportion_confint(336, 974, method="wilson"))                       # recall of A at its cost cut
print(mcnemar([[95, 108], [330, 441]], exact=False, correction=True))      # frauds, top-1% capacity
pv = [0.001, 0.004, 0.009, 0.012, 0.021, 0.04, 0.06, 0.2, 0.3, 0.5] + [0.6] * 10
print(multipletests(pv, 0.05, "bonferroni")[0].sum(), multipletests(pv, 0.05, "fdr_bh")[0].sum())
h = proportion_effectsize(0.95, 0.948)                                     # Cohen's h for 95% -> 94.8%
print(NormalIndPower().solve_power(effect_size=h, alpha=0.05, power=0.8))  # customers per arm

It prints the Wilson CI [0.316, 0.375] from Section 8, McNemar's 111.5, BH and Bonferroni's 2 and 1 rejections from Section 11, and about 190,000 customers per arm (via Cohen's arcsine effect size, close to Section 10's 186,411). DeLong isn't in sklearn or statsmodels; use the function above or the paired bootstrap.

Before you report a model's performance

• [ ] The split matches production: Rare, Repeated, Recent (Section 6)
• [ ] LEAKS audit done; test set used once (Section 7)
• [ ] The metric matches the job: Decide, Rank or Estimate (Section 1)
• [ ] CI and the number of positives reported (Section 8)
• [ ] A baseline is shown: flag nothing, current rules, current model
• [ ] Precision and cost at production base rate; enriched samples weighted back (Sections 5, 9)
• [ ] Only matured labels used; holdout labels for the blocked region; per-segment table (Section 9)
• [ ] Calibration checked if anyone consumes the probabilities (Section 4)

Remember: check your SLA: Scores, Label, Average. Scorers default to accuracy and thresholds default to 0.5; set both on purpose.

Next question → Where does evaluation make or break real systems, and what changes when the model is an LLM? (Section 15)

15. In the Real World, and How It Connects to LLMs

The question: how do fraud, credit, ads and search teams evaluate in practice, and why is evaluating an LLM the hardest version of this chapter?

The 30-second version: every industry has its own 1% problem and feedback loop: fraud and credit use precision at capacity, KS/Gini and random holdouts; ads use log loss and calibration because bids consume probabilities; search uses NDCG offline and interleaving online. LLM evaluation repeats every lesson at scale: benchmark contamination is leakage, pass@k needs an unbiased estimator, LLM-as-judge is a biased measuring instrument, and perplexity measures the training loss, not the task. First read: both tables.

Where evaluation decides things

How this chapter becomes LLM evaluation

Remember: every LLM evaluation failure is a chapter idea at scale: contamination is leakage, leaderboards are snooping, judges are biased instruments, and a 500-question benchmark has a ±4-point CI.

Next → The chapter on one page (Section 16). Then practice: the Derivations tab, then the Interview Bank.

16. The Chapter on One Page

One map of the whole chapter and every memory hook in one table. Use it before an interview, or to check a second read: if any row feels unfamiliar, go back to its section.

The memory map

Every hook in one place

Remember: if you can redraw the memory map on a blank page and say the hook for every row of the table, you have the chapter. Now practice.

17. Practice

The chapter ends here. Practice lives in two tabs, so this page stays a story you can read straight through.

• Ch 5 · Derivations: 20 derivation cards (precision from Bayes, AUC as a U-statistic, the binormal AUC, properness, the Brier decomposition, the F1 threshold, DeLong, McNemar, power, peeking, BH, CUPED, PSI, pass@k), each with its key move from the R-A-T-E-S toolkit and the common slip; about 24 formulas to memorize; and a mastery tracker.
• Ch 5 · Interview Bank: 39 questions in four levels, each answered in four beats; two worked research-level problems with figures (the binormal AUC and the peeking false-positive rate); 20 drills, each verified by exact computation or simulation; and 38 rapid-fire flashcards for the day before.

Suggested order: finish the chapter's second read → Derivations Cards 1–8 → Interview Bank Levels 1–2 → Cards 9–20 → Levels 3–4.

All chapter long we leaned on likelihoods (log loss), binomial proportions (recall's error bar), base rates as priors (precision at 1%), and a posterior (the true fraud probability). What exactly is a likelihood, and what does Bayes' rule really let us do? That is Chapter 6, Probability, Likelihood and Bayes.