Chapter 5 — Interview Bank

Every answer here follows the same four beats. Say the first two within 20 seconds; the last two are what turn a correct answer into a strong one.

Questions are stacked by difficulty. Each level assumes everything above it.

How questions are written here. Real interviewers rarely ask "define X". They describe a situation — a model, a chart, a stakeholder's claim — and watch whether you spot the concept underneath. So every question below opens with a scenario, then names the concept being tested, then gives the four beats. Practice by reading only the scenario and answering out loud before you look.

All numbers come from the chapter's running example: the twelve review transactions (4 frauds, 8 legit, drawn from a day of 400 with 1% fraud, so each legit row stands for 49.5 transactions of the day) and the 100,000 test transactions of the simulated book (974 frauds, base rate 0.974%), scored by model A (logistic regression on three risk signals, calibrated), model B (sees the consortium compromised-card flag, trained with balanced class weights, so its raw probabilities are inflated) and B-cal (B after Platt scaling).

Level 1 — Screen

Q1 · "99% accurate on day one"

The checkout PM sends a launch email: "Our new fraud model is 99.0% accurate on last month's transactions!" About 1% of transactions are fraud. You're on the thread. What does that number tell the company, and what should the email have reported instead?

• Concept tested: the accuracy paradox and the base-rate baseline.
• Intuition: when 99 of every 100 transactions are legit, a model that waves everything through is right 99 times out of 100.
• Answer: almost nothing. On the 100,000 test transactions, flagging nothing scores 0.9903 accuracy. Model A at sklearn's default threshold of 0.5 scores 0.9904 while catching 19 of 974 frauds (recall 0.0195). Report what the decision needs instead: precision and recall at the operating point (or precision at the review capacity), average precision, and every number next to the flag-nothing baseline.

TN

true negatives: legit transactions passed through

n

the number of test transactions (100,000)

\pi

the base rate: the share of transactions that are fraud (0.974%)

• Why: accuracy is dominated by the biggest box, the true negatives. Flag-nothing has balanced accuracy 0.5 and MCC 0 (sklearn's value for 0/0), which expose it at once (Card 2). Memory hook: doing nothing is 99% accurate.
• Next they'll ask: "If you could put one number in the email?" → It depends on the job (Decide, Rank, Estimate). For a review queue, precision at capacity; to compare rankers across thresholds, AP; for one threshold summary that uses all four boxes, MCC.

Q2 · The review team and the product team

Risk ops has 30 analysts who can review 1,000 transactions a day, out of 100,000. The checkout PM wants fewer good customers declined. Each asks you for "the" metric for the new fraud model. What do you give each of them, and why can't you maximize both?

• Concept tested: precision vs recall vs false-positive rate, and the threshold that trades them.
• Intuition: the threshold is one slider; pull it down to catch more fraud and more good customers fall into the net.
• Answer: risk ops gets precision@1,000 (how much of the queue is fraud) and recall@1,000 (how much fraud the queue holds). On the 100,000, B's top 1% is 42.5% fraud and holds 43.6% of all fraud; A's is 20.3% fraud and holds 20.8%. The PM gets the false-positive rate (the false-decline rate) and the approval rate. One threshold sets all of them, so pushing recall up pushes FPR up too.

TP

true positives: frauds flagged

FP

false positives: good customers flagged

FN

false negatives: frauds let through

• Why: P-P, R-R: precision divides by Predicted positives, recall by Real positives, and FPR by real negatives. Lowering the threshold adds rows to TP and FP at once (Card 2).
• Next they'll ask: "Who picks the threshold?" → Capacity picks it for the queue (the top 1,000). For auto-blocking, expected cost picks it (Q6). Pair every target with a guardrail that pulls the other way, or Goodhart's law bites: reward "fraud caught" alone and the team flags everything.

Q3 · "Explain AUC 0.87 to me"

The CFO, who has no statistics background, reads in a model review that model A has "AUC 0.87" on 100,000 test transactions. She asks: "So it's right 87% of the time?" Explain what the number means.

• Concept tested: AUC as a ranking probability.
• Intuition: pick one fraud and one good transaction at random and show both to the model; AUC is the chance it scores the fraud higher.
• Answer: no. AUC 0.868 means that for a randomly chosen fraud and a randomly chosen legit transaction, the model gives the fraud the higher score 86.8% of the time (ties count half). 0.5 is a coin flip; 1.0 is a perfect ranking. It doesn't count correct decisions at any threshold, and it says nothing about dollars.

P(s^+ > s^-)

the chance a random fraud scores above a random legit transaction

P(s^+ = s^-)

the chance their two scores tie (a tie counts half)

s^+

the score of a randomly chosen fraud

s^-

the score of a randomly chosen legit transaction

• Why: the area under the ROC staircase equals the share of fraud–legit pairs in the right order (Card 3). On the twelve review transactions, A puts 25 of the 32 pairs in the right order: 0.781.
• Next they'll ask: "Is 0.87 good?" → Compared with what: production (B, 0.848), the best possible on this data (the simulation's oracle, 0.923), and the part of the ranking the team actually uses, the very top (Q5).

Q4 · ROC looks great, PR looks awful

A teammate's dashboard for model A shows a ROC curve with AUC 0.868 and, on the next panel, a precision–recall curve with average precision 0.133. "One of these charts is lying," they say. Which one?

• Concept tested: the base rate enters precision but not TPR or FPR.
• Intuition: ROC grades the order inside each pile separately; PR asks how clean the flagged pile is, and the legit pile is a hundred times bigger.
• Answer: neither. TPR and FPR are each computed inside one class, so ROC ignores the 0.974% base rate. Precision divides by everything flagged, so each point of FPR is multiplied by 99,026 legit transactions. The PR baseline is the base rate itself: a random ranker has \text{AP} \approx \pi = 0.0097, so A's 0.133 is about 14 times random.

• Why: Bayes' rule on rates (Card 1); the PR baseline is \pi (Card 7). Memory hook: ROC asks about ranking; PR asks about purity.
• Next they'll ask: "Which one goes in the leadership deck?" → At 1% fraud, PR plus TPR at the FPR you can afford, because the review queue lives at the top of the list. Use ROC when classes are balanced or the base rate is unknown or changing.

Q5 · Same AUC, different models

Procurement compares two fraud models: AUC 0.868 and 0.848, "basically the same." But in its top 1% of scores the first holds 21% of all fraud and the second 44%. The procurement lead asks how that's possible, and which to buy for a review team that can only look at 1% of traffic.

• Concept tested: AUC averages over every threshold; ROC curves can cross.
• Intuition: AUC is a season average; the review team only plays one game, at the very top of the list.
• Answer: buy the 0.848 model (B). The two ROC curves cross once, at an FPR between 0.129 and 0.140. Below it B leads: at FPR 0.1%, B catches 33.0% of fraud and A 6.0%. A earns its AUC lead at FPRs above 14%, blocking far more good customers than any fraud team would. Compare at the operating point: precision@1% (0.425 vs 0.203), TPR at a fixed FPR, or partial AUC up to FPR 1% (McClish-standardized: 0.703 vs 0.572).

\int_0^1

add up the area under the ROC curve as FPR runs from 0 to 1

d\,\text{FPR}

a thin sliver of FPR: every sliver counts the same, wherever it sits

• Why: the integral weights every FPR equally, so a lead in a region nobody uses still buys AUC (Cards 3, 4).
• Next they'll ask: "Is B's lead at 1% real or luck?" → Compare the two on the same frauds with McNemar (Q22): at the top 1%, B alone catches 330 frauds and A alone 108, p \approx 5 \times 10^{-26}.

Q6 · Who picked 0.5?

*An engineer shipped the fraud model using model.predict(X). Blocking a good customer costs about $10 in friction and review; a missed fraud costs $200 on average. What threshold should the model use, and what has to be true for your answer to work?*

• Concept tested: the cost-optimal threshold (recap of Ch 3, Section 10).
• Intuition: block when the expected loss from letting it through beats the expected cost of a false alarm.
• Answer: block when p > 10/210 \approx 0.048, not 0.5. predict() uses 0.5, which assumes both errors cost the same. The rule needs three things: calibrated probabilities (on the simulation, whose average fraud is $216, the same rule gives t^* = 0.044; raw B cut there flags 98.6% of transactions and costs $9,766 per 1,000, against $1,231 for B-cal); a cost that doesn't vary by transaction (if it does, flag when p\cdot\text{amount} > c_\text{FP}: B-cal falls from $1,218 per 1,000 at its best single threshold to $1,012); and the same base rate as the data the probabilities came from.

p

the calibrated fraud probability of this transaction

c_{FN}

the cost of letting a fraud through: $200 on average

c_{FP}

the cost of blocking a good customer: about $10

\Longleftrightarrow

"exactly when": the two inequalities say the same thing

• Why: compare the expected cost of each action for one transaction with fraud probability p (Card 10).
• Next they'll ask: "What if you don't trust the probabilities?" → Read the threshold off the empirical expected-cost curve on validation data (A's minimum is $1,629 per 1,000 at t = 0.065), or recalibrate first (Platt, fit on held-out data).

Q7 · "Does 30% mean 30%?"

Finance wants to forecast next month's fraud losses by adding up the model's probabilities. The model ranks well (AUC 0.848), but its average predicted probability on a test month was 0.313, in a month where 0.974% of transactions were fraud. What do you tell finance, and how do you check a model's probabilities in general?

• Concept tested: calibration vs discrimination, and how calibration is measured.
• Intuition: a model can put every fraud in the right order and still shout every number 30 times too loud.
• Answer: don't sum these. This is raw B, trained with balanced class weights, so it predicts about 32 times the real fraud. Check calibration with: the mean prediction vs the observed rate (calibration in the large); a reliability diagram with equal-mass bins on log axes; Brier score and log loss, reported as skill against the base-rate forecast; ECE, stating the binning; and the calibration slope and intercept. Platt scaling on a held-out month fixes it: B-cal's mean is 0.0100, its Brier falls from 0.128 to 0.0068, and its AUC doesn't move.

P(y = 1 \mid \hat p = p)

the actual fraud rate among transactions the model scored at p

y

the true label: 1 if fraud, 0 if legit

\hat p

the model's predicted probability (hat = estimated)

p

any forecast value, e.g. 0.30

• Why: discrimination ranks, calibration tells the truth. Class weights shift the intercept (Ch 3, Section 8); the Brier decomposition separates the two properties (Card 9).
• Next they'll ask: "Where does calibration matter most?" → Where the probabilities are used: near the threshold for blocking decisions, everywhere for summing into a forecast (Q19).

Q8 · F1 everywhere

A team uses F1 as the single success metric for both its fraud model and its marketing-email propensity model, and sets each model's threshold by maximizing F1. When is F1 the wrong choice, and what would you use instead?

• Concept tested: what F1 ignores.
• Intuition: F1 is a referee who counts goals and missed shots but never looks at the rest of the pitch, or the price of each mistake.
• Answer: F1 never uses true negatives, weights precision and recall equally whatever the costs, and moves with the base rate. Its best threshold has nothing to do with money: B-cal's F1 peaks at a threshold of 0.257, while its expected cost is lowest at 0.057, less than a quarter of that. Use expected cost when costs are known, precision@k when capacity is fixed, F_\beta if misses are clearly worse (\beta > 1), and for the email model expected profit or uplift per message.

F_1

the F1 score: the harmonic mean of precision and recall

P

precision: the share of flagged transactions that are fraud

R

recall: the share of frauds that got flagged

F_\beta

the F-score that weights recall \beta times as much as precision

\beta

how much more recall matters than precision (\beta > 1: misses are worse)

• Why: TN is missing from the formula (Card 2), and the F1-optimal cut follows the F1 curve, not the cost curve (Card 11).
• Next they'll ask: "Where does F1 peak for a calibrated model?" → At half its best value, t^* = F_1^*/2: on the simulation's true probabilities the best F1 is 0.516, at t = 0.258 (Drill 7).

Q9 · MAPE on short trips

The delivery-time team (Ch 2's ETA model) reports MAPE 13.8% on 10,000 orders, much worse on short trips, and notices that models tuned on MAPE keep predicting too fast. They want to switch to RMSE "because it's standard." What do you recommend?

• Concept tested: each regression loss is minimized by a different summary of the outcome.
• Intuition: MAPE divides by the actual time, so being late on a 5-minute trip is a huge percentage, and the cheapest strategy is to promise early.
• Answer: pick the loss whose minimizer is the number the decision needs. RMSE → mean (staffing, totals), MAE → median (a typical trip), pinball → quantile (a promise). MAPE's best constant on these orders is 16.9 minutes, below the median of 20.1: it prefers under-forecasts. For a promise window, fit a q = 0.9 quantile regression: it covers 90.0% of orders, while the mean forecast is exceeded by half of them.

\ell_q(y, \hat y)

the pinball loss for one delivery at quantile level q

q

the target quantile, e.g. 0.9 for a "90% of the time" promise

y

the actual delivery time, in minutes

\hat y

the forecast delivery time (hat = estimated)

• Why: RMSE is minimized by the mean, MAE by the median, pinball loss at q by the q-th quantile (recap of Ch 1, Section 9: RMSE punishes, MAE forgives, MAPE breaks at zero).
• Next they'll ask: "What does RMSE notice that MAE doesn't?" → Rare disasters: storms on 1% of orders raise RMSE 1.65 times but MAE only 1.16 times.

Q10 · The vendor's 94% precision

A vendor's deck shows 94.1% precision at 80% recall, measured on a balanced 50/50 test set of fraud and legit transactions. Your fraud rate is 1%. What precision should you expect in production, and what do you ask the vendor for?

• Concept tested: prior shift: what carries over to a new base rate and what doesn't.
• Intuition: the same net cast into a pond with far more good fish brings up far more good fish.
• Answer: about 14%. Back out the FPR from the vendor's numbers: at \pi = 0.5, 0.941 = 0.8/(0.8 + \text{FPR}) gives \text{FPR} = 0.05. TPR and FPR carry over; precision doesn't. Ask the vendor for TPR and FPR (or the full ROC), the number of frauds behind them, and how the test set was built and dated.

• Why: TPR and FPR condition on the truth, so \pi isn't in them; precision conditions on the prediction (Card 1). Memory hook: recall and FPR travel; precision stays home.
• Next they'll ask: "What else does a 50/50 test set distort?" → AP, accuracy, F1 and cost. Model A's AP is 0.133 at the natural rate and 0.866 on a rebalanced set; its AUC is 0.868 on both.

Q11 · "Just use random 5-fold"

A new hire cross-validates the fraud model with shuffled 5-fold CV on six months of transactions. Cards appear many times, fraud tactics drift month to month, and chargebacks take up to 60 days. The CV AUC is 0.90. Is that number trustworthy?

• Concept tested: validation design: split like production.
• Intuition: a test is a dress rehearsal; rehearse with the same cast and last month's script and opening night will still surprise you.
• Answer: no. Next month brings new cards and new tactics, and a random split rehearses neither. On the chapter's card-level simulation: random split 0.904, stratified 5-fold 0.898, grouped by card 0.848, out-of-time 0.863, grouped and out-of-time 0.824, the number to believe. Use StratifiedGroupKFold by card, or train on the past and test on the future with a purge gap for the label window.
• Why: Rare → stratify, Repeated → group, Recent → split by time. The 0.055 drop from grouping is how much the random split memorized cards; the time split removes the drift the model hasn't seen. Adversarial validation agrees: train vs test is indistinguishable under the random split (AUC 0.500) and distinguishable out of time (0.582).
• Next they'll ask: "When is a random split fine?" → Independent rows, no repeated entities, a stable world. That is rare in fraud (Q25).

Q12 · Offline up, online flat

A new fraud model beat production offline by 2 AUC points (0.868 vs 0.848, 95% CI on the difference [0.005, 0.035]). After launch, fraud losses didn't move. The PM asks whether the offline metric is broken.

• Concept tested: the proxy ladder and why offline wins don't transfer.
• Intuition: the offline metric is a rung below the goal; you climbed a rung that doesn't lead where the money is.
• Answer: the metric isn't broken; it measured something else. Check in order: proxy mismatch (AUC gained at FPRs nobody operates at, Q5); feedback-loop labels (the offline test only had labels for what the old model let through, Q28); adaptation (fraudsters probe the new model and move); novelty; interference between arms; training–serving skew (a feature computed differently online); and plain power (was the online test big enough to see the effect?).
• Why: each rung of Goal → Online → Offline → Loss stands in for the one above, and each gap is a place to be fooled. Only a randomized test measures the goal itself (Section 10).
• Next they'll ask: "How would you have caught it before launch?" → Evaluate at the operating point, shadow-score live traffic to compare score distributions, and design the A/B test up front (Q30, Q34).

Level 2 — Onsite core

Q13 · Twelve transactions on the whiteboard

The interviewer writes the twelve review transactions: labels (1, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0) and model A's scores (0.90, 0.80, 0.70, 0.50, 0.40, 0.30, 0.25, 0.20, 0.15, 0.10, 0.05, 0.02). "At threshold 0.5, give me the confusion matrix, precision, recall, F1, FPR and MCC. Then tell me why MCC is so much lower than accuracy."

• Concept tested: confusion-matrix algebra.
• Intuition: four boxes, and every metric is one box divided by a row, a column or everything.
• Answer: A flags T1–T4: TP 2, FP 2, FN 2, TN 6. Precision 2/4 = 0.50, recall 2/4 = 0.50, F1 0.50, FPR 2/8 = 0.25, accuracy 8/12 = 0.667.

TP

true positives: frauds flagged (A at 0.5: 2)

TN

true negatives: legit transactions passed (6)

FP

false positives: legit transactions flagged (2)

FN

false negatives: frauds passed (2)

Accuracy 0.667 is exactly what flagging nothing scores on this sample (8 of 12 are legit), so A is no better than doing nothing by that measure. MCC is a correlation between prediction and truth, so it is zero for any no-skill rule and charges for both kinds of error.

• Why: MCC is the Pearson correlation of two 0/1 variables (Card 2); accuracy has no chance correction.
• Next they'll ask: "And on the day the sample came from?" → Each legit row stands for 49.5 transactions, so the 2 FPs become 99: precision 0.0198, accuracy 0.7475, MCC 0.057. Recall and FPR don't change.

Q14 · Count the pairs

Same twelve transactions and A's scores. "No libraries. Compute A's AUC two ways: by counting pairs, and with the rank-sum formula. How would a tie be counted?"

• Concept tested: AUC as a pair count and as a Mann–Whitney statistic.
• Intuition: line up every fraud against every legit and count the fights the fraud wins.
• Answer: 4 frauds \times 8 legit = 32 pairs. T1 beats all 8 legit, T3 beats 7 (not T2), T5 beats 6 (not T2, T4), T8 beats 4 (T9–T12): 25/32 = 0.78125. With ranks (1 = lowest score), A's frauds sit at ranks 12, 10, 8 and 5, so R_1 = 35:

R_1

the sum of the frauds' ranks, rank 1 = lowest score (A: 12 + 10 + 8 + 5 = 35)

n_1

the number of frauds (4)

n_0

the number of legit transactions (8)

A tie counts one half: tied scores get the average of their ranks, which adds exactly \tfrac12 per tied pair.

• Why: subtracting n_1(n_1 + 1)/2 removes the frauds' comparisons with each other, leaving the Mann–Whitney U, the number of fraud–legit pairs won (Card 3). Memory hook: rank, sum the fraud ranks, subtract, divide.
• Next they'll ask: "Model B?" → Frauds at ranks 12, 11, 10 and 1 (B scores T8 at 0.02, below every legit): R_1 = 34, \text{AUC} = 24/32 = 0.75.

Q15 · Prove the area is a probability

"You keep saying AUC is the chance a random fraud outranks a random legit. The ROC curve is a plot of TPR against FPR. Prove that its area equals that probability."

• Concept tested: the probabilistic meaning of the ROC area.
• Intuition: each step right on the ROC is one legit transaction; the height there is the share of frauds scored above it; averaging those heights over all legit is the pair count.
• Answer: let \bar F_1(t) = P(s^+ > t) and \bar F_0(t) = P(s^- > t) with density f_0. As the threshold t falls from +\infty to -\infty, the ROC traces (\text{FPR}, \text{TPR}) = (\bar F_0(t), \bar F_1(t)), and d\text{FPR} = f_0(t)\,dt:

\int_0^1

add up the area under the ROC curve as FPR runs from 0 to 1

d\,\text{FPR}

a thin sliver of FPR; here d\text{FPR} = f_0(t)\,dt

\int_{-\infty}^{\infty}

add up over every threshold value t

\bar F_1(t)

the share of fraud scores above t: the TPR at threshold t

f_0(t)

the density of legit scores at t

dt

a thin slice of threshold values

t

the threshold, sliding from +\infty down to -\infty

P(s^+ > t)

the chance a random fraud scores above t

P(s^+ > s^-)

the chance a random fraud outscores a random legit transaction

s^+

the score of a random fraud

s^-

the score of a random legit transaction

The last step conditions on the legit score s^- = t and averages over its density, using independence of the two draws. On data, each legit adds a column of width 1/n_0 whose height is the share of frauds above it; summing gives (\text{pairs won})/(n_1 n_0).

• Why: conditioning on one draw and integrating over it turns an area into a probability (Card 3).
• Next they'll ask: "Why does that matter in practice?" → AUC is then a two-sample U-statistic, so it has a known variance (DeLong, Hanley–McNeil, Card 13) and a known null distribution (Mann–Whitney).

Q16 · The squaring trick

A teammate squared model A's scores "to sharpen them" and reports that F1 at the production threshold of 0.05 fell from 0.169 to 0.131 on the 100,000 test transactions, so squaring "made the model worse." Another teammate says AUC is unchanged at 0.868, so nothing happened. Who's right?

• Concept tested: which metrics are invariant to monotone transforms.
• Intuition: squaring keeps every transaction in the same place in the queue; only the label on the ruler changed.
• Answer: the second, mostly. The ranking is identical, so AUC, AP, the whole ROC and PR curves and the best F1 over thresholds (0.208 before and after) don't change. Fixed-threshold F1 changed because the threshold effectively moved: s^2 > 0.05 is the same as s > \sqrt{0.05} \approx 0.224 on the old scale. What squaring does break is calibration, so log loss and Brier change too.

s

model A's original score

\sqrt t

the same cut on the old scale: s^2 > t means s clears the square root of t

g

any strictly increasing transform of the score (squaring, log, ranks)

\Longleftrightarrow

"exactly when": both cuts flag the same transactions

• Why: AUC depends only on the order of scores (Card 4). On the twelve review transactions, squaring A drops F1 at 0.5 from 0.50 to 0.333 and leaves the AUC (0.781) and max F1 (0.667) unchanged (Drill 4).
• Next they'll ask: "What isn't invariant?" → Anything that reads the score's value: thresholds, log loss, Brier, calibration curves. Max-F1 is invariant to transforms but not to the base rate.

Q17 · Draw both curves

"Sketch the ROC and PR curves of model B on the twelve review transactions. B's scores are (0.95, 0.60, 0.90, 0.40, 0.85, 0.35, 0.20, 0.02, 0.30, 0.10, 0.08, 0.05). Why does the PR curve zig-zag, why can't you join PR points with straight lines, and which number does sklearn report as AP?"

• Concept tested: curve shapes and average precision.
• Intuition: ROC walks up for each fraud and right for each legit; PR's precision jumps up with every fraud and sags with every legit.
• Answer: sorted by B: T1, T3, T5 (frauds), then all eight legit, then T8. ROC: three steps up to TPR 0.75 at FPR 0, a long walk right to FPR 1, then one step up: area 24/32 = 0.75. PR: precision 1.0 at recall 0.25, 0.5 and 0.75, then sagging from 0.75 to 0.273 as the legit rows arrive, then a jump to 0.333 at recall 1. sklearn's average_precision_score uses the step-wise sum:

\sum_k

add up over every cut-off k down B's ranked list

R_k

recall when the top k transactions are flagged

R_{k-1}

recall at the cut-off before, so R_k - R_{k-1} is the step in recall

P_k

precision when the top k transactions are flagged

Straight lines are wrong because precision TP/(TP + FP) is not linear in recall between two operating points (Davis and Goadrich, 2006). The trapezoid auc(recall, precision) gives 0.826 for B and 0.649 for A (vs AP 0.692): a different number.

• Why: AP is the mean of precision at each fraud's rank (Card 7).
• Next they'll ask: "What's a random model's AP?" → The base rate: precision \pi at every recall. On the twelve that's 0.333; on the 100,000, 0.0097.

Q18 · Log loss says A, Brier says B

On the twelve review transactions, A has the lower log loss (0.523 vs 0.576) but B has the lower Brier score (0.149 vs 0.181). "Both are proper scoring rules. How can they disagree, and which would you trust for a fraud-loss forecast?"

• Concept tested: proper scoring rules weight errors differently.
• Intuition: log loss is a judge who can't forgive one confident lie; Brier caps the penalty for any single row at 1.
• Answer: properness only says each rule is optimized by the true probabilities; between two imperfect forecasts, the rules can rank differently. B scores the $900 fraud T8 at 0.02. That one row costs -\ln 0.02 = 3.91 nats in log loss, but only (0.98)^2 = 0.96 in Brier, so B's three confident hits outweigh it under Brier. Without T8, B's log loss is 0.272. For a fraud-loss forecast, the confident miss on the biggest fraud is exactly the risk finance cares about, so log loss's verdict is the relevant one. With 12 rows, neither difference means much.

\frac1n

divide by n, the number of transactions scored (12): an average

\sum_i

add up over every transaction i

y_i

the true label of transaction i: 1 if fraud, 0 if legit

p_i

the predicted fraud probability for transaction i (B gives T8 0.02)

• Why: both expected scores are minimized at p = q (Card 8), but log loss is unbounded near 0 and 1 while Brier is bounded.
• Next they'll ask: "Why not just use absolute error of the probability?" → It's improper. At true q = 0.3, reporting 0.3 scores an expected 0.42 and reporting 0 scores 0.30, so it rewards rounding to 0 or 1.

Q19 · Brier 0.009 sounds amazing

A calibration report says model A's Brier score is 0.0090 on the 100,000 test transactions, "very close to the perfect 0." Decompose it and tell the reader what it means at a 0.974% base rate.

• Concept tested: the Brier decomposition and skill scores.
• Intuition: at 1% fraud even a forecaster who knows nothing scores near zero, so the score is mostly luck.
• Answer: with 200 equal-mass bins, \text{REL}\ 0.000018 - \text{RES}\ 0.00061 + \text{UNC}\ 0.00965 (plus a binning residual of -0.00006) = 0.0090. Almost all of it is UNC, the base rate's own noise, which no model can remove: a constant forecast of the base rate scores 0.00965. A is well calibrated (tiny REL) but resolves little (RES 0.00061). Its Brier skill score is 1 - 0.0090/0.00965 = 0.067: 7% better than knowing nothing. B-cal's skill is 0.296 and the oracle's 0.350.

\sum_k

add up over the forecast bins k

n

the total number of transactions scored

n_k

the number of transactions in bin k

f_k

the mean forecast in bin k

\bar o_k

the observed fraud rate in bin k (bar = average of the 0/1 outcomes)

\bar o

the overall fraud rate: the base rate

• Why: add and subtract each bin's observed rate inside the square; the cross term vanishes within bins (Card 9). Memory hook: Brier = Lie − Learn + Luck.
• Next they'll ask: "Does the binning matter?" → Yes, with continuous forecasts. With 10 bins B-cal's resolution reads 0.00029 instead of 0.00270, so always state the bins.

Q20 · ECE moved when we changed bins

Model A's expected calibration error is 0.00045 with 10 uniform bins and 0.00098 with 20 quantile bins on the same 100,000 transactions. A reviewer asks which is right, and adds: "Would a model that always predicts the base rate beat ours on ECE?"

• Concept tested: ECE as a binned estimator and its pitfalls.
• Intuition: ECE measures the gap between a ruler and the truth with a ruler of its own, and the second ruler's marks change the answer.
• Answer: neither is "right"; they estimate the same thing with different bins. Uniform bins crowd a 1% model: 98.7% of A's rows land in the first bin (0–0.1), hiding everything. More bins add noise to each bin's observed rate, which shows up as a gap even for a perfect model, so ECE rises with bin count (A reaches 0.00146 at 50 quantile bins). And yes: the constant base-rate forecast scores ECE 0.00034, lower than A, with AUC 0.5. ECE measures honesty, not usefulness.

• Why: \lvert \bar o_k - f_k\rvert is biased upward when \bar o_k is noisy, and the bins decide what is averaged together (Section 4). Report ECE with its binning, next to a proper score (Card 8) and a bin-free check.
• Next they'll ask: "What's bin-free?" → Regress y on \operatorname{logit}(\hat p): calibration slope (ideal 1) and intercept (ideal 0). A: 0.98 and −0.03. Raw B: 1.03 and −4.63, the right shape but odds about 100 times too high.

Q21 · Is 0.868 vs 0.848 real?

A's AUC is 0.868 and B's 0.848 on the same 100,000 test transactions, which hold 974 frauds. "Give me a confidence interval for the difference and a test. Why should the comparison be paired? And why do you keep talking about the 974 rather than the 100,000?"

• Concept tested: uncertainty of AUC; paired comparison.
• Intuition: both models face the same hard and easy frauds, so most of their luck is shared, and comparing them on the same rows cancels it.
• Answer: paired bootstrap (2,000 replicates, resampling row indices once per replicate and scoring both models on them): \Delta\text{AUC} = 0.0205, 95% CI [0.005, 0.035], 99.6% of replicates positive. DeLong: SE(A) 0.0055, SE(B) 0.0075, correlation 0.34, SE of the difference 0.0076, z = 2.70, p = 0.007. Ignoring the pairing gives SE 0.0093 and z = 2.21. An AUC's error bar is driven mostly by the smaller class, so your n is your frauds.

\text{Var}(\widehat{\text{AUC}}_A - \widehat{\text{AUC}}_B)

the variance of the estimated AUC gap between A and B on the same rows

\text{Var}_A

the variance of A's AUC estimate

\text{Var}_B

the variance of B's AUC estimate

\text{Cov}_{AB}

the covariance of the two AUCs: shared luck from the same frauds, so subtracting it is free precision

\widehat{\text{Var}}(\widehat{\text{AUC}})

DeLong's estimate of one AUC's variance (hat = estimated)

S_{10}

sample variance of each fraud's share of legit transactions it outranks

S_{01}

sample variance of each legit transaction's share of frauds that outrank it

• Why: AUC is a two-sample U-statistic; DeLong estimates its variance from each fraud's and each legit's structural components (Card 13). Positive covariance shrinks the variance of the difference.
• Next they'll ask: "Quick hand estimate?" → Hanley–McNeil from the AUC and the class counts: 0.0074 for A, conservative here (DeLong 0.0055, bootstrap 0.0054), because it assumes exponential-shaped score distributions.

Q22 · 60 disagreements

At the review capacity, two fraud models flag almost the same transactions. Among the frauds, A alone catches 40 and B alone catches 20; on every other fraud they agree. "Is A better, or is this noise?"

• Concept tested: McNemar's test for paired decisions.
• Intuition: frauds both models catch, or both miss, can't tell you which model is better; only the disagreements vote.
• Answer: under H_0 (equal catch rates), each of the 60 discordant frauds is a fair coin: A-only with probability \tfrac12.

\chi^2

McNemar's statistic: how lopsided the disagreements are

b

frauds caught by A only (40)

c

frauds caught by B only (20)

\sim

is distributed as, if the two catch rates were equal

\chi^2_1

the chi-squared distribution with 1 degree of freedom

p

the p-value: the chance of a split this lopsided if the models were equal

The exact binomial version, P(\lvert \text{Bin}(60, \tfrac12) - 30\rvert \ge 10), gives p \approx 0.0135. So A is better at this capacity, at the 5% level.

• Why: conditional on b + c, b \sim \text{Bin}(b + c, \tfrac12) under H_0; the concordant counts cancel from the comparison (Card 14).
• Next they'll ask: "What did the real test set show?" → At the top 1% of the 100,000, B alone caught 330 frauds and A alone 108: \chi^2 = 111.5, p \approx 5 \times 10^{-26}, so B is far better at capacity even though A has the higher AUC (Q5).

Q23 · The 0.98 feature

A new feature, "card flagged in the network's fraud report," lifts the honest out-of-time AUC on the card-level simulation from 0.824 to 0.980, and it holds 73% of permutation importance. "Walk me through how you'd decide whether to ship it."

• Concept tested: leakage audit: label proxies and known-later information.
• Intuition: a feature that knows the answer this well was probably written after the answer was known.
• Answer: treat it as a leak until proven otherwise. Ask the audit question of every feature: would I know this at the moment of decision? Check the feature's timestamp, not its name: a fraud report is filed after a chargeback or a customer complaint, so at authorization time it doesn't exist yet. Check how it's created (here it fires for 85% of frauds and 0.1% of legit, almost the label itself). Rebuild it as of each transaction's time from the feature store and re-evaluate out of time. If the point-in-time version still helps, ship that version.
• Why: leakage only ever flatters; one dominant feature and an AUC near 1 are its two loudest alarms. This is L (label proxy) and K (known later) of L-E-A-K-S (Section 7).
• Next they'll ask: "How do you stop this happening again?" → Point-in-time joins in the feature store, a single-feature AUC scan before every model review, and an out-of-time test as the reporting default.

Q24 · Target encoding before the split

Two bugs land in your review queue. (1) A teammate target-encoded merchant ID (each merchant's fraud rate) on the full dataset, then ran CV; AUC rose from 0.824 to 0.942. (2) Someone picked the 20 features most correlated with the label from 5,000 candidates using all 200 rows, then ran 5-fold CV and got 80% accuracy. The 5,000 features turned out to be pure noise. Diagnose both.

• Concept tested: all-data preprocessing leakage.
• Intuition: if the test rows helped build the features, the test already saw the answers.
• Answer: both are the A of L-E-A-K-S. (1) Each test row's own label is inside its merchant's encoded rate, so the feature partly is the label. Fix: out-of-fold target encoding, fit inside each training fold. (2) The selection step looked at every row's label, so the 20 "winners" are the noise columns that happen to match the labels everywhere, including the test folds. Moving the selection inside each fold drops accuracy to 0.535, chance plus noise (Ambroise and McLachlan, 2002).
• Why: every data-dependent step (scaling, imputation, encoding, selection, resampling) is part of the model and must be refit on training rows only. sklearn's Pipeline with cross_validate does it per fold.
• Next they'll ask: "Does scaling on all data leak too?" → In principle yes, a little (test means and SDs inform the transform); in practice it rarely moves results, but put it in the pipeline anyway, because the habit catches the steps that do leak.

Q25 · Random split 0.904, time split 0.863

On the card-level simulation, the same model scores AUC 0.904 on a random 80/20 split and 0.863 when trained on days 1–120 (January–April) and tested on days 151–180 (June). "Which do you report, why is there a gap, and what is the empty May for?"

• Concept tested: temporal validation, entity overlap and purging.
• Intuition: the random split lets the model study next month's cards and next month's fraud patterns before the exam.
• Answer: report 0.863, or better the grouped out-of-time 0.824 if cards repeat across periods. The gap has two sources: entity overlap (the same cards in train and test) and drift (the merchant-risk effect weakens over the 180 days). May is a purge. Labels resolve up to 60 days after the transaction, and fraud on a compromised card continues for weeks, so May rows carry outcomes and card histories that play out in June. Training through May reports 0.878, an optimistic number; the purge removes 19,984 rows.
• Why: Recent → split by time, and purge any training row whose label window overlaps the test period (López de Prado, 2018). An embargo after the test fold matters only in K-fold over time, where training folds can follow the test fold.
• Next they'll ask: "How would you implement the gap?" → TimeSeriesSplit(gap=…) with the gap at least the label window, or explicit date filters.

Q26 · Write AUC

*"Implement AUC in NumPy in O(n log n), handling ties, and check it against sklearn. Then tell me what roc_auc_score(y, model.predict(X)) actually computes."*

• Concept tested: the rank formula, and the hard-prediction mistake.
• Intuition: sort once, and the rank-sum counts every fraud–legit pair for free.
• Answer:

import numpy as np
from scipy.stats import rankdata
from sklearn.metrics import roc_auc_score, balanced_accuracy_score

def auc_rank(y, s):
    """AUC by the rank-sum formula. O(n log n); ties get average ranks (count 1/2)."""
    y = np.asarray(y, dtype=bool)
    r = rankdata(s)                                   # 1 = lowest score; ties averaged
    n1, n0 = y.sum(), (~y).sum()
    return (r[y].sum() - n1 * (n1 + 1) / 2) / (n1 * n0)

y = np.array([1, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0])            # the twelve review transactions
a = np.array([.90, .80, .70, .50, .40, .30, .25, .20, .15, .10, .05, .02])
b = np.array([.95, .60, .90, .40, .85, .35, .20, .02, .30, .10, .08, .05])
print(auc_rank(y, a), roc_auc_score(y, a))                     # 0.78125 0.78125

rng = np.random.default_rng(0)                                  # heavy ties: scores on a 0..4 grid
yt = rng.random(10_000) < 0.05; st = rng.integers(0, 5, 10_000) + yt
print(np.isclose(auc_rank(yt, st), roc_auc_score(yt, st)))      # True

hard = (b >= 0.5).astype(int)                                   # what model.predict() returns
print(roc_auc_score(y, hard), balanced_accuracy_score(y, hard)) # 0.8125 0.8125, not 0.75

roc_auc_score on 0/1 predictions sees a one-corner ROC, so it returns (TPR + TNR)/2, the balanced accuracy at 0.5. On the 100,000 that's 0.768 for B (TPR 0.696, FPR 0.159), against its real AUC of 0.848.

• Why: the hard classifier's ROC is two straight segments through (FPR, TPR) (Card 5); the rank formula is Card 3. Memory hook: SLA: Scores, Label, Average, and the S is "pass scores, not predictions."
• Next they'll ask: "And a bootstrap CI?" → Draw row indices with replacement once per replicate, compute both models' AUC on those rows, take the 2.5th and 97.5th percentiles of the difference (Q21).

Q27 · 1% capacity, dollars not counts

Review capacity is 1% of daily volume. From the cumulative gains table of the 100,000 test transactions (974 frauds): A's top 1,000 contain 203 frauds and B's 425. "Compute precision@k, recall@k and lift for both models. Then tell me why dollar recall might reverse a ranking like this."

• Concept tested: capacity metrics and dollar weighting.
• Intuition: the queue holds 1,000 cases; what matters is how much fraud, and how many fraud dollars, end up in it.
• Answer: precision@1% = 203/1{,}000 = 0.203 (A) and 0.425 (B). Recall@1% = 203/974 = 0.208 and 425/974 = 0.436. Lift = \text{precision}/\pi = 20.8 and 43.6 (a random queue would be 0.974% fraud).

k

the queue size: the top 1% = 1,000 transactions

\pi

the base rate: 0.974% of transactions are fraud

Dollar recall weights each fraud by its amount, and a model can rank many small frauds high and one huge one low. On the twelve review transactions, B ranks three frauds first but scores the $900 fraud below every legit one, so its top 3 hold $830 of $1,730 (48.0%), while A's top 3 hold $680 (39.3%). Here B still wins; on the 100,000 the dollar view agrees too (A 20.4%, B 44.3% of fraud dollars at 1%). Report both, because they don't have to agree.

• Why: count recall treats a $12 fraud like a $900 one; the cost curve doesn't (Card 10).
• Next they'll ask: "How would you rank by dollars directly?" → Sort by expected loss p\cdot\text{amount} instead of p: the amount-aware rule (flag if p\cdot\text{amount} > c_\text{FP}) cuts B-cal's cost per 1,000 from $1,218 to $1,012.

Level 3 — Senior

Q28 · We only learn about fraud we let through

The incumbent model (A) auto-declines its top 1% of transactions, so those never complete and never produce a chargeback. A challenger (B) is evaluated on the remaining labeled traffic, and its AUC comes out 0.823, below the 0.848 the team expected. "Is this comparison fair? Design an evaluation that is." This feedback-loop question is a staple of payments and fraud interviews.

• Concept tested: selection-biased labels and counterfactual evaluation.
• Intuition: you are grading the challenger only on the exam questions the incumbent chose not to answer.
• Answer: it isn't fair. A's blocks remove 20.8% of the test frauds from the labeled data, and the ones left are the frauds A found hard, so both models look worse: true AUCs A 0.868 and B 0.848, but on unblocked rows 0.841 and 0.823. The direction is set by the blocking policy; in general a challenger that resembles the incumbent can look better or worse than it is. The fix is a random holdout: let a small, logged fraction of would-be blocks through, then weight each labeled row by 1/P(it was let through).

\widehat{\text{metric}}_{\text{IPW}}

the metric estimated with inverse-propensity weights (hat = estimated)

w_i

the weight on labeled transaction i

P(\text{labeled}_i)

the chance transaction i was let through, and so got a label

With a 2% holdout, the IPW AUC of B averages 0.846 over 300 holdouts (truth 0.848), but with SD 0.014, because 2% of a 1% block rate labels only about 20 blocked rows (about 4 frauds) per 100,000. A 10% holdout cuts the SD to 0.006.

• Why: inverse-propensity weighting makes the labeled rows represent all traffic again, as long as every transaction had a known, non-zero chance of being labeled (Who, What, How, When: this is the WHO). Stripe's public talk on counterfactual evaluation describes this design (Manapat, 2015).
• Next they'll ask: "What does the holdout cost?" → Real fraud: per 1,000 transactions a day, a 2% holdout releases 0.2 would-be blocks, about $8.56 of fraud. Size the holdout like an experiment (Q30): bigger means cheaper error bars and dearer losses. An audited random review sample is the other source of unbiased labels.

Q29 · Last week looks clean

The fraud rate measured on transactions from the last 14 days is about half the long-run rate, and the new model's precision on those days looks half as good, or, in another team's framing, its "false alarms doubled." A teammate also wants to retrain on all data up to yesterday to catch the newest patterns. What's going on, and what do you advise?

• Concept tested: delayed labels (right-censoring).
• Intuition: chargebacks are letters still in the post; a quiet mailbox doesn't mean nobody wrote.
• Answer: chargeback delay here is about exponential with a mean of 20 days, so only half of 14-day-old fraud is labeled yet: F(14) = 1 - e^{-14/20} = 0.503. The measured fraud rate and precision are both understated by that factor (precision 0.069 measured vs 0.134 true for A at its cost cut, a ratio of 0.51). Measured recall is not biased when the delay is independent of the score (0.352 vs 0.345 true): its numerator and denominator lose the same share. Evaluate on matured cohorts only (95% maturity takes 20\cdot\ln 20 = 59.9 days), or divide young cohorts' counts by F(\text{age}). And don't retrain on immature labels: unlabeled frauds would enter training as legit, teaching the model that the newest fraud pattern is safe.

F(a)

the share of a cohort's fraud already labeled at age a

a

the cohort's age in days

e

Euler's number, about 2.718; e^{-a/20} is the share still unlabeled (mean delay 20 days)

• Why: this is the WHEN of Who, What, How, When. Precision divides by flagged rows, whose count doesn't wait for labels; recall's denominator is labeled fraud, which shrinks with the numerator (Drill 18).
• Next they'll ask: "But you need recent data to catch new fraud. Then what?" → Use early proxy labels (issuer fraud reports, analyst confirmations) with a study of how they map to final labels, train on matured data plus proxies, or use a survival model of the label delay.

Q30 · Design the live test

The VP wants an A/B test of the new fraud model, decided within four weeks. Approval rate is 95%; fraud is about 1% of transactions; fraud dollars per customer are heavy-tailed; chargebacks arrive over about two months. Each arm can get 50,000 customers a day. "Design it: unit, metrics, sample size, duration."

• Concept tested: A/B design end to end.
• Intuition: an umpire sets the rules before the first pitch, not after seeing the score.
• Answer: set UMPS before launch. Unit: the customer (or card), not the transaction, because declined customers retry and fraudsters probe; analyze per customer, with the delta method for ratio metrics. Metric: primary = fraud loss in basis points of volume (or expected cost); guardrails = approval rate, false declines, contacts. Power: to detect a 0.2-point drop in the 95% approval rate at \alpha = 0.05 two-sided with 80% power:

n

the customers needed in each arm

z_{0.975}

the normal cutoff for a two-sided 5% test: 1.96

z_{0.8}

the normal value for 80% power: 0.84

p

the approval rate: 0.95

\delta

the smallest change worth detecting: 0.002 (0.2 points)

\sigma^2

the per-customer variance, p(1-p) = 0.0475

That's 4 days of traffic. Fraud dollars are the hard metric: 0.1% of customers carry 53% of fraud dollars, so the raw per-customer SD ($247) needs about 15 times the sample of a metric winsorized at the 99.9th percentile (SD $63); winsorize and say so, or test the binary "any fraud loss". Stop rule: four weeks of exposure (whole weeks, past any novelty), then read guardrails at once and fraud loss when labels mature: a 30-day-old cohort is 77.7% labeled, a 60-day-old one 95.0%. Check sample ratio mismatch first (50,600 vs 49,400 on a 50/50 design gives \chi^2 = 14.4, p = 0.00015: stop and debug). Both arms share the random holdout from Q28 so their labels stay unbiased.

• Why: the difference of two means has SE \sigma\sqrt{2/n}; requiring the true \delta to sit 1.96 + 0.84 SEs from zero gives the formula (Card 15). Memory hook: 16\sigma^2/\delta^2 per arm (\alpha 0.05 two-sided, 80% power, \sigma^2 per unit, \delta absolute).
• Next they'll ask: "The VP says four weeks, not four months." → Decide on the guardrails and a fast-maturing proxy (issuer fraud reports, delay-adjusted fraud counts with F(\text{age})) at four weeks, with a pre-registered rule; confirm the fraud-loss effect on matured labels later, keeping a small holdback arm.

Q31 · Ship on day 5

The PM checks the A/B dashboard every morning of a planned 14-day test and wants to stop as soon as fraud loss shows p < 0.05. On day 5 it shows p = 0.03. "What's the real false-positive rate of this procedure, and what would you do instead?" Peeking is one of the most commonly reported A/B questions at large tech companies.

• Concept tested: optional stopping inflates type I error.
• Intuition: each look is another lottery ticket; the cumulative z-score is a random walk, and a wandering walk eventually crosses a fixed line.
• Answer: under no true effect, stopping at the first |z| > 1.96 over 14 daily looks gives a false-positive rate of about 0.22, not 0.05 (simulated, 100,000 A/A tests). Checking only up to day 5 already gives 0.14. Two looks give 0.083, exactly:

\text{Corr}

correlation: how tightly two quantities move together

Z_k

the cumulative z-statistic at look k

Z_l

the cumulative z-statistic at a later look l

n_k

customers in the data by look k

n_l

customers in the data by look l

P(\text{reject})

the chance of a false alarm across both looks when there is no effect

P

the probability of what's in the brackets

Z_1

the z-statistic at the first (halfway) look

Z_2

the z-statistic at the second (final) look

\rho

the correlation between the two looks: \sqrt{1/2} when equally spaced

Instead: keep the fixed 14-day horizon; or, if early stopping is valuable, pre-register a group-sequential design (Pocock: |z| > 2.413 at each of 5 looks; O'Brien–Fleming: 4.56, 3.23, 2.63, 2.28, 2.04), which brings the error back to 0.05; or use alpha spending (Lan–DeMets) or always-valid p-values (mSPRT; Johari et al., 2017) built into the platform.

• Why: the looks share data, so they're correlated but not identical; each extra look adds a little more chance to cross, and as looks \to \infty the rate \to 1 (Card 16; Worked problem 2).
• Next they'll ask: "Does stopping early also bias the effect size?" → Yes: a test that stops on a crossing stops on a lucky high, so the reported lift is inflated (a winner's curse). Stopping early for futility doesn't raise type I error; it only costs power.

Q32 · Two of twenty

The experiment dashboard tracks 20 metrics. Two are significant at 0.05: p = 0.004 and p = 0.012; the other 18 are above 0.05. The PM wants to announce both as wins. What do you say?

• Concept tested: multiple testing: FWER vs FDR.
• Intuition: buy twenty lottery tickets and one winning by chance isn't luck; it's the expected outcome.
• Answer: with 20 null metrics at 0.05 you expect 1.0 false alarm, and P(\text{at least one}) = 1 - 0.95^{20} = 0.64. Neither survives a correction. Bonferroni tests each at 0.05/20 = 0.0025: 0.004 fails. Benjamini–Hochberg compares the i-th smallest p-value with 0.05\cdot i/20: 0.004 > 0.0025 and 0.012 > 0.005, and no later p-value clears its cut, so zero rejections.

p_i

the p-value of metric i

\alpha

the overall false-alarm rate you allow: 0.05

m

the number of metrics tested: 20

p_{(1)}

the smallest p-value (brackets = sorted order)

p_{(k)}

the k-th smallest p-value: the last one rejected

k

how many metrics BH rejects

p_{(i)}

the i-th smallest p-value

i

a p-value's rank when sorted smallest first

q

the false discovery rate you allow: 0.05

Say: "Two nominal signals worth a pre-registered follow-up test, not wins." Pre-register one primary metric; treat the rest as guardrails or exploration.

• Why: the union bound gives Bonferroni \text{FWER} \le \alpha; BH controls the expected share of false discoveries at (m_0/m)q \le q under independence (Card 17). In 10,000 simulated 20-metric tests with 4 real effects, BH's FDR was 0.040 (theory 16/20 \times 0.05 = 0.04) and it found 63% of real effects, against Bonferroni's 49%.
• Next they'll ask: "Do segments count?" → Yes. "It worked for iOS users in Brazil" is one of dozens of implicit tests; correct for them, or treat it as a hypothesis for the next experiment.

Q33 · Four weeks is too long

Experiments on per-customer spend take four weeks to reach 80% power. Leadership wants answers in two. "What can you do without peeking or cheating?"

• Concept tested: variance reduction with CUPED.
• Intuition: much of the noise in this month's spend is predictable from last month's; subtract what you could have predicted and the treatment effect stands out.
• Answer: sample size scales with the variance, so halving the duration needs the variance halved. CUPED does it with a pre-experiment covariate X (last month's spend):

Y^{\text{cv}}

the CUPED-adjusted outcome for a customer

Y

the outcome during the test: this month's spend

X

the pre-experiment covariate: last month's spend

\theta

the regression slope of Y on X: how much to subtract per unit of X

\bar X

the average of X over all customers (bar = average)

\text{Cov}

covariance: how much two quantities move together

\text{Var}

variance: the spread of a quantity around its mean

\rho

the correlation between X and Y (about 0.71 here)

With \rho \approx 0.71 (\rho^2 = 0.5) two weeks suffice. In the chapter's simulation (100,000 customers per arm, \rho = 0.703), the variance ratio is 0.510 (theory 0.505), power at fixed n rises from 0.39 to 0.65, and the estimate stays unbiased. Stack other legal tools: winsorize heavy-tailed spend (it cut the needed sample 15-fold for fraud dollars in Q30), stratify randomization on the covariate, or move to a more sensitive metric that the business accepts as a proxy.

• Why: \theta is the regression slope minimizing \text{Var}(Y - \theta X); X is measured before randomization, so it has the same mean in both arms and the adjustment can't bias the effect (Card 18). Memory hook: CUPED subtracts what you could have predicted.
• Next they'll ask: "Can I use in-experiment engagement as the covariate?" → No. Anything the treatment can affect is forbidden: adjusting for it removes part of the effect, or biases it.

Q34 · Offline +2 AUC, online: more false declines

The challenger beat production offline by 2 AUC points (0.868 vs 0.848). In the A/B test, fraud losses didn't change, false declines rose significantly, and the challenger's week-1 lift on the fraud-detection rate was twice its week-4 lift. "Diagnose."

• Concept tested: offline–online gaps: operating point, threshold transfer, novelty, interference.
• Intuition: the challenger won a race on a track nobody uses, then was started from the wrong line.
• Answer: four suspects. (1) The AUC gain lives where nobody operates. These two models' ROC curves cross near FPR 0.13; at FPR 0.1% the challenger catches 6.0% of fraud and production 33.0%. At a realistic operating point it can be worse. (2) The threshold didn't transfer. If the old score cut was reused, a model with a different score scale flags a different share of traffic; more flags means more false declines. Set the challenger's threshold from its own calibrated probabilities or from capacity. (3) Novelty. A week-1 lift twice week 4's is a decaying effect (in the chapter's novelty simulation week 1 averages 4.0% against a true 2.0%); trust only the flat part. (4) Interference. Fraudsters probe and migrate to the weaker arm, so arm differences don't predict a full rollout.
• Why: the proxy ladder (Goal → Online → Offline → Loss): AUC averages over every FPR (Card 3), while the business pays at one point. The four Ps cover the reading: Primacy and Pollution here.
• Next they'll ask: "What should the offline metric have been?" → The one the decision uses: TPR at the FPR the business allows, precision at capacity, or expected cost at the threshold you will ship (Section 5).

Q35 · Same AUC, different customers

At the global cost-optimal threshold, model A's AUC is 0.869 for domestic customers and 0.860 for customers new to the country, but the false-decline rate is 2.2% for domestic customers and 6.2% for new-to-country customers. The model is calibrated in both groups. Compliance asks whether the model is fair and what you would change.

• Concept tested: subgroup metrics and the impossibility result.
• Intuition: the new-to-country customers' good transactions look novel (new device, foreign IP), so one threshold catches more of them.
• Answer: the false-decline rate is 2.8 times higher for new-to-country customers. Calibration and similar AUC don't prevent that: their true fraud rate is also higher (2.04% vs 0.98%) and their legit scores sit higher, so more legit rows cross one global threshold (flag rate 7.1% vs 2.5%). When base rates differ, a model cannot be calibrated within groups and have equal FPR and FNR across them unless it predicts perfectly (Kleinberg, Mullainathan and Raghavan, 2016; Chouldechova, 2017). Options: per-group thresholds to equalize FPR (Hardt, Price and Srebro, 2016), which trades away calibration-based cost optimality; better features that separate "new" from "risky" (travel notices, verified relocation); step-up authentication instead of declines for that group. Which trade to make is a policy decision; your job is to put the per-segment table in front of it.

• Why: with calibrated scores, FPR at a fixed threshold depends on how each group's legit scores are distributed, which differs when base rates and features differ (Section 9; Ch 29 owns fairness).
• Next they'll ask: "Should 'new to country' be a feature?" → It may reduce false declines by explaining the novelty, but using group membership directly can be legally restricted; that is a compliance decision, informed by both versions' per-segment metrics.

Q36 · PSI 0.44, labels in 60 days

Weeks after launch, the monitoring dashboard shows: merchant-risk input PSI 0.23 since week 8; device-novelty input PSI 0.44 in week 10; score PSI 0.10; and the share of transactions above the review threshold up from 2.5% to 4.9%. Chargebacks for these weeks won't arrive for about 60 days. "What do you check, in what order, and do you roll back?"

• Concept tested: drift monitoring before labels arrive.
• Intuition: follow the pipe left to right; a sudden jump in one input is usually a broken pipe, a slow drift is usually the world.
• Answer: DISCO, in order. Data quality first: device novelty jumping to PSI 0.44 in a single week looks like a feed problem, and here it is (an upstream bug sets it to 0 for 30% of rows): fix the feed or roll back to a safe default. Input drift: merchant risk crept up from week 5 (PSI crossed 0.1 in week 7, 0.23 by week 8), a gradual shift; score drift followed (0.095 in week 8), and the true fraud rate rose with it (1.0\% \to 1.66\%), so the queue growth is mostly the world, not the model. Calibration and Outcomes wait for matured labels; meanwhile use proxies (analyst confirm rate on reviewed cases, issuer reports). Roll back only for the bug; for the drift, recalibrate on fresh matured labels and plan a retrain.

\sum_i

add up over the score bins i

a_i

this week's (actual) share of transactions in bin i

e_i

the reference (expected) share in bin i, from training

\text{KL}(a\,\|\,e)

KL divergence: how surprised you'd be by shares a if you expected e

\text{KL}(e\,\|\,a)

the same surprise the other way round; PSI adds both directions

• Why: PSI is a symmetrized KL divergence over reference bins (Card 19). Memory hook: PSI: under 0.1 calm, 0.1–0.25 watch, over 0.25 act.
• Next they'll ask: "Why not alert on the KS test's p-value?" → At production volume every shift is significant: with a million rows, a 0.01-SD shift gives KS p = 6 \times 10^{-6} while PSI is 0.00009. Alert on size, with persistence (k days in a row) and seasonality adjustment.

Q37 · The leaderboard model

The team tried 20 model configurations, picked the one with the best 5-fold CV AUC, and reports that same CV AUC to leadership, with an error bar of "SD over folds / \sqrt{5}." "What's wrong with the number, and with the error bar?"

• Concept tested: selection optimism (winner's curse) and the variance of CV estimates.
• Intuition: the tallest of twenty people measured with a wobbly ruler was probably measured on a good day.
• Answer: the number is optimistic. If the 20 configurations were equally good and scored with independent errors of SE \sigma, the winner's expected score is 1.87\sigma above the truth (the expected maximum of 20 standard normals); at \sigma = 0.01 that's +0.019. In the chapter's snooping simulation, the winner picked on a 130-fraud test set scored 0.844 there and 0.837 on fresh data; over 40 test sets its optimism averaged 0.011 \pm 0.003, of which 0.006 was pure selection. The error bar is too narrow for unstable learners: the folds share training data, so they're correlated. Across 60 independent datasets of 5,000 rows, the true spread of gradient boosting's 5-fold AUC was 1.31 times the naive \text{SD}/\sqrt{K} (logistic regression 0.93, about right). Fix: report the winner on an untouched test set (or nested CV), with a bootstrap CI over test rows or groups.

\mathbb E

expected value: the average over many repeats of the whole experiment

\max_{j \le 20}

the largest over the 20 configurations

Z_j

configuration j's test-set luck, in standard errors

\sim

is distributed as

N(0, 1)

the standard normal: mean 0, SD 1

• Why: the max of noisy estimates is biased up (selection); and no unbiased estimator of the variance of K-fold CV exists (Bengio and Grandvalet, 2004). This is the S of L-E-A-K-S.
• Next they'll ask: "How do you compare two learners with CV, then?" → The 5\times 2 CV paired t-test (Dietterich, 1998) or the corrected resampled t-test (Nadeau and Bengio, 2003), which inflates the variance for train–test overlap.

Q38 · Testing a new ranker with less traffic

The merchant-search team in the consumer app has a new ranking model. An A/B test on conversion would need about eight weeks of traffic. "Is there a faster online method? What does it measure, and what does it miss?"

• Concept tested: interleaving.
• Intuition: instead of asking two crowds which restaurant they liked, give one person a plate with both chefs' dishes and see which they eat.
• Answer: team-draft interleaving: for each query, the two rankers take turns (a coin flip decides who picks first) adding their highest-ranked item not already in the list; each click is credited to the ranker that contributed the item. The ranker with more credited clicks wins that query, and a sign test over queries decides. Each user compares both rankers at once, so between-user variance cancels and it typically needs far fewer users than an A/B test (Chapelle, Joachims, Radlinski and Yue, 2012; Netflix's public write-up, 2017). Alternating picks also balances position bias between the two. It misses: business outcomes (it measures click preference, not conversion, revenue or retention), whole-page effects, and anything that isn't a ranked list. Use it to screen candidates fast, then confirm the winner with a regular A/B test on the goal metric.
• Why: a within-user comparison removes the biggest noise source, the difference between users; it's the same logic as pairing in Q21 and McNemar in Q22.
• Next they'll ask: "Offline first?" → NDCG@k on logged judgments (Ch 15 owns depth), keeping in mind that logged clicks are biased toward what the old ranker showed: the feedback loop of Q28 again.

Q39 · Evaluating the case-summary LLM

The fraud team built an LLM that summarizes cases for analysts. It scores 70% on a 500-question public summarization benchmark (the previous model scored 67%), wins most head-to-head comparisons judged by an LLM from the same model family, and has the lowest perplexity on internal case notes. A research lead asks which of these numbers you would trust, and what you would measure instead.

• Concept tested: LLM evaluation pitfalls: contamination, judge bias, perplexity, small-benchmark noise.
• Intuition: every number here is a chapter idea at scale: a leaky test, a biased instrument and a training loss mistaken for the task.
• Answer: none of them settles it. Benchmark: at 70% on 500 questions, \text{SE} = \sqrt{0.7 \times 0.3/500} = 0.020, so the 95% CI is about \pm 4 points; a 3-point gain is within noise unless analyzed paired, question by question, and a public benchmark may be in the pretraining data (contamination is leakage: check n-gram overlap and canary strings, prefer fresh held-out sets). LLM judge: judges favor the first position, longer answers and their own family's style; swap positions and average, control for length, use reference answers, and calibrate the judge against human labels (report agreement with Cohen's \kappa). Perplexity is token log loss, a proper score on next tokens, not a measure of summary quality. Instead: a human-labeled random sample of real cases scored against a rubric (accuracy of facts, missing red flags), with a CI ("your n is your questions"), plus an online test on analyst time-to-decision and decision accuracy.

• Why: the variance of a proportion (Card 12); judges are measuring instruments with bias (Section 15). Memory hook: judges like to go first, go long, and look like themselves. Resolving ±1 point needs about 8,000 questions.
• Next they'll ask: "The summaries feed an agent that takes actions. How do you score reliability?" → For tasks with a checkable answer, estimate pass@k without bias as 1 - C(n - c, k)/C(n, k) (Card 20), and report \text{pass}^k (all k attempts succeed) when the same action must work every time.

Level 4 — Research and quant

These questions hand you a score distribution, a base rate, a sampling scheme or a test procedure, and ask what a metric or an error rate will come out to. You answer on paper, with no data. There's one method, and every problem below uses it.

The recipe: Condition on the truth, then count.

1. Write the metric as a probability or an expectation over (score, label). \text{TPR} = P(s > t \mid y = 1). \text{Precision} = P(y = 1 \mid s > t). \text{AUC} = P(s^+ > s^-). The false-positive rate of a test procedure = P(\text{reject at any look} \mid H_0).
2. Condition on the true label, or on H_0. Everything then reduces to the base rate \pi and two class-conditional score distributions: F_1 for frauds and F_0 for legit. For a test, it reduces to the null distribution of the statistic.
3. Count or integrate. Use Bayes on rates for anything that divides by predicted positives. Use a two-sample comparison for anything defined over pairs. Use a binomial or a normal for anything that is a proportion.
4. Attach the error bar. Every metric here is a proportion or a U-statistic, so its variance is roughly p(1 - p)/n. Here n counts the units in the conditioning class: your frauds, your discordant pairs, your customers.
5. Sanity-check the limits. Then say how you'd detect the problem in real data and fix it. Check a random scorer (AUC 0.5, precision \pi), a perfect scorer, \pi \to 0, and one look vs infinitely many.

t

the threshold: flag when the score is above t

F_1(t)

the share of fraud scores at or below t (the fraud score distribution)

F_0(t)

the share of legit scores at or below t

\pi

the base rate: the share of transactions that are fraud

Ch 1's "Cov over Var, then expand" and Ch 3's "score to zero, in expectation" are the same move: write the population quantity first, then substitute the data-generating process. The derivations use the R-A-T-E-S toolkit. Rank-pair counting does AUC. Total probability does precision. Sampling variance does every error bar.

Worked problem 1 — "A random fraud outranks a random legit": the binormal model

A staple of research-scientist and quant loops: prove what AUC means, then show that a high AUC can sit next to a precision in single digits.

Setup: legit scores s^- \sim N(0, 1) and fraud scores s^+ \sim N(d, 1), independent, with d = 2. The base rate is \pi. (1) Prove that the area under the ROC curve equals P(s^+ > s^-). (2) Compute the AUC in closed form. (3) What happens to the AUC if every score is passed through \exp(\cdot)? (4) At the threshold that catches 80% of fraud, what are the FPR and the precision when \pi = 0.5 and when \pi = 0.01? (5) Now let fraud scores be N(d, 4). What changes?

Intuition first: AUC only asks which of two scores is bigger, so it is a statement about the difference of two normals. Any monotone squeeze of the scores keeps every order, so it can't move the AUC. Precision is different. It multiplies the FPR by the size of the legit pile, and at 1% fraud that pile is 99 times the fraud pile.

Solution.

(1) Slide the threshold t from +\infty down to -\infty. The ROC curve is the path of (\text{FPR}(t), \text{TPR}(t)) = (1 - F_0(t), 1 - F_1(t)). Change variables from FPR to t; d\text{FPR} = -f_0(t)\,dt, and the limits flip.

\int_0^1

add up the area under the ROC curve as FPR runs from 0 to 1

d\,\text{FPR}

a thin sliver of FPR; here d\text{FPR} = -f_0(t)\,dt

\int_{-\infty}^{\infty}

add up over every threshold value t

P(s^+ > t)

the chance a random fraud scores above t: the TPR at t

f_0(t)

the density of legit scores at t

dt

a thin slice of threshold values

P(s^+ > s^-)

the chance a random fraud outscores a random legit transaction

s^+

the score of a random fraud

s^-

the score of a random legit transaction

The middle integral averages "the chance a fraud beats t" over legit scores t, which is the chance a fraud beats a random legit (Card 3). With ties, each tied pair counts \tfrac12.

(2) The difference of two independent normals is normal.

s^+ - s^-

the gap between a random fraud's score and a random legit's

\sim

is distributed as

N(d,\; 1 + 1)

a normal with mean d and variance 2 (the two variances add)

d

the separation: how many SDs fraud scores sit above legit (2 here)

P(s^+ - s^- > 0)

the chance the fraud scores higher

\Phi

the standard normal CDF: the share of N(0, 1) below a value

Across 500 simulated datasets with 1,000 frauds and 99,000 legit, the mean AUC was 0.9212. The theory curve matches for d = 1 (0.760) and d = 3 (0.983) too (Card 6).

(3) Nothing happens. For any strictly increasing g, s^+ > s^- exactly when g(s^+) > g(s^-), so every pair keeps its verdict. In one simulated dataset the AUC was 0.9145 on the raw scores, 0.9145 after \exp(\cdot) and 0.9145 after replacing scores by their ranks (Card 4).

(4) Catching 80% of fraud means \text{TPR} = 1 - \Phi(t - d) = 0.8, so t = d - z_{0.8}. Then apply Bayes on rates (Card 1).

(5) The difference is now N(d, 1 + 4).

\sigma_1^2

the variance of fraud scores (4 now, so SD 2; legit stays at 1)

Simulation gives 0.814. The ROC is no longer symmetric. \text{TPR} = 1 - \Phi((t - 2)/2) and \text{FPR} = 1 - \Phi(t) are equal at t = -2, so the curve dips below the diagonal for FPR above \Phi(2) = 0.977. That is an improper ROC: there's a region where the score ranks worse than a coin. The cause is the wide fraud distribution, whose left tail puts some frauds below almost every legit.

What to say out loud: "AUC 0.92 and precision 6% are both true of the same model. AUC is conditional on the truth: it compares a fraud with a legit. Precision is conditional on the prediction: it asks who is in the flagged pile, and at 1% fraud that pile is mostly legit. Recall and FPR travel; precision stays home."

At d = 2 the binormal AUC (0.921) sits close to the oracle's 0.923 on our 100,000 transactions. A separation of about two standard deviations is roughly what the best possible fraud model in the running example achieves.

Follow-ups that separate senior from research-level answers:

• "What's the error bar on 0.921?" Hanley–McNeil gives 0.0059 at n_1 = 1{,}000 and n_0 = 99{,}000 (Card 13). The actual SD over the 500 simulated datasets was 0.0042. Hanley–McNeil assumes exponential-shaped scores and is conservative for binormal ones, so use DeLong or a bootstrap for the real number. Either way, the frauds set the width: your n is your frauds.
• "Why is AUC a Mann–Whitney statistic, and what does that buy you?" \text{AUC} = U/(n_1 n_0), where U counts the fraud–legit pairs a fraud wins. So under H_0 (scores unrelated to labels) it has a known null distribution: mean \tfrac12, with the variance below. That's a free test of "better than random".

\text{Var}(\widehat{\text{AUC}} \mid H_0)

the variance of the AUC when scores carry no signal

H_0

the null hypothesis: scores are unrelated to fraud

n_0

the number of legit transactions

n_1

the number of frauds

• "Is ranking by s the best you can do?" By Neyman–Pearson, the ranking that dominates every ROC is the likelihood ratio f_1(s)/f_0(s). With equal variances it is \exp(ds - d^2/2), which increases with s, so the raw score already gives the optimal ROC. With unequal variances the log-ratio is quadratic in s: very low scores are fraud-like too. Ranking by the likelihood ratio removes the improper dip and dominates the raw-score ROC.
• "AUC is 0.92, so precision must be decent?" No. The "AUC implies precision" step is a fallacy, because AUC never sees \pi. On our 100,000 transactions, A has the higher AUC (0.868 vs 0.848) and the far lower average precision (0.133 vs 0.399).
• "Where would a fraud team actually read the curve?" At low FPR. Under this model, TPR at FPR 1% is 1 - \Phi(2.326 - 2) = 0.372. The raw partial area up to FPR 1% is only 0.0026 out of a possible 0.01. Report TPR at the FPR you can afford, or the partial AUC.
• "What if the scores are rounded?" Ties appear, and the AUC must count them as halves. Rounding one simulated dataset's scores to integers gave 0.896 with half-credit for ties and 0.860 counting only strict wins, against 0.915 on the continuous scores. sklearn counts halves; a hand-rolled "count wins" loop doesn't.

How to identify it in real data:

• Plot the class-conditional score histograms: two piles, and the size of the legit pile.
• Probit-transform the empirical ROC. Under the binormal model, \Phi^{-1}(\text{TPR}) is a straight line in \Phi^{-1}(\text{FPR}), with slope 1/\sigma_1 and intercept d/\sigma_1. A slope far from 1 warns of unequal variances and an improper tail.
• Compute precision at the production \pi before quoting it. If the test set was balanced or enriched, its precision is not the one you'll see.

\Phi^{-1}

the inverse normal CDF (probit): turns a rate into a z-value

\sigma_1

the SD of fraud scores (legit SD = 1)

How to mitigate the precision collapse:

1. Operate at a fixed FPR, or a fixed capacity. Report TPR at FPR 0.1% or 1%, or precision@k for the review queue.
2. Put the PR curve and AP next to AUC. AP's random baseline is \pi, so it shows the 1% problem.
3. Convert precision with Card 1 to the production base rate before any number leaves the room.
4. Raise d, not the threshold. At a fixed \pi, only better separation lifts the whole PR curve: new signals like B's consortium flag, or better features.

Worked problem 2 — "Ship on the first p < 0.05": peeking

A staple of product data science and experimentation loops. Candidates who know the word "peeking" are asked to put a number on it.

Setup: an A/A test (no true effect) of the fraud model's approval rate runs for 14 days, with 10,000 customers per arm per day. Each morning the analyst runs a two-sided z-test on all data so far and stops at the first |z| > 1.96. (1) With 2 equally spaced looks, compute the false-positive rate exactly. (2) Why does it rise with more looks, and what is the limit? (3) What is it for 14 looks? (4) How do you keep early stopping without the inflation?

Intuition first: under H_0 the cumulative z-statistic wanders like a random walk. Each look is another lottery ticket, and a wandering path will eventually cross any fixed line.

Solution.

(1) Let Z_1 be the z-statistic on the first half of the data and Z_1' the one the second half would give alone. They are independent N(0, 1), and the full-data statistic averages them on the z scale. So the two looks are correlated by \sqrt{1/2} (Card 16).

Z_1

the z-statistic at the first look (first half of the data)

Z_1'

the z-statistic the second half alone would give; independent of Z_1

Z_2

the z-statistic at the second look (all the data)

\text{Corr}

correlation: how tightly the two looks move together

n_1

the sample size at the first look

n_2

the sample size at the second look (n_2 = 2n_1)

The false-positive rate is one minus the chance that both stay inside ±1.96, under a bivariate normal with that correlation.

\text{FPR}_2

the false-positive rate of a test checked at two looks

P

the probability of what's in the brackets

That's 0.05 from the final look alone, plus 0.033 from paths that crossed at the halfway look but would have ended inside. A million simulated A/A pairs gave 0.0832.

(2) Every extra look adds paths that cross at that look and nowhere else. Nearby looks are highly correlated, so each adds less than 5%, but the additions never stop. In the limit, the law of the iterated logarithm says a random walk's \lvert S_n\rvert/\sqrt{n} eventually exceeds any fixed constant. So with infinitely many looks the false-positive rate \to 1, slowly.

(3) Simulation, 100,000 A/A tests per row (50,000 for 100 looks). The 2-look value is exact; the others have no closed form.

The binary-outcome version (decline rate 5% in both arms, 10,000 per arm per day, 20,000 tests) gave 0.220 for 14 looks, so the normal shortcut is accurate. Day 1 contributes exactly the nominal 5%: 5,022 of the 22,088 false wins in 100,000 tests. Days 2–14 add the other 17 points.

(4) Spend \alpha on purpose. A group-sequential design fixes the looks and their boundaries in advance.

• Pocock: the same boundary at every look, 2.413 for 5 looks (a nominal p of 0.016 each). Simulated FPR 0.050.
• O'Brien–Fleming: very strict early and close to 1.96 at the end. For 5 looks the boundaries are 4.56, 3.23, 2.63, 2.28, 2.04. Simulated FPR 0.049.

c_k^{\text{OBF}}

the O'Brien–Fleming boundary at look k: reject if |z| exceeds it

c

the final-look boundary (2.04 for 5 looks)

K

the total number of looks

k

the current look, 1 to K

• Alpha spending (Lan–DeMets) lets the looks fall on any days, spending \alpha along a pre-chosen curve.
• Always-valid p-values (mSPRT; Johari et al., 2017) allow continuous monitoring at a known cost in power.

What to say out loud: "A p-value is only a p-value at the sample size you planned. Fourteen daily looks with a 1.96 line turn a 5% test into a 22% test. If we want to stop early, we choose the boundaries before launch."

Follow-ups that separate senior from research-level answers:

• "Can I look as long as I don't act?" Yes. Inflation comes from the stopping rule, not the glance. A dashboard that only displays always-valid intervals is safe.
• "What about stopping for futility?" Stopping because the effect looks like nothing can't create a false positive; it can only lower the type I error. Its cost is power: some true effects get abandoned.
• "Does a Bayesian decision rule fix it?" "Stop when P(B > A) > 95%" is still a stopping rule on a wandering statistic, and its frequentist false-positive rate also inflates with looks. A Bayesian posterior is coherent under optional stopping only as a statement of belief under its prior. It doesn't control the error rate of the shipping decision.
• "Is the effect estimate biased too?" Yes. Tests that stop early stop because the estimate was large, so stopped estimates are biased upward. It's the winner's curse at the stopping time. Report a bias-adjusted estimate, or the estimate at the planned horizon.
• "Unequal looks?" The correlation of looks at sample sizes n_k < n_l is \sqrt{n_k/n_l}. Spending functions handle unequal spacing; a fixed 1.96 line never does.
• "Twenty metrics, checked daily?" The two problems compound. Multiplicity across metrics (Card 17) stacks on multiplicity across time.

How to identify it in real data:

• Collect the p-values at which past tests were stopped. A spike just under 0.05, and stopping days clustered right after a crossing, are the signature.
• Run an A/A test with the team's real stopping habit and count how often it "wins".
• Check how often shipped wins shrink on re-test.

How to mitigate it:

1. Pre-register the horizon (sample size and duration) in the experiment platform, with the fraud test's label-maturity window included.
2. Build sequential methods into the platform: group-sequential boundaries, alpha spending or always-valid p-values.
3. Show always-valid confidence intervals on dashboards, so a morning glance can't trigger a fixed-horizon decision.
4. Audit: A/A tests, the p-value distribution of past launches, and re-tests of big wins.

Remember: Condition on the truth, then count. AUC conditions on the label, so it ignores \pi; precision conditions on the prediction, so it pays for \pi. Under H_0 the z-path is a random walk, so every unplanned look spends extra \alpha.

Level 4 drills

Twenty problems. Drills 1–4 cover ranking and AUC, 5–7 base rates and thresholds, 8–9 probabilities, 10–13 uncertainty, 14–17 experiments, and 18–20 labels, drift and LLMs. Cover the answer, run the recipe on paper, then check. Every answer was checked by exact computation, by simulation, or both; each answer says which.

Ranking and AUC — Drills 1–4

Drill 1 · The stub service and the coin flip

*During an integration test, the fraud scoring service is replaced by a stub that returns 0.5 for every transaction. A second stub returns a uniform random number. A third flags a random 30% of transactions as 1 and the rest as 0. The interviewer: "Each stub scores 1,000 frauds and 99,000 legit transactions. What AUC does roc_auc_score report for each, and why?"*

Concept tested: ties count \tfrac12; the AUC of random ranking.

• Answer: 0.5 for all three. The constant score gives exactly 0.5. Verified by simulation over 200 test sets: uniform scores averaged 0.4997 (SD 0.0090) and the random 30% flags 0.4989 (SD 0.0068).
• Intuition: a constant score ties every pair, and a tie is half a win. A random score wins half the pairs by symmetry.
• Why: constant: every one of the n_1 n_0 pairs is tied, so \text{AUC} = \tfrac12. Random: P(s^+ > s^-) = \tfrac12 when both come from the same distribution. Random flags: one ROC corner at \text{TPR} = \text{FPR} = 0.3, so \text{AUC} = (1 + 0.3 - 0.3)/2 = \tfrac12 (Card 5). The spread around \tfrac12 for the uniform stub is the Mann–Whitney null variance below; its simulated SD of 0.009 agrees with it. The 0/1 stub spreads less, because its many ties carry no information.

n_1

the number of frauds

\sum_{i \in \text{fraud}}

add up over every fraud i

\sum_{j \in \text{legit}}

add up over every legit transaction j

\mathbf 1

indicator: 1 if the bracket is true, else 0

s_i

the score of fraud i

s_j

the score of legit transaction j

\text{SD}(\widehat{\text{AUC}} \mid H_0)

the standard deviation of the AUC when scores carry no signal

Drill 2 · The rules engine's AUC

*The legacy rules engine outputs only "block" or "pass". On last quarter's labeled data it blocks 60% of frauds and 10% of legit transactions. A dashboard reports its AUC next to the new model's. "Model a hard classifier as a score that is 1 or 0, with TPR 0.6 and FPR 0.1. What AUC will roc_auc_score report? Is the comparison with a scored model fair?"*

Concept tested: the ROC of a hard classifier has one corner (Card 5).

• Answer: (1 + 0.6 - 0.1)/2 = 0.75, which is the balanced accuracy (exact). The comparison isn't fair: a scored model is judged over all thresholds, the rules engine at one point.
• Intuition: two thresholds exist, "flag everything" and "flag the 1s", so the ROC is two straight lines through one corner, and the area under them is a trapezoid sum.
• Why: the area under (0, 0) \to (0.1, 0.6) \to (1, 1) is a triangle plus a trapezoid.

• Next they'll ask: "What if someone feeds predict() into roc_auc_score?" → The same trap. Raw B's hard predictions at 0.5 give 0.768 on our 100,000 transactions (TPR 0.696, FPR 0.159) against its real AUC of 0.848. On the twelve review transactions they give 0.8125, above the real 0.75. A's hard predictions give 0.510, because A almost never scores above 0.5.

Drill 3 · AUC 0.30

A new model scores AUC 0.30 on the holdout, and a junior teammate wants to scrap it. "Suppose legit scores are N(0, 1) and fraud scores are N(d, 1) with d chosen so that the AUC is 0.30. Is the model useless? What is the AUC of 1 - \text{score}? What would you check first?"

Concept tested: \text{AUC}(-s) = 1 - \text{AUC}(s).

• Answer: not useless: 1 - \text{score} has AUC 0.70 (exact). Verified by simulation with d = -0.742: AUC 0.305, flipped 0.695, summing to exactly 1. Check the label encoding first: pos_label, the column of predict_proba, or a flipped target.
• Intuition: a model that is reliably wrong is reliably informative; turn it upside down.
• Why: reversing the order turns every won pair into a lost one, and ties stay ties.

s

the model's score; -s is the same score flipped upside down

P

the probability of what's in the brackets

Drill 4 · Squared scores and a moving F1

Take the twelve review transactions and square model A's scores (0.90 becomes 0.81, and so on). The interviewer: "What happens to the AUC? To F1 at threshold 0.5? To the best F1 over all thresholds? Which of these could a teammate truthfully call an improvement?"

Concept tested: monotone invariance (Card 4).

• Answer: AUC 0.781 before and after. F1 at 0.5 falls from 0.5 to 0.333. The best F1 over thresholds stays 0.667. None is an improvement; the fixed-threshold F1 changed only because the threshold moved relative to the scores (exact).
• Intuition: squaring keeps every order, so anything built from ranks is unchanged. A fixed cut at 0.5 now sits at what used to be 0.71.
• Why: after squaring only T1 (0.81) and T2 (0.64) clear 0.5, giving TP 1, FP 1, FN 3. The best threshold just moves to the square of the old one.

F_1

the F1 score at the 0.5 cut, after squaring the scores

Base rates and thresholds — Drills 5–7

Drill 5 · The vendor's balanced test set

A vendor demos a fraud model with 94% precision at 80% recall, measured on a balanced test set of 5,000 frauds and 5,000 legit. Your fraud rate is 1%. "Their model has TPR 0.8 and FPR 0.05. What precision should you expect in production? What do you ask the vendor for?"

Concept tested: Bayes on rates; precision stays home (Card 1).

• Answer: 0.941 on their set, 0.139 on yours (exact). Ask for TPR and FPR, or the full ROC, on data drawn like yours, and recompute precision at your \pi.
• Intuition: TPR and FPR are measured inside each class, so they travel. Precision counts both piles, and your legit pile is 99 times the fraud pile.
• Why: total probability over the two classes.

• Next they'll ask: "Same thing for our model A?" → At its cost-optimal threshold A has TPR 0.345 and FPR 0.022 on the test set, so its precision would read 0.940 on a 50/50 set and 0.138 at 1%.

Drill 6 · The do-nothing model

A model is deployed with a threshold so high it never fires. Over a day with 1% fraud, the monitoring job still reports metrics. "Compute accuracy, recall, precision, F1, MCC and balanced accuracy. Which are undefined, and what does sklearn return for them?"

Concept tested: the accuracy paradox and undefined ratios.

• Answer: accuracy 0.99, recall 0, specificity 1, balanced accuracy 0.5, F1 0. Precision is 0/0 (sklearn returns 0 with an UndefinedMetricWarning), and so is MCC (sklearn returns 0 silently). All exact; on our 100,000 transactions, flagging nothing scores 0.990 accuracy.
• Intuition: accuracy is mostly the big box, and doing nothing fills the big box perfectly.
• Why: TP = FP = 0. Precision divides by predicted positives (zero of them). MCC's denominator contains (TP + FP), which is zero. F1 = 2TP/(2TP + FP + FN) = 0/FN = 0.

n

the number of transactions

Drill 7 · Where F1 peaks

A calibrated fraud model's best achievable F1 on its population is 0.52. The analyst plans to search 1,000 thresholds on a small test set. "Without searching: where should the F1-maximizing threshold be? Derive it."

Concept tested: the F1-optimal threshold is half the best F1 (Card 11).

• Answer: t^* = F_1^*/2 \approx 0.26. Verified on the true probabilities of 4,000,000 simulated transactions: the expected F1 peaks at 0.516 at t = 0.258, and 0.516/2 = 0.258.
• Intuition: raise t while the next transaction you'd add is worth more than half your current F1. A flagged row with fraud probability p adds 2p to the numerator and 1 to the denominator, so it helps exactly when p > F1/2.
• Why: write the population F1 with G the distribution of calibrated probabilities. Then N(t) = \int_t p\,dG and D(t) = \int_t dG + \pi, so N' = -t\cdot g and D' = -g. Setting the derivative to zero gives t = N/D = \text{F1}/2.

F_1(t)

the expected F1 when flagging every probability above t

N

expected true positives per transaction when flagging above t: F1's numerator over 2

D

flagged share plus fraud share: F1's denominator

\frac{dF_1}{dt}

how fast F1 changes as you nudge the threshold t: a slope, not a fraction to divide

g

the density of the calibrated probabilities (the slope of G)

t^*

the best threshold (star = optimum)

F_1^*

the best achievable F1

• Next they'll ask: "Does it hold on a test set?" → Only roughly. The empirical F1 curve is flat near its peak and its argmax is noisy: on the 974-fraud test set B-cal's empirical argmax is 0.257, while the oracle's is 0.337.

Probabilities — Drills 8–9

Drill 8 · Brier by bins

A forecaster issues 10 fraud probabilities: five at 0.2 (1 of those transactions is fraud) and five at 0.6 (4 are fraud). "Compute the Brier score directly. Then decompose it into reliability, resolution and uncertainty, and check the identity."

Concept tested: the Murphy decomposition, Brier = Lie − Learn + Luck (Card 9).

• Answer: REL 0.02, RES 0.09, UNC 0.25, so \text{Brier} = 0.02 - 0.09 + 0.25 = 0.18. The direct computation also gives 0.18 (exact).
• Intuition: the 0.2 bin is honest (20% fraud). The 0.6 bin is too timid (80% fraud): that's the lie. The bins differ a lot from the 50% average: that's what was learned. The luck is the 50/50 base rate.
• Why: with forecasts constant within bins, the identity is exact.

Direct check: (4 \times 0.2^2 + 0.8^2 + 4 \times 0.4^2 + 0.6^2)/10 = (0.16 + 0.64 + 0.64 + 0.36)/10 = 0.18.

Drill 9 · Beating the base rate

A calibration report for model A says "Brier 0.0090, close to the perfect 0". Fraud is about 1% of traffic. "What Brier score and log loss does the constant forecast 'always say the base rate' get? What is A's Brier skill score, and how should the report be rewritten?"

Concept tested: \text{UNC} = \pi(1 - \pi), log loss of the base rate = entropy, skill scores.

• Answer: at exactly 1% the constant forecast scores Brier 0.0099 and log loss 0.056 nats (0.081 bits), both exact. At the test set's 0.974%, the values are 0.00965 and 0.0548. A's Brier skill score is 1 - 0.0090/0.00965 = 0.067. B-cal's is 0.296 and the oracle's 0.350. Rewrite: "A removes 7% of the base rate's squared error; B-cal removes 30%."
• Intuition: at a 1% base rate, every forecast's Brier is tiny, including doing nothing. The base rate is the bar to clear.
• Why: the constant forecast has no lie and learns nothing (REL = RES = 0), so its Brier is pure luck. Its log loss is the binary entropy.

Uncertainty — Drills 10–13

Drill 10 · \pm 2 points of recall

Compliance wants recall reported "to within two points". The model's recall is about 0.8 and fraud is 1% of traffic. "How many frauds do you need for a 95% CI of ±0.02, and how many transactions is that?"

Concept tested: the variance of a proportion; your n is your frauds (Card 12).

• Answer: 1,537 frauds \approx 153,700 transactions (exact). With the rule-of-thumb z = 2 it's 1,600 frauds.
• Intuition: recall is a proportion over frauds only, so the legit transactions add nothing to its precision.
• Why: set the half-width z\cdot\sqrt{p(1 - p)/n} equal to h and solve for n.

z

the normal multiplier for the confidence level: 1.96 for 95%

p

the recall you expect: 0.8

h

the target half-width of the CI: 0.02

Drill 11 · The error bar on an AUC

Two teams each report AUC 0.90. One test set has 100 frauds and 9,900 legit; the other has 1,000 frauds and 99,000 legit. "Give the Hanley–McNeil standard error for each. Which class size drives it? How far would you trust the formula?"

Concept tested: AUC as a U-statistic; its variance (Card 13).

• Answer: 0.0207 with 100 frauds and 0.0065 with 1,000 (exact). Raising n_0 from 9,900 to 99,000 at n_1 = 100 moves it only from 0.02067 to 0.02065: the frauds set the width. Verified by simulation (500 test sets each): for exponential scores, the model behind the formula, the SDs were 0.0206 and 0.0064. For binormal scores the SDs were 0.0154 and 0.0048, about 25% smaller. Treat Hanley–McNeil as a quick, often conservative approximation; use DeLong or a bootstrap for the real number.
• Intuition: each fraud is compared against thousands of legit transactions, so its "share of pairs won" is measured precisely. What stays noisy is which frauds you happened to draw.
• Why: in the formula, the n_0 term is divided by n_1 n_0, so it shrinks like 1/n_1; the n_1 term shrinks like 1/n_0 and vanishes.

\text{Var}(\widehat{\text{AUC}})

the variance of the estimated AUC (Hanley–McNeil)

A

the AUC itself, as one number (0.9 here)

Q_1

the chance two random frauds both outscore one random legit transaction

Q_2

the chance one random fraud outscores two random legit transactions

• Next they'll ask: "On our test set?" → For A, Hanley–McNeil gives 0.0074; DeLong gives 0.0055.

Drill 12 · Twenty models, one test set

A team tunes 20 configurations, all with the same true AUC, and reports the best test-set AUC. "Simplify: each test AUC is the truth plus independent N(0, 0.01^2) noise. How far above the truth is the reported winner, on average?"

Concept tested: the winner's curse; the expected maximum of normals.

• Answer: +1.87\ \text{SE} \approx +0.019 (exact: E[\max \text{ of 20 standard normals}] = 1.8675). Verified by simulation over 200,000 repetitions: mean gap 0.01868.
• Intuition: the winner is the configuration whose test-set luck was best, and that luck doesn't come back on fresh data.
• Why: the reported number is the maximum of 20 noisy estimates of the same quantity.

\mathbb E

expected value: the average over many repeats

\max_{1 \le j \le 20}

the largest over the 20 configurations

\widehat{\text{AUC}}_j

configuration j's test-set AUC

Z_j

configuration j's test-set error, in standard errors (SE 0.01)

• Next they'll ask: "Is it that bad in practice?" → Real configurations are correlated, which shrinks the gap. In our snooping simulation (20 configurations, a test set with 130 frauds, 40 repetitions), the picked winner's test AUC beat its fresh-sample AUC by 0.011 on average. Of that, 0.006 \pm 0.001 was selection. The fix is a test set touched once, or nested CV.

Drill 13 · Only the disagreements

At the review capacity, models A and B both flag most of the same frauds. Among the frauds where they disagree, A alone catches 40 and B alone catches 20. "Run McNemar's test with the continuity correction, then the exact version. Why are the frauds both models caught irrelevant?"

Concept tested: McNemar's test (Card 14).

• Answer: \chi^2 = 6.02, p = 0.014; exact binomial p = 0.0135 (exact). Without the continuity correction, \chi^2 = 6.67 and p = 0.0098. A simulation of the null (200,000 draws of \text{Bin}(60, \tfrac12)) confirmed the exact p: 0.0135.
• Intuition: concordant frauds say the models are alike; only the 60 frauds they split say which is better. Under H_0 each split is a fair coin.
• Why: conditional on b + c = 60 disagreements, b \sim \text{Bin}(60, \tfrac12) under equal error rates.

\chi^2

McNemar's statistic: how lopsided the disagreements are

b

frauds caught by A only (40)

c

frauds caught by B only (20)

p_{\text{exact}}

the exact two-sided p-value from the binomial

\text{Bin}(60, \tfrac12)

the count of heads in 60 fair coin flips: A-only frauds under equal catch rates

Experiments — Drills 14–17

Drill 14 · The approval-rate guardrail

The fraud model's A/B test has approval rate as a guardrail. Approval is 95%, and the business wants to detect a drop of 0.2 percentage points with 80% power at \alpha = 0.05, two-sided. "How many customers per arm? Derive the 16."

Concept tested: sample size, n \approx 16\sigma^2/\delta^2 per arm (Card 15).

• Answer: 190,000 per arm by the rule of thumb. The exact constant 2(1.96 + 0.84)^2 = 15.70 gives 186,411. Verified by simulation (2,000 tests at 186,411 per arm): power 0.793, against 0.800 in theory.
• Intuition: you need the true difference to sit about 2.8 standard errors from zero, and the SE of a difference of two means is \sigma\sqrt{2/n}.
• Why: \sigma^2 = p(1 - p) = 0.0475, \delta = 0.002.

\delta

the smallest change worth detecting: 0.002 (0.2 points)

z_{0.975}

the normal cutoff for a two-sided 5% test: 1.96

z_{0.8}

the normal value for 80% power: 0.84

\sigma

the per-customer SD of approval; \sigma^2 = p(1-p) = 0.0475

n

the customers needed in each arm

• Next they'll ask: "What can 50,000 per arm detect?" → An MDE of 0.0039, about 0.39 points. Halving \delta quadruples n.

Drill 15 · Two peeks

The analyst promises to look only twice: at half the planned sample and at the end, rejecting if either \lvert z\rvert > 1.96. "Under H_0, what is the overall false-positive rate?"

Concept tested: correlated looks (Card 16, Worked problem 2).

• Answer: 0.0831 (exact, bivariate normal). Verified by simulation with a million pairs: 0.0832.
• Intuition: the second look reuses half the first look's data, so the two tests are correlated but not identical, and the second adds about 3 points.
• Why: \text{Corr}(Z_1, Z_2) = \sqrt{n_1/n_2} = \sqrt{1/2}.

P(\text{reject})

the chance at least one of the two looks crosses ±1.96 with no real effect

N

a normal distribution, here for the pair of z-statistics

\mathbf 0

the mean: both z-statistics average 0 when there is no effect

Drill 16 · Twenty metrics

An experiment readout tracks 20 metrics. Sorted, the p-values are 0.001, 0.004, 0.009, 0.012, 0.021, 0.04, 0.06, 0.2, 0.3, 0.5 and ten at 0.6. "How many do you reject at 0.05 under Bonferroni? Under Benjamini–Hochberg? What does each procedure control?"

Concept tested: FWER vs FDR (Card 17).

• Answer: Bonferroni rejects 1; BH rejects 2 (exact). Bonferroni controls the family-wise error rate, the chance of any false alarm. BH controls the false discovery rate, the expected share of false alarms among rejections.
• Intuition: Bonferroni gives every metric the same tiny bar. BH raises the bar's height with rank, so a cluster of small p-values earns more rejections.
• Why: Bonferroni's cut is 0.05/20 = 0.0025, which only 0.001 clears. BH compares p_{(i)} with 0.0025\cdot i and rejects up to the largest i that passes: 0.001 \le 0.0025 and 0.004 \le 0.005, but 0.009 > 0.0075, 0.012 > 0.010, 0.021 > 0.0125, and every later p-value misses its cut too.

p_{(1)}

the smallest p-value (brackets = sorted order)

p_{(k)}

the k-th smallest p-value: the last one rejected

k

how many metrics BH rejects

p_{(i)}

the i-th smallest p-value

i

a p-value's rank when sorted smallest first

q

the false discovery rate you allow: 0.05

m

the number of metrics tested: 20

m_0

the number of metrics with no real effect (16 here)

• Next they'll ask: "Show me the trade-off." → Simulation, 10,000 readouts with 4 real effects and 16 nulls, independent: no correction has FWER 0.557 and power 0.851; Bonferroni FWER 0.037, power 0.491; BH FWER 0.135, power 0.627, and FDR 0.040, matching (16/20) \times 0.05 = 0.04.

Drill 17 · CUPED by hand

Spend tests take four weeks. Each customer's pre-period spend X correlates 0.7 with test-period spend Y, and \text{SD}(Y) = 2\cdot\text{SD}(X). "What \theta does CUPED use? By what factor do the variance and the required sample shrink? Is the effect estimate still unbiased?"

Concept tested: CUPED (Card 18).

• Answer: \theta = \rho\cdot\text{SD}(Y)/\text{SD}(X) = 1.4. The variance factor is 1 - \rho^2 = 0.51, so about 49% fewer customers (exact). Verified by simulation on a million Gaussian pairs: \theta 1.400, factor 0.510. Still unbiased, because X is measured before assignment.
• Intuition: subtract the part of each customer's outcome you could have predicted before the test started. What remains is the noise you couldn't predict, plus the treatment effect.
• Why: minimize \text{Var}(Y - \theta X) over \theta. Randomization makes E[\bar X_T - \bar X_C] = 0, so the adjustment removes no effect.

\theta^*

the best CUPED slope (star = optimum)

\text{Cov}

covariance: how much two quantities move together

\text{Var}

variance: the spread of a quantity around its mean

Y

the outcome during the test: this month's spend

X

the pre-experiment covariate: last month's spend

\rho

the correlation between X and Y (0.7)

\sigma_Y

the SD of Y

\sigma_X

the SD of X

• Next they'll ask: "With real, skewed spend?" → In the 100,000-per-arm simulation with lognormal spend, the realized \rho was 0.703, the per-unit variance ratio 0.510 (theory 0.505), and power rose from 0.39 to 0.65.

Labels, drift and LLMs — Drills 18–20

Drill 18 · Chargebacks that haven't arrived

Fraud labels come from chargebacks, which arrive after a delay. A teammate celebrates that the model's precision on transactions from 14 days ago is "half the false alarms of last quarter". "Let the delay be Exponential with mean 20 days, independent of the score. What share of 14-day-old fraud is labeled? How long until 95% is? What happens to measured precision and measured recall?"

Concept tested: delayed labels.

• Answer: 50.3% labeled at 14 days, and 59.9 days to reach 95% (exact). Verified by simulation: 50.4% and 59.9 days. Measured precision and the measured fraud rate are understated by about half. In the sim, A's precision on 14-day-old data read 0.069 against 0.134 true. Measured recall is not biased when the delay is independent of the score, only noisier: 0.352 against 0.345 true.
• Intuition: missing chargebacks turn flagged frauds into apparent false alarms, so precision drops. Recall divides labeled caught frauds by labeled frauds, and both shrink by the same factor.
• Why: with F(a) = 1 - e^{-a/20} the share labeled by age a:

F

the labeled-share curve; F(14) reads it at 14 days

F(a)

the share of a cohort's fraud already labeled at age a

a

the cohort's age in days

e

Euler's number, about 2.718

\text{recall}_{\text{meas}}

recall as measured on the partly labeled data

• Next they'll ask: "And the precision 'gain'?" → It's an artifact. Measured precision \approx F(a) \times true precision. Evaluate on cohorts at least 60 days old, and never train on unmatured labels as if they were negatives.

Drill 19 · PSI by hand

Monitoring bins this week's fraud scores by the reference quartiles, which held 25% each at launch. This week the four bins hold 15%, 20%, 30% and 35%. "Compute the PSI. Calm, watch or act? How does it relate to KL divergence?"

Concept tested: PSI as a symmetrized KL divergence (Card 19).

• Answer: PSI = 0.105, just over 0.1: watch (exact). The four terms are 0.051, 0.011, 0.009 and 0.034.
• Intuition: each bin pays (how much it moved) \times (the log of how much it moved), so the two bins that moved 10 points dominate.
• Why: expand the sum. It splits into \text{KL}(a\,\|\,e) + \text{KL}(e\,\|\,a) = 0.0512 + 0.0538, and for small shifts it is close to the \chi^2-type distance \sum (a - e)^2/e = 0.100.

\sum_i

add up over the 4 score bins i

a_i

this week's (actual) share in bin i

e_i

the reference (expected) share in bin i: 25% each

• Next they'll ask: "What does a real alert look like?" → In the monitoring simulation, the drifting merchant-risk input reached PSI 0.23 by week 8 (watch). An upstream bug zeroing 30% of the device-novelty input gave PSI 0.44 in week 10 (act). The score's PSI was 0.10 that week. Score PSI dilutes a broken input, because the score blends it with healthy features: watch inputs and score separately.

Drill 20 · pass@k

An LLM that writes fraud-rule code generates n = 10 candidate solutions per problem; for one problem, c = 3 pass the unit tests. The team reports pass@5 as 1 - (1 - 0.3)^5. "What is the unbiased pass@5? pass@1? Why not the team's formula?"

Concept tested: the unbiased pass@k estimator (Card 20).

• Answer: pass@5 = 1 - C(7, 5)/C(10, 5) = 1 - 21/252 = 0.917, and pass@1 = c/n = 0.3 (exact). The plug-in gives 0.832, which is the true pass@5 if the per-sample rate were exactly 0.3. As an estimator it is biased low in expectation. Verified by simulation (400,000 draws at p = 0.3): the plug-in averages 0.760; the unbiased estimator averages 0.832, the truth.
• Intuition: the unbiased version asks the question you mean: if I drew 5 of my 10 samples, how often would at least one pass?
• Why: averaging the "at least one correct" indicator over all C(n, k) subsets of size k is unbiased for pass@k. The plug-in applies the concave function 1 - (1 - p)^k to a noisy \hat p, so by Jensen's inequality its average falls below the truth.

\widehat{\text{pass@}k}

the unbiased estimate of pass@k: the chance at least one of k attempts passes

n

the candidate solutions generated for the problem (10)

c

the candidates that passed the unit tests (3)

k

the attempts allowed (5)

\binom{n}{k}

the number of ways to choose k of the n candidates

\hat p

the plug-in pass rate c/n (hat = estimated)

p

the true per-sample pass rate

• Next they'll ask: "Across a benchmark?" → On a 50-problem simulation (pass rates spread out, 4,000 repetitions), the truth was 0.496: the unbiased estimator averaged 0.496 and the plug-in 0.455. Report the error bar too: your n is your problems.

Metric cheat sheet

Error-bar traps (point estimate fine, uncertainty wrong): counting transactions instead of frauds (Drills 10–11); unpaired comparison of two models on the same rows (Card 13); Hanley–McNeil on binormal-like scores (Drill 11); "SD over folds / \sqrt{K}" for an unstable learner (Q37); a bootstrap that resamples transactions when cards repeat (resample cards).

Rapid fire — the day before

One line each. Cover the right column and answer out loud; anything you hesitate on, go back to its full question above.