Chapter 5 — Derivations

Evaluation interviews look like conversation: "why does precision collapse in production?", "is 0.868 vs 0.848 real?", "the PM wants to stop the test on day 5". But each of those questions has a short derivation underneath, and the interviewer is checking whether you can produce it. You will be asked to write a metric as a probability, condition on the truth, and count. Then you attach an error bar and say what happens in the limits. They ask because these derivations separate people who know the formulas from people who know where the formulas come from. Precision's dependence on the base rate is one line of Bayes. AUC as a pair probability is one swap of a sum. Brier's three parts, the F1 threshold of $F_1^*/2$ , the $16\sigma^2/\delta^2$ sample size, the 8.3% false-positive rate of two peeks and the $1 - \rho^2$ of CUPED are each fewer than ten lines. This tab has the five moves that produce all of them, the chapter's 25 formulas, and 20 cards. Each card states a result, derives it with every step shown, and checks the answer on the twelve review transactions or the 100,000-transaction simulation. It then names the slip candidates make and how the question is asked. Practice by covering everything below a card's Statement, deriving it on a blank page against a timer, then checking. The protocol at the end says how.

The toolkit

Every card checks itself on one of the chapter's two running examples. The numbers below come up again and again.

Fact	Twelve review transactions	100,000 test transactions
Frauds / legit	$n_1 = 4$ , $n_0 = 8$ , 32 pairs; each legit stands for 49.5 of the day's 396	974 frauds, base rate $\pi = 0.974\%$
A at threshold 0.5	TP 2, FP 2, FN 2, TN 6	—
B at threshold 0.5	TP 3, FP 1, FN 1, TN 7	—
AUC	A 25/32 = 0.78125, B 24/32 = 0.75	A 0.868, B 0.848, oracle 0.923
Average precision	A 0.6917, B 0.8333	A 0.133, B 0.399, oracle 0.494
Log loss / Brier	A 0.523 / 0.181; B 0.576 / 0.149	Brier: A 0.00900, raw B 0.128, B-cal 0.00679

Every derivation in this tab uses some combination of five moves. Hook: R-A-T-E-S.

R · Rank-pair counting. A threshold-free ranking metric is an average over (fraud, legit) pairs, a two-sample $U$ -statistic. Write it as a double sum of pair indicators, then count, swap the order of summation, or take expectations term by term.

\hat A = \frac{1}{n_1 n_0}\sum_{i=1}^{n_1}\sum_{j=1}^{n_0}\psi_{ij}, \qquad \psi_{ij} = \mathbf 1(s_i^+ > s_j^-) + \tfrac12\,\mathbf 1(s_i^+ = s_j^-)

$\hat A$: the AUC measured on the test rows (hat = estimated from data)
$n_1$: the number of frauds (4 among the twelve review transactions)
$n_0$: the number of legit transactions (8 in the review sample)
$\sum_{i=1}^{n_1}$: add up over every fraud $i$
$\sum_{j=1}^{n_0}$: add up over every legit transaction $j$
$\psi_{ij}$: the pair score: 1 if fraud $i$ outscores legit $j$ , ½ for a tie, else 0
$\mathbf 1$: indicator: 1 if the condition in brackets is true, else 0
$s_i^+$: the model's score for fraud $i$ (plus = the fraud class)
$s_j^-$: the model's score for legit transaction $j$ (minus = the legit class)

A · Add-and-subtract. Insert a term and its negative so a sum of squares splits into pieces whose cross term is zero. This is Ch 1's A move; here it splits the Brier score and finds CUPED's best $\theta$ .

\sum_{i=1}^{n}(f - y_i)^2 = \sum_{i=1}^{n}\big[(f - \bar y) + (\bar y - y_i)\big]^2 = n(f - \bar y)^2 + \sum_{i=1}^{n}(y_i - \bar y)^2 \qquad\Big(\text{since } \textstyle\sum_i(\bar y - y_i) = 0\Big)

$\sum_{i=1}^{n}$: add up over all $n$ transactions
$n$: the number of transactions scored
$f$: one constant fraud probability forecast for every transaction
$y_i$: transaction $i$ 's outcome: 1 if fraud, 0 if legit
$\bar y$: the share of transactions that are fraud (bar = average of the $y_i$ )
$\textstyle\sum_i$: add up over every transaction $i$

T · Total probability, then Bayes on rates. Anything that divides by predicted positives (precision, the share of false discoveries) is a posterior. Write the denominator by total probability over the true classes.

P(Y = 1 \mid F) = \frac{P(F \mid Y = 1)\,P(Y = 1)}{P(F \mid Y = 1)\,P(Y = 1) + P(F \mid Y = 0)\,P(Y = 0)}

$P$: the probability of the event in brackets; $\mid$ reads "given"
$Y$: the true class: $Y = 1$ for fraud, $Y = 0$ for legit
$F$: the event "the model flags this transaction"

E · Expected cost. For a decision rule, write the expected loss of each action given what you know, and take the cheaper one pointwise. For an aggregate metric, write it as a function of the threshold and set the derivative to zero.

\text{action}(x) = \arg\min_{a}\; \mathbb E\big[c(a, Y) \mid x\big] \qquad\Rightarrow\qquad \text{flag} \iff (1 - p)\,c_{FP} < p\,c_{FN}

$x$: everything known about the transaction: its features
$\arg\min_{a}$: the action $a$ (flag or pass) that makes what follows smallest
$c(a, Y)$: the cost of taking action $a$ when the truth is $Y$
$\mathbb E$: expected value: the average over what the truth could turn out to be
$p$: the model's calibrated probability that this transaction is fraud
$c_{FP}$: cost of a false positive: blocking a legit customer (about $10)
$c_{FN}$: cost of a false negative: letting a fraud through (its amount)

S · Sampling variance. Every metric here is a proportion, a mean, or a U-statistic. A proportion of $n$ independent 0/1 outcomes has variance $p(1 - p)/n$ . Sums add variances plus twice the covariances. Large-sample estimates are approximately normal.

\text{Var}(\hat p) = \frac{p(1 - p)}{n}, \qquad \text{Var}(X \pm Y) = \text{Var}(X) + \text{Var}(Y) \pm 2\,\text{Cov}(X, Y), \qquad \frac{\hat\theta - \theta}{\text{SE}(\hat\theta)} \;\dot\sim\; N(0, 1)

$\text{Var}$: variance: how much a number would wobble across repeated test sets
$\hat p$: a measured proportion, such as recall on the test frauds
$p$: the true proportion that $\hat p$ estimates
$n$: the number of independent 0/1 outcomes behind the proportion
$X$: any random quantity, such as one model's measured metric
$Y$: a second random quantity (not the class label here)
$\text{Cov}$: covariance: how much two quantities wobble together
$\hat\theta$: any estimate from data, such as a measured AUC
$\theta$: the true value that $\hat\theta$ estimates
$\text{SE}$: standard error: the standard deviation of the estimate
$\dot\sim$: is approximately distributed as (in large samples)
$N(0, 1)$: the standard normal bell curve: mean 0, variance 1

Which move does the work, card by card: R for Cards 3, 4, 5, 6, 13, 20; A for 2, 8, 9, 18, 19; T for 1, 4, 7, 16, 17, 20; E for 8, 10, 11; S for 6, 12, 13, 14, 15, 16, 18. When an unfamiliar question stalls you, run down the letters. Is it an average over pairs? Is there a mean to add and subtract? Does it divide by predicted positives? Is there a decision whose expected cost I can write? Is it a proportion whose variance I know?

Formula sheet

The confusion matrix

The rates (P-P, R-R)

\text{precision} = \frac{TP}{TP + FP}, \qquad \text{recall} = \text{TPR} = \frac{TP}{TP + FN}, \qquad \text{FPR} = \frac{FP}{FP + TN} = 1 - \text{specificity}

$TP$: true positives: frauds the model flags
$FP$: false positives: legit transactions the model flags
$FN$: false negatives: frauds the model lets through
$TN$: true negatives: legit transactions correctly passed
$\text{TPR}$: true-positive rate (recall): the share of frauds that get flagged
$\text{FPR}$: false-positive rate: the share of legit transactions that get flagged

Precision from Bayes (Card 1)

\text{precision} = \frac{\pi\,\text{TPR}}{\pi\,\text{TPR} + (1 - \pi)\,\text{FPR}}, \qquad \frac{\text{precision}}{1 - \text{precision}} = \frac{\pi}{1 - \pi}\cdot\frac{\text{TPR}}{\text{FPR}}

$\pi$: the base rate: the share of all transactions that are fraud (0.974% in test)

F1 and $F_\beta$ (Card 2)

F_1 = \frac{2PR}{P + R} = \frac{2\,TP}{2\,TP + FP + FN}, \qquad F_\beta = \frac{(1 + \beta^2)PR}{\beta^2 P + R} = \frac{(1 + \beta^2)\,TP}{(1 + \beta^2)\,TP + \beta^2 FN + FP}

$F_1$: the F1 score: the harmonic mean of precision and recall
$P$: precision: the share of flags that are really fraud
$R$: recall: the share of frauds that get flagged
$F_\beta$: the F-beta score: like F1, but recall weighs $\beta$ times as much
$\beta$: how many times more recall matters than precision (2 for F2)

Matthews correlation coefficient (Card 2)

\text{MCC} = \frac{TP\cdot TN - FP\cdot FN}{\sqrt{(TP + FP)(TP + FN)(TN + FP)(TN + FN)}} \in [-1, 1]

$\text{MCC}$: Matthews correlation: the correlation between the 0/1 flag and the 0/1 truth

Ranking

AUC as a pair probability, Gini and KS (Card 3)

\text{AUC} = P(s^+ > s^-) + \tfrac12 P(s^+ = s^-) = \int_0^1 \text{TPR}\;d\,\text{FPR}, \qquad \text{Gini} = 2\,\text{AUC} - 1, \qquad \text{KS} = \max_t\,\big[\text{TPR}(t) - \text{FPR}(t)\big]

$\text{AUC}$: area under the ROC curve of TPR against FPR
$P$: the probability of the event in brackets
$s^+$: the score of a randomly drawn fraud
$s^-$: the score of a randomly drawn legit transaction
$\int_0^1$: add up the area as FPR runs from 0 to 1
$d\,\text{FPR}$: a tiny step along the FPR axis
$\max_t$: the largest value over all thresholds $t$
$t$: the score threshold: flag when the score is above it
$\text{KS}$: Kolmogorov–Smirnov statistic: the widest gap between TPR and FPR

AUC from ranks (Card 3)

\text{AUC} = \frac{R_1 - n_1(n_1 + 1)/2}{n_1 n_0} = \frac{U}{n_1 n_0}, \qquad R_1 = \text{sum of the frauds' ranks (ascending, ties averaged)}

$R_1$: the sum of the frauds' ranks, with all scores ranked from lowest (rank 1) up
$n_1$: the number of frauds
$n_0$: the number of legit transactions
$U$: the Mann–Whitney count: (fraud, legit) pairs the fraud wins, ties as ½

AUC of a hard 0/1 classifier (Card 5)

\text{AUC} = \frac{1 + \text{TPR} - \text{FPR}}{2} = \frac{\text{TPR} + \text{TNR}}{2} = \text{balanced accuracy}, \qquad \text{AUC}(-s) = 1 - \text{AUC}(s)

$\text{TNR}$: true-negative rate: the share of legit transactions passed, $1 - \text{FPR}$
$s$: the model's score; $-s$ flips the ranking upside down

Binormal AUC (Card 6)

s^- \sim N(\mu_0, \sigma_0^2),\;\; s^+ \sim N(\mu_1, \sigma_1^2) \;\Rightarrow\; \text{AUC} = \Phi\!\left(\frac{\mu_1 - \mu_0}{\sqrt{\sigma_0^2 + \sigma_1^2}}\right), \qquad \text{equal unit variances: } \Phi(d/\sqrt2)

$\sim$: is distributed as
$N$: a normal (bell-curve) distribution with the (mean, variance) in brackets
$\mu_0$: the mean score of legit transactions
$\mu_1$: the mean score of frauds
$\sigma_0^2$: the variance of legit scores
$\sigma_1^2$: the variance of fraud scores
$\Phi$: the standard normal CDF: the share of a bell curve below its argument
$d$: the gap between the two class means, in standard deviations

Average precision (Card 7)

\text{AP} = \sum_k (R_k - R_{k-1})\,P_k = \frac{1}{n_1}\sum_{i=1}^{n_1}\frac{i}{r_i} \quad (r_i = \text{rank of the } i\text{-th fraud from the top}), \qquad \mathbb E[\text{precision}@k \mid \text{random}] = \pi

$\text{AP}$: average precision: the step-wise area under the precision–recall curve
$\sum_k$: add up over every cutoff $k$ down the ranked list
$k$: a cutoff: look at the top $k$ rows of the ranked list
$R_k$: recall within the top $k$ rows
$R_{k-1}$: recall within the top $k - 1$ rows, one row earlier
$P_k$: precision within the top $k$ rows
$\sum_{i=1}^{n_1}$: add up over every fraud, the $i$ -th found from the top
$i$: counts frauds from the top: the $i$ -th fraud in the ranked list
$r_i$: the rank of the $i$ -th fraud, counting from the top
$\mathbb E$: expected value: the average over many random rankings

NDCG@k (gain $g_i$ : 1 for fraud, or the fraud's amount; linear gains as in sklearn)

\text{DCG}@k = \sum_{i=1}^{k}\frac{g_{(i)}}{\log_2(i + 1)}, \qquad \text{NDCG}@k = \frac{\text{DCG}@k}{\text{DCG}@k \text{ of the ideal order}}

$\sum_{i=1}^{k}$: add up over the top $k$ positions in the ranked list
$g_{(i)}$: the gain at position $i$ : 1 for a fraud (or its amount), 0 for legit
$\log_2(i + 1)$: the position discount: rows lower in the list count less

Probabilities

Log loss and Brier (Card 8)

\text{LL} = -\frac1n\sum_i\big[y_i\ln p_i + (1 - y_i)\ln(1 - p_i)\big], \qquad \text{BS} = \frac1n\sum_i (p_i - y_i)^2, \qquad \text{constant } \pi:\; \text{LL} = H(\pi),\;\; \text{BS} = \pi(1 - \pi)

$\text{LL}$: log loss: the average surprise at the true outcomes
$\text{BS}$: Brier score: the average squared gap between forecast and outcome
$n$: the number of transactions scored
$\frac1n$: one over $n$ : turns the sum into an average over transactions
$\sum_i$: add up over every transaction $i$
$y_i$: transaction $i$ 's outcome: 1 if fraud, 0 if legit
$p_i$: the model's fraud probability for transaction $i$
$H(\pi)$: the entropy of the base rate: log loss of always forecasting $\pi$

Brier decomposition: Lie − Learn + Luck (Card 9)

\text{BS} = \underbrace{\sum_k \frac{n_k}{n}(f_k - \bar o_k)^2}_{\text{REL (lie)}} - \underbrace{\sum_k \frac{n_k}{n}(\bar o_k - \bar o)^2}_{\text{RES (learn)}} + \underbrace{\bar o(1 - \bar o)}_{\text{UNC (luck)}}

$\sum_k$: add up over every forecast bin $k$
$n_k$: the number of transactions in bin $k$
$f_k$: the fraud probability forecast for bin $k$
$\bar o_k$: the observed fraud rate in bin $k$ (bar = average of outcomes)
$\bar o$: the overall observed fraud rate

ECE and Brier skill

\text{ECE} = \sum_k \frac{n_k}{n}\,\lvert \bar o_k - f_k\rvert \;\le\; \sqrt{\text{REL}}, \qquad \text{BSS} = 1 - \frac{\text{BS}}{\text{BS}_{\text{base rate}}}

$\text{ECE}$: expected calibration error: the average gap between forecast and observed rate
$\text{REL}$: reliability: the squared calibration gap, the "lie" term
$\text{BSS}$: Brier skill score: the share of the base-rate forecast's Brier removed

Decisions

Cost-optimal threshold and the expected-cost curve (Card 10)

\text{flag} \iff p > t^\star = \frac{c_{FP}}{c_{FP} + c_{FN}}, \qquad \mathbb E[\text{cost}(t)] = \pi\,c_{FN}\,\big(1 - \text{TPR}(t)\big) + (1 - \pi)\,c_{FP}\,\text{FPR}(t)

$p$: the model's calibrated fraud probability for this transaction
$t^\star$: the cost-optimal threshold (star = the optimum)
$c_{FP}$: cost of a false positive: blocking a legit customer (about $10)
$c_{FN}$: cost of a false negative: the missed fraud's amount
$\mathbb E$: expected value: the average cost per transaction

The F1-optimal threshold (Card 11)

t^\star_{F_1} = \frac{F_1^\star}{2} \qquad\text{(population F1, calibrated probabilities)}

$t^\star_{F_1}$: the threshold that maximizes F1
$F_1^\star$: the best F1 any threshold achieves

Uncertainty

A proportion: variance, sample size, Wilson (Card 12)

\text{Var}(\hat p) = \frac{p(1 - p)}{n}, \qquad n = \frac{z^2 p(1 - p)}{h^2}, \qquad \text{Wilson: } \frac{\hat p + \frac{z^2}{2n} \pm z\sqrt{\frac{\hat p(1 - \hat p)}{n} + \frac{z^2}{4n^2}}}{1 + \frac{z^2}{n}}

$\hat p$: a measured proportion, such as recall on the test frauds
$p$: the true proportion that $\hat p$ estimates
$n$: the units in the metric's denominator: frauds, for recall
$z$: the normal quantile for the confidence level (1.96 for 95%)
$h$: the half-width you want for the confidence interval
$\text{Var}$: variance: the wobble across repeated test sets

Hanley–McNeil (Card 13)

\text{Var}(\hat A) \approx \frac{A(1 - A) + (n_1 - 1)(Q_1 - A^2) + (n_0 - 1)(Q_2 - A^2)}{n_1 n_0}, \qquad Q_1 = \frac{A}{2 - A}, \quad Q_2 = \frac{2A^2}{1 + A}

$\hat A$: the AUC measured on the test set (hat = estimated)
$A$: the true AUC
$Q_1$: the chance two random frauds both outscore one random legit
$Q_2$: the chance one random fraud outscores two random legit transactions

DeLong and the paired difference (Card 13)

\widehat{\text{Var}}(\hat A) = \frac{S_{10}}{n_1} + \frac{S_{01}}{n_0}, \qquad \text{Var}(\hat A_A - \hat A_B) = \text{Var}(\hat A_A) + \text{Var}(\hat A_B) - 2\,\text{Cov}(\hat A_A, \hat A_B)

$\widehat{\text{Var}}$: variance estimated from the data (hat = estimated)
$S_{10}$: sample variance, across frauds, of the share of legit each fraud beats
$S_{01}$: sample variance, across legit rows, of the share of frauds above each
$\hat A_A$: model A's measured AUC
$\hat A_B$: model B's measured AUC, on the same rows
$\text{Cov}$: covariance: how the two AUCs wobble together on shared rows

McNemar (Card 14)

\chi^2 = \frac{(|b - c| - 1)^2}{b + c} \;\sim\; \chi^2_1, \qquad b \mid (b + c) \sim \text{Bin}\big(b + c, \tfrac12\big) \text{ under } H_0

$\chi^2$: McNemar's test statistic (continuity-corrected)
$\chi^2_1$: the chi-squared distribution with 1 degree of freedom
$b$: the rows only model A gets right
$c$: the rows only model B gets right
$\sim$: is distributed as
$H_0$: the null hypothesis: both models have the same error rate

Experiments

Sample size per arm (Card 15)

n = \frac{2\,(z_{1-\alpha/2} + z_{1-\beta})^2\,\sigma^2}{\delta^2} \approx \frac{16\,\sigma^2}{\delta^2} \qquad (\alpha = 0.05 \text{ two-sided},\; 80\% \text{ power})

$n$: the units needed in each arm of the test
$z_{1-\alpha/2}$: the normal quantile for a two-sided test (1.96 at 5%)
$z_{1-\beta}$: the normal quantile for the power (0.84 for 80%)
$\alpha$: the false-positive rate the test allows (0.05)
$\sigma^2$: the per-unit variance of the metric
$\delta$: the smallest true difference you want to detect

Correlation of peeks (Card 16)

\text{Corr}(Z_k, Z_l) = \sqrt{n_k / n_l} \quad (n_k < n_l), \qquad \text{two equal looks: } P(\text{reject}) = 1 - P\big(|Z_1| \le 1.96,\, |Z_2| \le 1.96;\; \rho = \sqrt{1/2}\big) = 0.0831

$\text{Corr}$: correlation: how closely two quantities move together
$Z_k$: the cumulative z-statistic at look $k$
$Z_l$: the cumulative z-statistic at a later look $l$
$n_k$: the units collected by look $k$
$n_l$: the units collected by look $l$
$Z_1$: the z-statistic at the first look
$Z_2$: the z-statistic at the second look
$\rho$: the correlation between the two looks' z-statistics

Bonferroni and Benjamini–Hochberg (Card 17)

\text{Bonferroni: reject } p_i \le \frac{\alpha}{m} \;\Rightarrow\; \text{FWER} \le \alpha, \qquad \text{BH: reject } p_{(1)}, \dots, p_{(i^\star)},\;\; i^\star = \max\Big\{i : p_{(i)} \le \frac{i\,q}{m}\Big\} \;\Rightarrow\; \text{FDR} = \frac{m_0}{m}\,q

$p_i$: the p-value of test (metric) $i$
$m$: the number of tests (metrics) run
$\alpha$: the family-wise error rate allowed (0.05)
$p_{(1)}$: the smallest p-value
$p_{(i^\star)}$: the last p-value BH rejects
$i^\star$: the largest rank $i$ whose p-value is under its BH cutoff
$i$: a rank in the sorted p-values (1 = smallest)
$p_{(i)}$: the $i$ -th smallest p-value
$q$: the false discovery rate BH targets
$m_0$: the number of tests where nothing is really going on (true nulls)

CUPED (Card 18)

\hat\Delta_{\text{cv}} = \hat\Delta - \theta\,(\bar X_T - \bar X_C), \qquad \theta^\star = \frac{\text{Cov}(Y, X)}{\text{Var}(X)}, \qquad \text{Var}(\hat\Delta_{\text{cv}}) = (1 - \rho^2)\,\text{Var}(\hat\Delta)

$\hat\Delta$: the plain difference in means, treatment minus control
$\hat\Delta_{\text{cv}}$: the CUPED-adjusted difference (cv = control variate)
$\theta$: the weight on the pre-experiment covariate
$\bar X_T$: the average pre-experiment covariate in treatment (bar = mean)
$\bar X_C$: the average pre-experiment covariate in control
$\theta^\star$: the variance-minimizing weight (star = the optimum)
$Y$: the outcome metric, such as spend during the test
$X$: the pre-experiment covariate, such as last month's spend
$\rho$: the correlation between $Y$ and $X$

Monitoring and LLMs

PSI (Card 19)

\text{PSI} = \sum_i (a_i - e_i)\ln\frac{a_i}{e_i} = \text{KL}(a\,\Vert\,e) + \text{KL}(e\,\Vert\,a) \approx \sum_i \frac{(a_i - e_i)^2}{e_i}

$\sum_i$: add up over every score bin $i$
$a_i$: this week's share of transactions in bin $i$
$e_i$: the reference (expected) share in bin $i$
$\text{KL}(a\,\Vert\,e)$: KL divergence of this week's shares from the reference
$\text{KL}(e\,\Vert\,a)$: KL divergence the other way round
$\text{PSI}$: Population Stability Index: how far this week's score mix drifted

Unbiased pass@k (Card 20)

\widehat{\text{pass@}k} = 1 - \frac{\binom{n - c}{k}}{\binom{n}{k}} = 1 - \prod_{j = n - c + 1}^{n}\Big(1 - \frac{k}{j}\Big)

$\widehat{\text{pass@}k}$: the unbiased estimate of pass@k from one problem's samples
$n$: the number of samples generated for the problem
$c$: how many of those samples are correct
$k$: the number of attempts pass@k allows
$\binom{n - c}{k}$: the ways to pick $k$ samples that are all wrong
$\binom{n}{k}$: the ways to pick any $k$ of the $n$ samples
$\prod_{j = n - c + 1}^{n}$: multiply together the terms for $j$ from $n - c + 1$ to $n$
$j$: the running index of the product

Cards

★ = asked often, ★★ = asked at senior level, ★★★ = research level. Toolkit letters in Key move refer to R-A-T-E-S above.

Card 1 · Precision from Bayes: why it moves with the base rate ★

Statement. At any threshold, precision is set by three numbers: the base rate $\pi$ , the recall (TPR) and the false-positive rate. TPR and FPR are computed within a true class, so they don't depend on $\pi$ . Precision does, which is why a model's precision on an enriched or balanced test set says little about production.

\text{precision} = \frac{\pi\,\text{TPR}}{\pi\,\text{TPR} + (1 - \pi)\,\text{FPR}}

$\pi$: the base rate: the share of transactions that are fraud
$\text{TPR}$: recall: the share of frauds the model flags
$\text{FPR}$: the share of legit transactions the model flags

Derivation.

Name the events: Y = 1 is fraud, F is "flagged". The three rates are probabilities, and precision is the one that conditions on the prediction.

\pi = P(Y = 1), \qquad \text{TPR} = P(F \mid Y = 1), \qquad \text{FPR} = P(F \mid Y = 0), \qquad \text{precision} = P(Y = 1 \mid F)

$P$: the probability of the event in brackets; $\mid$ reads "given"
$Y$: the true class: $Y = 1$ for fraud, $Y = 0$ for legit
$F$: the event "the model flags this transaction"

Apply Bayes' rule to flip the conditioning.

P(Y = 1 \mid F) = \frac{P(F \mid Y = 1)\,P(Y = 1)}{P(F)}

Expand P(F) by total probability over the two true classes. Every flag is either a caught fraud or a blocked legit transaction.

P(F) = P(F \mid Y = 1)\,P(Y = 1) + P(F \mid Y = 0)\,P(Y = 0) = \pi\,\text{TPR} + (1 - \pi)\,\text{FPR}

Substitute to get the statement. Dividing precision by $1 - \text{precision}$ gives the odds form: posterior odds = prior odds × likelihood ratio. The likelihood ratio $\text{TPR}/\text{FPR}$ ( $\text{LR}^+$ ) is the model's contribution, and the prior odds come from the world.

\frac{\text{precision}}{1 - \text{precision}} = \frac{\pi\,\text{TPR}}{(1 - \pi)\,\text{FPR}} = \underbrace{\frac{\pi}{1 - \pi}}_{\text{world}}\cdot\underbrace{\frac{\text{TPR}}{\text{FPR}}}_{\text{model}}

Why TPR and FPR "travel". Under prior shift (the fraud rate changes but each class's score distribution doesn't), the flagged share of each class is unchanged. So TPR, FPR and any function of them alone (ROC, AUC, balanced accuracy) are unchanged too, and only the $\pi$ in the formula moves.

\text{TPR}(t) = P(s > t \mid Y = 1), \quad \text{FPR}(t) = P(s > t \mid Y = 0) \qquad\text{(neither contains } \pi\text{)}

$t$: the score threshold: flag when the score is above it
$s$: the transaction's model score

Check on the toy. A at 0.5 has TPR 0.5 and FPR 0.25. In the review sample $\pi = 4/12$ , which gives precision 0.50. On the day the sample came from, $\pi = 0.01$ , which gives precision 0.0198. The same model at the same threshold is 25 times less precise, and nothing about the model changed.

\text{sample: } \frac{\tfrac13(0.5)}{\tfrac13(0.5) + \tfrac23(0.25)} = 0.50, \qquad \text{day: } \frac{0.01(0.5)}{0.01(0.5) + 0.99(0.25)} = \frac{0.005}{0.2525} = 0.0198

B at 0.5 (TPR 0.75, FPR 0.125) goes from 0.75 on the sample to 0.0571 on the day. A vendor's TPR 0.8 and FPR 0.05 give 0.941 on a balanced test set and 0.139 at 1% (Drill 5).

\frac{0.5(0.8)}{0.5(0.8) + 0.5(0.05)} = 0.941, \qquad \frac{0.01(0.8)}{0.01(0.8) + 0.99(0.05)} = \frac{0.008}{0.0575} = 0.139

Check on the 100,000. At A's cost-optimal threshold, TPR = 0.345 and FPR = 0.0218. With the test base rate 0.00974, the formula gives 0.134, which equals the precision counted directly on the test rows.

\frac{0.00974 \times 0.345}{0.00974 \times 0.345 + 0.99026 \times 0.0218} = 0.134

The limit. As $\pi \to 0$ precision falls roughly in proportion to $\pi$ . To keep precision fixed while fraud gets rarer, the FPR has to fall just as fast.

\text{precision} \approx \frac{\pi}{1 - \pi}\cdot\frac{\text{TPR}}{\text{FPR}} \quad (\text{small } \pi), \qquad \text{fixed precision} \;\Rightarrow\; \text{FPR} \propto \pi

Key move. T: total probability for the denominator, Bayes for the flip.

Common slip. Treating precision as a property of the model alone and quoting it from a balanced, enriched or vendor test set. The related slip is the base-rate fallacy: reading P(flag | fraud) = 0.8 as "80% of flags are fraud".

How it's asked. "A vendor shows 95% precision at 80% recall on a balanced test set. We see 1% fraud. What precision do we get?" (Q10, Drill 5.) Strong answer: ask for TPR and FPR, convert with the formula, and get about 0.14 if FPR is 5%. Add that AP, accuracy and F1 shift for the same reason, while ROC and AUC don't. Ch 3, Card 13 converts the probabilities themselves.

Card 2 · F-scores and MCC ★

Statement. F1 is the harmonic mean of precision and recall. In counts it never mentions true negatives. $F_\beta$ is a weighted harmonic mean that counts recall $\beta$ times as much as precision. MCC is the Pearson correlation between the 0/1 prediction and the 0/1 truth, so it uses all four boxes.

F_1 = \frac{2\,TP}{2\,TP + FP + FN}, \qquad F_\beta = \frac{(1 + \beta^2)\,TP}{(1 + \beta^2)\,TP + \beta^2 FN + FP}, \qquad \text{MCC} = \frac{TP\cdot TN - FP\cdot FN}{\sqrt{(TP + FP)(TP + FN)(TN + FP)(TN + FN)}}

$F_1$: the F1 score: the harmonic mean of precision and recall
$F_\beta$: the F-beta score: recall counts $\beta$ times as much as precision
$\beta$: how many times more recall matters than precision (2 for F2)
$TP$: true positives: frauds the model flags
$FP$: false positives: legit transactions the model flags
$FN$: false negatives: frauds the model lets through
$TN$: true negatives: legit transactions correctly passed
$\text{MCC}$: Matthews correlation: the correlation of the 0/1 flag with the 0/1 truth

Derivation.

Write the harmonic mean through reciprocals, and write each reciprocal in counts.

F_1 = \frac{2}{\frac1P + \frac1R}, \qquad \frac1P = \frac{TP + FP}{TP}, \qquad \frac1R = \frac{TP + FN}{TP}

$\frac1P$: one over precision
$\frac1R$: one over recall
$P$: precision: the share of flags that are really fraud (not a probability here)
$R$: recall: the share of frauds that get flagged

Add and invert. TN has dropped out, so F1 cannot see the 99,000 legit transactions you correctly passed.

\frac1P + \frac1R = \frac{2\,TP + FP + FN}{TP} \;\Rightarrow\; F_1 = \frac{2\,TP}{2\,TP + FP + FN}

Why the harmonic mean: it collapses when either component is small, so a model can't buy a good F1 with recall alone. With P = 1 and R = 0.01 the harmonic mean is 0.0198, while the arithmetic mean would be 0.505.

\frac{2(1)(0.01)}{1 + 0.01} = 0.0198 \qquad\text{vs}\qquad \frac{1 + 0.01}{2} = 0.505

$F_\beta$ puts weight $\beta^2/(1 + \beta^2)$ on $1/R$ and $1/(1 + \beta^2)$ on $1/P$ . Expand in counts.

\frac{1}{F_\beta} = \frac{1}{1 + \beta^2}\cdot\frac{TP + FP}{TP} + \frac{\beta^2}{1 + \beta^2}\cdot\frac{TP + FN}{TP} = \frac{(1 + \beta^2)\,TP + FP + \beta^2 FN}{(1 + \beta^2)\,TP}

Invert. In the counts, $\beta^2$ multiplies FN, so each missed fraud costs $\beta^2$ times as much as a false alarm. Multiplying through by $PR$ gives the precision–recall form. F2 favors recall, F0.5 favors precision.

F_\beta = \frac{(1 + \beta^2)\,TP}{(1 + \beta^2)\,TP + \beta^2 FN + FP} = \frac{(1 + \beta^2)\,PR}{\beta^2 P + R}

MCC as a correlation. Code truth $y$ and prediction $\hat y$ as 0/1 over $n$ rows. The Pearson correlation needs $E[y\hat y]$ , the two means and the two variances, all of which are box counts over $n$ .

\mathbb E[y\hat y] = \frac{TP}{n}, \quad \mathbb E[y] = \frac{TP + FN}{n}, \quad \mathbb E[\hat y] = \frac{TP + FP}{n}, \quad \text{Var}(y) = \frac{(TP + FN)(FP + TN)}{n^2}, \quad \text{Var}(\hat y) = \frac{(TP + FP)(FN + TN)}{n^2}

$\mathbb E$: expected value: here the average over the $n$ rows
$y$: a row's truth: 1 if fraud, 0 if legit
$\hat y$: the model's 0/1 prediction for that row (hat = from the model)
$n$: the number of rows in the confusion matrix
$\text{Var}$: variance: the average squared distance from the mean

The covariance numerator, multiplied by $n^2$ , is $n\cdot\text{TP} - (\text{TP} + \text{FN})(\text{TP} + \text{FP})$ . Expand both products with $n = \text{TP} + \text{FP} + \text{FN} + \text{TN}$ (add-and-subtract in disguise). Everything cancels except two terms.

n\,TP - (TP + FN)(TP + FP) = (TP^2 + TP\,FP + TP\,FN + TP\,TN) - (TP^2 + TP\,FP + FN\,TP + FN\,FP) = TP\cdot TN - FP\cdot FN

Divide by the square root of the variance product (the $n^2$ factors cancel) to get MCC. It is 1 for perfect prediction, 0 for no association, $-1$ for perfectly wrong. It is symmetric in which class you call positive; F1 is not. It is the $\phi$ coefficient of the $2\times 2$ table, so $n\cdot\text{MCC}^2$ is the table's $\chi^2$ statistic.

\text{MCC} = \frac{TP\cdot TN - FP\cdot FN}{\sqrt{(TP + FP)(TP + FN)(TN + FP)(TN + FN)}}, \qquad n\cdot\text{MCC}^2 = \chi^2

$\chi^2$: the chi-squared statistic of the 2×2 table

Check on the toy at 0.5. For A, $P = R = 0.5$ , and when $P = R$ every $F_\beta$ equals both (F2 = F0.5 = 0.5). B's F1 is 0.75 and MCC is 0.625.

\text{A: } F_1 = \frac{4}{4 + 2 + 2} = 0.5, \;\; \text{MCC} = \frac{2\cdot 6 - 2\cdot 2}{\sqrt{4\cdot4\cdot8\cdot8}} = \frac{8}{32} = 0.25 \qquad \text{B: } F_1 = \frac{6}{6 + 1 + 1} = 0.75, \;\; \text{MCC} = \frac{21 - 1}{32} = 0.625

Both depend on the base rate through FP. Weight A's legit rows back to the day ( $\text{FP} = 2 \times 49.5 = 99$ , $\text{TN} = 297$ ) and F1 falls from 0.5 to 0.0381 and MCC from 0.25 to 0.057, while TPR and FPR are unchanged.

F_1^{\text{day}} = \frac{4}{4 + 99 + 2} = 0.0381, \qquad \text{MCC}^{\text{day}} = \frac{2\cdot 297 - 99\cdot 2}{\sqrt{101\cdot 4\cdot 396\cdot 299}} = 0.057

Key move. A: the expansion in step 7, plus the harmonic-mean reciprocals.

Common slip. Computing F1 as the arithmetic mean of P and R, or writing $\beta$ instead of $\beta^2$ in $F_\beta$ . The second slip is believing F1 or MCC travel across base rates: both contain FP, so they move with $\pi$ just as precision does (Card 1).

How it's asked. "Here are twelve transactions and A's scores. At 0.5, give me the confusion matrix, precision, recall, F1, FPR and MCC — and why is MCC so much lower than accuracy?" (Q13.) Strong answer: MCC 0.25 is a correlation, which is low because half the flags are wrong and half the frauds are missed, while accuracy is propped up by the easy legit box. Also mention that MCC is 0/0 for a model that flags nothing (sklearn returns 0).

Card 3 · AUC = P(s⁺ > s⁻) = U/(n₁n₀) ★★

Statement. The area under the empirical ROC staircase equals the share of fraud–legit pairs in which the fraud is scored higher, with ties counting one half. It also equals the Mann–Whitney $U$ statistic over $n_1 n_0$ , which the rank-sum formula computes in one sort. In the population, the area is the probability that a random fraud outranks a random legit transaction.

\text{AUC} = \frac{1}{n_1 n_0}\sum_{i=1}^{n_1}\sum_{j=1}^{n_0}\psi_{ij} = \frac{R_1 - n_1(n_1 + 1)/2}{n_1 n_0} \;\xrightarrow{\;n\to\infty\;}\; P(s^+ > s^-) + \tfrac12 P(s^+ = s^-)

$\text{AUC}$: area under the ROC curve of TPR against FPR
$n_1$: the number of frauds (4 in the review sample)
$n_0$: the number of legit transactions (8 in the review sample)
$\sum_{i=1}^{n_1}$: add up over every fraud $i$
$\sum_{j=1}^{n_0}$: add up over every legit transaction $j$
$\psi_{ij}$: the pair score: 1 if fraud $i$ outscores legit $j$ , ½ for a tie, else 0
$R_1$: the sum of the frauds' ranks, all scores ranked from lowest (rank 1) up
$\xrightarrow{\;n\to\infty\;}$: tends to, as the number of transactions grows without limit
$P$: the probability of the event in brackets (back to probability)
$s^+$: the score of a randomly drawn fraud
$s^-$: the score of a randomly drawn legit transaction

Derivation.

Build the empirical ROC by lowering the threshold past one score at a time. Passing a fraud moves the curve up by $1/n_1$ (TPR rises). Passing a legit transaction moves it right by $1/n_0$ (FPR rises).

\text{fraud passed: } \Delta\text{TPR} = \frac{1}{n_1}, \qquad \text{legit passed: } \Delta\text{FPR} = \frac{1}{n_0}

$\Delta$: the change in

Only right-steps add area, a strip of width $1/n_0$ at the current height. When the threshold passes legit transaction $j$ , the height is the share of frauds already above it.

\text{area} = \sum_{j=1}^{n_0}\frac{1}{n_0}\cdot\text{TPR just after passing } s_j^- = \sum_{j=1}^{n_0}\frac{1}{n_0}\cdot\frac{\#\{i : s_i^+ > s_j^-\}}{n_1}

$s_j^-$: the score of legit transaction $j$
$s_i^+$: the score of fraud $i$
$i$: which fraud, numbered 1 to $n_1$

Write the count as a sum of indicators. The area is then the double sum over pairs.

\text{area} = \frac{1}{n_1 n_0}\sum_{j=1}^{n_0}\sum_{i=1}^{n_1}\mathbf 1(s_i^+ > s_j^-)

$\mathbf 1$: indicator: 1 if the condition in brackets is true, else 0

Ties. If a fraud and a legit share a score, the threshold passes both at once and the curve moves diagonally. The triangle under that diagonal is half the rectangle, so a tied pair contributes $\tfrac12$ (the $\psi_{ij}$ of the toolkit).

\text{diagonal step of size } \tfrac{1}{n_0}\times\tfrac{1}{n_1}: \quad \text{area} = \tfrac12\cdot\frac{1}{n_1 n_0}

Population version. The ROC is the curve $(\text{FPR}(t), \text{TPR}(t)) = (\bar F_0(t), \bar F_1(t))$ , where $\bar F$ is the survival function and $f_0$ the legit score density. As $t$ falls from $+\infty$ to $-\infty$ , FPR rises from 0 to 1, and $d\,\text{FPR} = -f_0(t)\,dt$ . Substitute and flip the limits.

\int_0^1 \text{TPR}\;d\,\text{FPR} = \int_{+\infty}^{-\infty}\bar F_1(t)\,\big(-f_0(t)\big)\,dt = \int_{-\infty}^{\infty} P(s^+ > t)\,f_0(t)\,dt = P(s^+ > s^-)

$\int_0^1$: add up the area as FPR runs from 0 to 1
$d\,\text{FPR}$: a tiny step along the FPR axis
$\bar F_1(t)$: the share of frauds scoring above $t$ (here bar = 1 minus the CDF)
$f_0(t)$: the density of legit scores at $t$
$dt$: a tiny step in the threshold $t$

The last step conditions on the legit score $s^- = t$ and averages over its density; it needs $s^+$ and $s^-$ independent, which holds for one fraud and one legit drawn at random.

The rank-sum shortcut. Rank all $n = n_1 + n_0$ scores from lowest (1) to highest ( $n$ ). A fraud's rank is the number of scores at or below it. Split that into frauds at or below it (counting itself) and legit below it.

\text{rank}(s_i^+) = \#\{\text{frauds} \le s_i^+\} + \#\{j : s_j^- < s_i^+\}

$j$: which legit transaction

Sum over frauds. Among the frauds alone, the first counts are $1, 2, \dots, n_1$ , which sum to $n_1(n_1 + 1)/2$ . The second counts sum to $U$ , the number of pairs the frauds win.

R_1 = \sum_{i=1}^{n_1}\text{rank}(s_i^+) = \frac{n_1(n_1 + 1)}{2} + U \;\Rightarrow\; \text{AUC} = \frac{U}{n_1 n_0} = \frac{R_1 - n_1(n_1 + 1)/2}{n_1 n_0}

$U$: the Mann–Whitney count: (fraud, legit) pairs the fraud wins, ties as ½

With ties, give tied scores the average of their ranks. A tied fraud–legit pair then adds $\tfrac12$ to $U$ , matching step 4. One sort makes this $O(n \log n)$ , against $O(n_1 n_0)$ for pair counting.

Check on the toy by pairs. A's frauds beat 8, 7, 6 and 4 of the 8 legit transactions (T8 at 0.20 loses to T2, T4, T6 and T7). B's frauds beat 8, 8, 8 and 0, because B scores T8 at 0.02, below every legit.

\text{A: } \frac{8 + 7 + 6 + 4}{32} = \frac{25}{32} = 0.78125, \qquad \text{B: } \frac{8 + 8 + 8 + 0}{32} = \frac{24}{32} = 0.75

Check by ranks. A's frauds have ascending ranks 12, 10, 8, 5 ( $R_1 = 35$ ); B's have 12, 11, 10, 1 ( $R_1 = 34$ ). Subtract $n_1(n_1 + 1)/2 = 10$ .

\text{A: } \frac{35 - 10}{32} = 0.78125, \qquad \text{B: } \frac{34 - 10}{32} = 0.75

The null distribution comes free, which is why "is AUC above 0.5?" is the Wilcoxon rank-sum test. If scores are unrelated to fraud (and untied), $R_1$ is the sum of $n_1$ ranks drawn without replacement from $1, \dots, n$ . One rank has mean $(n + 1)/2$ and variance $(n^2 - 1)/12$ .

\mathbb E[R] = \frac{n + 1}{2}, \qquad \text{Var}(R) = \frac{(n + 1)(2n + 1)}{6} - \frac{(n + 1)^2}{4} = \frac{(n + 1)(n - 1)}{12}

$R$: the rank of one score drawn at random from $1, \dots, n$
$n$: all scored transactions, $n_1 + n_0$
$\mathbb E$: expected value: the average over random rankings

Without replacement, two different draws have covariance $c$ . All $n$ ranks always sum to the same total, so $0 = \text{Var}(\text{total}) = n\cdot\text{Var}(R) + n(n - 1)c$ , giving $c = -\text{Var}(R)/(n - 1)$ . Then the variance of the sum of $n_1$ draws follows.

\text{Var}(R_1) = n_1\text{Var}(R) + n_1(n_1 - 1)\,c = n_1\text{Var}(R)\,\frac{n - n_1}{n - 1} = \frac{n_1 n_0 (n + 1)}{12}

$c$: the covariance of two different ranks drawn without replacement (negative)

Divide by $(n_1 n_0)^2$ . Under $H_0$ the AUC has mean $\tfrac12$ and a variance set mostly by the smaller class: with $n_0 \gg n_1$ , $(n + 1)/(n_1 n_0) \approx 1/n_1$ .

\mathbb E[\widehat{\text{AUC}}] = \frac{n_1(n + 1)/2 - n_1(n_1 + 1)/2}{n_1 n_0} = \frac12, \qquad \text{Var}(\widehat{\text{AUC}} \mid H_0) = \frac{n + 1}{12\,n_1 n_0}

$\widehat{\text{AUC}}$: the AUC measured on the sample (hat = estimated)
$H_0$: the null hypothesis: scores are unrelated to fraud

Key move. R: count pairs, then rearrange the count.

Common slip. Three versions. Ranking from the top (rank 1 = highest) and still subtracting $n_1(n_1 + 1)/2$ gives $1 - \text{AUC}$ . Subtracting $n_0(n_0 + 1)/2$ instead of $n_1(n_1 + 1)/2$ is the second. The third is dropping ties, or counting them as wins, which matters for discretized scores or predict() output (Card 5).

How it's asked. "No libraries: compute A's AUC two ways, then prove that the area equals the probability a random fraud outranks a random legit." (Q14, Q15.) Strong answer: the staircase argument for the finite sample, the $\int \text{TPR}\,d\text{FPR}$ substitution for the population, and the rank sum as the $O(n \log n)$ implementation. Then add that $\text{AUC} = 0.5$ testing is Mann–Whitney.

Card 4 · What AUC is (and isn't) invariant to ★

Statement. AUC is unchanged by any strictly increasing transform of the scores and by any reweighting that treats every row of a class alike, including a change of base rate. Precision, F1 at a fixed threshold, log loss and Brier change under both. The best F1 over all thresholds survives a monotone transform but not a base-rate change. A non-strict transform such as isotonic regression can create ties and so move AUC slightly.

g \text{ strictly increasing} \;\Rightarrow\; \text{AUC}(g(s)) = \text{AUC}(s), \qquad \max_t F_1\big(g(s), t\big) = \max_t F_1(s, t), \qquad F_1\big(g(s), t\big) \ne F_1(s, t) \text{ in general}

$g$: a transform applied to every score, such as squaring it
$\max_t$: the best value over all thresholds $t$

Derivation.

A strictly increasing $g$ preserves every comparison and every tie, so every pair indicator $\psi_{ij}$ is unchanged. AUC is an average of those indicators (Card 3), so it's unchanged.

g(a) > g(b) \iff a > b, \qquad g(a) = g(b) \iff a = b \;\;\Rightarrow\;\; \psi_{ij}\big(g(s)\big) = \psi_{ij}(s)

$a$: any one model score
$b$: any other score

Class reweighting. Give every legit row weight $w_0$ and every fraud weight $w_1$ . The weighted AUC is a weighted average of pair indicators with every pair weighted $w_1 w_0$ , so the weights cancel. On the toy, counting each legit row 49.5 times leaves the AUCs at 0.78125 and 0.75. On the 100,000, a rebalanced 50/50 version gives A an AUC of 0.868, the same as the natural one.

\text{AUC}_w = \frac{\sum_i\sum_j w_1 w_0\,\psi_{ij}}{\sum_i\sum_j w_1 w_0} = \frac{w_1 w_0\sum_i\sum_j\psi_{ij}}{w_1 w_0\, n_1 n_0} = \text{AUC}

$\text{AUC}_w$: AUC with every row counted by its class weight
$\sum_i$: add up over every fraud $i$
$\sum_j$: add up over every legit transaction $j$
$w_1$: the weight on every fraud row
$w_0$: the weight on every legit row (49.5 to match the day)

A fixed threshold on $g(s)$ is a different threshold on $s$ , so it flags a different set. On the toy, squaring A's scores and cutting at 0.5 is the same as cutting the raw scores at $\sqrt{0.5} = 0.707$ . That flags T1 and T2 only, and F1 falls from 0.5 to 0.333. On the 100,000, F1 at a fixed 0.05 goes from 0.169 to 0.131 after squaring.

g(s) > 0.5 \iff s > g^{-1}(0.5) = 0.707: \quad TP = 1,\; FP = 1,\; FN = 3 \;\Rightarrow\; F_1 = \frac{2}{2 + 1 + 3} = 0.333

$g^{-1}$: the inverse transform: undoes $g$ (here the square root)

The best F1 over all thresholds only ever sees the sets "the top k rows", and a strictly increasing g leaves the order, and so every top-k set, unchanged. The maximum is the same: 0.667 on the toy before and after squaring (reached at k = 5 or k = 8).

\max_t F_1(g(s), t) = \max_{k}\; F_1\big(\text{top } k\big) = \max_t F_1(s, t), \qquad \text{toy: } F_1(\text{top } 5) = \frac{6}{6 + 2 + 1} = 0.667

$\max_{k}$: the best value over every cutoff $k$
$k$: a cutoff: flag the top $k$ rows

But max-F1 is not invariant to the base rate. Reweighting legit rows by $w_0$ multiplies FP by $w_0$ in every top- $k$ set, while TP and FN stay put.

F_1^{(w_0)}(\text{top } k) = \frac{2\,TP_k}{2\,TP_k + w_0\,FP_k + FN_k}

$F_1^{(w_0)}$: F1 with every legit row weighted by $w_0$
$TP_k$: frauds in the top $k$ rows
$FP_k$: legit rows in the top $k$
$FN_k$: frauds below the top $k$

AP is a base-rate metric for the same reason, because it averages precisions. Weighted to the day, the toy's AP falls from 0.692 to 0.272 for A and from 0.833 to 0.753 for B. Rebalancing the 100,000 to 50/50 lifts A's AP from 0.133 to 0.866.

\text{AP} = \frac{1}{n_1}\sum_{i}\frac{TP_{r_i}}{TP_{r_i} + w_0\,FP_{r_i}} \quad\text{depends on } w_0

$\text{AP}$: average precision: the mean precision at each fraud's rank
$TP_{r_i}$: frauds in the top $r_i$ rows, where $r_i$ is fraud $i$ 's rank
$FP_{r_i}$: legit rows in the top $r_i$

Log loss and Brier depend on the score values, so any transform that changes the values changes them. Platt scaling is strictly increasing: B's Brier falls from 0.128 to 0.00679, while its AUC change is exactly zero.

\Delta\text{AUC}_{\text{Platt}} = 0, \qquad \text{BS}: 0.128 \to 0.00679

$\text{BS}$: Brier score: the average squared gap between forecast and outcome

"Strictly" matters. Isotonic regression is a non-decreasing step function, so it maps many scores to the same value. Pairs that were wins become ties worth $\tfrac12$ . On the 100,000, isotonic maps the test set to 68 distinct values; AUC moves from 0.848 to 0.846, and AP falls 0.025 (0.399 to 0.374) because the ties land at the top of the list.

g(a) = g(b) \text{ for } a > b \;\Rightarrow\; \psi_{ij}: 1 \to \tfrac12

Key move. R for the invariances (pairs and top-k sets), T for the base-rate dependence of anything containing FP.

Common slip. Saying "AUC is invariant to any transform". It needs a strictly increasing one (squaring negative scores, or isotonic steps, break it). The other slip is reading an improvement in F1 at a fixed threshold after a transform as a better model: the threshold moved, not the model.

How it's asked. "A teammate took the square root of the scores and says F1 improved from 0.42 to 0.47 and AUC was unchanged, so it's a free win." (Q16, Drill 4.) Strong answer: AUC and max-F1 can't change; fixed-threshold F1 changed because $\sqrt{\cdot}$ moved the effective cut, which you could have done by choosing a different threshold. If probabilities are consumed downstream, $\sqrt{\cdot}$ also destroys calibration.

Card 5 · AUC of a hard classifier and of a random one ★★

Statement. A 0/1 classifier has a ROC with one corner at $(\text{FPR}, \text{TPR})$ , so its AUC is the balanced accuracy. A constant score gives 0.5 (every pair is a tie), and a score independent of the label gives 0.5 in expectation. Negating a score turns AUC into $1 - \text{AUC}$ .

\text{AUC}_{0/1} = \frac{1 + \text{TPR} - \text{FPR}}{2} = \frac{\text{TPR} + \text{TNR}}{2}, \qquad \text{AUC}(-s) = 1 - \text{AUC}(s)

$\text{AUC}_{0/1}$: the AUC of a model that only outputs 0 or 1
$\text{TNR}$: true-negative rate: the share of legit passed, $1 - \text{FPR}$

Derivation.

With scores in {0, 1}, a random fraud–legit pair falls into one of four cases. Their probabilities come from the two rates, and the two draws are independent.

P(\text{fraud } 1, \text{ legit } 0) = \text{TPR}(1 - \text{FPR}), \;\; P(1, 1) = \text{TPR}\cdot\text{FPR}, \;\; P(0, 0) = (1 - \text{TPR})(1 - \text{FPR}), \;\; P(0, 1) = (1 - \text{TPR})\,\text{FPR}

AUC counts the first case as a win and the two tied cases as halves.

\text{AUC} = \text{TPR}(1 - \text{FPR}) + \tfrac12\big[\text{TPR}\cdot\text{FPR} + (1 - \text{TPR})(1 - \text{FPR})\big]

Expand. The $\text{TPR}\cdot\text{FPR}$ terms cancel.

\text{AUC} = \text{TPR} - \text{TPR}\,\text{FPR} + \tfrac12\text{TPR}\,\text{FPR} + \tfrac12 - \tfrac12\text{TPR} - \tfrac12\text{FPR} + \tfrac12\text{TPR}\,\text{FPR} = \frac{1 + \text{TPR} - \text{FPR}}{2}

Geometric check. The ROC runs $(0, 0) \to (\text{FPR}, \text{TPR}) \to (1, 1)$ . The area is a triangle plus a trapezoid, and it gives the same answer.

\tfrac12\,\text{FPR}\cdot\text{TPR} + (1 - \text{FPR})\,\frac{\text{TPR} + 1}{2} = \frac{1 + \text{TPR} - \text{FPR}}{2}

Check on the toy. B's 0/1 predictions at 0.5 (TPR 0.75, FPR 0.125) give 0.8125, not B's AUC of 0.75; A's give 0.625, not 0.781. On the 100,000, roc_auc_score(y, predict(X)) gives 0.768 for B (TPR 0.696, FPR 0.159) against its true AUC 0.848. It gives 0.510 for A, which almost never scores above 0.5, against 0.868.

\text{B: } \frac{1 + 0.75 - 0.125}{2} = 0.8125, \qquad \text{A: } \frac{1 + 0.5 - 0.25}{2} = 0.625

A constant score ties every pair, so $\text{AUC} = \tfrac12$ exactly. A score independent of the label makes $s^+$ and $s^-$ exchangeable, so wins and losses are equally likely and AUC is $\tfrac12$ in expectation. A simulation with uniform random scores gives 0.4997, and one with a random 30% scored 1 gives 0.4989 (Drill 1).

P(s^+ > s^-) = P(s^- > s^+) \;\Rightarrow\; \text{AUC} = P(s^+ > s^-) + \tfrac12\big[1 - 2P(s^+ > s^-)\big] = \tfrac12

Negating the score swaps wins and losses and leaves ties alone. An AUC of 0.30 is a model with AUC 0.70 pointed the wrong way, usually a flipped label or a "probability of legit" column (Drill 3).

\mathbf 1(-a > -b) = \mathbf 1(b > a) \;\Rightarrow\; \text{AUC}(-s) = P(s^- > s^+) + \tfrac12 P(\text{tie}) = 1 - \text{AUC}(s)

A rules engine with TPR 0.6 and FPR 0.1 that outputs only block or pass will be reported with AUC 0.75 (Drill 2).

\frac{1 + 0.6 - 0.1}{2} = 0.75

Key move. R: four pair cases, ties at one half.

Common slip. Passing model.predict(X) to roc_auc_score. You get the balanced accuracy at threshold 0.5, which can sit anywhere relative to the real AUC (above it for B on the toy, far below it for A on the 100,000). The other slip is calling an AUC below 0.5 "worse than random" without checking the label encoding.

How it's asked. "Implement AUC in NumPy, then tell me what roc_auc_score(y, model.predict(X)) computes." (Q26.) Strong answer: (TPR + TNR)/2 at the default threshold, derived from the one-corner ROC. Add that you pass predict_proba[:, 1] or decision_function instead, the S of SLA (Scores, Label, Average).

Card 6 · The binormal AUC ★★★

Statement. If legit scores are $N(\mu_0, \sigma_0^2)$ and fraud scores are $N(\mu_1, \sigma_1^2)$ , independent, the AUC has a closed form. With equal unit variances and a gap $d$ , it is $\Phi(d/\sqrt{2})$ . The ROC is a straight line on probit axes, and unequal variances make it cross the diagonal (an "improper" ROC).

\text{AUC} = \Phi\!\left(\frac{\mu_1 - \mu_0}{\sqrt{\sigma_0^2 + \sigma_1^2}}\right), \qquad \Phi^{-1}(\text{TPR}) = a + b\,\Phi^{-1}(\text{FPR}), \quad a = \frac{\mu_1 - \mu_0}{\sigma_1},\; b = \frac{\sigma_0}{\sigma_1}

$\Phi$: the standard normal CDF: the share of a bell curve below its argument
$\Phi^{-1}$: its inverse (probit): turns a rate into a z-score
$\mu_0$: the mean legit score
$\mu_1$: the mean fraud score
$\sigma_0^2$: the variance of legit scores
$\sigma_1^2$: the variance of fraud scores
$\sigma_0$: the standard deviation of legit scores
$\sigma_1$: the standard deviation of fraud scores
$a$: the intercept of the ROC line on probit axes
$b$: the slope of the ROC line on probit axes

Derivation.

By Card 3, AUC is the probability that the difference of one fraud score and one legit score is positive (continuous scores, so no ties).

\text{AUC} = P(s^+ > s^-) = P(D > 0), \qquad D = s^+ - s^-

$D$: the score gap $s^+ - s^-$ between a random fraud and a random legit

A difference of independent normals is normal. Means subtract and variances add (toolkit S, with zero covariance).

D \sim N\big(\mu_1 - \mu_0,\; \sigma_1^2 + \sigma_0^2\big)

$\sim$: is distributed as
$N$: a normal distribution with the (mean, variance) in brackets

Standardize $D$ and use the symmetry of $\Phi$ .

P(D > 0) = P\!\left(Z > \frac{-(\mu_1 - \mu_0)}{\sqrt{\sigma_0^2 + \sigma_1^2}}\right) = \Phi\!\left(\frac{\mu_1 - \mu_0}{\sqrt{\sigma_0^2 + \sigma_1^2}}\right)

$Z$: a standard normal variable: mean 0, SD 1

Check with $\mu_0 = 0$ , $\mu_1 = d = 2$ and unit variances. Theory gives 0.9214; 500 simulated datasets of 100,000 rows at 1% fraud average 0.9212. For $d = 1$ and $d = 3$ the values are 0.760 and 0.983.

\Phi\big(2/\sqrt2\big) = \Phi(1.414) = 0.9214

The ROC on probit axes. At threshold $t$ , each rate is a normal tail. Apply $\Phi^{-1}$ to both and eliminate $t$ .

\text{TPR} = \Phi\!\Big(\frac{\mu_1 - t}{\sigma_1}\Big), \;\; \text{FPR} = \Phi\!\Big(\frac{\mu_0 - t}{\sigma_0}\Big) \;\Rightarrow\; t = \mu_0 - \sigma_0\,\Phi^{-1}(\text{FPR}) \;\Rightarrow\; \Phi^{-1}(\text{TPR}) = \frac{\mu_1 - \mu_0}{\sigma_1} + \frac{\sigma_0}{\sigma_1}\,\Phi^{-1}(\text{FPR})

In the $(a, b)$ form the AUC is $\Phi(a/\sqrt{1 + b^2})$ , the same number as step 3. This is how the binormal ROC is fit in practice: regress probit TPR on probit FPR.

\frac{a}{\sqrt{1 + b^2}} = \frac{(\mu_1 - \mu_0)/\sigma_1}{\sqrt{1 + \sigma_0^2/\sigma_1^2}} = \frac{\mu_1 - \mu_0}{\sqrt{\sigma_1^2 + \sigma_0^2}}

Unequal variances: fraud scores $N(2, 4)$ ( $\sigma_1 = 2$ ). The AUC is $\Phi(2/\sqrt{5}) = 0.814$ , and simulation gives 0.814. Here $a = 1$ and $b = \tfrac12$ , and the ROC meets the diagonal where $\Phi^{-1}(\text{TPR}) = \Phi^{-1}(\text{FPR}) = z$ . That happens at $\text{FPR} = \Phi(2) = 0.977$ , above which the curve dips below chance. A score that isn't monotone in the likelihood ratio can always be improved.

z = 1 + \tfrac12 z \;\Rightarrow\; z = 2 \;\Rightarrow\; \text{FPR} = \text{TPR} = \Phi(2) = 0.977

$z$: the shared probit value where the ROC meets the diagonal

Operating point. For $\text{TPR} = 0.8$ with equal variances, the threshold sits $z_{0.8} = 0.8416$ below the fraud mean. The FPR is the legit tail beyond it, and precision follows from Card 1.

t = 2 - 0.8416 = 1.158, \qquad \text{FPR} = 1 - \Phi(1.158) = 0.123, \qquad \text{precision} = \begin{cases}\dfrac{0.5(0.8)}{0.5(0.8) + 0.5(0.123)} = 0.866 & \pi = 0.5\\[2ex] \dfrac{0.01(0.8)}{0.01(0.8) + 0.99(0.123)} = 0.061 & \pi = 0.01\end{cases}

Monotone transforms don't move it (Card 4). On one simulated dataset the AUC is 0.9145 before and after $\exp(\cdot)$ and after a rank transform. Rounding the scores to integers creates ties: counting them as halves gives 0.896, while counting only strict wins drops it to 0.860.

Key move. R (AUC as P(D > 0)) and S (the variance of a difference adds).

Common slip. Forgetting that the variances add, and writing $\Phi(d) = \Phi(2) = 0.977$ instead of $\Phi(d/\sqrt{2})$ . The second slip is assuming every ROC is concave: with unequal variances the binormal ROC is improper near the top right.

How it's asked. "Legit scores $N(0, 1)$ , fraud scores $N(2, 1)$ . AUC in closed form? What happens under $\exp(\cdot)$ ? Precision at TPR 0.8 for $\pi = 0.5$ and 0.01?" (Worked problem 1.) Strong answer: 0.921, unchanged, and 0.866 vs 0.061. Then say that AUC 0.92 and precision 6% are both true of the same model, one conditioned on the truth and the other on the prediction.

Card 7 · Average precision and the PR baseline ★★

Statement. Average precision is the step-wise area under the PR curve. With no ties it equals the mean, over frauds, of the precision at each fraud's rank. A random ranker has expected precision $\pi$ at every cutoff, so the PR baseline is the base rate, not 0.5. You can't connect PR points with straight lines, because precision isn't linear between two operating points.

\text{AP} = \sum_k (R_k - R_{k-1})\,P_k = \frac{1}{n_1}\sum_{i=1}^{n_1} P@r_i = \frac{1}{n_1}\sum_{i=1}^{n_1}\frac{i}{r_i}

$\sum_k$: add up over every cutoff $k$ down the ranked list
$R_k$: recall within the top $k$ rows
$R_{k-1}$: recall within the top $k - 1$ rows, one row earlier
$P_k$: precision within the top $k$ rows
$P@r_i$: precision within the top $r_i$ rows
$r_i$: the rank of the $i$ -th fraud, counting from the top
$i$: counts frauds from the top: the $i$ -th fraud found

Derivation.

Sort by score, highest first, and walk down the list. Recall rises by $1/n_1$ at each fraud and is flat at each legit row, so only fraud positions contribute to the sum.

R_k - R_{k-1} = \begin{cases}1/n_1 & \text{row } k \text{ is a fraud}\\ 0 & \text{row } k \text{ is legit}\end{cases}

$k$: a position in the ranked list (1 = highest score)

At the $i$ -th fraud, found at rank $r_i$ , exactly $i$ of the top $r_i$ rows are fraud, so precision there is $i/r_i$ .

\text{AP} = \sum_{i=1}^{n_1}\frac{1}{n_1}\cdot P@r_i = \frac{1}{n_1}\sum_{i=1}^{n_1}\frac{i}{r_i}

Check on the toy. A's frauds sit at ranks 1, 3, 5, 8; B's at 1, 2, 3, 12.

\text{A: } \tfrac14\Big(\tfrac11 + \tfrac23 + \tfrac35 + \tfrac48\Big) = \tfrac{2.767}{4} = 0.6917, \qquad \text{B: } \tfrac14\Big(1 + 1 + 1 + \tfrac{4}{12}\Big) = 0.8333

The baseline. A random ranker's top $k$ is a uniformly random subset of size $k$ . Each row in it is fraud with probability $n_1/n$ , so the expected number of frauds is $k\pi$ and the expected precision is $\pi$ at every $k$ : a flat line at the base rate.

\mathbb E\big[\#\text{frauds in top } k\big] = k\,\frac{n_1}{n} \;\Rightarrow\; \mathbb E\big[P@k\big] = \frac{n_1}{n} = \pi

$P@k$: precision within the top $k$ rows

AP's exact expectation under random ranking. Write AP as a sum over rows $k$ of $y_k$ times the precision at $k$ . Then use $E[y_k] = \pi$ and, for two different rows, $E[y_k y_j] = n_1(n_1 - 1)/(n(n - 1))$ .

\text{AP} = \frac{1}{n_1}\sum_{k=1}^{n}\frac{y_k\sum_{j \le k}y_j}{k}, \qquad \mathbb E\Big[y_k\sum_{j\le k}y_j\Big] = \pi + (k - 1)\frac{n_1(n_1 - 1)}{n(n - 1)}

$\sum_{k=1}^{n}$: add up over every position in the ranked list
$y_k$: 1 if the row at position $k$ is fraud, else 0
$\sum_{j \le k}$: add up over the rows at or above position $k$
$\sum_{j\le k}$: add up over the rows at or above position $k$
$y_j$: 1 if the row at position $j$ is fraud, else 0

Sum over $k$ with $H_n = 1 + \tfrac12 + \dots + 1/n$ . The result is $\pi$ plus a small correction, about $(\ln n)/n$ , so at our sizes the random baseline is the base rate 0.0097. The correction matters only for tiny test sets like the toy, where the random AP is far above $\pi$ .

\mathbb E[\text{AP}] = \frac{1}{n_1}\Big[\pi H_n + \frac{n_1(n_1 - 1)}{n(n - 1)}(n - H_n)\Big] = \frac{n_1 - 1}{n - 1} + \frac{H_n}{n}\cdot\frac{n - n_1}{n - 1} \approx \pi + \frac{\ln n}{n}

$H_n$: the harmonic number $1 + \tfrac12 + \dots + \tfrac1n$

Read AP against that baseline. On the 100,000, A's AP of 0.133 is about 14 times random and B's 0.399 is about 41 times random, while their AUCs are 0.868 and 0.848. The PR view reverses the ROC verdict because B's lead is at the top of the list.

\frac{0.133}{0.00974} \approx 14, \qquad \frac{0.399}{0.00974} \approx 41

Why straight lines are wrong. Between two operating points you can reach any mixture by flagging a random fraction x of the extra rows. TP and FP are both linear in x, so recall is linear in x. Precision is a ratio of linear functions, so it isn't linear in recall. (On a ROC both axes are linear in x, which is why ROC interpolation is valid.)

TP(x) = TP_a + x\,\Delta TP, \quad FP(x) = FP_a + x\,\Delta FP \;\Rightarrow\; R(x) \text{ linear}, \quad P(x) = \frac{TP_a + x\,\Delta TP}{TP_a + FP_a + x(\Delta TP + \Delta FP)} \text{ not linear}

$x$: the fraction of the extra rows flagged, from 0 to 1
$TP(x)$: true positives when a fraction $x$ of the extra rows is flagged
$FP(x)$: false positives at that same mixture
$TP_a$: true positives at the first operating point
$FP_a$: false positives at the first operating point
$\Delta TP$: the extra true positives between the two operating points
$\Delta FP$: the extra false positives between the two operating points
$R(x)$: recall at the mixture $x$
$P(x)$: precision at the mixture $x$

Check on B's toy curve between the top 3 (TP 3, FP 0: R 0.75, P 1) and all 12 rows (TP 4, FP 8: R 1, P 0.333). Halfway ( $x = \tfrac12$ ), recall is 0.875 and precision is 3.5/7.5 = 0.467. The straight line would claim 0.667. Bridging a long run of false positives with a line overstates precision (Davis and Goadrich, 2006).

P\big(\tfrac12\big) = \frac{3 + 0.5}{3 + 0 + 0.5(1 + 8)} = \frac{3.5}{7.5} = 0.467 \quad\text{vs}\quad \frac{1 + 0.333}{2} = 0.667

$P\big(\tfrac12\big)$: precision halfway between the two points ( $x = \tfrac12$ )

The trapezoid in sklearn (auc(recall, precision)) is a different number with no probability behind it: 0.649 for A and 0.826 for B on the toy, against AP 0.692 and 0.833. Here it comes out lower, because it averages each new precision with the trough before it. Where it bridges long false-positive runs (step 9) it comes out higher. Report average_precision_score, the step-wise sum.

Key move. T: precision is a conditional rate, and its random-ranker value is the base rate.

Common slip. Drawing a 0.5 line as the "random" baseline on a PR plot (that's ROC's diagonal). The other slip is reporting the trapezoid area as AP, or interpolating PR points linearly.

How it's asked. "Sketch ROC and PR for model B on the toy. Why does PR zig-zag, why can't you connect the points with lines, and which AP does sklearn report?" (Q17, Q4.) Strong answer: each legit row drops precision without moving recall, which makes the sawtooth. Mixtures are linear in TP and FP, not in precision. sklearn reports the step-wise AP, and its random baseline is $\pi$ .

Card 8 · Proper scoring rules ★★

Statement. A scoring rule is proper if, when the truth is $\text{Bernoulli}(q)$ , your expected score is best when you report $p = q$ . It is strictly proper if $p = q$ is the only best report. Log loss and the Brier score are strictly proper. The absolute error $\lvert y - p\rvert$ is not: it rewards rounding to 0 or 1.

\arg\min_p\; \mathbb E_q\big[-y\ln p - (1 - y)\ln(1 - p)\big] = q, \qquad \arg\min_p\; \mathbb E_q\big[(y - p)^2\big] = q, \qquad \arg\min_p\; \mathbb E_q\,|y - p| \in \{0, 1\}

$\arg\min_p$: the report $p$ that makes what follows smallest
$\mathbb E_q$: the average over outcomes when fraud truly happens with probability $q$
$p$: the fraud probability you report
$q$: the true fraud probability

Derivation.

With y ~ Bernoulli(q), the expected score of a report p is a q-weighted average of the score when y = 1 and the score when y = 0.

S(p; q) = q\,s(p, 1) + (1 - q)\,s(p, 0)

$S(p; q)$: the expected score of reporting $p$ when the truth is $q$
$s(p, 1)$: the score for reporting $p$ when it is a fraud
$s(p, 0)$: the score for reporting $p$ when it is legit

Log loss. Differentiate the expected loss and set it to zero.

L(p) = -q\ln p - (1 - q)\ln(1 - p), \qquad L'(p) = -\frac{q}{p} + \frac{1 - q}{1 - p} = 0 \;\Rightarrow\; q(1 - p) = (1 - q)p \;\Rightarrow\; p = q

$L(p)$: the expected log loss of reporting $p$
$L'(p)$: how fast $L$ changes as you nudge $p$ : a slope, not a fraction

The second derivative is positive everywhere on (0, 1), so p = q is the unique minimum.

L''(p) = \frac{q}{p^2} + \frac{1 - q}{(1 - p)^2} > 0

$L''(p)$: the curvature: how fast that slope itself changes

The price of misreporting is the KL divergence from the truth to the report. It's zero only when p = q.

L(p) - L(q) = q\ln\frac{q}{p} + (1 - q)\ln\frac{1 - q}{1 - p} = \text{KL}(q\,\Vert\,p) \ge 0

$L(q)$: the expected log loss of reporting the truth $q$
$\text{KL}(q\,\Vert\,p)$: KL divergence: the extra log loss from reporting $p$ when the truth is $q$

Brier. Expand and complete the square (add-and-subtract $q^2$ ). The excess is $(p - q)^2$ , and $q(1 - q)$ is the part no forecaster can remove: the "luck" of Card 9.

B(p) = q(1 - p)^2 + (1 - q)p^2 = q - 2qp + p^2 = (p - q)^2 + q(1 - q)

$B(p)$: the expected Brier score of reporting $p$

Absolute error is linear in $p$ . A linear function is minimized at an end of the interval: report 0 whenever $q < \tfrac12$ . At 1% fraud, the absolute-error-optimal forecast is "never fraud" for every transaction.

A(p) = q(1 - p) + (1 - q)p = q + p(1 - 2q) \;\Rightarrow\; p^\star = \begin{cases}0 & q < \tfrac12\\ 1 & q > \tfrac12\end{cases}

$A(p)$: the expected absolute error of reporting $p$
$p^\star$: the best report (star = the optimum)

The score of knowing nothing. Reporting the base rate everywhere gives log loss equal to the entropy $H(\pi)$ and Brier equal to $\pi(1 - \pi)$ . At $\pi = 1\%$ these are 0.0560 nats and 0.0099; at the test base rate they are 0.0548 and 0.00965. That's why raw Brier numbers at 1% all look tiny, and why skill is reported against them (Card 9).

H(\pi) = -\pi\ln\pi - (1 - \pi)\ln(1 - \pi) = 0.0560, \qquad \pi(1 - \pi) = 0.0099 \qquad (\pi = 0.01)

$H(\pi)$: the entropy of the base rate: log loss of always reporting $\pi$

Two proper rules can still rank two imperfect forecasters differently, because they penalize errors differently. Log loss is unbounded as $p \to 0$ on a fraud; Brier is capped at 1. On the toy, log loss prefers A (0.523 vs 0.576) and Brier prefers B (0.181 vs 0.149). B's 0.02 on the $900 fraud T8 costs 3.912 nats alone; without T8, B's log loss would be 0.272.

-\ln(0.02) = 3.912 \;\;(\text{log loss on T8}), \qquad (1 - 0.02)^2 = 0.960 \;\;(\text{Brier on T8})

On the 100,000 both agree: oracle < B-cal < A < raw B. Log loss 0.0308 < 0.0362 < 0.0431 < 0.420, and Brier 0.00626 < 0.00679 < 0.00900 < 0.128.

Key move. E (minimize the expected score pointwise) and A (complete the square for Brier).

Common slip. Defining "proper" as "zero for a perfect model" (every reasonable score does that). The second slip is concluding that two proper scores must agree on which model is better; they agree only at the truth.

How it's asked. "On the toy, log loss says A, Brier says B. Both are proper. How can they disagree, and which do you trust for a fraud-loss forecast?" (Q18.) Strong answer: properness is about the minimizer, not the ranking of wrong forecasts. Log loss punishes one confident miss without limit, Brier caps it. For summing probabilities into a dollar forecast, look at calibration (Card 9) and at the confident misses on large frauds.

Card 9 · The Brier decomposition (Murphy) ★★★

Statement. Group forecasts into bins $k$ , with $n_k$ rows, forecast $f_k$ , observed fraud rate $\bar o_k$ and overall rate $\bar o$ . If forecasts are constant within each bin, the Brier score splits exactly into reliability minus resolution plus uncertainty: Lie − Learn + Luck.

\text{BS} = \underbrace{\sum_k \frac{n_k}{n}(f_k - \bar o_k)^2}_{\text{REL}} - \underbrace{\sum_k \frac{n_k}{n}(\bar o_k - \bar o)^2}_{\text{RES}} + \underbrace{\bar o(1 - \bar o)}_{\text{UNC}}

$\sum_k$: add up over every forecast bin $k$
$n_k$: the number of transactions in bin $k$
$n$: all transactions scored
$f_k$: the fraud probability forecast for bin $k$
$\bar o_k$: the observed fraud rate in bin $k$ (bar = average of outcomes)
$\bar o$: the overall observed fraud rate

Derivation.

Write the Brier score as a sum over bins of the squared misses within each bin.

\text{BS} = \frac1n\sum_k\sum_{i \in k}(f_k - y_i)^2

$\frac1n$: one over $n$ : turns the sum into an average
$\sum_{i \in k}$: add up over the transactions in bin $k$
$y_i$: transaction $i$ 's outcome: 1 if fraud, 0 if legit

Add and subtract the bin's observed rate $\bar o_k$ inside each square.

f_k - y_i = (f_k - \bar o_k) + (\bar o_k - y_i)

Square and sum within the bin. The cross term vanishes because the $y_i$ in bin $k$ average to $\bar o_k$ .

\sum_{i\in k}(f_k - y_i)^2 = n_k(f_k - \bar o_k)^2 + 2(f_k - \bar o_k)\underbrace{\sum_{i\in k}(\bar o_k - y_i)}_{=\,0} + \sum_{i\in k}(y_i - \bar o_k)^2

$\sum_{i\in k}$: add up over the transactions in bin $k$

For 0/1 outcomes, the within-bin sum of squares is $n_k \bar o_k(1 - \bar o_k)$ . So the Brier score is reliability plus the average within-bin variance.

\text{BS} = \sum_k\frac{n_k}{n}(f_k - \bar o_k)^2 + \sum_k\frac{n_k}{n}\,\bar o_k(1 - \bar o_k)

Split the total variance of $y$ with the same add-and-subtract, now around $\bar o$ . Total variance = within-bin + between-bin (the law of total variance), and the total variance of 0/1 outcomes is $\bar o(1 - \bar o)$ .

\bar o(1 - \bar o) = \frac1n\sum_k\sum_{i\in k}\big[(y_i - \bar o_k) + (\bar o_k - \bar o)\big]^2 = \sum_k\frac{n_k}{n}\bar o_k(1 - \bar o_k) + \sum_k\frac{n_k}{n}(\bar o_k - \bar o)^2

So the within-bin term is $\text{UNC} - \text{RES}$ . Substitute it into step 4 to get the statement. Resolution enters with a minus sign: the more your bins separate fraudulent from legit traffic, the lower the score.

\sum_k\frac{n_k}{n}\bar o_k(1 - \bar o_k) = \text{UNC} - \text{RES} \;\Rightarrow\; \text{BS} = \text{REL} - \text{RES} + \text{UNC}

$\text{UNC}$: uncertainty $\bar o(1 - \bar o)$ : the base rate's own noise, the "luck"
$\text{RES}$: resolution: how far bin rates spread from the overall rate, the "learn"
$\text{REL}$: reliability: the squared calibration gap, the "lie"

Check by hand (Drill 8). Five forecasts at 0.2 contain 1 fraud ( $\bar o_1 = 0.2$ ); five at 0.6 contain 4 ( $\bar o_2 = 0.8$ ); overall $\bar o = 0.5$ .

\text{REL} = \tfrac12(0)^2 + \tfrac12(0.6 - 0.8)^2 = 0.02, \quad \text{RES} = \tfrac12(0.3)^2 + \tfrac12(0.3)^2 = 0.09, \quad \text{UNC} = 0.25 \;\Rightarrow\; \text{BS} = 0.18

Directly: the first bin's squared misses are 0.64 + 4(0.04) = 0.80, the second bin's are 4(0.16) + 0.36 = 1.00, and 1.80/10 = 0.18.

Check on the 100,000 with 200 equal-mass bins. UNC is 0.00965 for everyone. A: REL 0.0000179, RES 0.000605, Brier 0.00900. Raw B: REL 0.122 and RES 0.00270, so B's problem is entirely the lie. B-cal: REL 0.0000100 and the same RES 0.00270 as raw B. Platt is monotone, so the rank-based bins and their observed rates are identical; only $f_k$ moves. The oracle's RES is 0.00327. So B learns about four and a half times as much as A, and recalibration removes its lie without touching what it learned.

\text{B-cal: } 0.0000100 - 0.00270 + 0.00965 \approx 0.00695 \quad\text{vs actual } 0.00679 \;\;(\text{residual } {-0.00016})

Why there's a residual. Real forecasts vary within a bin, and step 3's cross term is then not zero. The identity is exact only for binned-constant forecasts (A's forecasts replaced by their bin means satisfy it to rounding). The decomposition also depends on the bins: with 10 quantile bins at a 1% base rate, B-cal's measured RES collapses to 0.0003, because the top bin mixes the very risky with the merely risky.
Skill and ECE. The Brier skill score compares with the base-rate forecast, whose Brier is UNC. It is 0.067 for A and 0.296 for B-cal. ECE uses the absolute version of REL's gaps, and for the same bins $\text{ECE} \le \sqrt{\text{REL}}$ by Jensen's inequality.

\text{BSS} = 1 - \frac{\text{BS}}{\text{UNC}} \approx \frac{\text{RES} - \text{REL}}{\text{UNC}}, \qquad \text{ECE} = \sum_k w_k\,\lvert d_k\rvert \le \sqrt{\sum_k w_k d_k^2} = \sqrt{\text{REL}}

$\text{BSS}$: Brier skill score: the share of the base-rate forecast's Brier removed
$\text{ECE}$: expected calibration error: the average gap between forecast and observed rate
$w_k$: the share of rows in bin $k$ , $n_k/n$
$d_k$: bin $k$ 's calibration gap, observed rate minus forecast

Key move. A, twice: around $\bar o_k$ (step 2) and around $\bar o$ (step 5).

Common slip. Getting the sign of resolution wrong ("more resolution is worse"). Others: thinking a better model can reduce UNC (it's the base rate's noise), or treating ECE as the reliability term (ECE is unsquared and depends on the bins).

How it's asked. "The report says Brier 0.0090, close to the perfect 0. Decompose it and tell the reader what it means at a 1% base rate." (Q19, Drill 8.) Strong answer: almost all of 0.0090 is UNC = 0.0096, which nobody can remove. A's REL is tiny (it's calibrated), but its RES is small too, so its skill is only 0.07, while B-cal's is 0.30. Name the binning choice and the residual.

Card 10 · The cost-optimal threshold and the expected-cost curve ★

Statement. With calibrated probabilities, flag a transaction when the expected cost of passing it exceeds the expected cost of flagging it. That gives a threshold $c_\text{FP}/(c_\text{FP} + c_\text{FN})$ . When the cost of a miss is the fraud amount, the rule becomes amount-aware. Over a whole population, expected cost per transaction is a weighted sum of the miss rate and the FPR. Its minimum sits where the ROC slope equals the cost-weighted odds against fraud.

\text{flag} \iff p > \frac{c_{FP}}{c_{FP} + c_{FN}}, \qquad \mathbb E[\text{cost}(t)] = \pi\,c_{FN}\big(1 - \text{TPR}(t)\big) + (1 - \pi)\,c_{FP}\,\text{FPR}(t)

$p$: the calibrated fraud probability of this transaction
$c_{FP}$: cost of a false positive: blocking a legit customer (about $10)
$c_{FN}$: cost of a false negative: the missed fraud's amount
$\mathbb E$: expected value: the average cost per transaction

Derivation.

One transaction with calibrated fraud probability $p$ . Flagging costs $c_\text{FP}$ if it's legit, and passing costs $c_\text{FN}$ if it's fraud (a caught fraud and a passed legit cost 0 here).

\mathbb E[\text{cost} \mid \text{flag}] = (1 - p)\,c_{FP}, \qquad \mathbb E[\text{cost} \mid \text{pass}] = p\,c_{FN}

Flag when it's cheaper, and solve for p.

(1 - p)\,c_{FP} < p\,c_{FN} \iff c_{FP} < p\,(c_{FP} + c_{FN}) \iff p > t^\star = \frac{c_{FP}}{c_{FP} + c_{FN}}

$t^\star$: the cost-optimal threshold (star = the optimum)

With nonzero costs for correct decisions (a review fee $c_\text{TP}$ on caught fraud, $c_\text{TN}$ on passed legit), only the regrets matter.

t^\star = \frac{c_{FP} - c_{TN}}{(c_{FP} - c_{TN}) + (c_{FN} - c_{TP})}

$c_{TN}$: the cost of correctly passing a legit transaction
$c_{TP}$: the cost of catching a fraud, such as a review fee

Check on the 100,000: $c_\text{FP} =$ $10 and the mean fraud amount is $215.68, so $t^* = 0.0443$ . The empirical minima of the cost curves sit at 0.0648 for A and 0.0567 for B-cal. Cutting at $t^*$ costs $1,637 per 1,000 transactions for A against the curve's minimum of $1,629, and $1,231 for B-cal against $1,218: the curve is flat near its bottom.

t^\star = \frac{10}{10 + 215.68} = 0.0443

Amount-aware rule. A missed fraud costs its own amount, so $c_\text{FN}$ varies by row. Apply step 2 row by row. Small frauds are no longer worth a $10 friction cost, and large ones are flagged at lower probabilities. This rule costs $1,135 per 1,000 for A and $1,012 for B-cal, against $1,629 and $1,218 for the best single threshold.

\text{flag } i \iff p_i\cdot\text{amount}_i > (1 - p_i)\,c_{FP} \iff p_i > \frac{c_{FP}}{c_{FP} + \text{amount}_i}

$i$: which transaction we're deciding on
$p_i$: the calibrated fraud probability of transaction $i$

The population curve. A transaction ends up as a missed fraud with probability $\pi(1 - \text{TPR}(t))$ and as a blocked legit one with probability $(1 - \pi)\,\text{FPR}(t)$ . Multiply each by its cost.

\mathbb E[\text{cost}(t)] = P(Y = 1, s \le t)\,c_{FN} + P(Y = 0, s > t)\,c_{FP} = \pi\,c_{FN}\big(1 - \text{TPR}(t)\big) + (1 - \pi)\,c_{FP}\,\text{FPR}(t)

Minimize over t. Both rates fall as t rises, at rates given by the class densities.

\frac{d}{dt}: \;\; \pi\,c_{FN}\,f_1(t) - (1 - \pi)\,c_{FP}\,f_0(t) = 0 \;\Rightarrow\; \frac{f_1(t)}{f_0(t)} = \frac{(1 - \pi)\,c_{FP}}{\pi\,c_{FN}}

$\frac{d}{dt}$: how fast the expected cost changes as you nudge $t$ : a slope
$f_1(t)$: the density of fraud scores at $t$

Read it on the ROC. The ROC's slope is $d\text{TPR}/d\text{FPR} = f_1(t)/f_0(t)$ , so the optimum is where an iso-cost line of that slope touches the curve. Rarer fraud (smaller $\pi$ ) or cheaper false alarms move the optimum along the curve.

\frac{d\,\text{TPR}}{d\,\text{FPR}} = \frac{-f_1(t)}{-f_0(t)} = \frac{(1 - \pi)\,c_{FP}}{\pi\,c_{FN}}

$\frac{d\,\text{TPR}}{d\,\text{FPR}}$: the ROC's slope at this threshold

Close the loop. For a calibrated score, Bayes turns the likelihood ratio into posterior odds. Substituting step 7 gives back step 2: the population curve and the per-row rule agree.

\frac{p}{1 - p} = \frac{\pi f_1(t)}{(1 - \pi) f_0(t)} = \frac{\pi}{1 - \pi}\cdot\frac{(1 - \pi)\,c_{FP}}{\pi\,c_{FN}} = \frac{c_{FP}}{c_{FN}} \;\Rightarrow\; p = \frac{c_{FP}}{c_{FP} + c_{FN}}

On the toy, weighting matters as much as the rule. Unweighted, A's best cut (top 8, all frauds caught, 4 FP) costs $40. Counted at the day's proportions, each FP costs 49.5 × $10 = $495, so the same cut costs $1,980 and A's best becomes the top 1 at $1,310. B's top 3 costs $900 and wins.

\text{top } 8: \;\; 4 \times \$10 = \$40 \;\;(\text{sample}) \qquad 4 \times \$495 = \$1{,}980 \;\;(\text{day})

Key move. E: compare the two actions' expected costs, then differentiate the population version.

Common slip. Inverting the ratio ( $c_\text{FN}/(c_\text{FP} + c_\text{FN})$ ), or applying $t^*$ to uncalibrated scores. Raw class-weighted B has a mean probability of 0.313, so its 0.044 would flag almost everything; with an uncalibrated score, read the threshold off the cost curve instead. The last slip is computing costs on an enriched sample without weighting the negatives back.

How it's asked. "A blocked good customer costs about $10, a missed fraud averages $200. What threshold should the model use, and what must be true for that to work?" (Q6, Q27.) Strong answer: about 0.048, valid only if the probabilities are calibrated. Then upgrade to the amount-aware rule, and note that a review queue with fixed capacity replaces the threshold with "the top k".

Card 11 · The F1-optimal threshold is half the best F1 ★★★

Statement. For calibrated probabilities, the threshold that maximizes the population (expected) F1 is exactly half of that maximum F1 (Lipton, Elkan and Naryanaswamy, 2014). So the F1-optimal threshold is never above 0.5, and the weaker the model, the lower it sits.

t^\star = \frac{F_1^\star}{2}, \qquad F_1(t) = \frac{2\int_t^1 p\,dG(p)}{\int_t^1 dG(p) + \pi}

$t^\star$: the F1-optimal threshold
$F_1^\star$: the best F1 any threshold achieves
$\int_t^1$: add up over every score from $t$ to 1
$dG(p)$: the share of transactions with a score near $p$

Derivation.

Let G be the distribution of calibrated scores p, with density g, and flag p > t. Calibration means a row with score p is fraud with probability p. So the expected TP and FP per row are integrals over the flagged region.

\frac{\mathbb E[TP]}{n} = \int_t^1 p\,dG(p), \qquad \frac{\mathbb E[FP]}{n} = \int_t^1 (1 - p)\,dG(p), \qquad \frac{\mathbb E[TP + FN]}{n} = \pi

$\mathbb E$: expected value over the population of transactions
$dG$: the share of transactions with a score near $p$

Rewrite F1's denominator. 2TP + FP + FN = (TP + FP) + (TP + FN): the flagged count plus the fraud count. That gives F1 as a ratio N(t)/D(t) times 2.

F_1(t) = \frac{2\,TP}{(TP + FP) + (TP + FN)} = \frac{2N(t)}{D(t)}, \qquad N(t) = \int_t^1 p\,dG, \quad D(t) = \int_t^1 dG + \pi

$N(t)$: expected true positives per row: the fraud mass flagged above $t$
$D(t)$: the flagged share plus the base rate

Differentiate the limits. Raising t removes rows with score t from the flagged set.

N'(t) = -t\,g(t), \qquad D'(t) = -g(t)

$N'(t)$: how fast $N$ changes as you nudge $t$
$D'(t)$: how fast $D$ changes as you nudge $t$
$g(t)$: the density of scores at $t$

Apply the quotient rule.

F_1'(t) = 2\,\frac{N'D - ND'}{D^2} = 2\,\frac{-t\,g\,D + N\,g}{D^2} = \frac{2\,g(t)}{D(t)^2}\big[N(t) - t\,D(t)\big]

$F_1'(t)$: how fast F1 changes as you nudge the threshold $t$
$N'$: short for $N'(t)$
$D'$: short for $D'(t)$
$N$: short for $N(t)$
$D$: short for $D(t)$
$g$: short for $g(t)$ , the score density at $t$

Set the bracket to zero. At the optimum $t^* = N/D$ , and $\text{F1} = 2N/D$ , so $t^* = F_1^*/2$ .

N(t^\star) = t^\star D(t^\star) \;\Rightarrow\; t^\star = \frac{N(t^\star)}{D(t^\star)} = \frac{F_1(t^\star)}{2}

It is the unique maximum. The bracket $h(t) = N - tD$ has a negative derivative everywhere, so it is positive below $t^*$ (F1 rising) and negative above (F1 falling).

h'(t) = N'(t) - D(t) - t\,D'(t) = -t\,g - D + t\,g = -D(t) < 0

$h'(t)$: the slope of the bracket $h(t) = N(t) - t\,D(t)$

Check on a population of 4,000,000 transactions scored by the true probabilities: the maximum F1 is 0.516, reached at t = 0.258, and half of 0.516 is 0.258. On the 974-fraud test set, B-cal's empirical F1 peaks at 0.515 at t = 0.257. The oracle's empirical argmax lands at 0.337, because a finite-sample F1 curve is flat near its top and its argmax is noisy.

\frac{F_1^\star}{2} = \frac{0.516}{2} = 0.258 = t^\star

A weak model thresholds low. A's best F1 on the test set is 0.208, at t = 0.098, close to half of 0.208. A team that cuts A at 0.5 flags 24 transactions out of 100,000.
F1 as a hidden cost rule. Matching $t^*$ with Card 10's $c_\text{FP}/(c_\text{FP} + c_\text{FN})$ shows F1 behaves like a cost rule whose implied cost ratio depends on how good the model is. A better model implies cheaper misses, which is a strange property for a business metric.

\frac{c_{FP}}{c_{FP} + c_{FN}} = \frac{F_1^\star}{2} \;\Rightarrow\; \frac{c_{FN}}{c_{FP}} = \frac{2 - F_1^\star}{F_1^\star}

Key move. E: write the expected counts as integrals and set the derivative to zero.

Common slip. Defaulting to 0.5 for F1, or applying the rule to uncalibrated scores (raw B's empirical F1 peaks at 0.970 on its inflated scale). The other slip is quoting the identity for a finite test set's argmax, where it holds only approximately.

How it's asked. "A calibrated model's best achievable F1 is 0.52. At what threshold?" (Drill 7.) Strong answer: about 0.26, with the four-line quotient-rule derivation. Then say what F1 hides: it ignores TN and dollar costs, so with known costs use Card 10 instead.

Card 12 · Variance of a proportion and the sample size for a CI ★

Statement. A proportion over $n$ independent units has variance $p(1 - p)/n$ . For a 95% interval of half-width $h$ you need $n = z^2 p(1 - p)/h^2$ , where $n$ counts the units in the denominator of the metric: frauds for recall, legit for FPR, flagged rows for precision. "Your n is your frauds." For small $n$ or $p$ near 0 or 1, use the Wilson interval, which inverts the score test.

\text{Var}(\hat p) = \frac{p(1 - p)}{n}, \qquad n = \frac{z^2\,p(1 - p)}{h^2}

$\hat p$: the measured proportion, such as recall on the test frauds
$p$: the true proportion that $\hat p$ estimates
$n$: the units in the metric's denominator: frauds, for recall
$z$: the normal quantile for the confidence level (1.96 for 95%)
$h$: the half-width you want for the interval

Derivation.

Write the proportion as a mean of independent 0/1 outcomes, each with mean p.

\hat p = \frac1n\sum_{i=1}^{n}X_i, \qquad X_i \sim \text{Bernoulli}(p), \qquad \text{Var}(X_i) = \mathbb E[X_i^2] - p^2 = p - p^2 = p(1 - p)

$\mathbb E$: expected value: the long-run average
$\frac1n$: one over $n$ : turns the sum into an average
$\sum_{i=1}^{n}$: add up over every unit
$X_i$: 1 if unit $i$ is a hit (fraud $i$ caught), else 0

Independent terms add their variances, and the 1/n comes out squared.

\text{Var}(\hat p) = \frac{1}{n^2}\sum_{i=1}^{n}\text{Var}(X_i) = \frac{p(1 - p)}{n}

By the CLT $\hat p$ is close to normal, which gives the Wald interval.

\hat p \pm z\sqrt{\frac{\hat p(1 - \hat p)}{n}}, \qquad z = 1.96 \text{ for } 95\%

Fix the half-width $h$ and solve for $n$ . The worst case is $p = \tfrac12$ , which gives $n = z^2/(4h^2)$ .

h = z\sqrt{\frac{p(1 - p)}{n}} \;\Rightarrow\; n = \frac{z^2\,p(1 - p)}{h^2}

Check (Drill 10): recall near 0.8, half-width 0.02. You need 1,537 frauds, which is 153,700 transactions at 1% fraud.

n = \frac{1.96^2 \times 0.8 \times 0.2}{0.02^2} = \frac{0.6147}{0.0004} = 1{,}536.6 \;\to\; 1{,}537 \text{ frauds}

Wilson: instead of plugging $\hat p$ into the variance, keep the unknown $p$ and find every $p$ the score test would not reject.

|\hat p - p| \le z\sqrt{\frac{p(1 - p)}{n}} \iff (\hat p - p)^2 \le \frac{z^2}{n}\,p(1 - p)

Expand into a quadratic in p.

p^2\Big(1 + \frac{z^2}{n}\Big) - p\Big(2\hat p + \frac{z^2}{n}\Big) + \hat p^2 \le 0

The interval runs between the two roots. The center is pulled toward $\tfrac12$ , and the width never collapses to zero at $\hat p = 0$ .

p = \frac{\hat p + \frac{z^2}{2n} \pm z\sqrt{\frac{\hat p(1 - \hat p)}{n} + \frac{z^2}{4n^2}}}{1 + \frac{z^2}{n}}

Check on the toy: A's recall is 2/4. Wald gives [0.010, 0.990], almost the whole range. Wilson gives [0.150, 0.850]. Either way the toy can't tell A from B.

\text{Wilson: } \frac{0.5 + 0.480}{1.960} \pm \frac{1.96\sqrt{0.0625 + 0.0600}}{1.960} = 0.5 \pm 0.350 = [0.150,\; 0.850]

On the 100,000, A's recall at its cost threshold is 0.345 on 974 frauds, with Wilson CI [0.316, 0.375]. The width comes from the 974, not the 100,000. At the same recall, the binomial half-width is 0.092 at 100 frauds and 0.029 at 1,000. For an LLM benchmark of 500 questions at 70%, the SE is 0.0205 and the CI is ±0.040; ±1 point needs 8,068 questions.

\text{SE} = \sqrt{\frac{0.7 \times 0.3}{500}} = 0.0205, \qquad n_{\pm 0.01} = \frac{1.96^2 \times 0.21}{0.01^2} = 8{,}068

$n_{\pm 0.01}$: the number of questions needed for a ±1-point interval

Key move. S.

Common slip. Using all transactions as $n$ for recall, which makes the error bar about 10 times too narrow at 1% fraud. The second slip is using the Wald interval with 4 frauds, or at $\hat p = 0$ where it has zero width.

How it's asked. "Recall is about 0.8. How many frauds, and how many transactions, for $\pm 2$ points?" (Drill 10, Q21.) Strong answer: 1,537 frauds $\approx$ 154,000 transactions, and say which denominator each metric uses. Add that halving the half-width needs four times the frauds.

Card 13 · The variance of AUC: Hanley–McNeil and DeLong ★★★

Statement. An AUC is a two-sample U-statistic, and its variance comes from counting which pairs of pairs share a row. Hanley and McNeil (1982) close the formula by assuming exponential score distributions, which gives $Q_1$ and $Q_2$ as functions of $A$ . DeLong et al. (1988) estimate the same pieces from the data, and extend them to the covariance of two AUCs on the same rows, which is what a paired comparison needs.

\text{Var}(\hat A) = \frac{A(1 - A) + (n_1 - 1)(Q_1 - A^2) + (n_0 - 1)(Q_2 - A^2)}{n_1 n_0}, \qquad Q_1 = P(s^+ > s^-,\, s^{+\prime} > s^-), \quad Q_2 = P(s^+ > s^-,\, s^+ > s^{-\prime})

$\hat A$: the measured AUC (hat = estimated)
$A$: the true AUC
$Q_1$: the chance two random frauds both outscore one random legit
$Q_2$: the chance one random fraud outscores two random legit transactions
$s^{+\prime}$: the score of a second, different fraud
$s^{-\prime}$: the score of a second, different legit transaction

Derivation.

Write the AUC as an average of pair indicators (continuous scores, no ties). Each has mean A.

\hat A = \frac{1}{n_1 n_0}\sum_{i, j}\psi_{ij}, \qquad \psi_{ij} = \mathbf 1(s_i^+ > s_j^-), \qquad \mathbb E[\psi_{ij}] = A

$\sum_{i, j}$: add up over every (fraud $i$ , legit $j$ ) pair
$\psi_{ij}$: 1 if fraud $i$ outscores legit $j$ , else 0 (no ties here)

The variance of a double sum is the sum of all pairwise covariances.

\text{Var}(\hat A) = \frac{1}{(n_1 n_0)^2}\sum_{i, j}\sum_{k, l}\text{Cov}(\psi_{ij}, \psi_{kl})

$\sum_{k, l}$: add up over every second pair (fraud $k$ , legit $l$ )
$\psi_{kl}$: the pair indicator for fraud $k$ and legit $l$

Sort the (ij, kl) combinations into four cases by which rows they share. Rows that share nothing are independent, so only three cases survive.

Case	Shared rows	Count	$\text{Cov}(\psi_{ij}, \psi_{kl})$
Same pair	both	$n_1 n_0$	$A(1 - A)$
Two frauds, one legit	the legit	$n_1(n_1 - 1)n_0$	$Q_1 - A^2$
One fraud, two legit	the fraud	$n_1 n_0(n_0 - 1)$	$Q_2 - A^2$
Disjoint	none	the rest	0

\text{Var}(\psi_{ij}) = A - A^2, \qquad \text{Cov}(\psi_{ij}, \psi_{kj}) = \mathbb E[\psi_{ij}\psi_{kj}] - A^2 = Q_1 - A^2, \qquad \text{Cov}(\psi_{ij}, \psi_{il}) = Q_2 - A^2

$\text{Cov}$: covariance: how much two quantities wobble together
$\psi_{kj}$: another fraud $k$ against the same legit $j$
$\psi_{il}$: the same fraud $i$ against another legit $l$

Multiply each covariance by its count and divide by $(n_1 n_0)^2$ to get the statement.

\text{Var}(\hat A) = \frac{n_1 n_0\,A(1 - A) + n_1 n_0(n_1 - 1)(Q_1 - A^2) + n_1 n_0(n_0 - 1)(Q_2 - A^2)}{(n_1 n_0)^2}

$Q_1$ and $Q_2$ depend on the score distributions. Hanley–McNeil's model: legit scores $\text{Exponential}(\text{rate } \lambda)$ , fraud scores $\text{Exponential}(\text{rate } \mu)$ , $\mu < \lambda$ . First get $A$ , which fixes the ratio $r = \mu/\lambda$ .

A = P(s^+ > s^-) = \int_0^\infty \lambda e^{-\lambda x}\,e^{-\mu x}\,dx = \frac{\lambda}{\lambda + \mu} \;\Rightarrow\; r = \frac{\mu}{\lambda} = \frac{1 - A}{A}

$\int_0^\infty$: add up over every score from 0 upward
$\lambda$: the rate of the exponential legit-score distribution
$\mu$: the rate of the exponential fraud-score distribution (smaller, so higher scores)
$e$: Euler's number, 2.718…
$x$: a legit score value
$dx$: a tiny step in $x$
$r$: the rate ratio $\mu/\lambda$ , fixed by $A$

$Q_1$ : two frauds both above one legit. Condition on the legit score $x$ : each fraud clears it with probability $e^{-\mu x}$ , independently.

Q_1 = \mathbb E\big[e^{-2\mu s^-}\big] = \frac{\lambda}{\lambda + 2\mu} = \frac{1}{1 + 2r} = \frac{1}{1 + 2(1 - A)/A} = \frac{A}{2 - A}

$Q_2$ : one fraud above two legit. Condition on the fraud score $x$ : each legit is below it with probability $1 - e^{-\lambda x}$ .

Q_2 = \mathbb E\big[(1 - e^{-\lambda s^+})^2\big] = 1 - \frac{2\mu}{\mu + \lambda} + \frac{\mu}{\mu + 2\lambda} = 1 - \frac{2r}{1 + r} + \frac{r}{r + 2}

Simplify, using 1 + r = 1/A and r + 2 = (1 + A)/A.

Q_2 = \frac{(1 - r)(r + 2) + r(1 + r)}{(1 + r)(r + 2)} = \frac{2}{(1 + r)(r + 2)} = \frac{2}{\frac1A\cdot\frac{1 + A}{A}} = \frac{2A^2}{1 + A}

$\frac1A$: one over $A$

Check (Drill 11): $A = 0.9$ , 100 frauds, 9,900 legit. $Q_1 = 0.818$ , $Q_2 = 0.853$ , and the SE is 0.0207. With 1,000 frauds and 99,000 legit it's 0.0065, smaller by $\sqrt{10}$ . The legit count barely matters: 9,900 vs 99,000 legit at 100 frauds changes the SE in the fourth decimal.

\frac{0.09 + 99(0.8182 - 0.81) + 9{,}899(0.8526 - 0.81)}{100 \times 9{,}900} = \frac{0.09 + 0.810 + 422.0}{990{,}000} = 0.000427 \;\Rightarrow\; \text{SE} = 0.0207

Why: with $n_0 \gg n_1$ , the $Q_2$ term dominates and the variance is roughly $(Q_2 - A^2)/n_1$ . $Q_2 - A^2$ is the variance, across frauds, of "the share of legit transactions this fraud beats". Your n is your frauds.

\text{Var}(\hat A) \approx \frac{Q_2 - A^2}{n_1} + \frac{Q_1 - A^2}{n_0}, \qquad Q_2 - A^2 = \text{Var}_{s^+}\big(F_0(s^+)\big)

$\text{Var}_{s^+}$: the variance across randomly drawn frauds
$F_0(s^+)$: the share of legit scores below this fraud's score

DeLong replaces the model with data. For each fraud, $V_{10}(i)$ is the share of legit it beats; for each legit, $V_{01}(j)$ is the share of frauds above it. Both average to $\hat A$ . $\hat A - A$ is approximately the sum of their two centered means (its Hájek projection), so the variance is the two sample variances over their counts. This is step 10 with $Q_2 - A^2$ and $Q_1 - A^2$ estimated from the data.

V_{10}(i) = \frac{1}{n_0}\sum_j\psi_{ij}, \quad V_{01}(j) = \frac{1}{n_1}\sum_i\psi_{ij}, \qquad \widehat{\text{Var}}(\hat A) = \frac{S_{10}}{n_1} + \frac{S_{01}}{n_0}, \quad S_{10} = \widehat{\text{Var}}\big(V_{10}(i)\big)

$V_{10}(i)$: the share of legit transactions fraud $i$ beats
$V_{01}(j)$: the share of frauds scoring above legit $j$
$S_{10}$: the sample variance of the $V_{10}(i)$ across frauds
$S_{01}$: the sample variance of the $V_{01}(j)$ across legit rows
$\widehat{\text{Var}}$: variance estimated from the data

Paired comparison. Two models scored on the same rows have correlated components, and the covariance comes from the same projections. On the 100,000: $\text{SE}_A = 0.0055$ , $\text{SE}_B = 0.00745$ , covariance $1.40 \times 10^{-5}$ (correlation 0.34). The paired SE of the difference is 0.0076 against 0.0093 unpaired, so $z = 2.70$ and $p = 0.007$ . The paired bootstrap CI is [0.005, 0.035].

\text{SE}(\hat A_A - \hat A_B) = \sqrt{0.0055^2 + 0.00745^2 - 2(1.40\times10^{-5})} = 0.0076, \qquad z = \frac{0.0205}{0.0076} = 2.70

$\hat A_A$: model A's measured AUC
$\hat A_B$: model B's measured AUC, on the same rows
$z$: the paired test statistic: the AUC gap over its standard error
$\text{SE}$: standard error: the standard deviation of an estimate

Hanley–McNeil is a quick, often conservative approximation. It's exact only for its exponential model: simulation reproduces 0.0206 for exponential scores, but for binormal scores the true SD is about 25% smaller (0.0154). On the 100,000 it gives 0.0074 for A, against DeLong 0.0055 and bootstrap 0.0054. On the toy it gives 0.157 for A: four frauds can't separate anything.

Key move. R (count which pairs share a row) and S (variance of a sum of covariances).

Common slip. Taking $n = 100{,}000$ as the sample size. Others: comparing two AUCs as if independent ( $\sqrt{\text{SE}_A^2 + \text{SE}_B^2}$ ) instead of paired, or trusting Hanley–McNeil as exact.

How it's asked. "A is 0.868, B is 0.848 on the same 100,000 transactions, 974 frauds. Give me a CI for the difference and a test. Why paired?" (Q21, Drill 11.) Strong answer: DeLong z = 2.70, p = 0.007, CI about [0.005, 0.035]. Pairing removes the shared luck of which frauds landed in the test set. The error bar is set by the 974 frauds. Then add that a significant AUC gap says nothing about the top 1%, where B wins (Card 14).

Card 14 · McNemar's test ★★

Statement. To compare two classifiers' decisions on the same rows, only the rows where they disagree carry information. Under $H_0$ (equal error rates), the $b$ rows that only A gets right and the $c$ rows that only B gets right split like fair coin flips. The test statistic is $\chi^2_1$ in large samples, and an exact binomial test works for small ones.

b \mid (b + c) \sim \text{Bin}\big(b + c, \tfrac12\big), \qquad \chi^2 = \frac{(b - c)^2}{b + c}, \qquad \chi^2_{\text{cc}} = \frac{(|b - c| - 1)^2}{b + c}

$b$: rows (frauds) only model A gets right
$c$: rows (frauds) only model B gets right
$\chi^2$: McNemar's statistic, without the continuity correction
$\chi^2_{\text{cc}}$: McNemar's statistic with the continuity correction
$\text{Bin}$: binomial: the count of heads in that many fair coin flips

Derivation.

Tabulate paired outcomes on the same units: a = both right, b = only A right, c = only B right, d = both wrong.

\text{hit rate}_A = \frac{a + b}{n}, \qquad \text{hit rate}_B = \frac{a + c}{n}

$\text{hit rate}_A$: the share of rows model A gets right
$\text{hit rate}_B$: the share of rows model B gets right
$a$: rows both models get right
$n$: all rows compared

Subtract. The concordant counts a and d cancel, so they can't tell the models apart.

\text{hit rate}_A - \text{hit rate}_B = \frac{b - c}{n}

Under $H_0$ the two kinds of disagreement are equally likely. Given $m = b + c$ discordant rows, each is an "A only" with probability $\tfrac12$ , independently.

b \mid m \sim \text{Bin}\big(m, \tfrac12\big), \qquad \mathbb E[b \mid m] = \frac m2, \qquad \text{Var}(b \mid m) = \frac m4

$m$: the discordant rows, $b + c$
$\frac m2$: half the discordant rows: the expected $b$ under $H_0$
$\frac m4$: the binomial variance $m \cdot \tfrac12 \cdot \tfrac12$

Standardize $b$ and square. The continuity correction subtracts 1 from $\lvert b - c\rvert$ because $b$ moves in whole units.

z = \frac{b - m/2}{\sqrt{m/4}} = \frac{2b - m}{\sqrt m} = \frac{b - c}{\sqrt{b + c}}, \qquad \chi^2 = z^2 = \frac{(b - c)^2}{b + c} \;\sim\; \chi^2_1

$z$: $b$ standardized: its distance from $m/2$ in standard errors
$\sqrt m$: the square root of the discordant count
$\chi^2_1$: the chi-squared distribution with 1 degree of freedom

Check (Drill 13): at the same capacity, A alone catches 40 frauds and B alone catches 20. Uncorrected $\chi^2$ is 6.67 ( $p = 0.0098$ ), corrected is 6.02 ( $p = 0.014$ ), and the exact two-sided binomial $p$ is 0.0135.

\chi^2 = \frac{20^2}{60} = 6.67, \qquad \chi^2_{\text{cc}} = \frac{19^2}{60} = 6.02, \qquad p_{\text{exact}} = 2\,P\big(\text{Bin}(60, \tfrac12) \ge 40\big) = 0.0135

$p_{\text{exact}}$: the exact two-sided binomial p-value

On the 100,000, at the top-1% capacity, over the 974 frauds: both catch 95, A only 108, B only 330, neither 441. B is far better at capacity ( $\chi^2_\text{cc} = 111.5$ , $p \approx 5 \times 10^{-26}$ ), even though A has the higher AUC. The two tests ask different questions.

\chi^2_{\text{cc}} = \frac{(|108 - 330| - 1)^2}{108 + 330} = \frac{221^2}{438} = 111.5

Why not a two-proportion z-test? It treats the two hit rates as independent samples. They share the same frauds, and the shared 95 + 441 rows make the hit rates positively correlated. The unpaired variance is too large, so the unpaired test throws away power.

\text{Var}(\hat p_A - \hat p_B) = \text{Var}(\hat p_A) + \text{Var}(\hat p_B) - 2\,\text{Cov}(\hat p_A, \hat p_B), \qquad \text{Cov} > 0 \text{ when the models agree on most rows}

$\hat p_A$: model A's measured hit rate
$\hat p_B$: model B's measured hit rate, on the same frauds

Key move. S, on the conditional binomial of the discordant pairs.

Common slip. Including the concordant rows in the test. Others: running an unpaired two-proportion test on paired data, or using the $\chi^2$ approximation with fewer than about 25 discordant rows (use the exact binomial).

How it's asked. "At review capacity the two models agree on almost everything; on frauds, A alone catches 40 and B alone 20. Is A better?" (Q22.) Strong answer: $\chi^2_\text{cc} = 6.0$ , $p \approx 0.014$ , so yes at 5%. The test uses only the 60 disagreements, not the thousands of agreements. Add that it compares decisions at one operating point, while DeLong compares rankings over all thresholds.

Card 15 · A/B sample size: n ≈ 16σ²/δ² per arm ★

Statement. For a two-sided test at $\alpha = 0.05$ with 80% power, each arm needs $2(z_{1-\alpha/2} + z_{1-\beta})^2\sigma^2/\delta^2$ units, about $15.7\sigma^2/\delta^2$ , rounded to $16\sigma^2/\delta^2$ . Here $\sigma^2$ is the per-unit variance of the metric and $\delta$ the absolute difference to detect. Halving $\delta$ needs four times the units.

n = \frac{2\,(z_{1-\alpha/2} + z_{1-\beta})^2\,\sigma^2}{\delta^2} = \frac{15.70\,\sigma^2}{\delta^2} \approx \frac{16\,\sigma^2}{\delta^2} \;\text{ per arm}

$n$: the units needed in each arm
$z_{1-\alpha/2}$: the normal quantile for a two-sided test (1.96 at 5%)
$z_{1-\beta}$: the normal quantile for the power (0.84 for 80%)
$\sigma^2$: the per-unit variance of the metric
$\delta$: the smallest true difference you want to detect

Derivation.

Each arm has $n$ units with per-unit variance $\sigma^2$ . The difference of two independent means has the sum of their variances.

\hat\Delta = \bar Y_T - \bar Y_C, \qquad \text{Var}(\hat\Delta) = \frac{\sigma^2}{n} + \frac{\sigma^2}{n} = \frac{2\sigma^2}{n}, \qquad \text{SE} = \sigma\sqrt{2/n}

$\hat\Delta$: the measured difference, treatment mean minus control mean
$\bar Y_T$: the average outcome in the treatment arm (bar = mean)
$\bar Y_C$: the average outcome in the control arm
$\sigma$: the per-unit standard deviation of the metric

The test rejects when the z-statistic clears the critical value.

\text{reject } H_0 \iff \frac{|\hat\Delta|}{\text{SE}} > z_{1-\alpha/2}

$H_0$: the null hypothesis: no true difference between the arms

If the true difference is $\delta > 0$ , then $\hat\Delta/\text{SE}$ is approximately $N(\delta/\text{SE}, 1)$ . The opposite tail is negligible, so the power is one normal probability.

\text{power} \approx P\Big(\frac{\hat\Delta}{\text{SE}} > z_{1-\alpha/2}\Big) = \Phi\Big(\frac{\delta}{\text{SE}} - z_{1-\alpha/2}\Big)

Set the power to $1 - \beta$ , so the argument of $\Phi$ equals $z_{1-\beta}$ , and solve for $n$ .

\frac{\delta}{\sigma\sqrt{2/n}} = z_{1-\alpha/2} + z_{1-\beta} \;\Rightarrow\; n = \frac{2\,(z_{1-\alpha/2} + z_{1-\beta})^2\,\sigma^2}{\delta^2}

The constant at $\alpha = 0.05$ two-sided and 80% power: $z$ values 1.95996 and 0.84162.

2\,(1.95996 + 0.84162)^2 = 2 \times 2.80158^2 = 15.70

Check (Drill 14): approval rate 95%, detect a 0.2-point drop. A 0/1 metric has $\sigma^2 = p(1 - p) = 0.0475$ . The rule of thumb gives 190,000 per arm; the exact constant gives 186,411.

n \approx \frac{16 \times 0.0475}{0.002^2} = 190{,}000, \qquad \frac{15.70 \times 0.0475}{0.002^2} = 186{,}411

Turn it around for the smallest detectable effect at a given traffic. With 50,000 customers per arm, the MDE is 0.39 points.

\delta_{\min} = 2.80158\,\sigma\sqrt{2/n} = 2.80158 \times \sqrt{0.0475} \times \sqrt{2/50{,}000} = 0.00386

$\delta_{\min}$: the smallest effect the test can reliably detect (the MDE)

The square law: $n \propto \sigma^2/\delta^2$ . Halve $\delta$ and you need $4n$ . Any variance reduction (Card 18, or capping a heavy-tailed metric) cuts $n$ by the same factor.

Key move. S: the variance of a difference, then one normal quantile.

Common slip. Reading $16\sigma^2/\delta^2$ as the total sample (it's per arm), or plugging in $\sigma$ instead of $\sigma^2$ . Others: a one-sided $z$ , or computing $\sigma^2$ per transaction when the randomization unit is the customer.

How it's asked. "Approval rate is 95%. How many customers per arm to detect a 0.2-point change?" (Q30, Drill 14.) Strong answer: state the four conditions (per arm, two-sided 5%, 80% power, absolute $\delta$ ), then 190,000. Add that fraud dollars are heavy-tailed, so $\sigma^2$ explodes unless you cap or use a binary proxy. Labels must also mature before the test is read.

Card 16 · Peeking inflates the false-positive rate ★★★

Statement. If you recompute a cumulative $z$ -test at several looks and stop at the first $\lvert z\rvert > 1.96$ , the false-positive rate exceeds 5%, because each look is another chance to cross a fixed line. Under $H_0$ the $z$ -statistics at looks $k < l$ have correlation $\sqrt{n_k/n_l}$ . With two equally spaced looks the rate is exactly 0.0831. With continuous monitoring it goes to 1.

\text{Corr}(Z_k, Z_l) = \sqrt{\frac{n_k}{n_l}}, \qquad P(\text{reject at either of 2 equal looks}) = 1 - P\big(|Z_1| \le 1.96,\, |Z_2| \le 1.96\big) = 0.0831

$\text{Corr}$: correlation: how closely two quantities move together
$Z_k$: the cumulative z-statistic at look $k$
$Z_l$: the cumulative z-statistic at a later look $l$
$n_k$: the units collected by look $k$
$n_l$: the units collected by look $l$
$Z_1$: the z-statistic at the first look
$Z_2$: the z-statistic at the second look

Derivation.

Under $H_0$ the per-unit differences are i.i.d. with mean 0 and variance $\sigma^2$ . The cumulative sum after $n_k$ units gives the $z$ -statistic at look $k$ .

S_{n} = \sum_{u=1}^{n}X_u, \qquad Z_k = \frac{S_{n_k}}{\sigma\sqrt{n_k}} \sim N(0, 1)

$S_{n}$: the running sum of per-unit differences after $n$ units
$\sum_{u=1}^{n}$: add up over units $u = 1$ to $n$
$X_u$: the treatment-minus-control difference for unit $u$
$S_{n_k}$: the running sum at look $k$
$N(0, 1)$: the standard normal: mean 0, variance 1

Sums over overlapping windows share their first $n_k$ terms, and the later increments are independent of them. So the covariance is the variance of the shared part.

\text{Cov}(S_{n_k}, S_{n_l}) = \text{Cov}\Big(S_{n_k},\; S_{n_k} + \sum_{u = n_k + 1}^{n_l}X_u\Big) = \text{Var}(S_{n_k}) = n_k\sigma^2

$S_{n_l}$: the running sum at look $l$
$\sum_{u = n_k + 1}^{n_l}$: add up over the units after look $k$ , through look $l$

Divide by both standard deviations.

\text{Corr}(Z_k, Z_l) = \frac{n_k\sigma^2}{\sigma\sqrt{n_k}\cdot\sigma\sqrt{n_l}} = \sqrt{\frac{n_k}{n_l}}

Two equal looks: $\rho = \sqrt{1/2} = 0.707$ . Equivalently, $Z_2 = (Z_1 + Z_1')/\sqrt{2}$ , where $Z_1'$ is the independent second half.

Z_2 = \frac{S_{n} + (S_{2n} - S_n)}{\sigma\sqrt{2n}} = \frac{Z_1 + Z_1'}{\sqrt2}, \qquad \text{Corr}(Z_1, Z_2) = \frac{1}{\sqrt2}

$Z_1'$: the z-statistic of the independent second half alone
$S_{2n}$: the running sum after $2n$ units
$n$: the units per half

The false-positive rate is one minus the probability that both statistics stay inside the band. Evaluate the bivariate normal with $\rho = \sqrt{1/2}$ : 0.0831. A simulation gives 0.0832. Split it as 0.05 at the first look plus the 0.0331 of paths that were inside at look 1 and outside at look 2.

P(\text{reject}) = 1 - P\big(|Z_1| \le 1.96,\; |Z_2| \le 1.96\big) = 0.0831 = \underbrace{0.05}_{\text{look 1}} + \underbrace{0.0331}_{\text{inside, then out}}

More looks only add chances, so the rate never falls as looks are added. Simulated over 100,000 A/A runs (50,000 for 100 looks): 5 looks 0.140, 10 looks 0.195, 14 looks 0.221, 20 looks 0.246, 100 looks 0.373. No closed form exists beyond two looks; these come from the normal-increment simulation.

\{\text{reject with } K \text{ looks}\} \subseteq \{\text{reject with } K + 1 \text{ looks}\} \;\Rightarrow\; \text{FPR}_K \text{ non-decreasing in } K

$K$: the number of looks
$\subseteq$: is contained in: every rejection with $K$ looks is one with $K + 1$

The limit. The law of the iterated logarithm says the random walk's excursions grow like $\sqrt{2n \ln\ln n}$ , faster than $\sqrt{n}$ . So $Z_n = S_n/(\sigma\sqrt{n})$ exceeds any fixed $c$ infinitely often, and checking after every unit eventually rejects with probability 1.

\limsup_{n\to\infty}\frac{S_n}{\sigma\sqrt{2n\ln\ln n}} = 1 \;\text{a.s.} \;\Rightarrow\; \limsup_{n\to\infty} Z_n = \infty \;\Rightarrow\; P(\text{ever } |Z_n| > c) = 1

$\limsup_{n\to\infty}$: the level the sequence keeps coming back up to as $n$ grows
$Z_n$: the z-statistic after $n$ units
$n$: the units observed so far
$c$: any fixed boundary, such as 1.96

Fixes that keep early stopping. Raise the boundary so the whole procedure has 5%. Pocock uses one constant boundary, 2.413 for 5 looks, which gives 0.0499. O'Brien–Fleming uses $c_k = c\sqrt{K/k}$ with $c = 2.04$ : very strict early, close to 1.96 at the end. Its boundaries are 4.56, 3.23, 2.63, 2.28, 2.04, and its rate is 0.0493.

c_k^{\text{OBF}} = 2.04\sqrt{5/k}: \quad 4.56,\; 3.23,\; 2.63,\; 2.28,\; 2.04

$c_k^{\text{OBF}}$: the O'Brien–Fleming boundary at look $k$
$k$: which look: 1 to 5

Key move. S (covariance of overlapping sums) and T (the union of crossing events).

Common slip. Treating looks as independent and quoting $1 - 0.95^2 = 0.0975$ for two looks (too high: the looks are correlated), or $1 - 0.95^K$ in general. The opposite slip is believing a Bayesian posterior check is immune to peeking.

How it's asked. "The PM checks daily and stops at the first $p < 0.05$ ; on day 5 it's 0.03. What's the real false-positive rate?" (Q31, Worked problem 2, Drill 15.) Strong answer: derive $\sqrt{n_k/n_l}$ , compute two looks exactly (0.083), quote about 14% for 5 looks and 22% for 14. Fix it with a pre-registered horizon, group-sequential boundaries, alpha spending or always-valid $p$ -values. Add that stopping on a crossing also biases the effect estimate upward.

Card 17 · Bonferroni controls FWER; BH controls FDR ★★

Statement. With $m$ tests of which $m_0$ are true nulls, Bonferroni (reject $p_i \le \alpha/m$ ) keeps the probability of any false rejection, the family-wise error rate, at or below $\alpha$ , with no assumptions about dependence. Benjamini–Hochberg's step-up rule keeps the expected share of false rejections among rejections, the false discovery rate, at exactly $(m_0/m)q \le q$ for independent p-values.

\text{FWER} = P(V \ge 1) \le \alpha, \qquad \text{FDR} = \mathbb E\Big[\frac{V}{\max(R, 1)}\Big] = \frac{m_0}{m}\,q \le q

$\text{FWER}$: family-wise error rate: the chance of any false rejection
$V$: the number of false rejections
$\alpha$: the error rate allowed (0.05)
$\text{FDR}$: false discovery rate: the expected share of rejections that are false
$\mathbb E$: expected value: the average over repeated experiments
$R$: the total number of rejections
$m_0$: the number of true nulls: metrics where nothing really changed
$m$: the number of tests (metrics)
$q$: the false discovery rate BH targets

V is the number of false rejections and R the total number of rejections.

Derivation.

Bonferroni. A true null's p-value is uniform, so $P(p_i \le \alpha/m) = \alpha/m$ . The probability of a union is at most the sum of the probabilities.

\text{FWER} = P\Big(\bigcup_{i \in H_0}\{p_i \le \alpha/m\}\Big) \le \sum_{i\in H_0}P(p_i \le \alpha/m) = m_0\,\frac{\alpha}{m} \le \alpha

$\bigcup_{i \in H_0}$: at least one of these events, over the true nulls $i$
$H_0$: the set of true nulls
$p_i$: the p-value of test $i$
$\sum_{i\in H_0}$: add up over every true null $i$

Without correction, with 20 independent null metrics at 5%, the chance of at least one false alarm is 64%, and the expected number of false alarms is 1.

1 - 0.95^{20} = 0.642, \qquad \mathbb E[V] = 20 \times 0.05 = 1

BH. Sort the p-values $p_{(1)} \le \dots \le p_{(m)}$ , find the largest $i$ with $p_{(i)} \le iq/m$ , and reject the $i$ smallest. FDR adds up each null's contribution $1(\text{rejected})/R$ , split by the value of $R$ .

\text{FDR} = \sum_{j \in H_0}\mathbb E\Big[\frac{\mathbf 1(j \text{ rejected})}{R \vee 1}\Big] = \sum_{j\in H_0}\sum_{k=1}^{m}\frac1k\,P(j \text{ rejected},\, R = k)

$\sum_{j \in H_0}$: add up over every true null $j$
$\sum_{k=1}^{m}$: add up over every possible number $k$ of rejections
$R \vee 1$: the larger of $R$ and 1, so we never divide by 0
$j$: one true-null test
$k$: a possible number of rejections
$\frac1k$: one over $k$ : null $j$ 's share when $k$ tests are rejected

The key fact. If null $j$ is rejected and $R = k$ , its p-value is at most $kq/m$ . Moving $p_j$ anywhere below that cutoff (say to 0) leaves the same $k$ p-values under their cutoffs, so $R$ is still $k$ . So " $j$ rejected and $R = k$ " equals " $p_j \le kq/m$ " intersected with an event $C_k^{(j)}$ that depends only on the other $m - 1$ p-values: the procedure run with $p_j$ set to 0 rejects exactly $k$ .

\{j \text{ rejected},\; R = k\} = \Big\{p_j \le \frac{kq}{m}\Big\} \cap C_k^{(j)}

$p_j$: the p-value of null $j$
$C_k^{(j)}$: the event that BH, run with $p_j$ set to 0, rejects exactly $k$

Independence splits the probability, and uniformity gives the first factor.

P(j \text{ rejected},\, R = k) = P\Big(p_j \le \frac{kq}{m}\Big)\,P\big(C_k^{(j)}\big) = \frac{kq}{m}\,P\big(C_k^{(j)}\big)

The $k$ 's cancel. The events $C_1^{(j)}, \dots, C_m^{(j)}$ partition the outcomes of the other p-values (with $p_j = 0$ the procedure rejects at least one and at most $m$ ), so their probabilities sum to 1.

\sum_{k=1}^{m}\frac1k\cdot\frac{kq}{m}\,P\big(C_k^{(j)}\big) = \frac qm\sum_{k=1}^{m}P\big(C_k^{(j)}\big) = \frac qm

$\frac qm$: $q$ over $m$

Sum over the $m_0$ nulls.

\text{FDR} = m_0\cdot\frac{q}{m} = \frac{m_0}{m}\,q \le q

Check by hand (Drill 16). Sorted p-values 0.001, 0.004, 0.009, 0.012, 0.021, 0.04, 0.06, … with $m = 20$ and $q = \alpha = 0.05$ . Bonferroni's cutoff is 0.0025, so it rejects 1. BH's cutoffs are 0.0025, 0.005, 0.0075, 0.01, 0.0125, …. The first two p-values are under theirs and none after are, so BH rejects 2.

0.001 \le 0.0025 \;\checkmark, \quad 0.004 \le 0.005 \;\checkmark, \quad 0.009 > 0.0075, \quad 0.012 > 0.01, \quad 0.021 > 0.0125, \;\dots

Check by simulation: 20 metrics, 4 true effects (z shifted by 3), 16 nulls, 10,000 runs. FDR: none 0.163, Bonferroni 0.015, BH 0.040, matching (16/20)(0.05) = 0.04. FWER: 0.557, 0.037, 0.135. Power: 0.851, 0.491, 0.627. BH buys about a quarter more power than Bonferroni (0.627 vs 0.491) by letting a few false discoveries through.

Key move. T: the union bound for Bonferroni; total probability over R for BH.

Common slip. Saying BH controls FWER (it doesn't: 0.135 in the sim). Others: running BH step-down (stopping at the first p-value above its cutoff) instead of step-up (taking the largest i under its cutoff), or forgetting that segments and secondary metrics are tests too.

How it's asked. "Two of twenty metrics are significant, p = 0.004 and 0.012. The PM wants to report both as wins." (Q32, Drill 16.) Strong answer: neither survives Bonferroni (0.0025) or BH (0.0025 for the smallest, 0.005 for the second). Name what each controls and pre-register one primary metric. Add that BH's guarantee extends to positively dependent p-values (Benjamini–Yekutieli for arbitrary dependence).

Card 18 · CUPED ★★

Statement. Subtract from each unit's outcome the part predicted by a pre-experiment covariate $X$ . With $\theta = \text{Cov}(Y, X)/\text{Var}(X)$ , the variance falls by the factor $1 - \rho^2$ , where $\rho = \text{Corr}(Y, X)$ . The treatment-effect estimate stays unbiased for any fixed $\theta$ , because randomization balances a pre-treatment $X$ in expectation.

\hat\Delta_{\text{cv}} = (\bar Y_T - \bar Y_C) - \theta\,(\bar X_T - \bar X_C), \qquad \theta^\star = \frac{\text{Cov}(Y, X)}{\text{Var}(X)}, \qquad \text{Var}(\hat\Delta_{\text{cv}}) = (1 - \rho^2)\,\text{Var}(\hat\Delta)

$\hat\Delta_{\text{cv}}$: the CUPED-adjusted difference (cv = control variate)
$\theta$: the weight on the pre-experiment covariate
$\bar X_T$: the average pre-experiment covariate in treatment
$\bar X_C$: the average pre-experiment covariate in control
$\theta^\star$: the variance-minimizing weight (star = the optimum)
$Y$: the outcome metric, such as spend during the test
$X$: the pre-experiment covariate, such as last month's spend
$\rho$: the correlation between $Y$ and $X$

Derivation.

Unbiasedness first. X is measured before assignment, so random assignment makes its arm means equal in expectation.

\mathbb E[\bar X_T - \bar X_C] = 0 \;\Rightarrow\; \mathbb E[\hat\Delta_{\text{cv}}] = \mathbb E[\hat\Delta] - \theta\cdot 0 = \Delta \quad\text{for any fixed } \theta

$\Delta$: the true treatment effect

The per-unit variance of the adjusted outcome $Y - \theta X$ , expanded (toolkit S).

\text{Var}(Y - \theta X) = \text{Var}(Y) - 2\theta\,\text{Cov}(Y, X) + \theta^2\,\text{Var}(X)

Minimize over $\theta$ : a parabola, so set the derivative to zero.

\frac{d}{d\theta}: \; -2\,\text{Cov}(Y, X) + 2\theta\,\text{Var}(X) = 0 \;\Rightarrow\; \theta^\star = \frac{\text{Cov}(Y, X)}{\text{Var}(X)} = \rho\,\frac{\text{SD}(Y)}{\text{SD}(X)}

$\text{SD}$: standard deviation: the square root of the variance
$\frac{d}{d\theta}$: how fast the variance changes as you nudge $\theta$

Plug $\theta^*$ back in and write $\text{Cov}^2 = \rho^2\,\text{Var}(Y)\,\text{Var}(X)$ .

\text{Var}(Y - \theta^\star X) = \text{Var}(Y) - \frac{\text{Cov}(Y, X)^2}{\text{Var}(X)} = \text{Var}(Y)\,(1 - \rho^2)

Both arms shrink by the same factor, so $\text{Var}(\hat\Delta_\text{cv}) = 2\sigma^2(1 - \rho^2)/n$ . By Card 15, the required $n$ scales the same way.

n_{\text{cv}} = (1 - \rho^2)\,n

$n_{\text{cv}}$: the units needed per arm with CUPED
$n$: the units needed per arm without it

Check (Drill 17): $\rho = 0.7$ gives a factor 0.51, so 49% fewer customers. If $\text{SD}(Y) = 2\cdot\text{SD}(X)$ , $\theta = 0.7 \times 2 = 1.4$ .

1 - 0.7^2 = 0.51, \qquad \theta = 0.7 \times 2 = 1.4

Check by simulation: 100,000 customers per arm, lognormal spend, realized $\rho = 0.703$ , so theory says 0.505. The per-unit variance ratio is 0.510. The spread of 2,000 estimates gives $0.542 \pm 0.018$ (heavy tails make it noisy). Power rises from 0.39 to 0.65, and the mean estimate is off by −0.00004 against an MC SE of 0.00015, so it's unbiased.
Why $X$ must be pre-treatment. If treatment moves $X$ by $\tau_X$ , step 1 breaks and the estimate is biased by $-\theta\tau_X$ . Any covariate measured during the test is forbidden.

\mathbb E[\bar X_T - \bar X_C] = \tau_X \ne 0 \;\Rightarrow\; \mathbb E[\hat\Delta_{\text{cv}}] = \Delta - \theta\,\tau_X

$\tau_X$: how much the treatment itself shifts $X$

CUPED is Ch 1's regression adjustment: $\theta^*$ is the slope of $Y$ on $X$ , and $\hat\Delta_\text{cv}$ is the treatment coefficient in a regression of $Y$ on the treatment flag and centered $X$ . Estimating $\theta$ from the pooled data adds only an $O(1/n)$ bias.

Key move. A (the variance of $Y - \theta X$ , minimized like a least-squares slope) and S.

Common slip. Using a covariate measured during the experiment, or thinking CUPED changes the estimand (it changes only the noise). Other slips: setting $\theta = \rho$ (true only when $\text{SD}(Y) = \text{SD}(X)$ ), and claiming variance falls by $\rho$ instead of $\rho^2$ .

How it's asked. "Tests on spend take four weeks to reach power; leadership wants two. What can you do without peeking?" (Q33, Drill 17.) Strong answer: CUPED with pre-period spend, variance $\times (1 - \rho^2) \approx 0.51$ at $\rho = 0.7$ , so about half the sample. Add that capping heavy tails and stratified randomization stack with it.

Card 19 · PSI is a symmetrized KL divergence ★

Statement. The Population Stability Index compares this week's bin shares $a_i$ with the reference shares $e_i$ . It equals $\text{KL}(a\,\|\,e) + \text{KL}(e\,\|\,a)$ , the Jeffreys divergence: it is never negative and is zero only when the distributions match. For small shifts it is approximately a $\chi^2$ -type distance.

\text{PSI} = \sum_i (a_i - e_i)\ln\frac{a_i}{e_i} = \text{KL}(a\,\Vert\,e) + \text{KL}(e\,\Vert\,a) \approx \sum_i\frac{(a_i - e_i)^2}{e_i}

$\text{PSI}$: Population Stability Index: how far this week's mix drifted from the reference
$\sum_i$: add up over every bin $i$
$a_i$: this week's share of transactions in bin $i$
$e_i$: the reference (expected) share in bin $i$
$\text{KL}(a\,\Vert\,e)$: KL divergence of this week's shares from the reference
$\text{KL}(e\,\Vert\,a)$: KL divergence the other way round

Derivation.

Write the two KL divergences. Flip the log in the second to share a common $\ln(a_i/e_i)$ .

\text{KL}(a\,\Vert\,e) = \sum_i a_i\ln\frac{a_i}{e_i}, \qquad \text{KL}(e\,\Vert\,a) = \sum_i e_i\ln\frac{e_i}{a_i} = -\sum_i e_i\ln\frac{a_i}{e_i}

Add them.

\text{KL}(a\,\Vert\,e) + \text{KL}(e\,\Vert\,a) = \sum_i (a_i - e_i)\ln\frac{a_i}{e_i} = \text{PSI}

Every term is non-negative: $a_i - e_i$ and $\ln(a_i/e_i)$ always have the same sign. So $\text{PSI} \ge 0$ , with equality only if every $a_i = e_i$ .

a_i > e_i \Rightarrow \ln\frac{a_i}{e_i} > 0, \qquad a_i < e_i \Rightarrow \ln\frac{a_i}{e_i} < 0

Small shifts. Put $a_i = e_i + \delta_i$ and expand the log to first order.

(a_i - e_i)\ln\frac{a_i}{e_i} = \delta_i\ln\Big(1 + \frac{\delta_i}{e_i}\Big) \approx \delta_i\cdot\frac{\delta_i}{e_i} = \frac{\delta_i^2}{e_i}

$\delta_i$: the small shift in bin $i$ 's share, $a_i - e_i$

Check (Drill 19). Reference quartiles hold 25% each; this week holds 15%, 20%, 30%, 35%. PSI is 0.105, "watch". The $\chi^2$ approximation gives 0.100, and the two KLs are 0.0512 and 0.0538.

-0.10\ln 0.6 + (-0.05)\ln 0.8 + 0.05\ln 1.2 + 0.10\ln 1.4 = 0.0511 + 0.0112 + 0.0091 + 0.0336 = 0.105

A yardstick in standard deviations. For a unit normal shifted by $\delta$ , each KL is $\delta^2/2$ , so the continuous Jeffreys divergence is $\delta^2$ .

\text{KL}\big(N(\delta, 1)\,\Vert\,N(0, 1)\big) = \mathbb E_{x\sim N(\delta,1)}\Big[\frac{x^2 - (x - \delta)^2}{2}\Big] = \mathbb E\Big[\frac{2x\delta - \delta^2}{2}\Big] = \frac{2\delta^2 - \delta^2}{2} = \frac{\delta^2}{2}

$N(\delta, 1)$: a normal shifted by $\delta$ standard deviations
$\delta$: the shift of the mean, in standard deviations
$\mathbb E_{x\sim N(\delta,1)}$: the average over $x$ drawn from the shifted normal
$x$: a value drawn from the shifted normal
$\mathbb E$: the same average

Binning can only lose information (the data-processing inequality), so a decile PSI is at most $\delta^2$ . In the monitoring sim, x2's mean drifts to +0.5 SD by week 8 and its decile PSI is 0.231, just under $\delta^2 = 0.25$ . On this scale, the "watch" line at 0.1 is about a third of an SD and the "act" line at 0.25 is about half an SD. A's score PSI at week 8 is only 0.095, because x2 is one of three inputs. The week-10 bug that zeroes x3 for 30% of rows gives an input PSI of 0.44 (act) against a score PSI of 0.10.

\text{PSI}_{\text{deciles}} \le \delta^2 = 0.5^2 = 0.25 \qquad (\text{sim: } 0.231)

Empty bins make $\ln(a_i/e_i)$ infinite, so implementations add a small $\varepsilon$ to every share. PSI is a size, not a test. At production volumes a KS test calls every shift significant, so alert on the size.

Key move. A: add the two KL divergences so the logs share one ratio.

Common slip. Treating PSI as a hypothesis test with a p-value. Others: binning on this week's quantiles instead of the reference's, or reading the 0.1 and 0.25 thresholds as theory rather than credit-industry convention.

How it's asked. "Reference quartiles at 25% each; this week 15, 20, 30, 35. PSI, and do we act?" (Q36, Drill 19.) Strong answer: 0.105, watch. Name it as symmetric KL, give the $\approx \sum (a - e)^2/e$ shortcut, and say what you'd check next: is the score drifting too, did a pipeline change land, do the proxy labels move?

Card 20 · Unbiased pass@k ★★

Statement. Generate $n$ samples for a problem and count $c$ correct. Then $1 - C(n - c, k)/C(n, k)$ , the probability that $k$ samples drawn without replacement from those $n$ are all wrong, subtracted from 1, is an unbiased estimator of $\text{pass@}k = 1 - (1 - p)^k$ (Chen et al., 2021). The plug-in $1 - (1 - c/n)^k$ is biased low, by Jensen's inequality.

\widehat{\text{pass@}k} = 1 - \frac{\binom{n - c}{k}}{\binom{n}{k}}, \qquad \mathbb E\big[\widehat{\text{pass@}k}\big] = 1 - (1 - p)^k

$\widehat{\text{pass@}k}$: the unbiased estimate of pass@k from one problem's samples
$n$: the number of samples generated for the problem
$c$: how many of those samples are correct
$k$: the number of attempts pass@k allows
$\binom{n - c}{k}$: the ways to pick $k$ samples that are all wrong
$\binom{n}{k}$: the ways to pick any $k$ of the $n$ samples
$p$: the chance one sample is correct
$\mathbb E$: the average over random draws of the $n$ samples

Derivation.

The target. If each sample is correct with probability $p$ , independently, then $k$ samples all fail with probability $(1 - p)^k$ .

\text{pass@}k = 1 - (1 - p)^k

$\text{pass@}k$: the chance at least one of $k$ samples is correct

Among the $n$ generated samples, there are $C(n, k)$ subsets of size $k$ , and $C(n - c, k)$ of them contain only wrong samples. So the estimator's second term is the share of $k$ -subsets that are all wrong: an average of indicators over subsets.

\frac{\binom{n - c}{k}}{\binom{n}{k}} = \frac{1}{\binom{n}{k}}\sum_{\lvert S\rvert = k}\mathbf 1(\text{every sample in } S \text{ is wrong})

$\sum_{\lvert S\rvert = k}$: add up over every subset $S$ of $k$ samples
$S$: one subset of $k$ samples

Take expectations over the $n$ samples. Each fixed subset $S$ holds $k$ independent samples, so its indicator has expectation $(1 - p)^k$ . The average of $C(n, k)$ terms with the same expectation has that expectation.

\mathbb E\Big[\frac{\binom{n - c}{k}}{\binom{n}{k}}\Big] = \frac{1}{\binom{n}{k}}\sum_{\lvert S\rvert = k}(1 - p)^k = (1 - p)^k \;\Rightarrow\; \mathbb E\big[\widehat{\text{pass@}k}\big] = 1 - (1 - p)^k

Check (Drill 20): $n = 10$ , $c = 3$ , $k = 5$ . $C(7, 5) = 21$ and $C(10, 5) = 252$ , so pass@5 $= 0.917$ . For $k = 1$ it is $1 - 7/10 = 0.3$ , which is just $c/n$ .

1 - \frac{21}{252} = 1 - 0.0833 = 0.917

A numerically stable form for large n. Cancel the factorials, leaving a product of c terms.

\frac{\binom{n - c}{k}}{\binom{n}{k}} = \frac{(n - c)!\,(n - k)!}{n!\,(n - c - k)!} = \prod_{j = n - c + 1}^{n}\frac{j - k}{j} = \prod_{j = n - c + 1}^{n}\Big(1 - \frac kj\Big), \qquad \text{toy: } \tfrac38\cdot\tfrac49\cdot\tfrac{5}{10} = 0.0833

$\prod_{j = n - c + 1}^{n}$: multiply the terms for $j$ from $n - c + 1$ to $n$
$j$: the running index of the product
$\frac kj$: $k$ over $j$

The plug-in replaces p by c/n inside a concave function of p.

f(x) = 1 - (1 - x)^k, \qquad f''(x) = -k(k - 1)(1 - x)^{k - 2} < 0 \quad (k \ge 2)

$f(x)$: pass@k as a function of single-sample accuracy $x$
$f''(x)$: its curvature; negative means concave
$x$: a single-sample accuracy between 0 and 1

Jensen's inequality for a concave f: the average of f is at most f of the average. So the plug-in is biased low in expectation.

\mathbb E\big[f(c/n)\big] \le f\big(\mathbb E[c/n]\big) = f(p) = \text{pass@}k

$f$: the same function $1 - (1 - x)^k$

Check by simulation at $p = 0.3$ , $n = 10$ , $k = 5$ : the true pass@5 is 0.832. The unbiased estimator averages 0.832; the plug-in averages 0.760. (For this one problem the plug-in's value $1 - 0.7^5 = 0.832$ happens to equal the truth at $p = 0.3$ , which is why the bias claim is about averages.) Over a 50-problem benchmark with varied difficulty, the truth is 0.496, the unbiased estimator 0.496, and the plug-in 0.455.
The same counting gives $\text{pass}^k$ , the chance that all $k$ samples are correct, a reliability metric. Its estimator $C(c, k)/C(n, k)$ is unbiased for $p^k$ . With $c = 3 < k = 5$ it is 0.

\mathbb E\Big[\frac{\binom{c}{k}}{\binom{n}{k}}\Big] = p^k

$\binom{c}{k}$: the ways to pick $k$ samples that are all correct

Key move. R (count subsets, average indicators) and T (expectation term by term).

Common slip. Reporting the plug-in, or drawing the k samples with replacement. Others: computing C(n, k) directly for large n (overflow; use the product), and forgetting that the benchmark's error bar comes from the number of problems.

How it's asked. " $n = 10$ generations per problem, 3 pass. Unbiased pass@5, and why not $1 - 0.7^5$ ?" (Q39, Drill 20.) Strong answer: 0.917, derived as a share of all-wrong subsets. The plug-in is biased low in expectation because $1 - (1 - x)^k$ is concave. Add the SE over problems: your n is your questions.

Practice protocol

Read only the Statement. Cover the rest of the card.
Derive on a blank page against a timer: 5 minutes for ★, 10 for ★★, 15 for ★★★. Talk out loud as if an interviewer were listening; the narration is half the grade.
Land the check number. Every card has one: 25/32, 0.0198, 0.9214, 0.18, 0.258, 1,537, 0.0207, 6.02, 186,411, 0.0831, 1 and 2 rejections, 0.51, 0.105, 0.917. If your derivation doesn't reproduce it, the error is in your algebra, not the example.
Compare step by step. Mark the first step you missed or got wrong, and name the toolkit move it needed (R, A, T, E or S). Read the Common slip and check whether you made it.
Answer "How it's asked" cold, including the strong-answer extension. This is where senior and research rounds go.
Re-derive after 48 hours, then after a week, without looking. A card is mastered when you can do it twice in a row within time and hit the check.
Run the chains. Several cards are one argument continued; practice each chain in one sitting:

The base-rate chain: Card 1 → 4 → 7 (precision is a posterior → what travels and what stays home → why the PR baseline is $\pi$ ).
The pair chain: Card 3 → 5 → 6 → 13 (AUC as pairs → hard and random scores → the binormal closed form → the variance of the pair average).
The probability chain: Card 8 → 9 → 10 → 11 (proper scores → Lie − Learn + Luck → the cost threshold → the F1 threshold).
The error-bar chain: Card 12 → 13 → 14 (a proportion → a $U$ -statistic → discordant pairs).
The experiment chain: Card 15 → 16 → 17 → 18 (sample size → peeking → many metrics → CUPED).
The production chain: Card 19 → 20 (drift as KL → LLM metrics as counting).

Suggested order: Cards 1–8 after the chapter's second read, then Interview Bank Levels 1–2, then Cards 9–20, then Levels 3–4.

Mastery checklist

[ ] Card 1 · $\text{Precision} = \pi\text{TPR}/(\pi\text{TPR} + (1 - \pi)\text{FPR})$ by total probability; odds form prior odds $\times\ \text{LR}^+$ ; toy 0.50 → 0.0198; vendor 0.941 → 0.139; test-set check 0.134.
[ ] Card 2 · $\text{F1} = 2\text{TP}/(2\text{TP} + \text{FP} + \text{FN})$ from reciprocals; $F_\beta$ puts $\beta^2$ on FN; MCC as the 0/1 Pearson correlation (numerator $\text{TP}\cdot\text{TN} - \text{FP}\cdot\text{FN}$ ); toy 0.5/0.25 and 0.75/0.625; day-weighted F1 0.0381.
[ ] Card 3 · Staircase area = pair share; $\int \text{TPR}\,d\text{FPR} = P(s^+ > s^-)$ ; $R_1 - n_1(n_1 + 1)/2 = U$ ; toy 25/32 and 24/32 both ways; null variance $(n + 1)/(12 n_1 n_0)$ .
[ ] Card 4 · Strictly increasing $g$ keeps every pair and every top- $k$ set; class weights cancel; squared toy F1@0.5 $= 0.333$ , max-F1 0.667 both; isotonic ties: AUC −0.002, AP −0.025.
[ ] Card 5 · Four pair cases → $(1 + \text{TPR} - \text{FPR})/2$ ; toy B 0.8125 vs 0.75; predict() gives 0.768 (B) and 0.510 (A) on the 100,000; random 0.5; $\text{AUC}(-s) = 1 - \text{AUC}$ .
[ ] Card 6 · $s^+ - s^- \sim N(d, 2) \to \Phi(d/\sqrt{2}) = 0.9214$ (sim 0.9212); probit-linear ROC, $\Phi(a/\sqrt{1 + b^2})$ ; unequal variance 0.814, crossing at FPR 0.977; precision 0.866 vs 0.061.
[ ] Card 7 · $\text{AP} =$ mean of $i/r_i$ ; toy 0.6917 and 0.8333; random precision $= \pi$ at every $k$ ; $E[\text{AP}] \approx \pi + (\ln n)/n$ ; PR mixtures nonlinear (0.467 vs 0.667); trapezoid $\ne$ AP.
[ ] Card 8 · Log loss minimized at $p = q$ (regret = KL); $\text{Brier} = (p - q)^2 + q(1 - q)$ ; absolute error improper; $H(1\%) = 0.0560$ , $\pi(1 - \pi) = 0.0099$ ; toy disagreement from T8's 3.912 nats.
[ ] Card 9 · Add-and-subtract around $\bar o_k$ , then around $\bar o$ ; Drill 8: $0.02 - 0.09 + 0.25 = 0.18$ ; B and B-cal share RES 0.00270; binning residual; BSS 0.067 vs 0.296; $\text{ECE} \le \sqrt{\text{REL}}$ .
[ ] Card 10 · $(1 - p)c_\text{FP} < p\cdot c_\text{FN} \to t^* = c_\text{FP}/(c_\text{FP} + c_\text{FN}) = 0.0443$ ; amount-aware rule ($1,135 vs $1,629 for A); ROC slope $= (1 - \pi)c_\text{FP}/(\pi c_\text{FN})$ recovers $t^*$ ; toy $40 vs $1,980.
[ ] Card 11 · $\text{F1}(t) = 2N/D$ with $N' = -tg$ , $D' = -g$ ; bracket $N - tD \to t^* = F_1^*/2$ ; unique since $h' = -D$ ; 0.516 at 0.258; implied cost ratio $(2 - F_1^*)/F_1^*$ .
[ ] Card 12 · $p(1 - p)/n$ ; $n = z^2 p(1 - p)/h^2 =$ 1,537 frauds; Wilson from inverting the score test; toy [0.150, 0.850] vs Wald [0.010, 0.990]; 500-question SE 0.0205.
[ ] Card 13 · Three covariance cases with counts; exponential model gives $Q_1 = A/(2 - A)$ , $Q_2 = 2A^2/(1 + A)$ ; SE 0.0207 and 0.0065; DeLong components; paired SE 0.0076, z 2.70; HM conservative (0.0074 vs 0.0055).
[ ] Card 14 · Concordant rows cancel; $b \mid m \sim \text{Bin}(m, \tfrac12) \to (b - c)^2/(b + c)$ ; Drill 13: 6.02, p 0.014, exact 0.0135; 100,000 at capacity: 108 vs 330, $\chi^2$ 111.5.
[ ] Card 15 · $\text{SE} = \sigma\sqrt{2/n}$ ; power $\Phi(\delta/\text{SE} - z)$ ; $n = 2(z_{1-\alpha/2} + z_{1-\beta})^2\sigma^2/\delta^2$ , constant 15.70; 190,000 (exact 186,411); MDE 0.0039 at 50,000.
[ ] Card 16 · $\text{Cov}(S_{n_k}, S_{n_l}) = n_k\sigma^2 \to \sqrt{n_k/n_l}$ ; two looks $0.0831 = 0.05 + 0.0331$ ; 5/10/14/20/100 looks 0.140/0.195/0.221/0.246/0.373; LIL → 1; Pocock 2.413, OBF $2.04\sqrt{5/k}$ .
[ ] Card 17 · Union bound → $\text{FWER} \le m_0\alpha/m$ ; BH proof via $C_k^{(j)}$ and the $k$ 's cancelling → $\text{FDR} = m_0 q/m$ ; Drill 16: 1 vs 2 rejections; sim FDR 0.040, FWER 0.135 for BH.
[ ] Card 18 · Unbiased for any $\theta$ (pre-treatment $X$ ); minimize the parabola → $\theta^* = \text{Cov}/\text{Var}$ ; $\text{Var} \times (1 - \rho^2)$ ; $\rho = 0.7$ → 0.51, $\theta = 1.4$ ; sim 0.510, power 0.39 → 0.65; post-treatment bias $-\theta\tau_X$ .
[ ] Card 19 · $\text{PSI} = \text{KL}(a\,\|\,e) + \text{KL}(e\,\|\,a) \ge 0$ ; $\approx \sum (a - e)^2/e$ ; Drill 19: 0.105 ( $\chi^2$ 0.100); normal shift gives $\delta^2$ ; x2 week 8: $0.231 \le 0.25$ ; bug: input 0.44, score 0.10.
[ ] Card 20 · Share of all-wrong $k$ -subsets, expectation $(1 - p)^k$ → unbiased; $1 - 21/252 = 0.917$ ; product form; plug-in biased low by Jensen (0.760 vs 0.832 on average); $\text{pass}^k = C(c, k)/C(n, k)$ .