Chapter 3 — Interview Bank

Every answer here follows the same four beats. Say the first two within 20 seconds; the last two are what turn a correct answer into a strong one.

Questions are stacked by difficulty. Each level assumes everything above it.

How questions are written here. Real interviewers rarely ask "define X". They describe a situation — a model, a chart, a stakeholder's claim — and watch whether you spot the concept underneath. So every question below opens with a scenario, then names the concept being tested, then gives the four beats. Practice by reading only the scenario and answering out loud before you look.

Every number comes from the chapter's running example: the ten files (Applicant K has score 680, so x = 0), and the 20,000-applicant book with known truth \operatorname{logit} p = -2.6 - 1.5z + 0.6d - 0.5\ell, where z = (\text{score} - 680)/100, d = (\text{DTI} - 36)/10 and \ell = \ln(\text{income}/75). Variants of the book (NL, CC, RI, LN, NC, CAL, SHIFT, MC, ORD, POIS, LDA, L1) are the ones the chapter introduces.

Level 1 — Screen

Q1 · "Just reuse the price model"

A lending PM wants to predict default using the team's existing linear-regression pipeline, with default coded 0/1. On the ten review files it predicts 0.915 for Applicant A and −0.106 for Applicant J. The PM says: "Close enough — just clip at 0 and 1." What's wrong with the model, and what do you use instead?

• Concept tested: why the linear probability model (LPM) fails 0/1 labels.
• Intuition: a ruler keeps going past the edge of the page; probabilities have to stop at 0 and 1.
• Answer: three failures, B-S-S. Bounds: the line leaves [0, 1] (J at −0.106). Spread: a 0/1 outcome has variance p(1 - p), 0.25 at p = 0.5 but 0.09 at p = 0.1, so the equal-variance assumption fails by construction and the classical SEs are wrong. Shape: real risk is S-shaped and flattens near 0 and 1, which a line can't do. Clipping fixes only the first. Use logistic regression: a line for the log-odds, bent into a probability by the sigmoid.

\hat p

the predicted probability of default (hat = estimated from the ten files)

x

credit score in hundreds of points above 680 (Applicant K has x = 0)

\sigma

the sigmoid: bends any score into a probability between 0 and 1

e

Euler's number, about 2.718

• Why: squared error on 0/1 labels treats "too right" as an error, so rows far from the boundary pull the line around, and nothing in the model knows a probability must stop at 0 or 1 (Section 1).
• Next they'll ask: "Does anyone still use the LPM?" → Yes, economists, for average effects of a randomized treatment when fitted values stay in about 0.2–0.8, always with robust SEs (Ch 1 Q8). Never for individual probabilities.

Q2 · "What does 0.30 mean?"

An underwriter sees three numbers next to Applicant K in the model output: −0.824, 0.439 and 0.305. She asks: "Which of these is the probability of default, what are the other two, and why does the model need all three?"

• Concept tested: the P-O-L ladder — probability, odds, log-odds.
• Intuition: the same risk written on three rulers: one that stops at 0 and 1, one that stops at 0, and one that never stops.
• Answer: 0.305 is the probability. 0.439 is the odds, p/(1 - p): about 4 defaults for every 9 repayments. −0.824 is the log-odds, \ln(\text{odds}), the number the model's line actually predicts.

p

Applicant K's probability of default

\xrightarrow{\;p/(1-p)\;}

turn the probability into odds: divide it by its complement

\xrightarrow{\;\ln\;}

take the natural log: odds become log-odds

• Why: log-odds run from -\infty to +\infty, so a line can predict them without ever producing an impossible value; the sigmoid walks back down to a probability (Card 1).
• Next they'll ask: "Is 30.5% K's real risk?" → No. The ten files were sampled with every default kept, so this probability is inflated; after the case-control correction it is 5.6% (Q10, Section 8).

Q3 · Odds ratio of 0.063

A compliance analyst drafts a memo from the ten-file model: "The credit-score coefficient is −2.77 per 100 points, and e^{-2.77} = 0.063, so an applicant 100 points higher has a 6% chance of default." Correct the memo.

• Concept tested: exponentiated coefficients are odds ratios, not probabilities.
• Intuition: 0.063 is a multiplier on the odds, not a destination.
• Answer: +100 points multiplies the odds of default by 0.063, a 94% cut in the odds, whatever the starting point. What that does to the probability depends on where you start. For K, 0.305 becomes 0.027:

\text{odds}_{K+100}

Applicant K's odds of default after adding 100 points to K's credit score

Per 10 points the factor is e^{-0.277} = 0.758. And with ten rows the uncertainty is huge: the 95% profile-likelihood interval for the odds ratio is 0.00049 to 0.80.

• Why: log-odds add, so odds multiply: e^{w_0 + w_1(x + 1)} = e^{w_0 + w_1x} \cdot e^{w_1} (Card 8). Hook: log-odds add, odds multiply, probabilities depend.
• Next they'll ask: "The ten files over-represent defaults. Is the 0.063 biased too?" → No. Sampling on y moves only the intercept; odds ratios survive case-control sampling, which is why epidemiologists report them (Card 12). The book's own 100-point odds ratio, with three features, is 0.221.

Q4 · "DTI raises default by 60%"

The 20,000-applicant model has a DTI coefficient of 0.612 per 10 percentage points. A PM's slide says: "Every 10 points of DTI raises default risk by 60%." What should the slide say?

• Concept tested: the three scales of a coefficient; marginal effects; the divide-by-4 rule.
• Intuition: the coefficient is a push on the log-odds; how far the probability moves depends on where on the S-curve you stand.
• Answer: three correct sentences, one per scale. Log-odds: +0.612 per 10 points. Odds: × e^{0.612} = 1.84, so the odds rise 84%, not 60%. Probability: it depends; on average across the book's applicants, +5.3 percentage points per 10 points of DTI (the average marginal effect), and never more than \beta/4 = 15.3 points.

\frac{\partial p}{\partial x_j}

how fast the default probability changes as you nudge feature j, holding the rest fixed: a slope, not a fraction to divide

p

an applicant's predicted probability of default

x_j

feature j, here DTI in tens of percentage points

j

which feature

\beta_j

feature j's coefficient on the log-odds scale (0.612 for DTI)

\text{AME}_j

average marginal effect: feature j's probability slope averaged over all applicants

n

the number of applicants in the book (20,000)

\sum_i

add up over every applicant i

i

which applicant

\hat p_i

applicant i's fitted probability of default (hat = estimated)

• Why: p(1 - p) is at most 1/4, at p = 0.5, so the probability moves most for applicants near 50% and least for those near 0 (Card 8).
• Next they'll ask: "AME or the effect at the average applicant?" → Report the AME. The marginal effect at the means (MEM) is 3.9 points here, smaller, because the "average applicant" has p = 6.8\%, below the average p of 12%. "Is +84% odds the same as +84% risk?" → Only for rare outcomes, where odds ≈ probability.

Q5 · Accuracy 89% on day one

A new default model launches with 89.1% test accuracy and the PM is thrilled. The base rate is about 12%. "Why did the team train on log loss instead of accuracy," the PM asks, "and what does 89% even mean here?"

• Concept tested: log loss as the training objective; the base-rate baseline.
• Intuition: a model that approves everyone is already 88% "accurate"; accuracy can't see probabilities.
• Answer: on the test split, approving everyone scores 88.1%, so the model adds 1 point. Accuracy throws away the probability and keeps only which side of 0.5 it fell on. Log loss scores the probability itself: the model gets 0.287 against 0.364 for predicting the base rate to everyone.

\frac1n

divide by n, the number of applicants, to get an average

y_i

applicant i's outcome: 1 = defaulted, 0 = repaid

H(0.12)

the entropy at a 12% base rate: the log loss of predicting 12% for everyone

• Why: log loss is the negative log-likelihood, so minimizing it is maximum likelihood; it is smooth and convex in the weights, while accuracy is a step function with zero gradient almost everywhere (Card 2).
• Next they'll ask: "Then which metric do we report?" → Depends on the decision: log loss or Brier for probabilities, AUC for ranking, expected cost at the chosen cutoff for the business (Q8). Ch 5 owns metrics.

Q6 · "Squared error is simpler"

An ML engineer trains a sigmoid output with MSE loss in PyTorch. Training stalls: examples the model gets badly wrong barely change between epochs. "Squared error is simpler and it's what we used for regression," he says. Why does it stall, and what should the loss be?

• Concept tested: why cross-entropy, not squared error, through a sigmoid.
• Intuition: squared error goes deaf exactly when the model is confidently wrong, because the sigmoid is flat there.
• Answer: take a repayer (y = 0) the model scores at p = 0.99. With cross-entropy the gradient with respect to the score is p - y = 0.99. With squared error it is 2(p - y)\cdot p(1 - p) = 0.0196, fifty times weaker, and it shrinks toward 0 the more confident the mistake.

\frac{\partial}{\partial\eta}

how fast what follows changes as you nudge the score \eta, holding the rest fixed

\eta

the applicant's score (log-odds), which the sigmoid turns into p

y

the true label: 1 = default, 0 = repaid

• Why: the p(1 - p) factor from the sigmoid's derivative cancels in cross-entropy but not in squared error. Squared error through a sigmoid is also non-convex: on the ten files it has concave stretches (w_1 below −4.3 and above 1.1), and at w_1 = -15 its slope is 18 times flatter than cross-entropy's (Card 7). Hook: the 3 Cs — Convex, Confident mistakes punished, Calibrated.
• Next they'll ask: "What does a confident mistake cost under log loss?" → -\ln(0.01) = 4.61 for p = 0.01 on a true default, unbounded as p \to 0; squared error caps it at 0.98.

Q7 · ConvergenceWarning

*A batch pipeline fits LogisticRegression(max_iter=50) "to save time" on raw credit score, DTI and income in dollars, plus score² and DTI×income. lbfgs throws "failed to converge". A junior engineer raises max_iter to 10,000 and the warning goes away. Is that the right fix?*

• Concept tested: no closed form, so an iterative solver; conditioning and scaling; the 3 Ps.
• Intuition: features in wildly different units turn the loss into a long, narrow valley, and the solver zig-zags down it.
• Answer: it silences the symptom, not the cause. In scikit-learn 1.9.1 this raw design needs 98 lbfgs iterations (49–98 for the raw designs we tried, so the default max_iter = 100 just scrapes by and 50 fails). Standardized, the same fit takes 15. Put a StandardScaler in a pipeline.

\nabla^2 L

the Hessian: how each slope of the loss changes as you nudge each weight; the loss's curvature

X

the feature matrix: one row per applicant, one column per feature (score, DTI, income)

X^\top

the transpose of X: rows become columns

W

a diagonal matrix holding each applicant's p_i(1 - p_i)

• Why: logistic regression has no closed form (the score equations are nonlinear in w), so it is solved iteratively, and an iterative solver's speed is set by the Hessian's condition number (Ch 2, Section 4). Hook: the 3 Ps — Penalized by default, Predict cuts at 0.5, Plain lbfgs needs scaling.
• Next they'll ask: "Does scaling change the answer?" → For the plain MLE, no: predictions are identical. For sklearn's default, yes: it is L2-penalized (C = 1), and a penalty is not unit-free, so standardizing changes the fitted model (Ch 1 Q45).

Q8 · Is 0.5 the right cutoff?

*Operations calls .predict() on the default model and approves everyone predicted 0 — 97.2% of applicants. A defaulted loan loses about $6,000 of principal; declining a good borrower loses about $600 of profit. Is 0.5 the right cutoff, and if not, what is?*

• Concept tested: the cost-optimal threshold.
• Intuition: when a missed default costs ten times a lost customer, you should decline at a much lower risk than a coin flip.
• Answer: no. Decline when the expected cost of approving beats the expected cost of declining: p\cdot\$6{,}000 > (1 - p)\cdot\$600, so the cutoff is 9.1%. On the 6,000-applicant test split this approves 58.5% instead of 97.2% and cuts the cost per applicant from $604.6 to $311.1 (approving everyone costs $712.0).

c_{FN}

cost of a false negative: approving a defaulter, $6,000 of lost principal

c_{FP}

cost of a false positive: declining a good borrower, $600 of lost profit

t^*

the cost-minimizing threshold (star = the optimum)

• Why: 0.5 is optimal only when the two errors cost the same and the probabilities are calibrated (Card 16). Hook: cost of a false alarm over total cost. And .predict() is the second of the 3 Ps.
• Next they'll ask: "What if manual review exists?" → Three actions, two thresholds (Drill 16): with a $40 review that catches 80% of defaulters, approve below 0.83%, refer up to 31.1%, decline above. "Does moving the cutoff need a refit?" → Never; it needs calibrated probabilities.

Q9 · Three outcomes

Collections wants one model that predicts whether a loan will be prepaid, paid on time, or defaulted. A teammate proposes fitting three separate logistic regressions, one per outcome, and using whichever probability is largest. What would you fit, and why?

• Concept tested: softmax (multinomial) vs one-vs-rest vs ordinal.
• Intuition: three separate models each vote without knowing the others exist; a softmax makes the three share one pie.
• Answer: a multinomial (softmax) logistic regression: one score line per class, probabilities that are positive and sum to 1, one class (on time) as the reference. On the chapter's three-outcome variant, one-vs-rest's three probabilities sum to anywhere from 0.92 to 1.73 for the same loan; its normalized log loss (0.7828) is close to the multinomial's (0.7821), but it is not a coherent probability model.

p_k

the predicted probability of outcome k (prepaid, on time or defaulted)

k

which outcome class

\mathbf x^\top \mathbf w_k

class k's score: the loan's features times class k's own weights, summed (^\top = transpose)

\mathbf x^\top \mathbf w_j

class j's score

\sum_{j=1}^{K}

add up over all K classes, j = 1 to K

K

the number of classes (3)

\mathbf w_{\text{on time}}

the reference class's weights, fixed at zero

\mathbf 0

a vector of zeros

• Why: the softmax couples the classes through its denominator, and its loss is still −log of the probability given to what happened (Card 17). Hook: one-vs-rest votes, softmax shares, ordinal climbs.
• Next they'll ask: "And current / 30 days late / defaulted?" → Those are ordered, so a proportional-odds model: one line plus cut-points, 4 parameters instead of 6 on the chapter's ordinal variant (AIC 21,689.5 vs 21,715.1).

Q10 · Trained on 50/50

A data scientist undersampled repayers until the training set was 50/50 defaults "for balance". In production the model's average predicted default rate is 38.7% on a book where 12.0% actually default. The ranking looks fine. What happened, and how do you fix it without retraining?

• Concept tested: sampling on y shifts only the intercept.
• Intuition: throwing away most repayers makes defaults look common, but it doesn't change how much a lower score raises the odds.
• Answer: keeping every default and a share s_0 of repayers multiplies every applicant's odds by 1/s_0, so the intercept rises by \ln(1/s_0) and the slopes are untouched. Here s_0 = 0.137, so the intercept is 1.99 = \ln 7.31 too high. Subtract it: the mean prediction falls back from 38.7% to 12.0%.

P(y = 1 \mid \mathbf x, \text{sampled})

the chance an applicant with features \mathbf x is a default, among the rows kept for training

w_0

the population intercept

s_1

the share of defaults kept (all of them: 1)

s_0

the share of repayers kept (0.137 here)

\mathbf x^\top \mathbf w

the slope part of the score: features times weights, summed (^\top = transpose)

\hat w_0^{\text{prod}}

the corrected intercept to use in production

\hat w_0^{\text{sample}}

the intercept fitted on the 50/50 sample (hat = estimated)

• Why: Bayes' rule on odds: posterior odds = prior odds × likelihood ratio, and sampling scales the prior odds by s_1/s_0 for everyone (Card 12). Hook: sample on y, shift the intercept.
• Next they'll ask: "What if you only know the population base rate?" → Use the prior-shift formula on the odds (Card 13; Drill 7). "Is the shift always exact?" → Only if the logistic model is correctly specified (WP1).

Q11 · "Says 10%, 18% defaulted"

Last quarter, 24,396 applicants were scored between 8% and 12% (average 9.9%), and 17.8% of them defaulted. The risk lead asks: "Is the model broken? Should we rebuild it?"

• Concept tested: calibration and reliability diagrams; calibration vs discrimination.
• Intuition: a scale that reads 10 kg for an 18 kg bag is miscalibrated, but it may still sort bags by weight perfectly.
• Answer: the model is miscalibrated there: "10%" no longer means 10%. With 24,396 rows, 17.8% vs 9.9% is not noise. The first suspect is a level shift: in this scenario the true log-odds rose by 0.68 for everyone and the base rate climbed to 19.1%. Check a reliability diagram and the AUC on recent data. If the ranking is intact, recalibrate (intercept or Platt) on recent held-out loans; don't rebuild.

P

the actual share (probability) of what's in the brackets

\hat p(\mathbf x)

the model's predicted default probability for an applicant with features \mathbf x

q

any probability level, e.g. 10%

• Why: a uniform shift in log-odds moves every probability but reorders no one, so ranking metrics can't see it (Card 13).
• Next they'll ask: "Wasn't logistic regression calibrated by construction?" → Only on the data it was fit to: its intercept's score equation forces \sum\hat p = \sum y there (Card 6). A new population breaks it (Shift, Q36).

Q12 · Explain the decline

A declined applicant must receive the principal reasons for the decision (an adverse action notice under the US Equal Credit Opportunity Act). The product team asks: "How do we produce reasons from the logistic model, and what can go wrong?"

• Concept tested: additivity of log-odds; reason codes; points scaling.
• Intuition: the score is a sum of one term per feature, so you can see which terms cost the applicant the most.
• Answer: each feature contributes w_j(x_j - x_{\text{ref},j}) to the log-odds relative to a reference applicant (often the best-scoring or the approved average). Convert to points with the scorecard factor (20 points to double the odds gives 28.85 points per unit of log-odds) and report the features that lost the most points. An applicant with a 620 score (z = -0.6) against a reference at z = 0, on the book's model:

\Delta\eta_{\text{score}}

the change in log-odds caused by credit score alone, versus the reference applicant (\Delta = change)

• Why: the model is additive on the log-odds scale, so contributions are exact and need no approximation such as SHAP (Card 8).
• Next they'll ask: "What's the catch?" → Three traps: correlated features split one story across several reasons (or flip a sign, Q26); the reference point changes the ranking of reasons; and transformed or interacted features (splines, DTI×income) must be grouped back into one human-readable reason. Reasons must also be accurate and stable, and a protected attribute's proxy can't hide behind them (Q33).

Level 2 — Onsite core

Q13 · From coin flips to the loss

Whiteboard round. "Each of our ten applicants is a biased coin: applicant i defaults with probability p_i = \sigma(x_i^\top w). Write down the probability of the data we saw, and derive the loss you'd minimize. What is it at w = 0?"

• Concept tested: Bernoulli likelihood → cross-entropy (Card 2).
• Intuition: score each applicant by the probability the model gave to what actually happened, then multiply.
• Answer: one coin contributes p if it defaulted and 1 - p if it repaid, written p_i^{y_i}(1 - p_i)^{1 - y_i}. Multiply over independent rows, take logs, flip the sign:

\mathcal L(\mathbf w)

the likelihood: how probable the ten observed outcomes are under weights \mathbf w

\mathbf w

the weights (intercept and slope) as one vector; bold = vector

\prod_{i=1}^n

multiply the term after it over every applicant, i = 1 to n

i

which applicant (row)

n

the number of applicants (rows): 10 in the ten files

p_i

applicant i's probability of default, \sigma(\mathbf x_i^\top \mathbf w)

y_i

applicant i's outcome: 1 if they defaulted, 0 if they repaid

\sum_{i=1}^n

add up the term after it over every applicant, i = 1 to n

At w = 0 every p_i = 0.5, so the likelihood is 0.5^{10} = 0.000977 and the loss is 10 \ln 2 = 6.93 (mean \ln 2 = 0.693). The fitted model raises the log-likelihood to −4.36.

• Why: that sum is the binary cross-entropy, or log loss: "−log of the probability you gave to what happened". It is the same move as Ch 1, where Gaussian noise turned maximum likelihood into least squares; here a coin flip turns it into cross-entropy.
• Next they'll ask: "ML papers write \ln(1 + e^{-y\eta}). Same thing?" → Yes, with labels coded \tilde y \in \{-1, +1\}: \sum \ln(1 + e^{-\tilde y_i\eta_i}), a loss on the margin \tilde y\eta (the hinge loss of Ch 11 is its piecewise-linear cousin). "Why not squared error?" → Q6.

Q14 · The gradient on the board

"Now differentiate that loss with respect to w. Don't skip steps. What does the result remind you of, and what is it on the ten files at w = 0?"

• Concept tested: the gradient X^\top(p - y) (Card 3).
• Intuition: each feature column, weighted by how wrong we are on each row: features × misses.
• Answer: use \sigma' = \sigma(1 - \sigma) and the chain rule on one row; everything except the miss cancels:

\frac{\partial}{\partial \eta_i}

how fast what follows changes as you nudge applicant i's score, holding the rest fixed: a slope, not a fraction to divide

\eta_i

applicant i's score (log-odds), \mathbf x_i^\top \mathbf w

\sigma(\eta_i)

the sigmoid of applicant i's score: their probability of default p_i

\nabla_{\mathbf w}

the gradient with respect to the weights: the vector of all those slopes, one per weight; points uphill

\mathcal L

the likelihood, so -\ln\mathcal L is the total log loss

X

the feature matrix: one row per applicant, first column all 1s, second the score x

X^\top

the transpose of X: rows become columns

\mathbf p

all ten predicted probabilities stacked into one vector

\mathbf y

all ten 0/1 outcomes stacked into one vector

At w = 0, p = 0.5 everywhere: the intercept entry is \sum(0.5 - y_i) = 5 - 4 = 1, and the slope entry is 0.5\sum x_i - \sum x_iy_i = -0.2 + 2.3 = 2.1. So the gradient is (1, 2.1).

• Why: it is OLS's gradient X^\top(\hat y - y) with p in place of \hat y, so the same code, the same intuition, and the same "features × misses" picture carry over, all the way to softmax and every canonical GLM (Card 17).
• Next they'll ask: "Then set it to zero and solve?" → You can't in closed form: p = \sigma(Xw) is nonlinear in w. That is why logistic regression needs Newton or gradient descent (Q16).

Q15 · One bottom?

A colleague who trains neural networks worries that logistic regression, being "nonlinear", might also have local minima, so results could depend on initialization. Prove or disprove it — and say whether anything can still go wrong.

• Concept tested: convexity via the Hessian X^\top WX (Card 4).
• Intuition: the loss is a bowl, so there is only one bottom — but the bowl can be missing its bottom.
• Answer: disproved. Differentiate the gradient once more: the Hessian is X^\top WX with W = \operatorname{diag}(p_i(1 - p_i)), positive semi-definite for every w, so the loss is convex and every local minimum is global. On the ten files at w = 0 it is 0.25\cdot X^\top X:

\nabla^2

the Hessian: second derivatives, the curvature of what follows with respect to the weights

W

a diagonal matrix holding each applicant's weight p_i(1 - p_i)

\mathbf v

any direction you could move the weights in

\mathbf v^\top

\mathbf v transposed into a row (transpose: rows become columns)

\sum_i

add up over every applicant i

\mathbf x_i^\top \mathbf v

how far applicant i's features point along the direction \mathbf v

H_{\mathbf 0}

the Hessian at the starting weights \mathbf w = \mathbf 0

• Why: each row's curvature is a non-negative weight times a squared projection. With full column rank and every pᵢ strictly inside (0, 1), the loss is strictly convex, so any minimum is unique.
• Next they'll ask: "So it always converges?" → Not if the minimum doesn't exist. Under separation the loss keeps falling as \lVert w\rVert \to \infty: convex, but no bottom (Q17).

Q16 · One Newton step by hand

"Start the ten files at w = 0. Compute the first Newton step by hand, then tell me why people call it iteratively reweighted least squares."

• Concept tested: Newton = IRLS; the working response (Card 5).
• Intuition: at each step, pretend the problem is a weighted least-squares fit to a made-up target, solve it exactly, and repeat: weigh, regress, repeat.
• Answer: the step is H^{-1}X^\top(y - p), with H and the gradient from Q14–Q15:

\mathbf w_1

the weights after one Newton step

\mathbf 0

the starting weights: all zeros

Rearranged, every Newton step is a weighted least-squares fit of a working response zᵢ on X with weights pᵢ(1 − pᵢ):

\mathbf w_{\text{new}}

the weights after one Newton (IRLS) step

(X^\top W X)^{-1}

the matrix inverse of the Hessian: undoes it, like dividing by the curvature

\mathbf z

all the applicants' working responses as one vector

z_i

applicant i's working response: the current score nudged by how wrong the row is

At w = 0 every weight is 0.25 and z_i = \pm 2, so the first step is plain OLS of 4y - 2 on x: exactly 4 × the linear probability model minus 2 in the intercept (4 \times 0.382 - 2 = -0.471, 4 \times (-0.444) = -1.774).

• Why: a Newton step minimizes the local quadratic approximation, and for the log loss that quadratic is a weighted sum of squares (Card 5).
• Next they'll ask: "How many steps?" → The distance to the MLE goes 2.77, 0.991, 0.268, 0.0232, 0.000186, 1.2\times 10^{-8}: quadratic convergence, 5 steps (a "step below 10^{-8}" stopping rule fires on step 7). Gradient descent on standardized x needs 51 steps at \eta = 2 and 191 at \eta = 0.5 (Section 4).

Q17 · The coefficient that ran to 151

Someone fits statsmodels Logit on the ten review files with credit score and DTI (per 10 points). The output shows a DTI coefficient of 151 with a standard error of about 105,700, a ConvergenceWarning and a PerfectSeparationWarning. The Wald p-value is essentially 1. "So DTI doesn't matter?" What happened, and what do you report?

• Concept tested: complete separation; the MLE does not exist; Firth and L2 (Card 11).
• Intuition: if a line splits defaulters from repayers perfectly, the model can always be "more sure" by making the line steeper, so it never stops.
• Answer: every applicant above 39% DTI defaulted and every one below repaid, so the data are completely separated. Along the separating direction the log-likelihood rises toward 0 forever, and IRLS chases it: \lVert w\rVert grows ×1.6–1.8 per iteration at first, then by a steady +8.2 per iteration, until X^\top WX is numerically singular at iteration 40. The 151 is wherever the solver stopped; the SE is meaningless; and the Wald test calls the strongest predictor in the data useless. Report a finite, penalized estimate:

\max_{\mathbf w}

find the weights that make what follows largest

\ell(\mathbf w)

the log-likelihood at weights \mathbf w

\big|X^\top W X\big|

the determinant of the information matrix: its overall size, Firth's penalty

\hat w_{\text{DTI}}

the fitted DTI coefficient under Firth (hat = estimated)

C

sklearn's inverse penalty strength: big C, little penalty

• Why: with separation there is a w^* with \tilde y_ix_i^\top w^* > 0 for all rows, so \ell(c\cdot w^*) \uparrow 0 as c \to \infty (Card 11); the Jeffreys-prior penalty in Firth's method grows near that edge and keeps the estimate finite. This is the S in L-I-N-E-S.
• Next they'll ask: "How would you detect it without the warning?" → Coefficients in the tens with SEs in the thousands, \log L creeping to 0, or a linear-programming check for a separating w (Section 6; Drill 5). "Is the LR test still usable?" → Yes, it reads heights, not widths: a profile-likelihood bound on the effect is the honest report.

Q18 · Wald says no, LR says yes

On the ten files, credit score's Wald test gives p = 0.106 and the likelihood-ratio test gives p = 0.029. The credit committee wants one answer: "Does score predict default or not?" Which do you report, and why do they disagree?

• Concept tested: Wald vs LR vs score tests; small-sample behaviour (Card 10).
• Intuition: Wald measures the distance to zero with a ruler calibrated at the peak; on a lopsided hill that ruler is wrong.
• Answer: report the LR test (and a profile-likelihood interval). Three tests of w_1 = 0, hook W-H-S: Wald = Width (z = -1.62, p = 0.106), LR = Height (\text{LR} = 2 \times (-4.361 + 6.730) = 4.74, p = 0.029), Score = Slope (S = 3.96, p = 0.047).

W = \frac{\hat w_1^2}{\text{SE}(\hat w_1)^2}

the Wald statistic: squared distance of the slope from 0, in standard errors

\hat w_1

the fitted score slope, −2.766 per 100 points

LR

the likelihood-ratio statistic: twice the log-likelihood lost by forcing w_1 = 0

\ell(\hat{\mathbf w})

the log-likelihood at the full fit (−4.361)

\ell(\hat{\mathbf w}_0)

the log-likelihood with w_1 forced to 0, intercept only (−6.730)

\hat{\mathbf w}_0

the restricted fit: intercept only, w_1 = 0

S

the score (Rao) statistic: squared slope at the restricted fit, scaled by the information there

U(\hat{\mathbf w}_0)

the score: the slope of the log-likelihood in w_1 at the restricted fit

I(\hat{\mathbf w}_0)

the Fisher information (curvature) at the restricted fit

\chi^2_1

a chi-squared distribution with 1 degree of freedom

The 95% Wald interval for w_1 is [−6.12, 0.585] and includes 0; the profile-likelihood interval is [−7.63, −0.223] and doesn't.

• Why: all three approximate the log-likelihood by a parabola, at different points; on ten rows it falls steeply toward 0 and flattens toward -\infty, so Wald's SE (from the curvature at the peak) is too big. LR uses the actual heights.
• Next they'll ask: "Does Wald ever get worse as the effect gets stronger?" → Yes, the Hauck–Donner effect: the SE grows faster than the coefficient, so z falls (Drill 19); separation is the limit.

Q19 · Reading the Logit summary

The interviewer slides over the statsmodels summary of the 20,000-applicant fit: Pseudo R-squ. 0.2065, Log-Likelihood −5832.2, LL-Null −7350.4, LLR p-value 0.000, and z-values of −78.6, −28.9, 19.8 and −10.1. "Walk me through it. Is 0.21 a bad fit? Why z and not t?"

• Concept tested: McFadden R^2, the LR test against the null model, deviance, and asymptotic inference.
• Intuition: every line of the summary compares two heights on the same log-likelihood hill.
• Answer: Pseudo R^2 is McFadden's, 1 - \ell/\ell_{\text{null}} = 1 - 5832.2/7350.4 = 0.21. It is not a share of variance and runs much lower than OLS R^2; values of 0.2–0.4 are considered a very good fit. LLR p-value is the LR test that all three slopes are 0: 2 \times (7350.4 - 5832.2) = 3{,}036 on 3 df. z, not t, because inference is large-sample normal and there is no \sigma^2 to estimate. In deviance terms, the null deviance 14,701 drops to 11,664.5.

R^2_{\text{McF}}

McFadden's pseudo-R^2: the share of the null log-likelihood the model recovers

\ell(\hat{\mathbf w})

the log-likelihood at the fitted weights (−5832.2 on the book)

\ell_{\text{null}}

the log-likelihood of the intercept-only model (−7350.4 on the book)

LLR

the likelihood-ratio statistic against the null model, as statsmodels labels it

\sim

is distributed as

\chi^2_{k}

a chi-squared distribution with k degrees of freedom, k = the number of slopes tested (3)

• Why: the deviance plays the role of Ch 1's RSS and drops in deviance play the role of F-tests (Card 10). The truth here is known: even the true model can't make a 0/1 outcome predictable, so a "low" R^2 is the ceiling of the problem, not a flaw. On the test split the model's log loss is 0.287 against 0.364 for the base rate.
• Next they'll ask: "What would worry you in that output?" → converged: False, coefficients in the tens with huge SEs (separation, Q17), or a model compared with a different LL-Null (different rows). "AIC?" → 11,672.5; compare it only across models fit on the same rows.

Q20 · Risk accelerates above 43%

Underwriters insist risk jumps once DTI passes about 43%. The production model is linear in DTI on the log-odds scale, and its overall fit statistics look fine. How do you check the claim, and if it's true, what do you change?

• Concept tested: linearity in the log-odds — binned residuals, Box–Tidwell, splines.
• Intuition: a straight line forced through a hockey stick tilts too steeply in the middle and still misses at the end.
• Answer: on the chapter's NL variant (risk accelerates above 43%), the checks all agree. See: binned residuals by DTI sit above zero at low DTI, dip below in the high 30s to mid 40s, then shoot up; 75% of 20 bins fall outside the \pm 2 SE band (5% on the main book). Test: the Box–Tidwell term x\cdot\ln x (x = \text{DTI}/10) has z = 21.3; a 3-df spline in DTI improves the log-likelihood by \text{LR} = 455. Fix: a spline or a hinge term \max(0, (\text{DTI} - 43)/10), whose fitted coefficient is 2.61 against a true 2.5.

\text{logit}\,p

the log-odds of default for an applicant

\beta_d

the linear DTI coefficient

d

DTI in tens of points above 36%

\beta_h

the hinge coefficient: extra log-odds per 10 DTI points above 43% (fitted 2.61)

• Why: the linear fit compensates by steepening the DTI slope to 1.56 (truth 0.6) and still predicts log-odds 1.17 at 60% DTI where the truth is 3.09. Linearity is the L in L-I-N-E-S: break it and coefficients are biased in whole regions.
• Next they'll ask: "Why not a raw residual plot?" → Raw residuals for a 0/1 outcome are always two curved bands, whatever the truth; bin them. With the spline, only 5% of bins fall outside the band and HL rises to p = 0.75.

Q21 · Hosmer–Lemeshow p < 0.001 on 2M rows

A model validator rejects a PD model because its Hosmer–Lemeshow test gives p \approx 10^{-30} on 2 million loans. The model team says the misfit is tiny. Who's right, and what should the validation report show instead?

• Concept tested: goodness-of-fit tests at large n; plots first, tests second.
• Intuition: with 2 million rows a test can detect a crack too small to matter.
• Answer: both are, about different questions. In the chapter's large-n run, a mildly misspecified model (a small hockey stick above 43% DTI) gets \text{HL} = 160, p \approx 1.7\times 10^{-30}, yet its worst decile misses by 0.6 percentage points. The same misspecification at 20,000 rows gets p = 0.067. HL answers "can I detect misfit with this n?", not "is the misfit big enough to change a decision?". Report the reliability diagram with intervals, the size of the worst gaps, and whether they move any decision (threshold, pricing, expected loss).

\text{HL}

the Hosmer–Lemeshow statistic: total mismatch between observed and expected defaults across groups

\sum_{g=1}^{10}

add up over the 10 groups (deciles of \hat p)

g

which group: rows sorted by \hat p and cut into 10

O_g

the observed number of defaults in group g

E_g

the expected number of defaults in group g: the sum of its predicted probabilities

n_g

the number of loans in group g

\chi^2_8

a chi-squared distribution with 8 degrees of freedom (10 groups minus 2)

• Why: a test's power grows with n, so at large n every model fails; Ch 1's rule "if p is low, the assumption must go" needs the effect size next to it. HL also depends on the number of groups chosen.
• Next they'll ask: "Would the correct model pass at 2M rows?" → Its p-value would be uniform: in our single run the correct model drew p = 0.038, which is the luck of one draw, not evidence. That is the point: at this n, read magnitudes, not p-values.

Q22 · class_weight="balanced"

*A teammate switches on class_weight="balanced" in the PD model. Recall at the 0.5 cutoff jumps from 17% to 74%, and the team celebrates. A week later finance reports that the expected-loss forecast built on the model's probabilities is 3.2 times higher than before. Explain both.*

• Concept tested: weights equal sampling on y; the intercept shift \ln((1 - \pi)/\pi); calibration.
• Intuition: giving each default four times the vote of a repayer is like photocopying the defaults — the model learns that defaults are common.
• Answer: balanced weights are w_1 = n/(2n_1) = 4.16 and w_0 = n/(2n_0) = 0.568, which act exactly like keeping defaults at rate w_1 and repayers at rate w_0. The slopes stay put and the intercept rises by \ln(w_1/w_0) = \ln((1 - \pi)/\pi) = \ln 7.31 = 1.99 (simulated: 2.01). Every probability moves up: the mean \hat p goes from 0.120 to 0.389, so expected loss (\text{PD} \times \text{LGD} \times \text{EAD}) is inflated 3.2×. Recall at 0.5 rose because every probability crossed 0.5 more easily, not because the model got better. The AUC is unchanged (0.8118 vs 0.8118).

\Delta w_0

how far the intercept moves under balanced class weights (\Delta = change)

w_1

here the class weight on each default, n/(2n_1) = 4.16

w_0

here the class weight on each repayer, n/(2n_0) = 0.568

n_0

the number of repayers in the book

n_1

the number of defaults in the book

\pi

the base rate: the share of applicants who default (12.0%)

• Why: a weight is a fractional copy of a row, so the weighted score equation is the case-control score equation in expectation (Card 12). Hook: Shrink, Sample, Shift — this is Sample.
• Next they'll ask: "So never use weights?" → Not for a PD model without recalibrating. To get more recall, move the threshold instead (Q8); it does the same job without breaking the probabilities. If weights help SGD, recalibrate afterwards with Platt (here it undoes the shift: ECE 0.269 → 0.014).

Q23 · Probabilities squashed toward 12%

*A new product line has only 400 training loans. The team copied a config from a larger model, including C=0.003. The model never predicts much above 22% (the median top prediction across samples), even for obvious defaulters, though the true risk of the riskiest applicants reaches 93%. The mean prediction is right at 12%. What's going on?*

• Concept tested: sklearn's L2 penalty and the meaning of C; Shrink; the calibration slope.
• Intuition: a strong penalty pulls every slope toward zero, so every applicant's prediction collapses toward the average.
• Answer: sklearn minimizes C \times (\text{summed log loss}) + \tfrac12\lVert w\rVert^2, so the penalty's strength per row is about 1/(C\cdot n). C = 0.003 on 400 rows is a crushing penalty; the same C on 14,000 rows would be far weaker. The intercept isn't penalized, so its score equation still forces mean \hat p = \bar y (12%), but the spread collapses. Over 200 samples of 400 rows:

\min_{w_0,\mathbf w}

find the intercept and weights that make what follows smallest

w_0

the intercept (not penalized)

\ell_i(w_0, \mathbf w)

applicant i's log loss at these weights

\ell_i

applicant i's log loss

\lVert\mathbf w\rVert^2

the squared length of the weight vector: the sum of squared weights (the L2 penalty)

n

the number of training loans (400 in the scenario)

\frac1n

divide by n to get the average loss per loan

• Why: shrinkage costs calibration: predictions too timid give a calibration slope above 1, too extreme below 1 (Card 6, Section 9). Hook: big C, little penalty — and it's on by default.
• Next they'll ask: "So is the default C = 1 safe?" → Not automatically: at n = 400 it overfits (slope 0.72), at n = 14{,}000 it is practically no penalty. Choose C by cross-validated log loss (LogisticRegressionCV), and set C=np.inf for a plain MLE.

Q24 · Platt or isotonic?

You need to recalibrate a gradient-boosted PD model. Your team has 2,000 held-out loans; a sister team has 300,000. Which recalibration method should each team use, and on what data?

• Concept tested: Platt scaling vs isotonic regression; calibrating on held-out data.
• Intuition: Platt is a line, isotonic is a staircase: the line needs two numbers, the staircase needs data on every step.
• Answer: Platt fits q = \sigma(a\cdot\eta + b), a one-feature logistic regression on the model's score: 2 parameters, stable on small data, but it can only fix a logit-linear distortion. Isotonic fits the best non-decreasing step function: it fixes any monotone distortion but needs data. With 2,000 rows, start with Platt and compare isotonic by cross-validated log loss; with 300,000, isotonic. In the chapter's run, with 3,000 calibration rows isotonic was slightly better (ECE 0.011 vs Platt's 0.014); with 300 rows it set 45% of test predictions to exactly 0 and its log loss was 0.421 against Platt's 0.301.

q

the recalibrated probability

\sigma

the sigmoid: turns a score into a probability

a

Platt's fitted slope on the model's score

\eta

the model's score (logit) for an applicant

b

Platt's fitted intercept

g

here the isotonic map: a non-decreasing staircase from old probability to new

\hat p

the model's original predicted probability

• Why: both must be fit on rows the model never trained on — a held-out split or cross-fitted folds (CalibratedClassifierCV(cv=5)) — or they learn the training set's overconfidence. Both are monotone, so neither changes the AUC.
• Next they'll ask: "Does a logistic model need recalibrating at all?" → On its training population it is calibrated in the large by its score equation; it needs recalibration after Shrink, Sample or Shift (Q11, Q22, Q23, Q36). Boosted models usually do need it.

Q25 · Autopay and the 2×2 table

On the 20,000-applicant book: among autopay users, 1,057 defaulted and 10,313 repaid; among non-users, 1,349 defaulted and 7,281 repaid. The interviewer: "Logistic regression of default on an autopay dummy. Tell me the coefficient, its intercept and its standard error without fitting anything. Then: does autopay prevent default?"

• Concept tested: the MLE with one binary feature is the log odds ratio of the table.
• Intuition: with one dummy the model has one number per group, so it simply reproduces each group's default rate.
• Answer: the intercept is the log-odds for non-users, \ln(1{,}349/7{,}281) = -1.686. The coefficient is the log cross-product ratio, −0.592 (odds ratio 0.553), with SE 0.0438:

\hat\beta_1

the fitted autopay coefficient: the log odds ratio of the 2×2 table (hat = estimated)

\text{SE}(\hat\beta_1)

its standard error

Does autopay prevent default? Not on this evidence. In this book, autopay was simulated to depend on credit score and to have no effect of its own; adjusting for score, its coefficient falls to −0.026 (p = 0.60).

• Why: two parameters for two groups is a saturated model, so the score equations force \hat p to equal each group's observed rate (Card 6); the SE is the square root of the inverse cell counts (Card 9).
• Next they'll ask: "So should we push customers to autopay?" → Only a randomized test of an autopay nudge can tell (Ch 7). Better-credit customers choose autopay; that confounding is the whole association here.

Q26 · Two credit variables, wild signs

A risk model has "number of open trades" with a coefficient of +0.56 (SE 0.044). After adding "number of recent inquiries", which is correlated 0.95 with open trades, the open-trades coefficient becomes −0.18 and its SE triples to 0.137. The business asks which sign is right.

• Concept tested: multicollinearity in logistic regression; VIF; omitted-variable bias.
• Intuition: two witnesses telling the same story; before inquiries were in the model, open trades was speaking for both.
• Answer: the joint model is right about the sign, and imprecise. In this simulated design the truth is −0.3 for open trades and +0.8 for inquiries. Alone, open trades absorbed inquiries' effect (+0.56: Ch 1's omitted-variable bias). Together, the model can separate them only through the 10% of variance they don't share (1 - 0.95^2), so the SE inflates by about \sqrt{\text{VIF}}:

r

the correlation between open trades and recent inquiries (0.95)

• Why: the information matrix X^\top WX is nearly singular along the direction "trades minus inquiries", so that contrast has a huge variance (Card 9). This is the N in L-I-N-E-S.
• Next they'll ask: "What do you ship?" → If the goal is prediction, both, with an L2 penalty for stability. For a scorecard with reason codes, combine them into one "credit-seeking" feature or keep one, because a reason code saying "too few open trades" would mislead (Q12).

Q27 · Counting missed payments

Finance wants to predict the number of missed payments per borrower. Loans have been on the books anywhere from 6 to 24 months. A teammate fits OLS on the raw counts with score and DTI as features. What's wrong, and what's the right model?

• Concept tested: Poisson regression; the log link; the exposure offset; overdispersion (Card 17).
• Intuition: a loan watched for 24 months has four times the chances to miss a payment of one watched for 6; model the rate, then multiply by the time.
• Answer: OLS can predict negative counts, assumes constant variance (a count's variance grows with its mean) and ignores exposure. Fit a Poisson GLM with a log link and \log(\text{months}) as an offset, a feature whose coefficient is fixed at 1. On the chapter's POIS variant it recovers (−3.00, −0.890, 0.385) against the truth (−3.0, −0.9, 0.4); exponentiated, a rate ratio of 0.41 per 100 score points and 1.47 per 10 DTI points.

\mathbb E[\text{count}_i]

borrower i's expected number of missed payments (the average over repeated samples)

\text{months}_i

how long borrower i's loan has been on the books: the exposure

\mathbf x_i^\top\mathbf w

borrower i's score: the log of missed payments per month (^\top = transpose)

e

Euler's number, about 2.718

• Why: every GLM is a Distribution, a Line, and a Link: here Poisson, x^\top w and log. With the canonical log link the score equations are X^\top(y - \mu) = 0, so \sum\hat{\mu} = \sum y, as in logistic regression.
• Next they'll ask: "What if the variance is bigger than the mean?" → Check Pearson \chi^2/\text{df}: 1.00 on Poisson data, 1.52 on overdispersed data, where naive Poisson SEs are too small (the true SD of the score coefficient is 1.61× the naive SE). Use a negative binomial, quasi-Poisson or robust SEs. "What if you drop the offset?" → If exposure correlates with the features, the coefficients absorb it (Drill 20).

Level 3 — Senior

Q28 · We only see the loans we approved

The PD model is trained on five years of approved loans, because only approved loans have outcomes. Approval used the score plus an underwriter's judgment the model never sees. Strategy now wants to approve applicants below the old cutoff, using the model's predictions for them. What breaks, and what do you do?

• Concept tested: reject inference — selection on unobservables; the explore arm; bounds.
• Intuition: a low-score applicant was approved only if the underwriter saw something good, so the low-score loans you have are the best of a risky group.
• Answer: in the chapter's RI variant (approval needs z + 0.8u > -0.3, where u is the unseen underwriter signal), 61.7% are approved and default at 5.1%; the rejects would default at 27.7%. Among the approved, score and u are correlated −0.38 (−0.01 in the population), so the score coefficient flattens from −1.35 (everyone) to −0.92 (approved only), and the approved-only model predicts 10.9% for the rejects. Lower the cutoff on that forecast and losses come in at more than twice the plan.

• Why: this is selection on a collider (Ch 1's Berkson pattern): approval depends on score and u, so conditioning on it creates a negative score–u correlation. No model without u can reach 27.7%, because the rejects were selected by u. Parceling adds rows labeled with the model's own \hat p, which contribute exactly zero to the score equation, so it reproduces the approved-only fit to the last digit. Hook: Who, What, How, When — this is Who.
• Next they'll ask: "What do you recommend?" → Run an explore arm (approve a small random share below the cutoff, with a loss budget), use bureau data on rejects' loans elsewhere, report bounds, and lower the cutoff in steps while monitoring. "Which part of the problem does the explore arm fix?" → It fixes the slope and measures the rejects directly; a model without u still under-predicts them.

Q29 · Last quarter's loans look great

Loans booked in the last 3 months show a 3.1% default rate, and those booked 6 months ago 6.2%, against 12% across the book. The label is "charged off within 12 months". A teammate wants to retrain on all loans, including the recent ones, "to capture the latest applicant mix". What's the problem, and what do you do?

• Concept tested: immature labels and right-censoring; performance windows; vintages; survival models.
• Intuition: a loan booked last month hasn't had time to default; "no default yet" is not "repaid".
• Answer: recent loans' labels are censored, so they look safe because they are young. Mixed into training, they teach the model that whatever is recent (new channels, new products, current conditions) is low-risk, and they drag the intercept down. Fixes: a fixed performance window (train only on loans at least 12 months old); vintage analysis (cumulative default curves by booking month, compared at equal age); or a discrete-time survival model, one row per loan-month, which uses partial follow-up correctly.

P(\text{default in month } t \mid \text{survived to } t, \mathbf x)

the chance a loan defaults in month t after booking (1 to 12), given it got that far and has features \mathbf x

\mathbf x

the loan's features as a vector (bold = vector)

\alpha_t

the baseline log-odds of default in month t: one intercept per month

\mathbf x^\top\mathbf w

the features' part of the score: features times weights, summed (^\top = transpose)

• Why: this is the When in Who, What, How, When: the label's meaning depends on when it was observed. The 3-month and 6-month rates in the scenario are what the book's own default process produces at those ages.
• Next they'll ask: "Then how do you capture the latest mix?" → Use recent applicants' features for monitoring and reweighting (covariate shift, Ch 4), recalibrate the intercept when mature labels arrive (Q36), and use early-delinquency labels (30 days past due) as leading indicators with their own model.

Q30 · The control that changed the treatment effect

A randomized test sends an SMS payment reminder to half of 20,000 borrowers. The unadjusted log odds ratio of default is −0.31. Adding a pre-treatment risk score, which is independent of treatment by randomization, moves it to −0.40. A PM asks: "Randomization means no confounding, so how can a control change the effect? Which number is right?"

• Concept tested: non-collapsibility of odds ratios; conditional vs marginal estimands (Card 14).
• Intuition: a logistic coefficient lives on a scale set by the unexplained noise; leaving out a strong risk factor adds noise, so every coefficient shrinks, no confounding needed.
• Answer: both are right, for different questions. −0.40 is the conditional log-OR, for borrowers with the same risk score (the truth in this simulation is −0.4). −0.31 is the marginal, population-average log-OR. The risk difference is the same either way: −0.0394 unadjusted, −0.0394 as an adjusted average marginal effect. The approximation for the shrinkage:

\beta_{\text{marginal}}

the reminder's log odds ratio with the risk score left out (population-average)

\beta

the conditional log odds ratio, with the risk score in the model (−0.4)

\pi

the circle constant, 3.14159

\gamma

the left-out risk score's coefficient (1.0)

\sigma_Z^2

the variance of the risk score Z (2.25); here \sigma is a standard deviation, not the sigmoid

• Why: odds ratios are non-collapsible: averaging a logistic curve over Z gives a flatter curve, not a logistic curve with the same slope (Card 14). In OLS, an independent covariate can't move the coefficient. The \pi/8 formula is an approximation whose error depends on the base rate (7% short here, at 15%).
• Next they'll ask: "Which do you report?" → The risk difference (4 percentage points fewer defaults) for the PM, pre-specified. The adjusted analysis also has more power: z = -8.8 vs −7.8 here, and at n = 2{,}000 power is 80% vs 67%. Don't compare odds ratios across studies with different covariates (WP2).

Q31 · A 12-feature scorecard from 300

The regulator wants a short, stable, monotone scorecard. You have 300 candidate features, many of them near-duplicates (several versions of the bureau score, utilization over different windows). How do you get to about 12 features you can defend?

• Concept tested: L1 paths and selection instability; WOE/monotone binning; points scaling.
• Intuition: L1 picks a winner among identical twins almost by coin flip, so a "selected" feature can vanish on the next refit.
• Answer: L1 kills, L2 shrinks: an L1 path gives the short list. But on the chapter's L1 variant (the book plus 3 near-copies of the score, correlated 0.986, and 20 pure-noise features), cross-validation's best C keeps 9 features, 5 of them noise. A sparser rule (the smallest C within 0.001 of the best CV log loss) keeps 4 at a cost of 0.0005 in log loss. Under that sparser rule, over 100 bootstrap refits, score, DTI and income are kept every time, while the three score copies are kept 44%, 62% and 47% of the time, interchangeably. So: (1) cluster near-duplicates and keep one per cluster on business grounds; (2) L1 or stepwise on the reduced set, reporting bootstrap selection frequencies; (3) bin each kept feature into monotone weight-of-evidence bins; (4) scale to points.

Applicant K's corrected good:bad odds of 16.7 give 568 points.

• Why: the L1 penalty's corners sit on the axes, so among correlated features whichever is slightly ahead on this sample takes all the credit (Ch 1, Section 10). Selection is unstable even when predictions aren't.
• Next they'll ask: "Why monotone bins?" → The regulator and the applicant expect more delinquencies to never earn more points; monotone bins guarantee it, and they handle non-linearity (Q20) and missing values as their own bin. "Elastic net?" → Keeps correlated groups together instead of picking one.

Q32 · Logistic or gradient boosting?

On your lender's data, a gradient-boosted model beats the logistic scorecard by 2 AUC points on the validation set. Risk wants the GBM, compliance wants the scorecard, engineering wants whatever is simpler to run. You're asked for a recommendation.

• Concept tested: trade-offs: accuracy, calibration, explainability, adverse action, monotonicity, stability, governance.
• Intuition: 2 AUC points is worth money only if it survives a confidence interval, a time split, and the cost of explaining it.
• Answer: first check the 2 points are real: a confidence interval on the AUC difference (Ch 5), on an out-of-time split, and translated into expected cost at the operating threshold. They often shrink: on the chapter's hockey-stick NL book a default HistGradientBoosting scored 0.880 against the linear logistic model's 0.883 (truth 0.889). If the gain is real, the usual options are, in order: (1) improve the logistic model with what the GBM found — splines, hinges, a few interactions (Q20); (2) a GBM with monotone constraints, recalibrated, with SHAP-based reason codes validated for stability; (3) champion–challenger in production. Under model-risk rules (the Fed's SR 11-7), the GBM needs the same documentation, validation and monitoring, plus evidence that its reasons are accurate.
• Why: logistic regression is calibrated in the large by construction, exactly decomposable for reason codes (Q12), monotone by sign, and stable under refits; a GBM ranks better when interactions and thresholds are strong but needs post-hoc calibration and explanation.
• Next they'll ask: "What would make you pick the GBM outright?" → A large, stable, out-of-time gain in expected cost, monotone constraints that satisfy the business, and reason codes that compliance signs off. "A hybrid?" → Logistic regression on GBM-derived bins, or a GBM for ranking and a logistic layer for the reported PD.

Q33 · Race isn't in the model, so it's fair?

The PD model excludes race, sex and age. A fairness review finds that approval rates differ sharply between groups. The model owner says: "The model can't discriminate — it never sees those variables." Respond.

• Concept tested: proxies; disparate impact testing; what logistic regression can and can't do.
• Intuition: removing a column doesn't remove the information if other columns carry it.
• Answer: excluding protected attributes ("fairness through unawareness") doesn't prevent disparate outcomes. ZIP code, certain merchants, income patterns or even the bureau score itself can be correlated with a protected group, and logistic regression will use them for whatever predictive signal they carry. US fair-lending review looks at outcomes too (disparate impact): measure approval-rate ratios and error rates by group (using proxies such as surname-and-geography methods when the attribute isn't collected), test whether each feature's contribution is justified by risk, and search for a less discriminatory alternative — a model with similar performance and smaller disparities.
• Why: a coefficient answers "how much does this feature move the log-odds, holding the others fixed", not "is this feature a stand-in for something we may not use"; and different fairness criteria (equal approval rates, equal error rates, calibration within groups) generally can't all hold at once when base rates differ.
• Next they'll ask: "Just drop ZIP code?" → It may lower the disparity or simply move the signal to other proxies; measure before and after. Ch 29 owns the trade-offs.

Q34 · Ten billion impressions a day

An ads team trains a click-through model on 10 billion impressions a day with a 0.5% click rate. To fit in the budget they keep every click but only 1% of non-clicks. How would you train logistic regression at this scale, and how do you get calibrated click probabilities back for the auction?

• Concept tested: logistic regression at scale (SGD/FTRL, hashing, L1); negative-downsampling recalibration.
• Intuition: keeping 1% of non-clicks multiplies everyone's odds by 100; divide by 100 on the way out.
• Answer: train online with SGD-style updates, typically FTRL-Proximal with L1 for sparsity (McMahan et al., 2013), on hashed sparse features (the hashing trick), with per-coordinate learning rates. Downsampling is case-control sampling with s_1 = 1, s_0 = w = 0.01: the training sample is 33.4% clicks, and every probability must be mapped back (He et al., 2014):

q

the corrected click probability for the auction

p

the model's predicted probability (here of a click), trained on the downsampled data

w

the share of non-clicks kept (0.01)

A training-sample prediction of 0.5 becomes q = 0.0099.

• Why: sampling on y shifts only the intercept, by \ln(s_1/s_0) (Card 12). Hook: sample on y, shift the intercept. In an auction, bids are probability × value, so a 100× inflated probability overbids by 100×.
• Next they'll ask: "What else breaks calibration here?" → The L1 penalty (Shrink), position and selection effects (only shown ads get labels: the Who problem again), and drift; monitor predicted vs observed CTR by segment and recalibrate continuously. "Why not a deep model?" → Many teams do; the last layer is still a logistic regression on learned features (Q39).

Q35 · Chargebacks as labels

A fraud model uses chargebacks as its label. About 15% of true fraud is never charged back (customers don't notice), and about 2% of legitimate transactions are charged back anyway ("friendly fraud"). How does this label noise distort the model, and what can you do about it?

• Concept tested: misclassified labels; the probability floor and ceiling; attenuation; the noise-aware likelihood (Card 15).
• Intuition: if 15% of fraud is always recorded as clean, no transaction can ever look more than 85% fraudulent in the labels.
• Answer: with flip rates \alpha_1 = 0.15 (fraud recorded clean) and \alpha_0 = 0.02 (clean recorded fraud), the observed label follows a squeezed curve:

P(\tilde y = 1 \mid \mathbf x)

the chance the recorded label says fraud, for a transaction with features \mathbf x

\tilde y

the recorded (chargeback) label, which can be wrong; tilde = noisy version of the truth

\alpha_0

the chance a legitimate transaction is charged back anyway (0.02)

\alpha_1

the chance true fraud is never charged back (0.15)

\sigma

the sigmoid: turns a score into a probability

Running these flip rates on the chapter's loan book (LN variant): the naive fit flattens the slopes to 0.79, 0.80 and 0.86 of the clean fit, and still predicts up to 91.6% where the labels can't exceed 85%, because a logistic curve has no ceiling. A noise-aware likelihood with known \alpha's recovers (−2.61, −1.53, 0.621, −0.489) against the truth (−2.6, −1.5, 0.6, −0.5). And for fraud the floor matters most: at a true 1% fraud rate, the labeled rate is 0.02 + 0.83 \times 0.01 = 2.8\%, so most "fraud" labels are friendly fraud.

• Why: label noise is a How problem in Who, What, How, When: the outcome is measured badly, and unlike noise in a regression y it biases a classifier's slopes toward zero (Hausman, Abrevaya and Scott-Morton, 1998).
• Next they'll ask: "Where do the \alpha's come from?" → An audited sample (manual investigation of a random set of transactions), or chargeback reason codes; without them, report the model as predicting chargebacks, not fraud. "Does the ranking survive?" → Mostly: the observed probability is a monotone function of the true one, so the true ordering is preserved and the naive fit's AUC suffers far less than its probabilities do.

Q36 · Same ranking, wrong probabilities

After a change in the rate environment, the book's default rate falls from 11.9% to 6.6%. The PD model's AUC on new loans is unchanged at 0.81, but its average prediction is still 11.9% and the expected-loss forecast is far too high. The team proposes a full rebuild. What do you do instead?

• Concept tested: prior (base-rate) shift; intercept recalibration (Card 13).
• Intuition: everyone's risk fell by about the same factor, so the order is the same and only the level is wrong.
• Answer: correct the intercept by the change in log prior odds. On the chapter's SHIFT variant this moves every score by −0.651, cuts the ECE from 0.053 to 0.006, and leaves the AUC at 0.8115:

\eta_{\text{new}}

an applicant's corrected score (log-odds) for the new book

\eta_{\text{old}}

the old model's score for that applicant

\pi'

the new base rate: 6.57% default

\pi

here the old base rate the model was fit on (11.88%), not the circle constant

• Why: the formula is exact under label shift (P(x \mid y) fixed, only the base rate moves): Bayes on odds again (Card 13). Hook: Shrink, Sample, Shift — this is Shift.
• Next they'll ask: "Is it exact here?" → No, and say why: this variant changed P(y \mid x) with P(x) fixed (the true intercept fell by 0.8), so a 20% prediction should become 10.1% while the correction gives 11.5% (Drill 17). Close, not exact; with mature new labels, refit the intercept (or Platt) instead. "Keep the old threshold?" → Either correct the probabilities and keep t^*, or move the threshold by the same log-odds; never both.

Q37 · Small data: LDA, Naive Bayes or logistic?

A new product line has 300 loans and 25 defaults. A colleague says: "With this little data, use LDA — generative models are more efficient." The features are credit score, utilization (bounded, skewed), and a few binary flags. What do you fit?

• Concept tested: generative vs discriminative; LDA ⇒ logistic; efficiency vs robustness (Card 18).
• Intuition: generative draws the classes, discriminative draws the line: LDA gets the line for free if its picture of the classes is right, and gets it wrong if not.
• Answer: LDA's efficiency gain is real only when its assumption holds: two Gaussian classes with a shared covariance, in which case its posterior is exactly a logistic regression. In the chapter's LDA variant (one Gaussian feature, \pi_1 = 0.2), the logistic slope's variance at n = 200 was only 1.12× LDA's. Skewed and binary features break the Gaussian assumption, and then LDA's line is biased: with unequal class variances the true log-odds are quadratic (x^2 coefficient 0.375), and LDA's forced linear fit gave slope 0.60. With 25 events the real constraint is the E in L-I-N-E-S: few parameters, and a penalty or Firth's correction against the small-sample bias (+18% at n = 50, Drill 6).

y

the true class: 1 = default, 0 = repaid here

k

either class, 0 or 1

\sim

is distributed as

N(\boldsymbol\mu_k, \Sigma)

a normal (Gaussian) distribution with mean \boldsymbol\mu_k and covariance \Sigma

\boldsymbol\mu_k

the average feature vector of class k (bold = vector)

\Sigma

the covariance matrix of the features, shared by both classes

P(y = 1 \mid \mathbf x)

the chance an applicant with features \mathbf x defaults

P(y = 0 \mid \mathbf x)

the chance they repay

\mathbf x^\top

the features laid flat as a row (transpose: rows become columns)

\Sigma^{-1}

the matrix inverse of \Sigma: undoes it

\boldsymbol\mu_1

the average feature vector of defaulters

\boldsymbol\mu_0

the average feature vector of repayers

• Why: generative models estimate more (the class densities) and converge faster if right (Ng and Jordan, 2002; Efron, 1975); discriminative models assume less and stay consistent whatever x's distribution, as long as the log-odds are linear.
• Next they'll ask: "And Naive Bayes?" → A fast baseline that can beat logistic regression at very small n, but it double-counts correlated features, so its probabilities are overconfident; recalibrate if you use them (Ch 8). Also consider borrowing strength from the main book: fit there and recalibrate the intercept on the new line.

Q38 · A softmax over 100,000 classes

A team predicts merchant category from transaction text over 100,000 fine-grained categories, with a final softmax layer on a 512-dimensional embedding. Profiling shows the output layer dominates training time. Why, and what are your options?

• Concept tested: the cost of the softmax; sampled and hierarchical softmax; label smoothing and temperature.
• Intuition: to give one class its probability, the full softmax has to score and sum every other class too.
• Answer: each example needs all K = 100{,}000 logits for the denominator, and the gradient p - e_y is nonzero for every class: O(K\cdot d) per example, here 100{,}000 \times 512 \approx 5\times 10^7 multiply-adds, more than the rest of a small network. Options: sampled softmax (score the true class plus a few thousand sampled negatives, with a log-Q correction for the sampling distribution); negative sampling (binary logistic losses on positives vs sampled negatives); hierarchical softmax (a tree of binary decisions, O(\log K)); an adaptive softmax (full softmax for frequent classes, cheaper heads for the tail). At inference, top-k retrieval by maximum inner product search avoids scoring every class.

p_k

the predicted probability of category k

k

which category

e^{\eta_k}

e (about 2.718) raised to category k's logit: always positive

\eta_k

category k's logit (score)

\sum_{j=1}^{K}

add up over all K categories

j

a category index that runs over every category

K

the number of categories (100,000)

e^{\eta_j}

e raised to category j's logit

\frac{\partial(-\ln p_y)}{\partial \eta_k}

how fast the loss changes as you nudge category k's logit, holding the rest fixed: a slope, not a fraction to divide

p_y

the probability given to the true category

y

the true category

\mathbb 1[k = y]

1 if k is the true category, else 0

• Why: the softmax's normalizer couples all classes (Card 17); approximations replace the sum with an estimate, which is why they need a correction to stay unbiased.
• Next they'll ask: "Calibration with 100k classes?" → Temperature scaling (one parameter, Platt without the intercept) on held-out data; label smoothing during training also curbs overconfidence but changes what the probabilities mean. This is exactly the output layer of a language model (Q39).

Q39 · "An LLM is just logistic regression?"

A research interviewer: "Defend or attack this claim: next-token prediction in a large language model is just multinomial logistic regression." Take a position and support it with specifics.

• Concept tested: softmax on learned features; cross-entropy and perplexity; calibration; Bradley–Terry reward models and DPO as logistic losses.
• Intuition: the last layer is this chapter's model; everything below it is what this chapter can't do.
• Answer: defend the last layer, attack the whole. Given the final hidden state h, the next-token distribution is a softmax of Wh over a 50k–200k vocabulary: multinomial logistic regression, trained with this chapter's cross-entropy, with gradient p - e_y on the logits. Perplexity is e^{\text{loss}}: a loss of 2.0 nats is a perplexity of 7.39, and a uniform guess over 50,000 tokens has loss 10.82. Sampling temperature divides the logits, as in temperature scaling. Reward models are Bradley–Terry logistic regressions on reward differences, and DPO's loss is a logistic loss on a difference of log-probability ratios (Rafailov et al., 2023). But h is learned jointly with W, so the problem is non-convex; the features are the model; and the data are a dependent sequence, not i.i.d. rows.

P(\text{token}_t = k \mid \text{context})

the chance the next token at position t is vocabulary entry k, given the text so far

t

the position in the text

k

a vocabulary entry (token id)

e^{\mathbf w_k^\top \mathbf h_t}

e raised to token k's logit: its output weights times the hidden state, summed

e^{\mathbf w_j^\top \mathbf h_t}

the same for token j

\mathbf h_t

the final hidden state at position t: the learned features

\sum_{j}

add up over every token j in the vocabulary

P(A \succ B)

the chance a rater prefers response A over response B

r(A)

the reward model's score for response A

r(B)

the reward model's score for response B

• Why: cross-entropy is the likelihood of a categorical outcome, and minimizing it minimizes the KL divergence from the data's distribution to the model's (Card 2, Card 17). What logistic regression's guarantees (convexity, calibration in the large by the score equation) rely on is a fixed design matrix, which an LLM doesn't have.
• Next they'll ask: "Are LLMs calibrated, then?" → Pretrained models tend to be reasonably calibrated on next-token probabilities; preference tuning tends to make them overconfident, which temperature scaling partly fixes. "Where else does logistic regression appear?" → Linear probes on hidden states, and the logit lens.

Level 4 — Research and quant

These questions hand you a data-generating process (DGP): how the rows were drawn, sampled, labeled or selected. Then they ask what a logistic regression fitted to those rows converges to. There's one method, and every problem below uses it.

The recipe: score to zero, in expectation.

1. Write the true P(y = 1 \mid x) for the rows you actually fit. Do it after the sampling, the label noise, the omitted variable or the selection, not before.
2. Write the model's score. For logistic regression it is \sum x_i(y_i - \sigma(x_i^\top w)): features × misses (Card 3).
3. Set the expected score to zero. The MLE converges to the w^* that solves \operatorname{E}[x\,(P(y = 1 \mid x) - \sigma(x^\top w^*))] = 0.
4. If the true P is still logistic in x, read w* off exactly. That happens under case-control sampling, class weights and prior shift. If it isn't, approximate or simulate. That happens with omitted variables, label noise and misspecification. The tools are the probit approximation, a Taylor expansion, or a numerical solve.
5. Name the mechanism. Then say how you'd detect it, fix it, and whether it matters for the goal: ranking, probabilities or a causal effect.

\hat{\mathbf w}

the fitted weights (hat = estimated, bold = vector)

\mathbf w^*

what the fitted weights settle on as the rows grow (star = the limit)

\mathbb E

the average over the population the rows are drawn from

\mathbf x

an applicant's features as a vector, with a 1 for the intercept

P(y = 1 \mid \mathbf x)

the true chance of default for features \mathbf x, in the rows you fit

\sigma

the sigmoid: turns a score into a probability

\mathbf x^\top \mathbf w^*

the limiting model's score for features \mathbf x (^\top = transpose)

\mathbf 0

a vector of zeros

Ch 1's "Cov over Var, then expand" is this same recipe for a linear model, whose population score equation is \operatorname{E}[x(y - x^\top w)] = 0. The derivations use the L-O-G-I-T toolkit. Bayes on Odds does the exact cases. Taylor and the probit approximation do the rest.

Worked problem 1 — Oversampled defaults (case-control)

A staple of fintech credit loops. Defaults are rare, so modeling teams keep every default and only a fraction of repayers.

DGP: the 20,000-applicant book. The true model is \operatorname{logit} P(\text{default}) = -2.6 - 1.5z + 0.6d - 0.5\ell, where z is credit score in hundreds above 680, d is DTI in tens of points above 36%, and \ell is log income relative to $75k. The base rate is 12.0%. You keep every default and 20% of repayers (s_1 = 1, s_0 = 0.2) and fit logistic regression on (z, d, \ell). What do the fitted intercept and slopes converge to? Prove it. How do you get production probabilities back, and what changes if the model is misspecified?

Intuition first: sampling on y changes how common defaults look, not how much a credit score shifts the odds. Every applicant's odds get multiplied by the same factor, 1/0.2 = 5. A constant factor on the odds is a constant on the log-odds, and the intercept absorbs it.

Solution. Step 1 of the recipe: write the true P(y = 1 \mid x) among the rows you kept. Let S = 1 mean "kept". Bayes on odds:

P(y = 1 \mid \mathbf x, S = 1)

the chance of default for features \mathbf x, among the rows kept

P(y = 0 \mid \mathbf x, S = 1)

the chance of repayment, among the rows kept

S

1 if the row was kept in the sample, else 0

P(S = 1 \mid y = 1)

the chance a default is kept: s_1

P(S = 1 \mid y = 0)

the chance a repayer is kept: s_0

P(y = 0 \mid \mathbf x)

the chance of repayment in the full book

s_1

the share of defaults kept (1)

s_0

the share of repayers kept (0.2)

e^{\beta_0 + \mathbf x^\top\boldsymbol\beta}

the true odds of default: e (about 2.718) raised to the true log-odds; \beta_0 = -2.6 is the true intercept

\mathbf x^\top\boldsymbol\beta

the true slopes' part of the score: features times true slopes, summed

\boldsymbol\beta

the true slopes (−1.5, 0.6, −0.5) as a vector

Take logs. The kept rows still follow a logistic model in x, with the same slopes and a shifted intercept:

\beta_0 + \ln\frac{s_1}{s_0}

the kept rows' intercept: the true intercept \beta_0 plus the sampling offset

Recipe step 4: the sampled P is still exactly logistic in x, so the expected score is zero at w^* = (\beta_0 + \ln 5, \beta). The MLE is consistent for it. Sampling depends only on y, so it never touches the x-pattern inside each class (Card 12).

What to say out loud: the slopes still converge to the truth. Only the intercept moves, by \ln(s_1/s_0) = \ln 5 = 1.609. On one sample (5,896 rows kept, 2,406 of them defaults, sample base rate 40.8%), the fit gives an intercept of −0.985 and slopes (−1.49, 0.57, −0.48). The truth is −0.991 and (−1.5, 0.6, −0.5). Across 200 case-control samples the intercept shift averages 1.608 (sd 0.039) against the theoretical 1.609. The score slope averages −1.501 (sd 0.077). To get production probabilities, subtract \ln 5 from the intercept. Uncorrected, the model's mean \hat p on the full book is 32.5%. Corrected, it is 11.9%, against a true base rate of 12.0%. Hook: sample on y, shift the intercept.

Simulated on the main book's DGP with sim_casecontrol.py. Ch 1 Drill 27 did the one-number version of this problem. This is the proof, with the full feature set.

Follow-ups that separate senior from research-level answers:

• "What did you pay for throwing away 70% of the rows?" Less than you'd think. The sample keeps 29% of the rows, yet the score slope's SE only rises from 0.052 (full book) to 0.069, so the sample keeps about 57% of the information. The rows you dropped were confident repayers, and each carried little information, p(1 - p) (Card 9). That is why case-control designs exist.
• "Why not weight instead of shifting the intercept?" Weighting each kept repayer by 1/s_0 = 5 also works: the weighted fit gives intercept −2.593 and score slope −1.471. If the model is correct, the offset is more efficient. If it is misspecified, weighting is more robust: it targets the same best approximation as a fit on the full population (King and Zeng, 2001).
• "And if the model is misspecified?" Then the shift is no longer clean. Take the nonlinear book, where risk accelerates above 43% DTI, and fit it with a linear term. Case-control sampling moves the intercept by 1.709 instead of 1.609, and the DTI slope moves by −0.248 (from 1.548 to 1.300). The corrected mean \hat p is 16.9% against a true 17.3%. A misspecified model's best approximation depends on the mix of x, and sampling on y changes that mix. Under the correct model, at large n, the shift is 1.611 and every slope moves by less than 0.01.
• "Contrast with linear regression." In OLS, selecting on y biases the slope (Ch 1 Drill 4: 0.53 instead of 1). Logistic regression is special because odds ratios are symmetric in x and y: the odds ratio of y given x equals the odds ratio of x given y. Sampling on y preserves the second, so it preserves the first.
• "You don't know s_0, only the population base rate \pi." Solve for it: s_1/s_0 = [\pi_s/(1 - \pi_s)] / [\pi/(1 - \pi)], where \pi_s is the sample base rate (Card 13). If you know neither, you can rank applicants and report odds ratios, but you cannot recover absolute risk.
• "Small samples and rare events?" The MLE is biased away from zero in small samples (Drill 6). King and Zeng's rare-events correction adds a Firth-type bias correction on top of the prior correction.
• "Where else is this used?" Ads, where the negatives are downsampled. Keep 1% of non-clicks and the recalibration is q = p / (p + (1 - p)/w) with w = 0.01. A raw sample prediction of 0.5 becomes 0.0099 (Q34).

How to identify it in real data: the training base rate doesn't match the portfolio (40.8% vs 12.0%). The model's mean \hat p in production sits far above the observed default rate, while its ranking (AUC) looks fine. Check the data lineage for "all bads, sample of goods" pulls.

How to mitigate it:

1. Offset: subtract \ln(s_1/s_0) from the intercept, or pass it as an offset when fitting.
2. Weights at fit time: weight the kept repayers by 1/s_0. This is safer when you doubt the functional form.
3. Recalibrate on a random, unsampled holdout: refit the intercept with the slopes fixed, or apply Platt scaling (Section 9).
4. Document the sampling rates with the model, so that whoever deploys it can undo them.

Worked problem 2 — The randomized reminder and the independent covariate (non-collapsibility)

Asked in research-scientist and experimentation loops. It tests whether you know that "randomized" does not make the odds ratio the same with and without controls.

Setup: 20,000 borrowers are randomized 50/50 to an SMS payment reminder, T. A prognostic risk score Z \sim N(0, 1.5^2) is measured before randomization, so it is independent of T. The true model is \operatorname{logit} P(\text{default}) = -2.2 - 0.4\cdot T + 1.0\cdot Z, and the base rate is 14.8%. You fit (a) \text{default} \sim T and (b) \text{default} \sim T + Z. What does each treatment coefficient converge to? Is (a) biased? Which analysis has more power?

Intuition first: logistic coefficients live on a scale set by the noise the model can't explain. Leave Z out and its variation joins that noise. Then every coefficient shrinks toward zero to stay on the logistic scale. There is no confounding here: Z is balanced across arms by design. The shrinkage happens anyway.

Solution. Analysis (b) is correctly specified, so the coefficient on T \to -0.4. For (a), recipe step 1: average Z out of the true model within each arm.

P(y = 1 \mid T)

the chance of default in an arm: T = 1 reminder, T = 0 control

T

the treatment: 1 if the borrower got the SMS reminder, else 0

\mathbb E_Z

the average over the risk score Z's values

Z

the prognostic risk score, independent of T

\sim

is distributed as

N

a normal (bell-curve) distribution; its arguments are the mean and the variance

With one binary regressor the fit is saturated, so the coefficient is exactly the log-odds ratio of these two arm-level probabilities (Drill 2). Recipe step 4: the probit approximation \sigma(t) \approx \Phi(t\cdot\sqrt{\pi/8}) makes the integral solvable, because averaging a normal CDF over a normal shift just widens it (Card 14):

a + \gamma Z

the true score: a is everything kept in the model (-2.2 - 0.4T), \gamma Z the left-out part

\frac{a}{\sqrt{1 + \pi\gamma^2\sigma_Z^2/8}}

the kept score a, shrunk by the extra spread that Z adds

\gamma

Z's coefficient in the true model (1.0)

\pi

the circle constant, 3.14159

\sigma_Z^2

the variance of Z (2.25); here \sigma is a standard deviation, not the sigmoid

\hat\beta_T^{(a)}

the reminder's fitted coefficient in analysis (a), which leaves Z out

The exact arm probabilities are 16.8% (control) and 12.8% (reminder), so the exact limit is \operatorname{logit}(0.128) - \operatorname{logit}(0.168) = -0.312. Over 500 simulated trials, analysis (a) averages −0.314 and (b) averages −0.402. The \pi/8 formula undershoots by about 7% at this 15% base rate. Ch 1's latent-variance match, factor \sqrt{3.29/(3.29 + 2.25)} = 0.771, gives −0.308 and is closer here. Near a 50% base rate, \pi/8 is the closer one. Neither approximation is uniformly better.

What to say out loud: both estimates are right; they answer different questions. (b) is the conditional log-OR: the reminder's effect for borrowers with the same Z, −0.40. (a) is the marginal, population-average log-OR, −0.31. Neither is "biased" for its own estimand, and they differ although Z is independent of T. That is non-collapsibility. The risk difference doesn't move: −3.94 points unadjusted and −3.94 points as an average marginal effect from (b). Risk differences are collapsible; odds ratios are not.

Which has more power. Adjustment makes the treatment coefficient's SE larger, 0.0455 against 0.0401. It moves the estimate away from zero by even more: in all 500 replications, the adjusted estimate is further from zero. The net effect is a larger test statistic, z = -8.8 adjusted vs −7.8 unadjusted (Robinson and Jewell, 1991). At n = 20{,}000, both analyses reject in 100% of runs, so power can't tell them apart there. At n = 2{,}000, power is 80% adjusted vs 67% unadjusted, and at n = 4{,}000 it is 97% vs 93%.

Follow-ups that separate senior from research-level answers:

• "So which number goes in the launch memo?" The risk difference: defaults fall from 16.8% to 12.8%, about 3.9 points. It is collapsible, a PM can act on it, and adjustment makes it more precise. Across replications, its sd falls from 0.0049 unadjusted to 0.0043 as an adjusted AME. Report the conditional OR only if the audience wants the effect "for a borrower like this one".
• "Can I compare this OR with the one from last year's trial?" Only if both adjusted for the same covariates. ORs from studies with different covariate sets estimate different quantities. That makes meta-analyses of odds ratios treacherous (Gail, Wieand and Piantadosi, 1984).
• "Is this a logistic quirk?" Probit has the same issue (exactly, with Gaussian Z), and so do hazard ratios in Cox models. Linear models and risk differences don't.
• "Does the shrinkage depend on the base rate?" Yes. With \gamma\sigma_Z = 1.5, the attenuation ratio is 0.84 at a 5.5% base rate and 0.72 at 50% (Drill 10, Card 14). The \pi/8 formula ignores the intercept, which is why its error changes with the base rate.
• "How do I get a marginal OR with adjustment's precision?" Use standardization (g-computation). Fit (b), predict every borrower's \hat p with T set to 1 and then to 0, average each, and take the log-OR of the two averages. Use the bootstrap or the delta method for its SE.
• "And with a confounder instead?" Then (a) is confounded as well as non-collapsible, and the two effects add up. Randomization removes the first, never the second.

How to identify it in real data: first check the balance table. If the covariate is balanced across arms (here \operatorname{Corr}(T, Z) \approx 0) and the OR still moves when you add it, the cause is non-collapsibility, not confounding. Confirm it: the AME or risk difference stays put while the OR moves.

How to mitigate it:

1. Pre-register the covariate set in the analysis plan, so that nobody shops for the larger OR.
2. Report the risk difference or AME as the headline. Give the conditional OR as a secondary estimand and name it as such.
3. Use standardization to report a marginal effect on any scale with covariate-adjusted precision.
4. Never pool ORs across models with different controls.

Remember: score to zero, in expectation. If the sampled P is still logistic in x, the answer is exact: sampling on y moves only the intercept. If not, approximate it: an omitted variable shrinks every logistic coefficient, even an independent one, while the risk difference stays put.

Level 4 drills

Twenty problems. Drills 1–6 compute the likelihood and its information by hand. Drills 7–10 treat biased rows, Drills 11–14 scales and generative models, Drills 15–17 calibration and decisions, and Drills 18–20 information, tests and GLMs. Cover the answer, run the recipe on paper, then check. Every answer was checked by exact computation (sims/exact.py, or a deterministic calculation in sims/drills.py) or by simulation. "Verified by simulation" marks the second.

The likelihood by hand — Drills 1–6

Drill 1 · No features at all

The credit committee's ten files have four defaults. Before adding any feature, the interviewer wants the baseline: "Fit logistic regression with only an intercept. Give me \hat{\beta}_0 and its standard error, by hand, without software. Then tell me what the model predicts for every applicant."

Concept tested: the MLE of a Bernoulli on the logit scale; Fisher information.

• Answer: \hat{\beta}_0 = \operatorname{logit}(0.4) = \ln(0.4/0.6) = -0.405, with \operatorname{SE} = 1/\sqrt{10 \times 0.4 \times 0.6} = 0.645. Every applicant gets \hat p = 0.4. A numerical fit matches both numbers.
• Intuition: with no features, the best a model can say is "the share who defaulted", and the log-odds are just that share on the logit scale.
• Why: the intercept's score equation is \sum(y_i - \sigma(\beta_0)) = 0, so \sigma(\hat{\beta}_0) = \bar y. The information is n\cdot p(1 - p) (Card 9).

\sum_i

add up over every applicant i

i

which applicant

y_i

applicant i's outcome: 1 = default, 0 = repaid

\beta_0

here the intercept, the model's only parameter

\hat\beta_0

the fitted intercept (hat = estimated)

\bar y

the share of rows that defaulted (bar = average): 0.4 in the ten files

n

the number of rows: 10 files here

Drill 2 · One binary feature

Borrowers who enroll in autopay default less. On the 20,000-applicant book: 1,057 defaults and 10,313 repayments with autopay; 1,349 defaults and 7,281 repayments without. "What's the logistic coefficient on an autopay dummy, and its SE, without software? Then: the true DGP gives autopay no direct effect. Enrollment just rises with credit score. What happens when you add score?"

Concept tested: the MLE with one binary feature is the log cross-product ratio; confounding, as opposed to non-collapsibility.

• Answer: the coefficient is \ln[(1{,}057 \times 7{,}281)/(10{,}313 \times 1{,}349)] = -0.592, an odds ratio of 0.553, with \operatorname{SE} = \sqrt{1/1{,}057 + 1/10{,}313 + 1/1{,}349 + 1/7{,}281} = 0.0438. The intercept is \ln(1{,}349/7{,}281) = -1.686, the log-odds without autopay. A statsmodels fit matches all three. Adding score moves the autopay coefficient to −0.026 (SE 0.050, p = 0.60): it was all confounding. Verified by simulation (drills.py, seed 320).
• Intuition: with one dummy, the model has two free numbers for two groups, so it reproduces each group's default rate exactly. The coefficient is the log-odds gap between the groups.
• Why: the fit is saturated, so \hat p in each group equals its observed rate (Card 3, the one closed-form case). The SE is the delta-method SE of a log-OR from a 2×2 table. Here the move after adding score is confounding: autopay depends on score. In WP2 the covariate was independent of treatment and the OR still moved. Learn to tell the two apart with a balance check.

\hat\beta_1

the fitted autopay coefficient: the log odds ratio of the 2×2 table

\frac{a\,d}{b\,c}

the cross-product ratio: (defaults with autopay, 1,057 × repayments without, 7,281) over (repayments with, 10,313 × defaults without, 1,349)

\text{SE}(\hat\beta_1)

its standard error

\sqrt{\tfrac{1}{a} + \tfrac{1}{b} + \tfrac{1}{c} + \tfrac{1}{d}}

the SE of a log odds ratio: root of the summed inverse cell counts

Drill 3 · How many loans for a precise intercept?

Finance wants this year's portfolio default rate pinned down tightly on the log-odds scale, the scale the model's intercept lives on. "The base rate is about 12%. How many loans do you need for a 95% interval of ±0.05 on the log-odds of default?"

Concept tested: Bernoulli Fisher information on the logit scale; sample size.

• Answer: n = 1.96^2 / (0.05^2 \times 0.12 \times 0.88) \approx 14,551 loans. Verified by simulation: 95.1% of 20,000 simulated portfolios land within \pm 0.05. The sd of the log-odds estimate is 0.0254, against the theoretical 0.0255.
• Intuition: each loan carries information p(1 - p) = 0.106 about the log-odds. At a 12% base rate that is less than half the 0.25 a 50/50 outcome carries, so you need more than twice as many loans.
• Why: \operatorname{Var}(\operatorname{logit} \hat p) \approx 1/(n p(1 - p)) (Card 9). Set 1.96 \times \operatorname{SE} = 0.05 and solve for n.

n

the sample size: here, how many loans you need

\pi

here the base rate, about 12% (not the circle constant)

Drill 4 · Where to spend a test-below-cutoff budget

To learn the true slope of default on score, risk may approve a small randomized batch of applicants, at two score levels with half the batch at each. "If you could choose the two levels freely, at what predicted default probabilities should they sit to make the (intercept, slope) estimate as precise as possible, in the D-optimal sense? Then tell me why the answer is uncomfortable in lending."

Concept tested: Fisher information \propto p(1 - p)\cdot x^2; optimal design for logistic regression.

• Answer: put the two points at log-odds \eta = \pm 1.543, that is, p = 17.6\% and 82.4%. This is an exact numerical optimization in exact.py. The uncomfortable part: no lender wants half a test batch at an 82% predicted default rate. In practice the design is cost-constrained, keeps the upper point much lower, and accepts a wider interval.
• Intuition: rows near 50% carry the most information each, but two points close together give no leverage on a slope. Points far apart give leverage, but rows near 0% or 100% carry almost no information. ±1.54 on the log-odds scale is the compromise.
• Why: in canonical coordinates, the determinant of the information matrix is proportional to w_1w_2(\eta_1 - \eta_2)^2, where w = p(1 - p). For a symmetric design \pm c, both weights equal w(c), so the determinant is \propto 4c^2\cdot w(c)^2. Maximizing it is the same as maximizing c\cdot\sigma(c)(1 - \sigma(c)). Setting the derivative to zero gives c\cdot\tanh(c/2) = 1, so c = 1.543. (A common slip is to maximize c^2\cdot w(c), which gives c\cdot\tanh(c/2) = 2 and the wrong point, c \approx 2.4.)

\frac{d}{dc}

how fast what follows changes as you nudge c: a slope, not a fraction to divide

c\,\sigma(c)\big(1 - \sigma(c)\big)

the quantity to maximize: c, the design points' log-odds (\pm c), times the information p(1 - p) there

1 + c\,\big(1 - 2\sigma(c)\big)

that derivative written out, set to zero

c\,\tanh(c/2)

the same condition rewritten with the hyperbolic tangent

c = 1.543

the answer: design points at log-odds \pm 1.543, i.e. p = 17.6% and 82.4%

Drill 5 · DTI above 39%

In the committee's ten files, every applicant with DTI above 39% defaulted (A 48, B 42, D 45, F 40) and every one below repaid (C 30, E 38, G 25, H 33, I 22, J 18). A junior fits \text{default} \sim \text{DTI}. "What does the MLE do? Write the log-likelihood along the separating direction. What does Firth's method give instead?"

Concept tested: complete separation; the MLE does not exist; Firth's penalized likelihood.

• Answer: the MLE runs to infinity. Along \eta = c\cdot(\text{DTI} - 39)/10, the log-likelihood rises toward 0 as c grows, and never reaches it: −6.93 at c = 0, −3.90 at 1, −1.27 at 5, −0.26 at 20, −0.013 at 50. IRLS on DTI alone grows \lVert w\rVert about 1.6–1.8× per iteration for the first five iterations, then by a constant +10.8 per iteration. X^\top WX turns numerically singular at iteration 41. Firth's fit is finite: (−0.827, 2.826) on d = (\text{DTI} - 35)/10, with SEs (1.12, 1.77).
• Intuition: if a line splits the defaults from the repayers perfectly, you can always make the model more confident by making the line steeper, so there's no best steepness.
• Why: every row's margin \tilde y_i\cdot(\text{DTI}_i - 39)/10 is at least 0.1 (DTI 38 and 40 are the closest), so each loss term \ln(1 + e^{-c\cdot\text{margin}}) falls to 0 as c \to \infty. Firth maximizes \ell(w) + \tfrac12 \ln\lvert I(w)\rvert. The Jeffreys penalty falls to -\infty as the weights W \to 0, which stops the runaway (Card 11).

\ell(c\,\mathbf w^*)

the log-likelihood after stretching the separating direction c times

c

how far you stretch along the separating direction

\mathbf w^*

here a separating direction: weights that put every default on one side

\sum_{i=1}^{10}

add up over the ten files

e^{-c\,\tilde y_i (\text{DTI}_i - 39)/10}

e (about 2.718) raised to minus the stretched margin: shrinks to 0 as c grows

\tilde y_i

applicant i's label coded +1 (default) or -1 (repaid)

\text{DTI}_i

applicant i's debt-to-income ratio, in percent

\ell_{\text{Firth}}(\mathbf w)

Firth's penalized log-likelihood

\ell(\mathbf w)

the ordinary log-likelihood at weights \mathbf w

\mathbf w

the weights as one vector

\big|X^\top W X\big|

the determinant of the Fisher information: its overall size

X

the feature matrix: one row per file

X^\top

the transpose of X: rows become columns

W

a diagonal matrix of each file's p_i(1 - p_i)

Drill 6 · Small n, coefficients too big

A new product line has a few dozen loans. The analyst fits default on credit score alone and reports a steeper slope than the main book's. "Take the book's DGP with score as the only feature, so the true marginal slope is −2.03 per 100 points. Fit at n = 50, 100, 200, 1,000 and 5,000. Is the MLE unbiased? Which way does it miss, and is separation to blame?"

Concept tested: the MLE's small-sample bias away from zero; Firth's O(1/n) bias removal.

• Answer: the MLE is biased away from zero. The mean slope is −2.40 at n = 50 (+18.4%), −2.20 at n = 100 (+8.7%), −2.09 at n = 200 (+3.2%), −2.04 at n = 1{,}000 (+0.5%) and −2.03 at n = 5{,}000. Separation is not the cause: only 0.3% of the n = 50 samples separate (excluded from the MLE mean), and none at larger n. Firth's mean is −2.01 at n = 50 (−0.9%), and its sd is lower (1.06 vs 1.63). Verified by simulation (2,000 replications per n).
• Intuition: with about 6 defaults in 50 rows, a few lucky draws make the classes look cleaner than they are. The likelihood rewards that with a steeper slope, and nothing pulls it back.
• Why: the MLE's bias is O(1/n): it roughly halves each time n doubles (18.4 → 8.7 → 3.2). Firth's modified score subtracts that leading term. It adds h_i(\tfrac12 - p_i) to each residual, where h_i is the leverage from the weighted hat matrix (Card 11). Note that −2.03 is steeper than the full model's −1.5: z alone absorbs DTI and income, which are correlated with it.

U^*(\mathbf w)

Firth's modified score: the gradient of the penalized log-likelihood

\mathbf x_i

applicant i's features as a vector

p_i

applicant i's fitted probability of default

h_i

applicant i's leverage: the i-th diagonal entry of the weighted hat matrix in brackets

W^{1/2}

the square root of W: each row's \sqrt{p_i(1 - p_i)}

(X^\top W X)^{-1}

the matrix inverse of the information: undoes it

Biased rows — Drills 7–10

Drill 7 · Rebuilding the true base rate

A teammate undersampled repayers until the training set was 50/50 "for balance". The model now says p = 0.30 for an applicant. The portfolio default rate is 10%. "What probability should production use?"

Concept tested: prior correction on the odds scale (Cards 12–13).

• Answer: p \approx 0.0455. The model's odds are 0.30/0.70 = 0.4286. Multiply by the change in prior odds, (0.1/0.9)/(0.5/0.5) = 1/9, to get odds of 0.0476, so p = 0.0455.
• Intuition: the model learned a world where defaults are 50% of rows. Undo that by scaling its odds by how much rarer defaults really are.
• Why: sampling on y multiplies every row's odds by the same factor (WP1), so dividing it back out is exact if the model is otherwise right.

p

an applicant's probability of default

Drill 8 · Weights equal sampling

*Instead of resampling, a teammate passes class_weight="balanced" on the book, whose base rate is \pi = 12.0\%; the model is otherwise correct. "Without refitting, by how much does the intercept move? What happens to the slopes, the AUC and the average predicted default rate?"*

Concept tested: the weighted score equation; class weights act like case-control sampling.

• Answer: the intercept moves by \ln(w_1/w_0) = \ln((1 - \pi)/\pi) = \ln 7.31 = 1.99. The sim gives 2.005. Slopes move by at most 0.03, the AUC is unchanged (0.8118 vs 0.8118), and the mean \hat p jumps to 38.9% against a 12.0% base rate. Verified by simulation on the book.
• Intuition: giving each default 7.31 times the weight of a repayer is the same as seeing each default 7.31 times. That is case-control sampling with s_1/s_0 = 7.31.
• Why: balanced weights are w_k = n/(2n_k), so w_1/w_0 = n_0/n_1 = (1 - \pi)/\pi (here 4.156/0.568). In expectation, the weighted score equation is the ordinary score equation for a population whose odds are multiplied by w_1/w_0. Recipe step 4 applies exactly, and the shift lands in the intercept.

w_y

the class weight on a row with label y: w_1 for defaults, w_0 for repayers

y

the row's label: 1 = default, 0 = repaid

\mathbf x^\top\mathbf w

the model's score for features \mathbf x

P_w(y = 1 \mid \mathbf x)

the default probability in the reweighted population the fit sees

\beta_0

the true intercept

w_1

the class weight on each default, n/(2n_1) = 4.156

w_0

the class weight on each repayer, n/(2n_0) = 0.568

Drill 9 · 10% of labels flipped both ways

A servicing system mislabels 10% of defaults as repaid and 10% of repayments as defaults. "Let the true model be \operatorname{logit} P(y = 1) = x with x \sim N(0, 1), and flip each label with probability 0.1. What is the naive logistic slope? What's the largest probability the observed labels can ever show?"

Concept tested: misclassified labels; a probability ceiling and floor; attenuation.

• Answer: the observed P(\tilde y = 1 \mid x) = 0.1 + 0.8\sigma(x) is trapped in [0.1, 0.9]. The naive slope converges to 0.743, a numerical population solve. Verified by simulation: 0.741 on a million rows. A quick local approximation gives 0.8. The naive logistic fit does not respect the ceiling either: at x = 3 it predicts 0.903, while the observed-label probability is 0.862.
• Intuition: flipped labels pull every probability toward the middle. The S-curve gets squeezed between a floor and a ceiling, and a squeezed S looks flatter.
• Why: recipe step 1 gives the observed P, which is no longer logistic, so step 4 needs a solve. Near x = 0 the slope of \operatorname{logit} q is q'(0)/(q(0)(1 - q(0))) = 0.8\cdot 0.25/0.25 = 0.8. In the tails q saturates at 0.1 and 0.9, which flattens the global fit further, to 0.743. With known flip rates, the noise-aware likelihood recovers the truth (Card 15).

P(\tilde y = 1 \mid x)

the chance the recorded label says default at feature value x

x

the single feature, x \sim N(0, 1)

\alpha_0

the chance a true repayment is recorded as a default (0.1)

\alpha_1

the chance a true default is recorded as repaid (0.1)

\frac{d\,\text{logit}\,q}{dx}

how fast the observed log-odds change as you nudge x: a slope, not a fraction to divide

q

the observed-label probability, P(\tilde y = 1 \mid x)

Drill 10 · The independent risk factor

A bureau attribute Z predicts default but is independent of credit score, and your model leaves it out. "Let \operatorname{logit} p = -2.6 - 1.5z + Z with Z \sim N(0, 1.5^2), independent of z. What does the slope on z converge to when Z is omitted? Compare the \pi/8 approximation with Ch 1's latent-variance one. Does the average marginal effect move?"

Concept tested: non-collapsibility; the accuracy of the probit approximation depends on the base rate.

• Answer: the sim (n = 20{,}000, base rate 14.5%) gives −1.153, and the large-n limit is −1.162, a ratio of 0.775. \pi/8 predicts -1.5 \times 0.729 = -1.093. The latent-variance match predicts -1.5 \times 0.771 = -1.156, which is closer at this base rate. The AME barely moves: −0.1348 with Z, −0.1344 without. Verified by simulation, with the limit computed by integrating Z out exactly.
• Intuition: omitting Z adds spread to the hidden risk. The logistic scale is fixed, so the model shrinks the slope to fit the wider spread.
• Why: \sigma(t) \approx \Phi(t\sqrt{\pi/8}), and averaging a normal CDF over a normal shift divides its argument by \sqrt{1 + \text{variance}}. The approximation ignores the intercept, but the true attenuation depends on it: the ratio is 0.84 at a 5.5% base rate and 0.72 at 50% (Card 14).

\beta_z^{\text{omit}}

the score slope when Z is left out

\beta_z

the score slope with Z in the model (−1.5)

\pi

the circle constant, 3.14159

\pi^2/3

the variance of the standard logistic noise, about 3.29

Scales, generative models and softmax — Drills 11–14

Drill 11 · Logit vs probit coefficients

Two teams model the same default flag on the same feature. One reports a probit coefficient of 0.9, the other a logit coefficient of 1.5. A reviewer flags the reports as inconsistent. "Are they? What ratio would you expect, and why does it vary?"

Concept tested: the latent scales of logit and probit.

• Answer: they are consistent: 1.5/0.9 = 1.67. Theory gives a band. Matching slopes at the center gives \sqrt{8/\pi} = 1.60. The best single scale over the whole curve (minimax) gives 1.70, with a maximum probability error of 0.0095. Matching the sds of the latent noise gives \pi/\sqrt{3} = 1.81. On the 20,000-applicant book the empirical ratios run from 1.79 to 1.92 (z 1.88, DTI 1.90, income 1.92). The ratio rises as the base rate moves away from 50%.
• Intuition: both models put noise on a hidden risk score. The logistic noise has sd 1.81 and the probit noise has sd 1, so logit coefficients come out bigger by about that factor. The tails differ, so the factor depends on where the data sit.
• Why: match the slopes at 0: \sigma'(0) = 1/4 and \phi(0) = 1/\sqrt{2\pi}, so \beta_{\text{logit}}/\beta_{\text{probit}} = 4\phi(0) = \sqrt{8/\pi}. In the tails the logistic curve is heavier, so a rare outcome stretches the ratio toward and past \pi/\sqrt{3} (Card 1).

\beta_{\text{logit}}

the coefficient from a logit fit

\beta_{\text{probit}}

the coefficient from a probit fit on the same data

\varphi(0)

the standard normal density at 0, 1/\sqrt{2\pi} \approx 0.399: the probit curve's slope at its center

\sigma'(0)

the sigmoid's slope at 0, which is 1/4

Drill 12 · Gaussian classes, shared variance

A researcher argues that modeling what defaulters look like beats modeling the boundary. "Suppose X \mid y = 0 \sim N(0, 1) and X \mid y = 1 \sim N(1, 1), with \pi_1 = 0.2. Write the exact posterior log-odds. What does logistic regression converge to? Which is more efficient at n = 200?"

Concept tested: LDA ⇒ logistic posterior; generative vs discriminative efficiency.

• Answer: the log-odds are exactly linear: \beta_1 = 1, \beta_0 = -0.5 + \ln 0.25 = −1.886. Logistic regression is correctly specified and converges to it. On 50,000 rows it fits (−1.886, 1.004), and LDA's plug-in fits (−1.885, 1.001). At n = 200, over 1,000 replications, the logistic slope's variance is 1.12 times LDA's. LDA is more efficient because it uses the Gaussian shape. Verified by simulation.
• Intuition: two bell curves with the same width cross along a straight line, so the boundary model and the class model describe the same thing. The class model also uses where the points sit, not just which side they're on.
• Why: take the log of the ratio of the two Gaussian densities. The x^2 terms cancel when the variances are equal. Efron (1975) showed that logistic regression's efficiency loss grows with the separation between the classes. At \Delta = 1 it is small (Card 18).

P(y=1\mid x)

the chance of default for an applicant with feature value x

P(y=0\mid x)

the chance of repayment

x

an applicant's feature value (one Gaussian feature here)

\pi_1

the share of defaulters (0.2)

\pi_0

the share of repayers (0.8)

\mu_1

the defaulters' mean of x (1)

\mu_0

the repayers' mean of x (0)

\sigma^2

the shared within-class variance of x (1); here \sigma is a standard deviation, not the sigmoid

Drill 13 · Unequal variances

Same researcher, new data: defaulters' scores are more spread out than repayers'. "Let X \mid y = 0 \sim N(0, 1) and X \mid y = 1 \sim N(1, 2^2), with \pi_1 = 0.2. Is the log-odds still linear in x? What feature must a logistic model add, and with what coefficient?"

Concept tested: QDA ⇒ quadratic log-odds; misspecification of a linear model.

• Answer: no. The log-odds are quadratic: 0.375x^2 + 0.25x - 2.204. Add x^2. A logistic fit with x and x^2 recovers (−2.214, 0.240, 0.382). The x^2 term's LR statistic is 5,978, so the linear model is decisively rejected. Both linear fits are wrong: logistic gives slope 0.553, and LDA, which assumes equal variance, gives 0.603. Verified by simulation (n = 50{,}000).
• Intuition: a wide bell and a narrow bell cross twice, so the high-risk region is both tails, not one side of a line.
• Why: with unequal variances, the x^2 terms no longer cancel. The coefficient on x^2 is \tfrac12(1/\sigma_0^2 - 1/\sigma_1^2) = \tfrac12(1 - \tfrac14) = 0.375. Logistic regression with the right features stays correct. LDA's equal-variance assumption can't be repaired by adding features.

N(x; 1, 4)

the normal density at x with mean 1 and variance 4: how defaulters' scores spread

N(x; 0, 1)

the normal density at x with mean 0 and variance 1: how repayers' scores spread

Drill 14 · Softmax by hand

Collections' three-outcome model (prepaid / on time / defaulted) outputs logits (2, 1, 0) for a borrower who turned out "on time", the second class. "Give me the probabilities, the loss, and the gradient of the loss with respect to the logits. What happens if a bug adds 5 to every logit?"

Concept tested: the softmax, its cross-entropy gradient p - e_y, and shift invariance.

• Answer: p = (0.665, 0.245, 0.090). The loss is -\ln 0.245 = 1.408. The gradient is p - e_y = (0.665, -0.755, 0.090). Adding 5 to every logit changes nothing: the probabilities are identical. This is exact.
• Intuition: the gradient pushes the true class's logit up by how much probability it lacks (0.755), and pushes every other logit down by the probability it wrongly took.
• Why: with log-sum-exp, -\ln p_y = -\eta_y + \ln \sum_j e^{\eta_j}, so \partial/\partial\eta_k = p_k - 1[k = y] (Card 17). A constant cancels between numerator and denominator, which is why one class can serve as the reference. With K = 2, the softmax is the sigmoid of the logit difference: logits (1.3, 0.2) give 0.750 = \sigma(1.1).

p_k

the predicted probability of outcome k (prepaid, on time, defaulted)

k

which outcome

e^{\eta_k}

e (about 2.718) raised to outcome k's logit: always positive

\eta_k

outcome k's logit (score)

\sum_j

add up over every outcome j

e^{\eta_j}

e raised to outcome j's logit

\eta_j

outcome j's logit

(e^2,\, e^1,\, e^0)

e raised to each of the three logits (2, 1, 0)

e^2 + e + 1

their sum, the softmax denominator

\frac{\partial(-\ln p_y)}{\partial \eta_k}

how fast the loss changes as you nudge outcome k's logit, holding the rest fixed: a slope, not a fraction to divide

p_y

the probability given to the true outcome (on time)

y

the true outcome

\mathbb 1[k = y]

1 if k is the true outcome, else 0

Calibration and decisions — Drills 15–17

Drill 15 · Does L2 break average calibration?

Model validation asks whether the regularized PD model still reproduces the portfolio default rate. "sklearn's lbfgs fits L2-penalized logistic regression with an unpenalized intercept. On the training data, is mean \hat p = \bar y exactly? What about the slope equations? And with liblinear, which penalizes the intercept, at C = 0.001?"

Concept tested: which score equations survive a penalty.

• Answer: with lbfgs, yes, exactly. On the calibration split's training data, mean \hat p - \bar y is 2.8\times 10^{-10} at C = 0.01 and 1.7\times 10^{-10} at C = 0.001. The slope equations do not hold: X^\top(y - \hat p) = w/C instead of 0 (at C = 0.01, both sides equal (−82.2, 47.2, −21.3)). With liblinear at C = 0.001, mean \hat p is 21.7% against \bar y = 11.9\%, with an intercept of −1.37. Verified by simulation on the CAL split.
• Intuition: the penalty only pulls on what it touches. Leave the intercept free and it still lines up the average. Penalize it too, and it gets dragged toward log-odds 0, which means toward 50%.
• Why: sklearn minimizes \tfrac12\lVert w\rVert^2 + C\cdot\sum \text{loss}. Differentiate with respect to the unpenalized intercept and the penalty term vanishes (Card 6). Calibration in the large holds, but the spread of the probabilities shrinks: the calibration slope on test is 1.61 at C = 0.001.

\frac{\partial}{\partial b}

how fast the penalized loss changes as you nudge the intercept b, holding the rest fixed: a slope, not a fraction to divide

C

sklearn's inverse penalty strength: big C, little penalty

\bar{\hat p}

the average predicted probability on the training data

\bar y

the training default rate (bar = average)

\frac{\partial}{\partial \mathbf w}

how fast it changes as you nudge each slope weight in \mathbf w

\mathbf w

the slope weights, which the penalty pulls toward 0

\mathbf p

all predicted probabilities stacked into one vector

\mathbf y

all 0/1 outcomes stacked into one vector

Drill 16 · Threshold with a refer band

Declining a good borrower costs $600 of lost profit; approving a defaulter costs $6,000. Operations adds manual review: it costs $40 per file and catches 80% of the defaulters it sees; the rest are approved. "Find the thresholds for approve / refer / decline. What happens to Applicant K at 5.6%?"

Concept tested: expected-cost decisions with three actions.

• Answer: approve below 0.83%, refer between 0.83% and 31.1%, decline above 31.1%. Applicant K (5.6%) is referred. Under the two-action rule, t* = 600/6,600 = 9.1%, and K would be approved. This is exact.
• Intuition: each action's expected cost is a straight line in p. The cheapest action switches wherever two lines cross.
• Why: approve costs 6{,}000p. Refer costs 40 + 0.2 \times 6{,}000p = 40 + 1{,}200p. Decline costs 600(1 - p). Set neighboring lines equal and solve (Card 16, "cost of a false alarm over total cost" generalized).

Drill 17 · Next year's base rate

The model was fitted when the book's default rate was 12.0%. After a rate-environment change, this year's rate is 6.6%. The ranking still looks fine. "Correct the intercept. What happens to an applicant the model scores at 20%? And when is your correction exactly right?"

Concept tested: prior-shift correction on the logit scale; label shift vs concept shift.

• Answer: add \operatorname{logit}(0.066) - \operatorname{logit}(0.12) = −0.657 to the intercept, so 20% becomes 11.5%. With the calibration split's own rates (11.9% → 6.57%), the shift is −0.651 and 20% still becomes 11.5%. The correction cuts ECE from 0.053 to 0.006. It is not exact here: this variant's true intercept change was −0.8, which maps 20% to 10.1%. The formula assumes label shift, where P(x \mid y) is fixed and only the base rate changes. This variant changed P(y \mid x) with P(x) fixed. Exact formula, checked on the SHIFT simulation.
• Intuition: fewer defaults overall means everyone's odds fall by the same factor, if only the base rate moved.
• Why: under label shift, Bayes on odds multiplies every posterior odds by the change in prior odds (Card 13). If the relationship itself shifted, refit the intercept on recent data with the slopes fixed as an offset. On the SHIFT data that refit gives −0.757.

\Delta b

the change to add to the intercept (\Delta = change)

e^{-0.657}

e (about 2.718) to the power −0.657: the factor that shrinks the odds

p'

the corrected probability for the applicant scored at 20%

Information, tests and GLMs — Drills 18–20

Drill 18 · Fisher information of a Bernoulli

A quant interviewer: "A 0/1 outcome with probability p. Derive the Fisher information for p and for \theta = \operatorname{logit} p. Where is one observation most informative about \theta, and where is it least informative about p? Check your answer at p = 0.1 with n = 1{,}000."

Concept tested: Fisher information and how it changes under reparametrization.

• Answer: I(p) = 1/(p(1 - p)), smallest (4) at p = 0.5. I(\theta) = p(1 - p), largest (0.25) at p = 0.5. At p = 0.1: I(p) = 11.1 and I(\theta) = 0.09. Verified by simulation (n = 1{,}000, 50,000 reps): n\cdot\operatorname{Var}(\hat p) = 0.0901 against 1/I(p) = 0.09, and n\cdot\operatorname{Var}(\operatorname{logit} \hat p) = 11.27 against 1/I(\theta) = 11.1.
• Intuition: near 0 or 1, the proportion itself is pinned down tightly, but a tiny change in p is a big change in log-odds, so the log-odds are hard to pin down. Opposite scales, opposite answers.
• Why: differentiate the Bernoulli log-likelihood twice. Then apply the chain rule with dp/d\theta = p(1 - p) (Card 9). This is why logistic regression learns most from rows near 50% (Drill 4).

I(p)

the Fisher information about p from one observation

p

the probability of a 1 (default)

\mathbb E

the average over repeated draws of y

\frac{\partial^2}{\partial p^2}

the curvature: how fast the slope changes as you nudge p

y

the 0/1 outcome

I(\theta)

the Fisher information about the log-odds \theta

\theta

the log-odds, \operatorname{logit} p

\frac{dp}{d\theta}

how fast p changes as you nudge \theta; it equals p(1-p)

Drill 19 · The Hauck–Donner trap

A pilot has 100 borrowers per arm. Control has 10 defaults; the treated arm's defaults are k out of 100, and the interviewer raises k: 20, 40, 60, 80, 90, 95, 98, 99. "As the effect grows, what happens to the Wald z for the log-OR and to the LR statistic? Which would you trust, and what happens at k = 100?"

Concept tested: the Hauck–Donner effect; Wald vs LR (W-H-S).

• Answer: the Wald z rises, peaks and falls: 1.95, 4.58, 6.66, 8.60, 9.32 at k = 90, then 9.07, 7.72 and 6.41 at k = 99. The LR statistic rises monotonically: 3.99, 25.3, 59.4, 110, 147, 172, 191, 199. At k = 100 a cell is empty, the MLE is infinite (separation), and the Wald test loses all power. Trust LR. These are exact calculations from the stated tables.
• Intuition: Wald measures Width, the distance from 0 in standard errors. For a huge effect, the standard error grows faster than the estimate, so the distance in SEs shrinks. LR measures Height, the log-likelihood lost by forcing no effect, and that only grows.
• Why: \operatorname{SE}(\text{log-OR}) = \sqrt{1/a + 1/b + 1/c + 1/d}. The 1/b term (treated repayers) explodes as b \to 0, while the log-OR grows only like \ln(1/b) (Card 10).

z_{\text{Wald}}

the Wald z: the estimated log odds ratio divided by its standard error

k

the number of defaults in the treated arm, out of 100

\frac1k

one over k, the treated-defaults cell count

Drill 20 · Exposure offset

Finance models missed payments per borrower; loans have been on the book 6 to 24 months. The true rate is \exp(-3.0 - 0.9z + 0.4d) per month. "Borrowers A and B have z = d = 0; A has 6 months on the book, B has 24. What counts do you expect? What happens if you fit Poisson regression without the offset, (a) when months are independent of score and (b) when better-score borrowers have been on the book longer?"

Concept tested: the exposure offset in a Poisson GLM; omitted exposure as an omitted variable.

• Answer: expected counts are 6e^{-3} = 0.30 for A and 24e^{-3} = 1.19 for B. (a) Without the offset, with months independent of score, the slopes survive (z −0.889, d 0.391). The intercept absorbs \ln(\text{mean months}): −0.291 against −0.295 in theory. The dispersion rises to 1.14. (b) With months correlated with score (r = 0.84), the score coefficient is −0.918 with the offset but −0.368 without: the extra months of good borrowers cancel most of score's effect. Entering \ln(\text{months}) as a covariate instead gives it a coefficient of 0.96 \approx 1. Verified by simulation.
• Intuition: a borrower watched four times as long has four times the chances to miss. Leave that out, and score gets blamed (or credited) for exposure.
• Why: with a log link, a count over t months has mean t\cdot e^{x^\top\beta}, so \ln t enters the linear predictor with coefficient fixed at 1. Omitting it is omitted-variable bias on the log scale. It is harmless if exposure is independent of x and biased if it is correlated (Card 17).

\mathbb E[y \mid \mathbf x, t]

the expected missed payments for a borrower with features \mathbf x observed for t months

t

months on the book: the exposure

e^{\mathbf x^\top\boldsymbol\beta}

the monthly rate of missed payments: e (about 2.718) raised to the score

\mathbf x^\top\boldsymbol\beta

the borrower's log rate of missed payments per month

e^{-3}

the monthly rate at z = d = 0: e to the power −3, about 0.05

Bias-direction cheat sheet

Standard-error traps (estimate fine, uncertainty wrong): Wald under large effects or separation (Drill 19, Q18), overdispersed counts (Q27), clustered or grouped borrowers (Ch 1, Section 8), and adjusted vs unadjusted SEs that aren't comparable across estimands (WP2).

Rapid fire — the day before

One line each. Cover the right column and answer out loud; anything you hesitate on, go back to its full question above.

Review schedule: go through this table the day after you study, then 3 days later, a week later, and 3 weeks later. Any row you miss goes back to the start.