Chapter 3 — Derivations

Logistic regression is the model interviewers most often ask you to derive on a whiteboard. It is small enough to do in ten minutes, and every step carries over to softmax, GLMs and the output layer of an LLM. Onsite loops ask for the core four: the loss from a coin flip, the gradient $X^\top(p - y)$ , the Hessian $X^\top WX$ , and the Newton step that turns into IRLS. Senior loops add odds ratios, calibration and the three tests. Research and quant loops push into what happens when the data are not what the model assumes: why case-control sampling moves only the intercept, why the MLE runs to infinity under separation, why an independent variable you leave out still shrinks every coefficient, and what flipped labels do. The 18 cards below cover all of it. Each one has the statement, every step of the derivation, the one move that does the work, the slip candidates make, and how the question is really asked. Where a card can check itself, its last step does so on the chapter's ten loan files or the 20,000-applicant book. How to practice: read the statement only, derive on a blank page against a timer, then check every step and number here. Re-derive anything you missed 48 hours later (the protocol is at the end). Chapter 1's cards (the normal equation, unbiasedness, OVB) are not repeated; when a card leans on one, it says so in one line.

The toolkit

Five moves do every derivation in this chapter. Their initials spell the chapter's own function: L-O-G-I-T.

L — Likelihood. Write the probability the model gave to what actually happened, multiply over independent rows, and take logs.

P(y_i \mid \mathbf x_i) = p_i^{\,y_i}(1 - p_i)^{1 - y_i} \;\;\Rightarrow\;\; \ell(\mathbf w) = \sum_{i=1}^{n}\big[y_i\ln p_i + (1 - y_i)\ln(1 - p_i)\big]

$P(y_i \mid \mathbf x_i)$: the probability the model gives to applicant $i$ 's actual outcome, given their features
$y_i$: applicant $i$ 's label: 1 if they defaulted, 0 if they repaid
$\mathbf x_i$: applicant $i$ 's feature vector (bold = vector), with a leading 1 for the intercept
$p_i$: the model's predicted probability that applicant $i$ defaults
$\ell$: the log-likelihood: the log of the probability the model gives to all observed outcomes
$\mathbf w$: the weight vector: the intercept plus one slope per feature
$\sum_{i=1}^{n}$: add up over every applicant $i = 1$ to $n$
$n$: the number of applicants (rows): 10 on the toy loan files

O — Bayes on Odds. Posterior odds = prior odds × likelihood ratio. On the log scale the two add, so anything that changes only the prior lands only in the intercept.

\frac{P(y = 1 \mid \mathbf x)}{P(y = 0 \mid \mathbf x)} = \frac{\pi_1}{\pi_0}\cdot\frac{f(\mathbf x \mid y = 1)}{f(\mathbf x \mid y = 0)} \;\;\Rightarrow\;\; \text{logit}\,P(y = 1 \mid \mathbf x) = \ln\frac{\pi_1}{\pi_0} + \ln\frac{f_1(\mathbf x)}{f_0(\mathbf x)}

$P(y = 1 \mid \mathbf x)$: the probability of default for an applicant with features $\mathbf x$
$P(y = 0 \mid \mathbf x)$: the probability that the same applicant repays
$\mathbf x$: an applicant's feature vector (credit score, DTI, …)
$\frac{\pi_1}{\pi_0}$: the prior odds: defaulters per repayer in the population, before seeing any features
$f(\mathbf x \mid y = 1)$: how common these features are among defaulters (the class-conditional density)
$f(\mathbf x \mid y = 0)$: how common these features are among repayers
$f_1(\mathbf x)$: shorthand for $f(\mathbf x \mid y = 1)$
$f_0(\mathbf x)$: shorthand for $f(\mathbf x \mid y = 0)$

G — Gradient by the chain rule, with $\sigma' = \sigma(1 - \sigma)$ . The sigmoid's derivative is written in terms of itself, so the chain rule collapses.

\sigma'(\eta) = \sigma(\eta)\big(1 - \sigma(\eta)\big) \;\;\Rightarrow\;\; \frac{\partial p_i}{\partial \mathbf w} = p_i(1 - p_i)\,\mathbf x_i

$\sigma$: the sigmoid, $1/(1 + e^{-t})$ : turns log-odds into a probability (not a standard deviation here)
$\sigma'$: the sigmoid's slope (prime = derivative)
$\eta$: the linear predictor $\mathbf x^\top\mathbf w$ : the model's log-odds of default
$\frac{\partial p_i}{\partial \mathbf w}$: how fast applicant $i$ 's probability changes as you nudge each weight: a slope, not a fraction to divide

I — Information. The negative Hessian of the log-likelihood is the Fisher information. It gives the Newton step and, inverted, the standard errors.

I(\mathbf w) = -\nabla^2\ell(\mathbf w) = X^\top W X, \qquad \mathbf w \leftarrow \mathbf w + I(\mathbf w)^{-1}\nabla\ell(\mathbf w), \qquad \widehat{\text{Var}}(\hat{\mathbf w}) = I(\hat{\mathbf w})^{-1}

$I(\mathbf w)$: the Fisher information: how sharply the log-likelihood curves around $\mathbf w$
$\nabla^2\ell(\mathbf w)$: the Hessian: the matrix of second derivatives (curvatures) of the log-likelihood
$X^\top W X$: the weighted Gram matrix: each row's $\mathbf x_i\mathbf x_i^\top$ weighted by $p_i(1 - p_i)$ , summed
$X$: the design matrix: one row per applicant, first column all 1s
$W$: the diagonal matrix of Bernoulli variances $p_i(1 - p_i)$ , one per applicant
$I(\mathbf w)^{-1}$: the inverse information (matrix inverse): turns the slope into a Newton step
$\nabla\ell(\mathbf w)$: the gradient (score): the slopes of the log-likelihood, one per weight; points uphill
$\widehat{\text{Var}}(\hat{\mathbf w})$: the estimated variances and covariances of the fitted weights
$\hat{\mathbf w}$: the fitted weights: the MLE (hat = estimated from data)
$I(\hat{\mathbf w})^{-1}$: the inverse information at the fit; its diagonal holds the squared standard errors

T — Taylor and approximate. Expand the log-likelihood to second order around its peak, or swap the sigmoid for a scaled normal CDF so that Gaussian algebra goes through.

\ell(\boldsymbol\beta) \approx \ell(\hat{\boldsymbol\beta}) - \tfrac12(\boldsymbol\beta - \hat{\boldsymbol\beta})^\top I(\hat{\boldsymbol\beta})\,(\boldsymbol\beta - \hat{\boldsymbol\beta}), \qquad \sigma(t) \approx \Phi\big(\sqrt{\pi/8}\;t\big)

$\boldsymbol\beta$: the weight vector, written $\boldsymbol\beta$ instead of $\mathbf w$
$\hat{\boldsymbol\beta}$: the MLE: the weights at the peak of the log-likelihood
$I(\hat{\boldsymbol\beta})$: the information (curvature) at the peak
$t$: any log-odds value
$\Phi$: the standard normal CDF: the chance a standard normal falls below its argument
$\sqrt{\pi/8}$: the scale that makes $\Phi$ 's slope at 0 match the sigmoid's slope, $\tfrac14$

Which move does the work, card by card: L for Cards 2, 11, 15, 17; O for 12, 13, 16, 18; G for 1, 3, 6, 7, 8, 17; I for 4, 5, 9, 10, 11; T for 10, 11 and 14. When you're stuck on an unfamiliar question, run down the letters: can I write the likelihood? Is this Bayes on odds? Is it a chain rule through $\sigma$ ? Is it curvature? Do I need an approximation?

Notation used on every card.

Symbol	Meaning	On the ten files
$n$ , $X$	Rows; design matrix with a first column of 1s	$n = 10$ ; $X = [1, x]$ with $x = (\text{score} - 680)/100$
$w$ (or $\beta$ )	Weights; $w_0$ ( $\beta_0$ ) is the intercept	MLE (−0.824, −2.77)
$\eta_i = x_i^\top w$	Linear predictor, the log-odds	Applicant K has $x = 0$ , so $\eta = w_0$
$p_i = \sigma(\eta_i)$	Predicted probability of default	K: 0.305
$W$	$\operatorname{diag}(p_i(1 - p_i))$ , the Bernoulli variances	All 0.25 at $w = 0$
$\ell(w)$	Log-likelihood; the loss is $-\ell$	−4.36 at the MLE
ỹᵢ = 2yᵢ − 1	Label recoded to {−1, +1}	Defaulters +1, repayers −1
$\pi$	Base rate $P(y = 1)$	0.4 on the toy, 0.120 on the book

Formula sheet

Every formula the cards derive, in one place. Cover the block, say it, check.

The model

Sigmoid and logit (inverse pair)

p = \sigma(\eta) = \frac{1}{1 + e^{-\eta}}, \qquad \eta = \text{logit}\,p = \ln\frac{p}{1 - p}

$p$: the predicted probability that an applicant defaults
$\sigma$: the sigmoid: turns any log-odds into a probability (here $\sigma$ is the sigmoid, not a standard deviation)
$\eta$: the linear predictor $\mathbf x^\top\mathbf w$ : the applicant's log-odds of default
$e$: Euler's number, 2.718…; $e^{-\eta}$ means $\exp(-\eta)$
$\frac{p}{1 - p}$: the odds of default: chance of default over chance of repaying

Sigmoid derivative and symmetry

\sigma'(\eta) = \sigma(\eta)\big(1 - \sigma(\eta)\big) \le \tfrac14, \qquad 1 - \sigma(\eta) = \sigma(-\eta)

$\sigma'(\eta)$: the sigmoid's slope at $\eta$ (prime = derivative)

Odds and the odds ratio

\text{odds} = \frac{p}{1 - p} = e^{\eta}, \qquad \frac{\text{odds}(x_j + 1)}{\text{odds}(x_j)} = e^{\beta_j}

$\frac{\text{odds}(x_j + 1)}{\text{odds}(x_j)}$: the odds ratio: how the odds of default multiply when $x_j$ rises one unit
$x_j$: feature $j$ , e.g. the credit score in units of 100 points
$\beta_j$: the coefficient on feature $j$ : log-odds added per unit of $x_j$

The loss and its derivatives

Log loss (binary cross-entropy)

-\ell(\mathbf w) = -\sum_i \big[y_i\ln p_i + (1 - y_i)\ln(1 - p_i)\big] = \sum_i \big[\ln(1 + e^{\eta_i}) - y_i\eta_i\big]

$\ell(\mathbf w)$: the log-likelihood of weights $\mathbf w$ ; minus it is the log loss
$\ell$: the log-likelihood; $-\ell$ is the log loss training minimizes
$\mathbf w$: the weights: the intercept plus one slope per feature (bold = vector)
$\sum_i$: add up over every applicant $i$
$y_i$: applicant $i$ 's outcome: 1 = defaulted, 0 = repaid
$p_i$: applicant $i$ 's predicted default probability
$\eta_i$: applicant $i$ 's log-odds, $\mathbf x_i^\top\mathbf w$

Margin form, labels in {−1, +1}

-\ell(\mathbf w) = \sum_i \ln\big(1 + e^{-\tilde y_i\,\eta_i}\big), \qquad \tilde y_i = 2y_i - 1

$\tilde y_i$: the label recoded to ±1: +1 for a defaulter, −1 for a repayer

Gradient

\nabla_{\mathbf w}\big(-\ell\big) = X^\top(\mathbf p - \mathbf y)

$\nabla_{\mathbf w}$: the gradient with respect to the weights: one slope per weight; points uphill
$X$: the design matrix: one row per applicant, first column all 1s
$^\top$: transpose: rows become columns
$\mathbf p$: all $n$ predicted probabilities as one vector (bold = vector)
$\mathbf y$: all $n$ labels as one vector

Hessian

\nabla^2_{\mathbf w}\big(-\ell\big) = X^\top W X, \qquad W = \text{diag}\big(p_i(1 - p_i)\big) \succeq 0

$\nabla^2_{\mathbf w}$: the Hessian: the matrix of second derivatives (curvatures) with respect to the weights
$X^\top W X$: the weighted Gram matrix: each applicant's $\mathbf x_i\mathbf x_i^\top$ weighted by $p_i(1 - p_i)$
$W$: the diagonal matrix of each applicant's Bernoulli variance $p_i(1 - p_i)$
$\succeq 0$: is positive semi-definite: curves up, or stays flat, in every direction

Newton = IRLS step

\mathbf w_{\text{new}} = (X^\top W X)^{-1}X^\top W\mathbf z, \qquad \mathbf z = X\mathbf w + W^{-1}(\mathbf y - \mathbf p)

$\mathbf w_{\text{new}}$: the weights after one Newton step
$(X^\top W X)^{-1}$: the inverse of the weighted Gram matrix (matrix inverse: undoes it, like dividing by curvature)
$\mathbf z$: the working response: each applicant's linearized log-odds, which IRLS regresses on $X$
$W^{-1}$: the inverse of $W$ : divides each applicant's miss by its variance

Score equations at the MLE (calibration in the large)

X^\top(\mathbf y - \hat{\mathbf p}) = \mathbf 0 \;\;\Rightarrow\;\; \sum_i \hat p_i = \sum_i y_i, \qquad \sum_i \hat p_i\,x_{ij} = \sum_i y_i\,x_{ij}

$\hat{\mathbf p}$: the fitted probabilities at the MLE, as a vector (hat = estimated from data)
$\hat p_i$: applicant $i$ 's fitted default probability
$x_{ij}$: applicant $i$ 's value of feature $j$ (row $i$ , column $j$ of $X$ )

Reading the coefficients

Marginal effect and the divide-by-4 bound

\frac{\partial p}{\partial x_j} = \beta_j\,p(1 - p), \qquad \Big|\frac{\partial p}{\partial x_j}\Big| \le \frac{|\beta_j|}{4}

$\frac{\partial p}{\partial x_j}$: how fast the default probability changes as you nudge $x_j$ , holding the rest fixed: a slope, not a fraction to divide

Average marginal effect

\text{AME}_j = \frac{\beta_j}{n}\sum_{i=1}^{n}\hat p_i(1 - \hat p_i)

$\text{AME}_j$: the average marginal effect of feature $j$ : the slope $\partial p/\partial x_j$ averaged over applicants
$n$: the number of applicants
$\sum_{i=1}^{n}$: add up over applicants $i = 1$ to $n$

Inference

Fisher information of one Bernoulli

I(p) = \frac{1}{p(1 - p)}, \qquad I(\theta) = p(1 - p) \quad (\theta = \text{logit}\,p)

$I(p)$: the Fisher information one 0/1 outcome carries about $p$
$I(\theta)$: the information it carries about the log-odds $\theta$
$\theta$: the log-odds, $\text{logit}\,p$

Standard errors; the intercept-only case

\widehat{\text{Var}}(\hat{\mathbf w}) = (X^\top\hat W X)^{-1}, \qquad \text{SE}(\hat\beta_0)\big|_{\text{intercept only}} = \frac{1}{\sqrt{n\,\bar y(1 - \bar y)}} = \sqrt{\frac{1}{n_1} + \frac{1}{n_0}}

$\widehat{\text{Var}}(\hat{\mathbf w})$: the estimated covariance matrix of the fitted weights
$\hat{\mathbf w}$: the fitted weights: the MLE
$(X^\top\hat W X)^{-1}$: the inverse information, with $W$ evaluated at the fitted probabilities
$\text{SE}(\hat\beta_0)$: the standard error of the fitted intercept
$\bar y$: the share of applicants who defaulted (bar = mean)
$n_1$: the number of defaulters
$n_0$: the number of repayers

The three tests (W-H-S: Width, Height, Slope)

W = \frac{\hat\beta^2}{\widehat{\text{SE}}^2}, \qquad LR = 2\big(\ell(\hat{\boldsymbol\beta}) - \ell(\tilde{\boldsymbol\beta}_0)\big), \qquad S = \frac{U(\tilde{\boldsymbol\beta}_0)^2}{I(\tilde{\boldsymbol\beta}_0)}, \qquad \text{each} \to \chi^2_1

$W$: here the Wald statistic: the estimate's squared distance from 0, in SEs (not the weight matrix)
$\hat\beta$: the fitted coefficient being tested
$\widehat{\text{SE}}$: its estimated standard error
$LR$: the likelihood-ratio statistic: twice the log-likelihood lost by imposing the null
$\hat{\boldsymbol\beta}$: the unrestricted fit: the MLE of all weights
$\tilde{\boldsymbol\beta}_0$: the restricted fit: the null imposed, other weights re-estimated
$S$: the score statistic: the squared slope at the restricted fit, over its information
$U(\tilde{\boldsymbol\beta}_0)$: the score (slope of $\ell$ ) at the restricted fit
$I(\tilde{\boldsymbol\beta}_0)$: the information at the restricted fit
$\chi^2_1$: the chi-squared distribution with 1 degree of freedom: the null reference for all three

Deviance

D = -2\big(\ell(\hat{\mathbf w}) - \ell_{\text{sat}}\big) = -2\,\ell(\hat{\mathbf w}) \;\;(\text{0/1 data}), \qquad LR = D_{\text{null}} - D_{\text{model}}

$D$: the deviance: twice the log-likelihood gap between a perfect fit and the model
$\ell_{\text{sat}}$: the saturated model's log-likelihood: every row predicted perfectly; 0 for 0/1 data
$D_{\text{null}}$: the deviance of the intercept-only model
$D_{\text{model}}$: the deviance of the fitted model

Firth's penalized likelihood

\ell^*(\mathbf w) = \ell(\mathbf w) + \tfrac12\ln\det\big(X^\top W X\big), \qquad \sum_i\big(y_i - p_i + h_i(\tfrac12 - p_i)\big)\mathbf x_i = \mathbf 0

$\ell^*(\mathbf w)$: Firth's penalized log-likelihood (star = modified version)
$h_i$: applicant $i$ 's leverage: how strongly their row pulls on the fit
$\mathbf x_i$: applicant $i$ 's feature vector

Data bias

Case-control intercept shift

\beta_0^{\text{sample}} = \beta_0 + \ln\frac{s_1}{s_0}, \qquad \text{slopes unchanged}

$\beta_0^{\text{sample}}$: the intercept a fit on the case-control sample converges to
$\beta_0$: the population intercept
$s_1$: the chance a defaulter is kept in the training sample
$s_0$: the chance a repayer is kept

Class weights act like sampling

\beta_0^{\text{weighted}} = \beta_0 + \ln\frac{w_1}{w_0}, \qquad \text{balanced: } \frac{w_1}{w_0} = \frac{1 - \pi}{\pi}

$\beta_0^{\text{weighted}}$: the intercept a class-weighted fit converges to
$w_1$: the class weight on defaulters (not a model weight)
$w_0$: the class weight on repayers
$\pi$: the base rate of default, $P(y = 1)$ : 0.120 on the book

Prior-shift correction

\beta_0' = \beta_0 + \ln\frac{\pi'}{1 - \pi'} - \ln\frac{\pi}{1 - \pi}

$\beta_0'$: the corrected intercept for the new base rate
$\pi'$: the new base rate (e.g. next year's 6.6%)

Non-collapsibility (probit approximation)

\beta_{\text{omit } Z} \approx \frac{\beta}{\sqrt{1 + \pi\gamma^2\sigma_Z^2/8}} \qquad (Z \perp x)

$\beta_{\text{omit } Z}$: the slope on $x$ when $Z$ is left out of the model
$\beta$: the slope on $x$ in the full model, with $Z$ included
$Z$: another risk factor, independent of $x$ , that the smaller model leaves out
$x$: the feature that stays in the model (the scaled credit score)
$\pi$: here the circle constant 3.14159…, not the base rate
$\gamma$: $Z$ 's coefficient in the full model
$\sigma_Z$: the standard deviation of $Z$ (here $\sigma$ is a standard deviation, not the sigmoid)

Misclassified labels

P(\tilde y = 1 \mid \mathbf x) = \alpha_0 + (1 - \alpha_0 - \alpha_1)\,\sigma(\mathbf x^\top\boldsymbol\beta) \in (\alpha_0,\; 1 - \alpha_1)

$P(\tilde y = 1 \mid \mathbf x)$: the probability that the recorded label says default
$\alpha_0$: the chance a true repayer is recorded as a default
$\alpha_1$: the chance a true default is recorded as a repayment
$\mathbf x$: an applicant's feature vector, with a leading 1
$\boldsymbol\beta$: the true weights, intercept included

Decisions

Cost-optimal threshold

\text{decline} \iff p > t^* = \frac{c_{FP}}{c_{FP} + c_{FN}}

$t^*$: the cost-optimal threshold (star = the optimum)
$c_{FP}$: cost of a false alarm: declining a borrower who would have repaid ($600 lost profit)
$c_{FN}$: cost of a miss: approving a borrower who defaults ($6,000 lost principal)

Multiclass, GLMs and generative models

Softmax and its gradient

p_k = \frac{e^{\eta_k}}{\sum_j e^{\eta_j}}, \qquad \frac{\partial(-\ln p_y)}{\partial\eta_k} = p_k - \mathbb 1[k = y]

$p_k$: the predicted probability of outcome class $k$
$\eta_k$: the score (logit) for class $k$
$\sum_j$: add up over every class $j$
$\eta_j$: the score for class $j$
$\frac{\partial(-\ln p_y)}{\partial\eta_k}$: how fast the row's loss changes as you nudge class $k$ 's score, holding the others fixed
$p_y$: the probability given to the true class $y$
$\mathbb 1[k = y]$: 1 if $k$ is the true class, else 0

Canonical-link GLM score

\nabla_{\mathbf w}\,\ell = \frac{1}{\phi}X^\top(\mathbf y - \boldsymbol\mu), \qquad \text{Poisson with exposure: } \ln\mu_i = \ln t_i + \mathbf x_i^\top\boldsymbol\beta

$\phi$: the dispersion: 1 for Bernoulli and Poisson, the noise variance for Gaussian
$\boldsymbol\mu$: the vector of model means (for logistic regression, the probabilities $p_i$ )
$\mu_i$: borrower $i$ 's expected count
$t_i$: borrower $i$ 's exposure: months on the books

LDA ⇒ logistic

\boldsymbol\beta = \Sigma^{-1}(\boldsymbol\mu_1 - \boldsymbol\mu_0), \qquad \beta_0 = \ln\frac{\pi_1}{\pi_0} - \tfrac12\big(\boldsymbol\mu_1^\top\Sigma^{-1}\boldsymbol\mu_1 - \boldsymbol\mu_0^\top\Sigma^{-1}\boldsymbol\mu_0\big)

$\Sigma^{-1}$: the inverse of $\Sigma$ , the covariance matrix both classes share (matrix inverse: undoes it)
$\boldsymbol\mu_1$: the average feature vector of defaulters
$\boldsymbol\mu_0$: the average feature vector of repayers
$\pi_1$: the base rate of default
$\pi_0$: the share of applicants who repay
$\boldsymbol\beta$: the LDA slopes, one per feature

Cards

Card 1 · Sigmoid, odds and log-odds ★

Statement. The sigmoid is the inverse of the logit; its derivative is written in terms of itself; and it is symmetric about zero.

\sigma(z) = \frac{1}{1 + e^{-z}} = \text{logit}^{-1}(z), \qquad \sigma'(z) = \sigma(z)\big(1 - \sigma(z)\big), \qquad 1 - \sigma(z) = \sigma(-z)

$\sigma$: the sigmoid: turns log-odds into a probability (here $\sigma$ is the sigmoid, not a standard deviation)
$z$: any log-odds value, anywhere on the real line
$e$: Euler's number, 2.718…
$\text{logit}^{-1}$: the inverse of the logit: undoes log-odds back into a probability
$\sigma'$: the sigmoid's slope (prime = derivative)

Derivation.

Start from the log-odds and exponentiate to get the odds.

z = \ln\frac{p}{1 - p} \;\;\Rightarrow\;\; e^{z} = \frac{p}{1 - p}

$p$: the probability of default
$\frac{p}{1 - p}$: the odds of default: chance of default over chance of repaying

Solve for p: multiply out, collect p, divide.

e^{z}(1 - p) = p \;\;\Rightarrow\;\; p\,(1 + e^{z}) = e^{z} \;\;\Rightarrow\;\; p = \frac{e^{z}}{1 + e^{z}} = \frac{1}{1 + e^{-z}} = \sigma(z)

Symmetry: subtract from 1 and multiply top and bottom by $e^z$ .

1 - \sigma(z) = \frac{e^{-z}}{1 + e^{-z}} = \frac{1}{e^{z} + 1} = \sigma(-z)

Derivative: differentiate $(1 + e^{-z})^{-1}$ and split the result into two familiar factors.

\sigma'(z) = \frac{e^{-z}}{(1 + e^{-z})^2} = \underbrace{\frac{1}{1 + e^{-z}}}_{\sigma(z)}\cdot\underbrace{\frac{e^{-z}}{1 + e^{-z}}}_{1 - \sigma(z)} = \sigma(z)\big(1 - \sigma(z)\big)

The derivative is largest where $p(1 - p)$ is: at $p = \tfrac12$ , that is $z = 0$ , with value $\tfrac14$ . This is the divide-by-4 rule of Card 8.

\frac{d}{dp}\,p(1 - p) = 1 - 2p = 0 \;\Rightarrow\; p = \tfrac12, \qquad \max_z \sigma'(z) = \tfrac14

$\frac{d}{dp}$: how fast what follows changes as you nudge $p$ : a slope, not a fraction to divide

Check on Applicant K ( $x = 0$ , so $\eta = \hat{\beta}_0$ ): the three rungs of the P-O-L ladder.

\eta_K = -0.824 \;\Rightarrow\; \text{odds} = e^{-0.824} = 0.439 \;\Rightarrow\; p = \frac{0.439}{1.439} = \frac{1}{1 + e^{0.824}} = 0.305

$\eta_K$: Applicant K's log-odds: the fitted intercept $\hat\beta_0$ , since K's $x = 0$ (a 680 score)

Key move. Gradient by the chain rule (G): writing $\sigma'$ through $\sigma$ is what makes every later derivative collapse.

Common slip. Writing $\sigma' = \sigma(1 + \sigma)$ , or reading the odds (0.439) as the probability (0.305). In code, computing $1/(1 + e^{-z})$ for very negative $z$ overflows; stable code uses $e^z/(1 + e^z)$ there, or scipy.special.expit, and computes losses with logaddexp.

How it's asked. "Show that the sigmoid's derivative is $\sigma(1 - \sigma)$ ." A strong answer uses it at once: it is why the gradient collapses to $X^\top(p - y)$ (Card 3), why the per-row Fisher information is $p(1 - p)$ (Card 9), and why the effect on probability is at most $\beta/4$ (Card 8).

Card 2 · Bernoulli likelihood → cross-entropy ★

Statement. Maximizing the Bernoulli likelihood is minimizing the log loss (binary cross-entropy). With labels recoded to ±1 it is the "logistic loss" of ML papers.

-\ell(\mathbf w) = -\sum_{i=1}^{n}\big[y_i\ln p_i + (1 - y_i)\ln(1 - p_i)\big] = \sum_{i=1}^{n}\ln\big(1 + e^{-\tilde y_i\eta_i}\big), \qquad \tilde y_i = 2y_i - 1

$\ell$: the log-likelihood; $-\ell$ is the log loss (binary cross-entropy) training minimizes
$\mathbf w$: the weights: the intercept plus one slope per feature (bold = vector)
$\sum_{i=1}^{n}$: add up over every applicant $i = 1$ to $n$
$n$: the number of applicants: 10 on the toy loan files
$y_i$: applicant $i$ 's outcome: 1 = defaulted, 0 = repaid
$p_i$: applicant $i$ 's predicted default probability
$\tilde y_i$: the label recoded to ±1: +1 for a defaulter, −1 for a repayer
$\eta_i$: applicant $i$ 's log-odds, $\mathbf x_i^\top\mathbf w$

Derivation.

One applicant is a biased coin. The exponents pick the right factor: $p$ when $y = 1$ , $1 - p$ when $y = 0$ .

P(y_i \mid \mathbf x_i) = p_i^{\,y_i}(1 - p_i)^{1 - y_i}, \qquad p_i = \sigma(\mathbf x_i^\top\mathbf w)

$P(y_i \mid \mathbf x_i)$: the probability the model gives to applicant $i$ 's actual outcome
$\mathbf x_i$: applicant $i$ 's feature vector, with a leading 1 for the intercept
$^\top$: transpose: rows become columns, so $\mathbf x_i^\top\mathbf w$ is a dot product

Independent rows multiply.

L(\mathbf w) = \prod_{i=1}^{n} p_i^{\,y_i}(1 - p_i)^{1 - y_i}

$L(\mathbf w)$: the likelihood: the probability of all ten outcomes at once, as a function of $\mathbf w$
$\prod_{i=1}^{n}$: multiply over every applicant $i = 1$ to $n$

Logs turn the product into a sum; flip the sign to get a loss to minimize. Per row it is −log of the probability you gave to what happened.

-\ell(\mathbf w) = -\sum_{i=1}^{n}\big[y_i\ln p_i + (1 - y_i)\ln(1 - p_i)\big]

Rewrite in terms of $\eta$ , which Card 3 will use. Group the $y$ terms, then use $\ln(p/(1 - p)) = \eta$ and $1 - \sigma(\eta) = 1/(1 + e^\eta)$ (Card 1).

y\ln p + (1 - y)\ln(1 - p) = y\ln\frac{p}{1 - p} + \ln(1 - p) = y\,\eta - \ln\big(1 + e^{\eta}\big)

$y$: one applicant's outcome: 1 = default, 0 = repaid
$\eta$: that applicant's log-odds

Margin form: for $y = 1$ the row loss is $-\ln \sigma(\eta) = \ln(1 + e^{-\eta})$ ; for $y = 0$ it is $-\ln \sigma(-\eta) = \ln(1 + e^\eta)$ . Both are $\ln(1 + e^{-\tilde y\eta})$ .

y = 1:\; \ln(1 + e^{-\eta}), \qquad y = 0:\; \ln(1 + e^{\eta}) \qquad\Rightarrow\qquad \ln\big(1 + e^{-\tilde y\,\eta}\big)

$\tilde y$: the ±1 version of $y$ : +1 default, −1 repaid

Why the name: with the label as a distribution $q = (y, 1 - y)$ and the model as $(p, 1 - p)$ , the row loss is the cross-entropy $H(q, p)$ . Since $H(q, p) = H(q) + \operatorname{KL}(q \Vert p)$ and $H(q)$ does not depend on $w$ , minimizing log loss minimizes the KL divergence from data to model.

H(q, p) = -\sum_{c} q_c\ln p_c = H(q) + \text{KL}(q \,\Vert\, p)

$H(q, p)$: the cross-entropy: average surprise when outcomes follow $q$ but you predict with $p$
$\sum_{c}$: add up over the two outcomes $c$ : default and repay
$q_c$: the label's probability for outcome $c$ : 1 for what happened, 0 otherwise
$p_c$: the model's probability for outcome $c$
$H(q)$: the entropy of the label itself; it doesn't depend on $\mathbf w$
$\text{KL}(q \,\Vert\, p)$: the KL divergence: extra surprise from predicting with $p$ instead of $q$ ; zero when they match

Check on the ten files. At $w = 0$ every $p = 0.5$ , so the likelihood is $0.5^{10} = 0.000977$ and the loss is $10 \ln 2$ . At the MLE the loss falls to 4.36.

-\ell(\mathbf 0) = 10\ln 2 = 6.93 \;(\text{mean } 0.693), \qquad -\ell(\hat{\mathbf w}) = 4.36

$\hat{\mathbf w}$: the fitted weights: the MLE (hat = estimated from data)

Key move. Likelihood (L).

Common slip. Dropping the $(1 - y)$ term, so the loss only rewards defaulters. Or mixing label codings: $\ln(1 + e^{-y\eta})$ with $y \in \{0, 1\}$ gives the constant $\ln 2$ for every repayer, so the model never learns from them.

How it's asked. "Each applicant is a biased coin. Derive the loss you'd minimize." A strong answer adds the Chapter 1 parallel (Gaussian likelihood → squared error; Bernoulli → cross-entropy), the KL reading, and that a model that predicts only the base rate $\pi$ scores the entropy $H(\pi)$ . Every model must beat that baseline.

Card 3 · The gradient Xᵀ(p − y) ★

Statement. The gradient of the log loss is each feature column weighted by the miss on each row. It has the same shape as OLS's $X^\top(\hat y - y)$ , and setting it to zero has no closed-form solution.

\nabla_{\mathbf w}\big(-\ell(\mathbf w)\big) = \sum_{i=1}^{n}(p_i - y_i)\,\mathbf x_i = X^\top(\mathbf p - \mathbf y)

$\nabla_{\mathbf w}$: the gradient with respect to $\mathbf w$ : one slope per weight; points uphill
$X$: the design matrix: one row per applicant, first column all 1s (here $[1, x]$ )
$\mathbf p$: all ten predicted probabilities as one vector
$\mathbf y$: all ten labels as one vector

Derivation.

Start from the $\eta$ form of the loss (Card 2, step 4).

-\ell(\mathbf w) = \sum_{i=1}^{n}\big[\ln(1 + e^{\eta_i}) - y_i\eta_i\big], \qquad \eta_i = \mathbf x_i^\top\mathbf w

Differentiate one row with respect to its own $\eta$ : the derivative of $\ln(1 + e^\eta)$ is the sigmoid.

\frac{\partial}{\partial\eta_i}\big[\ln(1 + e^{\eta_i}) - y_i\eta_i\big] = \frac{e^{\eta_i}}{1 + e^{\eta_i}} - y_i = p_i - y_i

$\frac{\partial}{\partial\eta_i}$: how fast the bracket changes as you nudge row $i$ 's log-odds, holding the rest fixed: a slope, not a fraction to divide

Chain rule through $\eta_i = x_i^\top w$ , then sum over rows.

\frac{\partial\eta_i}{\partial\mathbf w} = \mathbf x_i \;\;\Rightarrow\;\; \nabla_{\mathbf w}(-\ell) = \sum_i (p_i - y_i)\,\mathbf x_i = X^\top(\mathbf p - \mathbf y)

$\frac{\partial\eta_i}{\partial\mathbf w}$: how row $i$ 's log-odds changes as you nudge each weight
$\sum_i$: add up over every applicant $i$

The same answer from the $p$ form, to show where the cancellation happens: $\sigma' = p(1 - p)$ cancels both denominators.

\frac{\partial}{\partial\mathbf w}\big[y\ln p + (1 - y)\ln(1 - p)\big] = \Big[\frac{y}{p} - \frac{1 - y}{1 - p}\Big]p(1 - p)\,\mathbf x = \big[y(1 - p) - (1 - y)p\big]\mathbf x = (y - p)\,\mathbf x

$\frac{\partial}{\partial\mathbf w}$: how fast the bracket changes as you nudge each weight
$\mathbf x$: one applicant's feature vector

Set it to zero. There are p equations, but w sits inside a sigmoid, so they are nonlinear in w and have no general closed form. OLS's are linear (Ch 1, Section 3).

\text{logistic: } \sum_i \mathbf x_i\big(y_i - \sigma(\mathbf x_i^\top\mathbf w)\big) = \mathbf 0 \qquad \text{OLS: } X^\top(\mathbf y - X\mathbf w) = \mathbf 0

Check on the ten files at $w = 0$ . Every $p = 0.5$ . Use $\sum y = 4$ , $\sum x = -0.4$ and $\sum xy = -2.3$ .

\nabla(-\ell)\big|_{\mathbf w = 0} = \begin{pmatrix} \sum_i (0.5 - y_i) \\ \sum_i x_i(0.5 - y_i) \end{pmatrix} = \begin{pmatrix} 5 - 4 \\ 0.5(-0.4) + 2.3 \end{pmatrix} = \begin{pmatrix} 1 \\ 2.1 \end{pmatrix}

$\nabla(-\ell)\big|_{\mathbf w = 0}$: the gradient of the log loss evaluated at all-zero weights
$x_i$: applicant $i$ 's scaled credit score, $(\text{score} - 680)/100$

The slope component is positive, so a downhill step makes $w_1$ negative. That matches $\hat{\beta}_1 = -2.77$ : higher scores, lower risk.

Key move. Gradient by the chain rule (G), with $\partial \ln(1 + e^\eta)/\partial\eta = \sigma(\eta)$ .

Common slip. The sign. The gradient of the log-likelihood is $X^\top(y - p)$ ; that of the loss is $X^\top(p - y)$ . Swap them and gradient descent climbs. Also, the mean loss has a $1/n$ in front; it changes the learning rate, not the solution.

How it's asked. "Differentiate the log loss. What does the gradient remind you of?" A strong answer: OLS's gradient, with $p$ in place of $\hat y$ (features × misses). The same $p - y$ appears for softmax (Card 17) and for every canonical-link GLM, and it is the error signal a neural network backpropagates from a cross-entropy output layer. One special case does have a closed form: a single binary feature, where the MLE is the log cross-product ratio of the 2×2 table (Bank Drill 2).

Card 4 · The Hessian and convexity ★★

Statement. The Hessian of the log loss is $X^\top WX$ , which is positive semi-definite. The loss is convex, and strictly convex when $X$ has full column rank. But a convex loss need not attain its minimum.

\nabla^2_{\mathbf w}\big(-\ell(\mathbf w)\big) = \sum_{i=1}^{n} p_i(1 - p_i)\,\mathbf x_i\mathbf x_i^\top = X^\top W X \succeq 0, \qquad W = \text{diag}\big(p_i(1 - p_i)\big)

$\nabla^2_{\mathbf w}$: the Hessian: the matrix of second derivatives (curvatures) with respect to the weights
$\mathbf x_i\mathbf x_i^\top$: row $i$ times itself as an outer product: one entry per pair of features
$X^\top W X$: the weighted Gram matrix: each row's $\mathbf x_i\mathbf x_i^\top$ weighted by $p_i(1 - p_i)$
$W$: the diagonal matrix of Bernoulli variances $p_i(1 - p_i)$ , one per applicant
$\succeq 0$: is positive semi-definite: curves up, or stays flat, in every direction

Derivation.

Differentiate the gradient $\sum(p_i - y_i)x_i$ once more. $y_i$ is a constant, and $\partial p_i/\partial w^\top = p_i(1 - p_i)x_i^\top$ (Card 1).

\nabla^2(-\ell) = \sum_i \mathbf x_i\,\frac{\partial p_i}{\partial\mathbf w^\top} = \sum_i p_i(1 - p_i)\,\mathbf x_i\mathbf x_i^\top = X^\top W X

$\nabla^2$: the Hessian: the matrix of second derivatives
$\frac{\partial p_i}{\partial\mathbf w^\top}$: how row $i$ 's probability changes with each weight, laid out as a row

PSD: sandwich with any vector v. Each weight is in (0, ¼], so every term is non-negative.

\mathbf v^\top X^\top W X\,\mathbf v = \sum_i p_i(1 - p_i)\,(\mathbf x_i^\top\mathbf v)^2 \;\ge\; 0

$\mathbf v$: any direction in weight space (a test vector)

Strictly positive: if $X$ has full column rank, $v \ne 0$ means $Xv \ne 0$ , so some $(x_i^\top v)^2 > 0$ . At any finite $w$ every weight is positive, so the sum is positive. A strictly convex function has at most one minimizer: no local minima, no ties.

\text{rank}(X) = p,\;\; \mathbf v \neq \mathbf 0 \;\Rightarrow\; \mathbf v^\top X^\top W X\,\mathbf v > 0

$\text{rank}(X) = p$: $X$ has full column rank: its $p$ columns are linearly independent

Existence is a separate question. As $\lvert\eta_i\rvert \to \infty$ the weights $p_i(1 - p_i) \to 0$ , and the bowl flattens at infinity. A strictly convex function can decrease forever without a minimum, like $e^{-t}$ . That is exactly separation (Card 11).

f(t) = e^{-t}: \;\; f''(t) = e^{-t} > 0, \;\; \inf_t f(t) = 0 \text{ but no } t \text{ attains it}

$f(t)$: a toy convex function, $e^{-t}$ , that has no minimum
$t$: its input, any real number
$f''(t)$: its curvature (second derivative)

A bound that sets the safe learning rate: $p(1 - p) \le \tfrac14$ gives $H \preceq \tfrac14X^\top X$ for every $w$ . For gradient descent on the mean loss, the Ch 2 stability rule $\eta < 2/\lambda_{\max}$ becomes:

X^\top W X \preceq \tfrac14 X^\top X \;\;\Rightarrow\;\; \eta_{\text{GD}} < \frac{2}{\lambda_{\max}\big(X^\top X/(4n)\big)}

$\preceq$: is no larger than, in every direction (a matrix inequality)
$X^\top X$: the unweighted Gram matrix of the features
$\eta_{\text{GD}}$: gradient descent's learning rate (not a log-odds)
$\lambda_{\max}$: the largest eigenvalue: the curvature along the steepest direction

Check on the ten files. At $w = 0$ , $W = \tfrac14I$ , so $H = \tfrac14X^\top X$ , with $n = 10$ , $\sum x = -0.4$ and $\sum x^2 = 4.84$ .

H\big|_{\mathbf w = 0} = \frac14\begin{pmatrix} 10 & -0.4 \\ -0.4 & 4.84 \end{pmatrix} = \begin{pmatrix} 2.5 & -0.1 \\ -0.1 & 1.21 \end{pmatrix}

$H$: the Hessian $X^\top W X$ , here at $\mathbf w = 0$ where every weight is $\tfrac14$

On standardized $x$ , $X^\top X/n = I$ , so $\lambda_{\max}(X^\top X/(4n)) = 0.25$ and the bound is $\eta < 8$ . Stopping when $\lvert\Delta \log L\rvert < 10^{-8}$ , gradient descent takes 13 steps at $\eta = 8$ , 51 at $\eta = 2$ and 191 at $\eta = 0.5$ , while $\eta = 16$ oscillates without diverging or converging. The bound is conservative: at the MLE the true curvature is smaller and the local bound is 12.5.

Key move. Information (I): the Hessian is a weighted Gram matrix.

Common slip. "Convex, so a unique minimum exists." Convexity gives at most one minimizer, not existence. Also writing $W = \operatorname{diag}(p)$ or $\operatorname{diag}(y - p)$ .

How it's asked. "A colleague says logistic regression can get stuck in local minima like a neural net. Prove or disprove." A strong answer gives the one-line PSD proof, then the caveat: under separation there is no minimum at all. Adding an L2 penalty makes the Hessian $X^\top WX + \lambda I \succ 0$ (strongly convex), so a finite minimizer always exists.

Card 5 · Newton = IRLS ★★

Statement. Newton's step for logistic regression is a weighted least-squares fit of a "working response" $z$ on $X$ , with weights $p(1 - p)$ . Repeat it: weigh, regress, repeat.

\mathbf w_{\text{new}} = (X^\top W X)^{-1}X^\top W\,\mathbf z, \qquad \mathbf z = X\mathbf w + W^{-1}(\mathbf y - \mathbf p), \quad z_i = \eta_i + \frac{y_i - p_i}{p_i(1 - p_i)}

$\mathbf w_{\text{new}}$: the weights after one Newton (IRLS) step
$(X^\top W X)^{-1}$: the inverse of the weighted Gram matrix (matrix inverse: undoes it)
$\mathbf z$: the working response: every applicant's linearized log-odds
$W^{-1}$: the inverse of $W$ : divides each miss by its variance
$z_i$: applicant $i$ 's working response

Derivation.

Newton on the loss (Ch 2, Section 9): $w \leftarrow w - H^{-1}g$ , with $g = X^\top(p - y)$ (Card 3) and $H = X^\top WX$ (Card 4).

\mathbf w_{\text{new}} = \mathbf w + (X^\top W X)^{-1}X^\top(\mathbf y - \mathbf p)

Write the current $w$ as $(X^\top WX)^{-1}X^\top WXw$ , so both terms share the same inverse.

\mathbf w_{\text{new}} = (X^\top W X)^{-1}\big[X^\top W X\mathbf w + X^\top(\mathbf y - \mathbf p)\big]

Factor $X^\top W$ out of the bracket. The term $W^{-1}(y - p)$ is what multiplying by $W$ undoes.

\mathbf w_{\text{new}} = (X^\top W X)^{-1}X^\top W\big[X\mathbf w + W^{-1}(\mathbf y - \mathbf p)\big] = (X^\top W X)^{-1}X^\top W\mathbf z

Recognize the form: it is Ch 1's normal equation with weights, the solution of weighted least squares.

(X^\top W X)^{-1}X^\top W\mathbf z = \arg\min_{\mathbf b}\;\sum_i W_{ii}\,(z_i - \mathbf x_i^\top\mathbf b)^2

$\arg\min_{\mathbf b}$: the $\mathbf b$ that makes what follows as small as possible
$\mathbf b$: candidate weights in the weighted least-squares fit
$W_{ii}$: the $i$ -th diagonal entry of $W$ : applicant $i$ 's weight, $p_i(1 - p_i)$

What z and W mean: z is a first-order Taylor expansion of logit(y) around the current p, and Wᵢᵢ is one over z's variance. So each step is the best weighted line through linearized log-odds.

\text{logit}(y_i) \approx \eta_i + (y_i - p_i)\frac{d\,\text{logit}\,p}{dp} = \eta_i + \frac{y_i - p_i}{p_i(1 - p_i)} = z_i, \qquad \text{Var}(z_i) \approx \frac{p_i(1 - p_i)}{p_i^2(1 - p_i)^2} = \frac{1}{W_{ii}}

$\frac{d\,\text{logit}\,p}{dp}$: how fast the log-odds changes as you nudge $p$ : equals $1/(p(1 - p))$

First step on the ten files from $w = 0$ . Every $p = \tfrac12$ and $W = \tfrac14I$ , so $z_i = (y_i - \tfrac12)/\tfrac14 = \pm 2$ . Equivalently, solve $H \Delta = X^\top(y - p) = (-1, -2.1)$ with $H$ from Card 4 ( $\det H = 2.5 \times 1.21 - 0.01 = 3.015$ ).

\Delta = \frac{1}{3.015}\begin{pmatrix} 1.21 & 0.1 \\ 0.1 & 2.5 \end{pmatrix}\begin{pmatrix} -1 \\ -2.1 \end{pmatrix} = \frac{1}{3.015}\begin{pmatrix} -1.42 \\ -5.35 \end{pmatrix} = \begin{pmatrix} -0.471 \\ -1.774 \end{pmatrix}

$\Delta$: the first Newton step from $\mathbf w = 0$ : the change in (intercept, slope)

Keep going. The error drops 2.77 → 0.991 → 0.268 → 0.0232 → 0.000186 → $1.2 \times 10^{-8}$ . In the last steps $e_{k+1} \approx 0.35 e_k^2$ : quadratic convergence, where the correct digits roughly double each step. Five steps land within $1.2 \times 10^{-8}$ of the MLE; a stop-when-the-step-is-below- $10^{-8}$ rule fires on step 7.

(0, 0) \to (-0.471, -1.774) \to (-0.722, -2.498) \to (-0.815, -2.743) \to (-0.8240, -2.7656) \to (-0.824116, -2.76581)

Key move. Information (I): the Newton step, rearranged.

Common slip. Forgetting to divide $(y - p)$ by $W$ in the working response, or letting $W^{-1}$ blow up as some $p_i \to 0$ or 1 (a symptom of separation; real code clips $p$ ). Also claiming "IRLS needs hundreds of steps". It is gradient descent that needs more (13 to 191 here).

How it's asked. "Compute the first Newton step on this data by hand", or "why is it called iteratively reweighted least squares?" A strong answer gives the cost (about $n\cdot p^2 + p^3$ per step, a handful of steps). It notes that for the canonical logit link Newton and Fisher scoring coincide, because the Hessian does not depend on $y$ . It also says that past $\sim 10^4$ features or with streaming data you switch to L-BFGS or SGD (Ch 2).

Card 6 · Score equations ⇒ calibration in the large ★★

Statement. At the MLE of a model with an intercept, the predicted probabilities add up to the number of defaults, and the model reproduces the default-weighted total of every feature. An L2 penalty on the slopes keeps the first fact; penalizing the intercept breaks it.

\sum_i \hat p_i = \sum_i y_i, \qquad \sum_i \hat p_i\,x_{ij} = \sum_i y_i\,x_{ij} \;\;\text{for every column } j

$\hat p_i$: applicant $i$ 's fitted default probability at the MLE (hat = estimated from data)
$x_{ij}$: applicant $i$ 's value in column $j$ of $X$
$j$: which column of $X$ (which feature) the equation is for

Derivation.

The MLE sets the gradient to zero (Card 3): one equation per column of X.

X^\top(\mathbf y - \hat{\mathbf p}) = \mathbf 0

$\hat{\mathbf p}$: all fitted probabilities as one vector

The intercept's column is all 1s, so its equation says the residuals sum to zero: mean $\hat p = \bar y$ .

\mathbf 1^\top(\mathbf y - \hat{\mathbf p}) = 0 \;\;\Rightarrow\;\; \frac1n\sum_i \hat p_i = \bar y

$\frac1n$: one over the number of applicants: turns the sum into an average
$\mathbf 1$: a vector of 1s: the intercept's column of $X$
$\bar y$: the share of applicants who defaulted (bar = mean): 0.4 on the ten files

Each feature column gives the second fact. For a 0/1 dummy column, it says the mean $\hat p$ within that category equals the category's observed rate: every category is calibrated.

\sum_i x_{ij}(y_i - \hat p_i) = 0 \;\;\overset{x_{ij} \in \{0,1\}}{\Longrightarrow}\;\; \frac{\sum_{i:\,x_{ij}=1}\hat p_i}{\#\{i: x_{ij}=1\}} = \frac{\sum_{i:\,x_{ij}=1} y_i}{\#\{i: x_{ij}=1\}}

$\Longrightarrow$: implies (when column $j$ is a 0/1 dummy)
$\sum_{i:\,x_{ij}=1}$: add up over the applicants in category $j$ (dummy = 1)
$\#\{i: x_{ij}=1\}$: the number of applicants in category $j$

Add an L2 penalty on the slopes only. The intercept's component is untouched, so mean $\hat p = \bar y$ survives; the slope equations no longer balance.

\nabla\Big[-\ell + \tfrac{\lambda}{2}\sum_{j \ge 1} w_j^2\Big] = \mathbf 0 \;\Rightarrow\; \sum_i (y_i - \hat p_i) = 0, \qquad \sum_i x_{ij}(y_i - \hat p_i) = \lambda w_j \neq 0

$\nabla$: the gradient: one slope per weight; points uphill
$\lambda$: the L2 penalty strength ( $1/C$ in sklearn)
$\sum_{j \ge 1}$: add up over the slopes only, skipping the intercept $w_0$
$w_j$: the weight on feature $j$

Penalize the intercept too (liblinear's default) and the first equation breaks.

\sum_i (y_i - \hat p_i) = \lambda w_0 \;\;\Rightarrow\;\; \bar{\hat p} \neq \bar y

$\lambda w_0$: the penalty's pull on the intercept $w_0$ , which breaks the balance
$\bar{\hat p}$: the average fitted probability over all applicants

What it does not guarantee: these are only as many linear constraints as $X$ has columns. A misspecified model can satisfy them and still be wrong inside bins of $\hat p$ , and none of it holds on new data whose base rate has moved (Card 13).

\text{guaranteed (training data): } X^\top(\mathbf y - \hat{\mathbf p}) = \mathbf 0 \qquad \text{not guaranteed: } P(y = 1 \mid \hat p = q) = q \;\text{ for every } q

$P(y = 1 \mid \hat p = q)$: the actual default rate among applicants the model scores at $q$
$q$: any predicted probability level (one bin of $\hat p$ )

Check. On the ten files the two equations hold to machine precision. On the book, sklearn minimizes $\tfrac12\lVert w\rVert^2 + C\cdot\sum(\text{log loss})$ , so step 4's $\lambda$ is $1/C$ : at $C = 0.01$ the slope equations read $X^\top(y - \hat p) = (-82.2, 47.2, -21.3) = w/C$ , while the intercept's equation still balances.

\text{toy: } \sum_i \hat p_i = 4.000 = \sum_i y_i, \quad \sum_i x_i\hat p_i = -2.300 = \sum_i x_i y_i \qquad \text{book, } C = 0.001: \;\; \bar{\hat p} - \bar y = 1.7\times 10^{-10}\;(\text{lbfgs}), \quad \bar{\hat p} = 0.217 \text{ vs } \bar y = 0.119\;(\text{liblinear})

$C$: sklearn's inverse penalty strength, $1/\lambda$

Key move. Gradient (G): the stationarity condition, read column by column.

Common slip. Reading this as "logistic regression is calibrated everywhere". It is calibrated in the large and along each feature, on the training data, if the intercept is free. Class weights and resampling break it too (Card 12).

How it's asked. "People say logistic regression is calibrated out of the box. Is that true, and when does it fail?" A strong answer derives it from the intercept's score equation, names what breaks it (Shrink, Sample, Shift: penalties, resampling or weights, base-rate drift), and notes that every canonical-link GLM shares the property: Poisson regression gives $\sum\hat{\mu} = \sum y$ (Card 17).

Card 7 · Why MSE through a sigmoid fails ★★

Statement. Squared error on a sigmoid output has a gradient that vanishes when the model is confidently wrong, and it is not convex. Cross-entropy has neither problem.

\frac{\partial}{\partial\eta}\big(y - \sigma(\eta)\big)^2 = -2\big(y - \sigma(\eta)\big)\,\sigma(\eta)\big(1 - \sigma(\eta)\big), \qquad \frac{\partial}{\partial\eta}\big[\ln(1 + e^{\eta}) - y\eta\big] = \sigma(\eta) - y

$\frac{\partial}{\partial\eta}$: how fast what follows changes as you nudge the log-odds $\eta$

Derivation.

Chain rule on squared error. Unlike Card 3, the $\sigma'$ factor has nothing to cancel against.

\frac{d}{d\eta}\big(y - \sigma\big)^2 = 2\big(y - \sigma\big)\cdot\big(-\sigma'\big) = -2\big(y - \sigma\big)\,\sigma(1 - \sigma)

$\frac{d}{d\eta}$: how fast what follows changes as you nudge $\eta$ (one variable, so $d$ rather than $\partial$ )

Confidently wrong: $y = 0$ but $\sigma = 0.99$ . The squared-error gradient is crushed by $\sigma(1 - \sigma) = 0.0099$ ; cross-entropy's is not. As $\sigma \to 1$ the first goes to 0 and the second to 1.

\text{MSE: } 2(0.99)(0.99)(0.01) = 0.0196 \qquad \text{CE: } 0.99 - 0 = 0.99 \qquad (\text{about } 50\times \text{ stronger})

Curvature of squared error in $\eta$ , for $y = 0$ , where the row loss is $\sigma^2$ . Use $\sigma'' = \sigma'(1 - 2\sigma)$ .

\frac{d^2}{d\eta^2}\sigma^2 = 2\sigma'^2 + 2\sigma\sigma'' = 2\sigma'\big[\sigma(1 - \sigma) + \sigma(1 - 2\sigma)\big] = 2\sigma^2(1 - \sigma)(2 - 3\sigma)

$\frac{d^2}{d\eta^2}$: the curvature in $\eta$ : how fast the slope itself changes
$\sigma''$: the sigmoid's second derivative, $\sigma'(1 - 2\sigma)$

That is negative whenever $\sigma > 2/3$ , so the row loss is concave there. Along a direction in $w$ , $\eta_i$ moves linearly, so a row that is confidently wrong contributes a concave stretch to the loss in $w$ . Once such rows dominate the sum, the total loss is concave along that direction: non-convex. (Step 6 shows that this happens on the ten files.)

\sigma > \tfrac23 \;\Rightarrow\; \frac{d^2}{d\eta^2}\sigma^2 < 0

Cross-entropy's row loss in $\eta$ has curvature $\sigma(1 - \sigma) > 0$ everywhere: convex for every $\eta$ .

\frac{d^2}{d\eta^2}\big[\ln(1 + e^{\eta}) - y\eta\big] = \sigma(\eta)\big(1 - \sigma(\eta)\big) > 0

Check on the ten files. Profile both losses over $w_1$ , holding $w_0$ at its best cross-entropy value for each $w_1$ . Squared error is concave on $w_1 \in [-15, -4.3]$ and $[1.1, 5]$ . At $w_1 = -15$ it sits on a plateau 18 times flatter than cross-entropy, which stays a bowl.

\frac{\partial\,\text{MSE}}{\partial w_1}\Big|_{w_1 = -15} = -0.0048, \qquad \frac{\partial\,\text{CE}}{\partial w_1}\Big|_{w_1 = -15} = -0.087, \qquad \frac{0.087}{0.0048} \approx 18

$\frac{\partial\,\text{MSE}}{\partial w_1}$: how fast the squared-error loss changes as you nudge the slope $w_1$
$\frac{\partial\,\text{CE}}{\partial w_1}$: the same slope for the cross-entropy loss
$w_1 = -15$: evaluated with the slope on the scaled credit score set to −15

Key move. Gradient by the chain rule (G): the $\sigma'$ that cancels in cross-entropy survives in squared error.

Common slip. Saying squared error is "wrong" for probabilities. As an evaluation metric it is the Brier score, a proper scoring rule (Ch 5). The problem is training through a sigmoid: non-convex, deaf to confident mistakes, and not the Bernoulli likelihood.

How it's asked. "An engineer trains a sigmoid output with MSE in PyTorch and learning stalls on badly wrong examples. Why?" A strong answer gives the $\sigma'$ factor, the 0.0196-vs-0.99 comparison, and the 3 Cs of cross-entropy (Convex, Confident mistakes punished, Calibrated). In deep nets, the same vanishing factor is why BCEWithLogitsLoss takes logits, not probabilities.

Card 8 · Odds ratios, marginal effects and the divide-by-4 rule ★

Statement. A one-unit change in $x_j$ adds $\beta_j$ to the log-odds, multiplies the odds by $e^{\beta_j}$ , and changes the probability by an amount that depends on where you start, never more than $\lvert\beta_j\rvert/4$ .

\frac{\text{odds}(x_j + 1)}{\text{odds}(x_j)} = e^{\beta_j}, \qquad \frac{\partial p}{\partial x_j} = \beta_j\,p(1 - p), \qquad \Big|\frac{\partial p}{\partial x_j}\Big| \le \frac{|\beta_j|}{4}, \qquad \text{AME}_j = \frac{\beta_j}{n}\sum_i \hat p_i(1 - \hat p_i)

$\frac{\text{odds}(x_j + 1)}{\text{odds}(x_j)}$: the odds ratio: how the odds of default multiply when $x_j$ rises one unit
$x_j$: feature $j$ , e.g. the credit score in units of 100 points
$\beta_j$: the coefficient on feature $j$ : log-odds added per unit of $x_j$
$\frac{\partial p}{\partial x_j}$: the marginal effect: how fast default probability changes as you nudge $x_j$ , rest fixed
$\text{AME}_j$: the average marginal effect: that slope averaged over all applicants

Derivation.

The log-odds is linear in $x$ , so raising $x_j$ by one unit adds $\beta_j$ , whatever the other features are.

\ln\text{odds}(\mathbf x) = \beta_0 + \sum_j \beta_j x_j \;\;\Rightarrow\;\; \ln\text{odds}(x_j + 1) - \ln\text{odds}(x_j) = \beta_j

$\beta_0 + \sum_j \beta_j x_j$: the log-odds: the intercept $\beta_0$ plus each coefficient times its feature
$\sum_j$: add up over every feature $j$

Exponentiate: the odds multiply by a constant, the odds ratio.

\frac{\text{odds}(x_j + 1)}{\text{odds}(x_j)} = e^{\beta_j}

On the probability scale, chain rule through $\sigma$ (Card 1). The factor $p(1 - p)$ depends on every feature, so the effect does too.

\frac{\partial p}{\partial x_j} = \sigma'(\eta)\,\frac{\partial\eta}{\partial x_j} = \beta_j\,p(1 - p)

$\frac{\partial\eta}{\partial x_j}$: how the log-odds moves per unit of $x_j$ : just $\beta_j$

Bound it: $p(1 - p) \le \tfrac14$ , with equality at $p = \tfrac12$ .

\Big|\frac{\partial p}{\partial x_j}\Big| \le \frac{|\beta_j|}{4}

Two summaries. The AME averages each row's derivative; the MEM evaluates the derivative at the mean row. They differ because $p(1 - p)$ is nonlinear (Jensen), and the AME is the one to report.

\text{AME}_j = \frac{\beta_j}{n}\sum_i \hat p_i(1 - \hat p_i), \qquad \text{MEM}_j = \beta_j\,\sigma(\bar{\mathbf x}^\top\hat{\mathbf w})\big(1 - \sigma(\bar{\mathbf x}^\top\hat{\mathbf w})\big)

$\text{MEM}_j$: the marginal effect at the mean: the slope at the average applicant
$\bar{\mathbf x}$: the average applicant's features (bar = mean)

Odds ratio vs risk ratio. Write each odds as p/(1 − p) and rearrange: the two agree only when both probabilities are small.

\text{RR} = \frac{p_1}{p_0} = \text{OR}\cdot\frac{1 - p_1}{1 - p_0} \;\approx\; \text{OR} \;\;\text{when } p_0, p_1 \ll 1

$p_1$: the default probability in one group (e.g. one unit higher on $x_j$ )
$p_0$: the default probability in the comparison group

Confidence interval for an OR: exponentiate the endpoints of the interval for $\beta$ . The map is monotone, so coverage carries over; the interval is not symmetric around the OR.

\big[e^{\hat\beta - 1.96\,\text{SE}},\; e^{\hat\beta + 1.96\,\text{SE}}\big]

$\hat\beta$: the fitted coefficient

Check on the ten files ( $\hat{\beta}_1 = -2.766$ per 100 points, K at $p = 0.305$ ).

\text{OR}_{100} = e^{-2.766} = 0.0629, \quad \text{OR}_{10} = e^{-0.2766} = 0.758, \quad \frac{\partial p_K}{\partial x} = -2.766 \times 0.305 \times 0.695 = -0.586, \quad \frac{\beta_1}{4} = -0.691

$\frac{\partial p_K}{\partial x}$: how fast Applicant K's default probability changes per unit of $x$ (100 score points)
$\beta_1$: the slope on the scaled credit score

Over all ten files the AME is −0.389 and the MEM −0.610. A full +100-point move takes K from 0.305 to 0.0269, a change of −0.278: the derivative overstates a large move because the curve flattens. The Wald interval for $\text{OR}_{100}$ is [0.0022, 1.80]; the profile interval is [0.00049, 0.800].

Key move. Gradient by the chain rule (G) once more: $\partial p/\partial x = \sigma' \cdot \beta$ .

Common slip. Reading a coefficient, or $\text{OR} - 1$ , as a change in probability ("an OR of 0.063 means a 6% chance"). Building the OR interval as $\text{OR} \pm 1.96\cdot\operatorname{SE}$ . Calling an odds ratio a risk ratio when the outcome is common.

How it's asked. "The credit-score coefficient is −2.77 per 100 points. What do you tell the credit committee?" A strong answer gives all three scales (log-odds add, odds multiply, probabilities depend), the divide-by-4 bound, and AMEs for humans. It adds that an interaction on the log-odds scale is not the interaction on the probability scale (Ai and Norton, 2003).

Card 9 · Fisher information and standard errors ★★

Statement. One Bernoulli carries information $1/(p(1 - p))$ about $p$ but $p(1 - p)$ about its log-odds. For logistic regression the information matrix is $X^\top WX$ , and its inverse gives the standard errors.

I(p) = \frac{1}{p(1 - p)}, \qquad I(\theta) = p(1 - p) \;\;(\theta = \text{logit}\,p), \qquad I(\mathbf w) = X^\top W X, \qquad \widehat{\text{Var}}(\hat{\mathbf w}) = (X^\top\hat W X)^{-1}

$I(p)$: the Fisher information one 0/1 outcome carries about $p$
$I(\theta)$: the information it carries about the log-odds $\theta$
$\theta = \text{logit}\,p$: $\theta$ is the log-odds of $p$
$I(\mathbf w)$: the information matrix for all the weights
$\widehat{\text{Var}}(\hat{\mathbf w})$: the estimated covariance matrix of the fitted weights
$(X^\top\hat W X)^{-1}$: the inverse information with $W$ at the fitted probabilities; its diagonal holds squared SEs

Derivation.

One row on the p scale: log-likelihood, score, and second derivative.

\ell(p) = y\ln p + (1 - y)\ln(1 - p), \qquad \ell'(p) = \frac{y}{p} - \frac{1 - y}{1 - p}, \qquad \ell''(p) = -\frac{y}{p^2} - \frac{1 - y}{(1 - p)^2}

$\ell'$: the slope of the log-likelihood (prime = derivative): the score
$\ell''$: the curvature of the log-likelihood (second derivative)

Fisher information is $-\operatorname{E}[\ell'']$ . Substitute $\operatorname{E}[y] = p$ .

I(p) = \frac{p}{p^2} + \frac{1 - p}{(1 - p)^2} = \frac1p + \frac{1}{1 - p} = \frac{1}{p(1 - p)}

$\frac1p$: one over $p$ : the information from the default term

On the log-odds scale $\theta$ , directly: $\ell(\theta) = y\theta - \ln(1 + e^\theta)$ (Card 2). The second derivative does not involve $y$ , so observed and expected information agree.

\ell''(\theta) = -\sigma(\theta)\big(1 - \sigma(\theta)\big) \;\;\Rightarrow\;\; I(\theta) = p(1 - p)

$\ell''(\theta)$: the curvature of one row's log-likelihood in the log-odds $\theta$
$\sigma(\theta)$: the sigmoid of $\theta$ : the probability $p$

Same answer by reparametrization: information scales with the squared Jacobian, $dp/d\theta = p(1 - p)$ .

I(\theta) = I(p)\Big(\frac{dp}{d\theta}\Big)^2 = \frac{p^2(1 - p)^2}{p(1 - p)} = p(1 - p)

$\frac{dp}{d\theta}$: how fast $p$ changes as you nudge the log-odds: the Jacobian $p(1 - p)$

Many rows with $\theta_i = x_i^\top w$ : information adds over independent rows, and the chain rule brings in $x_ix_i^\top$ . It equals the Hessian of Card 4.

I(\mathbf w) = \sum_i p_i(1 - p_i)\,\mathbf x_i\mathbf x_i^\top = X^\top W X

Why the inverse information is the variance. At the true $w$ , the score $U(w) = X^\top(y - p)$ has mean zero because $\operatorname{E}[y] = p$ . Its variance is the information, because $\operatorname{Var}(y_i) = p_i(1 - p_i)$ and the rows are independent. This is the information identity: the variance of the score equals the expected curvature.

\mathbb E\big[U(\mathbf w)\big] = X^\top(\mathbb E[\mathbf y] - \mathbf p) = \mathbf 0, \qquad \text{Var}\big(U(\mathbf w)\big) = X^\top\,\text{Var}(\mathbf y)\,X = X^\top W X = I(\mathbf w)

$\mathbb E$: the average over repeated samples of outcomes (expectation)
$U(\mathbf w)$: the score: the gradient of the log-likelihood, $X^\top(\mathbf y - \mathbf p)$
$\text{Var}$: the variance (or covariance matrix): spread over repeated samples

$U$ is a sum of $n$ independent mean-zero vectors, so for large $n$ the central limit theorem makes it approximately $N(0, I)$ . The MLE solves $U(\hat w) = 0$ . Expand around the truth to first order, using $-\partial U/\partial w^\top = I$ . Then $\hat w - w$ is a fixed matrix times an approximately normal vector, and its covariance sandwiches $I$ between two copies of $I^{-1}$ . (Ch 6 makes this formal.)

\mathbf 0 = U(\hat{\mathbf w}) \approx U(\mathbf w) - I(\mathbf w)(\hat{\mathbf w} - \mathbf w) \;\Rightarrow\; \hat{\mathbf w} - \mathbf w \approx I^{-1}U \;\Rightarrow\; \text{Var}(\hat{\mathbf w}) \approx I^{-1}\,I\,I^{-1} = (X^\top W X)^{-1}

$U(\hat{\mathbf w})$: the score at the MLE, zero by definition
$I$: the information matrix $I(\mathbf w)$
$I^{-1}$: the inverse information (matrix inverse: undoes it)
$U$: the score $U(\mathbf w)$

So the SEs are the square roots of the diagonal of $(X^\top\hat W X)^{-1}$ , with $W$ evaluated at the estimate.

\hat{\mathbf w} \overset{a}{\sim} N\big(\mathbf w,\; (X^\top W X)^{-1}\big), \qquad \text{SE}(\hat w_j) = \sqrt{\big[(X^\top\hat W X)^{-1}\big]_{jj}}

$\overset{a}{\sim}$: is approximately distributed as, for large $n$
$N$: a normal distribution (mean, then variance or covariance)
$\text{SE}(\hat w_j)$: the standard error of fitted weight $j$
$_{jj}$: take the $j$ -th diagonal entry

Intercept only: $X$ is a column of 1s and every $\hat p = \bar y$ . The information is $n\cdot\bar y(1 - \bar y)$ , which rearranges into the counts of each class.

\text{SE}(\hat\beta_0) = \frac{1}{\sqrt{n\,\bar y(1 - \bar y)}} = \sqrt{\frac{1}{n\bar y} + \frac{1}{n(1 - \bar y)}} = \sqrt{\frac{1}{n_1} + \frac{1}{n_0}}

$\text{SE}(\hat\beta_0)$: the standard error of the fitted intercept, intercept-only model
$n_1$: the number of defaulters (4 on the ten files)
$n_0$: the number of repayers (6 on the ten files)

Where the information lives. On the log-odds scale, a row at $p = 0.5$ carries the most and a row at $p = 0.01$ almost none. On the $p$ scale it is the reverse. A rare event pins $p$ down in absolute terms but leaves its log-odds vague. That is why rare-outcome models need many events, not just many rows.

p = 0.5:\;\; I(\theta) = 0.25,\;\; I(p) = 4 \qquad\qquad p = 0.01:\;\; I(\theta) = 0.0099,\;\; I(p) = 101

Check on the ten files, intercept only (4 defaults in 10), then the sample size for a precise intercept on the book. With the score feature, inverting $X^\top\hat W X$ gives SEs 0.931 and 1.710.

\hat\beta_0 = \ln\frac{0.4}{0.6} = -0.405, \qquad \text{SE} = \frac{1}{\sqrt{10 \times 0.4 \times 0.6}} = \sqrt{\tfrac14 + \tfrac16} = 0.645, \qquad n_{\pm 0.05} = \frac{1.96^2}{0.05^2 \times 0.12 \times 0.88} = 14{,}551

$\hat\beta_0$: the fitted intercept
$n_{\pm 0.05}$: the sample size that pins the book's intercept within ±0.05, at 95% confidence

A simulation at that n puts 95.1% of intercept estimates within ±0.05 of the truth.

Key move. Information (I).

Common slip. Giving $I(p)$ when asked about the log-odds, or forgetting to square the Jacobian. Assuming observed = expected information for every link: true for the canonical logit, false for probit.

How it's asked. "Derive the Fisher information of a Bernoulli. Where is a 0/1 outcome most informative?" A strong answer says it depends on the parameter: rows near 50% teach the most about log-odds, while rows near 0 or 1 pin down $p$ itself. It links to design: the D-optimal two points for learning a slope sit at $\eta = \pm 1.543$ , that is $p = 0.176$ and 0.824 (Bank Drill 4).

Card 10 · Wald, likelihood-ratio and score tests; deviance ★★

Statement. Three tests of $H_0: \beta = \beta_0$ measure the same log-likelihood three ways: Wald = Width (distance from $\hat{\beta}$ in SEs), LR = Height (log-likelihood lost by imposing $H_0$ ), Score = Slope (steepness at $\beta_0$ ). All three tend to $\chi^2_1$ under $H_0$ and agree when the log-likelihood is quadratic.

W = \frac{(\hat\beta - \beta_0)^2}{\widehat{\text{SE}}(\hat\beta)^2}, \qquad LR = 2\big(\ell(\hat{\boldsymbol\beta}) - \ell(\tilde{\boldsymbol\beta}_0)\big), \qquad S = \frac{U(\tilde{\boldsymbol\beta}_0)^2}{I(\tilde{\boldsymbol\beta}_0)}, \qquad D = -2\big(\ell - \ell_{\text{sat}}\big)

$W$: here the Wald statistic: squared distance of $\hat\beta$ from $\beta_0$ , in SEs (not the weight matrix)
$\hat\beta$: the fitted coefficient being tested
$(\hat\beta - \beta_0)^2$: the squared gap between the estimate and $\beta_0$ , the value $H_0$ claims
$\widehat{\text{SE}}(\hat\beta)$: the estimated standard error of $\hat\beta$
$LR$: the likelihood-ratio statistic: twice the log-likelihood lost by imposing $H_0$
$\hat{\boldsymbol\beta}$: the unrestricted fit: the MLE of all weights
$\tilde{\boldsymbol\beta}_0$: the restricted fit: $H_0$ imposed, other weights re-estimated
$S$: the score statistic: the squared slope at the restricted fit, over its information
$U(\tilde{\boldsymbol\beta}_0)$: the score (slope of $\ell$ ) at the restricted fit
$I(\tilde{\boldsymbol\beta}_0)$: the information at the restricted fit
$D$: the deviance: twice the log-likelihood gap between a perfect fit and the model
$\ell_{\text{sat}}$: the saturated model's log-likelihood: each row given its own probability $y_i$

Here $\tilde{\beta}_0$ is the fit with $H_0$ imposed (other parameters re-estimated), $U$ is the score (the gradient of $\ell$ ) and $I$ the information for the tested parameter.

Derivation.

Taylor-expand $\ell$ around $\hat{\beta}$ (scalar case), where $\ell'(\hat{\beta}) = 0$ and $-\ell'' \approx I$ .

\ell(\beta_0) \approx \ell(\hat\beta) - \tfrac12\,I\,(\hat\beta - \beta_0)^2

$\ell(\beta_0)$: the log-likelihood at $\beta_0$ , the value $H_0$ claims

So the height test is approximately the width test: $\text{LR} \approx \text{curvature} \times \text{distance}^2$ .

LR = 2\big(\ell(\hat\beta) - \ell(\beta_0)\big) \approx I\,(\hat\beta - \beta_0)^2 = \frac{(\hat\beta - \beta_0)^2}{\text{SE}^2} = W

Taylor-expand the score around $\hat{\beta}$ : the slope at $\beta_0$ is approximately curvature × distance, so the slope test matches too.

U(\beta_0) \approx U(\hat\beta) - I\,(\beta_0 - \hat\beta) = I\,(\hat\beta - \beta_0) \;\;\Rightarrow\;\; S = \frac{U(\beta_0)^2}{I} \approx I\,(\hat\beta - \beta_0)^2 = W

$U(\beta_0)$: the score (slope of $\ell$ ) at the null value $\beta_0$
$(\beta_0 - \hat\beta)$: the gap from the estimate to the null value
$(\hat\beta - \beta_0)$: the estimate's distance from the null value $\beta_0$

Under $H_0$ , $\hat{\beta} \approx N(\beta_0, 1/I)$ (Card 9, step 7), so $I(\hat{\beta} - \beta_0)^2$ is a squared standard normal. All three tend to $\chi^2_1$ . They part ways when $\ell$ is far from quadratic: small $n$ , large effects, near-separation.

\sqrt{I}\,(\hat\beta - \beta_0) \;\overset{a}{\sim}\; N(0, 1) \;\;\Rightarrow\;\; W,\; LR,\; S \;\overset{a}{\sim}\; \chi^2_1

$\chi^2_1$: the chi-squared distribution with 1 degree of freedom: a squared standard normal

Deviance. The saturated model gives each row its own probability equal to $y_i$ . For 0/1 data that log-likelihood is zero ( $0\cdot\ln 0 = 0$ ), so the deviance is just $-2\ell$ , and the LR test is the drop in deviance.

\ell_{\text{sat}} = \sum_i\big[y_i\ln y_i + (1 - y_i)\ln(1 - y_i)\big] = 0 \;\;\Rightarrow\;\; D = -2\ell(\hat{\mathbf w}), \qquad LR = D_{\text{null}} - D_{\text{model}}

$D_{\text{null}}$: the deviance of the intercept-only model
$D_{\text{model}}$: the deviance of the fitted model

Check on the ten files, $H_0: \beta_1 = 0$ . Wald and LR come straight from the fit.

z = \frac{-2.766}{1.710} = -1.618,\;\; W = 2.62,\;\; p = 0.106 \qquad LR = 13.460 - 8.723 = 4.74,\;\; p = 0.0295

$z$: here the Wald z-statistic: the estimate divided by its SE
$p = 0.106$: the Wald p-value: the chance of a statistic this extreme if $H_0$ were true
$p = 0.0295$: the LR test's p-value

The score test needs only the null fit (intercept only, every $p = 0.4$ ). The information for $\beta_1$ must be adjusted for the estimated intercept: take the Schur complement of the 2×2 information matrix at $p = 0.4$ (entries $0.24\cdot n$ , $0.24\cdot\sum x$ , $0.24\cdot\sum x^2$ ).

U = \sum_i x_i(y_i - 0.4) = -2.3 - 0.4(-0.4) = -2.14, \qquad I_{11\cdot 0} = 0.24(4.84) - \frac{\big(0.24(-0.4)\big)^2}{0.24(10)} = 1.158

$I_{11\cdot 0}$: the information for $\beta_1$ after adjusting for the estimated intercept (a Schur complement)

S = \frac{(-2.14)^2}{1.158} = 3.96, \qquad p = 0.0467

$p = 0.0467$: the score test's p-value

Wald says "not significant" ( $p = 0.106$ ); LR (0.0295) and score (0.0467) say it is. The profile-likelihood interval for $\beta_1$ is [−7.63, −0.22], which excludes 0; the Wald interval [−6.12, 0.59] does not. The log-likelihood is skewed, and Wald's quadratic approximation misses it.

The Hauck–Donner effect. Take a 2×2 table with 10/100 control defaults and 20, 40, 60, 80, 90, 95, 98, 99 treated defaults out of 100. As the effect grows, the Wald z rises and then falls, because the SE at $\hat{\beta}$ grows faster than $\hat{\beta}$ . The LR statistic rises monotonically. Under separation the Wald test fails outright.

\text{Wald } z:\; 1.95,\; 4.58,\; 6.66,\; 8.60,\; \mathbf{9.32},\; 9.07,\; 7.72,\; 6.41 \qquad LR:\; 3.99,\; 25.3,\; 59.4,\; 110,\; 147,\; 172,\; 191,\; 199

Key move. Taylor (T), with Information (I) supplying the curvature.

Common slip. Computing the score test at $\hat{\beta}$ instead of at the restricted fit; forgetting to adjust the score test's information for nuisance parameters; assuming the saturated log-likelihood is zero for grouped binomial data (it isn't); reporting a Wald interval when the profile is visibly skewed.

How it's asked. "On ten loan files, Wald $p = 0.106$ and LR $p = 0.029$ . Which do you report to the credit committee?" A strong answer draws the W-H-S picture, reports LR (and the profile interval) for small $n$ or large effects, and adds two facts. LR is invariant to reparametrization (log-odds vs odds) and Wald is not. The score test needs only the null fit, which makes it cheap for screening many candidate features.

Card 11 · Separation: the MLE does not exist; Firth's fix ★★★

Statement. If some direction $w^*$ puts every defaulter on one side of a hyperplane and every repayer on the other, the log-likelihood rises toward 0 along that ray forever: no finite MLE exists. Firth's penalized likelihood always has a finite maximizer and removes the MLE's $O(1/n)$ bias.

\exists\,\mathbf w^*:\; \tilde y_i\,\mathbf x_i^\top\mathbf w^* > 0 \;\;\forall i \;\;\Rightarrow\;\; \ell(c\,\mathbf w^*) \uparrow 0 \text{ as } c \to \infty, \qquad \ell^*(\mathbf w) = \ell(\mathbf w) + \tfrac12\ln\det\big(X^\top W X\big)

$\mathbf w^*$: a separating direction: every defaulter on one side, every repayer on the other
$\forall i$: for every applicant $i$
$c$: how far you walk along that direction (a scale factor)
$\ell^*(\mathbf w)$: Firth's penalized log-likelihood (star = modified version)
$W$: the diagonal matrix of Bernoulli variances $p_i(1 - p_i)$ (not the Wald statistic)

Derivation.

Margin form (Card 2): the log-likelihood depends on $w$ only through the margins $m_i = \tilde y_ix_i^\top w$ , and every term is negative.

\ell(\mathbf w) = -\sum_i \ln\big(1 + e^{-m_i}\big) < 0 \;\;\text{for every finite } \mathbf w

$m_i$: applicant $i$ 's margin, $\tilde y_i\mathbf x_i^\top\mathbf w$ : positive when the model sides with the actual outcome

Walk along the separating ray $w = c\cdot w^*$ , where every margin $m_i^* = \tilde y_ix_i^\top w^* > 0$ . The derivative in $c$ is a sum of positive terms, so $\ell$ rises for every $c$ .

\frac{d}{dc}\,\ell(c\,\mathbf w^*) = \sum_i m_i^*\,\sigma(-c\,m_i^*) > 0

$\frac{d}{dc}$: how fast the log-likelihood changes as you walk further along the ray
$m_i^*$: applicant $i$ 's margin along $\mathbf w^*$ , positive for every row

Each term goes to 0 as $c \to \infty$ , so $\ell \to 0$ , its upper bound (a likelihood can't exceed 1). The supremum is approached but never attained at a finite $w$ : there is no MLE. Along the way every $p_i(1 - p_i) \to 0$ , so $X^\top WX$ turns singular and the Wald SEs explode.

\lim_{c\to\infty}\ell(c\,\mathbf w^*) = 0 = \sup_{\mathbf w}\ell(\mathbf w), \qquad W_{ii} = \sigma(m_i)\sigma(-m_i) \to 0

Why IRLS then adds a constant each iteration. Far out, $\ln(1 + e^{-cm}) \approx e^{-cm}$ , and the rows with the smallest margin $m$ dominate. Newton on $e^{-cm}$ in $c$ moves by exactly $1/m$ per step, whatever $c$ is.

f(c) = e^{-c\,m}: \quad c_{\text{new}} = c - \frac{f'(c)}{f''(c)} = c - \frac{-m\,e^{-cm}}{m^2 e^{-cm}} = c + \frac1m

$f(c)$: the far-out loss of the smallest-margin rows, as a function of $c$
$m$: the smallest margin among the rows
$c_{\text{new}}$: $c$ after one Newton step
$f'(c)$: its slope in $c$
$f''(c)$: its curvature in $c$
$\frac1m$: one over the smallest margin: the fixed size of each step in $c$

Check on the ten files, DTI alone. Every applicant above 39% DTI defaulted, so $w^* \propto (\text{DTI} - 39)/10$ separates them; the smallest margin is 0.1 (F at 40, E at 38). Along that ray the log-likelihood climbs toward 0 and never reaches it.

c = 0,\, 1,\, 2,\, 5,\, 10,\, 20,\, 50: \qquad \ell(c\,\mathbf w^*) = -6.93,\; -3.90,\; -2.57,\; -1.27,\; -0.680,\; -0.256,\; -0.0134

Under IRLS on $d = (\text{DTI} - 35)/10$ , the ray is $(\text{DTI} - 39)/10 = d - 0.4$ , the direction (−0.4, 1) in (intercept, slope). $\lVert w\rVert$ first grows ×1.6–1.8 per iteration for about five iterations, then by a constant: step 4's $1/m = 10$ along that direction.

\Delta\|\mathbf w\| \;\to\; \frac{1}{m_{\min}}\,\big\|(-0.4,\; 1)\big\| = 10 \times \sqrt{1.16} = 10.8 \;\text{ per iteration}

$\Delta\|\mathbf w\|$: how much the length of the weight vector grows per IRLS iteration
$m_{\min}$: the smallest margin, 0.1

$X^\top WX$ becomes numerically singular at iteration 41. With score + DTI the constant is 8.2 and the fit breaks at iteration 40. statsmodels returns coefficients (−58.5, 32.6, 151) with SEs of about 44,000, 54,000 and 106,000, plus ConvergenceWarning and PerfectSeparationWarning.

Firth's fix: add half the log-determinant of the information. Along any separating ray $W \to 0$ , so $\ln \det(X^\top WX) \to -\infty$ while $\ell$ stays bounded by 0. The penalized objective falls at infinity, and its maximum is finite.

\ell^*(c\,\mathbf w^*) = \underbrace{\ell(c\,\mathbf w^*)}_{\le\, 0} + \underbrace{\tfrac12\ln\det\big(X^\top W(c)X\big)}_{\to\, -\infty} \;\to\; -\infty

$W(c)$: the weight matrix $W$ at the point $c\,\mathbf w^*$ ; its entries shrink to 0

The modified score. Differentiate the penalty with the log-determinant rule. Only $W$ depends on $w$ , and each $W_{ii} = p_i(1 - p_i)$ has derivative $\sigma'(1 - 2p)$ times $x_{ij}$ .

\frac{\partial}{\partial w_j}\ln\det A = \text{tr}\Big(A^{-1}\frac{\partial A}{\partial w_j}\Big), \qquad \frac{\partial (X^\top W X)}{\partial w_j} = \sum_i \frac{\partial W_{ii}}{\partial w_j}\,\mathbf x_i\mathbf x_i^\top, \qquad \frac{\partial W_{ii}}{\partial w_j} = p_i(1 - p_i)(1 - 2p_i)\,x_{ij}

$\frac{\partial}{\partial w_j}$: how fast what follows changes as you nudge weight $w_j$
$A$: any invertible matrix; here $X^\top W X$
$\frac{\partial A}{\partial w_j}$: how each entry of $A$ changes as you nudge $w_j$
$\frac{\partial (X^\top W X)}{\partial w_j}$: how the information matrix changes as you nudge $w_j$
$\frac{\partial W_{ii}}{\partial w_j}$: how applicant $i$ 's weight $p_i(1 - p_i)$ changes as you nudge $w_j$

Use $\operatorname{tr}(A^{-1}xx^\top) = x^\top A^{-1}x$ and name the leverage $h_i = p_i(1 - p_i)x_i^\top(X^\top WX)^{-1}x_i$ , the diagonal of the logistic hat matrix.

\frac{\partial}{\partial w_j}\tfrac12\ln\det(X^\top W X) = \tfrac12\sum_i \mathbf x_i^\top(X^\top W X)^{-1}\mathbf x_i\;p_i(1 - p_i)(1 - 2p_i)\,x_{ij} = \sum_i h_i\big(\tfrac12 - p_i\big)x_{ij}

$h_i$: applicant $i$ 's leverage: how strongly their row pulls on the fit

U^*(\mathbf w) = \sum_i\big(y_i - p_i + h_i(\tfrac12 - p_i)\big)\,\mathbf x_i = \mathbf 0

$U^*(\mathbf w)$: Firth's modified score

In words: each row is nudged toward $\tfrac12$ in proportion to its leverage, as if it carried an extra $h_i/2$ of a default and $h_i/2$ of a repayment. $\tfrac12 \ln \det I(w)$ is the log of the Jeffreys prior, so Firth's estimate is that prior's posterior mode.

Why the penalty removes the bias, in one dimension. Take a single canonical parameter $\theta$ with $\ell(\theta) = n[\bar y\theta - K(\theta)]$ . For the Bernoulli, $K(\theta) = \ln(1 + e^\theta)$ , $K' = p$ , $K'' = p(1 - p)$ and $K''' = p(1 - p)(1 - 2p)$ . The score $U = n(\bar y - K')$ has mean 0. Its derivatives $U' = -nK''$ and $U'' = -nK'''$ are not random. Expand $U(\hat{\theta}) = 0$ to second order in $\delta = \hat{\theta} - \theta$ , then take expectations, using $\operatorname{E}[\delta^2] \approx 1/(nK'')$ .

0 = U - nK''\,\delta - \tfrac12 nK'''\,\delta^2 \;\;\Rightarrow\;\; nK''\,\mathbb E[\delta] \approx -\tfrac12 nK'''\cdot\frac{1}{nK''} \;\;\Rightarrow\;\; b(\theta) \equiv \mathbb E[\hat\theta - \theta] \approx -\frac{K'''}{2n\,K''^2}

$K''$: the Bernoulli variance $p(1 - p)$ : second derivative of $K(\theta) = \ln(1 + e^\theta)$
$K'''$: its third derivative, $p(1 - p)(1 - 2p)$ : the skew
$\theta$: the log-odds (the canonical parameter), $\text{logit}\,p$
$\delta$: the estimation error, $\hat\theta - \theta$
$b(\theta)$: the MLE's bias at $\theta$
$\hat\theta$: the MLE of the log-odds

Firth's (1993) idea is to subtract $I(\theta)\cdot b(\theta)$ from the score, so the root moves back by the bias. That correction is exactly the derivative of $\tfrac12 \ln I(\theta)$ , the Jeffreys penalty.

U^*(\theta) = U(\theta) - I(\theta)\,b(\theta) = U(\theta) + nK''\cdot\frac{K'''}{2nK''^2} = U(\theta) + \frac{K'''}{2K''} = \frac{d}{d\theta}\Big[\ell(\theta) + \tfrac12\ln\underbrace{nK''(\theta)}_{I(\theta)}\Big]

$U^*(\theta)$: Firth's corrected score for $\theta$
$\frac{d}{d\theta}$: how fast the bracket changes as you nudge $\theta$

For the Bernoulli the bias is $-(1 - 2p)/(2n p(1 - p))$ . When $p < \tfrac12$ the true log-odds is negative and the bias is negative too, so the MLE is pushed away from zero. Intercept only, the leverages are all $1/n$ , and step 7's modified score solves in closed form as "add $\tfrac12$ to each class count".

\sum_i\Big(y_i - p + \tfrac1n(\tfrac12 - p)\Big) = 0 \;\Rightarrow\; \hat p_{\text{Firth}} = \frac{n_1 + \tfrac12}{n + 1}; \qquad \text{ten files: } \ln\frac{4.5}{6.5} = -0.368 \;\text{ vs MLE } -0.405, \quad b = -\frac{0.2}{2(10)(0.24)} = -0.042

$\hat p_{\text{Firth}}$: Firth's estimate of the default rate (intercept-only model)
$b$: the estimated bias of the MLE intercept
$\tfrac1n$: each row's leverage $h_i$ when the model is intercept-only

The MLE minus its estimated bias, $-0.405 + 0.042 = -0.364$ , lands next to Firth's −0.368.

Check. Firth on DTI alone, $d = (\text{DTI} - 35)/10$ , gives SEs (1.12, 1.77). On score + DTI it gives (−0.563, −0.104, 2.18). L2 at $C = 1$ on standardized features is also finite, (−0.695, −0.535, 1.22), but shrunk toward zero by design. The bias claim holds in simulation on the book's single-feature DGP over 2,000 samples per $n$ . Only 0.3% of the $n = 50$ samples separate, so separation does not cause the MLE's +18% bias there. It is the $O(1/n)$ bias of step 8, and it falls to +8.7% at $n = 100$ and +0.5% at $n = 1{,}000$ .

\text{Firth, DTI alone: } (-0.827,\; 2.83) \qquad n = 50:\;\; \bar{\hat\beta}_{\text{MLE}} = -2.40\;(+18\%), \;\; \bar{\hat\beta}_{\text{Firth}} = -2.01\;(-0.9\%), \;\; \beta_{\text{true}} = -2.03

$\bar{\hat\beta}_{\text{MLE}}$: the MLE slope averaged over 2,000 simulated samples
$\bar{\hat\beta}_{\text{Firth}}$: Firth's slope, averaged the same way
$\beta_{\text{true}}$: the true slope in the simulation

Key move. Likelihood (L) along a ray, then Information (I) as the penalty, with Taylor (T) for the bias it removes.

Common slip. "The loss is convex, so the optimizer's answer is the minimum." It stopped at max_iter. Other slips: trusting a huge $\hat{\beta}$ with a huge SE; dropping the separating feature, which is the strongest predictor you have; missing quasi-complete separation (ties on the boundary), where only some coefficients run off.

How it's asked. "statsmodels prints a DTI coefficient of 151 with $\operatorname{SE} \approx 106{,}000$ on a 10-row review sample. What happened, and what do you do?" A strong answer gives the ray argument and detection (huge $\lvert\hat\beta\rvert$ and SE, warnings, a linear-programming feasibility check for $w$ with $\tilde y_ix_i^\top w \ge 1$ ). For the fix: Firth, L2, merging sparse categories, exact logistic regression, or a profile-likelihood one-sided bound. Then, which to use: Firth when you need inference, L2 when you need predictions.

Card 12 · Case-control sampling shifts only the intercept ★★

Statement. If cases are kept with probability $s_1$ and controls with probability $s_0$ , independently of $x$ , the sample still follows a logistic model with the same slopes. Only the intercept moves, by $\ln(s_1/s_0)$ . Class weights $w_1, w_0$ do the same thing with $\ln(w_1/w_0)$ .

\text{logit}\,P(y = 1 \mid \mathbf x, S = 1) = \Big(\beta_0 + \ln\frac{s_1}{s_0}\Big) + \mathbf x^\top\boldsymbol\beta, \qquad \text{weights: } \beta_0 \to \beta_0 + \ln\frac{w_1}{w_0}

$P(y = 1 \mid \mathbf x, S = 1)$: the default probability among applicants who made it into the training sample
$\beta_0$: the population intercept
$s_1$: the chance a defaulter is kept in the training sample
$s_0$: the chance a repayer is kept
$\boldsymbol\beta$: the slopes, one per feature
$w_1$: the class weight on defaulters (not a slope)
$w_0$: the class weight on repayers (not the intercept)

Derivation.

Sampling depends on the outcome only. $S = 1$ means "in the training set".

P(S = 1 \mid y = 1, \mathbf x) = s_1, \qquad P(S = 1 \mid y = 0, \mathbf x) = s_0

$P(S = 1 \mid y = 1, \mathbf x)$: the chance a defaulter makes it into the training sample ( $S = 1$ means kept)
$P(S = 1 \mid y = 0, \mathbf x)$: the chance a repayer makes it in

Bayes' rule for the probability of default among sampled rows.

P(y = 1 \mid \mathbf x, S = 1) = \frac{s_1\,P(y = 1 \mid \mathbf x)}{s_1\,P(y = 1 \mid \mathbf x) + s_0\,P(y = 0 \mid \mathbf x)}

$P(y = 1 \mid \mathbf x)$: the population probability of default for features $\mathbf x$
$P(y = 0 \mid \mathbf x)$: the population probability of repaying

Take odds: the denominator cancels, leaving the population odds times a constant that does not depend on x.

\frac{P(y = 1 \mid \mathbf x, S = 1)}{P(y = 0 \mid \mathbf x, S = 1)} = \frac{s_1}{s_0}\cdot\frac{P(y = 1 \mid \mathbf x)}{P(y = 0 \mid \mathbf x)} = \frac{s_1}{s_0}\,e^{\beta_0 + \mathbf x^\top\boldsymbol\beta}

$P(y = 0 \mid \mathbf x, S = 1)$: the probability of repaying among sampled applicants

Take logs. The sample model is logistic with the same $\beta$ and a shifted intercept. It is correctly specified on the sample, so the sample MLE's expected score is zero at exactly these values: the fit is consistent for them.

\text{logit}\,P(y = 1 \mid \mathbf x, S = 1) = \beta_0 + \ln\frac{s_1}{s_0} + \mathbf x^\top\boldsymbol\beta

Correct it by subtracting the offset from the fitted intercept, or by fitting with $\ln(s_1/s_0)$ as a fixed offset. For negative downsampling in ads (keep every click, a fraction $r$ of non-clicks: $s_1 = 1$ , $s_0 = r$ ), the same algebra in probabilities is the recalibration of He et al. (2014).

\hat\beta_0^{\text{pop}} = \hat\beta_0^{\text{sample}} - \ln\frac{s_1}{s_0}, \qquad q = \frac{p}{p + (1 - p)/r}

$\hat\beta_0^{\text{pop}}$: the corrected intercept for the full population
$\hat\beta_0^{\text{sample}}$: the intercept fitted on the sample
$q$: the corrected probability; $p$ is the prediction of the model trained on the downsampled data
$r$: the fraction of non-clicks kept in training

Class weights. The weighted MLE solves $\operatorname{E}[w_y\, x(y - \sigma(x^\top v))] = 0$ . Conditioning on $x$ , the bracket vanishes for every $x$ when:

w_1 P(1 - \sigma) - w_0(1 - P)\,\sigma = 0 \;\;\Rightarrow\;\; \frac{\sigma}{1 - \sigma} = \frac{w_1}{w_0}\cdot\frac{P}{1 - P} \;\;\Rightarrow\;\; \text{logit}\,\sigma = \text{logit}\,P + \ln\frac{w_1}{w_0}

$P$: the true default probability at this $\mathbf x$ , $P(y = 1 \mid \mathbf x)$
$\sigma$: the weighted fit's predicted probability, the sigmoid $\sigma(\mathbf x^\top\mathbf v)$

"Balanced" weights are wₖ = n/(2nₖ), so their ratio is the inverse class ratio.

\frac{w_1}{w_0} = \frac{n/(2n_1)}{n/(2n_0)} = \frac{n_0}{n_1} = \frac{1 - \pi}{\pi}

$\pi$: the base rate of default, $P(y = 1)$

Check on the ten files. They came from a batch of 48: all 4 defaults ( $s_1 = 1$ ) and 6 of the 44 repayers ( $s_0 = 6/44$ ).

\ln\frac{s_1}{s_0} = \ln\frac{44}{6} = 1.992, \qquad \hat\beta_0^{\text{pop}} = -0.824 - 1.992 = -2.817, \qquad p_K = \sigma(-2.817) = 0.0564 \;\;(\text{was } 0.305)

$\sigma$: the sigmoid: turns log-odds into a probability
$p_K$: Applicant K's corrected default probability

On the book: keep every default and 20% of repayers, so the offset is $\ln 5 = 1.609$ and the sample intercept should be $-2.6 + 1.609 = -0.991$ . One sample gives −0.985. Over 200 samples the shift averages 1.608 (SD 0.039), and the slopes average (−1.501, 0.604, −0.505) against the truth (−1.5, 0.6, −0.5). Balanced weights on the book are $w_1 = 4.16$ and $w_0 = 0.568$ , a ratio of $7.31 = (1 - 0.1203)/0.1203$ . That predicts a shift of $\ln 7.31 = 1.99$ ; the sim gives 2.01. Mean $\hat p$ becomes 0.389 against a base rate of 0.120, while AUC is unchanged (0.812).

Key move. Bayes on Odds (O).

Common slip. Expecting biased slopes, as when OLS selects on $y$ (Ch 1, Section 7); here the slopes survive because the odds ratio is unchanged by outcome-based sampling. Other slips: subtracting $\ln(s_0/s_1)$ , the wrong sign; forgetting that the correction needs $s_1/s_0$ (if prevalence is unknown, only relative risk is recoverable); assuming it holds exactly under misspecification. On the nonlinear book fitted linearly, the shift came out 1.709 and the DTI slope moved by −0.25.

How it's asked. "You trained on every default and 20% of repayers. What do the coefficients converge to? Prove it, then tell me how you'd serve probabilities." A strong answer is Bayes on odds in three lines, then the offset, then the caveats: misspecification, unknown prevalence, and the same algebra for class weights and for the ads team's negative downsampling. It also mentions King and Zeng's (2001) rare-events correction.

Card 13 · Prior-shift correction of probabilities ★★

Statement. If the base rate moves from $\pi$ to $\pi'$ while the class-conditional distributions $f(x \mid y)$ stay fixed (label shift), multiply every prediction's odds by the change in prior odds. Equivalently, add the log prior-odds change to the intercept.

\text{odds}'(\mathbf x) = \text{odds}(\mathbf x)\cdot\frac{\pi'/(1 - \pi')}{\pi/(1 - \pi)} \quad\Longleftrightarrow\quad \beta_0' = \beta_0 + \ln\frac{\pi'}{1 - \pi'} - \ln\frac{\pi}{1 - \pi}

$\text{odds}'(\mathbf x)$: the odds of default for features $\mathbf x$ under the new base rate
$\text{odds}(\mathbf x)$: the odds under the old base rate
$\pi'$: the new base rate
$\Longleftrightarrow$: is equivalent to
$\beta_0'$: the corrected intercept

Derivation.

Bayes on odds in the old world (toolkit O).

\text{odds}(\mathbf x) = \frac{\pi}{1 - \pi}\cdot\frac{f(\mathbf x \mid y = 1)}{f(\mathbf x \mid y = 0)}

$f(\mathbf x \mid y = 1)$: how common these features are among defaulters
$f(\mathbf x \mid y = 0)$: how common these features are among repayers

In the new world only the prior changes; the likelihood ratio is the same by assumption.

\text{odds}'(\mathbf x) = \frac{\pi'}{1 - \pi'}\cdot\frac{f(\mathbf x \mid y = 1)}{f(\mathbf x \mid y = 0)}

Divide: the likelihood ratio cancels, leaving a constant factor. Take logs for the intercept form.

\frac{\text{odds}'(\mathbf x)}{\text{odds}(\mathbf x)} = \frac{\pi'/(1 - \pi')}{\pi/(1 - \pi)} \;\;\Rightarrow\;\; \ln\text{odds}'(\mathbf x) = \ln\text{odds}(\mathbf x) + \ln\frac{\pi'}{1 - \pi'} - \ln\frac{\pi}{1 - \pi}

Back to a probability.

p' = \frac{\text{odds}'}{1 + \text{odds}'}

$p'$: the corrected default probability

Card 12 is the special case where the "old world" is the case-control sample: its prior odds are the population's times $s_1/s_0$ .

\frac{\pi_{\text{sample}}}{1 - \pi_{\text{sample}}} = \frac{s_1}{s_0}\cdot\frac{\pi}{1 - \pi}

$\pi_{\text{sample}}$: the default rate in the case-control sample

Check, 50/50 training: a model trained on a 50/50 undersample outputs 0.30; the population rate is 10%.

\text{odds}' = \frac{0.3}{0.7}\cdot\frac{0.1/0.9}{0.5/0.5} = \frac{0.4286}{9} = 0.0476 \;\;\Rightarrow\;\; p' = 0.0455

Check, next year's book: the base rate moves from 12.0% to 6.6%.

\Delta\beta_0 = \ln\frac{0.066}{0.934} - \ln\frac{0.12}{0.88} = -0.657, \qquad 0.20 \to \frac{0.25\,e^{-0.657}}{1 + 0.25\,e^{-0.657}} = 0.115

$\Delta\beta_0$: the change to the intercept

The same formula with the calibration split's exact rates gives −0.651, and it cuts the old model's ECE on the new book from 0.053 to 0.006, with AUC unchanged at 0.812. But the simulated shift was not pure label shift: the data-generating process lowered the true intercept by 0.8 with $P(x)$ fixed, which changes $P(y \mid x)$ . Under that truth a 20% prediction should become $\sigma(\ln 0.25 - 0.8) = 10.1\%$ , not 11.5%. The formula is exact only under its assumption; here it still removes most of the miscalibration.

Key move. Bayes on Odds (O).

Common slip. Scaling probabilities linearly ( $0.20 \times 6.6/12 = 0.11$ ) instead of odds; that is only close for rare events. Correcting the probabilities and moving the threshold, which double-counts the shift. Assuming label shift when the relationship itself changed.

How it's asked. "The base rate fell from 12% to 6.6%. Ranking is unchanged but expected loss is overstated. Fix it without refitting." A strong answer gives the intercept correction, says which kind of shift it assumes, and recommends checking on recent labeled loans: refitting just the intercept as an offset gave −0.757 here. When new labels aren't available yet, $\pi'$ can be estimated from the model's own predictions with an EM procedure (Saerens et al., 2002).

Card 14 · Non-collapsibility: omitting an independent variable ★★★

Statement. In logistic regression, leaving out a risk factor Z shrinks the coefficient on x toward zero even when Z is independent of x. Through the probit approximation of the sigmoid:

\text{logit}\,P(y = 1 \mid x, Z) = \beta_0 + \beta x + \gamma Z,\;\; Z \sim N(0, \sigma_Z^2),\;\; Z \perp x \;\;\Rightarrow\;\; \text{logit}\,P(y = 1 \mid x) \approx \frac{\beta_0 + \beta x}{\sqrt{1 + \pi\gamma^2\sigma_Z^2/8}}

$P(y = 1 \mid x, Z)$: default probability given the score $x$ and the risk factor $Z$
$x$: the applicant's scaled credit score
$Z$: another risk factor, independent of $x$ , that the smaller model leaves out
$\beta$: the slope on $x$ with $Z$ in the model
$\gamma$: $Z$ 's coefficient
$\sim$: is distributed as
$\sigma_Z$: the standard deviation of $Z$ (here $\sigma$ is a standard deviation, not the sigmoid)
$P(y = 1 \mid x)$: default probability given the score alone, averaged over $Z$
$\pi$: here the circle constant 3.14159…, not the base rate

Derivation.

The model without Z must fit the marginal probability, which averages the sigmoid over Z. Because Z ⊥ x, its distribution is the same at every x.

P(y = 1 \mid x) = \mathbb E_Z\big[\sigma(a + \gamma Z)\big], \qquad a = \beta_0 + \beta x

$\mathbb E_Z$: the average over the left-out risk factor $Z$
$a$: the part of the log-odds the score explains

Replace $\sigma$ with a scaled normal CDF whose slope matches at 0: $\sigma'(0) = \tfrac14$ and the derivative of $\Phi(\lambda t)$ at 0 is $\lambda/\sqrt{2\pi}$ .

\sigma(t) \approx \Phi(\lambda t), \qquad \frac{\lambda}{\sqrt{2\pi}} = \frac14 \;\Rightarrow\; \lambda = \frac{\sqrt{2\pi}}{4} = \sqrt{\pi/8}

$t$: any log-odds value
$\Phi$: the standard normal CDF: the chance a standard normal falls below its argument
$\lambda$: the scale that matches the probit's slope at 0 to the sigmoid's $\tfrac14$

The Gaussian average of a probit is exact. Write $\Phi(\cdot)$ as $P(U \le \cdot)$ with $U \sim N(0, 1)$ independent of $Z$ . Then $U - \lambda\gamma Z$ is normal with variance $1 + \lambda^2\gamma^2\sigma_Z^2$ .

\mathbb E_Z\big[\Phi(\lambda a + \lambda\gamma Z)\big] = P\big(U - \lambda\gamma Z \le \lambda a\big) = \Phi\Big(\frac{\lambda a}{\sqrt{1 + \lambda^2\gamma^2\sigma_Z^2}}\Big)

$U$: a standard normal draw, independent of $Z$
$P$: the probability of the event in brackets

Undo the approximation, $\Phi(\lambda u) \approx \sigma(u)$ , with $u = a/\sqrt{1 + \lambda^2\gamma^2\sigma_Z^2}$ and $\lambda^2 = \pi/8$ .

P(y = 1 \mid x) \approx \sigma\Big(\frac{\beta_0 + \beta x}{\sqrt{1 + \pi\gamma^2\sigma_Z^2/8}}\Big)

Read off the slope: the marginal log-odds is still linear in x, with every coefficient divided by the same factor, which is greater than 1. Contrast OLS, where Z simply averages out, which is why OLS coefficients are collapsible.

\beta_{\text{omit }Z} \approx \frac{\beta}{\sqrt{1 + \pi\gamma^2\sigma_Z^2/8}} \qquad\text{vs OLS:}\quad \mathbb E[y \mid x] = \beta_0 + \beta x + \gamma\,\mathbb E[Z] = \beta_0 + \beta x

$\beta_{\text{omit }Z}$: the slope on $x$ when $Z$ is left out
$\mathbb E[y \mid x]$: the average outcome at score $x$ (what OLS fits)

It is an approximation, and its error depends on the base rate. The large-n limit of the attenuation ratio, for $\gamma\sigma_Z = 1.5$ , is 0.84 at a 5.5% base rate, 0.774 at 14.6%, 0.725 at 34% and 0.716 at 50%, while $\pi/8$ predicts 0.729 at all of them. So $\pi/8$ is close near 50% and too pessimistic for rare outcomes. Chapter 1's cruder latent-variance match (Ch 1, Drill 26) can do better at low base rates. It comes from the latent-variable picture (Section 2): default when $y^* = a + \gamma Z + \varepsilon > 0$ , with $\varepsilon$ standard logistic, whose variance is $\pi^2/3 \approx 3.29$ . Omitting $Z$ folds $\gamma Z$ into the noise. Treat the combined noise as a rescaled logistic with the same variance. Dividing $y^*$ by the scale factor puts it back on the unit-logistic scale, and every coefficient shrinks by the same factor.

\text{Var}(\varepsilon + \gamma Z) = \frac{\pi^2}{3} + \gamma^2\sigma_Z^2 \;\;\Rightarrow\;\; \beta_{\text{omit }Z} \approx \beta\sqrt{\frac{\pi^2/3}{\pi^2/3 + \gamma^2\sigma_Z^2}}

$\varepsilon$: the standard logistic noise in the latent default score

Check on the NC variant ( $\gamma = 1$ , $\sigma_Z = 1.5$ , $\beta = -1.5$ , base rate 14%).

\frac{1}{\sqrt{1 + \pi(2.25)/8}} = \frac{1}{\sqrt{1.884}} = 0.729 \;\Rightarrow\; -1.093, \qquad \sqrt{\frac{3.29}{3.29 + 2.25}} = 0.771 \;\Rightarrow\; -1.156

The simulation gives −1.153 (large-n limit −1.162, ratio 0.775), so here the latent-variance match is closer. The average marginal effect barely moves (−0.1348 with $Z$ , −0.1344 without): AMEs and risk differences are collapsible. In the randomized-reminder problem (Bank WP2), the true conditional log-OR is −0.4. $\pi/8$ predicts −0.291 unadjusted; the simulation gives −0.314 unadjusted and −0.402 adjusted. The risk difference is −0.0394 either way.

Key move. Taylor and approximate (T): swap $\sigma$ for a probit, use exact Gaussian algebra, swap back.

Common slip. Calling it confounding or omitted-variable bias; no correlation is needed. Writing $\pi^2/8$ instead of $\pi/8$ . Presenting the approximation as exact, or as always better than the latent-variance match. Believing randomization protects the odds ratio: it protects the marginal effect, not the conditional OR.

How it's asked. "In a randomized SMS-reminder test, adding a pre-treatment risk score that is independent of treatment moves the log-OR from −0.31 to −0.40. Which is right?" A strong answer: both, for different estimands (marginal vs conditional). You can't compare ORs across models or studies with different covariates. Report risk differences or AMEs to the PM. Adjusting still pays: $z = -8.8$ vs −7.8, and at $n = 2{,}000$ power rises from 67% to 80% (Robinson and Jewell, 1991; Gail, Wieand and Piantadosi, 1984).

Card 15 · Misclassified labels ★★★

Statement. If a true default is recorded as a repayment with probability $\alpha_1$ , and a true repayment as a default with probability $\alpha_0$ , the observed-label probability is squeezed into $(\alpha_0, 1 - \alpha_1)$ , it is no longer logistic, and a naive fit flattens every slope. The noise-aware likelihood recovers $\beta$ when the $\alpha$ 's are known.

P(\tilde y = 1 \mid \mathbf x) = \alpha_0 + (1 - \alpha_0 - \alpha_1)\,\sigma(\mathbf x^\top\boldsymbol\beta), \qquad \alpha_0 = P(\tilde y = 1 \mid y = 0),\;\; \alpha_1 = P(\tilde y = 0 \mid y = 1)

$P(\tilde y = 1 \mid \mathbf x)$: the probability that the recorded label says default
$\tilde y$: the recorded label, possibly flipped (not the ±1 recoding)
$\alpha_0$: the chance a true repayer is recorded as a default
$\alpha_1$: the chance a true default is recorded as a repayment

Derivation.

Total probability over the true label, with flips that depend only on the true label.

P(\tilde y = 1 \mid \mathbf x) = P(\tilde y = 1 \mid y = 1)\,P(y = 1 \mid \mathbf x) + P(\tilde y = 1 \mid y = 0)\,P(y = 0 \mid \mathbf x) = (1 - \alpha_1)\,\sigma + \alpha_0(1 - \sigma)

Collect the $\sigma$ terms.

q(\mathbf x) \equiv P(\tilde y = 1 \mid \mathbf x) = \alpha_0 + (1 - \alpha_0 - \alpha_1)\,\sigma(\mathbf x^\top\boldsymbol\beta)

$q(\mathbf x)$: the observed-label default probability at $\mathbf x$
$q$: shorthand for $q(\mathbf x)$

Floor and ceiling: as $\sigma$ runs over $(0, 1)$ , $q$ runs over $(\alpha_0, 1 - \alpha_1)$ . No logistic curve has a floor or a ceiling, so the naive model is misspecified.

\sigma \to 0 \Rightarrow q \to \alpha_0, \qquad \sigma \to 1 \Rightarrow q \to 1 - \alpha_1

Local slope on the observed log-odds scale: differentiate $\operatorname{logit} q$ with respect to $\eta$ .

\frac{d\,\text{logit}\,q}{d\eta} = \frac{q'(\eta)}{q(1 - q)} = \frac{(1 - \alpha_0 - \alpha_1)\,\sigma(1 - \sigma)}{q(1 - q)}

$\frac{d\,\text{logit}\,q}{d\eta}$: how fast the observed log-odds moves as you nudge the true log-odds $\eta$
$q'(\eta)$: the slope of $q$ in $\eta$

With symmetric noise $\alpha_0 = \alpha_1 = \alpha$ , at the center ( $\eta = 0$ , $\sigma = q = \tfrac12$ ) the slope is $1 - 2\alpha$ . In the tails, $q$ flattens against its floor and ceiling and the local slope falls toward 0. The best logistic fit averages these local slopes over the data, so it typically lands below $(1 - 2\alpha)\beta$ : attenuation. Step 7 confirms this on a case where the exact limit is computable.

\alpha_0 = \alpha_1 = \alpha,\; \eta = 0: \quad \frac{d\,\text{logit}\,q}{d\eta} = \frac{(1 - 2\alpha)\cdot\frac14}{\frac14} = 1 - 2\alpha

$\alpha$: the common flip rate when $\alpha_0 = \alpha_1$

Fix: if the $\alpha$ 's are known (from an audit of labels) or estimable (a clean validation subsample), maximize the likelihood of the observed labels under step 2's $q$ (Hausman, Abrevaya and Scott-Morton, 1998).

\ell(\boldsymbol\beta) = \sum_i\big[\tilde y_i\ln q_i + (1 - \tilde y_i)\ln(1 - q_i)\big], \qquad q_i = \alpha_0 + (1 - \alpha_0 - \alpha_1)\,\sigma(\mathbf x_i^\top\boldsymbol\beta)

$\tilde y_i$: applicant $i$ 's recorded 0/1 label, possibly flipped
$q_i$: applicant $i$ 's observed-label probability

Check, symmetric case: $\alpha_0 = \alpha_1 = 0.1$ , true slope 1, $x \sim N(0, 1)$ , intercept 0. The naive slope's exact population limit sits below the center slope, as step 5 says (simulation on $10^6$ rows: 0.741).

1 - 2\alpha = 0.8 \;>\; \beta_{\text{naive}} = 0.743, \qquad q \in (0.1,\; 0.9)

$\beta_{\text{naive}}$: the slope a plain logistic fit converges to on the flipped labels

Check on the book's LN variant ( $\alpha_0 = 0.02$ , $\alpha_1 = 0.15$ , true base rate 0.1182). The sim records an observed base rate of 0.120.

\mathbb E[\tilde y] = 0.02 + (1 - 0.02 - 0.15)\times 0.1182 = 0.1181, \qquad q(\mathbf x) < 1 - 0.15 = 0.85

Naive slopes shrink to 0.79 (score), 0.80 (DTI) and 0.86 (log income) of their clean-label values. The noise-aware MLE gives (−1.53, 0.621, −0.489) against the truth (−1.5, 0.6, −0.5).

Key move. Likelihood (L): write the probability of the label you actually observed.

Common slip. Carrying over Chapter 1's "noise in y only blurs". In OLS, mean-zero noise in y leaves slopes unbiased; flipped 0/1 labels are not mean-zero noise, and they bias slopes. Thinking symmetric noise is harmless. Forgetting the ceiling when reading the riskiest applicants' probabilities.

How it's asked. "Fraud labels come from chargebacks: about 15% of fraud is never charged back and about 2% of chargebacks are friendly fraud. What does that do to your model?" A strong answer gives the ceiling (0.85) and the attenuation. It notes that ranking can survive ( $q$ is increasing in $\sigma$ when the $\alpha$ 's are constant) while probabilities can't. The fix is a noise-aware likelihood with $\alpha$ 's from an audit, and if the flip rates depend on $x$ there is no clean fix without validation data.

Card 16 · The cost-optimal threshold ★

Statement. With c_FN the cost of approving a borrower who defaults and c_FP the cost of declining one who would have repaid, decline exactly when the predicted probability exceeds the cost of a false alarm over the total cost.

\text{decline} \iff p\,c_{FN} > (1 - p)\,c_{FP} \iff p > t^* = \frac{c_{FP}}{c_{FP} + c_{FN}}

$c_{FN}$: cost of approving a borrower who defaults ($6,000 lost principal)
$c_{FP}$: cost of declining one who would have repaid ($600 lost profit)
$t^*$: the cost-optimal threshold (star = the optimum)

Derivation.

Expected cost of approving: you lose c_FN only if the applicant defaults.

\mathbb E[\text{cost} \mid \text{approve}] = p\,c_{FN}

Expected cost of declining: you lose c_FP only if they would have repaid.

\mathbb E[\text{cost} \mid \text{decline}] = (1 - p)\,c_{FP}

Decline when the first exceeds the second, and solve for p.

p\,c_{FN} > c_{FP} - p\,c_{FP} \;\;\Rightarrow\;\; p\,(c_{FN} + c_{FP}) > c_{FP} \;\;\Rightarrow\;\; p > \frac{c_{FP}}{c_{FP} + c_{FN}}

Equal costs give t* = ½. That is the only case where the 0.5 default of predict() is optimal, and only if p is calibrated.

c_{FP} = c_{FN} = c \;\;\Rightarrow\;\; t^* = \frac{c}{2c} = \tfrac12

$c$: the common cost when both errors cost the same

Three actions. Add "refer to manual review": it costs $40, catches 80% of defaulters, and lets good applicants through. Its expected cost is $40 + 0.2 \times 6{,}000p$ . Compare it with each neighbor.

\text{refer: } 40 + 1{,}200\,p; \qquad \text{approve} < \text{refer} \iff 6{,}000\,p < 40 + 1{,}200\,p \iff p < \frac{40}{4{,}800} = 0.83\%

\text{refer} < \text{decline} \iff 40 + 1{,}200\,p < 600(1 - p) \iff p < \frac{560}{1{,}800} = 31.1\%

Check with the chapter's costs, c_FP = $600 of lost profit and c_FN = $6,000 of lost principal.

t^* = \frac{600}{600 + 6{,}000} = \frac{1}{11} = 0.0909

Applicant K's naive 0.305 means decline (expected cost $1,829 if approved vs $417 if declined). The case-control-corrected 0.0564 (Card 12) means approve ($339 vs $566). With the refer band, K's 5.6% lies between 0.83% and 31.1%, so K is referred.

Key move. Bayes on Odds (O), in decision form: compare expected costs, and the cut sits where the odds equal the cost ratio, p/(1 − p) = c_FP/c_FN.

Common slip. Inverting the ratio, c_FN/(c_FP + c_FN) = 0.909. Thresholding uncalibrated probabilities: the same rule declines K on 0.305 and approves K on 0.056. Mixing up which error is the "false positive"; here positive means "predicted to default", so declined.

How it's asked. "Losses on a default are ten times the profit on a good loan. Where do you cut?" A strong answer: t* = 1/11, and the threshold is a business input, so moving it never needs a refit. It only works with calibrated probabilities (Card 6, Card 13). Under prior shift, adjust either the probabilities or the cutoff, never both. Ch 5 owns the metrics built from the confusion matrix.

Card 17 · Softmax, its gradient, and GLMs with canonical links ★★

Statement. Softmax turns $K$ scores into probabilities. The gradient of its cross-entropy with respect to the scores is $p - e_y$ , so the weight gradient is again $X^\top(P - Y)$ . It is shift-invariant, and for $K = 2$ it is the sigmoid. Every GLM with a canonical link has the same score, $X^\top(y - \mu)$ .

p_k = \frac{e^{\eta_k}}{\sum_{j=1}^{K}e^{\eta_j}}, \qquad \frac{\partial(-\ln p_y)}{\partial\eta_k} = p_k - \mathbb 1[k = y], \qquad \nabla_{\mathbf w}\,\ell_{\text{GLM}} = \frac1\phi\,X^\top(\mathbf y - \boldsymbol\mu)

$p_k$: the predicted probability of outcome class $k$
$\eta_k$: the score (logit) for class $k$
$\sum_{j=1}^{K}$: add up over all $K$ classes
$K$: the number of outcome classes
$\eta_j$: the score for class $j$
$\frac{\partial(-\ln p_y)}{\partial\eta_k}$: how fast the row's loss changes as you nudge class $k$ 's score, holding the others fixed
$p_y$: the probability given to the true class $y$
$\mathbb 1[k = y]$: 1 if $k$ is the true class, else 0
$\ell_{\text{GLM}}$: a GLM's log-likelihood
$\phi$: the dispersion: 1 for Bernoulli and Poisson
$\frac1\phi$: one over the dispersion
$\boldsymbol\mu$: the vector of model means (for logistic regression, the probabilities)

Derivation.

Write the row loss as a log-sum-exp minus the true class's score.

-\ln p_y = -\eta_y + \ln\sum_{j=1}^{K}e^{\eta_j}

$\eta_y$: the true class's score

Differentiate with respect to each score. The log-sum-exp's derivative is the softmax itself.

\frac{\partial(-\ln p_y)}{\partial\eta_k} = -\mathbb 1[k = y] + \frac{e^{\eta_k}}{\sum_j e^{\eta_j}} = p_k - \mathbb 1[k = y]

$\sum_j$: add up over every class $j$

Chain rule through $\eta_k = x^\top w_k$ and stack the rows: $P$ is the $n \times K$ matrix of probabilities, $Y$ the one-hot labels. It is the same shape as Card 3.

\nabla_{\mathbf w_k}(-\ell) = \sum_i (p_{ik} - y_{ik})\,\mathbf x_i \;\;\Rightarrow\;\; \nabla_{W}(-\ell) = X^\top(P - Y)

$\nabla_{\mathbf w_k}$: the gradient with respect to class $k$ 's weight vector
$p_{ik}$: applicant $i$ 's probability for class $k$
$y_{ik}$: 1 if applicant $i$ 's outcome is class $k$ , else 0
$\nabla_{W}$: the gradient with respect to all classes' weights at once (one column per class)
$P$: the $n \times K$ matrix of predicted probabilities
$Y$: the $n \times K$ matrix of one-hot labels

Convexity: $\partial p_k/\partial\eta_j = p_k(1[k = j] - p_j)$ , so the Hessian in $\eta$ is $\operatorname{diag}(p) - pp^\top$ , the covariance matrix of a one-hot draw. It is PSD.

\mathbf v^\top\big(\text{diag}(\mathbf p) - \mathbf p\mathbf p^\top\big)\mathbf v = \sum_k p_k v_k^2 - \Big(\sum_k p_k v_k\Big)^2 = \text{Var}_{k\sim\mathbf p}(v_k) \ge 0

$\mathbf p$: one row's vector of class probabilities
$\sum_k$: add up over classes $k$
$v_k$: entry $k$ of the test direction $\mathbf v$
$\text{Var}_{k\sim\mathbf p}(v_k)$: the variance of $v_k$ when class $k$ is drawn with probabilities $\mathbf p$

Shift invariance: adding c to every score changes nothing, so one class's weights are redundant. Fix a reference class (or penalize, which picks the sum-to-zero solution).

\frac{e^{\eta_k + c}}{\sum_j e^{\eta_j + c}} = \frac{e^{c}e^{\eta_k}}{e^{c}\sum_j e^{\eta_j}} = p_k

$c$: any constant added to every score

$K = 2$ : divide top and bottom by $e^{\eta_1}$ .

p_1 = \frac{e^{\eta_1}}{e^{\eta_1} + e^{\eta_0}} = \frac{1}{1 + e^{-(\eta_1 - \eta_0)}} = \sigma(\eta_1 - \eta_0)

$p_1$: the probability of class 1 (default) when there are two classes
$\eta_1$: class 1's score
$\eta_0$: class 0's score

GLMs. Write the distribution in exponential-family form, with mean $\mu = b'(\theta)$ . The Bernoulli fits it with $\theta = \operatorname{logit} p$ and $b(\theta) = \ln(1 + e^\theta)$ .

f(y;\theta) = \exp\Big\{\frac{y\theta - b(\theta)}{\phi} + c(y, \phi)\Big\}, \qquad p^{y}(1 - p)^{1 - y} = \exp\big\{y\,\text{logit}\,p - \ln(1 + e^{\text{logit}\,p})\big\}

$f(y;\theta)$: the probability of outcome $y$ , written in exponential-family form
$\theta$: the canonical parameter: the log-odds for the Bernoulli, $\ln \mu$ for Poisson
$b(\theta)$: the cumulant function; its slope $b'(\theta)$ is the mean
$c(y, \phi)$: a normalizing term that does not involve $\theta$

A canonical link sets $\theta_i = \eta_i = x_i^\top w$ . Differentiating, the score is $X^\top(y - \mu)/\phi$ , for Gaussian ( $\theta = \mu$ , $b = \theta^2/2$ : OLS), Bernoulli ( $b' = \sigma$ : Card 3) and Poisson ( $\theta = \ln \mu$ , $b = e^\theta$ ) alike. With an intercept, $\sum\hat{\mu} = \sum y$ , as in Card 6.

\frac{\partial\ell_i}{\partial\mathbf w} = \frac{y_i - b'(\theta_i)}{\phi}\,\mathbf x_i = \frac{y_i - \mu_i}{\phi}\,\mathbf x_i

$\frac{\partial\ell_i}{\partial\mathbf w}$: how row $i$ 's log-likelihood changes as you nudge each weight
$b'(\theta_i)$: row $i$ 's mean, from the cumulant's slope
$\mu_i$: row $i$ 's model mean: $p_i$ for Bernoulli, the expected count for Poisson

Exposure offset for counts: a borrower on the books for $t_i$ months has expected count $t_i \times \text{rate}$ . The log link turns that into $\ln t_i$ with its coefficient fixed at 1.

\mu_i = t_i\,e^{\mathbf x_i^\top\boldsymbol\beta} \;\;\Rightarrow\;\; \ln\mu_i = \ln t_i + \mathbf x_i^\top\boldsymbol\beta

$t_i$: borrower $i$ 's exposure: months on the books

Check. Scores (2, 1, 0) with the second class true; adding 5 to every score leaves p unchanged; two classes with scores (1.3, 0.2) reduce to a sigmoid.

\mathbf p = \frac{(e^2,\, e^1,\, e^0)}{e^2 + e + 1} = (0.665,\; 0.245,\; 0.090), \qquad -\ln 0.245 = 1.41, \qquad \mathbf p - \mathbf e_2 = (0.665,\; -0.755,\; 0.090), \qquad \sigma(1.3 - 0.2) = 0.750

$\mathbf e_2$: the one-hot vector for class 2, the true class

On the Poisson variant, two borrowers with the same features, on the books 6 and 24 months, have expected counts 0.299 and 1.19: four times the exposure, four times the count. The fitted Poisson model also satisfies $\sum\hat{\mu} = \sum y$ to $10^{-11}$ .

Key move. Gradient (G) through the log-sum-exp; Likelihood (L) in exponential-family form for the GLM step.

Common slip. Getting the true class's sign wrong (it is $p_y - 1$ , negative). Computing softmax without subtracting the max score first (overflow). Believing one-vs-rest probabilities sum to 1; on the three-outcome book their sums range from 0.92 to 1.73. Fitting counts without the offset when exposure varies.

How it's asked. "Derive the gradient of softmax cross-entropy with respect to the logits." A strong answer gives $p - e_y$ in two lines, then shift invariance and the log-sum-exp trick, and notes it is the gradient at the output layer of every classifier network and LLM. Research loops follow up with "now generalize": the canonical-link score $X^\top(y - \mu)$ .

Card 18 · LDA ⇒ logistic posterior ★★★

Statement. If each class is Gaussian with its own mean and a shared covariance $\Sigma$ , the posterior log-odds is exactly linear in $x$ : a logistic regression. With unequal covariances it is quadratic (QDA).

\mathbf x \mid y = k \sim N(\boldsymbol\mu_k, \Sigma) \;\;\Rightarrow\;\; \text{logit}\,P(y = 1 \mid \mathbf x) = \underbrace{\ln\frac{\pi_1}{\pi_0} - \tfrac12\big(\boldsymbol\mu_1^\top\Sigma^{-1}\boldsymbol\mu_1 - \boldsymbol\mu_0^\top\Sigma^{-1}\boldsymbol\mu_0\big)}_{\beta_0} + \mathbf x^\top\underbrace{\Sigma^{-1}(\boldsymbol\mu_1 - \boldsymbol\mu_0)}_{\boldsymbol\beta}

$k$: the class: 1 = default, 0 = repaid
$\boldsymbol\mu_k$: the average feature vector of class $k$
$\Sigma$: the covariance matrix of the features, shared by both classes
$\Sigma^{-1}$: its inverse (matrix inverse: undoes it)
$\pi_1$: the base rate of default
$\pi_0$: the share of applicants who repay
$\boldsymbol\mu_1$: the average feature vector of defaulters
$\boldsymbol\mu_0$: the average feature vector of repayers

Derivation.

Bayes on odds (toolkit O).

\text{logit}\,P(y = 1 \mid \mathbf x) = \ln\frac{\pi_1}{\pi_0} + \ln f_1(\mathbf x) - \ln f_0(\mathbf x)

$f_1$: the Gaussian density of defaulters' features, $f(\mathbf x \mid y = 1)$
$f_0$: the Gaussian density of repayers' features

The Gaussian log-density. With a shared $\Sigma$ , the normalizing constant is the same for both classes and cancels in the difference.

\ln f_k(\mathbf x) = -\tfrac12(\mathbf x - \boldsymbol\mu_k)^\top\Sigma^{-1}(\mathbf x - \boldsymbol\mu_k) + \text{const}

$f_k$: class $k$ 's Gaussian density

Expand the quadratic form. $\Sigma^{-1}$ is symmetric, so the cross terms combine.

-\tfrac12(\mathbf x - \boldsymbol\mu_k)^\top\Sigma^{-1}(\mathbf x - \boldsymbol\mu_k) = -\tfrac12\mathbf x^\top\Sigma^{-1}\mathbf x + \mathbf x^\top\Sigma^{-1}\boldsymbol\mu_k - \tfrac12\boldsymbol\mu_k^\top\Sigma^{-1}\boldsymbol\mu_k

Subtract class 0 from class 1. The $-\tfrac12 x^\top\Sigma^{-1}x$ term is identical for both and cancels, leaving a linear function of $x$ : exactly the statement.

\ln f_1 - \ln f_0 = \mathbf x^\top\Sigma^{-1}(\boldsymbol\mu_1 - \boldsymbol\mu_0) - \tfrac12\big(\boldsymbol\mu_1^\top\Sigma^{-1}\boldsymbol\mu_1 - \boldsymbol\mu_0^\top\Sigma^{-1}\boldsymbol\mu_0\big)

Unequal covariances: the quadratic terms and the normalizing constants no longer cancel. In one dimension the log-odds gains an $x^2$ term.

\text{logit}\,P = \ln\frac{\pi_1}{\pi_0} - \ln\frac{\sigma_1}{\sigma_0} + \tfrac12\Big(\frac{1}{\sigma_0^2} - \frac{1}{\sigma_1^2}\Big)x^2 + \Big(\frac{\mu_1}{\sigma_1^2} - \frac{\mu_0}{\sigma_0^2}\Big)x - \tfrac12\Big(\frac{\mu_1^2}{\sigma_1^2} - \frac{\mu_0^2}{\sigma_0^2}\Big)

$P$: the probability of default given $x$
$x$: the single feature in this one-dimensional example
$\sigma$: here a class's standard deviation of $x$ , not the sigmoid
$\sigma_1$: the standard deviation of $x$ among defaulters
$\sigma_0$: the standard deviation of $x$ among repayers
$\mu_1$: the mean of $x$ among defaulters
$\mu_0$: the mean of $x$ among repayers

Check, shared variance: $\mu_0 = 0$ , $\mu_1 = 1$ , $\sigma = 1$ , $\pi_1 = 0.2$ .

\beta_1 = 1, \qquad \beta_0 = \ln\frac{0.2}{0.8} - \tfrac12(1 - 0) = -1.386 - 0.5 = -1.886

$\beta_1$: the slope on $x$

On 50,000 draws, logistic regression gives (−1.886, 1.004) and LDA (−1.885, 1.001).

Check, unequal variances ( $\sigma_1 = 2$ ): $x^2$ coefficient $\tfrac12(1 - \tfrac14) = 0.375$ , $x$ coefficient $1/4 = 0.25$ , intercept $\ln 0.25 - \ln 2 - 1/8 = -2.204$ . Logistic regression with an $x^2$ feature recovers (−2.21, 0.240, 0.382).

The converse fails. Linear log-odds does not imply Gaussian classes: any exponential family with a shared dispersion gives it. That is the trade. LDA uses the joint likelihood and is more efficient when the Gaussian story holds (Efron, 1975, studied the gap). Logistic regression assumes less and is safer when it doesn't hold. Over 1,000 samples of $n = 200$ from step 6's classes:

\frac{\text{Var}(\hat\beta_{\text{logistic}})}{\text{Var}(\hat\beta_{\text{LDA}})} = \frac{0.2167^2}{0.2051^2} = 1.12

$\hat\beta_{\text{logistic}}$: the slope fitted by logistic regression
$\hat\beta_{\text{LDA}}$: the slope fitted by LDA

On the unequal-variance data both linear fits are misspecified and they no longer agree: slope 0.553 for logistic regression, 0.603 for LDA. Adding $x^2$ to the logistic fit raises the log-likelihood by an LR statistic of 5,978.

Key move. Bayes on Odds (O).

Common slip. Dropping the prior term $\ln(\pi_1/\pi_0)$ from the intercept. Claiming the quadratic term cancels without a shared $\Sigma$ . Claiming the converse ("logistic works, so the classes must be Gaussian").

How it's asked. "Show that LDA gives a linear decision boundary, and relate it to logistic regression." A strong answer: same functional form, different estimators. LDA models $P(x \mid y)$ and the base rate, then applies Bayes (generative); logistic regression models $P(y \mid x)$ directly (discriminative). Hook: generative draws the classes, discriminative draws the line. Add that Naive Bayes reaches its (higher) asymptotic error faster with little data (Ng and Jordan, 2002; Ch 8).

Practice protocol

Blank page, statement only. Cover everything below the statement. Write the derivation from memory, numbering the steps.
Timer. ★ cards: 3 minutes. ★★: 6 minutes. ★★★: 10 minutes. That is roughly what an interviewer allows before moving on.
Check every step, then the number. Compare line by line. Then redo the card's check on the running example by hand: Card 5's first step (−0.471, −1.774), Card 10's three p-values, Card 12's 0.305 → 0.0564. A derivation that can't reproduce its number isn't finished.
Name the key move and the slip out loud. Say which L-O-G-I-T letter did the work and the slip you'd watch for. Then give the "How it's asked" strong answer in under a minute, as if to an interviewer.
Mark and re-derive. Mark any card where you looked, stalled or slipped. Re-derive marked cards after 48 hours, then after a week. A card is mastered when you get it right, inside the time, twice in a row on separate days.
Pair with the bank. After each card, answer its linked questions from the Interview Bank:

Card	Interview Bank
1 · Sigmoid, odds, log-odds	Q2, Q3, Drill 11
2 · Likelihood → cross-entropy	Q13, Drill 1
3 · Gradient	Q14, Q25, Drill 2
4 · Hessian, convexity	Q15
5 · Newton = IRLS	Q7, Q16
6 · Score equations, calibration	Q11, Drill 15
7 · Why not MSE	Q6
8 · Odds ratios, marginal effects	Q3, Q4, Q12
9 · Fisher information	Drills 1, 3, 4, 18
10 · Wald, LR, score	Q18, Q19, Drill 19
11 · Separation, Firth	Q17, Drills 5, 6
12 · Case-control, weights	Q10, Q22, Q34, WP1, Drill 8
13 · Prior shift	Q36, Drills 7, 17
14 · Non-collapsibility	Q30, WP2, Drill 10
15 · Misclassified labels	Q35, Drill 9
16 · Threshold	Q8, Drill 16
17 · Softmax, GLMs	Q9, Q27, Q38, Drills 14, 20
18 · LDA ⇒ logistic	Q37, Drills 12, 13

Suggested order, matching the chapter: Cards 1–6 first (the model, loss, gradient, Hessian, IRLS, calibration), then Levels 1–2 of the bank, then Cards 7–18, then Levels 3–4.

Mastery checklist

[ ] Card 1. Inverted the logit into the sigmoid, derived $\sigma' = \sigma(1 - \sigma)$ and $1 - \sigma(z) = \sigma(-z)$ , and traced K across 0.305 → 0.439 → −0.824.
[ ] Card 2. Derived the log loss from a coin flip, rewrote it as $\sum[\ln(1 + e^\eta) - y\eta]$ and as $\ln(1 + e^{-\tilde y\eta})$ , and explained "cross-entropy" through KL.
[ ] Card 3. Derived $X^\top(p - y)$ both ways, said why there is no closed form, and computed [1, 2.1] at $w = 0$ on the ten files.
[ ] Card 4. Derived $X^\top WX$ , proved it PSD, explained strict convexity vs existence, and got the learning-rate bound $\eta < 8$ on standardized $x$ .
[ ] Card 5. Turned the Newton step into weighted least squares on z, explained z and W, and computed the first step (−0.471, −1.774) by hand.
[ ] Card 6. Derived $\sum\hat p = \sum y$ from the intercept's score equation, showed L2 on slopes keeps it and a penalized intercept breaks it.
[ ] Card 7. Showed the MSE gradient's extra $\sigma(1 - \sigma)$ factor (0.0196 vs 0.99) and its concavity for $\sigma > 2/3$ .
[ ] Card 8. Derived the odds ratio, the marginal effect, the $\beta/4$ bound and the AME, and read −2.77 on all three scales.
[ ] Card 9. Derived $I(p) = 1/(p(1 - p))$ and $I(\theta) = p(1 - p)$ , got $I(w) = X^\top WX$ , showed $\operatorname{Var}(U) = I$ and $\operatorname{Var}(\hat w) \approx I^{-1}$ by the sandwich, and computed the intercept-only SE 0.645.
[ ] Card 10. Showed $W \approx \text{LR} \approx S$ by Taylor, defined deviance, and reproduced $p = 0.106$ , 0.0295 and 0.0467 on the ten files.
[ ] Card 11. Proved $\ell$ rises forever along a separating ray, explained the constant $+1/m$ Newton increment, derived Firth's modified score, and showed the Jeffreys penalty cancels the 1-D bias $-K'''/(2nK''^2)$ (ten files: −0.405 → −0.368).
[ ] Card 12. Derived the $\ln(s_1/s_0)$ intercept shift by Bayes on odds and the $\ln(w_1/w_0)$ shift for class weights, and corrected K from 0.305 to 0.0564.
[ ] Card 13. Derived the prior-shift correction, stated its label-shift assumption, and computed 0.30 → 0.0455 and 20% → 11.5%.
[ ] Card 14. Derived $\beta/\sqrt{1 + \pi\gamma^2\sigma^2/8}$ through the probit approximation and the latent-variance match, and said which is closer where ( $\pi/8$ near 50%; latent-variance at the NC variant's 14%).
[ ] Card 15. Derived $\alpha_0 + (1 - \alpha_0 - \alpha_1)\sigma$ , the floor and ceiling, the $1 - 2\alpha$ local slope, and wrote the noise-aware likelihood.
[ ] Card 16. Derived t* = c_FP/(c_FP + c_FN) = 0.0909, the refer band (0.83%, 31.1%), and K's three decisions.
[ ] Card 17. Derived $p - e_y$ , shift invariance, $K = 2$ → sigmoid, and the canonical-link score $X^\top(y - \mu)$ with an exposure offset.
[ ] Card 18. Showed the shared-covariance quadratic cancels, wrote $\beta$ and $\beta_0$ for LDA, and the $x^2$ term (0.375) when variances differ.