Chapter 1 — Derivations

Seventeen derivations cover every regression proof an interviewer can reasonably ask for; Cards 14–17 are for research and quant roles. Each card has the same five parts: the statement, the derivation (4–8 lines), the key move that unlocks it, the common slip, and how it's asked. Learn the four tools first — every card is those tools in a different order.

The toolkit: four moves

Every derivation below is some combination of these four. Memory hook: D-V-T-A — Differentiate, Variance, Trace, Add-and-subtract.

1. Differentiate (matrix calculus). Two rules cover all of regression:

\frac{\partial}{\partial \mathbf w}\,(\mathbf a^\top \mathbf w) = \mathbf a \qquad \frac{\partial}{\partial \mathbf w}\,(\mathbf w^\top A \mathbf w) = (A + A^\top)\mathbf w = 2A\mathbf w \;\text{ if } A \text{ is symmetric}

$\frac{\partial}{\partial \mathbf w}$: how fast the expression after it changes as you nudge each weight, holding the rest fixed: a slope, not a fraction to divide
$\mathbf w$: the weight vector, one weight per feature (bold = a vector)
$\mathbf a$: a fixed vector that doesn't depend on $\mathbf w$
$^\top$: transpose: rows become columns, so $\mathbf a^\top \mathbf w$ is a single number
$A$: a fixed square matrix that doesn't depend on $\mathbf w$

Intuition: these are the matrix versions of $d(ax)/dx = a$ and $d(ax^2)/dx = 2ax$ .

2. Variance of a linear transform. For a fixed matrix $A$ and random vector $\varepsilon$ :

\mathbb E[A\boldsymbol\varepsilon] = A\,\mathbb E[\boldsymbol\varepsilon] \qquad \text{Var}(A\boldsymbol\varepsilon) = A\,\text{Var}(\boldsymbol\varepsilon)\,A^\top

$\mathbb E$: expected value: the average over many repeated draws of the noise
$\boldsymbol\varepsilon$: the noise vector: every house's error (luck, haggling) stacked in one column
$\text{Var}$: covariance matrix: each entry's variance on the diagonal, pairwise covariances off it

Intuition: the scalar rule $\text{Var}(a\varepsilon) = a^2\,\text{Var}(\varepsilon)$ , with $a^2$ written as $A \cdot A^\top$ . Its covariance cousin, used in every Level 4 problem: Cov is bilinear — $\text{Cov}(X + Z, X + u)$ splits into four pairwise terms, and independence zeroes the cross terms.

3. The trace trick. For a quadratic form in a random vector with $\text{Var}(\varepsilon) = \sigma^2 I$ :

\mathbb E[\boldsymbol\varepsilon^\top M \boldsymbol\varepsilon] = \sigma^2\,\text{tr}(M) \qquad \text{tr}(AB) = \text{tr}(BA)

$M$: any fixed $n \times n$ matrix (later, the residual maker $I - H$ )
$\sigma^2$: the noise variance: how far each house's price scatters around the true line
$\text{tr}$: trace: the sum of a matrix's diagonal entries
$B$: another fixed matrix, shaped so both $AB$ and $BA$ exist

Intuition: a scalar equals its own trace, and the trace lets you rotate matrices so the expectation lands on $\varepsilon\varepsilon^\top$ .

4. Add and subtract. To split an error into parts, insert a term and remove it again: $(a - c) = (a - b) + (b - c)$ . Expand the square and show the cross term has zero expectation. This one move gives every bias–variance decomposition.

Two matrix facts you'll also need: the hat matrix $H = X(X^\top X)^{-1}X^\top$ and $M = I - H$ are both symmetric and idempotent ( $HH = H$ — projecting twice changes nothing), and $\operatorname{tr}(H) = p$ .

Formulas to memorize

Memorize these 22 cold. Everything else in the chapter can be derived from them in a minute or two. Each label says which card proves it, so a formula you forget can always be rebuilt.

Fitting

1. Simple slope and intercept (Card 1)

w = \frac{S_{xy}}{S_{xx}} = r\,\frac{s_y}{s_x}, \qquad b = \bar y - w\,\bar x

$w$: the fitted slope: $k of price per extra sq ft
$S_{xy}$: $\sum_i (x_i - \bar x)(y_i - \bar y)$ : how size and price move together
$S_{xx}$: $\sum_i (x_i - \bar x)^2$ : how spread out the sizes are
$r$: the correlation between size and price, from −1 to 1
$s_y$: the standard deviation of the prices
$s_x$: the standard deviation of the sizes
$b$: the fitted intercept: the line's price for a 0 sq ft house
$\bar y$: the average price over all houses (bar = mean)
$\bar x$: the average size

2. Normal equation (Card 2)

X^\top X\,\hat{\mathbf w} = X^\top \mathbf y \;\Rightarrow\; \hat{\mathbf w} = (X^\top X)^{-1}X^\top\mathbf y

$X$: the design matrix: one row per house, one column per feature
$^\top$: transpose: rows become columns
$X^\top X$: the $p \times p$ table of feature sums of squares and cross-products
$\hat{\mathbf w}$: the fitted weight vector (hat = estimated from data, bold = a vector)
$\mathbf y$: the column of all $n$ actual prices
$(X^\top X)^{-1}$: the inverse of $X^\top X$ : the matrix that undoes it

3. Hat matrix, leverage, orthogonality (Cards 2, 7)

H = X(X^\top X)^{-1}X^\top, \qquad h_{ii} = [H]_{ii}, \qquad \textstyle\sum_i h_{ii} = p, \qquad X^\top \mathbf e = \mathbf 0

$H$: the hat matrix: turns actual prices into fitted ones, $\hat{\mathbf y} = H\mathbf y$
$h_{ii}$: house $i$ 's leverage: how strongly its own price pulls its own fit
$[H]_{ii}$: the $i$ -th diagonal entry of $H$ (row $i$ , column $i$ )
$\textstyle\sum_i$: add up over every house $i$
$p$: the number of fitted weights (features, counting the intercept)
$\mathbf e$: the residual vector: each house's actual minus fitted price

4. Variance decomposition and $R^2$ (Card 4)

\text{TSS} = \text{ESS} + \text{RSS}, \qquad R^2 = 1 - \frac{\text{RSS}}{\text{TSS}}, \qquad R^2_{\text{adj}} = 1 - (1 - R^2)\frac{n-1}{n-p}

$\text{TSS}$: total sum of squares: how much prices vary around their mean
$\text{ESS}$: explained sum of squares: the part of that variation the line captures
$\text{RSS}$: residual sum of squares: the part left over, $\sum_i e_i^2$
$R^2$: the share of price variation the line explains, from 0 to 1
$R^2_{\text{adj}}$: adjusted $R^2$ : docked for each extra feature, so useless features can't inflate it
$n$: the number of houses (rows)

Inference

5. Coefficient covariance (Card 6)

\text{Var}(\hat{\mathbf w}\mid X) = \sigma^2 (X^\top X)^{-1}, \qquad \text{one feature: } \text{Var}(\hat w) = \frac{\sigma^2}{S_{xx}}

$\text{Var}$: variance (a covariance matrix for a vector): how much it wobbles across repeated samples
$\mid X$: given $X$ : the houses' features held fixed, only the noise redrawn
$\sigma^2$: the noise variance: how far prices scatter around the true line
$\hat w$: the fitted slope when size is the only feature

6. Robust (sandwich) covariance (Card 6)

\text{Var}(\hat{\mathbf w}\mid X) = (X^\top X)^{-1}X^\top \Omega X(X^\top X)^{-1}

$\Omega$: the noise covariance matrix: each house's own error variance on the diagonal, possibly unequal

7. Unbiased noise variance (Card 7)

s^2 = \frac{\text{RSS}}{n - p}, \qquad \mathbb E[\text{RSS}] = (n - p)\,\sigma^2

$s^2$: the estimate of the noise variance $\sigma^2$ , built from the residuals
$\mathbb E$: expected value: the average over many repeated samples

8. t statistic and confidence interval (Card 6)

t = \frac{\hat w_j}{\text{SE}(\hat w_j)}, \qquad \hat w_j \pm t_{0.975,\,n-p}\cdot\text{SE}(\hat w_j)

$t$: the t statistic: how many standard errors the weight sits from 0
$\hat w_j$: the fitted weight on feature $j$
$\text{SE}(\hat w_j)$: its standard error: the typical wobble of $\hat w_j$ across samples
$t_{0.975,\,n-p}$: the 97.5th percentile of a t distribution with $n - p$ degrees of freedom (≈ 2 for large $n$ )

9. Confidence vs prediction interval (Cards 6, 12)

\text{CI: } \hat y_0 \pm t\,s\sqrt{\mathbf x_0^\top(X^\top X)^{-1}\mathbf x_0} \qquad \text{PI: } \hat y_0 \pm t\,s\sqrt{1 + \mathbf x_0^\top(X^\top X)^{-1}\mathbf x_0}

$\hat y_0$: the predicted price for a new house
$t$: here the critical value $t_{0.975,\,n-p}$ from formula 8
$s$: $\sqrt{s^2}$ : the estimated noise standard deviation, in $k
$\mathbf x_0$: the new house's feature vector

10. Moulton factor (Card 16)

\frac{\text{Var}_{\text{true}}(\hat w)}{\text{Var}_{\text{naive}}(\hat w)} \approx 1 + (m - 1)\,\rho_x\,\rho_\varepsilon

$\text{Var}_{\text{true}}(\hat w)$: the slope's actual variance when errors within a group are correlated
$\text{Var}_{\text{naive}}(\hat w)$: the variance the usual formula reports, assuming independent errors
$m$: the number of houses in each group (e.g. each neighbourhood)
$\rho_x$: how correlated the feature is within a group (1 if shared, like a school rating)
$\rho_\varepsilon$: how correlated the errors are within a group

Bias

11. Omitted-variable bias (Card 9)

\mathbb E[\hat\beta_1] = \beta_1 + \gamma\,\delta, \qquad \delta = \frac{\text{Cov}(x, z)}{\text{Var}(x)}

$\hat\beta_1$: the fitted slope on size when the omitted feature is left out
$\beta_1$: the true effect of size on price
$\gamma$: the omitted feature's true effect on price
$\delta$: the slope from regressing the omitted feature on size: how they move together
$\text{Cov}$: covariance: how two quantities move together
$x$: the included feature (size)
$z$: the omitted feature (e.g. neighbourhood quality)

12. Attenuation from measurement error (Card 10)

\text{plim}\,\hat\beta = \beta\cdot\frac{\sigma^2_{x^*}}{\sigma^2_{x^*} + \sigma^2_u}

$\text{plim}$: probability limit: where the estimate settles with endless data
$\hat\beta$: the slope fitted on the noisily measured size
$\beta$: the true slope of price on size
$\sigma^2_{x^*}$: the variance of the true sizes $x^*$ (the signal)
$\sigma^2_u$: the variance of the measurement noise $u$ in recorded size

13. Fixed-effects (within) estimator (Card 14)

y_{it} - \bar y_i = \beta\,(x_{it} - \bar x_i) + (\varepsilon_{it} - \bar\varepsilon_i)

$y_{it}$: house $i$ 's price at its sale in year $t$ (each house sells several times)
$\bar y_i$: house $i$ 's average price over its sales
$x_{it}$: house $i$ 's feature at that sale (e.g. size after renovations)
$\bar x_i$: house $i$ 's average of that feature
$\varepsilon_{it}$: the noise on that sale
$\bar\varepsilon_i$: house $i$ 's average noise over its sales

14. Probit with an omitted independent variable (Card 17)

\hat\beta \to \frac{\beta}{\sqrt{1 + \gamma^2\sigma_z^2}} \qquad (\text{logit: replace } 1 \text{ by } \pi^2/3, \text{ approximately})

$\hat\beta$: the probit slope fitted with the independent variable $z$ left out
$\beta$: the true probit slope on $x$
$\gamma$: the omitted variable $z$ 's true effect on the hidden score
$\sigma_z^2$: the variance of the omitted variable $z$
$\pi^2/3$: the variance of the logistic error, which logit fixes in place of probit's 1

Regularization and prediction

15. Ridge (Card 11)

\hat{\mathbf w}_{\text{ridge}} = (X^\top X + \lambda I)^{-1}X^\top\mathbf y, \qquad X^\top X = I: \; \hat{\mathbf w}_{\text{ridge}} = \frac{\hat{\mathbf w}_{\text{OLS}}}{1 + \lambda}

$\hat{\mathbf w}_{\text{ridge}}$: the ridge weights: least squares plus a penalty on large weights
$\lambda$: the penalty strength you choose; bigger shrinks the weights more
$I$: the identity matrix: 1s on the diagonal, 0s elsewhere
$^{-1}$: matrix inverse: undoes the matrix
$\hat{\mathbf w}_{\text{OLS}}$: the plain least-squares weights, no penalty

16. Lasso soft threshold (Card 11)

X^\top X = I: \quad \hat w_j^{\text{lasso}} = \text{sign}(\hat w_j)\,\max\Big(|\hat w_j| - \frac{\lambda}{2},\, 0\Big)

$\hat w_j^{\text{lasso}}$: the lasso weight on feature $j$
$\text{sign}$: +1 if positive, −1 if negative: keeps the weight's direction
$|\hat w_j|$: the size of the OLS weight, ignoring its sign

17. Expected prediction error (Card 12)

\mathbb E\big[(y_0 - \hat f(\mathbf x_0))^2\big] = \sigma^2 + \text{Bias}^2 + \text{Var}

$y_0$: the actual price of a new house
$\hat f(\mathbf x_0)$: the model's prediction for that house (hat = learned from training data)
$\text{Bias}^2$: squared gap between the average prediction and the true price function

18. Training vs test error (Card 15)

\mathbb E[\text{train MSE}] = \sigma^2\Big(1 - \frac{p}{n}\Big), \qquad \mathbb E[\text{test MSE}] = \sigma^2\Big(1 + \frac{p}{n}\Big)

$\text{train MSE}$: mean squared error on the houses the model was fit to
$\text{test MSE}$: mean squared error on fresh prices at those same houses

Diagnostics

19. Leave-one-out residual (Card 13)

e_{(-i)} = \frac{e_i}{1 - h_{ii}}

$e_{(-i)}$: house $i$ 's leave-one-out residual: its miss when the line is fit without it
$e_i$: house $i$ 's ordinary residual from the full fit

20. Cook's distance (Card 13)

D_i = \frac{e_i^2\,h_{ii}}{p\,s^2\,(1 - h_{ii})^2}

$D_i$: Cook's distance: how far all fitted prices move if house $i$ is dropped

21. Variance inflation factor

\text{VIF}_j = \frac{1}{1 - R_j^2}

$\text{VIF}_j$: variance inflation factor: how much collinearity inflates $\hat w_j$ 's variance
$R_j^2$: the $R^2$ from regressing feature $j$ on all the other features

22. Matrix-calculus rules (Tool 1)

\frac{\partial (\mathbf a^\top\mathbf w)}{\partial\mathbf w} = \mathbf a, \qquad \frac{\partial (\mathbf w^\top A\mathbf w)}{\partial\mathbf w} = (A + A^\top)\mathbf w, \qquad \text{Var}(A\boldsymbol\varepsilon) = A\,\text{Var}(\boldsymbol\varepsilon)\,A^\top

$\partial$: how fast the top changes as you nudge the bottom, holding the rest fixed: a slope, not a fraction to divide
$\mathbf a$: a fixed vector that doesn't depend on $\mathbf w$
$\mathbf w$: the weight vector, one weight per feature (bold = a vector)
$A$: a fixed square matrix that doesn't depend on $\mathbf w$
$\boldsymbol\varepsilon$: the noise vector: every house's error stacked in one column

Fitting the line — Cards 1–4

Card 1 — OLS for one feature, by hand

Statement. For $\hat y = b + wx$ , least squares gives:

b = \bar y - w\,\bar x, \qquad w = \frac{S_{xy}}{S_{xx}}

$b$: the fitted intercept: the line's price for a 0 sq ft house
$w$: the fitted slope: $k of price per extra sq ft
$\bar y$: the average price over all $n$ houses (bar = mean)
$\bar x$: the average size
$S_{xy}$: $\sum_i (x_i - \bar x)(y_i - \bar y)$ : how size and price move together
$S_{xx}$: $\sum_i (x_i - \bar x)^2$ : how spread out the sizes are

Derivation.

L(b, w) = \sum_i (y_i - b - w x_i)^2

$L$: the loss: the total squared miss, as a function of $b$ and $w$
$\sum_i$: add up over every house $i$
$y_i$: house $i$ 's actual sale price
$x_i$: house $i$ 's size

\frac{\partial L}{\partial b} = -2\sum_i (y_i - b - w x_i) = 0 \;\Rightarrow\; b = \bar y - w\bar x

$\frac{\partial L}{\partial b}$: how fast the loss changes as you nudge $b$ , holding $w$ fixed: a slope, not a fraction to divide

\text{substitute: } L = \sum_i \big[(y_i - \bar y) - w(x_i - \bar x)\big]^2 \;\Rightarrow\; \frac{\partial L}{\partial w} = -2\sum_i (x_i - \bar x)\big[(y_i - \bar y) - w(x_i - \bar x)\big] = 0

$\frac{\partial L}{\partial w}$: how fast the loss changes as you nudge $w$ , with $b$ already solved out

w = \frac{\sum_i (x_i - \bar x)(y_i - \bar y)}{\sum_i (x_i - \bar x)^2} = \frac{S_{xy}}{S_{xx}}

Key move. Solve for $b$ first. Substituting it back centers the data, and the slope falls out in one line.

Common slip. Writing $\sum x_i y_i / \sum x_i^2$ . That's the slope of a line forced through the origin, not this one.

How it's asked. "Derive the least-squares estimates." · "Why does the fitted line pass through $(\bar x, \bar y)$ ?" (Because $b = \bar y - w\bar x$ is exactly that statement.)

Card 2 — The normal equation

Statement. Minimizing the squared error gives the normal equation; the solution is unique when $X$ has full column rank:

\min_{\mathbf w}\lVert \mathbf y - X\mathbf w\rVert^2 \;\Rightarrow\; X^\top X\,\hat{\mathbf w} = X^\top\mathbf y

$\mathbf w$: the weight vector, one weight per feature (bold = a vector)
$\lVert$: length of a vector (closed by $\rVert$ ); squared, it's the sum of squared entries
$\mathbf y$: the column of all $n$ actual prices
$X$: the design matrix: one row per house, one column per feature
$^\top$: transpose: rows become columns
$X^\top X$: the $p \times p$ table of feature sums of squares and cross-products
$\hat{\mathbf w}$: the fitted weights (hat = estimated from data)

Derivation.

L(\mathbf w) = (\mathbf y - X\mathbf w)^\top(\mathbf y - X\mathbf w) = \mathbf y^\top\mathbf y - 2\mathbf w^\top X^\top\mathbf y + \mathbf w^\top X^\top X\mathbf w

$L$: the loss: the sum of squared residuals, now a function of the weight vector $\mathbf w$

\nabla L = -2X^\top\mathbf y + 2X^\top X\mathbf w = 0 \;\Rightarrow\; X^\top X\,\hat{\mathbf w} = X^\top \mathbf y \;\Rightarrow\; \hat{\mathbf w} = (X^\top X)^{-1}X^\top\mathbf y

$\nabla L$: the gradient: the vector of slopes of $L$ , one per weight; points uphill
$(X^\top X)^{-1}$: the inverse of $X^\top X$ : the matrix that undoes it

\nabla^2 L = 2X^\top X \succeq 0 \;\Rightarrow\; \text{convex: a minimum, unique if } X^\top X \text{ is invertible}

$\nabla^2 L$: the Hessian: the matrix of second derivatives, how the slopes themselves change
$\succeq 0$: is positive semi-definite: curves up (or stays flat) in every direction

Rearranged, the first-order condition says the residuals are orthogonal to every column:

X^\top(\mathbf y - X\hat{\mathbf w}) = X^\top\mathbf e = \mathbf 0

$\mathbf e$: the residual vector: each house's actual minus fitted price

Key move. Expand, then use Tool 1. The two middle terms combine because $y^\top X w$ is a scalar, so it equals its own transpose $w^\top X^\top y$ .

Common slip. Writing $(XX^\top)^{-1}$ instead of $(X^\top X)^{-1}$ (check dimensions: $X^\top X$ is $p \times p$ ), or assuming invertibility without full rank.

How it's asked. "Derive OLS in matrix form. When is it unique? Why is it a minimum and not a maximum?"

Card 3 — Least squares is maximum likelihood

Statement. With independent Gaussian noise, maximizing the likelihood over $w$ is minimizing RSS, and the MLE of the noise variance divides by $n$ :

y_i = \mathbf w^\top\mathbf x_i + \varepsilon_i,\; \varepsilon_i \sim N(0, \sigma^2) \;\Rightarrow\; \hat{\mathbf w}_{\text{MLE}} = \hat{\mathbf w}_{\text{OLS}}, \qquad \hat\sigma^2_{\text{MLE}} = \frac{\text{RSS}}{n}

$\mathbf x_i$: house $i$ 's feature vector (size, bedrooms, …)
$\varepsilon_i$: house $i$ 's error: how far its price lands from the true line
$\sim$: is distributed as
$N(0, \sigma^2)$: a normal (Gaussian) bell curve centred at 0 with variance $\sigma^2$
$\sigma^2$: the noise variance: how far prices scatter around the true line
$\hat{\mathbf w}_{\text{MLE}}$: the weights that make the observed prices most likely
$\hat{\mathbf w}_{\text{OLS}}$: the least-squares weights
$\hat\sigma^2_{\text{MLE}}$: the most-likely estimate of the noise variance
$\text{RSS}$: residual sum of squares: $\sum_i e_i^2$
$n$: the number of houses

Derivation.

\mathcal L = \prod_i \frac{1}{\sqrt{2\pi\sigma^2}} \exp\!\Big(-\frac{(y_i - \mathbf w^\top \mathbf x_i)^2}{2\sigma^2}\Big)

$\mathcal L$: the likelihood: how probable the observed prices are, given $\mathbf w$ and $\sigma^2$
$\prod_i$: multiply together over every house $i$ (the errors are independent)
$\pi$: the constant 3.14159…, part of the normal curve's height

\log\mathcal L = -\frac{n}{2}\log(2\pi\sigma^2) - \frac{1}{2\sigma^2}\sum_i (y_i - \mathbf w^\top\mathbf x_i)^2

\text{max over } \mathbf w \iff \text{min } \textstyle\sum_i (y_i - \mathbf w^\top \mathbf x_i)^2; \qquad \frac{\partial \log\mathcal L}{\partial \sigma^2} = -\frac{n}{2\sigma^2} + \frac{\text{RSS}}{2\sigma^4} = 0 \Rightarrow \hat\sigma^2_{\text{MLE}} = \frac{\text{RSS}}{n}

$\textstyle\sum_i$: add up over every house $i$
$\frac{\partial \log\mathcal L}{\partial \sigma^2}$: how fast the log-likelihood changes as you nudge the noise variance
$\sigma^4$: $(\sigma^2)^2$ : the noise variance squared

Key move. The log turns a product into a sum, and only the exponent contains $w$ .

Common slip. Thinking the MLE of $\sigma^2$ is unbiased. It divides by $n$ and is biased low (Card 7).

How it's asked. "Why squared error?" · "What loss would Laplace noise give?" (absolute error — its log-density is $-\lvert e \rvert / b$ ) · "Is the MLE always unbiased?"

Card 4 — TSS = ESS + RSS, and R² = r²

Statement. With an intercept, total variation splits into explained plus residual; in simple regression, $R^2$ is the squared correlation:

\text{TSS} = \text{ESS} + \text{RSS}, \qquad R^2 = r^2 \;\;(\text{one feature})

$\text{TSS}$: total sum of squares: how much prices vary around their mean
$\text{ESS}$: explained sum of squares: the part of that variation the line captures
$R^2$: the share of price variation the line explains
$r$: the correlation between size and price

Derivation.

\text{TSS} = \sum_i (y_i - \bar y)^2 = \sum_i \big[(\hat y_i - \bar y) + e_i\big]^2 = \text{ESS} + \text{RSS} + 2\sum_i (\hat y_i - \bar y)\,e_i

$\hat y_i$: the line's fitted price for house $i$
$e_i$: house $i$ 's residual: $y_i - \hat y_i$

\sum_i (\hat y_i - \bar y) e_i = \hat{\mathbf w}^\top X^\top \mathbf e - \bar y \sum_i e_i = 0 - 0 \quad (\text{orthogonality; } \textstyle\sum e_i = 0 \text{ needs the intercept})

$\textstyle\sum$: add up over every house

\text{simple case: } \hat y_i - \bar y = w(x_i - \bar x) \Rightarrow \text{ESS} = w^2 S_{xx} = \frac{S_{xy}^2}{S_{xx}} \Rightarrow R^2 = \frac{\text{ESS}}{\text{TSS}} = \frac{S_{xy}^2}{S_{xx}S_{yy}} = r^2

$S_{yy}$: $\sum_i (y_i - \bar y)^2$ : how spread out the prices are (equals TSS)

Key move. Add and subtract $\hat y$ (Tool 4); orthogonality kills the cross term.

Common slip. Using $1 - \text{RSS}/\text{TSS}$ for a model without an intercept. The cross term no longer vanishes, so the "decomposition" fails and $R^2$ can go negative.

How it's asked. "Show $R^2 = r^2$ in simple regression." · "When can $R^2$ be negative?"

Trusting the line — Cards 5–8

Card 5 — OLS is unbiased

Statement. If the model is linear, $X$ has full rank and the errors are exogenous, OLS is unbiased:

\mathbf y = X\boldsymbol\beta + \boldsymbol\varepsilon,\;\; \mathbb E[\boldsymbol\varepsilon\mid X] = \mathbf 0 \;\Rightarrow\; \mathbb E[\hat{\mathbf w}\mid X] = \boldsymbol\beta

$\mathbf y$: the column of all $n$ actual prices
$X$: the design matrix: one row per house, one column per feature
$\boldsymbol\beta$: the true weights of the real-world model (unknown)
$\boldsymbol\varepsilon$: the noise vector: every house's error stacked in one column
$\mathbb E$: expected value: the average over many repeated samples
$\mid X$: given $X$ : the houses' features held fixed, only the noise redrawn
$\hat{\mathbf w}$: the OLS weights fitted from the data (hat = estimated)

Derivation.

\hat{\mathbf w} = (X^\top X)^{-1}X^\top(X\boldsymbol\beta + \boldsymbol\varepsilon) = \boldsymbol\beta + (X^\top X)^{-1}X^\top\boldsymbol\varepsilon

$^\top$: transpose: rows become columns
$(X^\top X)^{-1}$: the inverse of $X^\top X$ : the matrix that undoes it

\mathbb E[\hat{\mathbf w} \mid X] = \boldsymbol\beta + (X^\top X)^{-1}X^\top\,\mathbb E[\boldsymbol\varepsilon \mid X] = \boldsymbol\beta \qquad \text{(then average over } X\text{: } \mathbb E[\hat{\mathbf w}] = \boldsymbol\beta)

Key move. Substitute the true model into the estimator. Every OLS proof starts here: estimate = truth + weighted noise.

Common slip. Saying it needs normality or constant variance (it doesn't), or that $E[\varepsilon] = 0$ is enough (it isn't — the noise must average zero at every $X$ , which is exogeneity).

How it's asked. "Prove OLS is unbiased. Which assumptions did you use, and which didn't you?"

Card 6 — The variance of OLS, standard errors, and the sandwich

Statement. With equal-variance, uncorrelated errors the covariance is $\sigma^2 (X^\top X)^{-1}$ ; in general it's the sandwich:

\text{Var}(\boldsymbol\varepsilon\mid X) = \sigma^2 I \Rightarrow \text{Var}(\hat{\mathbf w}\mid X) = \sigma^2 (X^\top X)^{-1}; \qquad \text{Var}(\boldsymbol\varepsilon\mid X) = \Omega \Rightarrow (X^\top X)^{-1}X^\top\Omega X (X^\top X)^{-1}

$\text{Var}$: variance; for a vector, the covariance matrix (variances on the diagonal)
$\sigma^2$: the noise variance: how far prices scatter around the true line
$I$: the identity matrix: 1s on the diagonal, so equal variances and no correlation
$\Omega$: the general noise covariance: per-house variances may differ, errors may correlate

Derivation.

\hat{\mathbf w} - \boldsymbol\beta = A\boldsymbol\varepsilon, \quad A = (X^\top X)^{-1}X^\top \;\Rightarrow\; \text{Var}(\hat{\mathbf w}\mid X) = A\,\text{Var}(\boldsymbol\varepsilon\mid X)\,A^\top

$A$: the fixed matrix that turns noise into estimation error (Tool 2's $A$ )

\Omega = \sigma^2 I: \;\; \sigma^2 (X^\top X)^{-1}X^\top X (X^\top X)^{-1} = \sigma^2 (X^\top X)^{-1} \qquad \text{general: } (X^\top X)^{-1}X^\top \Omega X(X^\top X)^{-1}

\text{simple case: } \text{Var}(\hat w) = \frac{\sigma^2}{S_{xx}}, \;\; \text{SE}(\hat w) = \frac{s}{\sqrt{S_{xx}}}; \qquad \text{Var}(\mathbf x_0^\top \hat{\mathbf w}) = \sigma^2\,\mathbf x_0^\top (X^\top X)^{-1}\mathbf x_0

$\hat w$: the fitted slope on size, with one feature
$S_{xx}$: $\sum_i (x_i - \bar x)^2$ : how spread out the sizes are
$\text{SE}(\hat w)$: standard error: the estimated typical wobble of $\hat w$ across samples
$s$: the estimated noise standard deviation, from the residuals
$\mathbf x_0$: a new house's feature vector

The last line gives the confidence interval for the mean at $x_0$ . A new observation adds its own $\sigma^2$ , which is the "1 +" in the prediction interval.

Key move. Tool 2 with $A = (X^\top X)^{-1}X^\top$ . The bread is $(X^\top X)^{-1}X^\top$ , the filling is $\Omega$ — hence "sandwich".

Common slip. Simplifying to $\sigma^2 (X^\top X)^{-1}$ without stating that it assumes equal variance and uncorrelated errors.

How it's asked. "Derive the variance of the OLS estimator." · "What changes under heteroscedasticity?" · "Why is a prediction interval wider than a confidence interval?"

Card 7 — Why divide by n − p

Statement. The expected residual sum of squares is $(n - p)\sigma^2$ , so dividing by $n - p$ gives an unbiased noise estimate:

\mathbb E[\text{RSS}] = (n - p)\,\sigma^2 \;\Rightarrow\; s^2 = \frac{\text{RSS}}{n - p}, \quad \mathbb E[s^2] = \sigma^2

$\text{RSS}$: residual sum of squares: the total squared miss, $\mathbf e^\top\mathbf e$
$n$: the number of houses
$p$: the number of fitted weights (counting the intercept)
$s^2$: the noise-variance estimate: RSS divided by $n - p$

Derivation.

\mathbf e = (I - H)\mathbf y = M(X\boldsymbol\beta + \boldsymbol\varepsilon) = M\boldsymbol\varepsilon \quad (\text{since } MX = 0)

$\mathbf e$: the residual vector: each house's actual minus fitted price
$H$: the hat matrix $X(X^\top X)^{-1}X^\top$ : turns prices into fitted prices
$M$: the residual maker $I - H$ : turns prices into residuals

\text{RSS} = \mathbf e^\top \mathbf e = \boldsymbol\varepsilon^\top M^\top M \boldsymbol\varepsilon = \boldsymbol\varepsilon^\top M \boldsymbol\varepsilon \quad (M \text{ symmetric, idempotent})

\mathbb E[\text{RSS}] = \sigma^2\,\text{tr}(M) = \sigma^2\big(n - \text{tr}(H)\big), \quad \text{tr}(H) = \text{tr}\big((X^\top X)^{-1}X^\top X\big) = \text{tr}(I_p) = p

$\text{tr}$: trace: the sum of a matrix's diagonal entries
$I_p$: the $p \times p$ identity matrix

Key move. Residuals are a projection of the noise, $e = M\varepsilon$ ; then the trace trick (Tool 3). Rotating inside the trace turns $\operatorname{tr}(H)$ into $\operatorname{tr}(I_p)$ .

Common slip. Saying $\operatorname{tr}(H) = n$ . $H$ is $n \times n$ but has rank $p$ .

How it's asked. "Why $n - p$ degrees of freedom?" · "Prove $s^2$ is unbiased." · "Why does the sample variance divide by $n - 1$ ?" (Same proof with $X$ = a column of 1s, $p = 1$ .)

Card 8 — Gauss–Markov: OLS is BLUE

Statement. Under linearity, full rank, exogeneity and spherical errors, every linear unbiased estimator has at least the variance of OLS:

\text{Var}(\boldsymbol\varepsilon\mid X) = \sigma^2 I,\;\; \tilde{\mathbf w} = C\mathbf y,\;\; \mathbb E[\tilde{\mathbf w}] = \boldsymbol\beta \;\Rightarrow\; \text{Var}(\tilde{\mathbf w}) - \text{Var}(\hat{\mathbf w}_{\text{OLS}}) \succeq 0

$\tilde{\mathbf w}$: any rival estimator that is linear in the prices (tilde = an alternative)
$C$: the fixed matrix defining that rival: how it weights each price
$\hat{\mathbf w}_{\text{OLS}}$: the ordinary least-squares weights
$\succeq 0$: is positive semi-definite: the rival's variance is never smaller, in any direction

Derivation.

\tilde{\mathbf w} = C\mathbf y, \quad C = (X^\top X)^{-1}X^\top + D \;\Rightarrow\; \mathbb E[\tilde{\mathbf w}\mid X] = (I + DX)\boldsymbol\beta

$D$: how the rival differs from OLS: $C$ minus the OLS matrix

\text{unbiased for every } \boldsymbol\beta \;\Rightarrow\; DX = 0

\text{Var}(\tilde{\mathbf w}\mid X) = \sigma^2 CC^\top = \sigma^2\big[(X^\top X)^{-1} + DD^\top\big] \quad (\text{cross terms contain } DX = 0)

Since $DD^\top$ is positive semi-definite, $\text{Var}(\tilde w) \ge \text{Var}(\hat w)$ , with equality only when $D = 0$ .

Key move. Write any competitor as "OLS + $D$ ". Unbiasedness forces $DX = 0$ , which deletes the cross terms, leaving OLS's variance plus something non-negative.

Common slip. Forgetting where the conditions entered: $\sigma^2 I$ is used in $\text{Var}(Cy) = \sigma^2 CC^\top$ . Without it, OLS stops being best.

How it's asked. Research and quant roles: "Prove Gauss–Markov." Elsewhere: "Why is OLS called BLUE, and when does it stop being best?"

Bias, regularization, and prediction — Cards 9–13

Card 9 — Omitted-variable bias

Statement. If the truth includes $z$ but you regress $y$ on $X$ alone, the estimate absorbs $z$ 's effect through its correlation with $X$ :

\mathbf y = X\boldsymbol\beta + \mathbf z\gamma + \boldsymbol\varepsilon \;\Rightarrow\; \mathbb E[\hat{\mathbf w}] = \boldsymbol\beta + \boldsymbol\delta\,\gamma, \qquad \boldsymbol\delta = (X^\top X)^{-1}X^\top\mathbf z

$\mathbf y$: the column of all actual prices
$X$: the features you did include, one row per house
$\boldsymbol\beta$: the true weights on those features
$\mathbf z$: the omitted feature for every house (e.g. neighbourhood quality)
$\gamma$: the omitted feature's true effect on price
$\boldsymbol\varepsilon$: the noise vector: every house's error
$\mathbb E$: expected value: the average over many repeated samples
$\hat{\mathbf w}$: the weights fitted without $\mathbf z$ (the short regression)
$\boldsymbol\delta$: slopes from regressing $\mathbf z$ on $X$ : how the omitted feature tracks the included ones
$^\top$: transpose: rows become columns
$(X^\top X)^{-1}$: the inverse of $X^\top X$ : the matrix that undoes it

Derivation.

\hat{\mathbf w} = (X^\top X)^{-1}X^\top(X\boldsymbol\beta + \mathbf z\gamma + \boldsymbol\varepsilon) = \boldsymbol\beta + \underbrace{(X^\top X)^{-1}X^\top \mathbf z}_{\boldsymbol\delta}\,\gamma + (X^\top X)^{-1}X^\top \boldsymbol\varepsilon

\mathbb E[\hat{\mathbf w}\mid X, \mathbf z] = \boldsymbol\beta + \boldsymbol\delta\gamma; \qquad \text{one feature: } \delta = \frac{\text{Cov}(x, z)}{\text{Var}(x)}

$\mid X, \mathbf z$: given both included and omitted features: only the noise is redrawn
$\delta$: with one feature: the slope from regressing $z$ on $x$
$\text{Cov}$: covariance: how two quantities move together
$\text{Var}$: variance: how spread out a quantity is
$x$: the included feature (size)
$z$: the omitted feature (neighbourhood quality)

Key move. Card 5's substitution, but with the long (true) model plugged into the short regression.

Common slip. Forgetting that bias needs both $\gamma \ne 0$ and $\delta \ne 0$ , or getting the sign wrong. Sign of bias $= \operatorname{sign}(\gamma) \times \operatorname{sign}(\delta)$ .

How it's asked. "Derive omitted-variable bias." · Worked problem 2 in the Interview Bank (NBA height).

Card 10 — Attenuation from measurement error

Statement. If you observe $x = x^* + u$ with independent noise $u$ , the slope converges to $\beta$ shrunk by the reliability ratio $\lambda$ :

y = \beta x^* + \varepsilon,\;\; x = x^* + u \;\Rightarrow\; \text{plim}\,\hat w = \beta\lambda, \qquad \lambda = \frac{\text{Var}(x^*)}{\text{Var}(x^*) + \text{Var}(u)} < 1

$y$: a house's price
$\beta$: the true slope of price on true size
$x^*$: the house's true size (never seen)
$\varepsilon$: the price noise
$x$: the size you actually record: true size plus measurement noise
$u$: the measurement noise in size, independent of everything else
$\text{plim}$: probability limit: where the estimate settles with endless data
$\hat w$: the slope fitted on recorded size
$\lambda$: the reliability ratio: the share of recorded size's variance that is real signal

Derivation.

\text{plim}\,\hat w = \frac{\text{Cov}(y, x)}{\text{Var}(x)} = \frac{\text{Cov}(\beta x^* + \varepsilon,\; x^* + u)}{\text{Var}(x^*) + \text{Var}(u)} = \beta\,\frac{\text{Var}(x^*)}{\text{Var}(x^*) + \text{Var}(u)}

\text{reverse regression: } \frac{\text{Var}(y)}{\text{Cov}(x, y)} = \frac{\beta^2\text{Var}(x^*) + \text{Var}(\varepsilon)}{\beta\,\text{Var}(x^*)} \ge \beta \;\Rightarrow\; \beta\lambda \le \beta \le \frac{\text{Var}(y)}{\text{Cov}(x,y)} \;\; (\beta > 0)

$\frac{\text{Var}(y)}{\text{Cov}(x, y)}$: the reverse-regression slope flipped: regress size on price, then take 1 over it

Key move. "Cov over Var", then bilinearity (Tool 2): of the four cross-covariances, only $\text{Cov}(x^*, x^*)$ survives in the numerator, while $\text{Var}(u)$ stays in the denominator.

Common slip. Saying noise in $y$ also biases the slope (it doesn't — it only appears in the numerator as $\text{Cov}(\text{noise}, x) = 0$ ).

How it's asked. Worked problem 1 in the Interview Bank (the reported Upstart question) · "Can you bound the true effect without knowing the noise variance?"

Card 11 — Ridge's closed form, and lasso's soft threshold

Statement. Ridge has a closed form that's always invertible for $\lambda > 0$ and is biased toward zero; with orthonormal features ridge divides and lasso soft-thresholds:

\hat{\mathbf w}_R = (X^\top X + \lambda I)^{-1}X^\top\mathbf y; \qquad X^\top X = I:\;\; \hat w_j^{R} = \frac{\hat w_j}{1 + \lambda}, \quad \hat w_j^{L} = \text{sign}(\hat w_j)\max\big(|\hat w_j| - \tfrac{\lambda}{2}, 0\big)

$\hat{\mathbf w}_R$: the ridge weights: least squares plus a penalty on squared weights
$\lambda$: the penalty strength you choose; bigger shrinks the weights harder
$I$: the identity matrix: 1s on the diagonal, 0s elsewhere
$^{-1}$: matrix inverse: undoes the matrix
$\hat w_j^{R}$: the ridge weight on feature $j$
$\hat w_j$: the plain OLS weight on feature $j$
$\hat w_j^{L}$: the lasso weight on feature $j$
$\text{sign}$: +1 if positive, −1 if negative: keeps the weight's direction
$|\hat w_j|$: the size of the OLS weight, ignoring its sign

Derivation.

L = \lVert \mathbf y - X\mathbf w\rVert^2 + \lambda\lVert\mathbf w\rVert^2 \;\Rightarrow\; -2X^\top\mathbf y + 2X^\top X\mathbf w + 2\lambda\mathbf w = 0 \;\Rightarrow\; \hat{\mathbf w}_R = (X^\top X + \lambda I)^{-1}X^\top\mathbf y

$L$: the ridge loss: squared misses plus $\lambda$ times the squared weight length
$\lVert$: length of a vector (closed by $\rVert$ ); squared, it's the sum of squared entries
$\mathbf w$: the weight vector being chosen, one weight per feature

\mathbb E[\hat{\mathbf w}_R] = (X^\top X + \lambda I)^{-1}X^\top X\,\boldsymbol\beta \ne \boldsymbol\beta; \qquad X^\top X = I: \;\; \hat{\mathbf w}_R = \frac{\hat{\mathbf w}_{\text{OLS}}}{1 + \lambda}, \;\; \text{Var} = \frac{\sigma^2}{(1+\lambda)^2}

$\hat{\mathbf w}_{\text{OLS}}$: the plain least-squares weights, no penalty
$\sigma^2$: the noise variance: how far prices scatter around the true line

\text{lasso, } X^\top X = I: \;\; \min_{w_j}\, (w_j - \hat w_j)^2 + \lambda |w_j| \;\Rightarrow\; \hat w_j^{\text{lasso}} = \text{sign}(\hat w_j)\,\max\!\big(|\hat w_j| - \tfrac{\lambda}{2},\, 0\big)

$w_j$: a candidate weight on feature $j$ , being chosen
$\hat w_j^{\text{lasso}}$: the lasso weight on feature $j$ : the winning candidate

Invertibility: $X^\top X$ has eigenvalues $\ge 0$ , so $X^\top X + \lambda I$ has eigenvalues $\ge \lambda > 0$ . In the SVD view, ridge shrinks each direction by a factor, and the weakly-determined directions (small singular values $d_j$ , the collinear ones) shrink the most:

X = UDV^\top \;\Rightarrow\; X\hat{\mathbf w}_R = \sum_j \mathbf u_j\,\frac{d_j^2}{d_j^2 + \lambda}\,\mathbf u_j^\top\mathbf y

$U$: left singular vectors: orthonormal directions across houses, as columns
$D$: diagonal matrix of singular values $d_j$ : how much $X$ stretches each direction
$V$: right singular vectors: orthonormal directions across features
$\sum_j$: add up over every direction $j$
$\mathbf u_j$: the $j$ -th column of $U$
$d_j$: the $j$ -th singular value; small means a weakly determined (collinear) direction

Key move. Card 2 plus a $2\lambda w$ term. For lasso, split into cases $w > 0$ and $w < 0$ ; any $\lvert \hat w \rvert \le \lambda/2$ lands exactly on zero.

Common slip. Penalizing the intercept, or forgetting to standardize features first.

How it's asked. "Derive ridge. Why is it always invertible? Show it's biased." · "Why does lasso give exact zeros but ridge doesn't?"

Card 12 — Bias–variance decomposition of prediction error

Statement. For a new point $y_0 = f(x_0) + \varepsilon_0$ and a model $\hat f$ trained on independent data:

\mathbb E\big[(y_0 - \hat f(\mathbf x_0))^2\big] = \underbrace{\sigma^2}_{\text{irreducible}} + \underbrace{\big(f(\mathbf x_0) - \mathbb E\hat f(\mathbf x_0)\big)^2}_{\text{bias}^2} + \underbrace{\text{Var}\big(\hat f(\mathbf x_0)\big)}_{\text{variance}}

$y_0$: the actual price of a new house
$\mathbf x_0$: that new house's features
$\hat f$: the model trained on one sample of houses (hat = learned from data)
$f$: the true price function: the average price for given features

Derivation.

y_0 - \hat f = (f - \hat f) + \varepsilon_0 \;\Rightarrow\; \mathbb E[(y_0 - \hat f)^2] = \mathbb E[(f - \hat f)^2] + \sigma^2 \quad (\varepsilon_0 \text{ independent, mean } 0)

$\varepsilon_0$: the new house's own noise, independent of the training data

f - \hat f = (f - \mathbb E\hat f) + (\mathbb E\hat f - \hat f) \;\Rightarrow\; \mathbb E[(f - \hat f)^2] = (f - \mathbb E\hat f)^2 + \text{Var}(\hat f) \quad (\text{cross term has mean } 0)

For a correctly specified OLS model the bias is zero, and the variance averaged over the training points is:

\frac{1}{n}\sum_i \text{Var}(\hat y_i) = \frac{\sigma^2}{n}\,\text{tr}(H) = \frac{\sigma^2 p}{n}

$n$: the number of training houses
$\sum_i$: add up over every training house $i$
$\hat y_i$: the fitted price for training house $i$
$\text{tr}(H)$: trace of the hat matrix: its diagonal sum, which equals $p$
$p$: the number of fitted weights

So each extra feature costs $\sigma^2/n$ of expected error — the precise reason more features can hurt.

Key move. Add and subtract twice (Tool 4): once with $f$ , once with $E[\hat f]$ .

Common slip. Forgetting the irreducible $\sigma^2$ , or confusing this with the estimator decomposition $\text{MSE}(\hat\theta) = \text{bias}^2 + \text{variance}$ (that one has no noise term).

How it's asked. "Derive the bias–variance trade-off." · "Why can adding features make test error worse?"

Card 13 — Leave-one-out CV without refitting

Statement. Each leave-one-out residual comes from a single fit, so LOOCV needs no refitting:

e_{(-i)} = \frac{e_i}{1 - h_{ii}}, \qquad \text{LOOCV} = \frac{1}{n}\sum_{i=1}^{n}\Big(\frac{e_i}{1 - h_{ii}}\Big)^2

$e_{(-i)}$: house $i$ 's leave-one-out residual: its miss when the line is fit without it
$e_i$: house $i$ 's ordinary residual from the full fit
$h_{ii}$: house $i$ 's leverage: the $i$ -th diagonal entry of the hat matrix
$\text{LOOCV}$: leave-one-out cross-validation error: the average squared leave-one-out miss
$\sum_{i=1}^{n}$: add up over every house, $i = 1$ to $n$

Derivation. With $A = X^\top X$ , removing row $i$ is a rank-one update. Sherman–Morrison gives:

(A - \mathbf x_i\mathbf x_i^\top)^{-1} = A^{-1} + \frac{A^{-1}\mathbf x_i\mathbf x_i^\top A^{-1}}{1 - h_{ii}}, \qquad h_{ii} = \mathbf x_i^\top A^{-1}\mathbf x_i

$A$: shorthand for $X^\top X$ from the full data
$\mathbf x_i$: house $i$ 's feature vector (row $i$ of $X$ , stood up as a column)
$A^{-1}$: the inverse of $A$ : the matrix that undoes it

\hat{\mathbf w}_{(-i)} = \hat{\mathbf w} - \frac{A^{-1}\mathbf x_i\, e_i}{1 - h_{ii}} \;\Rightarrow\; y_i - \mathbf x_i^\top\hat{\mathbf w}_{(-i)} = e_i + \frac{h_{ii}\,e_i}{1 - h_{ii}} = \frac{e_i}{1 - h_{ii}}

$\hat{\mathbf w}_{(-i)}$: the weights refit with house $i$ left out
$y_i$: house $i$ 's actual price

The same pieces give Cook's distance — a big residual times high leverage (both formulas checked numerically against a brute-force refit):

D_i = \frac{e_i^2\,h_{ii}}{p\,s^2\,(1 - h_{ii})^2}

$D_i$: Cook's distance: how far all fitted prices move if house $i$ is dropped
$s^2$: the noise-variance estimate, $\text{RSS}/(n - p)$

Key move. Sherman–Morrison for a rank-one change to $X^\top X$ .

Common slip. Dropping the $(1 - h_{ii})$ — high-leverage points have LOO errors much larger than their in-sample residuals.

How it's asked. "How would you compute LOOCV for linear regression efficiently?" · "Why are high-leverage points' residuals misleadingly small?"

Research-level extensions — Cards 14–17

Card 14 — Fixed effects, and why they worsen attenuation

Statement. Demeaning within each unit removes a fixed trait $a_i$ and gives an unbiased slope under strict exogeneity; with noisy $x$ , the within estimator is attenuated more than pooled OLS:

y_{it} = \beta x_{it} + a_i + \varepsilon_{it} \;\Rightarrow\; \mathbb E[\hat\beta_{\text{FE}}] = \beta; \qquad \text{noisy } x:\;\; \lambda_{\text{within}} < \lambda_{\text{pooled}}

$y_{it}$: house $i$ 's price at its sale in year $t$ (each house sells several times)
$\beta$: the true slope: price change per unit of the feature
$x_{it}$: house $i$ 's feature at that sale (e.g. size after renovations)
$a_i$: house $i$ 's fixed trait: everything constant about it (location, view), unobserved
$\varepsilon_{it}$: the noise on that sale
$\mathbb E$: expected value: the average over many repeated samples
$\hat\beta_{\text{FE}}$: the fixed-effects (within) slope estimate
$x$: the feature (e.g. size), across all sales
$\lambda_{\text{within}}$: the reliability ratio after demeaning: the share of $x$ 's variance that is signal
$\lambda_{\text{pooled}}$: the same ratio for the raw, pooled data

Derivation.

\ddot y_{it} = y_{it} - \bar y_i = \beta\,\ddot x_{it} + \ddot\varepsilon_{it} \quad (a_i - \bar a_i = 0) \qquad \hat\beta_{\text{FE}} = \frac{\sum_{i,t} \ddot x_{it}\,\ddot y_{it}}{\sum_{i,t} \ddot x_{it}^2}

$\ddot y_{it}$: the demeaned price: this sale's price minus house $i$ 's average (double dot = demeaned)
$\bar y_i$: house $i$ 's average price over its sales (bar = mean)
$\ddot x_{it}$: the demeaned feature: this sale's value minus house $i$ 's average
$\ddot\varepsilon_{it}$: the demeaned noise
$\bar a_i$: house $i$ 's average fixed trait, which is just $a_i$
$\sum_{i,t}$: add up over every house $i$ and every sale $t$

\text{observed } \tilde x = x + v: \quad \text{plim}\,\hat\beta_{\text{FE}} = \beta\,\frac{\text{Var}(\ddot x)}{\text{Var}(\ddot x) + \text{Var}(\ddot v)}

$\tilde x$: the recorded feature, with measurement noise (tilde = observed with noise)
$x$: the feature's true value
$v$: measurement noise, independent across sales
$\text{plim}$: probability limit: where the estimate settles with endless data
$\text{Var}$: variance: how spread out a quantity is
$\ddot x$: the true feature after demeaning within each house
$\ddot v$: the noise after demeaning within each house

Demeaning removes all between-unit signal in $x$ but only a $1/T$ share of the (independent) noise, so the within reliability ratio is lower than the pooled one.

Key move. Subtract each unit's mean — equivalent to adding a dummy per unit (Frisch–Waugh–Lovell).

Common slip. Forgetting that fixed effects need strict exogeneity: errors uncorrelated with $x$ in every period. A lagged $y$ violates it (Nickell bias, Drill 25).

How it's asked. "What does a fixed-effects model control for, and what can't it fix?" · Drill 13.

Card 15 — The training-vs-test gap (optimism)

Statement. For a correctly specified OLS model with $p$ parameters, evaluated at the training $x$ 's:

\mathbb E[\text{train MSE}] = \sigma^2\Big(1 - \frac{p}{n}\Big), \qquad \mathbb E[\text{test MSE}] = \sigma^2\Big(1 + \frac{p}{n}\Big)

$\text{train MSE}$: mean squared error on the houses the model was fit to
$\text{test MSE}$: mean squared error on fresh prices at those same houses
$\sigma^2$: the noise variance: how far prices scatter around the true line
$p$: the number of fitted weights
$n$: the number of training houses

Derivation.

\mathbb E[\text{train MSE}] = \frac{\mathbb E[\text{RSS}]}{n} = \frac{(n - p)\sigma^2}{n} \quad (\text{Card 7})

$\text{RSS}$: residual sum of squares: the total squared training miss

\mathbb E\Big[\tfrac{1}{n}\lVert \mathbf y_{\text{new}} - X\hat{\mathbf w}\rVert^2\Big] = \sigma^2 + \tfrac{1}{n}\mathbb E\lVert X(\hat{\mathbf w} - \boldsymbol\beta)\rVert^2 = \sigma^2 + \tfrac{\sigma^2}{n}\text{tr}(H) = \sigma^2\Big(1 + \frac{p}{n}\Big)

$\lVert$: length of a vector (closed by $\rVert$ ); squared, it's the sum of squared entries
$\mathbf y_{\text{new}}$: fresh prices for the same houses, with new noise
$X$: the design matrix: one row per training house
$\hat{\mathbf w}$: the weights fitted on the training prices
$\boldsymbol\beta$: the true weights
$\text{tr}(H)$: trace of the hat matrix: its diagonal sum, which equals $p$

Key move. The trace trick twice (Tool 3): once on $M = I - H$ for the training error, once on $H$ for the prediction variance.

Common slip. Thinking the gap is $p\sigma^2/n$ . It's $2p\sigma^2/n$ — training error sits $p\sigma^2/n$ below $\sigma^2$ and test error $p\sigma^2/n$ above it.

How it's asked. "Why does training error underestimate test error, and by how much?" · "Where do Mallows' Cp and AIC come from?"

Card 16 — The Moulton factor

Statement. With groups of size $m$ , $x$ constant within groups and within-group error correlation $\rho$ , clustering inflates the slope's variance:

\text{Var}(\hat w) = \text{Var}_{\text{naive}}(\hat w)\cdot\big[1 + (m - 1)\rho\big]; \qquad \text{in general } 1 + (m-1)\,\rho_x\,\rho_\varepsilon

$\text{Var}(\hat w)$: the slope's actual variance when errors are correlated within groups
$\text{Var}_{\text{naive}}(\hat w)$: the variance the usual formula reports, assuming independent errors
$m$: the number of houses in each group (e.g. each neighbourhood)
$\rho$: the correlation between two houses' errors in the same group
$\rho_x$: how correlated the feature is within a group (1 if shared, like a school rating)
$\rho_\varepsilon$: the within-group error correlation (the $\rho$ above)

Derivation. With centered $x$ , $\hat w - \beta = \sum_i x_i \varepsilon_i / \sum_i x_i^2$ . Group $g$ contributes $m$ identical x-values $\tilde x_g$ :

\text{Var}\Big(\sum_i x_i\varepsilon_i\Big) = \sigma^2 \sum_g \tilde x_g^2\big[m + m(m-1)\rho\big], \qquad \Big(\sum_i x_i^2\Big)^2 = \Big(m\sum_g \tilde x_g^2\Big)^2

$\text{Var}$: variance: how spread out a quantity is across repeated samples
$\sum_i$: add up over every house $i$
$x_i$: house $i$ 's centred feature value
$\varepsilon_i$: house $i$ 's error
$\sum_g$: add up over every group $g$
$\tilde x_g$: the feature value shared by every house in group $g$

\text{Var}(\hat w) = \frac{\sigma^2}{m\sum_g \tilde x_g^2}\,\big[1 + (m-1)\rho\big] = \text{Var}_{\text{naive}}(\hat w)\times\big[1 + (m-1)\rho\big]

Key move. Tool 2 with a block-diagonal error covariance: each group adds $m$ variance terms and $m(m - 1)$ covariance terms.

Common slip. Assuming clustering only matters when $\rho$ is large. With $m = 100$ and $\rho = 0.05$ , the variance is already about 6× the naive one.

How it's asked. "Why do we cluster standard errors?" · Drill 24.

Card 17 — Non-collapsibility: omitting an independent variable in probit and logit

Statement. In probit, omitting a variable $z$ that is independent of $x$ still shrinks $x$ 's coefficient; logit behaves almost the same way:

y^* = \beta x + \gamma z + \varepsilon,\;\; z \perp x \;\Rightarrow\; \hat\beta_{\text{probit, omit } z} \to \frac{\beta}{\sqrt{1 + \gamma^2\sigma_z^2}}

$y^*$: the hidden score: buyer keenness; the house sells above asking when it's positive
$\beta$: the true effect of $x$ on that score
$x$: the included feature (e.g. size)
$\gamma$: the true effect of the omitted variable $z$ on the score
$z$: a variable left out of the model, independent of $x$
$\varepsilon$: the noise in the score, standard normal in probit
$\perp$: is independent of
$\hat\beta_{\text{probit, omit } z}$: the probit slope fitted with $z$ left out
$\sigma_z^2$: the variance of $z$

Derivation.

y = \mathbf 1[\beta x + \gamma z + \varepsilon > 0], \;\; \varepsilon \sim N(0,1) \;\Rightarrow\; \gamma z + \varepsilon \sim N\big(0,\, 1 + \gamma^2\sigma_z^2\big)

$y$: 1 if the house sells above asking, 0 if not
$\mathbf 1$: 1 if the condition in brackets is true, else 0
$\sim$: is distributed as
$N(0,1)$: the standard normal: mean 0, variance 1
$N$: a normal distribution, with (mean, variance) in the brackets

P(y = 1 \mid x) = \Phi\Big(\frac{\beta x}{\sqrt{1 + \gamma^2\sigma_z^2}}\Big) \;\Rightarrow\; \hat\beta_{\text{probit, omit } z} \to \frac{\beta}{\sqrt{1 + \gamma^2\sigma_z^2}}

$P(y = 1 \mid x)$: the probability the house sells above asking, given $x$
$\Phi$: the standard normal CDF: turns a score into a probability

With $\beta = \gamma = 1$ and $\text{Var}(z) = 4$ , that's $1/\sqrt{5} \approx 0.447$ (simulation: 0.445). For logit, replace 1 with $\pi^2/3$ inside the square root; it's an approximation because the sum is no longer exactly logistic (Drill 26: about 0.67 predicted, 0.61 simulated).

Key move. Fold the omitted variable into the error, then rescale so the error has the variance the model insists on.

Common slip. Carrying over the OLS rule "an independent omitted variable causes no bias". In OLS the slope is unaffected; in probit and logit it shrinks.

How it's asked. "Adding an uncorrelated control changed my logistic coefficient. Why?" · "Can you compare odds ratios across models with different controls?"

Practice protocol

Reading a derivation feels like understanding it; reproducing it is what an interview tests. Practice like this:

Blank page, out loud. Write the statement, then derive while explaining each line as if to an interviewer.
Stuck? Peek at the key move only, never the full derivation. Then try again.
Log every slip next to the card's Common slip. Your own slips are the ones you'll repeat under pressure.
Review the day after, 3 days later, a week later, and 3 weeks later. A card you miss goes back to day 1.

Level it's asked at	Cards	Target from a blank page
Onsite core	1, 2, 3, 4	Under 3 minutes each
Senior	5, 6, 7, 9, 12	Under 4 minutes each
Research / quant	8, 10, 11, 13, 14–17	Under 5 minutes each

Mastery tracker — tick a card when you've done it from memory, under the target time, twice on different days:

[ ] Card 1 — OLS by hand
[ ] Card 2 — Normal equation
[ ] Card 3 — Least squares = MLE
[ ] Card 4 — $\text{TSS} = \text{ESS} + \text{RSS}$ , $R^2 = r^2$
[ ] Card 5 — OLS is unbiased
[ ] Card 6 — Variance and the sandwich
[ ] Card 7 — Why $n - p$
[ ] Card 8 — Gauss–Markov
[ ] Card 9 — Omitted-variable bias
[ ] Card 10 — Attenuation
[ ] Card 11 — Ridge and lasso
[ ] Card 12 — Bias–variance of prediction error
[ ] Card 13 — LOOCV shortcut
[ ] Card 14 — Fixed effects and attenuation
[ ] Card 15 — Training-vs-test gap
[ ] Card 16 — Moulton factor
[ ] Card 17 — Non-collapsibility (probit and logit)