Chapter 4 — Interview Bank

Every answer here follows the same four beats. Say the first two within 20 seconds; the last two are what turn a correct answer into a strong one.

Questions are stacked by difficulty. Each level assumes everything above it.

How questions are written here. Real interviewers rarely ask "define overfitting". They describe a situation — a model, a chart, a stakeholder's claim — and watch whether you spot the concept underneath. So every question below opens with a scenario, then names the concept being tested, then gives the four beats. Most scenarios use the chapter's two running examples: the bike-share season (30 days, true curve \sin(2\pi x), noise \sigma = 0.3, rerun 500 times and scored on 10,000 new days) with its six-day toy, and the 100 stations × 80 descriptors. Every number comes from the chapter's simulations. Practice by reading only the scenario and answering out loud before you look.

Level 1 — Screen

Q1 · 99% on train, 71% on holdout

A junior data scientist's churn classifier scores 99% accuracy on its training customers and 71% on a holdout month. The PM says: "We're clearly missing signal — let's add the 200 behavioral features engineering just built." What's going on, and will more features close the gap?

• Concept tested: overfitting and the generalization gap.
• Intuition: the model has memorized the training customers, quirks included, and the quirks don't come back next month.
• Answer: a 28-point gap between training and holdout is the signature of overfitting (high variance), not missing signal. Hook: train tells bias, gap tells variance — 99% on train says bias isn't the problem; the gap says variance is. Adding 200 features adds flexibility, which typically widens the gap. First try: regularize (a penalty, shallower trees), simplify, or get more customers; then compare on cross-validation, not training accuracy.
• Why: training error can only fall as the family grows, because the bigger family contains the smaller one. On the chapter's six-day toy, the degree-5 curve has training error exactly 0 and next-year error 0.258, while the cubic scores 0.041 — 6.3 times better — with a worse training fit (Card 1).
• Next they'll ask: "When would more features help?" → When train and holdout are both bad and close together: high bias (Q5, Q6).

Q2 · "Report the training error of 0.031"

The forecasting lead fit a degree-15 polynomial to this season's 30 days of ride counts. Its training MSE is 0.031 (\text{thousand rides})^2, and she wants to put "expected error 0.031" in next year's staffing plan. What do you tell her, and what number should go in the plan instead?

• Concept tested: risk vs empirical risk; optimism; a held-out estimate with its standard error.
• Intuition: a curve graded on the exam it studied from always looks like a genius.
• Answer: 0.031 is empirical risk, the error on the days the curve was chosen to fit. What the plan needs is risk, the expected error on new days. On 10,000 new days this curve scores 0.888, nearly 29 times worse. Report a held-out or cross-validated error with its SE: the cubic, which CV picks, scores 0.114 on new days with an SE of 0.0016.

R

risk: a curve's expected squared error on new days

f

a candidate curve: a rule from time of year to predicted rides

\mathbb{E}

average over new random days (on the right, also over possible seasons)

Y

a new day's rides, in thousands (random: capital letter)

X

that new day's time of year (random)

\hat R_n

empirical risk: the average squared error on the n training days (hat = computed from data)

n

the number of training days (30 this season)

\sum_{i=1}^{n}

add up over training days i = 1 to n

y_i

the rides logged on training day i

x_i

the time of year of training day i

\hat f

the curve the fit picked for its low training error, e.g. the degree-15 polynomial

• Why: the fit picked the curve with the lowest training error, so its training error is the minimum of many noisy scores, and that minimum is biased low (Card 1).
• Next they'll ask: "Where does the remaining error come from?" → Above the noise floor \sigma^2 = 0.09: Family, Sample, Search — the family can't reach the truth, 30 days can't pin down the best member, the optimizer may not find it.

Q3 · Explain it to the CFO

The CFO asks why the team rejected the forecasting model that was "more accurate on last year's data" in favor of a simpler one. She wants an answer with no math. What do you say?

• Concept tested: bias and variance in plain words; the noise floor.
• Intuition: an archer who adjusts after every gust of wind hits last year's targets perfectly and next year's badly.
• Answer: "Every forecast misses for three reasons. Some days are unpredictable — weather, events — and no model fixes that. A model that's too simple misses the same way every year: that's bias, a choice we make. A model that's too flexible bends to last year's random bumps, so its forecast would change a lot if last year had gone slightly differently: that's variance, and we pay for it in data. The 'more accurate' model fit last year's bumps. On days it hadn't seen, it was six times worse than the one we chose."
• Why: expected error = \text{noise} + \text{bias}^2 + \text{variance} — Noise you can't beat, Bias you chose, Variance you paid for (Card 2). Ch 1's picture: bias = aim, variance = steady hand.
• Next they'll ask: "So should we always pick the simplest model?" → No: too simple is bias. Pick the complexity that minimizes error on held-out data (Q9, Q21).

Q4 · Degree 15 nails all 30 days

An analyst shows a degree-15 polynomial that passes almost through every one of this season's 30 days (training MSE 0.031) and wants to ship it for next year's staffing. The curve looks great in the middle of the year. Do you ship it?

• Concept tested: the U-curve; why flexible fits explode at the edges.
• Intuition: a polynomial forced near 30 noisy points spends its spare wiggles where nothing holds it down — between points and past the ends.
• Answer: no. On new days it scores 0.888 on this season and 0.379 on average over 500 seasons, against 0.106 for the cubic. The damage is concentrated at the edges: over 500 seasons its variance is 0.037 on June 20 (mid-data) and 1.61 on March 16 (past the last training day), 44 times larger. Fitting the training data closely is not the goal.
• Why: test error is a U in flexibility: it falls while flexibility captures real shape (degree 0 → 3: 0.582 → 0.106), then rises as it fits noise (degree 15: 0.379; degree 18: 8.54).
• Next they'll ask: "How would you pick the degree?" → Cross-validation (Q21, Q27). "Can you keep degree 15 and tame it?" → Ridge brings it to 0.137 (Q8, Q17), but the cubic still wins (Q36).

Q5 · Which model is underfitting?

Two ride models are on the dashboard. Model A: training MSE 0.28, test 0.29. Model B: training 0.043, test 0.38. The noise floor is 0.09. Ops asks which one to fix first, and how.

• Concept tested: diagnosing bias vs variance from two numbers.
• Intuition: A is a ruler laid across a wave — consistently wrong; B is a scribble through every point — right on the days it saw, wrong elsewhere.
• Answer: A underfits (high bias): training error is far above the floor and the gap is tiny, so give it a richer family or better features (a seasonal term). B overfits (high variance): training error is below the floor and the gap is 0.34, so restrain it (a penalty, a lower degree) or give it more data. These are the season's degree 1 and degree 15. Fix B first only if you must keep its family; otherwise replace both with the cubic (0.106).

• Why: train tells bias, gap tells variance. Training error mostly measures how far the family is from the truth; the gap measures how much the fit chased this sample's noise (Card 3).
• Next they'll ask: "Training error below the noise floor — is that possible?" → Yes, and it's a warning sign: the fit has absorbed noise. For least squares, \mathbb{E}[\text{train MSE}] = \sigma^2(1 - p/n) (Q15).

Q6 · $200k for 10× more days

A data vendor offers ten times more labeled days of ride data for $200k. Your current model is a straight line in time of year. Its learning curves — training and validation error against the number of training days — are flat and meet at about 0.29. Do you buy?

• Concept tested: reading learning curves; more data cures variance, never bias.
• Intuition: a ruler can't follow a wave no matter how many points of the wave you show it.
• Answer: no — not for this model. Curves that meet high above the noise floor mean high bias. The line's floor is exact: \sigma^2 plus the \text{bias}^2 of the best possible line through a full sine wave, 0.09 + 0.5 - 3/\pi^2 = 0.286. Ten times more data moves the line's test error from 0.290 (n = 30) to 0.284 (n = 1{,}000). Spend on a better feature instead: the cubic on today's 30 days already scores 0.107.

• Why: more data shrinks variance, roughly \sigma^2\cdot \text{df}/n, and leaves bias untouched (Card 18). Hook: more data cures variance, never bias.
• Next they'll ask: "And for a degree-12 model?" → Its gap is still closing: 0.146 at n = 30, 0.091 at n = 1{,}000, overtaking the cubic from n = 200. There the data is worth pricing — fit a power law first (Q39).

Q7 · Short list or best forecast?

The expansion team wants "the 5 station features that drive demand" for a new city. Ops wants the most accurate ride forecast per station. You have 80 candidate descriptors (population within 250 m, 500 m, 1 km, transit stops, offices…) on 100 stations, in correlated groups. One model or two, and which penalty?

• Concept tested: ridge vs lasso vs elastic net; prediction vs interpretation.
• Intuition: ridge turns every knob down a little, lasso switches most of them off, elastic net switches off whole groups together.
• Answer: two goals, so probably two outputs. For the forecast, the penalty with the best held-out error: on the stations elastic net scores 4.67, lasso 4.78, ridge 5.05 (irreducible 4.0). For the short list, lasso or elastic net, but report groups and selection frequencies, not five names: with correlated descriptors lasso picks one member of a group almost arbitrarily, and 100 bootstrap refits gave 100 different selected sets.
• Why: L1 kills, L2 shrinks (Ch 1); among correlated features, lasso picks, ridge spreads, elastic net groups (Cards 9–10).
• Next they'll ask: "Can I put p-values on the five survivors?" → Not naive ones (Q31).

Q8 · The regularization slider

An internal forecasting tool exposes "regularization strength" as a slider on a degree-15 ride model. The PM asks: "What happens at zero, what happens at the maximum, and where should I leave it?"

• Concept tested: the \lambda path: bias rises, variance falls.
• Intuition: the slider sets how much the model trusts this season's bumps.
• Answer: at 0 the model is plain least squares (test 0.379, almost all variance). As \lambda \to \infty every coefficient goes to zero and it predicts the season average every day (test 0.582, all bias). In between, test error is a U: on the season the bottom is \lambda = 5.62, test 0.137. Leave it where cross-validation says, or at the largest \lambda within one SE of that minimum.

• Why: per direction, ridge's \text{bias}^2 rises from 0 and its variance falls from \sigma^2/d^2 (Card 7).
• Next they'll ask: "Why does least squares at \lambda = 0 have 16 df with only 15 features?" → The unpenalized intercept adds one (Q18).

Q9 · A new best model every seed

A teammate chose the polynomial degree with a single random 80/20 split of this season's 30 days. Rerunning with a different seed changed the "best" degree from 3 to 15. She asks which split to trust.

• Concept tested: the variance of a validation estimate; K-fold CV.
• Intuition: judging a model on six days is like judging a basketball player on six shots.
• Answer: neither. Six validation days give a very noisy score, so the argmin jumps around: over 20 seeds the 80/20 split chose 10 different degrees, from 3 to 15. Use K-fold CV, which scores every day once and averages: reshuffling 10-fold CV 20 times picked degree 3 eighteen times and degree 5 twice. With so few rows, also prefer the simplest model within one SE (Q21).
• Why: this is the second of the 3 Ls of a CV number — Lots of noise (Card 11).
• Next they'll ask: "Is the K-fold number itself reliable?" → Less noisy, but its naive SE is too small, because folds share training rows (Q24, Drill 17).

Q10 · Tuned on the test set

A colleague tried 10 values of \lambda, picked the one with the best test-set score, and put that score in the launch doc as "expected production error". Is that number honest?

• Concept tested: the Single-use promise; selection optimism.
• Intuition: if you take the best of ten noisy measurements, you've partly selected the luckiest.
• Answer: no. Once the test set chose \lambda, its score is a training number for that choice and is biased low. Even if all ten settings were equally good, with true error 0.200 and independent validation noise SD 0.010, the winner reports 0.185 on average (Drill 14). Split the roles: tune on validation (or CV), test once on untouched data — or nested CV.

\mathbb{E}

average over many repeats of the validation noise

\min_{k \le K}

the smallest over the K settings tried

\hat R_k

setting k's test-set score: its true error plus noise

K

the number of \lambda values tried (10)

\mu

the true error every setting shares (0.200)

\sigma_v

the SD of the noise in each score (0.010)

\max_{k \le K}

the largest over the K settings

Z_k

setting k's noise in standard units: a standard normal draw

• Why: a test score predicts future error only if it was used once: the third of Same, Separate, Single-use. The bias is the Lucky minimum (Card 11).
• Next they'll ask: "What if you try 500 configurations?" → The optimism grows like \sigma_{v}\cdot \sqrt{2 \ln K} (Q35). Hook: tune inside, test outside (Q23).

Q11 · "Isn't early stopping a hack?"

An MLE stops gradient descent when validation loss starts rising and restores the best checkpoint. A skeptical PM calls it "ad hoc — you're just quitting early". Is it principled?

• Concept tested: early stopping as regularization.
• Intuition: gradient descent learns the strong, well-measured patterns first and the weak, noisy ones last; stopping early keeps the first and skips the second.
• Answer: it's principled. For a linear model, stopping gradient descent from zero after t steps multiplies each data direction by 1 - (1 - \eta d^2)^t, a filter shaped like ridge's d^2/(d^2 + \lambda) with \lambda roughly 1/(\eta t). On the season's degree-15 fit, test error bottoms at 0.130 after 3 steps, against ridge's best 0.137 and least squares' 0.379. With a patience rule on a 20/10 split, the restored weights scored 0.159 on new days, against 2.07 had training run to convergence.
• Why: strong directions (large d^2) converge in a few steps; weak ones barely move (Card 13). It's the S in PANDAS: Penalty, Augmentation, Noise, Dropout, Averaging, Stopping early.
• Next they'll ask: "So it is ridge?" → Same shape and nearly the same minimum, but the 1/(\eta t) map is rough: the two test curves differ by up to 0.10 at matched t (Drill 7).

Q12 · Great in summer, bad in winter

The ride model was trained on a pilot that logged 80% of its days in spring and summer. It forecasts summer well and misses badly from November to February. What's happening, and what do you do?

• Concept tested: covariate shift; extrapolation; importance weighting.
• Intuition: the model studied for the summer exam and is being tested all year.
• Answer: covariate shift: the distribution of x (which days you see) differs between training and deployment, while the rule p(y \mid x) is the same. It bites when the model is misspecified: a line spends its accuracy where the training days are. Reweight each training day by w(x) = p_{\text{test}}(x)/p_{\text{train}}(x): 0.625 for summer days, 2.5 for winter days. At n = 200 that moves the line from 0.314 to 0.293 (no-shift reference 0.291). The cost: an effective sample of 64% of n, so 30 pilot days count as 19.2.

w

the importance weight of a training day: how much more common such days are at deployment

x

a day's time of year

p_{\text{test}}

how common days like x are at deployment (density 1: the whole year evenly)

p_{\text{train}}

how common they were in the pilot: 1.6 in spring and summer, 0.4 in autumn and winter

• Why: weighted training loss has the same expectation as test loss (Card 17). It's the Same promise broken; hook X moves, Y moves, the rule moves — this is X moving.
• Next they'll ask: "What if the pilot had no winter days at all?" → No weight can fix missing support; it's extrapolation (the line's winter error is 1.94, degree 5's median 8,349). Collect winter data (Q33).

Level 2 — Onsite core

Q13 · Derive the decomposition

Whiteboard, first onsite round. "A new point at x_0 has y = f(x_0) + \varepsilon. Your fitted \hat{f} is random because the training set is. Derive the expected squared error at x_0 as \text{noise} + \text{bias}^2 + \text{variance}. Tell me exactly where each cross term goes and what assumption kills it."

• Concept tested: the bias–variance decomposition (Card 2).
• Intuition: the miss is three arrows laid end to end — the day's luck, the procedure's aim, this fit's wobble — and the three arrows are uncorrelated.
• Answer: write \bar{f} = \mathbb{E}[\hat{f}(x_0)] and add and subtract f and \bar{f}:

y

the rides on a new day at x_0

\hat f

the fitted curve's prediction at x_0: random, because the training set is

f

the true curve's value at x_0

\bar f

the average prediction over all training sets, \mathbb{E}[\hat f(x_0)] (bar = average): where the procedure aims

\varepsilon

the new day's noise: mean 0, variance \sigma^2, independent of the training data

\mathbb{E}

average over training sets and the new day's noise

\sigma^2

noise variance: the new day's luck no curve can beat (0.09 on the season)

The fourth term is zero because \mathbb{E}[\varepsilon] = 0. The fifth is zero because the new day's noise is independent of the training data (so of \hat{f}) and has mean zero. The sixth is zero because f - \bar{f} is a constant and \mathbb{E}[\bar{f} - \hat{f}] = 0 by the definition of \bar{f}. What's left: \sigma^2 + \text{bias}^2 + \operatorname{Var} (\hat{f}(x_0)).

• Why: the only real assumption is that the test noise is independent of the training set. Expectations are over both the training draw and the new noise (Card 2).
• Next they'll ask: "Does this hold for 0-1 loss?" → No additive version (Q38). "Average over x_0?" → Integrate each term over the test distribution of x: that's the season's table.

Q14 · Six days, by hand

The interviewer writes the toy on the board: six training days at months m = 1, 3, 5, 7, 9, 11, true curve f(m) = \sin(2\pi m/12), noise \sigma = 0.3, design fixed. "Compute the straight line's \text{bias}^2, variance and expected squared error at July 20 (m = 4). Then tell me how the cubic compares."

• Concept tested: the decomposition computed exactly for a linear smoother.
• Intuition: for least squares, the average fit is just the fit to the true values, so bias needs no simulation.
• Answer: fit the line to the true values (0.5, 1, 0.5, −0.5, −1, −0.5): slope -12/70 = -0.1714, intercept 1.0286, so \mathbb{E}[\hat{y}(4)] = 0.343 against the truth 0.866: bias −0.523, \text{bias}^2 0.274. The variance comes from Ch 1's prediction-variance formula with centered months (S_{\text{mm}} = 70):

\text{Var}

variance over fresh noise on the six fixed training days

\hat y(4)

the fitted line's prediction at month 4 (July 20)

n

the number of training days, 6

\bar m

the average training month, 6 (bar = average)

S_{mm}

the spread of the months, \sum (m_i - \bar m)^2 = 70

\hat y

the line's prediction for July 20

• Why: \hat{y} = Hy is linear in y, so \mathbb{E}[\hat{y}] = Hf and \operatorname{Var} (\hat{y}_0) = \sigma^2\cdot x_0^\top(X^\top X)^{-1}x_0 (Card 2; Ch 1, Card 6). Degree 5 aims perfectly and still loses to the cubic: it pays more variance than the cubic pays in bias.
• Next they'll ask: "On this season the cubic and degree 5 predicted 0.791 and 0.801 for July 20 — nearly the same. Why prefer the cubic?" → One season's prediction is one draw; the decomposition is about the procedure over many seasons.

Q15 · How optimistic is training error?

A degree-9 polynomial (10 coefficients) fit to 30 days reports training MSE 0.061; you know \sigma = 0.3 and the model's bias is negligible. "What test MSE do you expect? How do Mallows' Cp and AIC use that gap — and does the answer change if new days come at random times?"

• Concept tested: training optimism for least squares (Card 3); Cp and AIC as rent (Card 4).
• Intuition: each fitted coefficient bends the curve toward the training noise, making old days look better and new days worse by the same amount.
• Answer: for a correct model with fixed design and new noise at the same x's:

\mathbb{E}

average over fresh noise at the same 30 days

\sigma^2\Big(1 - \frac{p}{n}\Big)

expected training MSE: the noise variance \sigma^2 (0.09) shrunk by the share p/n the fit absorbs

\sigma^2\Big(1 + \frac{p}{n}\Big)

expected test MSE at the same days: the noise variance \sigma^2 inflated by p/n

p

the number of coefficients (10 for degree 9)

n

the number of training days (30)

The season simulation gives 0.0612 for training and 0.124 on random new days. Cp adds the rent back: \text{Cp} = \text{train MSE} + 2\sigma^2 p/n = 0.061 + 0.060 \approx 0.121. AIC charges the same 2 per df on the log-likelihood scale. For Gaussian random x, the test error is larger, \sigma^2(1 + p/(n - p - 1)) without an intercept — it blows up as p \to n (Drill 2, Q30).

• Why: \mathbb{E}[\text{RSS}] = (n - p)\sigma^2 (Ch 1, Card 7), and test error adds the average prediction variance \sigma^2p/n (Card 3).
• Next they'll ask: "Do Cp and AIC pick the same model?" → Not always. On this season Cp (\sigma^2 known), AICc and BIC pick degree 3, but plain Gaussian AIC picks degree 15: with \hat{\sigma}^2 estimated from a shrinking RSS, a rent of 2 per df is too cheap at n = 30. Use AICc or CV at small n.

Q16 · "kNN has no parameters, so it can't overfit"

An interviewer says: "k-nearest-neighbors regression has no fitted parameters, so it can't overfit. Agree?" Use 1-NN on the 30-day season to answer with numbers.

• Concept tested: degrees of freedom as covariance; kNN bias and variance (Cards 4–5).
• Intuition: 1-NN copies each training day's value exactly, noise and all — as flexible as a model can be.
• Answer: disagree. Capacity is how hard the fit chases the noise, \text{df} = \sum \operatorname{Cov} (\hat{y}_i, y_i)/\sigma^2. For kNN, \hat{y} = Sy with 1/k in each row's neighbor columns, so \text{df} = \operatorname{tr} (S) = n/k: 1-NN uses all 30 df (Monte Carlo: 30.05). Its training error is 0 and its test error 0.183; k = 5 (6 df) scores 0.124.

\sigma^2

noise variance: day-to-day luck (0.09 on the season)

\sum_{i=1}^{n}

add up over the n training days

\text{Cov}

covariance over noise redraws: how much the fit at day i moves with that day's rides

\hat y_i

kNN's fitted value at training day i

y_i

the rides observed on day i

\text{tr}

trace: the sum of a matrix's diagonal entries

S

kNN's smoother matrix: 1/k in each row's k neighbor columns, so \hat{\mathbf y} = S\mathbf y

k

the number of neighbors averaged

\hat f(x_0)

kNN's forecast for day x_0

\text{Var}

variance over noise redraws on the same 30 days

• Why: the optimism formula 2\sigma^2\cdot \text{df}/n holds for any fit: at k = 1 it's 2 \times 0.09 \times 30/30 = 0.18 — the full gap between training error 0 and the in-sample test error 2\sigma^2 (Card 4). Hook: Count it, Correlate it, Shatter it — counting fails, the covariance works.
• Next they'll ask: "How do bias and variance move with k?" → Variance \sigma^2/k falls; bias grows as neighbors reach further from x_0. Best k at June 20 is 5 (Drill 1).

Q17 · Ridge in the SVD basis

"Derive the ridge solution, then rotate into the singular-value basis of X and show me what ridge does to each direction. Which directions does it shrink most, and why are those the right ones to shrink?"

• Concept tested: ridge as a spectral filter (Card 6).
• Intuition: ridge keeps what the data measured firmly and discounts what it barely measured.
• Answer: minimizing \lVert y - Xw\rVert^2 + \lambda \lVert w\rVert^2 gives (X^\top X + \lambda I)\hat{w} = X^\top y. With X = UDV^\top:

\hat{\mathbf w}_\lambda

ridge's vector of weights at penalty \lambda (bold = vector of all of them)

X

the design matrix: one row per training day, one column per feature

(X^\top X + \lambda I)^{-1}

the inverse (it undoes the matrix) of the penalized cross-product matrix

X^\top X

the features' cross-products; diagonal in the SVD basis

X^\top

the transpose of X: rows become columns

I

the identity matrix: 1s on the diagonal, 0s elsewhere

\lambda

the ridge penalty strength

\mathbf y

the vector of all training days' rides

\sum_j

add up over the data's directions j, one per singular value

\mathbf v_j

direction j in weight space (column j of V)

\mathbf u_j^\top\mathbf y

how much of the rides vector points along direction j

\mathbf u_j

direction j across the training days (column j of U)

d_j

singular value j: how much the data spread along direction j

Least squares has factor 1 everywhere; ridge multiplies direction j by d_j^2/(d_j^2 + \lambda), which is \approx 1 when d_j^2 \gg \lambda and \approx 0 when d_j^2 \ll \lambda. It shrinks the weak (small-d) directions most. Those are the right ones: the least-squares coefficient along direction j has variance \sigma^2/d_j^2, so weak directions are mostly noise. On the season's degree-15 design, 13 directions have d^2 between 26.7 and 47.6 and two have 1.47 and 0.59; at \lambda = 5.62 the strong ones keep 83–89% and the weak ones 21% and 10%.

• Why: in the V basis X^\top X is diagonal, so ridge decouples into one-weight problems. Hook: shrink the weak directions most.
• Next they'll ask: "Compare PCR." → PCR's factor is 1 or 0 per direction (a hard threshold); ridge is its soft version. "Bias of ridge?" → Q8, Card 7.

Q18 · "df = 7.8" on the model card

A model card reads: "Ridge on 15 polynomial features plus intercept, \lambda = 31.6, effective degrees of freedom 7.8." A reviewer asks how 15 features can have 7.8 degrees of freedom, and how you'd check the number without trusting the formula.

• Concept tested: effective df of ridge (Cards 4, 6).
• Intuition: a penalized coefficient that can barely move doesn't count as a full degree of freedom.
• Answer: ridge is a linear smoother, so \text{df} = \operatorname{tr} (S): the shrink factors summed, plus 1 for the unpenalized intercept. On the season's degree-15 design at \lambda = 31.6, each of the 13 strong directions keeps about half (0.46–0.60) and the two weak ones under 5%, totalling 7.80.

\sum_{j=1}^{15}

add up over the 15 directions of the degree-15 design

To check it independently, use the covariance definition: simulate many noise draws on the same design, refit, and compute \sum_i \operatorname{Cov} (\hat{y}_i, y_i)/\sigma^2. At the season's best \lambda = 5.62 the formula gives 12.44 and Monte Carlo 12.45.

• Why: \operatorname{Cov} (Sy, y) = S\sigma^2, so the covariance df equals the trace exactly (Card 4).
• Next they'll ask: "Lasso's df?" → The expected number of non-zero coefficients (Zou, Hastie and Tibshirani, 2007), even though lasso is not linear in y.

Q19 · Why exact zeros? The KKT view

"Without drawing the diamond: derive the condition under which lasso sets a coefficient exactly to zero, and the smallest \lambda at which every coefficient is zero. Use the loss \lVert y - Xw\rVert^2 + \lambda \lVert w\rVert_1."

• Concept tested: subgradient optimality for lasso (Card 9).
• Intuition: a weight leaves zero only when its feature's pull on the current residual is stronger than the fixed price \lambda/2.
• Answer: the subgradient of |w| at 0 is any value in [−1, 1], so the KKT conditions are:

\mathbf x_j^\top(\mathbf y - X\hat{\mathbf w})

how strongly station feature j lines up with the current residual (actual minus fitted rides)

\lambda

the lasso penalty strength

\text{sign}

the sign: +1 if positive, -1 if negative

\hat w_j

lasso's weight on station feature j

At \hat{w} = 0 the residual is y, so everything stays at zero while \lambda \ge \lambda_{\max} = 2\cdot \max_j\lvert x_j^\top y\rvert. On station replicate 0, \lambda_{\max} = 512 (sklearn alpha 2.56, because sklearn divides the loss by 2n). With orthonormal features this becomes soft-thresholding:

z_j

feature j's least-squares coefficient (with orthonormal features)

\big(|z_j| - \tfrac{\lambda}{2}\big)_+

shrink \lvert z_j\rvert by \lambda/2 but stop at 0 (the _+ keeps only positive values)

\mathbf x_j^\top\mathbf y

how strongly feature j lines up with the rides vector

• Why: the kink at zero gives a whole interval of subgradients, so zero can be optimal for a range of pulls; a smooth penalty (ridge) has subgradient 0 only at w = 0, so zero is optimal only when the pull is exactly zero.
• Next they'll ask: "z = (3, 0.8, -1.5), \lambda = 2?" → (2, 0, -0.5), \lambda_{\max} = 6 (Drill 10).

Q20 · Lasso keeps changing its mind

Bootstrapping the station lasso, the "population within 250 m / 500 m / 1 km" block gets a different winner each time; the VP wants to know which radius is "the" driver so the site team can use it. What's going on, and what do you deliver instead?

• Concept tested: lasso instability under correlation; elastic net's grouping; stability selection.
• Intuition: three nearly identical witnesses — lasso calls one to the stand at random and dismisses the others.
• Answer: with correlation 0.8 within a block, the features carry nearly the same information, and the L1 penalty prefers one large weight to several small ones, so which member wins depends on the noise. On station replicate 0, 100 bootstrap refits gave 100 different selected sets; true feature 1 was selected 11% of the time by lasso and 53% by elastic net. Averaged over the 10 true features: 76% (lasso) vs 95% (elastic net); over the 70 noise features, 9% vs 14%. Deliver groups ("population density near the station matters; the radius can't be separated"), selection frequencies, and elastic net for prediction.

\mathbf w

the vector of weights on the 80 station descriptors

\lVert\mathbf y - X\mathbf w\rVert^2

the sum of squared misses over the stations

\lambda_1

strength of the L1 (lasso) penalty

\lVert\mathbf w\rVert_1

L1 norm: the sum of the absolute weights

\lambda_2

strength of the L2 (ridge) penalty

\lVert\mathbf w\rVert_2^2

squared L2 length: the sum of the squared weights

\hat w_i

elastic net's fitted weight on feature i

\hat w_j

elastic net's fitted weight on feature j

\lVert\mathbf y\rVert_1

the sum of the absolute values of the rides vector

\rho_{ij}

the correlation between features i and j (0.8 inside a block)

• Why: the L2 part makes the objective strictly convex and gives correlated features similar weights — the grouping effect (Zou and Hastie, 2005; Card 10). Lasso picks, ridge spreads, elastic net groups.
• Next they'll ask: "A formal version of the frequencies?" → Stability selection: refit on many half-samples and keep features selected above a threshold (Meinshausen and Bühlmann, 2010). "Two identical copies?" → Drill 11.

Q21 · Read this CV curve

You're shown one season's 10-fold CV curve over polynomial degree, with SE bars: the minimum is 0.0737 at degree 5 (SE 0.0149); degree 3 scores 0.0879 (SE 0.0136). (This is season replicate 13; on "this season", replicate 0, the minimum and the 1-SE rule both land on degree 3.) "Which degree do you ship, and why?"

• Concept tested: the 1-SE rule; the SE of a CV estimate.
• Intuition: a lead smaller than the measurement's own wobble isn't a lead.
• Answer: degree 3. Its CV error, 0.0879, is within one SE of the minimum (0.0737 + 0.0149 = 0.0886), so the two are statistically indistinguishable, and degree 3 is simpler: lower variance, easier to defend. On that season the true test errors are 0.1076 (degree 3) and 0.1084 (degree 5), so nothing was lost. Over 500 seasons, plain 10-fold picks degree 3 in 267 and the 1-SE rule in 436, and the mean test error of the chosen model falls from 0.1121 to 0.1087.
• Why: SE at each setting = SD of the K fold errors / \sqrt{K}. 1-SE: the simplest model within one standard error of the best (Hastie, Tibshirani and Friedman; Card 11). "Simplest" must be defined in advance: lowest df, lowest degree or largest \lambda.
• Next they'll ask: "Is that SE right?" → Too small: the fold errors are correlated because training sets overlap (Drill 17).

Q22 · Scaler and selector before the split

*Code review: a pipeline fits StandardScaler and SelectKBest(k=10) on all 50 rows (1,000 candidate features), then calls cross_val_score on a linear model. The CV R^2 is 0.46. "What's wrong, and which of the two mistakes is worse?"*

• Concept tested: leakage inside CV; supervised vs unsupervised preprocessing.
• Intuition: the selector looked at the answers of the held-out fold before the exam.
• Answer: both steps learned from the held-out rows, but SelectKBest is far worse because it is supervised: it chose features by their correlation with this y, including the held-out rows. The chapter reproduces it on pure noise (no fit can beat MSE 1.0): screening on all rows gives a CV MSE of 0.54 — "R^2 = 0.46" from nothing, below 1.0 in 99% of replicates — while screening inside each training fold gives the honest 1.43. The scaler barely matters: refitting it on all rows changes the least-squares CV curve by exactly 0 and ridge's by about 0.003. Put both inside a Pipeline.
• Why: CV is honest only if every step that learns from data runs inside the training folds (Ambroise and McLachlan, 2002). Least squares with an intercept is invariant to affine rescaling of columns, so an unsupervised scaler can't leak into it. Hook: the 3 Ss — Scaler inside, Shuffle explicitly, Same \lambda \ne same alpha.
• Next they'll ask: "Other supervised steps that leak?" → Target encoding, supervised imputation, oversampling (SMOTE) before splitting, and choosing a model on the same folds you report.

Q23 · Is the tuned CV score honest?

*The team tunes polynomial degree × ridge alpha with GridSearchCV (96 configurations, 10-fold) on this season and reports best_score_ = 0.122 as the expected production error. "Is that number biased? Which way? How would you report it?"*

• Concept tested: the Lucky minimum vs Less data; nested CV.
• Intuition: you're reporting the best of 96 noisy grades — but each grade came from a model trained on less data.
• Answer: two biases pull in opposite directions. Less data (each fit sees 90% of the rows) makes CV pessimistic; the Lucky minimum (best of many) makes it optimistic. Which wins depends on how many configurations you search. On the stations, a 19-value \lambda search reports best_score_ 0.09 ± 0.06 above the true error (300 replicates), while a 760-configuration search reports it 0.83 ± 0.15 below (40 replicates). Report nested CV: the whole search inside each outer training fold, scored on the outer fold. It stayed honest to slightly pessimistic in both searches (+0.19, +0.25). Here: best_score_ 0.122, test on new days 0.113, nested 0.160 (outer folds 0.010–0.385, so with 30 rows any single number is rough).
• Why: the outer fold never influenced any choice (Card 11). The 3 Ls: Less data, Lots of noise, Lucky minimum. Tune inside, test outside.
• Next they'll ask: "Which model do you ship after nested CV?" → Rerun the inner search on all the data and ship that; nested CV estimates the procedure, not one fitted model.

Q24 · LOOCV or 10-fold?

"Which K would you use for 30 days of data? For 30 million rows? And for a ridge model specifically — does that change your answer?"

• Concept tested: the bias, variance and cost of K; the LOO shortcut and GCV (Cards 11–12).
• Intuition: bigger K trains on more data (less pessimistic) but makes nearly identical fits (noisier average, more cost).
• Answer:

30 days: 10-fold or LOO. 30 million rows: one large holdout already has a tiny SE; K-fold is for scarce data. For ridge at fixed \lambda, LOO is free — fit once and use the leverages (on the season it matches brute-force refits to 3 \times 10^{-11}):

\hat y_{(-i)}

the prediction for day i from a fit that left day i out

\hat y_i

the prediction for day i from the fit on all days

S_{ii}

day i's leverage: the diagonal entry of S, its pull on its own fit

\frac1n

divide by n: with the sum, the average over the days

\sum_i

add up over every day i

S

ridge's smoother matrix, mapping observed rides to fitted rides

• Why: CV at K estimates the risk of a model trained on n(K - 1)/K rows. For \text{error}(n) = 1 + 20/n at n = 100: truth 1.200, 5-fold 1.250, 10-fold 1.222, LOO 1.202 (Drill 15).
• Next they'll ask: "GCV is cheaper still — use it?" → Only with similar leverages. On this season GCV picks \lambda = 0.32 for the degree-15 ridge (test 0.52) because edge days have far more leverage than \operatorname{tr} (S)/n; LOO and 10-fold pick 5.62 (test 0.137).

Q25 · Noise made validation better

An engineer adds Gaussian jitter with SD 0.1 to every standardized input on every epoch, and validation error on a linear model drops. A teammate says it's luck. "Prove what this is equivalent to for a linear model with sum-of-squares loss, and give the \lambda for 30 rows."

• Concept tested: noise injection = ridge (Card 14).
• Intuition: a model that must give the same answer for slightly blurred inputs can't afford steep weights.
• Answer: replace X by X + \Delta with \Delta's entries i.i.d. N(0, s^2), and take the expectation over \Delta:

\mathbb{E}

average over the random jitter \Delta

\Delta

the jitter: fresh N(0, s^2) noise added to every input

\mathbf w

the linear model's weight vector

\lVert \mathbf y - X\mathbf w\rVert^2

the ordinary sum of squared misses on the clean inputs

s^2

the variance of the jitter (0.01 for SD 0.1)

\lVert\mathbf w\rVert^2

squared length of the weights: the sum of squared weights

So jitter is ridge with \lambda = n\cdot s^2 = 30 \times 0.01 = 0.3 (\lambda = s^2 for the mean loss — say which). On the season's degree-15 design, least squares on 200 jittered copies per row and ridge with \lambda = 0.3 give coefficients within 0.003 of each other and test errors of 0.534 and 0.533 (Drill 6).

• Why: the cross term vanishes because \Delta has mean zero; \mathbb{E}[\lVert\Delta w\rVert^2] = \textstyle\sum_i \mathbb{E}[(\delta_i^\top w)^2] = n s^2\lVert w\rVert^2 (Bishop, 1995).
• Next they'll ask: "Is augmentation the same thing?" → Only for isotropic Gaussian noise on a linear model. Augmentation encodes a specific invariance; a wrong invariance adds bias.

Q26 · Dropout on a linear model

Research-flavored onsite: "Apply inverted dropout to a linear regression's inputs: each input is kept with probability q and divided by q. What objective are you minimizing in expectation? What's the equivalent ridge \lambda for standardized features on 100 rows with q = 0.8?"

• Concept tested: dropout = per-feature ridge (Card 15).
• Intuition: randomly hiding features punishes leaning hard on any one of them, most for the features with the most energy.
• Answer: with \tilde{x}_{ij} = x_{ij}\delta_{ij}/q, \delta \sim \text{Bernoulli}(q), each \tilde{x}_{ij} has mean x_{ij} and variance x_{ij}^2(1 - q)/q, independently. So:

\mathbb{E}

average over the random dropout masks

\tilde X

the inputs after inverted dropout: each kept with probability q and divided by q

q

keep probability: the chance each input survives (0.8)

\sum_j

add up over features j

\sum_i

add up over training rows i

x_{ij}

feature j's value on row i

w_j

the weight on feature j

With standardized features on 100 rows, \sum_ix_{ij}^2 = 100, so \lambda = (0.2/0.8) \times 100 = 25 for every feature (Drill 19). On the season's design the Monte Carlo dropout objective (8.769 ± 0.006) matches the formula (8.758).

• Why: \mathbb{E}[(y_i - \tilde{x}_i^\top w)^2] = (y_i - x_i^\top w)^2 + \operatorname{Var} (\tilde{x}_i^\top w), and the variance sums over independent masks (Wager, Wang and Liang, 2013).
• Next they'll ask: "What must you do at inference?" → Turn dropout off (model.eval()); inverted dropout already rescaled during training, so no correction is needed.

Q27 · Code the U-curve

"You have 15 minutes and NumPy only. Here are 30 days, one every 12 days, with y = \sin(2\pi x) + N(0, 0.3^2). Fit degrees 0–15 by least squares, print train and test MSE on 10,000 new days, then choose the degree by 10-fold CV."

• Concept tested: writing the U-curve and K-fold CV from scratch.
• Intuition: features → fit → score on held-out rows → rotate.
• Answer:

import numpy as np

rng = np.random.default_rng(404)
f = lambda x: np.sin(2 * np.pi * x)
x_test = rng.uniform(0, 1, 10_000); y_test = f(x_test) + rng.normal(0, 0.3, 10_000)
x = (np.arange(1, 31) - 0.5) / 30
y = (f(x) + rng.normal(0, 0.3, (500, 30)))[0]                 # this season

def poly(x, d):                                                # Legendre: same fits as raw powers, well conditioned
    return np.polynomial.legendre.legvander(2 * x - 1, d)

def fit(x, y, d):
    w = np.linalg.lstsq(poly(x, d), y, rcond=None)[0]          # QR/SVD under the hood, never inv(X'X)
    return lambda xn: poly(xn, d) @ w

for d in range(16):
    m = fit(x, y, d)
    print(d, round(np.mean((y - m(x)) ** 2), 4), round(np.mean((y_test - m(x_test)) ** 2), 4))

folds = np.array_split(np.random.default_rng(0).permutation(30), 10)   # shuffled folds
def cv(d):
    return np.mean([np.mean((y[te] - fit(np.delete(x, te), np.delete(y, te), d)(x[te])) ** 2) for te in folds])
print("10-fold CV picks degree", int(np.argmin([cv(d) for d in range(16)])))

It prints this season's U (degree 0: 0.754 / 0.583; 3: 0.092 / 0.114; 9: 0.070 / 0.136; 15: 0.031 / 0.888) and picks degree 3.

• Why it's written this way: shuffled folds (rows sorted by day would make every fold extrapolate); lstsq instead of inverting X^\top X; Legendre features because raw powers of x \in[0, 1] are nearly collinear at degree 15 (Section 12).
• Next they'll ask: "Now measure \text{bias}^2 and variance." → Refit on many simulated seasons at fixed x, average the predictions on a test grid: \text{bias}^2 is the squared gap between the average and f, variance the spread (Section 12; Q13).

Level 3 — Senior

Q28 · The bigger model generalizes better

A research lead shows a station model with 1,000 descriptors fit to 100 stations: zero training error, and test error 0.351 — better than every model with between 40 and 500 features. "This contradicts everything you just told me about bias and variance. Reconcile it."

• Concept tested: double descent; minimum-norm interpolation; benign overfitting (Card 16).
• Intuition: at p = n there is exactly one way to hit every point, and it's the wildest; with many more features there are infinitely many ways, and the minimum-norm rule picks the calmest.
• Answer: nothing is contradicted; parameter count was the wrong capacity measure. With p > n, least squares via the pseudo-inverse (and gradient descent from zero) returns the smallest-norm interpolator. On the chapter's design — 20 strong descriptors, then 980 weak "embedding" features — the median test error is 0.306 at p = 10 (the classical best), 47.4 at p = n = 100, and falls back to 0.351 at p = 1{,}000. At the threshold the smallest singular value of X is near zero (0.009) and \lVert \hat{w}\rVert explodes (42.8); past it the many weak directions absorb the noise thinly.

\hat{\mathbf w}

the minimum-norm weights: the interpolating fit least squares returns when p > n

X

the design: 100 stations by p descriptors

X^{+}

the pseudo-inverse of X: the best "undo" when X has no true inverse

\mathbf y

the vector of station rides

X^\top

the transpose of X: rows become columns

(XX^\top)^{-1}

the inverse of the n \times n matrix of inner products between stations

\arg\min

the \mathbf w that makes what follows smallest

\lVert\mathbf w\rVert

the length (norm) of a weight vector

\mathbf w

any candidate weight vector

• Why: test error = \sigma^2 + \text{bias}^2 + \text{variance} at every p. Overfitting is benign when the signal lives in a few strong directions and a long tail of weak directions soaks up the noise (Bartlett, Long, Lugosi and Tsigler, 2020). With a flat signal spread over 400 features, every model with p < 100 is worse than predicting the mean (1.25), and the largest model wins (1.09).
• Next they'll ask: "Why does gradient descent find the minimum-norm solution?" → Its iterates are combinations of the rows of X, so they never leave the row space, and the only interpolator there is X^+y (Card 16; WP2).

Q29 · Validation error spikes at one model size

Sweeping the hidden width of a small network, the team sees validation loss spike when the parameter count is roughly the number of training examples, then fall again as width grows. "Diagnose it. What would make the spike disappear — and could adding training data make it worse?"

• Concept tested: the interpolation threshold; ridge and early stopping remove the spike; sample-wise double descent.
• Intuition: the spike is a variance explosion along directions the data barely measures; anything that damps those directions flattens it.
• Answer: it's the interpolation threshold: the model can just barely fit the training set, and the only fit that does is wild. Fixes: a tuned ridge penalty or weight decay, early stopping, or moving away from the threshold. In the chapter's simulation, ridge with \lambda chosen by CV stays between 0.309 and 0.354 for every p from 5 to 1,000, with no spike (Nakkiran, Venkat, Kakade and Ma, 2021). And yes, more data can hurt near the threshold: with p = 100 fixed, the min-norm test error rises from 1.01 at n = 50 to 2.91 at n = 90 and 6.69 at n = 105, then falls to 0.756 at n = 150.
• Why: ridge's factor d^2/(d^2 + \lambda) kills the near-zero singular directions where the variance lives (Card 6). The spike appears along three knobs: model size, sample size, and training time (epoch-wise; Nakkiran et al., 2019).
• Next they'll ask: "Seen it in a polynomial?" → Degree 12's learning curve spikes to 16.3 at n = 15, with 13 coefficients and 15 points (Section 5).

Q30 · Negative test R^2 with 80 features

Least squares on 100 stations × 80 descriptors gives a test R^2 of −0.53 on 10,000 new stations: worse than predicting the mean. Training R^2 looked excellent. "Why so bad? What do you fit instead, and what do you report?"

• Concept tested: random-X optimism near p \approx n; regularization for wide data.
• Intuition: 80 coefficients from 100 rows leave some directions barely measured, and new stations land squarely in them.
• Answer: with random new stations and an intercept, the exact expected test error of least squares is

\mathbb{E}

average over random new stations, random designs and noise

\sigma^2

noise variance (4 on the stations)

n

the number of training stations (100)

p

the number of descriptors (80)

(simulated 22.2; 21.96 ± 0.03 over 3,000 designs), against 14.5 for the mean. At the same 100 stations (fixed X) it would be only \sigma^2(1 + p/n) = 7.24: the blow-up is from new x's. Fit a penalized model and report a held-out error with its SE: ridge 5.05 (27 effective df), lasso 4.78, elastic net 4.67, against an irreducible 4.0.

• Why: the factor p/(n - p - 2) comes from \mathbb{E}[(X^\top X)^{-1}] for Gaussian rows, which explodes as p \to n (Card 3; Drill 2).
• Next they'll ask: "Why ridge rather than dropping features?" → Hard selection among correlated blocks is unstable (Q20); ridge spreads weight and degrades gracefully. If effects are dense, ridge beats lasso (WP1).

Q31 · "Significant drivers" after lasso

An analyst runs lasso with CV on the station data, refits OLS on the surviving features, and presents three "significant drivers (p < 0.01)" to executives. "Are those p-values valid? How big is the problem, and what should she have done?"

• Concept tested: post-selection inference.
• Intuition: the same data chose the features because they looked strong, then graded them on how strong they looked.
• Answer: invalid. In a pure-noise check (true \beta = 0, 400 replicates), lasso-CV selected at least one feature in 44% of runs, and in 39% of all runs at least one survivor had a naive p < 0.05, against the 5% a valid test allows; among selected features, 41% looked significant. Sample splitting fixes it: select on 50 stations, test on the other 50, and the rate drops to 7.5% (2.0% with Bonferroni over the features tested). Other fixes: selective inference conditioned on the selection event (Lee, Sun, Sun and Taylor, 2016), simultaneous intervals over all selections (Berk et al., 2013), stability selection for the list itself.
• Why: naive p-values assume the model was fixed before seeing the data (Ch 1, Q66 raised the warning; this is its size).
• Next they'll ask: "Are the refitted coefficients biased too?" → Yes, away from zero for the survivors (winner's curse); lasso's own estimates are biased toward zero by \lambda/2. Report held-out estimates.

Q32 · CV said 0.12, spring says 0.15

The ride cubic's 10-fold CV error was 0.12 on last season. Since e-bikes launched this spring, its production error has run at about 0.15 for six weeks. "Walk me through the investigation in order, and tell me what you'd find."

• Concept tested: the three promises behind a test score; shift vs leakage vs tuning optimism.
• Intuition: a validation score is a promise with three clauses; find the clause that broke.
• Answer: check Same, Separate, Single-use in turn:

1. Single-use. Was the CV number tuned on? The degree was chosen by the same CV, but the lucky-minimum optimism on the season is small (−0.0055 on average), and CV at degree 3 is if anything pessimistic. Not this.
2. Separate. Any duplicated days, shared groups or future features across folds? Leakage makes CV look too good, so check, but it can't explain a stable model degrading after a date.
3. Same. Did x move, y move, or the rule move? Adversarial validation (train a classifier to tell last season's days from this spring's) tests whether x moved; here the calendar coverage is the same, so it should hover near 0.5 (the chapter gets 0.49 with no shift, 0.65 under the summer-heavy pilot). Residuals plotted over time show a systematic undershoot at the peaks: concept drift. With seasonal swings 1.3 times larger, the cubic's error goes from 0.106 to 0.153, and the extra 0.045 is exactly the new bias, the average of (0.3\cdot \sin 2\pi x)^2.

Fix: retrain on recent weeks with time-weighting, and monitor residuals (Ch 28). Reweighting can't help: p(y | x) changed.

• Why: X moves, Y moves, the rule moves — only the first is fixed by importance weights (Card 17).
• Next they'll ask: "How would you have caught it before six weeks?" → Monitor residuals and input drift on live data; hold out the most recent period as a time-based test (Ch 5).

Q33 · Importance weights of 200

To correct covariate shift, a teammate estimates density-ratio weights with a domain classifier. A handful of training rows get weights above 200, and the weighted model's estimates swing wildly between retrains. "What's happening, and what do you do?"

• Concept tested: effective sample size, weight clipping, overlap, and when not to weight.
• Intuition: a few rows are being asked to stand in for a whole region of deployment the training data barely covers.
• Answer: the effective sample size has collapsed. It counts how many equally weighted rows give the same variance:

\sum_i

add up over training rows i

w_i

training row i's importance weight

n

the number of training rows

\mathbb{E}_{\text{train}}

average over rows drawn like the training data

w

a training row's importance weight

\mu

the size of the shift, in standard deviations

e^{-\mu^2}

the share of the sample that survives: \exp(-\mu^2)

At a shift of 1 SD the ESS is 37% of n; at 2 SDs it's 1.8% — so heavy-tailed that a two-million-draw Monte Carlo estimate of it comes out at 2.2%, itself unreliable (Drill 13). Do, in order: check overlap (if deployment has regions with no training support, no weight fixes it: collect data there); clip or self-normalize the weights; ask whether you need weights at all. Weighting helps a misspecified model and mostly adds variance to a well-specified one: on the season at n = 200 it improves the line in 90% of replicates and degree 5 in only 39%.

• Why: importance weighting is unbiased for the test loss but its variance scales with \mathbb{E}[w^2] (Card 17). Hook: weight by test over train; check the ESS.
• Next they'll ask: "How do you estimate the weights?" → Train a classifier on train vs target rows: w(x) = [P(\text{target} \mid x)/P(\text{train} \mid x)] \times(n_{\text{train}}/n_{\text{target}}), then use importance-weighted CV (Sugiyama, Krauledat and Müller, 2007).

Q34 · Four docks, one day

Each day's ride count is reported by 4 docks with tiny reporting errors, so the 30-day season becomes 120 rows. A random 80/20 row split gives a 1-NN model a test MSE of 0.0019; on truly new days it scores 0.184. "Explain the gap, the general lesson, and the fix."

• Concept tested: leakage as fake generalization.
• Intuition: the test set is full of the training set's twins, so 1-NN is grading itself.
• Answer: a random row split puts a twin of almost every test row (99.7%) in training, and 1-NN predicts each test row with its own twin. The reported error is 98 times too optimistic. Degree 15 looks 6.5 times better than it is (0.069 vs 0.448). Split by group (here, by day): 1-NN's grouped error is 0.207, close to new days' 0.184.

• Why: it breaks the Separate promise. Degree 15's grouped error is heavy-tailed (holding out edge days forces extrapolation from 24 days), so quote its median.
• Next they'll ask: "Where does this happen for real?" → Multiple rows per user, patient or session; near-duplicate images; augmented copies split across folds. Ch 5 owns the leakage taxonomy and grouped and time-based splits.

Q35 · 760 configurations, one winner

A team scored 760 hyperparameter configurations (\lambda \times \text{feature subsets}) on the station data with CV and shipped the one with the best score. In production it underperformed its reported CV error by a wide margin. "How big a bias would you have expected, and how do you prevent it?"

• Concept tested: the winner's curse in model selection.
• Intuition: the more lottery tickets you buy, the luckier your best ticket looks.
• Answer: if K configurations were equally good and their validation scores had independent noise with SD \sigma_{v}, the winner's expected score sits below the truth by \sigma_{v} times the expected maximum of K standard normals (1.54 for K = 10), which grows like \sqrt{2 \ln K}:

\mathbb{E}

average over many repeats of the validation noise

\min_{k \le K}

the smallest over the K configurations

\hat R_k

configuration k's validation score: true error plus noise

K

the number of configurations tried (760 here)

\mu

the true error every configuration shares

\sigma_v

the SD of the noise in each validation score

\max_{k \le K}

the largest over the K configurations

Z_k

configuration k's noise in standard units: a standard normal draw

Real scores are correlated (the configurations share data), so the bias is smaller than the independent bound, but on the stations it was still large: over 40 replicates the 760-configuration search's best_score_ was 0.83 ± 0.15 below the true error, while nested CV was 0.25 ± 0.19 above. Prevent it: nested CV or an untouched holdout for the final number; search fewer configurations or prefer simpler ones (1-SE); a reusable holdout that adds noise to each answer (Dwork et al., 2015).

• Why: it's the Lucky minimum, and it scales with the number of looks (Card 11; Drill 14).
• Next they'll ask: "With only 19 values of \lambda?" → The Less-data pessimism won: best_score_ was 0.09 ± 0.06 above the truth (Q23).

Q36 · Penalty, more data, or a better feature?

You have one quarter. The degree-15 ride model with tuned ridge reaches 0.137 on new days; a plain cubic reaches 0.106; a vendor offers 30× more days; the noise floor is 0.09. "Allocate the quarter, and justify it with the decomposition."

• Concept tested: decomposition-driven decisions; a penalty is not a substitute for the right family.
• Intuition: spend where the error is, and you can only ever win the part above the floor.
• Answer: ship the cubic now. Split each model's error:

The cubic has only 0.016 left above the floor, so that is the most any investment can buy. The vendor's data cuts variance: at n = 1{,}000 the cubic reaches 0.0946, just above its own floor of 0.0944 (\sigma^2 + \text{its bias}^2), and degree 12 reaches 0.091. A better feature (a seasonal sine term) would also remove the cubic's 0.0044 of bias. So: ship the cubic, price the data against at most 0.016 of MSE, and spend the analyst time on features first.

• Why: ridge cut degree 15's variance from 0.288 to 0.035 but had to pay 0.012 in bias to do it; the cubic needs no such trade. More data cures variance, never bias (Card 18).
• Next they'll ask: "How would you estimate those terms on real data?" → Learning curves (bias floor and variance gap) and bootstrap refits (variance), since you never see the truth.

Q37 · Weight decay in Adam

*Training a transformer, one teammate adds \lambda \lVert w\rVert^2 to the loss and uses Adam; another uses AdamW with weight_decay=0.1. "Are these the same regularizer? Which one behaves like ridge, and why does it matter?"*

• Concept tested: L2 penalty vs decoupled weight decay (Ch 2, Section 8).
• Intuition: in Adam, the L2 gradient goes through the same per-weight rescaling as the data gradient, so weights with large gradient history are barely decayed.
• Answer: not the same. With an L2 term, the gradient \lambda w enters Adam's moment estimates and is divided by \sqrt{\hat v}, so the effective decay differs per weight and is weakest exactly where gradients are large. AdamW applies the decay directly to the weights, outside the adaptive step, so every weight shrinks by the same fraction per step — the "shrink toward zero" of ridge:

\mathbf w

the network's weights, updated each step

\eta

the learning rate (step size)

\hat{\mathbf m}

Adam's bias-corrected first moment: a running average of the gradient it is fed

\hat{\mathbf v}

Adam's bias-corrected second moment: a running average of that gradient squared

\nabla L

the gradient of the data loss: one slope per weight; points uphill

\lambda

the L2 penalty strength

\epsilon

a tiny constant that prevents division by zero

\gamma

AdamW's decoupled decay rate (weight_decay)

Here \hat m[\cdot] and \hat v[\cdot] are Adam's bias-corrected moment estimates of whatever gradient they're fed, and \gamma is weight_decay. For plain SGD the two coincide: an L2 term \lambda \lVert w\rVert^2 is weight decay at rate \gamma = 2\lambda. Decoupled weight decay is the standard in transformer and LLM recipes (Loshchilov and Hutter, 2019).

• Why: it's the P in PANDAS, and a penalty is only "ridge" if the optimizer doesn't rescale it.
• Next they'll ask: "Does weight decay matter if you train to convergence on a scale-invariant network?" → It changes the effective learning rate through the weight norm (Ch 17); and in grokking experiments weight decay helps the late jump to generalization (Power et al., 2022).

Q38 · Bias–variance for classification

Research interviewer: "For squared error, test error = \text{noise} + \text{bias}^2 + \text{variance}. Does that hold for 0-1 loss? Can variance ever help a classifier? What survives for log loss?"

• Concept tested: the decomposition beyond squared error.
• Intuition: with a vote, being consistently wrong is worse than being randomly wrong — a wobbly procedure that's wrong on average sometimes lands right.
• Answer: no additive decomposition holds for 0-1 loss. Take a noise-free point whose true class is 1, and a procedure that predicts 1 with probability q over training sets. Its error is 1 - q. If the procedure's majority vote is right (q > \tfrac 12), extra variance pulls q toward \tfrac 12 and raises error. If the majority vote is wrong (q < \tfrac 12: the point is "biased"), extra variance pulls q toward \tfrac 12 and lowers error. Domingos (2000) writes this as \text{loss} = \text{noise} + \text{bias} + c\cdot\text{variance} with c = +1 on unbiased points and −1 on biased ones (two classes); Friedman (1997) showed the same interaction for probability estimates near the boundary. For log loss an analogous decomposition exists, with the "average model" a normalized geometric mean of the predicted distributions (Heskes, 1998).
• Why: squared error is the only common loss where the cross terms vanish exactly (Card 2).
• Next they'll ask: "Then why does bagging help classifiers?" → It cuts variance on points whose majority vote is already right, which is most of them for a decent base model; it can hurt on the biased points (Ch 10).

Q39 · Predict the loss at 64× data

A research lead shows validation loss from three pretraining runs at 1k, 4k and 16k units of data: 0.1632, 0.1316, 0.1158. "Predict the loss at 64k, and tell me when this extrapolation breaks."

• Concept tested: power-law learning curves; scaling laws (Card 18).
• Intuition: each 4× of data buys a fixed fraction of the remaining gap to the floor.
• Answer: fit \text{error}(n) = a + b\cdot n^{-\alpha}. The successive differences are 0.0316 and 0.0158, so each 4× shrinks the excess by 2 = 4^\alpha, giving \alpha = 0.5. The excess at 1k is 2 \times 0.0316 = 0.0632, so a = 0.1632 - 0.0632 = 0.100. At 64k the excess is 0.0632/8 = 0.0079: predicted loss 0.108.

e

validation loss as a function of the amount of data

n

the smallest run's data size (1k units)

\alpha

the power-law exponent: how fast extra data pays off

It breaks when the assumptions behind the curve break: data repeated across epochs instead of fresh, a change in data mixture or distribution, a model too small to use the data (the floor a depends on model size), or the irreducible floor reached. In LLMs these are scaling laws: Kaplan et al. (2020) reported loss falling as power laws in parameters, data and compute with exponents of about 0.076, 0.095 and 0.050; Hoffmann et al. (2022) fit L(N, D) = E + A/N^\alpha + B/D^\beta with reported E = 1.69, \alpha = 0.34, \beta = 0.28.

• Why: for a fixed-design least-squares smoother, excess error \approx \sigma^2\cdot \text{df}/n, so \alpha = 1; real curves often have \alpha < 1, because the effective complexity a model can use grows with the data (Card 18). On the season, degree 12's fitted \alpha is 1.09 with a = 0.0903, the noise floor.
• Next they'll ask: "Three points determine three parameters — how confident are you?" → Not very: fit on more scales, put intervals on the extrapolation, and check the residuals of the fit at the largest runs (Drill 16).

Level 4 — Research and quant

These questions hand you a setup — a design, a prior, a noise level, a fitting rule — and ask for the exact expected error, or the exact knob that minimizes it. In this chapter almost every estimator is linear in y: least squares, ridge, early-stopped gradient descent, PCR and the minimum-norm interpolator. That gives one method, and the worked problems and most drills below use it.

The recipe: Rotate, filter, split, sum.

1. Rotate into the basis where the problem decouples: the SVD X = UDV^\top for a linear smoother, or plain coordinates for an orthonormal design. Along direction j the least-squares coordinate is z_j = u_j^\top y/d_j = \theta_j + \text{noise} of variance \sigma^2/d_j^2, with \theta = V^\top \beta.
2. Filter: write the estimator as a per-direction multiplier c_j on that coordinate — 1 for OLS, d_j^2/(d_j^2 + \lambda) for ridge, 1 - (1 - \eta d_j^2)^t for early stopping, 0 or 1 for PCR and for the minimum-norm fit beyond rank, a soft threshold for lasso.
3. Split each direction's error into \text{bias}^2 and variance: (1 - c_j)^2\theta_j^2 + c_j^2\sigma^2/d_j^2.
4. Sum over directions (a trace), add the noise floor \sigma^2, and take the expectation the question asks for — over the noise, the prior on \beta, or the random design.
5. Name the mechanism, then say how you'd see it in real data and which knob fixes it.

\mathbb E

average over the noise (and over the prior or the random design when the question asks)

c_j

the filter: the multiplier the estimator puts on direction j (1 for OLS)

z_j

the least-squares coordinate along direction j: truth plus noise

\theta_j

the true coefficient along direction j, \theta = V^\top\beta

\frac{\sigma^2}{d_j^2}

the noise variance of z_j: noise variance \sigma^2 over the data's spread d_j^2 along direction j

d_j

singular value j: how much the data spread along direction j

R

test risk: the expected squared error on a new point

\sum_j

add up over directions j

s_j

how much the test distribution uses direction j (1 if test x is isotropic)

\sigma^2

noise variance: the floor no estimator can beat

Here s_j = v_j^\top\Sigma v_j weights each direction by how much the test distribution of x (covariance \Sigma) uses it; s_j = 1 when test x is isotropic. The sum is exact when the errors in different directions are uncorrelated — always true for the noise part, and true for the bias part under an isotropic prior on \beta or isotropic test x; otherwise add the cross terms. Ch 1's recipe, "Cov over Var, then expand", is the one-direction case: one feature, filter 1. Non-linear procedures (kNN, cross-validation, importance weighting, model selection) skip the rotation and use steps 3–5 or a direct expectation. The hook is the derivations toolkit, S-E-A-T (SVD, Expectation, Add and subtract, Trace), run in order.

Worked problem 1 — Bayes-optimal ridge on the dense station data

A standard research-screen question type: "derive the optimal ridge penalty under a Gaussian prior". The station setting is this chapter's.

Setup: the expansion team models 100 existing stations' standardized daily rides from 80 station descriptors (unit variance, in 16 correlated blocks of 5 with within-block correlation 0.8, no intercept needed). This time nobody believes in a short list: every descriptor has a small effect, \beta_j \sim N(0, \tau^2) independently with \tau^2 = 0.05, and the noise variance is \sigma^2 = 4. You fit ridge, \lVert y - Xw\rVert^2 + \lambda \lVert w\rVert^2. Which \lambda minimizes the expected test error on a new station, averaged over plausible \beta? Prove it. How much does it beat least squares, and what does a \lambda ten times too big or too small cost? When would you prefer lasso?

Intuition first: \lambda is an exchange rate between the data and the prior. The noise variance \sigma^2 says how loudly the data lie; the prior variance \tau^2 says how big real effects tend to be. When the data lie loudly relative to the size of real effects, trust them less and shrink more: \lambda = \sigma^2/\tau^2, noise over prior.

Solution. Run the recipe.

1. Rotate. Take the SVD X = UDV^\top. The prior is rotation-invariant, so \theta = V^\top \beta \sim N(0, \tau^2I) too, and the least-squares coordinates decouple.

\mathbf u_j^\top\mathbf y

how much of the rides vector lies along direction j

\eta_j

the noise in direction j's coordinate

\tau^2

prior variance: how big each true coefficient tends to be (0.05)

\sigma^2

noise variance on each station's rides (4)

\sim

is distributed as

N(0, \tau^2)

normal with mean 0 and variance \tau^2 = 0.05: the prior on each true coefficient

N

a normal (Gaussian) distribution, written N(\text{mean}, \text{variance})

1. Filter. Any estimator of the form \hat{w} = \sum_j c_jz_jv_j; ridge is c_j = d_j^2/(d_j^2 + \lambda).
2. Split, averaging over the prior as well as the noise (E):

\mathbb E

average over the prior on \theta_j and over the noise

(1 - c_j)^2\,\tau^2

squared bias averaged over the prior: the share of the truth the filter drops, squared, times the prior variance \tau^2

1. Sum. A new station x \sim N(0, \Sigma) has expected squared error \sigma^2 + (\hat w - \beta)^\top\Sigma(\hat w - \beta). The error \hat{w} - \beta = \sum_j(c_jz_j - \theta_j)v_j has independent, mean-zero coordinates, so every cross term between directions averages to zero and only the diagonal weights s_j = v_j^\top\Sigma v_j survive.

\mathbb E[R \mid X]

expected test risk for this design X, averaged over the prior and the noise

\sum_{j=1}^{p}

add up over all p directions, one per station descriptor (p = 80)

(1 - c_j)^2\tau^2

direction j's squared bias, averaged over the prior (variance \tau^2)

\mathbf v_j

direction j in weight space (column j of V)

\Sigma

the covariance matrix of a new station's descriptors

1. Minimize term by term. Each bracket is a parabola in c_j, and its minimizer does not depend on the weight s_j in front of it.

c_j^*

the best filter for direction j (star = the optimum)

\frac{\tau^2}{\tau^2 + \sigma^2/d_j^2}

prior variance over prior-plus-noise variance: the share of z_j worth trusting

\sigma^2/\tau^2

noise over prior: the penalty that reproduces that filter

\lambda^*

the Bayes-optimal ridge penalty

\frac{\sigma^2}{\tau^2}

noise variance \sigma^2 = 4 over prior variance \tau^2 = 0.05

One \lambda gives the optimal filter in every direction at once. That proves ridge with this \lambda is the best of all per-direction filters. The stronger claim — best among all estimators — is Card 8: the posterior is Gaussian, its mean is exactly ridge with \lambda = \sigma^2/\tau^2, and the posterior mean minimizes expected squared error in any PSD metric M (add and subtract the posterior mean; the cross term vanishes).

1. The numbers. At \lambda*, each bracket equals the posterior variance \sigma^2\tau^2/(\sigma^2 + \tau^2d_j^2); at \lambda = 0 (OLS) it is \sigma^2/d_j^2, and the random-design expectation is Card 3's inverse-Wishart mean.

R(\lambda^*)

expected test risk of ridge at the optimal penalty

\mathbb E

average over random designs X

\text{tr}

trace: the sum of a matrix's diagonal entries

(X^\top X + \lambda^* I)^{-1}

the inverse of ridge's penalized cross-product matrix at \lambda^*

R_{\text{OLS}}

expected test risk of least squares

\frac{p}{n - p - 1}

least squares' variance inflation on random designs: p features against n - p - 1 spare rows

n

the number of training stations (100)

The simulated risk curve over a grid of 44 \lambda values has its minimum at exactly \lambda = 80, as does the exact curve (sims/sim_dense.py).

What to say out loud: the Bayes-optimal penalty is noise over prior, 4/0.05 = 80, and it's not a tuning heuristic — under this prior, ridge at \lambda = 80 is the posterior mean, so no estimator does better on average: 5.21 against an irreducible 4. Least squares, with 80 coefficients on 100 random stations, pays \sigma^2\cdot p/(n - p - 1) = 16.8 in variance and scores 20.8 — worse than predicting zero (8.0). The risk curve is flat near its minimum: a \lambda ten times off costs about 1.0, against 15.5 for no penalty at all. Getting the order of magnitude right is what matters.

Follow-ups that separate senior from research-level answers:

• "You don't know \tau^2." Estimate it. Empirical Bayes: marginally y \sim N(0, \tau^2XX^\top + \sigma^2I), so maximize that likelihood over (\tau^2, \sigma^2) and plug in \hat\lambda = \hat\sigma^2/\hat\tau^2 (sklearn's BayesianRidge does this). Or cross-validate: here LOOCV picked a median \lambda of 80 and scored 5.32, so estimating \lambda from the data cost about 0.12 over the oracle. James–Stein (Ch 1, Drill 33) is the same move for an orthonormal design: it estimates the shrink factor from \lVert x\rVert^2.
• "Does \lambda depend on the test distribution?" No. The weights s_j = v_j^\top\Sigma_{\text{test}} v_j change with the test covariates, but step 5's minimizer never looked at them. Under pure covariate shift (same p(y \mid x)) the same \lambda stays optimal; only the achieved risk changes. Dobriban and Wager (2018) state the same fact in proportional asymptotics: the optimal ridge penalty does not depend on the feature covariance.
• "Does \lambda depend on n?" Not in the sum-of-squares convention: \lambda^* = \sigma^2/\tau^2 at any n. What changes is the effective* shrinkage d_j^2/(d_j^2 + \lambda^*), because d_j^2 grows roughly in proportion to n — more data, less shrinkage. In the mean-loss convention, (1/n)\lVert y - Xw\rVert^2 + \lambda \lVert w\rVert^2, the same answer reads \lambda^* = \sigma^2/(n\tau^2) = 0.8. Say which convention you're in (sklearn's Ridge uses the sum; Lasso and glmnet use 1/(2n)).
• "So with a Laplace prior, the best estimator is lasso?" No. The lasso is the posterior mode under a Laplace prior. The Bayes-optimal estimator for squared error is the posterior mean, which is never exactly zero (Card 9). Lasso's case rests on sparsity of the truth, not on being Bayes-optimal.
• "When does lasso win?" When the truth is sparse. On the SPARSE stations (10 effects of 0.5, 70 zeros) lasso-CV scores 4.78 and elastic net 4.67 against ridge-CV's 5.05. On these dense stations it flips: on the same 150 draws, lasso-CV is worse than ridge-CV by 0.318 ± 0.030. The prior that matches the world wins.
• "What does the posterior give you beyond the point estimate?" A covariance, \sigma^2(X^\top X + \lambda^*I)^{-1}, so prediction intervals for a new station come free. Ridge alone gives none.
• "The derivation assumed no intercept and no standardization. Does that matter?" It's why \lambda^* = 80 is exact here: the prior applies to the fitted coefficients as written. If you standardize on the training rows, the prior is on rescaled coefficients and \lambda* is only approximately σ²/τ²; leave the intercept unpenalized either way.

How to identify it in real data: you can't see \beta, but three signs point to a dense, small-effects world. The CV curve for ridge has a broad, flat minimum. Lasso's CV-chosen model keeps most features, with no clear gap in the path between strong and weak entries. A marginal-likelihood fit gives a \hat{\tau}^2 comparable across features, with no feature standing far above the rest. When the ridge \hat{\lambda} from CV and \hat{\sigma}^2/\hat{\tau}^2 from empirical Bayes agree within a factor of a few, the Gaussian prior is a reasonable description.

How to mitigate a mis-set \lambda:

1. Search \lambda on a log grid with K-fold CV or the exact LOOCV shortcut (RidgeCV); the curve is flat in \log\lambda near the minimum, so a coarse grid is enough.
2. Empirical Bayes (maximize the marginal likelihood), which also returns \hat{\sigma}^2 and \hat{\tau}^2.
3. GCV as a cheap alternative, but check leverages first: on the season, GCV's "average leverage" approximation picked \lambda = 0.32 (test 0.52) when LOOCV and 10-fold picked 5.62, because the edge days have far more leverage than \operatorname{tr} (S)/n (Card 12).
4. Compare against lasso and elastic net by CV before committing: the prior's shape matters more than fine-tuning \lambda.

Worked problem 2 — The spike at p = n (minimum-norm interpolation)

The research-lab version of "why do overparameterized models generalize?". It is asked as a derivation.

Setup: features are isotropic Gaussian, x \sim N(0, I_p), and there are n = 100 training rows. First the outcome is pure noise: \beta = 0, \sigma^2 = 1. Then add signal: for each p, \beta is a random direction in ℝᵖ with \lVert \beta \rVert = 1, and \sigma^2 = 0.25. You fit least squares with the pseudo-inverse, \hat{w} = X^+y, which for p > n is the interpolator with the smallest norm. Derive the expected test risk as a function of p from 10 to 1,000. Why does it blow up at p = n? Where does it come back to? What does ridge with a tuned \lambda do?

Intuition first: right at p = n there is exactly one weight vector that passes through all 100 noisy points, and it is the wildest: the data have almost no spread along one direction, and fitting the noise exactly along that direction takes a huge coefficient. Below the threshold, least squares averages the noise over n - p spare rows; above it, the minimum-norm rule spreads the noise thinly over p - n spare directions. Either way, distance from the threshold is what calms the fit.

Solution. Run the recipe.

1. Rotate. X = UDV^\top has r = \min(n, p) non-zero singular values d_1 \ge \ldots \ge d_{r}.
2. Filter. The pseudo-inverse keeps each of the r row-space directions with filter 1 (z_j = u_j^\top y/d_j) and puts filter 0 on the p - n null-space directions, which the data cannot see.
3. Split. Kept directions: no bias, variance \sigma^2/d_j^2. Dropped directions: all signal lost, so \text{bias}^2\ \theta_j^2, no variance.
4. Sum. A new x \sim N(0, I) gives risk \sigma^2 + \lVert \hat{w} - \beta \rVert^2, so

\sigma^2

noise variance (1 for pure noise, 0.25 with signal)

\mathbb E

average over random Gaussian designs (and noise)

\sum_{j=1}^{r}

add up over the r kept directions

r

the number of non-zero singular values, \min(n, p)

\lVert P_{\text{null}}\,\boldsymbol\beta\rVert^2

the squared size of the part of \boldsymbol\beta the data cannot see: the bias

P_{\text{null}}

projection onto the null space: the p - n directions the data cannot see

\boldsymbol\beta

the true weight vector (bold = vector)

For p < n the middle term is \operatorname{tr} ((X^\top X)^{-1}), a p \times p inverse Wishart with n degrees of freedom; there is no null space. For p > n it is \operatorname{tr} ((XX^\top)^{-1}), an n \times n inverse Wishart with p degrees of freedom — the same formula with n and p swapped (Card 16, via the Trace trick).

(X^\top X)^{-1}

the inverse of the p \times p feature cross-product matrix

(XX^\top)^{-1}

the inverse of the n \times n matrix of row inner products

\frac{n}{p - n - 1}

past the threshold: n rows against p - n - 1 spare features

p < n - 1

below the threshold: fewer features p than rows n

p > n + 1

past the threshold: more features p than rows n

n

the number of training rows (100)

The null space of X is a uniformly random (p - n)-dimensional subspace, so it captures, on average, a fraction 1 - n/p of \lVert \beta \rVert^2. Together:

\lVert\boldsymbol\beta\rVert^2

the signal's total size: the squared length of the true weights (1 here)

\dfrac{p}{n - p - 1}

least squares' variance inflation below the threshold: p features against n - p - 1 spare rows

\dfrac{n}{p - n - 1}

the min-norm fit's variance inflation past the threshold: n rows against p - n - 1 spare features

\dfrac{n}{p}

the share of \boldsymbol\beta the n-dimensional row space can see; the rest is lost as bias

1. Why it blows up. The smallest singular value of an n \times p Gaussian matrix is about \lvert \sqrt{n} - \sqrt{p}\rvert, which goes to zero at p = n. One coordinate of noise is divided by an almost-zero number. For n - 1 \le p \le n + 1 the expectation is infinite: the inverse-Wishart mean doesn't exist.

\lVert\hat{\mathbf w}\rVert^2

the squared length of the fitted weights

d_{\min}

the smallest singular value of X

\sqrt n

the square root of the number of rows n

\sqrt p

the square root of the number of features p

p

the number of features (10 to 1,000)

1. Ridge caps every direction. With filter d^2/(d^2 + \lambda), a direction's variance is \sigma^2d^2/(d^2 + \lambda)^2 \le \sigma^2/(4\lambda), whatever d is (Card 16, step 8). With the Bayes penalty for this prior (\beta a random direction \approx N(0, I/p), so \lambda = noise over prior = p\sigma^2/\lVert \beta \rVert^2), there is no spike at all.

Pure-noise rows use 800 designs per p (3,000 for 85 \le p \le 115; sims/sim_dd_null.py); signal rows use 300 (600 near the threshold; sims/sim_dd_wp2.py). Next to the threshold the risk is so heavy-tailed that the simulated means wander: at p = 105 with signal, 7.06 against an exact 6.55; at p = 98 with pure noise, 94.9 against 99, while the median there is only 42.0.

What to say out loud: the spike is not caused by having many parameters. At p = 1{,}000 there are ten times more parameters than rows and pure-noise risk is 1.11, almost the noise floor; at p = 95 it is 24.75. What matters is the smallest singular value of X, which hits zero at p = n. Past the threshold the minimum-norm rule spreads the noise over p dimensions, so variance falls like n/(p - n - 1), but it can see only the n-dimensional slice of \beta, so bias rises like \lVert \beta \rVert^2(1 - n/p). In this isotropic world the second descent bottoms out near p = 200 (1.0025) and never gets back to the small model (0.278 at p = 10). Ridge with the Bayes \lambda rises smoothly in p with no spike. The bias–variance decomposition held at every p; what failed was counting parameters as capacity.

Follow-ups that separate senior from research-level answers:

• "Does gradient descent find this solution?" From zero, yes: every gradient X^\top(Xw - y) lies in the row space, so the iterates never leave it, and the only interpolator in the row space is X^+y. At p = 300, 20,000 steps of GD from zero end within 10^{-12} of the pseudo-inverse. From a random start they end 1.42 away — exactly the norm of the start's null-space part, which GD never touches (Card 16). Stopping early is extra regularization: after 100 steps GD is still 0.48 from the interpolator.
• "Is optimal ridge monotone in p?" In this isotropic setting, yes: 0.277 at p = 10 rising to 1.170 at p = 1{,}000, with no bump anywhere. Nakkiran, Venkat, Kakade and Ma (2021) prove that optimally tuned ridge removes double descent in such settings. The rise itself is honest: with \lVert \beta \rVert fixed and spread over more directions, each coefficient is harder to learn from 100 rows.
• "Is there a sample-wise version?" Yes: fix p = 100 and grow n. Minimum-norm risk goes 1.01 (n = 50) → 2.91 (n = 90) → 6.69 (n = 105) → 0.756 (n = 150) → 0.500 (n = 200); the exact values are 1.005, 2.85, 6.50, 0.760, 0.503. More data can hurt near n \approx p (Q29). Degree 12's spike at n = 15 in the season's learning curve is the same effect.
• "Then when is interpolation benign?" When the spectrum is anisotropic: a few strong directions carry the signal and a long tail of many weak directions absorbs the noise (Bartlett, Long, Lugosi and Tsigler, 2020). In the chapter's core + embedding design (20 strong features, 980 weak ones), the interpolator at p = 1{,}000 scores 0.351 against a classical best of 0.306. In the flat design the largest model is best of all (1.09 at p = 400, against 1.25 for predicting zero). Isotropic features, as here, give no such tail, so interpolation is never better than the best small model.
• "What are the effective degrees of freedom of the interpolator?" For p \ge n it reproduces y exactly, so the in-sample smoother is S = XX^+ = I_n and \text{df} = n — capped at n however large p gets. In-sample df stops growing at the threshold, but out-of-sample variance keeps depending on the spectrum. That's why df on the training rows can't see the second descent.
• "Why should a deep-learning researcher care?" The same spike is reported along model size, training time (epoch-wise) and dataset size in networks, and it is strongest with label noise (Nakkiran et al., 2019). Weight decay and early stopping, the same caps as step 6, are what make it disappear in practice.

How to identify it in real data:

• A validation spike when the parameter count, or the effective df, is close to the number of training rows.
• Training error reaching exactly zero at the same point.
• A jump in \lVert \hat{w}\rVert: in the core + embedding design it goes from 0.99 at p = 10 to 42.8 at p = 100.
• A tiny smallest singular value or a huge condition number of X.
• Predictions that swing when one training row is removed.

How to mitigate it:

1. Ridge or weight decay with a tuned \lambda: in the core + embedding design, ridge-CV stays between 0.309 and 0.354 for every p, including p = n.
2. Early stopping from small initial weights.
3. Move away from the threshold: collect more rows, or go far past it with many weak features.
4. Truncate the spectrum (PCR to k well below n), which deletes the near-zero directions outright.

Remember: Rotate, filter, split, sum. With a Gaussian prior, \lambda = \text{noise over prior} is optimal in every direction at once, for any test distribution of x. The spike at p = n is the smallest singular value, not the parameter count; past it, minimum norm picks the calmest fit. Ridge caps every direction's variance at \sigma^2/(4\lambda).

Level 4 drills

Twenty setups that reduce to a page of algebra. Drills 1, 2, 12 and 18 cover bias, variance and capacity; 3–7 and 19 ridge and its disguises; 8 interpolation; 9–11 LOOCV and sparsity; 13 and 20 shift; 14–17 what a validation number or a learning curve really tells you. Cover the answer, run the recipe on paper, then check. Every answer was computed exactly (sims/exact.py, or sims/sim_dd_null.py for Drill 8). Answers marked "verified by simulation" were also confirmed by Monte Carlo (sims/drills.py, seed 4400 + drill number, or the chapter sim named); "checked with sklearn" means the library reproduced the hand answer. All variables are independent and N(0, 1) unless stated.

Drill 1 · How many neighbors at the peak?

Ops wants a quick June 20 staffing number from last season's 30 days and proposes "average the k days closest to June 20". The season has one day every 1/30 of a year, x_i = (i - 0.5)/30, and June 20 (x_0 = 0.25) is itself a design point. The truth is f(x) = \sin(2\pi x), which peaks at 1 there, and the noise variance is \sigma^2 = 0.09. "For k = 1, 3, 5, 7, 9 symmetric neighbors, what are the bias and variance of the forecast at June 20, and which k minimizes the expected error?"

Concept tested: kNN bias–variance (Card 5).

• Answer: variance \sigma^2/k = 0.09, 0.03, 0.018, 0.0129, 0.01. Bias 0, −0.0146, −0.0433, −0.0855, −0.140. Excess error (\text{bias}^2 + \text{variance}) 0.090, 0.0302, 0.0199, 0.0202, 0.0296, so k = 5 (verified by simulation: 0.0900, 0.0301, 0.0198, 0.0201, 0.0296). Add \sigma^2 = 0.09 for the expected squared error.
• Intuition: at a peak every neighbor is lower than the peak, so each extra day drags the forecast down; but each extra day also dilutes the noise. The best k is where the next day's dilution stops paying for its drag.
• Why: k = 2m + 1 neighbors at spacing h = 1/30 average \cos(2\pi j/30) for j = -m \ldots m, which sums in closed form (a Dirichlet kernel). A second-order Taylor expansion gives the familiar f''h^2 shape: f'' = -4\pi^2, so f''h^2/6 = -0.00731, and k = 9 (m = 4) gives −0.146 against the exact −0.140.

\text{bias}_k

the bias of the k-neighbor average at June 20

k

the number of neighbors averaged, k = 2m + 1

\sum_{j=-m}^{m}

add up over the neighbors, from m steps before June 20 to m after

j

a neighbor's offset from June 20, in design steps

m

the number of neighbors on each side

\pi

pi, 3.14159…; 2\pi j/30 turns j steps into an angle of the yearly cycle

f''(x_0)

the curvature of the true curve at June 20: -4\pi^2

h

the spacing between design days, 1/30 of a year

\text{Var}_k

the variance of the k-neighbor average

\sigma^2

noise variance, 0.09 on the season

• Next they'll ask: "Same k everywhere?" → No: where the curve is nearly straight (f'' \approx 0, near the equinoxes) bias is tiny and a larger k pays; at the edges of the data the neighbors are all on one side, so bias becomes first-order in h. Over the whole season k = 5 is also best (test 0.124).

Drill 2 · 80 features, 100 rows

A colleague fits OLS with 80 station descriptors to 100 stations and is pleased with the training error. The interviewer strips it down: "Assume the linear model is correct, no intercept, noise variance \sigma^2 = 1, n = 100, p = 80. What is the expected training MSE, the expected test MSE on the same 100 x's with fresh noise, and the expected test MSE on new stations with Gaussian x?"

Concept tested: fixed-X vs random-X optimism (Card 3).

• Answer: training 1 - p/n = 0.20; fixed-X test 1 + p/n = 1.80; random-X test 1 + p/(n - p - 1) = 1 + 80/19 = 5.21 (verified by simulation: 0.202, 1.799, 5.19 ± 0.02).
• Intuition: the fit spends 80 degrees of freedom chasing noise, which flatters the training rows and hurts the same rows by the same amount. New x's hurt far more: a random design with 80 features on 100 rows has some nearly flat directions, and new stations probe exactly those.
• Why: \mathbb{E}[\text{RSS}] = (n - p)\sigma^2; at the same x's the test error adds the prediction variance \sigma^2\cdot \operatorname{tr} (H)/n = \sigma^2p/n. For new x \sim N(0, I) the variance is \sigma^2\cdot \operatorname{tr} ((X^\top X)^{-1}), and the inverse-Wishart mean is I/(n - p - 1).

\mathbb E

average over the noise (for the random case, also over new designs)

\sigma^2

noise variance (1 here)

\frac pn

the share of the rows the coefficients use up, p/n = 0.8

p

the number of features (80)

n

the number of training stations (100)

• Next they'll ask: "With an intercept?" → \sigma^2(1 + 1/n)(1 + p/(n - p - 2)). On the stations (\sigma^2 = 4) that is 22.0, confirmed at 21.96 ± 0.03 over 3,000 designs, and the 50-replicate OLS average is 22.2 — worse than predicting the mean, 14.5 (Q30).

Drill 3 · Shrinking one coefficient

A pricing team has one well-isolated feature whose true coefficient they believe is about 1, and their least-squares estimate has variance 0.25. "With an orthonormal design (one coordinate, d^2 = 1), \beta = 1 and \sigma^2 = 0.25, what ridge \lambda minimizes the MSE of the coefficient, and how much does it beat OLS?"

Concept tested: per-direction bias/variance; \lambda^* = \sigma^2/\beta^2.

• Answer: \lambda^* = \sigma^2/\beta^2 = 0.25, shrink factor 1/(1 + 0.25) = 0.8; \text{bias}^2 0.04 + variance 0.16 = 0.20 against OLS 0.25 — a 20% cut (verified by simulation: 0.2002 vs 0.2505).
• Intuition: shrinking by 20% costs 0.2 of bias (0.04 squared) and saves 36% of the variance (0.25 → 0.16).
• Why: step 3 of the recipe with filter c = d^2/(d^2 + \lambda). Setting the derivative to zero gives \lambda^* = \sigma^2/\beta^2 whatever d is, and the minimized MSE is \sigma^2\beta^2/(\sigma^2 + \beta^2d^2) — WP1's posterior variance with \beta^2 in place of \tau^2.

\text{MSE}(\lambda)

the coefficient's mean squared error at ridge penalty \lambda

\lambda

the ridge penalty strength

\beta

the true coefficient (1)

d

the design's singular value along this coordinate (d^2 = 1)

\sigma^2

noise variance (0.25)

\lambda^*

the penalty that minimizes the coefficient's MSE (star = the optimum)

\text{MSE}_{\text{OLS}}

the least-squares coefficient's MSE, \sigma^2/d^2

• Next they'll ask: "But \lambda^* needs \beta, which you don't know." → That's why \lambda is chosen by CV, or by a prior (WP1). The oracle \lambda^* shows the size of the possible gain.

Drill 4 · Ridge on the toy line

Back to the six training days. On centered months the line's slope has S_{\text{mm}} = \sum(m - \bar{m})^2 = 70; the best line through the true curve has slope \beta = -12/70 per month; the noise variance is \sigma^2 = 0.09. "What \lambda minimizes the MSE of the ridge slope as an estimate of \beta, and how much does it gain over least squares?"

Concept tested: \lambda^* = \sigma^2/\beta^2 in one dimension; a strong signal needs little shrinkage.

• Answer: \lambda^* = 0.09/(12/70)^2 = 3.0625; the slope's MSE falls from 0.001286 to 0.001232, a gain of only 4.2%.
• Intuition: the slope is measured with a signal-to-noise ratio \beta^2d^2/\sigma^2 of about 23, so there is little variance to buy back.
• Why: Drill 3's formula with d^2 = 70. The relative MSE is 1/(1 + \sigma^2/(\beta^2d^2)) = 0.958. Compare Drill 3, whose ratio \beta^2d^2/\sigma^2 = 4 gave a 20% gain: the gain from shrinkage is 1/(1 + SNR) of the OLS error.

\sigma^2

noise variance (0.09)

\beta

the best-line slope, -12/70 thousand rides per month

\text{MSE}(\lambda^*)

the ridge slope's MSE at the best penalty

Drill 5 · Count the effective df

A model card says "ridge, 4 features, effective df 2.5", and a reviewer asks how that can be. The design's squared singular values are d^2 = 100, 10, 1 \text{ and } 0.1, and \lambda = 1. "Compute the effective degrees of freedom, and say what each term means."

Concept tested: \text{df}(\lambda) = \textstyle\sum d^2/(d^2 + \lambda) (Cards 4 and 6).

• Answer: 0.990 + 0.909 + 0.5 + 0.091 = 2.49 (verified by simulation: the covariance df \sum \operatorname{Cov} (\hat{y}_i, y_i)/\sigma^2 over 200,000 noise draws is 2.492).
• Intuition: each direction contributes the fraction of itself that ridge keeps. The strong direction is a full degree of freedom; the weakest is less than a tenth of one.
• Why: \hat{y} = Sy with S = U \operatorname{diag} (d_j^2/(d_j^2 + \lambda)) U^\top, and \text{df} = \operatorname{tr} (S).

S_\lambda

ridge's smoother matrix at penalty \lambda: fitted values = S_\lambda\,\mathbf y

Drill 6 · Jitter equals ridge

An engineer adds Gaussian noise of standard deviation 0.1 to every standardized input on every pass, trains with the sum of squared errors over the 30 season days, and validation error drops. "For a linear model, what explicit penalty is this, and what is \lambda?"

Concept tested: noise injection = ridge (Card 14).

• Answer: ridge with \lambda = n\cdot s^2 = 30 \times 0.01 = 0.3 for the sum loss (\lambda = s^2 = 0.01 for the mean loss). Verified by simulation on the season's degree-15 design: least squares on 200 jittered copies per day and ridge with \lambda = 0.3 give coefficients within 0.003 of each other and test MSEs of 0.534 and 0.533 (plain least squares: 0.888).
• Intuition: a fit that leans on a weight w_j suffers every time feature j wobbles, so large weights get punished in proportion to the jitter.
• Why: expand the expected loss; the cross term has mean zero and the square of the jitter leaves s^2\lVert w\rVert^2 per row.

\mathbb E_\Delta

average over the random jitter \Delta

\Delta

the jitter: fresh N(0, s^2) noise added to every standardized input

\mathbf y

the vector of training rides

X

the clean standardized inputs (30 days by degree-15 features)

\mathbf w

the weight vector

\lVert\mathbf y - X\mathbf w\rVert^2

the ordinary sum of squared misses on clean inputs

\mathbb E

average over the jitter

n

the number of training rows (30)

s^2

the variance of the jitter (0.01 for SD 0.1)

\lVert\mathbf w\rVert^2

squared length of the weights: the sum of squared weights

• Next they'll ask: "Jitter in raw units?" → Each feature gets its own penalty n\cdot s_j^2w_j^2, so features measured in small units are penalized more. Jitter on standardized features is what gives ordinary ridge.

Drill 7 · When to stop is what \lambda to pick

An MLE runs gradient descent on \tfrac 12 \lVert y - Xw\rVert^2 from w = 0 with \eta = 0.01 and stops at t = 100, and calls it "unregularized". The data have three directions with d^2 = 10, 1 \text{ and } 0.1. "How much of the least-squares solution has GD recovered along each direction, and how does that compare with ridge at \lambda = 1/(\eta t)?"

Concept tested: early stopping as spectral filtering (Card 13).

• Answer: GD's filter 1 - (1 - \eta d^2)^t = 1.000, 0.634, 0.095; ridge with \lambda = 1/(\eta t) = 1 gives d^2/(d^2 + 1) = 0.909, 0.5, 0.091. A direct run of 100 GD steps reproduces GD's three numbers to six digits.
• Intuition: gradient descent fits strong directions first and weak ones slowly — shrink the weak directions most, done by the clock instead of by a penalty.
• Why: in the SVD basis each coordinate of GD from zero obeys a one-dimensional recursion. For weak directions both filters are \approx \eta td^2 = d^2/\lambda, so they agree. For strong ones GD's filter reaches 1 exponentially fast while ridge's approaches it only like 1 - \lambda/d^2, so the map \lambda = 1/(\eta t) is rough: the filters cross 1/2 at the same d^2 only if \lambda = \ln 2/(\eta t).

\theta_j^{(t)}

gradient descent's coefficient along direction j after t steps from zero

\eta

the step size (0.01)

t

the number of gradient-descent steps (100)

• Next they'll ask: "How rough on real data?" → On the season's degree-15 fit both have the same U and nearly the same minimum (GD 0.130, ridge 0.137), but at matched t their test curves differ by up to 0.10; with a smaller step the best-fitting constant is \lambda \approx 0.6/(\eta t).

Drill 8 · Fitting pure noise

*A sanity check: a researcher regresses a pure-noise target on p random Gaussian features with n = 100 rows and uses np.linalg.lstsq (the minimum-norm solution). The interviewer: "Isotropic X, \beta = 0, \sigma^2 = 1. What is the expected test risk at p = 50, 90, 95, 105, 110, 200 \text{ and } 1{,}000?"*

Concept tested: minimum-norm risk on both sides of the threshold (Card 16, WP2).

• Answer: 2.02, 11.0, 24.75, 26.0, 12.1, 2.01, 1.11 (verified by simulation: 2.02, 11.05, 24.7, 25.5, 12.0, 2.01, 1.11).
• Intuition: with nothing to learn, every unit of risk above 1 is variance, and variance is set by how close the smallest singular value is to zero.
• Why: WP2 with \beta = 0. Below n, p/(n - p - 1); above n, n/(p - n - 1). The two sides mirror each other around p = n.

p

the number of pure-noise features

Drill 9 · LOOCV of the average

A junior analyst wants a leave-one-out score for the simplest forecast — "predict the season average" — on the six toy days, whose training MSE is 0.5467. "Can you get LOOCV without refitting six times?"

Concept tested: the LOO shortcut for a linear smoother.

• Answer: \text{LOOCV} = (n/(n - 1))^2 \times \text{training MSE} = 1.44 \times 0.5467 = 0.787 (matches brute-force refitting).
• Intuition: leaving a day out moves the mean away from that day by 1/(n − 1) of its residual, so every LOO residual is the training residual stretched by n/(n − 1).
• Why: the mean is a smoother with S_{ii} = 1/n for every row, and the shortcut divides each residual by 1 - S_{ii} (Card 12).

y_i

the rides on toy day i

\bar y_{(-i)}

the average of the other five days: the forecast with day i left out

\bar y

the average of all six days (bar = average)

n

the number of days (6)

Drill 10 · Soft-thresholding by hand

With an orthonormal design, OLS gives z = (3, 0.8, -1.5). "Fit lasso with \lambda = 2 in the convention \lVert y - Xw\rVert^2 + \lambda \lVert w\rVert_1. What is the solution, and what is the smallest \lambda that sets every coefficient to zero?"

Concept tested: KKT conditions, soft-thresholding, \lambda_{\max} (Card 9).

• Answer: \hat{w} = (2, 0, −0.5); \lambda_{\max} = 2\cdot \max \lvert z_j\rvert = 6. Checked with sklearn (Lasso with \text{alpha} = \lambda/(2n) returns (2, 0, -0.5) and all zeros just above \lambda = 6).
• Intuition: lasso slides every coefficient \lambda/2 = 1 toward zero and stops it there; 0.8 doesn't survive the slide.
• Why: with X^\top X = I the problem separates by coordinate; the subgradient condition 2(z_j - w_j) \in \lambda \cdot \partial \lvert w_j\rvert gives the soft threshold.

\hat w_j

lasso's coefficient on feature j

\text{sign}

the sign: +1 if positive, -1 if negative

z_j

the OLS coefficient on feature j: here 3, 0.8 and -1.5

\Big(|z_j| - \frac{\lambda}{2}\Big)_+

shrink \lvert z_j\rvert by \lambda/2 but stop at 0 (the _+ keeps only positive values)

\lambda

the lasso penalty strength (2)

\lambda_{\max}

the smallest penalty that sets every coefficient to zero

\max_j

the largest over the coefficients j

Drill 11 · Two copies of one feature

A feature store accidentally exposes the same column twice (x_1 = x_2 = x). With one copy, OLS gives a coefficient of 1.0, and \lVert x\rVert^2 = 100. "Fit ridge (\lambda = 20), lasso (\lambda = 20) and elastic net (\lambda_1 = \lambda_2 = 20) on both copies. What coefficients do you get?"

Concept tested: the grouping effect (Card 10); lasso picks, ridge spreads, elastic net groups.

• Answer: ridge (0.455, 0.455); lasso: any same-sign split with total 0.9 — not unique (sklearn returns (0.9, 0)); elastic net (0.409, 0.409). All three checked with sklearn.
• Intuition: the fit only sees the sum w_1 + w_2. The L2 penalty is smallest when the sum is split evenly; the L1 penalty is the same for every same-sign split, so lasso doesn't care.
• Why: with s = w_1 + w_2 and x^\top y = \lVert x\rVert^2 = 100, the squared error is 100(1 - s)^2 + \text{const}. Its derivative with respect to either copy is -200(1 - s). Set each copy's stationarity condition to zero: ridge adds 2\lambda w, lasso adds \lambda \cdot \operatorname{sign} (w), elastic net adds both. For ridge and elastic net, w_1 = w_2 = w by symmetry and strict convexity; for lasso only the total s is pinned down.

w

the weight on each copy (equal for ridge and elastic net, by symmetry)

s

lasso's total weight on the two copies, w_1 + w_2

Drill 12 · The rent for kNN

An analyst says kNN "has no parameters, so its training error is honest". On the season (n = 30, \sigma^2 = 0.09) they use k = 5. "What is the expected gap between the in-sample test error and the training error?"

Concept tested: \text{optimism} = 2\sigma^2\cdot\text{df}/n (Card 4).

• Answer: \text{df} = n/k = 6; \text{gap} = 2 \times 0.09 \times 6/30 = 0.036 (verified by simulation: optimism 0.0359, covariance df 6.02).
• Intuition: each prediction averages its own noisy y with weight 1/k, so the fit chases its own noise by 1/k at every row.
• Why: kNN is a linear smoother with S_{ii} = 1/k (each point is its own neighbor), so \operatorname{tr} (S) = n/k, and the covariance formula turns df into optimism.

\sum_i

add up over the 30 season days

\text{Cov}

covariance over noise redraws: how much the fit at day i moves with that day's rides

\hat y_i

kNN's fitted value at day i

y_i

the rides observed on season day i

\sigma^2

noise variance (0.09)

S

kNN's smoother matrix: 1/k in each row's k neighbor columns

\frac nk

n/k: the number of days over the number of neighbors

k

the number of neighbors averaged (5)

n

the number of season days (30)

\mathbb E

average over noise redraws on the same 30 days

Drill 13 · How much data survives reweighting?

Training traffic comes from one region and deployment from another, shifted by \mu standard deviations on a key feature. "Training x \sim N(0, 1), deployment x \sim N(\mu, 1). Write the importance weights, and give the effective-sample-size fraction for \mu = 1 and \mu = 2."

Concept tested: importance weights and ESS (Card 17).

• Answer: w(x) = e^{\mu x - \mu^2/2}; \text{ESS}/n = 1/\mathbb{E}_{\text{train}}[w^2] = e^{-\mu^2} = 0.368 at \mu = 1 (verified by simulation: 0.367) and 0.0183 at \mu = 2 (exact; a 2-million-draw Monte Carlo gives 0.022, see below).
• Intuition: a shift of one standard deviation keeps about a third of your data; two keeps under 2%, because a handful of rows in the far tail carry almost all the weight.
• Why: the ratio of two Gaussian densities is an exponential tilt, and \mathbb{E}[e^{aX}] = e^{a^2/2}.

w

the importance weight of a training row at x

x

the key feature's value

\varphi

the standard normal density (the bell curve)

\mu

the size of the shift, in standard deviations

e

the exponential base, 2.718…: e^{a} is \exp(a)

\mathbb E

average over training rows, X \sim N(0, 1)

X

a random training value of the feature

n

the number of training rows

• Next they'll ask: "Why can't simulation confirm \mu = 2?" → Estimating \mathbb{E}[w^2] needs \mathbb{E}[w^4] = e^{6\mu^2} = e^{24} to be finite and moderate; it is finite but astronomically large, so even 2 million draws give a noisy answer. The same heavy tail makes a weighted model unstable — the lesson of Q33: clip or self-normalize, and check overlap.

Drill 14 · The best of ten equal models

A team tries ten hyperparameter settings that, unknown to them, all have the same true error 0.200. Each validation estimate has independent noise with standard deviation 0.010. They report the minimum. "What reported error do you expect?"

Concept tested: the Lucky minimum (winner's curse in model selection).

• Answer: 0.200 - 0.010 \times \mathbb{E}[\text{max of 10 standard normals}] = 0.200 - 0.010 \times 1.539 = 0.185 (verified by simulation: 0.18461).
• Intuition: the minimum of ten noisy scores is, by construction, the one with the luckiest noise.
• Why: \min \text{ of the estimates} = 0.200 - 0.010 \times \max \text{ of ten } N(0, 1) \text{ draws}, by symmetry.

\mathbb E

average over many repeats of the validation noise

\min_{k \le 10}

the smallest of the ten estimates

\hat e_k

setting k's validation estimate: true 0.200 plus noise

\max_{k \le 10}

the largest of the ten draws

Z_k

setting k's noise in standard units: a standard normal draw

• Next they'll ask: "With 500 configurations?" → The expected maximum of K normals grows roughly like \sqrt{2 \ln K}, so the optimism keeps growing with every configuration tried (Q35). Report a number from data the search never touched: a fresh holdout or nested CV.

Drill 15 · How pessimistic is K-fold?

A model's learning curve is known to be \text{error}(n) = 1 + 20/n, and you have n = 100 rows. "What do 5-fold, 10-fold and leave-one-out CV estimate on average, compared with the true error of the model fitted on all 100?"

Concept tested: CV trains on less (Card 11) — the Less data L.

• Answer: truth error(100) = 1.200; 5-fold estimates error(80) = 1.250; 10-fold error(90) = 1.222; LOO error(99) = 1.202.
• Intuition: each CV fit sees only (K − 1)/K of the rows, so CV measures a slightly worse model than the one you'll ship.
• Why: the expected fold error is the learning curve evaluated at the training-fold size; the pessimism is the learning curve's drop between n(K − 1)/K and n.

\mathbb E[\text{CV}_K]

the K-fold CV estimate, averaged over datasets

K

the number of folds

n

the number of rows (100)

• Next they'll ask: "On the season?" → At degree 3, 10-fold CV averages 0.1148 against a true 0.1063 over 500 seasons: pessimistic by 0.0085. On the six toy days the pessimism is so large that 3-fold CV prefers the line.

Drill 16 · Extrapolating a learning curve

A research lead has validation errors of 0.1632, 0.1316 and 0.1158 at 1k, 4k and 16k training examples and asks whether buying 64k is worth it. "Fit \text{error} = a + b\cdot n^{-\alpha} and predict the error at 64k."

Concept tested: power-law learning curves (Card 18).

• Answer: the differences 0.0316 and 0.0158 halve per 4× → 4^\alpha = 2 → \alpha = 0.5; b\cdot(1\text{k})^{-\alpha} = 0.0316/(1 - 1/2) = 0.0632; asymptote a = 0.1632 - 0.0632 = 0.100; at 64k, 0.100 + 0.0632/8 = 0.108.
• Intuition: with three points spaced by a constant factor, the ratio of successive improvements gives the exponent, and the exponent gives the floor.
• Why: subtracting consecutive terms removes a; the ratio removes b.

e

validation error as a function of training size

n

the smallest run's size (1k examples)

\alpha

the power-law exponent: how fast extra data pays off

a

the asymptote: the error with unlimited data

b

the scale of the extra error at small data

• Next they'll ask: "When does this break?" → When the data distribution changes as you scale (new sources, repeated data), when you approach the noise floor, or near an interpolation threshold, where more data can briefly hurt (WP2). Fit the curve on the regime you'll extrapolate from (Q39).

Drill 17 · Overlapping folds

A team reports 10-fold CV error ± SE, with \text{SE} = (\text{SD of the 10 fold errors})/\sqrt{10}, and claims two models differ "by three SEs". The folds share most of their training rows, so the fold errors are correlated. "Model the fold errors as equicorrelated with SD s and pairwise correlation \rho = 0.3. By how much does the naive SE understate the true SE of the CV mean?"

Concept tested: the variance of a CV estimate (Card 11) — the Lots of noise L.

• Answer: if s is known, the true SE is larger by \sqrt{1 + (K - 1)\rho} = \sqrt{3.7} = 1.92× (verified by simulation: 1.93). If, as in practice, s is estimated from the fold errors themselves, that SD is deflated by 1 - \rho and the understatement grows to \sqrt{3.7/0.7} = 2.30× (simulation 2.37; the gap is Jensen's inequality, since \mathbb{E}[\text{SD}] < \sqrt{\mathbb{E}[\text{Var}]}).
• Intuition: ten correlated fold errors carry less information than ten independent ones, and correlated values also look closer together than they are.
• Why: variance of a mean of equicorrelated terms; and the expected sample variance of equicorrelated terms is s^2(1 - \rho).

\text{Var}

variance across datasets

\bar e

the CV mean: the average of the K fold errors (bar = average)

s^2

the variance of each fold error (s is its SD)

K

the number of folds (10)

\rho

the correlation between any two fold errors (0.3)

\mathbb E

average across datasets

\hat s^2

the sample variance of the fold errors, computed from the folds themselves (hat = estimated)

• Next they'll ask: "On real data?" → On the season at degree 3, the SD of the 10-fold estimate across 500 seasons is 1.25 times the average naive SE. There is no unbiased estimator of this variance from a single K-fold run (Bengio and Grandvalet, 2004); use repeated CV with care, or nested CV and a held-out set.

Drill 18 · How many labelings can a line draw?

A research interviewer probes capacity: "Take four points in general position in the plane and a classifier that is any line (one side positive). How many of the 16 labelings can it realize? What is the VC dimension of lines in the plane? And of the one-parameter family sign(sin(ωx)) on the real line?"

Concept tested: shattering and VC dimension (Card 19) — Count it, Correlate it, Shatter it.

• Answer: 14 of 16 (for four points in convex position the two "diagonal" labelings fail; for a triangle with a point inside, "inside vs the rest" fails both ways); so \text{VC} = 3 (three points not on a line give all 8 labelings). \operatorname{sign} (\sin(\omega x)) has infinite VC dimension: it shatters x_i = 10^{-i} for any m (checked by enumeration for m = 5).
• Intuition: a line can cut off any one point or any adjacent pair, but not two opposite corners. A sine wave with one free frequency can wiggle as fast as it likes, so one parameter can encode any labeling.
• Why: the 14 is an exact count (a linear-programming feasibility check per labeling). For the sine family, choose \omega from the labels' digits: with y_i \in \{-1, +1\},

\omega

the sine's frequency, built from the labels

\pi

pi, 3.14159…

\sum_{i=1}^{m}

add up over the m points

y_i

the label of point i, -1 or +1

i

which point: point i sits at x_i = 10^{-i}

m

the number of points to shatter

\text{sign}

the sign: +1 if positive, -1 if negative

• Next they'll ask: "So parameter count is useless?" → It is a crude proxy. VC is a worst case over labelings; covariance df measures how hard a given procedure chases noise; and a procedure (penalty, early stopping, minimum norm) can make a huge family behave like a small one (Section 10).

Drill 19 · Dropout's hidden penalty

A research-flavored onsite: inverted dropout with keep-probability q = 0.8 is applied to the inputs of a linear regression with n = 100 rows of standardized features (so \sum_ix_{ij}^2 = 100 for every j), sum-of-squares loss. "What objective is minimized in expectation, and what ridge \lambda does it correspond to?"

Concept tested: dropout = per-feature ridge (Card 15).

• Answer: the squared error plus \sum_j((1 - q)/q)(\sum_ix_{ij}^2)w_j^2, so \lambda = (0.2/0.8) \times 100 = 25 on every feature (verified by simulation: over 200,000 dropout masks the averaged objective is 1,440.9 ± 0.3 against the formula's 1,441.4, within two standard errors).
• Intuition: inverted dropout keeps the prediction right on average but adds variance proportional to each used feature's energy, and that variance is exactly a ridge penalty.
• Why: with \tilde{x}_{ij} = x_{ij}\delta_{ij}/q, the mean is x_{ij} and the variance is x_{ij}^2(1 - q)/q; the expected squared error is the clean loss plus that variance summed over rows.

\mathbb E_\delta

average over the random dropout masks \delta

\tilde X

the inputs after inverted dropout: each kept with probability q and divided by q

q

keep probability (0.8)

\sum_j

add up over features j

\sum_i

add up over training rows i

x_{ij}

feature j's value on row i

w_j

the weight on feature j

• Next they'll ask: "Non-standardized features?" → Each weight gets its own penalty proportional to its feature's energy, which is ridge after standardizing. And dropout must be off at prediction time (model.eval()).

Drill 20 · The summer-heavy pilot

Back to the bike-share pilot: 80% of its 30 training days fell in the spring–summer half of the year and 20% in the autumn–winter half, uniformly within each half; the model will forecast every day of next year equally. "What importance weight does each summer and winter day get, and how many effective days of deployment-like data does the pilot hold?"

Concept tested: importance weights and ESS on the running example (Card 17).

• Answer: w = p_{\text{test}}/p_{\text{train}} = 0.5/0.8 = 0.625 for summer days and 0.5/0.2 = 2.5 for winter days; \text{ESS}/n = 1/(0.8 \times 0.625^2 + 0.2 \times 2.5^2) = 0.64, so the 30 days are worth 19.2 (verified by simulation: 0.640).
• Intuition: the winter days are scarce and each speaks for four times as many deployment days as a summer day; leaning on a few rows costs a third of the sample.
• Why: \text{ESS} = (\textstyle\sum w_i)^2/\textstyle\sum w_i^2 \approx n/\mathbb{E}_{\text{train}}[w^2], since \mathbb{E}_{\text{train}}[w] = 1.

\mathbb E_{\text{train}}

average over the pilot's training days

w

a training day's importance weight: 0.625 in summer, 2.5 in winter

n

the number of pilot days (30)

• Next they'll ask: "Does weighting help here?" → Only where the model is wrong. At n = 200 the misspecified line improves from 0.314 to 0.293 (a model trained on uniform days gets 0.290), while the well-specified cubic barely moves (0.0980 → 0.0977). A domain classifier separates pilot days from deployment days with AUC 0.65 and estimates the weights at 0.667 and 2.27.

Error-source cheat sheet

Estimate traps (the model is fine, the number about it is wrong): CV pessimism (Drill 15), the lucky minimum (Drill 14), correlated folds (Drill 17), heavy-tailed importance weights (Drill 13), leakage across the split (Q34) and post-selection p-values (Q31).

Rapid fire — the day before

One line each. Cover the right column and answer out loud; anything you hesitate on, go back to its full question above.

Review schedule: go through this table the day after you study, then 3 days later, a week later, and 3 weeks later. Any row you miss goes back to the start.