Workflow, splits, and leakage

1. Intuition

Splits. You train on one portion of the data, tune choices on a second (validation), and report final performance once on a third (test). The test set only estimates production performance if it differs from training the way production will:

• Random split: fine when examples are independent and identically distributed (i.i.d.).
• Stratified split: random, but preserving class proportions; important when one class is rare.
• Time split: train on the past, test on the future. Required for forecasting or any system where tomorrow's data informs nothing about yesterday.
• Group split: all rows of the same user, patient, or device go into the same split. Otherwise the model can recognize the entity rather than learn the pattern.

Leakage happens when training uses information that won't be available at prediction time:

• Target leakage: a feature recorded after the outcome ("refund issued" when predicting fraud).
• Train-test contamination: preprocessing (scaling, imputation, feature selection) fit on all data, or duplicates across splits.
• Group/temporal leakage: the same entity or future information appears on both sides of the split.

Baselines. Before any model, measure what a trivial rule achieves: predict the mean, the majority class, last week's value. A model that can't beat it isn't learning anything useful.

2. Visualization

Two measurements from the script (synthetic data):

The last two rows use labels that are random, unrelated to the features. A random split still reports 90% accuracy, because each patient's ten nearly identical rows land in both train and validation.

3. The math

Symbols

What a test score estimates

The approximation holds only if test examples are drawn like production examples and played no role in building $f$. Every leak breaks the second condition; a mismatched split breaks the first.

K-fold cross-validation

Split the training data into $K$ folds; train on $K-1$, validate on the held-out fold, rotate, and average. It uses data efficiently for model selection, but the folds must respect the same rules as the final split (time-ordered folds for time series, group folds for grouped data).

Worked example: is this feature legal?

Predicting at 9:00 on day $T$ whether a delivery will be late:

4. Implementation

You will build a preprocessing pipeline from scratch (imputer, scaler, feature selector, model), a k-fold cross-validator, and then use them to manufacture 90% accuracy out of pure noise by fitting one step in the wrong place. Seeing the fake score is the fastest way to stop trusting unexplained ones.

Build it from scratch

Every preprocessing step has two phases. fit learns statistics (medians, means, standard deviations) and transform applies them. The rule that prevents leakage is that fit only ever sees training rows; validation and test rows are transformed with the training statistics.

class MedianImputer:
    def fit(self, X, y=None):
        self.medians_ = np.nanmedian(X, axis=0)           # learned from TRAINING rows only
        return self

    def transform(self, X):
        return np.where(np.isnan(X), self.medians_, X)

class Standardizer:
    def fit(self, X, y=None):
        self.mean_, self.std_ = X.mean(axis=0), X.std(axis=0) + 1e-12
        return self

    def transform(self, X):
        return (X - self.mean_) / self.std_

Predict: a test row has a feature value 3 standard deviations above the training mean. After transform, is it exactly 3.0, or recomputed from the test set's own mean?

Some steps look at the labels. A feature selector that keeps the columns most correlated with y is supervised, so it can leak the labels it was fit on.

class TopKByCorrelation:
    """Keep the k features most correlated with y. It looks at y, so it can leak."""
    def __init__(self, k):
        self.k = k

    def fit(self, X, y):
        Xc, yc = X - X.mean(0), y - y.mean()
        corr = np.abs(Xc.T @ yc) / (np.linalg.norm(Xc, axis=0) * np.linalg.norm(yc) + 1e-12)
        self.keep_ = np.argsort(-corr)[: self.k]
        return self

    def transform(self, X):
        return X[:, self.keep_]

A pipeline chains the steps so they are fit together, on the same rows, every time. Passing the whole pipeline into cross-validation means each fold refits every step on its own training part.

class Pipeline:
    def __init__(self, *steps):
        self.transforms, self.model = steps[:-1], steps[-1]

    def fit(self, X, y):
        for t in self.transforms:
            X = t.fit(X, y).transform(X)                   # fit, then pass the transformed data on
        self.model.fit(X, y)
        return self

    def predict(self, X):
        for t in self.transforms:
            X = t.transform(X)                             # transform only: reuse training statistics
        return self.model.predict(X)

K-fold cross-validation from section 3: shuffle once, cut into k folds, train a fresh model on k − 1 of them, score the held-out one, average.

def kfold_accuracy(make_model, X, y, k=5, seed=0):
    idx = np.random.default_rng(seed).permutation(len(y))
    folds = np.array_split(idx, k)
    scores = []
    for i in range(k):
        te = folds[i]
        tr = np.concatenate([folds[j] for j in range(k) if j != i])
        model = make_model().fit(X[tr], y[tr])
        scores.append(np.mean(model.predict(X[te]) == y[te]))
    return float(np.mean(scores))

Now the trap. 100 rows, 5,000 random features, coin-flip labels. Select the top 20 features on all 100 rows, then cross-validate.

    Xn = rng.normal(size=(100, 5000))
    yn = rng.integers(0, 2, 100)
    keep = TopKByCorrelation(20).fit(Xn, yn).keep_         # WRONG: selection saw every label
    leaky = kfold_accuracy(LogisticRegression, Xn[:, keep], yn)
    honest = kfold_accuracy(lambda: Pipeline(TopKByCorrelation(20), LogisticRegression()), Xn, yn)

Predict: the labels are random, so the true accuracy of any model is 50%. What do leaky and honest come out to?

Use the library

The production version is the same shape: ColumnTransformer routes numeric and categorical columns to their own steps, make_pipeline chains them, and the cross-validation splitters encode the split rules from section 3.

from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GroupKFold, TimeSeriesSplit, cross_val_score, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.dummy import DummyClassifier

# Stratified hold-out split (i.i.d. data)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)

# Every preprocessing step lives INSIDE the pipeline, so it is fit on training folds only
pre = ColumnTransformer([
    ("num", make_pipeline(SimpleImputer(strategy="median"), StandardScaler()), numeric_cols),
    ("cat", OneHotEncoder(handle_unknown="ignore"), categorical_cols),
])
model = make_pipeline(pre, LogisticRegression(max_iter=1000))
baseline = cross_val_score(DummyClassifier(strategy="most_frequent"), X_train, y_train, cv=5).mean()
score = cross_val_score(model, X_train, y_train, cv=5).mean()

# Grouped and temporal data: use the matching splitters
cross_val_score(model, X, y, cv=GroupKFold(5), groups=user_ids)
cross_val_score(model, X_time_sorted, y_time_sorted, cv=TimeSeriesSplit(5))

What scikit-learn adds: handle_unknown="ignore" for categories first seen in production, stratified folds by default for classifiers, and one object you can pickle so serving runs exactly the transforms training ran (the cure for training-serving skew).

Run it

From guide/:

uv run code/02-ml-workflow/pipeline_scratch.py

missing values: 193 of 2000
scratch pipeline 5-fold accuracy: 0.748
noise labels: selection outside CV 0.900, selection inside CV 0.560
sklearn Pipeline, selection inside CV: 0.590
sklearn Pipeline on the real-signal data: 0.750
checks pass: leakage inflates noise to look like signal

All data is synthetic. Full script: code/02-ml-workflow/pipeline_scratch.py. The target-leak and group-leak demonstrations and the figure are in code/02-ml-workflow/workflow.py.

5. Engineering

Before training. Write the problem statement: decision, prediction unit, target definition, prediction time, metric, and baseline. Version the dataset. Do EDA: missing values, duplicates (especially near-duplicates across splits), outliers, label balance, time coverage.

During training. Use pipelines so preprocessing is part of the model. Track experiments (code version, data version, configuration, seed, metrics). Look at the test set once, at the end.

After training. Error analysis: slice performance by segment, time, and input type; read the worst errors. Then deployment (batch scoring versus an online API), and monitoring of input distributions, prediction distributions, and delayed ground-truth performance.

Reproducibility. A result you can't regenerate from data version, code version, config, and seed should not ship.

Watch out

Leakage smells. A score far better than published work or a strong baseline; one feature with overwhelming importance; near-perfect validation scores on a hard problem; a feature whose name contains "status," "final," "outcome," or a timestamp after the event.

Common mistakes

• Random splits on time series or grouped data.
• Scaling, imputing, or selecting features before splitting.
• Re-using the test set to make decisions until it is effectively a validation set.
• Skipping the baseline, so nobody knows whether 0.82 is good.
• No plan for monitoring, so drift is discovered by customers.

6. Knowledge check

Practice exercise. You predict next-month churn for each customer using data up to the end of this month. Your dataset has 24 monthly snapshots per customer. Describe a split that is both time-correct and group-correct.

Implementation challenge. Take any classification dataset, deliberately add a leaky feature (the label plus noise), and show the inflated cross-validation score. Then write a check that flags any single feature whose solo cross-validated score is suspiciously close to the full model's.

Summary

• The workflow is frame, collect, explore, split, baseline, train, evaluate, deploy, monitor.
• Split the way production differs: random for i.i.d., time for forecasting, group for entities.
• Leakage (target, contamination, group, or temporal) inflates scores; ask whether each feature exists at prediction time.
• Keep preprocessing in pipelines; beat a baseline; look at the test set once.

Next: Overfitting and the bias-variance trade-off

Related: Classification metrics · Reliability, cost, and observability

Parts of this lesson are adapted from AI Engineering from Scratch by Rohit Ghumare (MIT license, notice): ML Pipelines.

Interview angle