An 0.98 AUC on a problem where a good model should score 0.80 is not good news. It's usually the first symptom of data leakage — information about the target sneaking into your features through a channel that won't exist when the model runs in production. Leakage is dangerous precisely because it makes your offline metrics look better, not worse, so the normal instinct to trust a strong validation score works against you.

The most common source: leakage through time

If a feature is computed using data from after the prediction point — even by a few hours — the model learns a shortcut that doesn't exist at inference time. A classic version: predicting whether a loan will default using a feature like "number of collection calls made," when collection calls only happen after a loan is already flagged as troubled. The feature is a near-perfect predictor precisely because it's downstream of the outcome, not upstream of it.

This isn't always obvious from the column name. "Account status at time of data pull" sounds innocent until you realize the data pull happened after the outcome was already known for closed accounts.

Leakage through aggregate statistics computed on the whole dataset

Normalizing a feature using the mean and standard deviation of the entire dataset — including the test set — leaks test-set information into training. It's a small leak individually, but it's why validation scores computed this way tend to run optimistic versus true production performance:

python
# Leaky: statistics computed across train + test together
scaler = StandardScaler().fit(X_full)
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Correct: fit only on train, apply to test
scaler = StandardScaler().fit(X_train)
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)

The same principle applies to imputation values, target encodings, and feature selection — any statistic derived from data must be derived only from the training fold, then applied unchanged to validation and test.

If your dataset has near-duplicate rows — the same customer appearing multiple times, or multiple images of the same object from different angles — and a random split puts some copies in train and others in test, the model can effectively memorize the specific entity rather than learning the general pattern. The fix is grouping the split by entity, so all rows for a given customer or object land entirely in one side:

python
from sklearn.model_selection import GroupShuffleSplit

gss = GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
train_idx, test_idx = next(gss.split(X, y, groups=df["customer_id"]))

A checklist that catches most leakage before it costs you

  • Does any feature use data timestamped after the prediction point?
  • Were any dataset-wide statistics (mean, std, target encodings) computed before the train/test split, rather than after?
  • Could the same entity appear in both train and test due to a naive random split?
  • Is there a feature that, on inspection, is suspiciously predictive on its own — one column with unusually high importance relative to everything else?

That last point is the fastest practical detector: run feature importance after every training run, and treat any single feature dominating importance as worth manually justifying, not celebrating. In a legitimate model, predictive power is usually spread across several features; a single feature explaining almost everything is far more often a leak than a genuine signal.

Leakage doesn't announce itself with an error message — it shows up as a model that quietly underperforms its validation score the moment it meets real, chronologically honest data. Building the leakage checks into your pipeline, rather than trusting a single suspiciously good number, is what keeps that gap from surprising you after launch.