Teams routinely spend weeks tuning a model architecture and a single afternoon auditing the labels it's trained on, which is roughly backwards in terms of expected return. A model architecture change might buy you a point or two of accuracy. Fixing systematically wrong labels can buy you far more, because every hour spent training on bad labels is an hour spent teaching the model a pattern that isn't real — and no amount of tuning fixes a target that's wrong.
Label noise doesn't average out the way people assume
The comforting assumption is that random label noise "averages out" and the model learns the true pattern anyway, given enough data. This is true only for genuinely random noise at moderate rates, and it's rarely what real label noise looks like. Real label noise is usually systematic: a particular annotator consistently mislabels a specific edge case, a labeling instruction was ambiguous for a whole category of examples, or an automated labeling heuristic fails identically on a specific data segment every time. Systematic noise doesn't average out — it teaches the model a specific, reproducible wrong pattern for that segment, and you'll only notice as a persistent blind spot in production that no amount of additional random noise-free data elsewhere fixes.
Finding likely label errors without manually reviewing everything
You don't need to hand-review an entire dataset to find where labels are probably wrong. Training a model, then looking at where it disagrees with the label with high confidence, is a strong signal — either the model has learned something genuinely wrong, or the label is:
model.fit(X_train, y_train)
probs = model.predict_proba(X_train)
predicted = probs.argmax(axis=1)
confidence = probs.max(axis=1)
suspects = (
(predicted != y_train) & (confidence > 0.9)
)
review_queue = X_train[suspects]A high-confidence disagreement between model and label, on training data the model has already seen, is disproportionately likely to be a label error rather than a hard example — a genuinely hard example tends to produce low-confidence, uncertain predictions, not confident disagreement. This doesn't replace a real audit, but it turns "review everything" into "review this specific, much smaller, high-yield subset first."
Consistency matters as much as correctness
Two annotators can each be individually reasonable and still disagree with each other on a meaningful fraction of examples, especially on genuinely ambiguous cases near a category boundary. Measuring inter-annotator agreement — Cohen's kappa for two annotators, Fleiss' kappa for more — tells you whether your labeling task itself is well-specified before you spend money scaling up collection:
| Kappa range | Interpretation |
|---|---|
| < 0.4 | Poor agreement — labeling instructions likely ambiguous |
| 0.4 - 0.6 | Moderate — worth investigating disagreement patterns |
| 0.6 - 0.8 | Good — some genuine edge-case ambiguity remains |
| > 0.8 | Strong agreement — task is well-specified |
Low kappa isn't just a data quality problem, it's information: it usually means the category boundary itself needs a clearer written definition, and no amount of additional annotator training fixes an ambiguous instruction.
Distribution mismatch between labeled data and production data
A dataset can have perfectly correct labels and still be low quality for your purposes, if the examples it contains don't represent what the model will actually see in production. Historical fraud data collected before a major product change, or labeled images that skew toward well-lit studio conditions when production traffic is mostly low-light phone cameras, both produce a model that's accurate on its own training distribution and disappointing in the real one. This is a data quality problem, not a modeling problem, and no architecture change fixes it — the fix is collecting or reweighting data to actually match production conditions.
Where to actually spend the marginal hour
Given a fixed amount of time before a model needs to ship, the return on an hour spent auditing high-confidence label disagreements, checking inter-annotator agreement on ambiguous categories, or comparing training data distribution against production traffic tends to exceed the return on an additional hour of hyperparameter tuning, especially once tuning has already found a reasonable configuration. Model quality has a ceiling set by label quality; past a certain point, better tuning just gets you closer to that ceiling faster — it doesn't raise it.