Gradient boosting builds trees in sequence, each fitting the residual (more precisely, a Newton step on a second-order approximation). XGBoost made that a system people could run on billion-scale tabular problems: a regularized objective (shrinkage, leaf penalties), handling of missing values as learned default directions, weighted quantile sketch for approximate splits, and cache-aware, out-of-core, distributed implementations. Chen and Guestrin won Kaggle mindshare because the software matched the math.
In 2026, LightGBM and CatBoost are cousins; neural tabular models sometimes win. The bake-off still starts with a boosted tree. If you jump to a 50M-parameter transformer on 20 numeric columns, you are skipping the paper that still pays the bills.
Regularization is first-class
The objective penalizes leaf weights and tree complexity, not only training loss. max_depth, min_child_weight, subsample, colsample_bytree, lambda, eta are how you stop memorizing IDs. Early stopping on a validation set is part of the method. If you set n_estimators=5000 with no early stop, you will overfit and then blame boosting.
Missing values: XGBoost can send NaNs down a learned branch. That is a feature. It is also a leakage path if "missing" encodes the label (a field only filled after the event). Audit missingness.
Systems choices change accuracy
Histogram / approximate splits vs exact greedy disagree on small data. Random seeds and column sampling disagree more than people admit. Pin versions. Distributed training must split rows without leaking groups (time, user). A random row split on a user-keyed table is not an eval.
Monotone constraints and interaction constraints exist because production models need to not reverse a known causal slope. Use them when legal or domain rules demand it; they are in the spirit of a controllable tree ensemble, not a black net.
A worked leakage
You include customer_lifetime_days that is computed with the label date. XGBoost finds a perfect split. AUC 0.99. Production AUC 0.51. Feature importance looked "reasonable." Boosted trees are extremely good at leakage. They are not at fault. Your split is.
Failure modes
One-hot exploding high-cardinality IDs; use native categorical handling or target encoding carefully.
Class imbalance without scale_pos_weight or the right metric (use PR-AUC).
Training on shuffled time series.
Comparing to an untuned MLP and declaring deep learning stronger.
Calibration and monotonicity
Boosted trees can be overconfident. If you ship probabilities, use a validation-set calibrator (Platt or isotonic) and never fit it on the train AUC set. Monotone constraints are how you encode "risk cannot fall as this credit variable rises" without a custom net. They reduce some interactions; that is the point. Export to a portable format (JSON, treelite) and run a golden-row test in the serving language. Python AUC is not a C++ scorer.
For ranking tasks use the ranking objectives rather than forcing a binary label. For count data, Poisson-style objectives exist. The default reg:squarederror on a 0/1 label is a common lazy mismatch. Pick the objective that matches the label semantics.
What you can borrow
- Start tabular problems with regularized gradient-boosted trees and early stopping.
- Treat missingness, sampling, and column subsample as core knobs, not extras.
- Pin the system (histogram vs exact, version) because it is part of the model.
- Hunt leakage before you hunt architecture.
- Reach for nets when you have unstructured data, huge interaction depth that trees cannot express, or a need to share a representation with other modalities.