Kaggle Playground Competitions
Competition ML run as a controlled experiment: a five-model committee (LightGBM, XGBoost, CatBoost, GRU, RealMLP) trained fold-parallel on A100s, blended by Nelder-Mead and stacked reaching 0.951 ROC-AUC on F1 pit-stop prediction, with every fashionable 'AUC trick' ablated before it ships.
Skills involved
What This Is
Submissions to Kaggle's Playground Series, run as controlled experiments rather than notebook archaeology. Every competition gets the same treatment: full EDA before a single model is trained, a documented feature pipeline, a multi-model committee, and an ablation requirement before any "trick" makes it into the baseline.
The repo currently holds two competitions end to end S6E3 (telecom churn, binary classification) and S6E5 (F1 pit-stop prediction, ROC-AUC).
S6E5 — Will This Driver Pit On The Next Lap?
Binary prediction of PitNextLap from F1 telemetry, scored on ROC-AUC. This is where the infrastructure got serious.
Training runs on Modal. Five A100-40GB workers in parallel, one per fold, via modal.starmap. Before the folds spawn, the orchestrator runs Optuna (6 min LightGBM, 4 min XGBoost) to tune hyperparameters. Each fold then trains six models:
- LightGBM and XGBoost on the Pipeline A tree features
- GRU (PyTorch) on sequences grouped by driver × race × stint
- RealMLP three ways: on Pipeline A, on raw Pipeline B features, and on the two concatenated
After the folds return, the orchestrator assembles OOF predictions, runs a Nelder-Mead blend with six warm starts to find ensemble weights, then trains a LightGBM L2 stacker on top of the OOF predictions plus raw features. Everything commits to a Modal volume with a run registry, so any run is reproducible and comparable to any other.
The feature pipeline (163 tree features + 14 categorical). Priors computed on the training fold only global, driver and circuit pit rates, compound stint percentiles, lap-bin pit rates. Rolling 3/5/10-lap windows per driver × race for lap-time mean, standard deviation, delta and position change. Safety-car proxies, gap-to-car-ahead and undercut viability, fuel-corrected degradation, tyre-life and race-phase bins.
Adversarial validation AUC is 0.502 train and test are indistinguishable, so local CV is trustworthy. That number is the licence to trust everything else.
What Killed The Score, And Why That's The Interesting Part
AUC dropped from 0.951 to roughly 0.8 when pseudo-labelling and target encoding went in together.
The dataset has a 96% train-test overlap of Driver-Race-Year groups. Pseudo-labels on repeated-identity synthetic data carry identity information they inflate OOF CV without generalising at all. The fold-safe target encoding implementation was correct; combined with pseudo-labels it still contributed to distribution shift.
CUDA-compiled LightGBM was also tried and abandoned: it added real build complexity and did not improve scores. CPU LightGBM on 16 vCPUs is sufficient. The repo now carries an explicit rule not to change the 0.951 baseline without a controlled ablation.
The AUC Myth Table
ROC-AUC is purely rank-based it only cares about the order of predicted scores. That single fact decides which commonly-cited tricks actually move anything:
- Hard-negative isolation / weighting real lever, implemented
- Rank-based ratio and difference features real lever, feeding the MLP
- Class weights over SMOTE already in place,
scale_pos_weight ≈ 4 - Ranking loss (lambdarank / pairwise) needs query groups; this is row-wise binary classification with no natural group, and forcing one breaks
predict_proba - Label-noise / Isolation-Forest filtering risky on this data, not adopted
- Platt / isotonic calibration a myth for AUC: monotone transforms cannot change rank order, so they cannot change the metric
Hard-negative weighting is a two-pass GBDT: a fast 300-tree LightGBM scores training rows in-sample, then negatives it ranks confidently high and positives it ranks low get a sample-weight ramp up to 3×. This sharpens exactly the overlap zone where AUC is decided. It's off by default per the repo's ablation rule.
S6E3 — Telecom Churn
The earlier, simpler pipeline, and a good baseline for what the standard playbook gets you.
- Features: service counts, tenure-derived terms (average monthly charge, charge deviation, charges per service, tenure squared and binned), binary service flags, and categorical interactions (contract × internet, contract × payment, partner × dependents, senior × contract)
- Encoding: label-encoded on combined train + test so no unseen labels appear at inference
- Models: LightGBM, XGBoost and CatBoost, all at 3000 estimators / lr 0.03 with real regularisation, under 5-fold stratified CV
- Ensemble: grid search over blend weights on OOF predictions
What's Next
- Cross-run blending as a first-class step rather than a post-hoc script
- Multi-seed everything (currently 3407 and 42) for stability
- Rank-space geometric and power-mean blending as label-free proxies for LB gain
Last updated Jun 1, 2026
← Back to all projects