Training Pipelines

Training Pipelines#

A model once shipped at a major ride-sharing company that looked exceptional in offline evaluation. AUC was up, NDCG was up, the team celebrated. Six weeks later, someone noticed that a feature used during training was computed using future trip data that wouldn't exist at serving time. The model had learned to cheat. Production metrics had been quietly degrading the entire time, and the offline eval never caught it because it was poisoned by the same leakage.