Feature Engineering at Scale

Feature Engineering at Scale#

A team at a major e-commerce company spent months building a recommendation model. Offline metrics looked excellent. They shipped it. Within days, click-through rates were worse than the model it replaced. The culprit wasn't the model architecture or the training data. It was the features. The batch pipeline computing "user's recent browsing history" ran every 24 hours. In production, the model was scoring requests against features that were sometimes 23 hours stale. The "recent" browsing history it saw during training looked nothing like what it saw in production.