How to measure model success in production

Short-term targets and more out-of-time samples.

Work with the business and your data to develop relevant short-term targets. For example, if a sale takes 6 months after first-contact and the sale is the modeled target, set first-contact as short-term target for measurement. Set other steps in the funnel as a short-term target. Then run your analysis with these. Similar to predicting a charge-off of a loan. If a model is used to give a loan and the loan is 2 years, we do not wait 2 years from model deployment to get the first day of measurements. That could be a lot of bad loans. Perhaps half of the people who typically charge off do so within 12 months. And then 75% charge off within 20 months. You need to examine what is relevant and how predictive those short-term targets are of the modeled target. Use your training data and SMEs.

Also, if your process takes time, then you did not use the most up-to-date data in modeling. For example, if a loan has a 24 month modeled target, the model could not have used the latest 24 months, since that does not have a mature target (assuming this is not a survivor model - then this paragraph does not hold). But now the model can be evaluated on that data as each month progresses. The model did not score that data for action, but the model can be evaluated. Perhaps some loans would not have been given with the new model, but this is better than waiting 2 years to determine that the model is bad.