How often do I need to run ML and AI?

Comments

  • AMuresan
    AMuresan

    This depends on a few factors.

    Firstly there are two ways to "run" ML. One is to train a machine learning model to fit new data and learn new patterns. The other, is to use a trained model to score new observation and get predictions.

    For scoring: This should be run as often as your business intends to act on the results.

    Note that if the action is passive, i.e. when the customer comes to store we use the last score to determine the offer they may be eligible for, then some analysis should be done to determine what interval of scoring maintains model performance. Normally I would recommend the same scoring frequency as the data snapshotting frequency, and no longer than the target action window.

    For training: Here I would suggest setting up a monitoring system of the predictions being made by the scoring process and retraining the model when performance begins to degrade.

    Note since the model may be being used to change the behaviour of the customers it is predicting for it is important to look at the performance on control groups. For example, if a model is meant to predict customers canceling, and the business action is to offer them a retention offer, then the people the model things are most likely to cancel are the ones getting an offer and hopefully not leaving. As a result performance would look bad. Bu to see if the model would have been right without intervention, performance must be measured on a hold out control group.

Categories