When it comes to using machine learning (ML) models in your business operations, one of the important questions to ask are: Why use a machine learning model? And if it has a positive effect on your business, how much better is it compared to normal, i.e. not using a model?
To answer these questions, this is where the Gain chart and Lift chart, collectively known as Gain-Lift charts, will help us.
First, a brief primer: supervised ML models for binary classification are trained to predict probabilities for an event, e.g., the probability of a customer buying a product is predicted as 71% (or 0.71). Using 50% as the passing mark, we can then label this customer as a positive event (i.e., a buyer). So, if there are 100 customers being processed by the model, each one of them will have their own predicted probability and be labeled as buyers or non-buyers.
With this covered, let’s continue on to Gain charts.
Gain Chart
The Gain chart represents how much percentage of the events of interest (e.g., product buyers, credit card defaulters, etc.) the ML model can cover with its predictions.
Now, imagine a scenario where our marketing or communications department needs to target potential credit card defaulters to encourage them to pay their bills, before they default. But if we send mailers to all cardholders, both defaulters and non-defaulters, we’re wasting our resources. How do we reach more of the card defaulters and less of the others? By using our ML model.
Look at the orange line in the chart below. It represents the percentage of Class 1, that is credit card defaulters who did not make their bill payments, as predicted by the ML model. The blue line represents Class 0 (non-defaulters). And the x-axis represents the percentage of cardholders in our sample: 0.2 refers to 20%, 0.4 to 40%, and so on until 1.0 refers to 100%.
Using our ML model, if we target the top 20% of the cardholders (sorted by their predicted probabilities of defaulting), we can cover 42% of the defaulters. Compare this to a blind approach (where we randomly choose who to contact), and only 20% of defaulters are reached.
If we choose the top 40%: refer to the 0.4 mark on the x-axis, and you can see that on the orange line, we will cover 60% of defaulters. Compared to normal/random (40% cover), isn’t this a better use of your limited marketing resources?
How to create a Gain chart
- Arrange events by their predicted probability from highest to lowest.
- For each decile: Count number of positive events (column D)
- Cumulative sum of positive events (column E) / Total positive events (450) = Gain (column F)
Gain chart is cumulative and adds up to 1 or 100%.
For this example, we see that if we target the top 50%, we can get 88% of the events of interest (potential buyers, cases, etc.) compared to baseline (random) approach which reaches only 50%.
Besides the percentage of the population covered, we can also gauge the effect of the model in relative terms: e.g., how many times is it better than normal without using a model. How much ‘lift’ does it provide on results?
Lift Chart
Back to the example on credit card defaulters, the lift chart below shows that at 20% of the population, using the ML model will reach 2.2 times more defaulters than baseline.
In the gain chart above, at 20% population, we get 42% of overall defaulters compared to 20% by non-model approach. That is about a multiple of 2. So, the lift chart is just another way of calculating the relative effectiveness of using a ML model compared to not using one.
How to create a Lift chart
- Arrange events by their predicted probability from highest to lowest.
- For each decile: Count number of positive events
- Cumulative sum of positive events with model (column E) / Cumulative sum of events (column C) = Lift (column G)
For this example, at 20% population, the ML model is 2.3 times better at identifying or reaching the customers of interest.
Conclusion
Gain charts and lift charts are two ways of calculating how a ML model can help your business to better identify or target customers (or events of interest), especially in marketing contexts.
How to improve your predictive model?
Not all data are equal. You can have better predictions by having more relevant data to train your model, e.g., data that relates to the responsiveness of your customers. Like how often do they open and click on links in your marketing emails? What is their NPS score?
Besides demographic data about your customers, behavioral data such as your customers’ interactions with your product or with your marketing efforts, can give a stronger indication of their potential to buy.
Make your words mean something 