How to Design a Score KPI

A common need in data is to translate a collection of outcome variables into a "Score" of some kind that can be used to identify areas of opportunity. For example, a relevance score for online shopping so a user who is shopping can receive recommendations that are valuable to them. If you have a collection of variables you want to be able to sort on in aggregate, what you need is a score! Having worked on several of these kinds of problems, I have a general methodology that allows me to very efficiently develop an intuitive score.

The first step is typically to identify a list of candidate input variables. Initially this list will be just brainstormed items typically gathered through interviews. To make this more intuitive, we can imagine you are trying to create a customer value score. If you want to score customers, you might care about a variety of things:

• Average purchase dollars

• Median purchase dollars

• Total number of historical purchases

• Average days between orders

• Standard deviation of days between orders

• Number of complaints

• Number of reviews

Once you have gathered a list of input variables, define a total target score range. For example, 0 to 100 or -100 to +100 are some common ranges. With the range defined, we can allocate the total points available to the different variables (called "weights"). For example, 20 points to the average purchase dollars, 5 to median purchase dollars, and so on.

With the variables selected and the priorities between the different variables, we have one final step to be able to combine all the inputs for a single customer and get a total "Score." Namely, transforming the raw input data into something more manageable. My approach is to take the input, and truncate it to a meaningful range by designing lower and upper bounds. We then scale it from 0 to 1 and multiply it by the weight. Each weighted, scaled, truncated, input is then summed with all the others.

For example, we might notice that the highest average purchase dollars is $1000, but only 2% of customers have an average purchase dollars above $150. On the lower end, we may see that only 5% of customers have an average purchase dollars below $50. Using these values, we can truncate the range so any customer with average purchase dollars below $50 gets a score of 0, and anyone with an average purchase dollars above $150 gets a score of 20. By truncating the inputs we are able to get meaningful differences between different customers for a single input, while keeping many positive features of linearity.

Overall, this approach produces a simple, intuitive, and implementable score. It also allows the relevant team to have productive conversations around what metrics matter most and how we should quantify those differences.