Measuring Customer Engagement: It’s Not Easy and It Takes Time

Here’s what’s easy: Measuring the effect of individual engagements like Web page views, email opens, paid and organic search clicks, call center interactions, Facebook likes, Twitter follows, tweets, retweets, referrals, etc. Here’s what’s hard: Understanding the combined effect of your promotions across all those channels. Many marketers turn to online attribution methods to assign credit for all or part of an individual order across multiple online channels. es as the independent variables.

Here’s what’s easy: Measuring the effect of individual engagements like Web page views, email opens, paid and organic search clicks, call center interactions, Facebook likes, Twitter follows, tweets, retweets, referrals, etc.

Here’s what’s hard: Understanding the combined effect of your promotions across all those channels.

Many marketers turn to online attribution methods to assign credit for all or part of an individual order across multiple online channels. Digital marketing guru Avinash Kaushik points out the strengths of weaknesses of various methods in his blog, Occam’s Razor in “Multichannel Attribution: Definitions, Models and a Reality Check” and concludes that none are perfect and many are far from it.

But online attribution models look to give credit to an individual tactic rather than measuring the combined effects of your entire promotion mix. Here’s a different approach to getting a holistic view of your entire promotion mix. It’s similar to the methodology I discussed in the post “Use Market Research to Tie Brand Awareness and Purchase Intent to Sales,” and like that methodology, it’s not something you’re going to be able to do overnight. It’s an iterative process that will take some time.

Start by assigning a point value to every consumer touch and every consumer action to create an engagement score for each customer. This process will be different for every marketer and will vary according to your customer base and your promotion mix. For illustration’s sake, consider the arbitrary assignments in the table in the media player, at right.

Next, perform this preliminary analysis:

  1. Rank your customers on sales volume for different time periods
    —previous month, quarter, year, etc.
  2. Rank your customers on their engagement score for the same periods
  3. Examine the correlation between sales and engagement
    —How much is each point of engagement worth in sales $$$?

After you’ve done this preliminary scoring, do your best to isolate customers who were not exposed to specific elements of the promotion mix into control groups, i.e., they didn’t engage on Facebook or they didn’t receive email. Compare their revenue against the rest of the file to see how well you’ve weighted that particular element. With several iterations of this process over time, you will be able to place a dollar value on each point of engagement and plan your promotion mix accordingly.

How you assign your point values may seem arbitrary at first, but you will need to work through this iteratively, looking at control cells wherever you can isolate them. For a more scientific approach, run a regression analysis on the customer file with revenue as the dependent variable and the number and types of touches as the independent variables. The more complete your customer contact data is, the lower your p value and the more descriptive the regression will be in identifying the contribution of each element.

As with any methodology, this one is only as good as the data you’re able to put into it, but don’t be discouraged if your data is not perfect or complete. Even in an imperfect world, this exercise will get you closer to a holistic view of customer engagement.

Sex and the Schoolboy: Predictive Modeling – Who’s Doing It? Who’s Doing it Right?

Forgive the borrowed interest, but predictive modeling is to marketers as sex is to schoolboys. They’re all talking about it, but few are doing it. And among those who are, fewer are doing it right. In customer relationship marketing (CRM), predictive modeling uses data to predict the likelihood of a customer taking a specific action. It’s a three-step process.

Forgive the borrowed interest, but predictive modeling is to marketers as sex is to schoolboys.

They’re all talking about it, but few are doing it. And among those who are, fewer are doing it right.

In customer relationship marketing (CRM), predictive modeling uses data to predict the likelihood of a customer taking a specific action. It’s a three-step process:

1. Examine the characteristics of the customers who took a desired action

2. Compare them against the characteristics of customers who didn’t take that action

3. Determine which characteristics are most predictive of the customer taking the action and the value or degree to which each variable is predictive

Predictive modeling is useful in allocating CRM resources efficiently. If a model predicts that certain customers are less likely respond to a specific offer, then fewer resources can be allocated to those customers, allowing more resources to be allocated to those who are more likely to respond.

Data Inputs
A predictive model will only be as good as the input data that’s used in the modeling process. You need the data that define the dependent variable; that is, the outcome the model is trying to predict (such as response to a particular offer). You’ll also need the data that define the independent variables, or the characteristics that will be predictive of the desired outcome (such as age, income, purchase history, etc.). Attitudinal and behavioral data may also be predictive, such as an expressed interest in weight loss, fitness, healthy eating, etc.

The more variables that are fed into the model at the beginning, the more likely the modeling process will identify relevant predictors. Modeling is an iterative process, and those variables that are not at all predictive will fall out in the early iterations, leaving those that are most predictive for more precise analysis in later iterations. The danger in not having enough independent variables to model is that the resultant model will only explain a portion of the desired outcome.

For example, a predictive model created to determine the factors affecting physician prescribing of a particular brand was inconclusive, because there weren’t enough dependent variables to explain the outcome fully. In a standard regression analysis, the number of RXs written in a specific timeframe was set as the dependent variable. There were only three independent variables available: sales calls, physician samples and direct mail promotions to physicians. And while each of the three variables turned out to have a positive effect on prescriptions written, the “Multiple R” value of the regression equation was high at 0.44, meaning that these variables only explained 44 percent of the variance in RXs. The other 56 percent of the variance is from factors that were not included in the model input.

Sample Size
Larger samples will produce more robust models than smaller ones. Some modelers recommend a minimum data set of 10,000 records, 500 of those with the desired outcome. Others report acceptable results with as few as 100 records with the desired outcome. But in general, size matters.

Regardless, it is important to hold out a validation sample from the modeling process. That allows the model to be applied to the hold-out sample to validate its ability to predict the desired outcome.

Important First Steps

1. Define Your Outcome. What do you want the model to do for your business? Predict likelihood to opt-in? Predict likelihood to respond to a particular offer? Your objective will drive the data set that you need to define the dependent variable. For example, if you’re looking to predict likelihood to respond to a particular offer, you’ll need to have prospects who responded and prospects who didn’t in order to discriminate between them.

2. Gather the Data to Model. This requires tapping into several data sources, including your CRM database, as well as external sources where you can get data appended (see below).

3. Set the Timeframe. Determine the time period for the data you will analyze. For example, if you’re looking to model likelihood to respond, the start and end points for the data should be far enough in the past that you have a sufficient sample of responders and non-responders.

4. Examine Variables Individually. Some variables will not be correlated with the outcome, and these can be eliminated prior to building the model.

Data Sources
Independent variable data
may include

  • In-house database fields
  • Data overlays (demographics, HH income, lifestyle interests, presence of children,
    marital status, etc.) from a data provider such as Experian, Epsilon or Acxiom.

Don’t Try This at Home
While you can do regression analysis in Microsoft Excel, if you’re going to invest a lot of promotion budget in the outcome, you should definitely leave the number crunching to the professionals. Expert modelers know how to analyze modeling results and make adjustments where necessary.