Marketing Metrics Aren’t Baseball Scores

Lester Wunderman is called “the Father of Direct Marketing” — not because he was the first one to put marketing offers in the mail, but because he is the one who started measuring results of direct channel efforts in more methodical ways. His marketing metrics are the predecessors of today’s measurements.

Lester Wunderman is called “the Father of Direct Marketing” — not because he was the first one to put marketing offers in the mail, but because he is the one who started measuring results of direct channel efforts in more methodical ways. His marketing metrics are the predecessors of today’s measurements.

Now, we use terms like 1:1 marketing or digital marketing. But, in essence, data-based marketing is supposed to be looped around with learnings from results of live or test campaigns. In other words, playing with data is an endless series of learning and relearning. Otherwise, why bother with all this data? Just do what you gut tells you to do.

Even in the very beginning of the marketer’s journey, there needs to a step for learning. Maybe not from the results from past campaigns, but something about customer profiles and their behaviors. With that knowledge, smart marketers would target better, by segmenting the universe or building look-alike or affinity models with multiple variables. Then a targeted campaign with the “right” message and offers would follow. Then what? Data players must figure out “what worked” (or what didn’t work). And the data journey continues.

So, this much is clear; if you do not measure your results, you are really not a data player.

But that doesn’t mean that you’re supposed to get lost in an endless series of metrics, either. I sometimes see what is commonly called “Death by KPI” in analytically driven organizations. That is a case where marketers are too busy chasing down a few of their favorite metrics and actually miss the big boat. Analytics is a game of balance, as well. It should not be too granular or tactical all of the time, and not too high in the sky in the name of strategy, either.

For one, in digital marketing, open and clickthrough rates are definitely “must-have” metrics. But those shouldn’t be the most important ones for all, just because all of the digital analytics toolsets prominently feature them. I am not at all disputing the value of those metrics, by the way. I’m just pointing out that they are just directional guidance toward success, where the real success is expressed in dollars, pounds and shillings. Clicks lead to conversions, but they are still a few steps away from generating cash.

Indeed, picking the right success metrics isn’t easy; not because of the math part, but because of political aspects of them, too. Surely, aggressive organizations would put more weight onto metrics related to the size of footprints and the rate of expansion. More established and stable companies would put more weight on profitability and various efficiency measures. Folks on the supply side would have different ways to measure their success in comparison to sales and marketing teams that must move merchandise in the most efficient ways. If someone is dedicated to a media channel, she would care for “her” channel first, without a doubt. In fact, she might even be in direct conflicts with fellow marketers who are in charge of “other” channels. Who gets the credit for “a” sale in a multi-channel environment? That is not an analytical decision, but a business decision.

Even after an organization settles on the key metrics that they would collectively follow, there lies another challenge. How would you declare winners and losers in this numbers game?

As the title of this article indicates, you are not supposed to conclude one version of creative beat the other one in an A/B test, just because the open rate was higher for one by less than 1%. This is not some ballgame where a team becomes a winner with a walk-away homerun at the bottom of the 11th inning.

Differences in metrics should have some statistical significance to bear any meaning. When we compare heights of a classroom full of boys, will we care for differences measured in 1/10 of a millimeter? If you are building a spaceship, such differences would matter, but not when we measure the height of human beings. Conversion rates, often expressed with two decimal places, are like that, too.

I won’t get too technical about it here, but even casual decision-makers without any mathematical training should be aware of factors that determine statistical significance when it comes to marketing-related metrics.

  • Expected and Observed Measurements: If it is about open, clickthrough and conversion rates, for example, what are “typical” figures that you have observed in the past? Are they in the 10%to 20% range, or something that is measured in fractions? And of course, for the final measure, what are the actual figures of opens, clicks and conversions for A and B segments in test campaigns? And what kind of differences are we measuring here? Differences expressed in fractions or whole numbers? (Think about the height example above.)
  • Sample Size: Too often, sample sizes are too small to provide any meaningful conclusions. Marketers often hesitate to put a large number of target names in the no-contact control group, for instance, as they think that those would be missed revenue-generating opportunities (and they are, if the campaign is supposed to work). Even after committing to such tests, if the size of the control group is too small, it may not be enough to measure “small” differences in results. Size definitely matters in testing.
  • Confidence Level: How confident would you want to be: 95% or 90%? Or would an 80% confidence level be good enough for the test? Just remember that the higher the confidence level that you want, the bigger the test size must be.

If you know these basic factors, there are many online tools where you can enter some numbers and see if the result is statistically significant or not (just Google “Statistical Significance Calculator”). Most tools will ask for test and control cell sizes, conversion counts for both and minimum confidence level. The answer comes out as bluntly as: “The result is not significant and cannot be trusted.”

If you get an answer like that, please do not commit to a decision with any long-term effects. If you want to just declare a winner and finish up a campaign as soon as possible, sure, treat the result like a baseball score of a pitchers’ duel. But at least be aware that the test margin was very thin. (Tell others, too.)

Here’s some advice related to marketing success metrics:

  • Always Consider Statistical Significance and do not make any quick conclusions with insufficient test quantities, as they may not mean much. The key message here is that you should not skip the significance test step.
  • Do Not Make Tests Too Complicated. Even with just 2-dimensional tests (e.g., test of multiple segments and various creatives and subject lines), the combination of these factors may result in very small control cell sizes, in the end. You may end up making a decision based on less than five conversions in any given cell. Add other factors, such as offer or region, to the mix? You may be dealing with insignificant test sizes, even before the game starts.
  • Examine One Factor at a Time in Real-Life Situations. There are many things that may have strong influences on results, and such is life. Instead of looking at all possible combinations of segments and creatives, for example, evaluate segments and creatives separately. Ceteris paribus (“all other factors held constant,” which would never happen in reality, by the way), which segment would be the winner, when examined from one angle?
  • Test, Learn and Repeat. Like any scientific experiments, one should not jump to conclusions after one or two tests. Again, data-based marketing is a continuous loop. It should be treated as a long-term commitment, not some one-night stand.

Today’s marketers are much more fortunate in comparison to marketers of the past. We now have blazingly fast computers, data for every move that customers and prospects make, ample storage space for data, affordable analytical toolsets (often for free), and in general, more opportunities for marketers to learn about new technologies.

But even in the machine-driven world, where almost everything can be automated, please remember that it will be humans who make the final decisions. And if you repeatedly make decisions based on statistically insignificant figures, I must say that good or bad consequences are all on you.

Marketing Success Metrics: Response or Dollars?

It’s tempting to ask about whether marketing success metrics should be response rates or money. But you don’t need to ask marketers what they want. Basically, they want everything.

It’s tempting to ask about whether marketing success metrics should be response rates or money. But you don’t need to ask marketers what they want. Basically, they want everything.

They want big spenders who also visit frequently, purchasing flagship products repeatedly. For a long time (some say “lifetime”). Without any complaint. Paying full price, without redeeming too many discount offers. And while at it, minimal product returns, too.

Unfortunately, such customers are as rare as a knight in white armor. Because, just to start off, responsiveness to promotions is often inversely related to purchase value. In other words, for many retailers, big spenders do not shop often, and frequent shoppers are often small item buyers, or worse, bargain-seekers. They may just stop coming if you cut off fat discount deals. Such dichotomy is quite common for many types of retailers.

That is why a seasoned consultants and analysts ask what brand leaders “really” want the most in marketing success metrics. If you have a choice, what is more important to you? Expanding the customer base or increasing the customer value? Of course, both are very important goals — and marketing success metrics. But what is the first priority for “you,” for now?

Asking that question upfront is a good defensive tactic for the consultant, because marketers tend to complain about the response rate when the value target is met, and complain about the revenue size when goals for click and response rates are achieved. Like I said earlier, they want “everything, all the time.”

So, what does a conscientious analyst do in a situation like this? Simple. Set up multiple targets and follow multiple marketing success metrics. Never hedge your bet on just one thing. In fact, marketers must follow this tactic as well, because even CMOs must answer to CEOs eventually. If we “know” that such key marketing success metrics are often inversely correlated, why not cover all bases?

Case in point: I’ve seen many not-so-great campaign results where marketers and analysts just targeted the “best of the best” segment — i.e., the white rhinoceros that I described in the beginning — in modeled or rule-based targeting. If you do that, the value may be realized, but the response rate will go down, leading to disappointing overall revenue volume. So what if the average customer value went up by 20%, when only a small group of people responded to the promotion?

A while back, I was involved in a case where “a” targeting model for a luxury car accessory retailer tanked badly. Actually, I shouldn’t even say that the model didn’t work, because it performed exactly the way the user intended. Basically, the reason why the campaign based on that model didn’t work was the account manager at the time followed the client’s instructions too literally.

The luxury car accessory retailer carried various lines of products — from a luxury car cover costing over $1,000 to small accessories priced under $200. The client ordered the account manager to go after the high-value target, saying things like “who cares about those small-timers?” The resultant model worked exactly that way, achieving great dollar-per-transaction value, but failing at generating meaningful responses. During the back-end analysis, we’ve found that the marketer indeed had very different segments within the customer base, and going only after the big spenders should not have been the strategy at all. The brand needed a few more targets and models to generate meaningful results on all fronts.

When you go after any type “look-alikes,” do not just go after the ideal targets in your head. Always look at the customer profile reports to see if you have dual, or multiple universes in your base. A dead giveaway? Look at the disparity among the customer values. If your flagship product is much more expensive than an “average” transaction or customer value in your own database, well, that means most of your customers are NOT going for the most expensive option.

If you just target the biggest spenders, you will be ignoring the majority of small buyers whose profile may be vastly different from the whales. Worse yet, if you target the “average” of those two dichotomous targets, then you will be shooting at phantom targets. Unfortunately, in the world of data and analytics, there is no such thing as an “average customer,” and going after phantom targets is not much different from shooting blanks.

On the reporting front — when chasing after often elusive targets — one must be careful not to get locked into a few popular measurements in the organization. Again, I recommend looking at the results in every possible way to construct the story of “what really happened.”

For instance:

  • Response Rate/Conversion Rate: Total conversions over total contacted. Much like open and click-through rate, but I’d keep the original denominator — not just those who opened and clicked — to provide a reality check for everyone. Often, the “real” response rate (or conversion rate) would be far below 1% when divided by the total mail volume (or contact volume). Nonetheless, very basic and important metrics. Always try to go there, and do not stop at opens and clicks.
  • Average Transaction Value: If someone converted, what is the value of the transaction? If you collect these figures over time on an individual level, you will also obtain Average Value per Customer, which in turn is the backbone of the Lifetime Value calculation. You will also be able to see the effect of subsequent purchases down the line, in this competitive world where most responders are one-time buyers (refer to “Wrestling the One-Time Buyer Syndrome”).
  • Revenue Per 1,000 Contacts: Revenue divided by total contacts multiplied by 1,000. This is my favorite, as this figure captures both responsiveness and the transaction value at the same time. From here, one can calculate net margin of campaign on an individual level, if the acquisition or promotion cost is available at that level (though in real life, I would settle for campaig- level ROI any time).

These are just three basic figures covering responsiveness and value, and marketers may gain important intelligence if they look at these figures by, but not limited to, the following elements:

  • Channel/Media
  • Campaign
  • Source of the contact list
  • Segment/Selection Rule/Model Score Group (i.e., How is the target selected)
  • Offer and Creative (hopefully someone categorized an endless series of these)
  • Wave (if there are multiple waves or drops within a campaign)
  • Other campaign details such as seasonality, day of the week, daypart, etc.

In the ultimate quest to find “what really works,” it is prudent to look at these metrics on multiple levels. For instance, you may find that these key metrics behave differently in different channels, and combinations of offers and other factors may trigger responsiveness and value in previously unforeseen manners.

No one would know all of the answers before tests, but after a few iterations, marketers will learn what the key segments within the target are, and how they should deal with them discriminately going forward. That is what we commonly refer to as a scientific approach, and the first step is to recognize that:

  • There may be multiple pockets of distinct buyers,
  • Not one type of metrics will tell us the whole story, and
  • We are not supposed to batch and blast to a one-dimensional target with a uniform message.

I am not at all saying that all of the popular metrics for digital marketing are irrelevant; but remember that open and clicks are just directional indicators toward conversion. And the value of the customers must be examined in multiple ways, even after the conversion. Because there are so many ways to define success — and failure — and each should be a lesson for future improvements on targeting and messaging.

It may be out of fashion to say this old term in this century, but that is what “closed-loop” marketing is all about, regardless of the popular promotion channels of the day.

The names of metrics may have changed over time, but the measurement of success has always been about engagement level and the money that it brings.