Why Modeling Beats Rule-Based Segmentation

I have been talking about “employing all available data” for targeting and customer insights for some time now. So allow me to pick a different bone today. Let’s forget the data part, and talk about the methodology. When machines can build models super-fast, aversion to modeling only limits the users. After all, I am not asking any marketers to get a degree in statistics. I am just asking them to consider modeling techniques.

I cringe when I hear “rule-based” segments are sitting on top of so-called state-of-the-art campaign engines. This is year 2018 A.D. It’s the age of abundant data with an ample number of tools and options to harness their true powers. And marketers are still making up rules now? It’s time for marketers to embrace modeling.

I wonder what most of the rules marketers are using are made of. Recency? Certainly, but how recent is recent enough?

Frequency? Sure, why not? The more the merrier, right? But in what timeframe? Are you counting transactions, orders or items? Or just some “events”?

Monetary? Hmm, that’s tricky. Are we using an individual-level lifetime total amount, value of the last transaction, average spending per transaction, average spending amount per year, or what? Don’t tell me you don’t even have individual-level summary data. No customer is just a reflection of her last transaction.

Actually, if a company is using some RFM (Recency, Frequency, Monetary Value) data for targeting, that is not so bad. At least it’s taking a look at what actually happened in terms of monetary transactions, not just clicks and page views, along with basic demographic data.

I have been talking about “employing all available data” for targeting and customer insights for some time now. So allow me to pick a different bone today. Let’s forget the data part, and talk about the methodology. When machines can build models super-fast, aversion to modeling only limits the users. After all, I am not asking any marketers to get a degree in statistics. I am just asking them to consider modeling techniques, as this data industry has moved forward from the days when some basic RFM rule sets used to get a passing grade.

Let’s look at the specific reasons why marketers should consider modeling techniques more seriously and ditch rule-based segmentation.

Reason No. 1: Variable Selection

We are surrounded by data, as every move that anyone makes is digitized now. When you describe a buyer, you may need to evaluate hundreds, if not thousands, of data points. Even if you are just using simple set of demographic data without any behavioral data, we are talking about over 100 variables to consider out of the gate.

Let’s say you want to build a rule to find a good segment for the sale of luxury cruises. How would you pick the most predictable variable for that one purpose? Income and age? That is not a bad start, but that is like using just two colors out of a crayon box containing 80 colors.

Case in point: Do you really believe that the main difference between luxury cruisers and luxury car buyers is “income”? Guess what, those buyers are all rich. You must dig much deeper than that.

Marketers often choose variables that they can easily understand and visualize. Unfortunately, the goal of the targeting exercise should be effectiveness of targeting, not easy comprehension by the marketer.

We often find obscure variables in models. They may “seem” obscure, as a human being would never have instinctively picked them. But mathematics doesn’t care for our opinions. In modeling, variables are picked for their predictive power, nothing else. The bonus is that this is exactly how new patterns are discovered.

We hear tidbits such as “People who tend to watch more romantic comedies are more likely to rent cars over the weekend,” “Aggressive investors are less likely to visit family restaurants” or “High-value customers for a certain teenage apparel company are more likely to be seasonal buyers with high item counts per customer, but relatively lower transaction counts.”

These are the contributing factors found through vigorous mathematical exercises, not someone’s imagination or intuition. But they should always make sense in the end (unless of course, there were errors). Picking the right predictor is indeed the most important step in modeling.

Reason No. 2: Weight Factor

Let’s say that by chance, a user stumbled upon a set of useful predictors of certain customer behavior. Let’s go back to the last example of the teenage apparel company’s high-value customer model. In that one sentence, I listed: seasonality (expressed in number of transactions by month, regardless of year), number of item counts per customer (with time limits, such as past 36 months), and number of transactions per customer.

In real life, there would be a far greater number of variables that would pass the initial variable selection process. But for simplicity’s sake, let’s just review these three variables.

Now tell me, which one of these three variables is the most important predictor of this high-value customer model? (Please don’t say they are all equally important.) Model scores are made of selected variables multiplied by the weight of each, as not all predictors carry the same level of predictability. Some may even be “negatively” correlated to the ideal behavior that we are going after. In this example alone, we saw that the number of items was positively related to the high value, while the number of transactions are negatively related. When investigating further about this “strange” correlation, we found out that most of the high-value customers are trained by the marketer to wait for a big sale, and then buy lots of items in one transaction.

The main trouble with the “rule-based” segmentation or targeting exercise is that human beings put arbitrary weight (or importance) on each variable, even if “the right” variables were picked — mostly by chance — in the first place.

The modeling process reveals the actual balance among all important predictors, with the sole purpose of maximizing predictability. Conversely, I have never met a person who can “imagine” the dynamics of two or three variables, let alone 10 to 20 (the typical number of variables in models).

Forget about the recent emergence of machine learning; with or without human statisticians, modeling techniques have been beating rudimentary rules by end-users for decades. If solely left to humans, the No. 1 predictor of any human behavior would be the income of the target. But that is just a reflection of human perception and a one-dimensional way of looking at a complex composition of human behavior. You don’t believe you can explain the difference between a Lexus buyer and a Mercedes buyer with just income, do you?

Reason No. 3: Banding

Much of data are composed of numbers and figures. The rest of them are called categorical variables (i.e., data that cannot be added or subtracted, such as product category or channel description).

Let’s assume that income — not my first pick, as you can see — is found to be predictable for mid- to high-scale female accessory buyers. Surely, different ranges of income would behave differently in such models. If the income is too low, they won’t be able to afford such items. Too high, then the buyer may have moved on to even more expensive handbags. So, the middle ground may seem to be the ideal target. The trouble is that now you have to describe that middle group in terms of actual dollars. Exactly where does that ideal range begin and end? To make it even more complicated, what about regional biases in buying power? Can one set of banding explain the whole thing? We’ve gone way past any intuitive grouping.

Moving onto categorical variables, one of the most predictable variables in any B2B modeling is the SIC code. There are thousands of variations in any one field, and they are definitely not numbers (although they look like them). How would one go about putting them into ideal groups to describe the target (such as “loyal customers”)?

If you are selling expensive computer servers, one may put “Agricultural, Fishing and Mining” as a low priority group. Then, how about all those variations in huge groups, such as “Retail,” “Business Service” or “Manufacturing,” with hundreds of sub-categories? Let’s just say that I’ve never met a human being who went beyond the initial two-digit SIC code in their heads. Good luck creating an effective group with that one variable with rudimentary methods.

Grouping “values” that move together in terms of predictability is not simple. In fact, that is exactly why computers were invented. Don’t struggle with such jobs.

These are just a few reasons why we must rely on advanced modeling techniques to navigate through complex data. The benefits of modeling are plenty (refer to “Why Model?”). Compared to our gut feelings, statistical models are much more accurate and consistent. They also reveal previously unseen patterns in data. Because they are summarized answers to specific questions, users do not have to consider hundreds of factors, but just one model score at a time. In the current marketing environment, when things move at a light speed, who has time to consider hundreds of data points in real-time? Machine learning — leading to full automation — is just a natural extension of modeling.

Each model score is a summary of hundreds of contributing factors. “Responsiveness to email campaigns for a European cruise vacation” is a complex question to answer, especially when we all go through daily data overload. But if the answer is in the form of a simple score (say, one through 10), any user who understands “high is good, low is bad” can make a sound decision at the time of campaign execution.

Marketers already have ample amounts of data and advanced campaign tools. Running such machines with some man-made segmentation rules from the last century is a real shame. No one is asking marketers to become seasoned data scientists; they just need to be more open to advanced techniques. With firm commitments, we can always hire experts, or in the near future, machines that will do the mathematical jobs for us. But marketers must move out of old fashioned rule-based marketing first.

Deliver an Outstanding CX When Customers Aren’t Talking to You

Customers like self-service. This makes sense when you already have a relationship with a company and you’re just trying to execute a transaction as quickly and efficiently as possible. I do this with American Airlines, interacting with a kiosk to check in rather than an agent.

I just saw the headline, “Consumers Like Self-service More Than Associate Interaction, Reveals Survey.” The gist of the research is if consumers have a choice they’re more likely to tap self-service technology vs. interacting with a retail sales associate, according to a SOTI survey.

This makes sense when you already have a relationship with a company and you’re just trying to execute a transaction as quickly and efficiently as possible. I do this with American Airlines, interacting with a kiosk to check in rather than an agent. I use the self-check-out at Harris Teeter (a grocery store). And I use an app on my phone or an ATM more frequently than a teller at my bank. I also have more than a 20-year relationship with each of these brands.

If they don’t know me after 20 years, they aren’t listening. And, after 20 years, if I’m not pleased with an experience, I’ll let them know about it.

If I didn’t already have a relationship with the airline, the bank or the grocery store, I don’t think I’d trust their other distribution channels. I certainly wouldn’t be familiar with them, they’d be less convenient to use, and I likely would not use them — it would no longer be the most efficient way for me to do what I need to get done.

  • What are you doing to engage your customers and provide an outstanding customer experience?
  • Are you providing a product or service that addresses a problem or concern of your customer?
  • Do you make it easy for your customer to buy?

Do you, or your customer-facing employees, try to engage your customer in a conversation along these lines:

  • What’s driving you to buy our product?
  • What problem are you trying to solve?
  • Have you used our product before?
  • How’s our service?
  • What can we do to improve our product or service?

A lot has been written recently about how customers don’t want to have a relationship with a brand. However, a brand is not a person.

I believe customers do want to have a relationship with a representative of the brand. Someone with whom they can share a comment or suggestion and know that it will be heard and acted upon.

Typically, the people interacting with your customers are your employees.

Do you encourage your employees to engage customers in a conversation to learn more about their needs and wants? What they’re happy with and what can be improved?

Your customers probably don’t want to talk to you because you’ve shown no interest in talking to them. They may have no emotional connection to your brand and don’t care whether you succeed or not.

I have an 11-year relationship with Chipotle and fill out an online feedback form after every visit because I do have an emotional connection with the brand and I want to see it delivering an outstanding customer experience.

You may send a customer satisfaction survey or mine sales data, but have you, or your employees, had a conversation with the customer?

People like to do business with people they know, like and trust. People also don’t care how much you know until they know how much you care about them. This is done person-to-person, not by analyzing data. This is how you build an emotional connection between a customer and a brand.

This is a function of having empathy and being sincerely concerned about why the customer is buying your product vs. that of your competitors — B2B or B2C.

Customers want relationships with brands and product and service providers on their terms. They want to be able to talk with a real person with some knowledge and authority if they have a question, suggestion or complaint.

The customer wants what they want when they want it, and it’s up to the service provider to figure out what it is.

Empower employees to find out what your customers and prospects want to know and how they want to find out about it.

By finding out how different customers want to learn about your products and services, you’ll be able to differentiate and segment your customers; thereby, providing them with more relevant information of value.

You must provide your customers the options they want in order to keep them satisfied. If you don’t, they will find someone else who will. In order to understand your customers’ needs and wants, you need to have a relationship with them, so you’ll be able to fulfill their needs on an ongoing basis.

If customers don’t want to talk to you, it may be because they don’t have a need right now, or they’re pressed for time. However, they are not saying they never want to talk to you or give you feedback.

Don’t stop trying to have a relationship with your customers. Don’t stop trying to gather real customer insights. Be there for your customers when they’re ready to talk.  If you’re not, they’ll go to someone who is.

The Keyword in ‘Customer Journey’ Is ‘Customer’

The keyword in “Customer Journey” is “customer,” not “journey.” In fact, in this Omni-channel world, the word “journey” doesn’t even do much justice to what that journey study should be all about; there is no simple linear timeline about any of it anymore.

The keyword in “Customer Journey” is “customer,” not “journey.” In fact, in this omnichannel world, the word “journey” doesn’t even do much justice to what that journey study should be all about; there is no simple linear timeline about any of it anymore.

We often think about the customer journey in this fashion: awareness, research, engagement, transaction, feedback and, ever-important, repeat-purchase. This list is indeed a good start.

However, if you look at this list as a consumer, not as a marketer, do you personally go through all of these steps in this particular order? On a conceptual level, yes, but in the world where everyone is exposed to over five types of screens and interactive devices every day, old-fashioned frameworks based on linear timelines don’t always hold water.

I, as a consumer, often do research using my phone at the place of purchase. I may feel rewarded even before any actual purchase. I may provide feedback about my “experience” before, during or after a transaction. And being a human being with emotions, my negative feedback may not be directly correlated to my loyalty to the brand. (Actually, I am writing this piece while flying on an airline with which I have a premiere status, and to which I often provide extremely negative reviews.)

People are neither linear nor consistent. Especially when we are connected to devices with which we research, interact, transact and complain anytime, anywhere. The only part that is somewhat linear is when we put something in the shopping basket, make a purchase, and keep or return the item. So, this timeline view, in my opinion, is just a good guideline. We need to look at the customer journey from the customer’s angle, as well.

Understanding customer behavior is indeed a tricky business, as it requires multiple types of data. If we simplify it, we may put the key variables into three major categories. For a 3-dimensioal view (as I often do in a discussion), put your left hand out and assign each of the following dimensions to your thumb, index finger and the middle finger:

  • Behavioral Data: What they showed interest in, browsed, researched, purchased, returned, subscribed to, etc. In short, what they actually did.
  • Demographic Data: What they look like, in terms of demographic and geo-demographic data, such as their age, gender, marital status, income, family composition, type of residence, lifestyle, etc.
  • Attitudinal Data: Their general outlook on life, religious or political beliefs, priorities in life, reasons why they like certain things, purchase habits, etc.

One may say these data types are highly correlated to each other, and more often than not, they are indeed highly correlated. But not exactly so, and not all the time. Just because one keeps purchasing luxury items or spending time and money on expensive activities, and he is enjoying a middle-age life style living in posh neighborhood, we can’t definitely claim that he is politically conservative. Sometimes we just have to stop and ask the person.

On top of that, what people say they do and what they actually do are often not the same. Hence, these three independent axis of data types to describe a person.

If we have all three types of data about a person, prediction of that person’s intention — or his journey for commercial purposes or otherwise — will become incredibly accurate. But, unfortunately for marketers, asking “everyone about everything” simply isn’t feasible.

Even the most thorough survey is based on a relatively small sample response. One great thing about traditional primary research is that we often get to know who the respondents are. On the other hand, if we rely on social media to “listen,” we get to have opinions from far more people. But the tricky part there is that we don’t get to know who is speaking, as PII (personally identifiable information) is heavily guarded by the social media handlers. Basically, it isn’t easy to connect the dots completely when it comes to attitudinal data. (Conversely, connecting the dots between the behavioral data and demographic data is much simpler, provided with a decent data collection mechanism.)

Now let’s go back to the timeline view of the customer journey for an initial framework. Let’s list the key items in a general order for a simpler breakdown (though things may not be totally linear nowadays), and examine types of data available in each stage. The goal here is to find the point of entry for this difficult task of understanding the “end-to-end” customer journey in the most comprehensive way.

Listing typical data types associated with these entries:

  1. Awareness: Source (where from), likes/followings, clicks other digital trails, survey results, social media data, etc.
  2. Research: Browsing data, search words/search results, browsing length, page/item views, chats, etc.
  3. Engagement: Shopping basket data, clicks, chats, sales engagements, other physical trails at stores, etc.
  4. Transaction: Product/service (items purchased), transaction date, transaction amount, delivery date, transaction channel, payment method, region/store, discounts, renewals, cancelations, etc.
  5. Feedback: Returns, complaints and resolutions, surveys, social media data, net promotor score, etc.
  6. Repeat-purchase: Transaction data summarized on a customer level. The best indicator of loyalty.

Now, looking back at the three major types of data, let’s examine these data related to journey stages in terms of the following criteria:

  • Quality: Are data useful for explaining customer behaviors and predicting their next moves and future values? To explain their motives?
  • Availability: Do you have access to the data? Are they readily available in usable forms?
  • Coverage: Do you have the data just for some customers, or for the most of them?
  • Consistency: Do you get to access the data at all times, or just once in a while? Are they in consist forms? Are they consistently accurate?
  • Connectivity: Can you connect available data on a customer level? Or are they locked in silos? Do you have the match-key that connects customer data regardless of the data sources?

With these criteria, the Ground Zero of the most useful source in terms of understanding customers is transaction data. They are usually in the most usable formats, as they are mostly numbers and figures around the product data of your business. Sometimes, you may not get to know “who” made the purchase, but in comparison to other data types, hands-down, transaction data will tell you the most compelling stories about your customers. You’ll have to tweak and twist them to derive useful insights, but the field of analytics has been evolving around such data all along.

If you want to dig deeper into the “why” part of the equation (or other qualitative elements), you would need to venture into non-transactional, more attitudinal data. For the study of online journey toward conversion, digital analytics is undoubtedly in its mature stage, though it only covers online behaviors. Nonetheless, if you really want to understand customers, start with what they actually purchased, and then expand the study from there.

We rarely get to have access to all of the behavioral, demographic, and attitudinal data. And under those categories, we can think of a long list of subcategories, as well. Cross all of that with the timeline of the journey — even a rudimentary one — and having readily usable data from all three angles at all stages is indeed a rare event.

But that has been true for all ages of database marketing. Yes, those three key elements may move independently, but what if we only get to have one or two out of the three elements? Even if we do not have attitudinal data for a customer’s true motivation of engagement, the other two types of data — behavioral, which is mostly transaction and digital data, and demographic data, which can be purchased in large markets like the U.S. — can provide at least directional guidance.

How do you think the political parties target donors during election cycles? They at times have empirical data about someone’s political allegiance, but many times they “guess” using behavioral and demographic data along with modeling techniques, without really “asking” everyone.

Conversely, if you get to have access to attitudinal data of “some” people with known identities, we can build models to project such valuable information to the general population, only using a “common” set of variables (mostly demographic data). For instance, we may only get a few thousand respondents revealing their sentiment toward a brand or specific stances (for example, being a “green” conscience customer). We can use common demographic variables to project such a tendency to everyone. Would such a “bridging technique” be perfect? Like I mentioned in the beginning, no, not always. Will having such inferred information be much better than not knowing anything at all? Absolutely.

Without a doubt, understanding the customer journey is an important part of marketing. How else would you keep them engaged at all stages of purchases, leading them to loyalty?

The key is not to lose focus on the customer-centric side of analytics. Customer journey isn’t even perfectly sequential anymore. It should be more about “customer experience” regardless of the timeline. And to get to that level of constant relevancy, start with the known customer behaviors, and explain away “what works” in all channel engagements for each stage.

Channel or stage-oriented studies have their merits, but they won’t lead marketers to a more holistic view of customers. After all, high levels of awareness and ample clicks are just good indicators of future conversions; they do not instantly guarantee loyalty or profitability. Transaction data tend to reveal more stable paths to longevity of customer relationship.

You may never get to have explicit measurements of loyalty consistently; but luckily for us, customers vote with their money. Unlock the transaction data first, and then steadily peel away to the “why” part.

I am not claiming that you will obtain the answer to the “causality” question with just behavioral data; but for marketing purposes, I’d settle for “highly correlated” elements anytime. Because marketing activities can happen successfully without pondering upon the “why” question, if actionable shortcuts to loyalty are revealed through sold transaction data.

How Direct Mail Is Your Little Engine That Could

Because direct mail is the little engine that could for your marketing funnel, it can sustain you through troubled times. Direct mail is most powerful when used in a long-term, multi-touch plan. The average prospect needs to see your mail piece seven to 10 times before buying from you.

direct mail
“Mailboxes in ivy,” Creative Commons license. | Credit: Flickr by Ryan McFarland

Because direct mail is the little engine that could for your marketing funnel, it can sustain you through troubled times. Direct mail is most powerful when used in a long-term, multi-touch plan. The average prospect needs to see your mail piece seven to 10 times before buying from you. So a well-planned direct mail program includes multiple drops with various mailers and postcards. Then once the prospect makes a purchase from you, you move the consumer into your customer retention mail program. These types of programs are extremely effective and can be counted on to consistently generate sales.

Are you taking advantage of direct mail programs throughout the year? Do you mail consistently? Do you have a plan? Depending on what you are selling and who your customer base is, it will determine what your direct mail plan should be. The more data you are capturing on your customers, the better you will be able to target them with direct mail.

So what should a basic prospect direct mail plan look like?

  • List — Purchase a multi-use list of prospects based on what you know about your customers such as demographics, psychographics and more.
  • Message — Prospects need to learn who you are, what you do and see testimonials from current customers. You are trying to convert them to customers.
  • Offer — You need to create offers that will resonate with your prospects. What is in it for them?
  • Format — To be most effective, alternate formats for each mailing so that each prospect will get a letter, postcard and self-mailer over the course of your program. You can use formats more than once, but always make sure to add something fresh and new to each mailing. Sending the same thing over and over again does not get you the results you need.
  • Schedule — This will really vary depending on what you are selling, more expensive purchases are made less frequently vs. some items that need to be purchased all the time. The general rule of thumb is once a month to once every other month for high-ticket items and twice a month for more frequent purchases.

So what should a basic customer direct mail plan look like?

  • List Pull as much information as you have on each customer. You can use their purchase history to get your direct mail highly targeted.
  • Message — Customers should get messaging that is applicable to them and what they buy. You can suggest add-ons that complement what they have already bought or items that other people like them have purchased.
  • Offer — Customers love coupons on items that they buy. You can also give them special offers on new items they have not previously purchased from you but are likely to buy.
  • Formats — Just like prospects you should vary the formats of direct mail you are sending to customers.
  • Schedule — Customers should have a more scaled-back schedule than prospects. They know who you are and how to buy from you so send to them less frequently. We recommend at most once a month.

Are you ready to get started planning your ongoing direct mail campaigns? By constantly feeding your pipeline with your direct mail prospects and customers, your marketing funnel will always be generating sales. Get excited about your direct mail programs and create some really fun direct mail pieces. When you get creative you stand out more and get remembered. Make you direct mail campaigns real profit-drivers. Have you had a very successful long-term campaign? We would love to hear about it.

For Millennials, Direct Marketing Books Aren’t Catching Up

Lucy and Ethel on the Chocolate Factory conveyor belt faced a daunting task. That’s what it must like to try to create a current direct response marketing book.

direct marketing books
Direct marketing books | Credit: Chuck McLeester

Lucy and Ethel on the Chocolate Factory conveyor belt faced a daunting task. That’s what it must like to try to create a current direct response marketing book.

I’ve been teaching as an adjunct for more than 10 years — mostly advertising research and marketing courses. But only recently have I had the opportunity to teach a class devoted entirely to direct response. When I began to put the course together for Rowan University, I was looking for a general direct marketing book that students could acquire inexpensively (used on Amazon or another used book site) that I would supplement with additional resources. I came up empty.

The available direct marketing books rely heavily on mail as a medium with a lot of content about lists and crafting direct mail letters. I could only imagine the eye-rolls I’d be looking at standing in front of a group of Millennials talking about direct mail lists.

The standards I’ve relied on for years, Ed Nash’s “Direct Marketing” and Bob Stone’s “Successful Direct Marketing Methods” (updated by Ron Jacobs) haven’t been revised since 2000 and 2007, respectively. Lisa Spiller and Martin Baier had published a textbook for Pearson, but the third and most recent edition from 2010 is out of print. Some books, like Dave Shepherd’s “The New Direct Marketing” (1999) and Arthur Hughes’s “The Complete Database Marketer” are focused on database and response modeling, the precursors to algorithmic targeting. Richard Tooker’s “The Business of Database Marketing” is very practitioner-focused, and other titles are specific to subsets of direct marketing, like “Managing Customer Relationships” by Don Peppers and Martha Rogers (2011).

The newer online marketing books focus on driving clicks, web analytics and retargeting, but they don’t address the fundamental principles of allowable acquisition cost and customer lifetime value.

There’s nothing that brings together online marketing with traditional direct response principles.

I combed through my library of DR books, reached out to publishers and even purchased a few things on Amazon.

For a moment, and just a moment, I considered writing one. Then I realized that with the amount of new information coming at marketers every day, I would be stuffing chocolates in my mouth and under my hat — like Lucy and Ethel.

Data’s $20B Role in Marketing

Right on cue. My last blog post happened to discuss Europe’s forthcoming “Data Freeze.” Enter a new U.S. study that articulates just how large the use of data for smarter marketing really is stateside — to the tune of $20 billion plus.

Third Party Data Study - Selected Chart
Credit: Data & Marketing Association by Winterberry Group

Right on cue. My last blog post happened to discuss Europe’s forthcoming “Data Freeze.”

Enter a new U.S. study that articulates just how large the use of data for smarter marketing really is stateside — to the tune of $20 billion plus.

The Data & Marketing Association and Interactive Advertising Bureau’s Data Center of Excellence commissioned the “State of Data 2017” study [available as a download], conducted by Winterberry Group. According to the foreword:

“…marketers and publishers looking to become ‘data centric’ have had little choice but to embark on that titanic change effort without the benefit of clear and complete intelligence; the inherent complexity of data and its myriad applications has previously made accurate reporting — on how users are investing in data, putting it to work and evolving their marketing approaches in turn — too challenging to accurately compile.

“This report represents the first industry-wide effort to address that gap. By providing credible, practitioner-informed insight, we hope to demystify how U.S. companies are investing in audience data (and its associated support functions), helping practitioners benchmark their own spending against industry norms and establish a firmer basis for future investments.”

If 2018 will be the year of third-party data quality, this study perhaps underscores why: Third-party audience data spending will top nearly $10.1 billion this year, in omnichannel ($3.5 billion), transactional ($3.0 billion), digital ($2.8 billion), specialty ($0.9 billion) and identity categories ($0.6 billion). Another $10.1 billion will be spent on various data “activation” solutions, from integration, processing and hygiene ($4.3 billion); to hosting and management ($4.2 billion); to analytics, modeling and segmentation ($1.6 billion).

In short, marketers are investing heavily on knowing prospects and customers better — and communicating intelligently with them to meet demands and expectations. For more and more brands and organizations in both consumer and business-to-business markets, third-party data is essential in this process — online, offline and omnichannel. But it’s indeed complex.

The scope of the study includes commercially licensable data and/or audience segments, as well as third-party data solutions that seek to activate or apply any combination of first-, second- or third-party data. It does not include data for “insourced” product development, aggregated data for market research, data for custom audiences that bundled inside “walled gardens” of social media platforms and other publishers, and enterprise data usage not related to advertising, marketing and media.

The study is helpful in providing benchmarks for companies as they evaluate their own third-party data dynamics in advertising, marketing and media planning — but I can’t help appreciate this snapshot on a wider economic basis. Responsible data collection for more relevant engagement with customers is a $20 billion business – a substantial and likely growing slice of all ad and marketing spend. [Early next month, Winterberry Group’s Bruce Biegel will present firsthand a “2018 Media Outlook” for direct, digital and data — and how they compare to overall media spending.]

If CMOs increasingly are judged on business effectiveness, on how advertising and marketing performs in this context, then gaining prowess with data — including third-party data — is fast becoming table stakes. Building out data-driven marketing capabilities will serve them well.

Third-party data and activation is indeed fuel for consumer engagement and business growth. This reality — documented in this study — needs to be understood, recognized and respected far beyond the C-suite. But let’s start with the C-suite.

Data, Data Everywhere: Nary a Bargain to Find?

Stephen Yu’s recent and extremely thought-provoking piece on AI started me wondering once again about the dangers of data overload and whether we’ll ever really, really understand the purchasing decisions people make, how they make them and be able to track them accurately.

Data mining
“Big_Data_Prob,” Creative Commons license. | Credit: Flickr by KamiPhuc

Stephen Yu’s recent and extremely thought-provoking piece on AI started me wondering once again about the dangers of data overload and whether we’ll ever really, really understand the purchasing decisions people make, how they make them and be able to track them accurately.

Because today’s machines gobble data and — like my dog eats anything he can get jaws around — we marketers seem to search for more and more bytes in the hope that sifting through this mega data will hold the keys to the holy grail of maximum profitability. Perhaps it will. But as a disciple of Lester Wunderman, I can’t let go of his oft-expressed prescient warning that “Data is an expense. Knowledge is a bargain.”

Admittedly, when this was first expressed, data was one hell of a lot more expensive to keep and handle than it is today and shaking knowledge out of it was very difficult. But that’s hardly the point. Our trade press is now overflowing with titles like “Planning and Measuring Social Media Campaigns” (Sysomos), the “Email Marketing Metric You May Not Know” and unnumbered guides to the customer journey. But I’m still waiting for the definitive article that leaves all of the peripheral data by the side of the road and presents a usable and believable knowledge-based metric model to measure the cost of each step in the journey from awareness through to final purchase. In today’s multi-media environment, that’s the metric model we are all waiting for. Will we ever get it? Will AI provide it? I’m not so sure.

There is historically a different focus between top management whose attention is quite sensibly on macro numbers and operational marketers who know that it is the micro numbers that spotlight big opportunities. The ROMI, the return on the total marketing investment, is the bottom line for both: How much did we earn for how much marketing money invested? Simple.

But at what milestones in the customer journey did the momentum toward purchase increase and at what others did the potential customer take a turn away from purchase and why? That’s the type of data we need if we are to optimize our practice and it will surely impact the ROMI. Sadly in many cases, we will never know.

Recently, some of my Brazilian colleagues created a very strong email campaign as the first stage in persuading well-segmented prospects to clickthrough to a website to register interest and gain a price advantage in making a major purchase. The client reported that while the website was receiving a lot of activity, only a tiny fraction came as the expected clickthrough from the emails. The client was understandably angry and it didn’t make any sense.

Every adult Brazilian has a unique CPF number, which is regularly requested and used to identify the individual in financial transactions. It’s rather like an American Social Security number. Because my colleagues were fortunate enough to have the CPFs of the prospects to whom the emails had been sent and as registration on the website also required a CPF, it was a relatively easy task to compare the two groups to determine how many of the registrants had been sent the emails, even if they hadn’t availed themselves of the clickthrough option. It turned out to be a happily large percentage.

While research has been undertaken to determine why, any measurement of the relation of emails to registrations and their cost would have been both misguided and meaningless. If the marketers had decided to stop using the emails because, as they said, ”emails didn’t generate any response,” they would have been making a critical error.

Perhaps that’s a long way around the issue of just why, with all of the enormous data and sophisticated tools at our disposal, we just can’t develop a meaningful metric model that reliably tracks the prospect along the path to becoming a customer. And it argues that while AI will certainly add valuable knowledge, getting inside the head of a prospect and truly understanding his/her actions is a long way off.

Creating a Persona Menu (for You)

Personas are like menu items, each representing key characteristics of target customers that marketers need to know to push their products.

“93H,” Public Domain license. | Credit: Flickr by saul saulete

I have been writing about the importance of using modeling techniques for personalization for some time now (refer to “Personalization Is About the Person” and “Segments vs. Personas”). If I may summarize the whole idea down to a 15-second pitch:

  • We need modeling because we will never know everything about everybody, and;
  • Selfishly for marketers, it is much simpler to assign personas to product groups and related contents than to have to deal with an obscene amount of customer data and a long list of content details at the same time.

Simply, personas are like menu items, each representing key characteristics of target customers that marketers need to know to push their products.

One may say, “Hey, I just put in SKU-level data into some personalization engine!” To which, I must ask, “Do you also put in unrefined oil into your beloved automobile?” I didn’t think so. Not that ruining some personalization engine will break anyone’s heart. But it may annoy the heck out of your customers by treating them as extensions of their immediate purchases, not as living, breathing human beings.

I’ve actually met someone from a software company at a conference who claimed to be able to create hundreds of thousands of combinations of SKU-level transaction data and content data. If you have a few hundred thousand SKUs and tens of thousands of pictures and creative items, well, the number of combinations will be quite large. Not exactly the number of stars in the universe, but quite unmanageable, enough for marketers to just “let go” and leave it all to the machine on a default setting. So, even if someone automated the process of combining such data (with some built-in rules, I’m sure), how would any marketer – and recipients of messages – make sense out of it all?

That type of shotgun approach is the mother of all of those annoying “personalizations,” like offers of the very same items that you just purchased. For such rudimentary methods, it might actually be a great achievement to offer a yoga mat to someone who just bought a yoga mat. Hey, they are in the same category after all, categorically speaking, right?

The key to humanization of marketing messages is to make them about the customers, not about marketers, products or channels. And that kind of high-level personalization requires, well, a real human touch. That means, each block of information must be bite-sized so that human beings – i.e., marketers – can process and consume it easily.

When I first came to America (a long time ago), it wasn’t so easy to go through menu items in a typical diner. Too many items! How can I pick just “one” of those items that matches my appetite and mood of the day? Now imagine a menu that goes on for hundreds of thousands of lines. And you have to act fast on it, too.

Personas, or architypes as some may call them, are the bridges between obscene amounts of data points and yet another large set of pictures and content. The idea is to have a manageable number of personas to make it easier for us to match the right content to the right target.

I bet most content libraries are not crazy big, but large enough. But on that side, it is what it is. You will not cut out some valuable digital assets just because the inventory got big. So, we have to make the personal data – especially behavioral and transactional data – more compact to facilitate easy assignment, as in “Show this picture of a glass of red wine next to a juicy steak” to a persona called “Wine Enthusiast” or “Fine Dining.” The assignment itself would be as simple as saving a room for persona designation in the content library (if you don’t even have a content library, we need to talk).

Then, how would you come up with the right list of personas for “you”? Having done this a few times for many companies in various industries on a national level, I have some tips to share.

  1. Be Product-Centric: Anyone who has been reading my articles about personalization will be surprised by this one, as I have been screaming “customer-centric marketing” all along. But, in the end, we are doing all of this to sell more of our products to customers. Think about the products you want to push, then think about the types of characteristics that you would love to know about customers to push those products in a relevant way.

Trying to sell cutting-edge products? Then you may need personas such as “Early adopter.” Selling value-based items? You may want “Bargain-seekers.” Pushing travel items? Try “Frequent business traveler” or “Family vacation” personas. Dealing with high net-worth people? Well, go beyond simple income-select and try “Globetrotter,” “Luxury car,” “Heavy stock investor,” etc., depending on what you are selling. By the way, these luxury personas may or may not be related to one another, as human beings are much more complex than their income levels.

  1. Be Creative: Models can be built if you have data for “some” people who have actually behaved in a certain way to be used as targets. That limitation aside, you can be as creative you want to be.

For example, if you are in the telecommunications industry, expand the typical triple-play offering, and dig deeper into “why” people would need broadband service. Is it because someone is an “Avid gamer,” “Heavy VOIP user,” “Frequent international caller,” part of a “Big family,” “Home office worker” and/or “On-demand movie watcher”? If you can differentiate these traits, you don’t have to push broadband Internet services with brute force. You can now show reasons why they need over 100 megabits per second service.

If you are dealing with mostly female customers (who are, by the way, responsible for the bulk of economic activities on a national level), one can imagine categories that start with various health and beauty items, going all of the way to yoga and fitness personas. In between those, add any persona that is an ideal target for the products you are trying to sell, be it “Fashion enthusiast,” “Children’s interests,” “Gardening enthusiast,” “Organic food,” “Weight watchers,” Gourmet Cooking,” “Family entertainment,” etc., etc. The keys is to describe the buyer, not the product.

  1. Start Small, but be bolder as the list grows: In the beginning, you may have to prove that personalization using model-based personas really works. Yes, building a persona is as simple as building a propensity model (in essence, they are exactly those), but that doesn’t mean that you start the effort with 50 persons. Pick the product that you really want to push, or characteristics that you need to know in order to resonate with your core customers, and build a few personas as a starter (say five to 10). You may find some data limitations along the way, but as you go through the list, your team (or analytics partners) will definitely gain momentum.

Then you can be bold. I’ve seen retailers who routinely maintain over 100 personas for just one major product category. And I’ll bet that list didn’t grow that big overnight, either.

Also, when you are in an expansion mode, just add items when in doubt. Think about the users of those personas, not mathematical differences among models. Do you know the difference between Kung Pao Chicken and Diced Chicken with Hot Peppers? Just peanuts on top. But restaurants have them both because customers expect to see them.

Similarly, there may be only slight differences between “Conservative Investor” and “Annuity Investor” personas. But the users of those personas may grab one or the other because of their targeting need at the moment. Or whatever inspired their marketing spirit. Think in terms of user-friendliness, not mathematical purity.

  1. Do Not Go Out of Control: When I was leading a product development team in a prominent data compiling company in the U.S., our team developed about 140 personas covering the entire country for various behavioral categories, including investment, travel, sports (both active participation and being a fan of), telecomm, donation, politics, etc. One of our competitors tried to copy that idea, and failed miserably. Why? It had built too many models.

For instance, if you are building personas for the cruise industry in general, you may need just “Luxury cruise” and “Family cruise” for starters. Those are good enough for initial prospecting. Then, if you must get deeper into cross-selling for coveted “onboard spending,” then you may get into “Adventure-seeker,” “Family entertainment,” “Gourmet,” “Wine enthusiast,” “Shopping expedition,” “Luxury entertainment,” “Silver years,” “Young parents,” etc., for customization of offers.

My old copycats with too many models had developed separate models for “each” cruise fleet and brand. How were they going to use all of that? One brand at a time, with one company as a user group? Why not build a custom model as needed, then? Surely that would be more effective if the model is to target a specific brand or fleet. Anyway, my competitors ended up building a few thousand models, for any known brand out there in every industry, seriously limiting the chance those personas would be used by marketers.

As I mentioned in the beginning, this is about matching offers (or content) to the right people at the right time. If you go out of control, it will be very difficult to do that kind of match-making. If your persona list is just big for the sake of being big, well, how is that any different from using the raw data? You’ve got to know when to stop, too. The key is “not too small, and not too big,” for humans and machines alike.

  1. Update Periodically: Like any menu, persona lists go out of date. Some items may not have been used actively. Some may become obsolete as business models and core product lines go through changes. And models do go stale, as well. You may not have to review this all of the time, and there will be staple menu items, like spaghetti with meatballs in a restaurant. But it will be prudent to go through the menu once in awhile. If not because of the product, then because of people’s attitudes about it changing.
  2. Evangelize: It would be a shame if the data and analytics people did all of this work and marketers didn’t use it fully. These personas are in essence mathematical summaries of “lots of” data in compact forms. They can be used in targeting (for selecting the right target for specific product offers), and for personalization of offers and messages based on dominant characteristic of the target (e.g., show different pictures to “Adventure-seeker” and “Family entertainment” personas, even if they are about to board the same ship). Continuously educate your fellow marketers that using personas is as easy as using any other type of data, except that they are compressed model scores with no missing values.

The personalization game is complex. It may look easy if you just buy an off-the-shelf personalization engine, set up some rules with unrefined data and let it run. While it’s better than sending uniform message to everyone, that kind of rudimentary approach is far less than ideal, not to mention the annoyance factor.

To maximize the power of all available data and the personalization engine itself, we must compress the data in forms of personas. Resultant messaging will be far more relevant to your target audience as, for one, a persona is a built-in mechanism for the personal touch. If you set the menu up as a bridge between data and people, that is.

The Art of Data Categorization

Machine-learning is getting better at recognition and categorization by leaps and bounds, for sure. My dog has a Facebook page — don’t ask why — and on his last birthday, Facebook correctly converted his age to dog years.

Do you know why some reports are unbearably long and filled with numbers that are irrelevant to decision-making? It is mostly because there are serious misalignments between the desired level of detail in reporting and actual data categorization. Raw data, with very few exceptions, are rarely ready for decision-making (through various reports) or statistical modeling (an important part of what we often call advanced analytics).

Machine-learning is getting better at recognition and categorization by leaps and bounds, for sure. My dog has a Facebook page — don’t ask why — and on his last birthday, Facebook correctly converted his age to dog years. I kept hearing that machines have a hard time separating dog and cat pictures, but apparently such an obstacle has been overcome (or do they just use dog years for cats, too?).

In any case, do machines understand the “purpose” of categorization and tagging, as well? Does it understand why it is even necessary to put my dog’s age in dog years? That is an entirely different matter, and whether the work is done by humans or machines, I have seen time and time again that categorization efforts with clear purposes result in improvement in analytics and prediction.

Let’s take an example of the hot topic of personalization. Folks who have read my previous articles may already know that I am not even nearly impressed with various marketing efforts under the banner of personalization today. Most are done on a product level, with raw product-level data, when the personalization must foremost be about the person.

Even at a basic level of personalization, consumers on the receiving end often suspect that some personalization engines don’t even consider categories of products, as a suggested product is often irrelevant, dubious or even stupid (as in, “Hey, I just bought that exact item! Why are they offering it to me again?”). I can think of many reasons why that happens (mostly around data and analytics), but the first wrong gear often is that data are not properly categorized.

Results of analytical efforts for personalization and other complex challenges certainly improve when clean data enters the system. The reasons why most analysts spend the majority of their valuable time in data preparation — or even give up to use some granular data — is mostly because input data are unclean, unstructured or uncategorized.

Allow me to share some categorization rules that I have developed based on countless trials and errors during my co-op database days, when we had to put tens of millions of SKUs from over 1,500 sources into one consistent list of categories, solely for the purpose of analytics for individual-level targeting. Whether the actual categorization is done by humans or machines is not the issue; they all have to “learn” what the proper category is to be assigned for each item, and that starts with a proper categorization framework.

The rules I am introducing here are for personal-level targeting and customization of messages; therefore, “customer-centric” at the core. You may need to develop separate frameworks, if the goals are different. Problem statements such as “What will be the most popular product next season?” for instance, would require product-centric categorization. Nonetheless, this framework will be useful when setting up your own, as well.

Without further ado, let’s dive into the list:

Categorize the Buyers, Not the Product

This may not sound intuitive, but it is the first item to remember when setting up a goal-oriented categorization framework. If it is for personalization, and if you are creating a 360-degree view of customers for that purpose, don’t stop there and convert the product-level information into descriptors of buyers. And categorizing items with this goal in mind results in a vastly different — and far more predictable — outcome.

For instance, some items in a nautical catalog, such as a wall-mounted weather station (displaying temperature, air pressure, humidity, etc. on a fancy panel), can also be purchased from an executive gift catalog or website. When assigning categories for items like that, think about the context of the purchase, not just SKU descriptions, to avoid cases where you end up sending nautical catalogs to casual gift buyers. When in doubt, imagine how many purposes baking soda serves; think about the context of the purchase to describe the buyer, depending on the specific purpose (e.g., baking, personal hygiene, deodorization of a refrigerator, domestic cleaning, etc.).

Also consider the price scale and purpose of the purchase, so that you do not end up putting a cheap, everyday lamp and a state-of-the-art home decor lamp in the same category, leading to seriously misaligned offers. You must look beyond simple product descriptions.

The More Specific, the Better

Basically, don’t be lazy and put a 4K TV under “Home Electronics” and call it a day. For apparel items, gender break is the easy part, but sub-categories are even more important for prediction. Most modern product categorization schema are multilevel, like Home Electronics>Home Theater>TV>4K TV. So use it fully.

I’m not saying that all the minute details are helpful for analytics; I’m just emphasizing that one can combine categories later in the process. But if things are lumped up to begin with, one cannot break them apart without resorting back to the source data.

You will be better off if this type of effort is performed as early in the process as possible. Don’t create some big homework for everyone — especially for the analysts — for later. 

Consistency Over Accuracy

This may sound weird as well, but consistently wrong data may be more predictable than “sometimes” accurate data. Assigning the same item to multiple categories creates all kinds of havoc in reporting and prediction downstream. We may argue forever if a certain type of luxury handbag belongs in a category, with no clear winner in the end. The key point is that one should not go back and forth with established categorization rules.

If you can’t settle the fight, then use multiple tags for an item (I don’t recommend it personally). In any case, to machines and algorithms, those categories are just a numeric representation of where they belong, without any judgement. Don’t spend too much energy on making human sense out of every assignment. We can always change the “label” at the reporting stage.

Categorize Only as Much as It Matters

When categorizing items for targeting and reporting, we do not have to create a new schema that covers the entire spectrum of items. If targeting is the end-goal, you don’t even have to touch the items that did not sell very well, as there are not many buyers behind them. Going further, it is alright to categorize the top 20 percent of the items in terms of popularity (i.e., number of transactions or revenue dollar amount), if it covers over 80 percent of the customer behaviors. Yes, I said don’t be lazy under No. 2, but there is no point in spending energy categorizing small items that may not even move analytical needles later. In other words, know when to stop and use the “All other” category for insignificant ones.

Cut Out the Noise

Not every little detail matters in analytics. For example, the “color” of an item may matter a great deal for inventory management (as in “Hey, we are running low on the toasters in Ferrari red!”). But unless you are thinking about targeting people who only purchase items in red, you may not need such details for customized communication and offers. Break down the elements that make up an item, and go only as far as your specific goal calls for. Consult with analysts when in doubt.

Be Inventive

Creating the category buckets is the first important step of categorization efforts. This is where one must “imagine” what type of category would be useful for reporting and prediction later. Simple food labels could lead to all kinds of interesting “behavioral” categories that may be extremely useful when personalizing offers (refer to “Freeform Data Are Not Exactly Free”). This may sound contradictory to No. 5, but hitting the right balance between “too much” and “too little” is indeed the human function — for now — that I was talking about.


Analytics, as we’ve been saying for a long time, is a “garbage-in-garbage-out” business. But in the age of abundant and ubiquitous data, some “seemingly” useless data can be truly predictable. If we don’t think about “data refinement” — of which categorization is a big part — analysts will end up beating down a few popular variables, or worse yet, push down the raw data through some analytical engine “hoping” for some good results.

If the current state of personalization is any indication, most available data must be refined in more systematic and rigorous fashion, whether done by machines or humans. And until the machine catches up with us in the area of creativity, intuition, as well as logical deduction, we will have to be the ones who set up the framework.

Data became too big and complex and customers became too demanding for marketers to leave anything to chance. Even your off-the-self personalization engine will run better with well-categorized data. So, commit to that step, set up proper frameworks and rules, and move onto automation once the organization is ready for it.

AI may take over the world soon, but different types of thinking machines will have to work together to make various marketing efforts truly fruitful. And categorization, along with predictive analytics, is an important component. That is, if you as a consumer believe that machine-driven personalization can use a “human touch.”

Delivering on the Marketing Promise

We all know that promises are made to be kept. And let’s assume that most marketers are intent on delivering the promises they make, even if the promotional wording of those promises may be somewhat exaggerated.

marketing management
“community-manager,” Creative Commons license. | Credit: Flickr by Enrique Martinez Bermejo

We all know that promises are made to be kept.

And let’s assume that most marketers are intent on delivering the promises they make, even if the promotional wording of those promises may be somewhat exaggerated.

The problem is that unless we can truly control every step in the journey from the first promotional articulation through to the timely receipt of the goods or service and payment, Murphy’s law — “anything that can go wrong, will go wrong” — may come into play. I remember many years ago making a unique “Act Now: One Day Only” offer for a book club membership drive and being hit by the season’s worst snowstorm at the end of the “One Day Only.” We waited and waited for the response: one day, two days and only on the third day did the mailed orders begin to trickle in. Of course, it never caught up with expectation.

Not long ago, a Brazilian marketing company had launched a major campaign for magazine subscriptions using, as a medium, promotional inserts in a bank’s monthly credit card charges’ mailing. The bank promised that 100 percent of its invoices would have the insert. When response was well below tested expectations, it was discovered that only about 60 percent of the promotional pieces had been inserted: Someone in the lettershop had mislaid boxes of the printed inserts and never alerted anyone, lest it slow the tightly scheduled invoice mailing. Had the marketer endured the boredom and personally paid a visit to the facility when the job was being run, a significant and very expensive disaster could have been averted.

We have no way of managing the customer’s expectation other than scrupulously delivering what we have promised or even a little more than we have promised — just in case Murphy is hanging around. We all know that one of Amazon’s greatest strengths is its delivery follow-through. It doesn’t only “ask” for — it almost insists — on customer feedback. It carefully monitors every step of the process and listens and responds to comments, whether bouquets or brickbats.

Sadly, in my experience, not enough companies listen carefully to the recordings of telephone interactions the law requires them to announce and proactively respond to about customer complaints.

We are at a strange time in marketing’s history.

We have more tools than ever before, and these allow levels of sophistication not even dreamed of only a couple of decades ago in the age of Addressograph plates and before computers were on every desk. But it seems that the promise of the future — super technology to deal efficiently with all the minutiae of the selling, purchasing and payment processes — often falls short of keeping that promise.

The easy thing to do is to blame it on “those lazy, overpaid, long-haired techies” and software that “doesn’t do what they promised it would do.” But, as the saying is, “the fault lies not with the Gods but with ourselves.”

As managers of data-driven marketing enterprises or service companies, most of us have come a long way from the days when management by walking around (such as visiting the lettershop facility in the earlier example) was in vogue. That meant actually seeing if what was happening where the real work is done, away from our elegant offices, matches the promising PowerPoint presentations we see in the conference room.

Our customers want and need us. They applaud with their purchases, in the convenience and economy of the digital world where everything is immediately available and even better than promised.

But for those of us who have left the “reality” down on the shop floor and manage by keeping an eye on our ever-fancier dashboards, it might be good to remember the anecdote about a possible future airline flight whose passengers were told that the flight was historic, the first one to have no crew. The joke about that flight is the announcement promised: “This flight will be flown by a faultless new technology and nothing can go wrong … go wrong … go wrong.”

We would do well to make sure that our promises are being kept the old-fashioned way — walking around.