How to Find New Customers, Based on Current Customers, With a Targeted Mail List

When you need to acquire new customers, purchasing a targeted mail list is the way to reach them. However, some lists are better than others. We talked about five types of prospecting lists in the last post; now, we will discuss analytics for profiling and modeling lists.

Your mailing list is critical to your mailing results. When you need to acquire new customers, purchasing a targeted mail list is the way to reach them. However, some lists are better than others. We talked about five types of prospecting lists in the last post; now, we will discuss analytics for profiling and modeling lists.

The better your list is targeted, the better your response rate will be.

  • Descriptive Analytics: Is a profile that describes common attributes of your customers and helps to target, based on demographic lookalikes. The market penetration of each attribute shows the comparison between customers and overall population living in the same geo area, with the same attributes, where each element is examined separately. Basically, you will see who your best customers are and find prospects just like them.
  • Predictive Analytics: Is a model that finds how two or more groups are similar or dissimilar. For example, buyers vs. non-buyers; or responders vs. non-responders. Then it assigns a score that represents a probability-to-act, based on the interaction of several attributes. That way, you can get a better idea of who buys what in order to find more people like them.

So why would you want to try one of these options? You can expect an improved response rate, more effective cross-sell and up-sell opportunities, and the ability to build better loyalty programs, because you understand people better. These processes help you identify prospects who “look like” your best customers.

Profiling allows you to profile your best customers (B2C or B2B) and find out what makes them different from others in your target market. You can target new prospects who are the most likely to respond, purchase, or renew, based on your customer data. You can gain precise information about your customers, based on the statistical analysis of key activities. Finally, you will understand the lifetime value of customers, including their probability to respond and purchase products, with a highly advanced model.

Predictive modeling is a process that analyzes the past and current activities or behaviors of two groups to improve future results. This is accomplished between the comparisons of two groups of data. The differences are assessed to identify whether a pattern exists and if it is likely to repeat itself. Scores can be applied to prospect data purchases, or to segment client data for marketing.

Both provide great opportunities for you to target and reach prospects who are more likely to be interested in what you are selling. This way, your offer resonates with them and compels action. This is another way to increase your ROI, as well as save money. You are mailing to only qualified people, so there are less pieces to print and mail. Keep in mind that your customer list is going to get the best response rates, but a highly targeted list like these will have higher response rates than an average purchase list. Are you ready to profile and model your list?

 

For Our Security, Does the FBI Need a Predictive Model?

If you’ve ever worked with a predictive model, you know it is not static, but an iterative effort that requires constant testing, tweaking and feeding of additional data points. It’s a living, breathing tool that is extremely useful in helping to determine where you should best spend your marketing investment for the highest return. This same premise could be used to predict the likelihood of terrorist activity — and therefore be a useful tool in our global war on terror.

Data ScientistIf you’ve ever tried to improve your direct marketing response rates, you’ve probably considered the use of a predictive model.

Predictive modeling is a process that uses data mining and probability to forecast outcomes. The model is made up of a number of variables about your customers: demographic variables (gender, age, household income, etc.), lifestyle variables (smoker, frequent flyer, etc.) and behavioral variables (last date of purchase, purchase amount, SKU, etc.). Each variable is weighted as to its likelihood to predict a specific outcome (like a future purchase) and a statistical model is then formulated. The model is then overlaid on your customer file and every customer is ranked based on their likelihood to respond to an offer, take your desired action, and even predict the average purchase amount.

If you’ve ever worked with a predictive model, you know it is not static, but an iterative effort that requires constant testing, tweaking and feeding of additional data points. It’s a living, breathing tool that is extremely useful in helping to determine where you should best spend your marketing investment for the highest return.

This same premise could be used to predict the likelihood of terrorist activity — and therefore be a useful tool in our global war on terror.

Think about it for just a minute.

The recent bombing in Manchester, U.K. might have been prevented if only the suspect had been higher on the terrorism watch list.

While authorities noted that the suspect (and his family) were on the list, it was added that there are “thousands” on the watch list and there isn’t enough manpower to track them all. Fair enough. But let’s consider those variables that may have predicted that something was about to happen and that, perhaps, he should have moved higher up on that list.

  • The suspects father was linked to a well-known militant Islamist group in Libya
  • His two brothers have been separately arrested on suspicion of terrorism offences
  • He was reported to authorities two years ago “because he [was] thought to be involved in extremism and terrorism”
  • Two friends separately called the police counter-terrorism hotline five years ago and again last year
  • Neighbors had called authorities within the last year, noting that the family had flown a flag for a short time that was black and had writing on it similar to jihadists

The final variable is that the suspect had traveled to both Syria and Libya — the latter only a few weeks before returning to the U.K. and launching his attack. Libya is well known as a terrorist hotbed, so add all the previous variables and the “traveled in May 2017 to Libya” variable would probably catapult this guy to the top of the model.

But why doesn’t such a database exist?

Well, privacy concerns, for one. While consumers — and in particular, Americans — argue about their privacy rights, they are already part of every large consumer database whether they realize it or not. If you’ve ever purchased a home, opened a credit card, paid a tax bill, enrolled in a public school, joined a Frequent Flyer program, registered a purchase for warranty coverage, made a political contribution or subscribed to a magazine, you’re in the Experian or Equifax master file.

In many countries around the world, these same kinds of consumer databases exist, so imagine if these files were combined, and then appended with data variables from law enforcement databases and ticket sales from airline databases. Add in databases about weapon and ammunition purchases, and surely there are enough predictive variables that would allow an analyst to build a model that would determine a way to help prioritize security watch lists, and aid in keeping our world just a little bit safer?

Privacy advocates get itchy just thinking about it.

And, of course, there are those concerned about how this wealth of information could be abused, or how hackers could infiltrate and release confidential information.

But as I head through another security check at my airline gate, and I hear more news about losing the ability to work on my laptop or read my Kindle while in the air, I have to think there’s got to be a better way than the seemingly randomization of these security measures. And it seems that a predictive model might be the answer — but since it depends on consumer data at its core, the future is uncertain without it.

Patients Aren’t Ready for Treatment?

The key is to an effective prescription is to listen to the client first. Why do they lose sleep at night? What are their key success metrics? What are the immediate pain points? What are their long-term goals? And how would we reach there within the limits of provided resources

In my job of being “a guy who finds money-making opportunities using data,” I get to meet all kinds of businesspeople in various industries. Thanks to the business trend around analytics (and to that infamous “Big Data” fad), I don’t have to spend a long time explaining what I do any more; I just say I am in the field of analytics, or to sound a bit fancier, I say data science. Then most marketers seem to understand where the conversation will go from there. Things are never that simple in real life, though, as there are many types of analytics — business intelligence, descriptive analytics, predictive analytics, optimization, forecasting, etc., even at a high level — but figuring what type of solutions should be prescribed is THE job for a consultant, anyway (refer to “Prescriptive Analytics at All Stages”).

The key is to an effective prescription is to listen to the client first. Why do they lose sleep at night? What are their key success metrics? What are the immediate pain points? What are their long-term goals? And how would we reach there within the limits of provided resources and put out the fire at the same time? Building a sound data and analytics roadmap is critical, as no one wants to have an “Oh dang, we should have done that a year ago!” moment after a complex data project is well on its way. Reconstruction in any line of business is costly, and unfortunately, it happens all of the time, as many marketers and decision-makers often jump into the data pool out of desperation under organizational pressure (or under false promises by toolset providers, as in “all your dreams will come true with this piece of technology”). It is a sad sight when users realize that they don’t know how to swim in it “after” they jumped into it.

Why does that happen all of the time? At the risk of sounding like a pompous doctor, I must say that it is quite often the patient’s fault, too; there are lots of bad patients. When it comes to the data and analytics business, not all marketers are experts in it, though some are. Most do have a mid-level understanding, and they actually know when to call in for help. And there are complete novices, too. Now, regardless of their understanding level, bad patients are the ones who show up with self-prescribed solutions, and wouldn’t hear about any other options or precautions. Once, I’ve even met a client who demanded a neural-net model right after we exchanged pleasantries. My response? “Whoa, hold your horses for a minute here, why do you think that you need one?” (Though I didn’t quite say it like that.) Maybe you just came back from some expensive analytics conference, but can we talk about your business case first? After that conversation, I could understand why doctors wouldn’t appreciate patients who would trust WebMD over living, breathing doctors who are in front of them.

Then there are opposite types of cases, too. Some marketers are so insecure about the state of their data assets (or their level of understanding) that they wouldn’t even want to hear about any solutions that sound even remotely complex or difficult, although they may be in desperate need of them. A typical response is something like “Our datasets are so messy that we can’t possibly entertain anything statistical.” You know what that sounds like? It sounds like a patient refusing any surgical treatment in an ER because “he” is not ready for it. No, doctors should be ready to perform the surgery, not the patient.

Messy datasets are surely no excuse for not taking the right path. If we had to wait for a perfect set of data all of the time, there wouldn’t be any need for statisticians or data scientists. In fact, we need such specialists precisely because most data sets are messy and incomplete, and they need to be enhanced by statistical techniques.

Analytics is about making the best of what we have. Cleaning dirty and messy data is part of the job, and should never be an excuse for not doing the right thing. If anyone assumes that simple reports don’t require data cleansing steps because the results look simple, nothing could be further from the truth. Most reporting errors stem from dirty data, and most datasets — big or small, new or old — are not ready to be just plugged into analytical engines.

Besides, different types of analytics are needed because there are so many variations of business challenges, and no analytics is supposed to happen in some preset order. In other words, we get into predictive modeling because the business calls for it, not because a marketer finished some basic Reporting 101 class and now wants to move onto an Analytics 202 course. I often argue that deriving insights out of a series of simple reports could be a lot more difficult than building models or complex data management. Conversely, regardless of the sophistication level, marketers are not supposed to get into advanced analytics just for intellectual curiosity. Every data and analytics activity must be justified with business purposes, carefully following the strategic data roadmap, not difficulty level of the task.

Sex and the Schoolboy: Predictive Modeling – Who’s Doing It? Who’s Doing it Right?

Forgive the borrowed interest, but predictive modeling is to marketers as sex is to schoolboys. They’re all talking about it, but few are doing it. And among those who are, fewer are doing it right. In customer relationship marketing (CRM), predictive modeling uses data to predict the likelihood of a customer taking a specific action. It’s a three-step process.

Forgive the borrowed interest, but predictive modeling is to marketers as sex is to schoolboys.

They’re all talking about it, but few are doing it. And among those who are, fewer are doing it right.

In customer relationship marketing (CRM), predictive modeling uses data to predict the likelihood of a customer taking a specific action. It’s a three-step process:

1. Examine the characteristics of the customers who took a desired action

2. Compare them against the characteristics of customers who didn’t take that action

3. Determine which characteristics are most predictive of the customer taking the action and the value or degree to which each variable is predictive

Predictive modeling is useful in allocating CRM resources efficiently. If a model predicts that certain customers are less likely respond to a specific offer, then fewer resources can be allocated to those customers, allowing more resources to be allocated to those who are more likely to respond.

Data Inputs
A predictive model will only be as good as the input data that’s used in the modeling process. You need the data that define the dependent variable; that is, the outcome the model is trying to predict (such as response to a particular offer). You’ll also need the data that define the independent variables, or the characteristics that will be predictive of the desired outcome (such as age, income, purchase history, etc.). Attitudinal and behavioral data may also be predictive, such as an expressed interest in weight loss, fitness, healthy eating, etc.

The more variables that are fed into the model at the beginning, the more likely the modeling process will identify relevant predictors. Modeling is an iterative process, and those variables that are not at all predictive will fall out in the early iterations, leaving those that are most predictive for more precise analysis in later iterations. The danger in not having enough independent variables to model is that the resultant model will only explain a portion of the desired outcome.

For example, a predictive model created to determine the factors affecting physician prescribing of a particular brand was inconclusive, because there weren’t enough dependent variables to explain the outcome fully. In a standard regression analysis, the number of RXs written in a specific timeframe was set as the dependent variable. There were only three independent variables available: sales calls, physician samples and direct mail promotions to physicians. And while each of the three variables turned out to have a positive effect on prescriptions written, the “Multiple R” value of the regression equation was high at 0.44, meaning that these variables only explained 44 percent of the variance in RXs. The other 56 percent of the variance is from factors that were not included in the model input.

Sample Size
Larger samples will produce more robust models than smaller ones. Some modelers recommend a minimum data set of 10,000 records, 500 of those with the desired outcome. Others report acceptable results with as few as 100 records with the desired outcome. But in general, size matters.

Regardless, it is important to hold out a validation sample from the modeling process. That allows the model to be applied to the hold-out sample to validate its ability to predict the desired outcome.

Important First Steps

1. Define Your Outcome. What do you want the model to do for your business? Predict likelihood to opt-in? Predict likelihood to respond to a particular offer? Your objective will drive the data set that you need to define the dependent variable. For example, if you’re looking to predict likelihood to respond to a particular offer, you’ll need to have prospects who responded and prospects who didn’t in order to discriminate between them.

2. Gather the Data to Model. This requires tapping into several data sources, including your CRM database, as well as external sources where you can get data appended (see below).

3. Set the Timeframe. Determine the time period for the data you will analyze. For example, if you’re looking to model likelihood to respond, the start and end points for the data should be far enough in the past that you have a sufficient sample of responders and non-responders.

4. Examine Variables Individually. Some variables will not be correlated with the outcome, and these can be eliminated prior to building the model.

Data Sources
Independent variable data
may include

  • In-house database fields
  • Data overlays (demographics, HH income, lifestyle interests, presence of children,
    marital status, etc.) from a data provider such as Experian, Epsilon or Acxiom.

Don’t Try This at Home
While you can do regression analysis in Microsoft Excel, if you’re going to invest a lot of promotion budget in the outcome, you should definitely leave the number crunching to the professionals. Expert modelers know how to analyze modeling results and make adjustments where necessary.

Big Data Must Get Smaller

Like many folks who worked in the data business for a long time, I don’t even like the words “Big Data.” Yeah, data is big now, I get it. But so what? Faster and bigger have been the theme in the computing business since the first calculator was invented. In fact, I don’t appreciate the common definition of Big Data that is often expressed in the three Vs: volume, velocity and variety. So, if any kind of data are big and fast, it’s all good? I don’t think so. If you have lots of “dumb” data all over the place, how does that help you? Well, as much as all the clutter that’s been piled on in your basement since 1971. It may yield some profit on an online auction site one day. Who knows? Maybe some collector will pay good money for some obscure Coltrane or Moody Blues albums that you never even touched since your last turntable (Ooh, what is that?) died on you. Those oversized album jackets were really cool though, weren’t they?

Like many folks who worked in the data business for a long time, I don’t even like the words “Big Data.” Yeah, data is big now, I get it. But so what? Faster and bigger have been the theme in the computing business since the first calculator was invented. In fact, I don’t appreciate the common definition of Big Data that is often expressed in the three Vs: volume, velocity and variety. So, if any kind of data are big and fast, it’s all good? I don’t think so. If you have lots of “dumb” data all over the place, how does that help you? Well, as much as all the clutter that’s been piled on in your basement since 1971. It may yield some profit on an online auction site one day. Who knows? Maybe some collector will pay good money for some obscure Coltrane or Moody Blues albums that you never even touched since your last turntable (Ooh, what is that?) died on you. Those oversized album jackets were really cool though, weren’t they?

Seriously, the word “Big” only emphasizes the size element, and that is a sure way to miss the essence of the data business. And many folks are missing even that little point by calling all decision-making activities that involve even small-sized data “Big Data.” It is entirely possible that this data stuff seems all new to someone, but the data-based decision-making process has been with us for a very long time. If you use that “B” word to differentiate old-fashioned data analytics of yesteryear and ridiculously large datasets of the present day, yes, that is a proper usage of it. But we all know most people do not mean it that way. One side benefit of this bloated and hyped up buzzword is data professionals like myself do not have to explain what we do for living for 20 minutes anymore by simply uttering the word “Big Data,” though that is a lot like a grandmother declaring all her grandchildren work on computers for living. Better yet, that magic “B” word sometimes opens doors to new business opportunities (or at least a chance to grab a microphone in non-data-related meetings and conferences) that data geeks of the past never dreamed of.

So, I guess it is not all that bad. But lest we forget, all hypes lead to overinvestments, and all overinvestments leads to disappointments, and all disappointments lead to purging of related personnel and vendors that bear that hyped-up dirty word in their titles or division names. If this Big Data stuff does not yield significant profit (or reduction in cost), I am certain that those investment bubbles will burst soon enough. Yes, some data folks may be lucky enough to milk it for another two or three years, but brace for impact if all those collected data do not lead to some serious dollar signs. I know how the storage and processing cost decreased significantly in recent years, but they ain’t totally free, and related man-hours aren’t exactly cheap, either. Also, if this whole data business is a new concept to an organization, any money spent on the promise of Big Data easily becomes a liability for the reluctant bunch.

This is why I open up my speeches and lectures with this question: “Have you made any money with this Big Data stuff yet?” Surely, you didn’t spend all that money to provide faster toys and nicer playgrounds to IT folks? Maybe the head of IT had some fun with it, but let’s ask that question to CFOs, not CTOs, CIOs or CDOs. I know some colleagues (i.e., fellow data geeks) who are already thinking about a new name for this—”decision-making activities, based on data and analytics”—because many of us will be still doing that “data stuff” even after Big Data cease to be cool after the judgment day. Yeah, that Gangnam Style dance was fun for a while, but who still jumps around like a horse?

Now, if you ask me (though nobody did yet), I’d say the Big Data should have been “Smart Data,” “Intelligent Data” or something to that extent. Because data must provide insights. Answers to questions. Guidance to decision-makers. To data professionals, piles of data—especially the ones that are fragmented, unstructured and unformatted, no matter what kind of fancy names the operating system and underlying database technology may bear—it is just a good start. For non-data-professionals, unrefined data—whether they are big or small—would remain distant and obscure. Offering mounds of raw data to end-users is like providing a painting kit when someone wants a picture on the wall. Bragging about the size of the data with impressive sounding new measurements that end with “bytes” is like counting grains of rice in California in front of a hungry man.

Big Data must get smaller. People want yes/no answers to their specific questions. If such clarity is not possible, probability figures to such questions should be provided; as in, “There’s an 80 percent chance of thunderstorms on the day of the company golf outing,” “An above-average chance to close a deal with a certain prospect” or “Potential value of a customer who is repeatedly complaining about something on the phone.” It is about easy-to-understand answers to business questions, not a quintillion bytes of data stored in some obscure cloud somewhere. As I stated at the end of my last column, the Big Data movement should be about (1) Getting rid of the noise, and (2) Providing simple answers to decision-makers. And getting to such answers is indeed the process of making data smaller and smaller.

In my past columns, I talked about the benefits of statistical models in the age of Big Data, as they are the best way to compact big and complex information in forms of simple answers (refer to “Why Model?”). Models built to predict (or point out) who is more likely to be into outdoor sports, to be a risk-averse investor, to go on a cruise vacation, to be a member of discount club, to buy children’s products, to be a bigtime donor or to be a NASCAR fan, are all providing specific answers to specific questions, while each model score is a result of serious reduction of information, often compressing thousands of variables into one answer. That simplification process in itself provides incredible value to decision-makers, as most wouldn’t know where to cut out unnecessary information to answer specific questions. Using mathematical techniques, we can cut down the noise with conviction.

In model development, “Variable Reduction” is the first major step after the target variable is determined (refer to “The Art of Targeting“). It is often the most rigorous and laborious exercise in the whole model development process, where the characteristics of models are often determined as each statistician has his or her unique approach to it. Now, I am not about to initiate a debate about the best statistical method for variable reduction (I haven’t met two statisticians who completely agree with each other in terms of methodologies), but I happened to know that many effective statistical analysts separate variables in terms of data types and treat them differently. In other words, not all data variables are created equal. So, what are the major types of data that database designers and decision-makers (i.e., non-mathematical types) should be aware of?

In the business of predictive analytics for marketing, the following three types of data make up three dimensions of a target individual’s portrait:

  1. Descriptive Data
  2. Transaction Data / Behavioral Data
  3. Attitudinal Data

In other words, if we get to know all three aspects of a person, it will be much easier to predict what the person is about and/or what the person will do. Why do we need these three dimensions? If an individual has a high income and is living in a highly valued home (demographic element, which is descriptive); and if he is an avid golfer (behavioral element often derived from his purchase history), can we just assume that he is politically conservative (attitudinal element)? Well, not really, and not all the time. Sometimes we have to stop and ask what the person’s attitude and outlook on life is all about. Now, because it is not practical to ask everyone in the country about every subject, we often build models to predict the attitudinal aspect with available data. If you got a phone call from a political party that “assumes” your political stance, that incident was probably not random or accidental. Like I emphasized many times, analytics is about making the best of what is available, as there is no such thing as a complete dataset, even in this age of ubiquitous data. Nonetheless, these three dimensions of the data spectrum occupy a unique and distinct place in the business of predictive analytics.

So, in the interest of obtaining, maintaining and utilizing all possible types of data—or, conversely, reducing the size of data with conviction by knowing what to ignore, let us dig a little deeper:

Descriptive Data
Generally, demographic data—such as people’s income, age, number of children, housing size, dwelling type, occupation, etc.—fall under this category. For B-to-B applications, “Firmographic” data—such as number of employees, sales volume, year started, industry type, etc.—would be considered as descriptive data. It is about what the targets “look like” and, generally, they are frozen in the present time. Many prominent data compilers (or data brokers, as the U.S. government calls them) collect, compile and refine the data and make hundreds of variables available to users in various industry sectors. They also fill in the blanks using predictive modeling techniques. In other words, the compilers may not know the income range of every household, but using statistical techniques and other available data—such as age, home ownership, housing value, and many other variables—they provide their best estimates in case of missing values. People often have some allergic reaction to such data compilation practices siting privacy concerns, but these types of data are not about looking up one person at a time, but about analyzing and targeting groups (or segments) of individuals and households. In terms of predictive power, they are quite effective and results are very consistent. The best part is that most of the variables are available for every household in the country, whether they are actual or inferred.

Other types of descriptive data include geo-demographic data, and the Census Data by the U.S. Census Bureau falls under this category. These datasets are organized by geographic denominations such as Census Block Group, Census Tract, Country or ZIP Code Tabulation Area (ZCTA, much like postal ZIP codes, but not exactly the same). Although they are not available on an individual or a household level, the Census data are very useful in predictive modeling, as every target record can be enhanced with it, even when name and address are not available, and data themselves are very stable. The downside is that while the datasets are free through Census Bureau, the raw datasets contain more than 40,000 variables. Plus, due to the budget cut and changes in survey methods during the past decade, the sample size (yes, they sample) decreased significantly, rendering some variables useless at lower geographic denominations, such as Census Block Group. There are professional data companies that narrowed down the list of variables to manageable sizes (300 to 400 variables) and filled in the missing values. Because they are geo-level data, variables are in the forms of percentages, averages or median values of elements, such as gender, race, age, language, occupation, education level, real estate value, etc. (as in, percent male, percent Asian, percent white-collar professionals, average income, median school years, median rent, etc.).

There are many instances where marketers cannot pinpoint the identity of a person due to privacy issues or challenges in data collection, and the Census Data play a role of effective substitute for individual- or household-level demographic data. In predictive analytics, duller variables that are available nearly all the time are often more valuable than precise information with limited availability.

Transaction Data/Behavioral Data
While descriptive data are about what the targets look like, behavioral data are about what they actually did. Often, behavioral data are in forms of transactions. So many just call it transaction data. What marketers commonly refer to as RFM (Recency, Frequency and Monetary) data fall under this category. In terms of predicting power, they are truly at the top of the food chain. Yes, we can build models to guess who potential golfers are with demographic data, such as age, gender, income, occupation, housing value and other neighborhood-level information, but if you get to “know” that someone is a buyer of a box of golf balls every six weeks or so, why guess? Further, models built with transaction data can even predict the nature of future purchases, in terms of monetary value and frequency intervals. Unfortunately, many who have access to RFM data are using them only in rudimentary filtering, as in “select everyone who spends more than $200 in a gift category during the past 12 months,” or something like that. But we can do so much more with rich transaction data in every stage of the marketing life cycle for prospecting, cultivating, retaining and winning back.

Other types of behavioral data include non-transaction data, such as click data, page views, abandoned shopping baskets or movement data. This type of behavioral data is getting a lot of attention as it is truly “big.” The data have been out of reach for many decision-makers before the emergence of new technology to capture and store them. In terms of predictability, nevertheless, they are not as powerful as real transaction data. These non-transaction data may provide directional guidance, as they are what some data geeks call “a-camera-on-everyone’s-shoulder” type of data. But we all know that there is a clear dividing line between people’s intentions and their commitments. And it can be very costly to follow every breath you take, every move you make, and every step you take. Due to their distinct characteristics, transaction data and non-transaction data must be managed separately. And if used together in models, they should be clearly labeled, so the analysts will never treat them the same way by accident. You really don’t want to mix intentions and commitments.

The trouble with the behavioral data are, (1) they are difficult to compile and manage, (2) they get big; sometimes really big, (3) they are generally confined within divisions or companies, and (4) they are not easy to analyze. In fact, most of the examples that I used in this series are about the transaction data. Now, No. 3 here could be really troublesome, as it equates to availability (or lack thereof). Yes, you may know everything that happened with your customers, but do you know where else they are shopping? Fortunately, there are co-op companies that can answer that question, as they are compilers of transaction data across multiple merchants and sources. And combined data can be exponentially more powerful than data in silos. Now, because transaction data are not always available for every person in databases, analysts often combine behavioral data and descriptive data in their models. Transaction data usually become the dominant predictors in such cases, while descriptive data play the supporting roles filling in the gaps and smoothing out the predictive curves.

As I stated repeatedly, predictive analytics in marketing is all about finding out (1) whom to engage, and (2) if you decided to engage someone, what to offer to that person. Using carefully collected transaction data for most of their customers, there are supermarket chains that achieved 100 percent customization rates for their coupon books. That means no two coupon books are exactly the same, which is a quite impressive accomplishment. And that is all transaction data in action, and it is a great example of “Big Data” (or rather, “Smart Data”).

Attitudinal Data
In the past, attitudinal data came from surveys, primary researches and focus groups. Now, basically all social media channels function as gigantic focus groups. Through virtual places, such as Facebook, Twitter or other social media networks, people are freely volunteering what they think and feel about certain products and services, and many marketers are learning how to “listen” to them. Sentiment analysis falls under that category of analytics, and many automatically think of this type of analytics when they hear “Big Data.”

The trouble with social data is:

  1. We often do not know who’s behind the statements in question, and
  2. They are in silos, and it is not easy to combine such data with transaction or demographic data, due to lack of identity of their sources.

Yes, we can see that a certain political candidate is trending high after an impressive speech, but how would we connect that piece of information to whom will actually donate money for the candidate’s causes? If we can find out “where” the target is via an IP address and related ZIP codes, we may be able to connect the voter to geo-demographic data, such as the Census. But, generally, personally identifiable information (PII) is only accessible by the data compilers, if they even bothered to collect them.

Therefore, most such studies are on a macro level, citing trends and directions, and types of analysts in that field are quite different from the micro-level analysts who deal with behavioral data and descriptive data. Now, the former provide important insights regarding the “why” part of the equation, which is often the hardest thing to predict; while the latter provide answers to “who, what, where and when.” (“Who” is the easiest to answer, and “when” is the hardest.) That “why” part may dictate a product development part of the decision-making process at the conceptual stage (as in, “Why would customers care for a new type of dishwasher?”), while “who, what, where and when” are more about selling the developed products (as in “Let’s sell those dishwashers in the most effective ways.”). So, it can be argued that these different types of data call for different types of analytics for different cycles in the decision-making processes.

Obviously, there are more types of data out there. But for marketing applications dealing with humans, these three types of data complete the buyers’ portraits. Now, depending on what marketers are trying to do with the data, they can prioritize where to invest first and what to ignore (for now). If they are early in the marketing cycle trying to develop a new product for the future, they need to understand why people want something and behave in certain ways. If signing up as many new customers as possible is the immediate goal, finding out who and where the ideal prospects are becomes the most imminent task. If maximizing the customer value is the ongoing objective, then you’d better start analyzing transaction data more seriously. If preventing attrition is the goal, then you will have to line up the transaction data in time series format for further analysis.

The business goals must dictate the analytics, and the analytics call for specific types of data to meet the goals, and the supporting datasets should be in “analytics-ready” formats. Not the other way around, where businesses are dictated by the limitations of analytics, and analytics are hampered by inadequate data clutters. That type of business-oriented hierarchy should be the main theme of effective data management, and with clear goals and proper data strategy, you will know where to invest first and what data to ignore as a decision-maker, not necessarily as a mathematical analyst. And that is the first step toward making the Big Data smaller. Don’t be impressed by the size of the data, as they often blur the big picture and not all data are created equal.