Election Polls and the Price of Being Wrong 

The thing about predictive analytics is that the quality of a prediction is eventually exposed — clearly cut as right or wrong. There are casually incorrect outcomes, like a weather report failing to accurately declare at what time the rain will start, and then there are total shockers, like the outcome of the 2016 presidential election.

screen-shot-2016-11-17-at-1-03-34-pmThe thing about predictive analytics is that the quality of a prediction is eventually exposed — clearly cut as right or wrong. There are casually incorrect outcomes, like a weather report failing to accurately declare the time it will start raining, and then there are total shockers, like the outcome of the 2016 presidential election.

In my opinion, the biggest losers in this election cycle are pollsters, analysts, statisticians and, most of all, so-called pundits.

I am saying this from a concerned analyst’s point of view. We are talking about colossal and utter failure of prediction on every level here. Except for one or two publications, practically every source missed the mark by more than a mile — not just a couple points off here and there. Even the ones who achieved “guru” status by predicting the 2012 election outcome perfectly called for the wrong winner this time, boldly posting a confidence level of more than 70 percent just a few days before the election.

What Went Wrong? 

The losing party, pollsters and analysts must be in the middle of some deep soul-searching now. In all fairness, let’s keep in mind that no prediction can overcome serious sampling errors and data collection problems. Especially when we deal with sparsely populated areas, where the winner was decisively determined in the end, we must be really careful with the raw numbers of respondents, as errors easily get magnified by incomplete data.

Some of us saw that type of over- or under-projection when the Census Bureau cut the sampling size for budgetary reasons during the last survey cycle. For example, in a sparsely populated area, a few migrants from Asia may affect simple projections like “percent Asians” rather drastically. In large cities, conversely, the size of such errors are generally within more manageable ranges, thanks to large sample sizes.

Then there are human inconsistency elements that many pundits are talking about. Basically everyone got so sick of all of these survey calls about the election, many started to ignore them completely. I think pollsters must learn that at times, less is more. I don’t even live in a swing state, and I started to hang up on unknown callers long before Election Day. Can you imagine what the folks in swing states must have gone through?

Many are also claiming that respondents were not honest about how they were going to vote. But if that were the case, there are other techniques that surveyors and analysts could have used to project the answer based on “indirect” questions. Instead of simply asking “Whom are you voting for?”, how about asking what their major concerns were? Combined with modeling techniques, a few innocuous probing questions regarding specific issues — such as environment, gun control, immigration, foreign policy, entitlement programs, etc. — could have led us to much more accurate predictions, reducing the shock factor.

In the middle of all this, I’ve read that artificial intelligence without any human intervention predicted the election outcome correctly, by using abundant data coming out of social media. That means machines are already outperforming human analysts. It helps that machines have no opinions or feelings about the outcome one way or another.

Dystopian Future?

Maybe machine learning will start replacing human analysts and other decision-making professions sooner than expected. That means a disenfranchised population will grow even further, dipping into highly educated demographics. The future, regardless of politics, doesn’t look all that bright for the human collective, if that trend continues.

In the predictive business, there is a price to pay for being wrong. Maybe that is why in some countries, there are complete bans on posting poll numbers and result projections days — sometimes weeks — before the election. Sometimes observation and prediction change behaviors of human subjects, as anthropologists have been documenting for years.

Author: Stephen H. Yu

Stephen H. Yu is a world-class database marketer. He has a proven track record in comprehensive strategic planning and tactical execution, effectively bridging the gap between the marketing and technology world with a balanced view obtained from more than 30 years of experience in best practices of database marketing. Currently, Yu is president and chief consultant at Willow Data Strategy. Previously, he was the head of analytics and insights at eClerx, and VP, Data Strategy & Analytics at Infogroup. Prior to that, Yu was the founding CTO of I-Behavior Inc., which pioneered the use of SKU-level behavioral data. “As a long-time data player with plenty of battle experiences, I would like to share my thoughts and knowledge that I obtained from being a bridge person between the marketing world and the technology world. In the end, data and analytics are just tools for decision-makers; let’s think about what we should be (or shouldn’t be) doing with them first. And the tools must be wielded properly to meet the goals, so let me share some useful tricks in database design, data refinement process and analytics.” Reach him at stephen.yu@willowdatastrategy.com.

9 thoughts on “Election Polls and the Price of Being Wrong ”

  1. Great overview explaining the many reasons why pollsters were unable to call this election accurately, not the least of which being that while Trump won the electoral vote, he lost the popular vote so the majority of the pollsters did ‘t get everything wrong.

    With regard to your comment about Nate Silver and his team at FiveThirtyEight, who achieved “guru” status by predicting the 2012 election outcome perfectly, your comment about his boldly posting a confidence level of 70% before the election is off-base. A 70% confidence level, essentially a one standard deviation event, means that the outcome is expected to occur roughly 2/3 of the time with the opposite outcome occurring the other 1/3 of the time. Accordingly, a 70% confidence level is indicative of a prediction that’s not particularly certain.

    When I ran the MIT Blackjack Team, the minimum confidence level we would accept for each venture before raising funds from investors was a two standard deviation event (i.e. a 95% confidence level). In other words, we projected our Return on Investment to investors while allowing for a break-even outcome 5% or less of the time. Anyone with a background in statistics knows that one standard deviation events happen all of the time (i.e. 1/3 of the time) and that two standard deviation events happen in 1 in 20 times, which is still too often to ever feel safe.

    Yes, Nate Silver and his team at FiveThirtyEight, along almost every other pollster, got this election wrong. But at least FiveThirtyEight knew their prediction had a nearly 1/3 chance of being wrong.

    1. It is an honor to get to have an online chat with someone who ran the Blackjack Team! As for the popular vote, yes, I hear that Hillary is now winning by over 2MM votes, but I still standby my statement about “colossal failure” as all the pollsters, including Mr. Silver, posted projections on a state level. And regarding that 70% (which he changed to 68% the day before the election, which is coincidentally much closer to 68.2% or 1 standard deviation), I still think that prediction is considered to be “way off”, looking at the results. And I’m sure most of his followers knew that FiveThirtyEight had 1/3 chance to be wrong, too. But I guess wishful thinking got the better of most us?

      1. If a few pollsters were wrong and a few were right then we’d consider this a near miss. Individually Nate Silver was forced to wxplain his work that way — but hey he has a business to run and there are time you just can’t say you made a mistake (I mean I would not want to be in his shoes). At the same time all these folks must already be scheming and budgeting to over-sampling in rural, low median school year counties so this “fail” won’t happen again. Seems there is a whole new cast of swing states screaming out for phone surveys!!! Marketing campaigns and products bomb. The captains sailing Hillary’s ship fed completely wrong coordinates into their GPS. The message was wrong, and so was the target. They’ll need to dry dock their boat for a total overhaul. Let’s hope they do it quickly and reliably!!! Thanks, Steven

  2. Another excellent piece Stephen and another reminder that when the ‘image’ advertisers bombard us with ‘research’ both quantitative and qualitative, we would do well not to take them too literally or seriously and keep more than a grain of salt at the ready.

    Measurable response with quantifiable data beats ‘research’ every time. We should be glad we are in the accountable marketing business.

    1. Thanks! I remember Lester Wunderman’s quote about “measurement” being the first qualification for being a good direct marketer.

      1. If Lester wasn’t always right, he certainly was right most of the time. He named my book ‘Accountable Marketing’ because as he said simply: that’s what we do.

        1. Indeed! That is such a good way to put it: “Accountable Marketing”. I remember him talking about measuring number of steps that he climbed as a delivery person and the tip amount that he received. He was “measuring” everything even back then! Interesting conclusion was that the correlation looked like a reverse-bell-curve thanks to the income factor for the top floor dwellers. It is amazing that he did all that in his head.

  3. Sorry for the late comment, BUT, we’ve all had some time to think about this. The polling failure might be in how the questions were asked? My gut feeling is many Trump supporters didn’t share their true feelings with pollsters. Why? Because the MSM made it clear Trump was undesirable, or at best completely unfit for office. These voters don’t care what nice people in suits say on TV, they wanted a change. There’s also data that indicates many were voting for the first time in their lives, so that group may not be on polling databases?

Leave a Reply

Your email address will not be published. Required fields are marked *