Hidden cultural bias led to seriously flawed election forecasts. Enterprises face similar challenges as they harness their Big Data and put it to work.

ZDNet: By Tom Foremski for Tom Foremski: IMHO | November 9, 2016

For several weeks I was receiving daily messages on my phone from the New York Times’ Upshot column, which confidently predicted Trump’s chances of winning at around 5 percent most days.

The Upshot data was crunched from many different polls and fed into a special algorithm based on historical and other relevant data. Other organizations also used reams of Big Data to feed their analytical models and were coming to similar predictions: Trump would lose.

So how was it that all these sophisticated analytical models with access to high-quality data got the election forecast so wrong?

Jim Rutenberg in the New York Times writes that there was a cultural bias.

“Journalists didn’t question the polling data when it confirmed their gut feeling that Mr. Trump could never in a million years pull it off. They portrayed Trump supporters who still believed he had a shot as being out of touch with reality. In the end, it was the other way around.”

There’s an important lesson for enterprises here is that simply getting access to all your Big Data is not enough. It won’t result in valuable business predictions unless the analysis is the right one.

Domain knowledge counts for a tremendous amount of success with Big Data because analysis matters and knowing the right questions to ask comes from experience. The right analytical model is vital but being aware of cultural bias in those models is key.

See also: Big data: Three ways to make your project a success | The IoT security doomsday is lurking, but we cannot talk about it properly | How to stop people being the weakest link in enterprise security

However, the inclusion of a cultural bias into analytical models is not something to avoid because if a company knows the culture of its customers it can uncover emerging markets or changes in buying habits more rapidly than others.

The Big Data industry promises businesses that they can uncover new sources of revenues — and they can. But the election has shown that Big Data is useless if the analysis is flawed.

It’s the hidden cultural bias that’s dangerous and can lead to wildly inaccurate predictions. Knowing that there will always be some hidden cultural bias means the better design of analytical models.

Fortunately, analytical models can learn and adapt and can be run against each other, to give management a good understanding of their business and their options for future performance.

It’s too late for the pollsters but their analytical tools will certainly be a lot sharper next time.

ZDNet: By Tom Foremski for Tom Foremski: IMHO | November 9, 2016



[[ Breaking News: New York Times polling guru Nate Cohn said the Paper of Record “fudged” numbers on election night so Donald Trump’s shocking upset wouldn’t contradict projections.
The Upshot election forecasting model started off giving Hillary Clinton an 85 percent chance to win the election, a forecast based on polling data. By the end of Election Night, (only a few hours), the same model gave Donald Trump a 95 percent chance of winning. An actual 180* turn around…Impossible…but there it is. ]]