Big Data Challenges

Big Data

Big Data has been touted as the answer to many problems. Currently many organisations are collecting everything they can get their hands on in order to try and make sense of it, either now, or eventually. And then of course to leverage that for profit, advantage, protection, whatever…

 

But there is a massive flaw in one aspect of this. For instance, you can determine that there is a statistical correlation between two sets of numbers, but that doesn’t mean it is causative.

Tyler Vigen

Tyler Vigen

Tyler Vigen has written Spurious Correlations to demonstrate just this issue. Below are 2 examples.

Spurious Correlation #1

Spurious Correlation #1

So it looks like the US spending on science correlates with suicides. But does it cause it? Or is this merely a coincidence?

 

Spurious Correlation #2

Spurious Correlation #2

Same here. Would suggest spelling bees stick to short words only to limit the damage they could be doing.

 

And this is what we need to be careful of. Now this doesn’t mean our own thinking can’t be fooled. Rolf Dobelli in The Art of Thinking Clearly documents 99 more common cognitive biases we can suffer from. Think of these as bugs in our thinking algorithms. An example is Confirmation Bias. Most people believe they are above average. Nearly half of them are wrong. This is called Illusory Superiority.

 

So the danger is concluding there is a cause and effect relationship when there is not. Leading to decisions being made and outcomes not working out as expected. There are no simple answers here. The above examples are selected because it is easy to conclude there isn’t a direct relationship. It is harder to discern when the data is big and an auto-correlating algorithm is coming to its own conclusions.

 

The Art of Thinking Clearly

The Art of Thinking Clearly

And to round off, here is Rolf Dobelli talking about The Art of Thinking Clearly.

 

 

Successful Endeavours specialise in Electronics Design and Embedded Software Development, focusing on products that are intended to be Made In AustraliaRay Keefe has developed market leading electronics products in Australia for more than 30 years. This post is Copyright © 2018 Successful Endeavours Pty Ltd.