Why is Hadoop failing?

Have companies adopted Hadoop without understanding it? How can companies access more data whilst protecting customer privacy? Is data socialisation the next big thing in data analytics? Is your personal data being used to influence your actions?

--

Why is Hadoop failing? Gartner research director Nick Heudecker has predicted that 70 per cent of Hadoop deployments made this year will fail to deliver either the expected cost savings or hoped-for new revenue. 😳 Hadoop manages big data using distributed storage and processing and is essentially designed to scale across thousands of machines. But the average company does not have enough information to make it an effective framework - a typical company could fit every bit of information they have on a single one terabyte hard drive.

Building on this, George Hill argues that "companies have adopted the platform without understanding it and then failed to get the right people or data to make it work properly, which has led to disillusionment and its apparent stagnation". Companies should either seek an alternative to effectively deal with their data, or consider how they can obtain access to larger data volumes to make full use of Hadoop. 🤔 (Side note - need more clarity on Hadoop? Read Adam Shepherd's explanation.)

But how can companies access more data whilst protecting customer privacy? The benefits of companies sharing data is clear, but logistically how data should be shared remains elusive, argues Penny Crosman. 👍 Being able to open up datasets across companies is set to be one of the biggest barriers over the next few years.

Is data socialisation (the process of making sure the right data is in front of the right person, at the right time) the next big thing in data analytics? Bernard Marr argues that organisations need to stop keeping their data siloed and democratise access to business data:

"Embedding data and analytics throughout an organisation, and ensuring its effects can be measured on every process, is often a more productive approach than attempting to impose data-directives in a top-down, centralised manner."

Is your personal data being used to influence your actions? 😳 Excellent write-up by Juuso Parkkinen discussing how "AI systems that feed on personal data have made it possible to influence people's behaviour". Of course not everyone is affected, but as shown in recent political events, altering the attitudes of "a large enough fraction of undecided people can be enough" to swing an election. 👀

Miscellaneous

Facebook launched 'Prophet' - its free forecasting tool for Python and R. Andrew Gelman had a play with the code to create the viz below of log-scale views of American football player Peyton Manning's Wikipedia page. Read more here. 🏈