Is Hadoop Right For Everyone?
Across many organisations we find ourselves answering the same questions around Hadoop technology.We understand that senior level executives read broad sheets, the business press – and even the IT press -for the next buzzword to take to the monthly board meeting. But, just because ‘sexy’ organisations likeGoogle, Facebook and Yahoo talk about the power of Hadoop, it doesn’t mean it’s appropriate for your business.
When your IT Director/CTO comes back from his next board meeting saying ‘we need a Hadoop Big Data strategy’, remember there is often a reason why your data currently lives where it does.
As always, technology alone is never the answer:
“In order for a big data initiative to be successful, it must deliver a positive return on investment. Yet, the investments necessary to be successful are widely misunderstood. Investments are not just in technology, but also in people with the right skill sets.”
Stephen Brobst (Teradata CTO)
Over last 12 months we have witnessed first hand a number of businesses embark on the Hadoop journey.
Several common issues seem to be emerging:
1) User alienation – what will your business users do when they can’t easily query their data anymore? You invested in their ability to query data sets through SQL, and now you’re going to throw all that away and ask them to learn Java, Pig & Hive? Good luck with that!
2) Lack of IT skills – does your current IT department have the skills to drive Hadoop? In order to maximise your Hadoop environment you will need skills in things like Linux, Java, Python or C. And that’s just to get started.
3) Inappropriate solution – do you *really* have large amounts of unstructured data that need to be transformed into a more useable format? Always keep in mind your source systems will be producing 99% structured data, and that common-or-garden relational databases have been good at this stuff for a long time.
Hadoop is being tested across many industry sectors in proof-of-concepts (POCs) and labs to see if it is the silver bullet to reduce the cost burden of IT on the CFO. But, in our experience so far, it might be open source technology (“Yay, free!”) backed by some well funded vendors such as Hortonworks, Cloudera and Pivotal, but Hadoop still has a long way to go in terms solving mainstream data management challenges.
Watch this space as we work with more organisations sucking on the Hadoop lollipop….