Most often, one hears remarks that Big Data implementation is a failure. This requires a reset of expectations. Big Data is all hype. With this post, I would like to share my views.
In the last 10 years, I’ve spent 5 ½ years in the consulting space and the remaining years with end user clients. By no means do I consider myself an expert; however, I would like to elaborate what I learnt through out these years across both sides of the fence.
Let us understand why do Big Data projects fail. Before doing that, let us agree on the definition of a failure. In my views, a project or an implementation is considered a failure if it does not deliver the perceived benefits.
Most Big Data projects fail because there is very little or no business initiatives and no proper use cases identified. These projects usually run as a technical proposal or the CIO’s initiative. If one looks at the data value chain, the most important part of the chain is Data Operationalisation and/or Data Monetisation. If a project runs as a technical initiative, the Data Monetisation part is often ignored and hence, there is no ROI on the investments.
Most of the Consulting Companies sell Big Data solutions as a silver bullet solution to all the enterprises’ problems and this creates lots of hype. When hype meets reality, the outcome is often not pleasant.
An ideal approach to this should be as:
Client - “I want to implement a Big Data solution”
Consultant- “That could be the solution, but let us discuss the problem first”.
Having been in the consulting space for 5 ½ years, I can say from experience that once you establish yourself as a trusted advisor, the business is only going to grow. From an end-user’s perspective my advice is to use a common-sense approach. If something is too good to be true then probably it is not true. This will help both parties to differentiate between hype and reality.
A few other useful pieces of advice from experience:
Harjot Singh has over 22 years of experience in Data and Digital Transformation in Asia, Pacific, Africa and Middle East. He is currently working as SVP/ Head of Data Engineering for a Digital Banking Fintech in Dubai.