Leveraging The Big Data

The term 'Big Data' refers to huge amount of data (in Terabytes or even in Petabytes) that a company has access to. It not only refers to stored data but also the data which is being continuously streamed in through various sensors and other data collection points. However, 'Big Data' doesn't represents this huge data but the opportunity that it holds.

Big data has moved from a concept to reality because:

• Information technologies has made it possible to extract (from database, image, audio, video, sensors etc..), manage and process the data without significant latency.
• Advances in data analysis like machine learning algorithms which can provide predictive intelligence and pattern analysis to mine unstructured data.

Everyone now understands the Big Data and the value it holds. The challenge lies in dealing with it. There are two approaches to leverage the Big Data:

1. Start with the problem statement: Define the need or problem statement first and then work backwards to identify the analysis and data required. This approach works when there is clear understanding of the business need. If some of the data required to meet the business need is not available, efforts are made to collect that data or redefine the business need.
2. Dig for Gold: Executives are not always clear what can be done out of the big data. In that case, the suggested approach is to play with the data and derive some sense out of it. While experimenting by combining the data and applying algorithms, one gathers insight and increases the chances of finding the 'Gold'.

One can argue in favour of either of the approaches. However, success of both the approaches hinges on the 'Data Scientist'. It is an emerging profile with the required skillset in applying maths, statistics and algorithms to the data and spotting patterns, trends. A data scientist also acts as the bridge between the Executive who understands the business need and a database engineer. Companies are increasing relying on a data scientists to make sense of the unwieldy data sets in their organisation.

Big Data is the new gold mine for organisations. And tools are techniques are evolving to effectively mine it. Though organisations realise that they cannot ignore it, not everyone really knows where exactly the gold lies. Irrespective of the approach taken, perseverance and experimentation are key to unlock the value in big data.