Big Data seems to be the pervasive buzzword these days so I thought I would take a moment to shed some light on this topic being that Data & Decision Analysis and Predictive Analytics are big interests of mine.
What is Big Data?
First and foremost, it’s data that is being collected from numerous sources at a very rapid pace and comes in a variety of forms; accordingly, analysts and industry “experts” describe Big Data using 3V’s – Volume, Variety and Velocity. Before we go any further, let me remind everyone that this description of data is not new – many analysts and experts described the vast amount of data made available during the early dot-com days using the 3V’s as well.
So what’s the big deal? What’s all the hype about?
Short answer – in my view, the ‘Big’ in ‘Big Data’ describes not the data but organizations’ challenge in making sense of the voluminous data and deriving insights or strategic value from that data.
I am a strong believer of the notion that data, big or small, is just data. An excel spreadsheet with 1000 records is no different than a 100 terabyte database with 100 million records if you don’t have the right people, the right technology, or the right processes in place to analyze and derive insights from the information.
In order to compete on analytics and gain strategic value from data, organizations’ talent recruiting plans should include, at the minimum, the following three key resource: 1) Leaders & Managers – who understand the organization’s strategy and objectives in terms of KPIs and metrics, 2) Enterprise Data Warehouse/BI Experts – who can implement the best technology to retrieve, store, and maintain the data, 3) Analytics & Insights Professionals – who understand the data (functionally, not technically), identify the data needed to answer a business question (described by the Leaders & Managers), articulate the data needs to IT in order to retrieve the data, and finally, analyse the data using basic and advanced statistical techniques and derive actionable insights.
I defer to the Enterprise Data Warehouse and BI experts on the technical specifications, but, fundamentally, organizations need to focus on managing the data by securing technology to store the data in more efficient formats for rapid access, split data sets into manageable chunks based on function or need, use NoSQL for large, non-shrinkable data volumes, and archive unused or limited use data to make space for value-adding data. Additionally, organizations need to procure enterprise licenses for and train the Analytics & Insights on statistical/analytical software such as R (open-source) and SAS, in order to perform the basic and advanced analytical techniques on powerful, dedicated servers.
The final critical component to competing on analytics is having the right processes and methodologies in place. Regardless of what framework you come up with, at the minimum, that framework should consist of the following activities and be repeatable: 1) defining your organization’s strategy and corresponding KPIs and metrics, 2) monitoring those leading and lagging metrics (this should include information governance and master data management), 3) hypothesizing key drivers that impact those metrics, and finally, 4) confirming/validating your hypothesis using the information available to you. Doesn’t this look very similar to the scientific method?
My recommendation to everyone intimidated by ‘Big Data’ is to just take the ‘Big’ out of it and focus your efforts on coming up with a comprehensive information management strategy that considers the aforementioned items. And always ask yourself, “Am I setting up the organization to get the most value out of the data, big or small?”
So, there you have it. As always, would love to hear your thoughts, so please feel free to leave a comment!