Big Data's Managerial Challenges

In my last post, I declared that Big Data Needs to Scale. In particular, CIOs and technology leaders have to get beyond the data management challenges of the volume, velocity, and variety of data and step up to the leadership, management, and talent challenges in using the data to drive decisions. CIOs need to help justify the investment in the organizational change efforts to support big data analytics. They also must cultivate new skills and roles such as data stewards and data scientists.

Harvard Business to the Rescue

Harvard Business' October/2012 issue has several gem articles on Big Data and Big Data: The Management Revolution by Andrew McAfee and Erik Brynjolfsson is a must read. Here are a couple of highlights:

 

On Big Data ROI

The more companies characterized themselves as data-driven, the better they performed on objective measures of financial and operational results. In particular, companies in the top third of their industry in the use of data-driven decision making were, on average, 5% more productive and 6% more profitable than their competitors.

On Big Data managerial challenges

The technical challenges of using big data are very real. But the managerial challenges are even greater. “What do the data say?” Where did the data come from?,” “What kinds of analyses were conducted?,” and “How confident are we in the results?”  

The article ends with five management challenges, and while technology is on the list, it is not in the pole position. I suggest reading the article to get the answer.

About those Managerial Challenges

In my post, Dear Spreadsheet Jockey, Welcome to Big Data I suggested that the era of using spreadsheets as the Swiss army knife of data is over. More advanced tools with such business friendly names like ETL, Data Quality, MDM, and BI are delivering more functionality and data processing transparency, at a cost of developing new talent, expertise, processes, and governance. So an investment in Big Data - even if your data is not that "big", is a journey. CIOs have to train their data subject matter experts to play out the more defined role of data steward, where they are given tools to maintain the integrity of data sources. We have to make sure that Data Scientists are not this generation's spreadsheet jockeys with shiny new tools. Scientists have to be educated on not just providing the answers and insights, but on delivering dashboards and other discovery tools to equip decision makers to make more informed, data driven decisions.

My next post on this topic will cover Data Scientists. Until then, the media is finally catching on to the managerial challenges and while some of these posts are hyped up, they do provide some insight.

Companies would do better at satisfying and retaining customers if they spent less time worrying about big data and more time making good use of ‘small data’ — already-available information from simple technology solutions — to become more flexible, informative, and helpful. - You can't fix stupid, and other arguments against big data.

So, when the advertisements claim that big data will transform your business, remember that big data brings the potential for transformation, not the actual transformation. - The big data fairy tale

Pursuing big data with small, targeted steps can actually be the fastest, least expensive, and most effective way to go.- Big data doesn't work if you ignore the small things that matter