Tuesday, December 11, 2012

What Is Big Data? The Real Challenges Beyond Volume, Velocity and Variety

Big Data's Challenges
I was asked this week by a colleague, "What is your definition of Big Data"? Here's how I responded.

The technology industry describes Big Data in technology terms such as Volume, Velocity and Variety of data or too much data to store and process in traditional data warehouse infrastructure. This might help make technologists aware of the challenges in data management around large, changing, and unstructured data sets. It helps technology vendors sell new capabilities to address these challenges, the media to rally CIOs to invest in them and build awareness around the challenges attracting talent to analyze them.

But that's not a good business definition of Big Data.

Big Data for All Businesses

Every enterprise already collects lots of data, but under utilizes it for intelligence, insight, and decision making. Data exists in disparate databases that can't easily be connected and analyzed holistically. Unstructured data in the form of documents, spreadsheets and presentations exist that are largely used for individual or departmental needs and rarely coalesced and analyzed. Enterprises that have deployed collaboration platforms have the opportunity to better leverage the networks and intelligence of its employees by analyzing relationships, contributions, and consumption on these platforms. Data collected in workflow solutions such as ERP and CRM are rarely merged and analyzed for intelligence. Email is certainly one of the largest repositories of unstructured data.

Bottom line is, all enterprises already have big data. By my definition, Big Data is not defined by its data management challenges, but by the organization's capabilities in analyzing the data, deriving intelligence from it, and leveraging it to make forward looking decisions. It should also be defined by the organization's capability in creating new data streams and aggregating them into its data warehouses.

Volume, variety, and velocity of data define the overall size and complexity of Big Data's data management challenges and for some, the size requires a different architecture than traditional data warehouses. But the real Big Data challenge, and the challenge for all business regardless of size or complexity is in transforming to a data driven organization driven by analysis and insight of existing and new data streams.

See my previous posts on this subject including Big Data, Big CIO Opportunity, Big Data Needs to Scale, Big Data's Managerial Challenges, and my Top Five Tools of Big Data Analytics.

4 comments:

  1. Excellent post Isaac. Since we (I) at Gartner first introduced the "3Vs" over 12 years ago (ref: http://goo.gl/wH3qG), we have also enhanced our definition of Big Data to include the use cases of enhanced decision making, insights and optimization. --Doug Laney, VP Research, Gartner, @doug_laney

    ReplyDelete
    Replies
    1. Doug - Thanks for the comments. My fear is that when Big Data's hype wears off, business executives will only recall the data management attributes. Some industries like healthcare will see transformational changes by utilizing better data analysis but only if individual businesses invest in data science.

      Delete
  2. also improved our meaning of Big Information to consist of the use situations of improved making decisions, ideas and marketing. Outstanding publish Isaac. Since we (I) at Gartner first presented the "3Vs" over 12 decades

    from an essays site

    ReplyDelete
  3. look.....essay writing service
    Great post! Interesting information and cute writing style. essay writing service
    but essay writing service I dont know why I always visit here =))

    ReplyDelete

Share