Thursday, May 16, 2013

Transforming to a Big Data Organization

What's Driving your Big Data Transformation Priorities?
What's more important? Big Data infrastructure including cloud computing, storage, Hadoop and other data processing engines? Is it the algorithms used to automate and derive intelligence from the data? Or is it the talent of the data scientists, the visualizations that they develop, the stories that they tell, and the questions that they answer?

If you believe the technology companies marketing Big Data solutions or a large number of technology media journalists they will  focus on Big Data infrastructure. Organizations that already have a strong practice in analytics and reporting are more likely to focus on improving the talent, tools, and capabilities of their data scientists. Companies that already have a strong software practice and develop algorithms for other strategic needs are more likely to pursue machine learning and other data mining algorithms.

 

What's Driving your Big Data Transformation Priorities?

 

From my perspective, it appears that organizational strengths and dynamics are driving the priorities in Big Data capability investments. This isn't necessarily a bad thing and probably smart for organizations by investing in their strengths. But it isn't sufficient and at some point, a balanced set of investments and changes over a longer period of activity will likely be required.

In other words, while leveraging your organizational strengths may be an easy on ramp to Big Data insights, it probably isn't a complete holistic approach. At some point, many organizations that truly want to become "data driven" must transform or invest in new capabilities.

So in thinking about this transformation, here are some simple questions and guidelines on where to focus efforts

  • How big is your data? - The bigger the data, the more likely you will need to look at infrastructure to store, process, and manage larger data sets.
  • How fast do you need results? - The more likely your business derives value in presenting results faster for direct revenue, real time decision making or competitive advantage, the more likely you will need algorithms to directly churn data to drive other systems or decisions. Examples include Hadoop and other machine learning APIs.
  • How complex is your data? - Complexity comes in many forms. It may be unstructured data, data that has rich relational metadata requiring subject matter expertise, or data that's sparse and has other quality issues. In these situations, organizations will likely look to its subject matter experts and ideally data scientists to help provide insights, but they need to work differently. Data scientists are not glorified spreadsheet jockeys and produce different results than statisticians or BI analysts.

But you can see these are not mutually exclusive capabilities. If your data is big, you certainly are going to need algorithms to process the data and people to interpret the results. If you need fast results, you will need to consider the architecture to best process the data. If your data is complex, eventually you'll want to develop algorithms to help automate some of the results. Then, there are other challenges beyond the volume, variety, and velocity of data, and organizations need to consider how to scale their big data practices. I believe that this transformation is a big opportunity for CIOs looking to bring Big Data intelligence and analytics to their organizations.

So Big Data is not a solution or a capability. It is a transformation journey of developing new capabilities through people, process, and technology.

1 comment:

  1. This is a great article. I recently have moved my business to a new location and I have been working on data transformation. I know there are systems that can help me transfer my data in any electrical format to anther format that is easily accessible.

    ReplyDelete