The Agile Data Organization - Balancing Responsibilities in Data Science Programs

If you've read this blog or have seen me speak at a conference, then you know I am a strong proponent of self-service BI programs. I've posted on principles of self-service BI programs, attributes of data driven organizations, and how to avoid data landfills among many other big data topics all aimed to get business teams successful competing and driving decisions with data.

But success isn't driven by technologies, data practices, or the value of the underlying data alone. It is people and organizational structure that truly drive success and yet this is where I see many organizations make classic missteps. The problem is in balancing responsibilities and making decisions on who does what steps in the data management practices, and who owns what decisions.

Three classic mistakes

Here are some of the missteps some leaders and organizations make when considering how to manage big data or self-service BI programs:

• An overreached business team that tries to cut out IT from all or the majority of data management practices. In other words, data scientists on business teams trying to to turn "self service" to "complete control"


• An overgoverning IT team that tries to provide technologies and identify structured business practices on every step from data gathering, to processing to delivery.


• An overzealous PMO that tries to identify and label every part of the process and formally assign responsibility and decision making before the practice is in place and business value determined.

Hopefully you can visualize what's happening here. If you elect to be the overzealous PMO, you have a lot of up front work to define structure, process, roles, and responsibilities. If you choose not to predefine a structured practice with roles and responsibilities defined, then the organization will evolve its practice through experimentation and attempts to provide value. This is generally a good "agile" evolution, however, it, can lead to an imbalance depending on who has more organizational power and controls. Undisciplined business teams with little IT participation can lead to the first scenario and an overly controlling, "technology first" approach yields the second scenario.

What's the Solution To Getting A Balanced Business and IT Data Organization?

First and foremost, organizations need to recognize that this is not a unique problem to self service BI or data science programs. Agile product and software delivery teams are almost always cross-functional between Business and IT. The heart of agile is the product owner managing a backlog of features and enhancements, defining minimally viable solutions, working with IT on implementation scenarios, and prioritizing planning and development stories. Strong agile teams also have mechanisms to express and prioritize technical debt, larger business investments, and more significant infrastructure changes. 

The same practice can be applied to Agile Data Organizations, except that instead of prioritizing features, organizations look to prioritize big data questions. What questions provide value to stakeholders and customers that are worth answering? How do we attribute value and estimate feasibility on answering the question? How do we factor in other work such as loading in new data sets, data cleansing efforts, or improvements in data processing?

The next step is to get a team working together on discovery efforts. Once a multidisciplinary group understands priorities, there is a stronger likelihood that they will work together and disregard organizational boundaries and responsibilities.

Want to get started? See my related post on agile leadership practices to help data scientists.

But that's the start. There are some fundamental differences between software and data analytics that also contribute to the organizational discord. More to come!