Agile Data Scientist, A Disciplined Hiker or Reckless Hunter?

In my last post, I provided some guidance on why and how agile data practices can lead to better Business and IT collaboration. Agile aligns priorities, focuses multidisciplinary teams to hit goals, and enables teams to self organize and figure out individual responsibilities. As the team succeeds, management can then map our roles, responsibilities, and other governance considerations.

Data Insights are a Journey

As I ended my last post, I suggested that there are some fundamental differences between agile teams delivering a software driven product versus the analytics and insights that data science teams strive to deliver. Product teams are optimizing against scope, cost, and time to complete their delivery and agile teams often fix time and cost while making tactical decisions on scope. Unlike product and software deliveries, data teams are less likely to have structured deliverables like product launches or software releases. As I explained to CIOs contemplating their data strategy, data science and analytics is closer to a journey and not a destination or milestone. This key difference leads to a different way of thinking about agile data teams.

Hiking or Hunting Your Way to Insights

Picture a hiker in the wilderness who is trying to find the most interesting locations to photograph and is using her skills and tools to find vistas, waterfalls, and wildlife. When the hiker has a clean line of site to an interesting destination, she will move with vigor to capture it. Other times, she will navigate the dangers of the wilderness in a search, often stopping to check her gear, set up camp, or completing other necessities needed for a long term journey.

Data scientists are in the search for insights, and much like a nature photographer, know they've found something insightful when they see it. Until then they are on a search or hunt using a combination of their skills and data tools to support their discovery efforts. 

Now there are some very disciplined hikers who are well equipped and methodical in their approach. When faced with adversity, they have the tools and skills to address challenges without compounding to the risks they are facing. There are also more adventurous hikers that act more like reckless hunters; they are so fixed on the kill that will take on additional risk in order to meet their objectives. (Note: I should point out that there certainly are reckless hikers and many disciplined hunters out there. My point in the analogy is to illustrate differences in both behavior and persona.)  

Agile Data Scientists

The same is true for data scientists. Complexity lies in the form of slow data processing tools, technical difficulties in getting data integrated, structural issues with how the data is stored, data quality issues and other impediments that complicate data discovery efforts. Some data scientists will collaborate to improve the underlying tools, data structures, data processes, or other infrastructure barriers in order to achieve their current and future goals. Other more scrappy scientists focused on just getting the job done will engage in bad data practices, create silo databases or perform adhoc analytics.

So there in lies two major differences between product and data agile practices:

• Product organizations march to milestones like launches and software releases, data science is more of a journey.
• Agile product teams evolve products around a stable set of platforms and infrastructure. Data scientists have to choose if and when to be disciplined because there are many tools that can easily bypass defined data structures and practices.

These two differences are key to managing data science practices and big data technologies. More to come!