Monday, February 23, 2015

Ten Ways To Improve IT Culture with Agile, DevOps, Data, and Collaboration

Over the last decade I've had the privilege of working with several IT teams that transformed themselves from minimal credibility with the business to delivering award winning customer facing products while improving other IT business services. Different groups with different issues and opportunities, but what stands out with all of them is their willingness to change and improve their own culture

So coming off several posts on DevOps and how its practices combined with agile development can lead to an improved execution, I thought to share some incites on how to recognize and improve IT culture.

Signs that IT culture is Heading in the Right Direction

  1. Agile teams get things done first, worry about process mechanics later - When introducing agile practices - or even when agile has been used for some time and the team is looking to mature its practice - I find that strong teams will think agile, focus on execution first, and address mechanics of the process second. So for example, a new agile team should gets its backlog going first and commit, worry about story points and how to handle unfinished stories at the end of a sprint later. A mature agile organization with multiple teams prioritizing stories, figures out the communication mechanics between teams during the sprint, and formalizes communication practices by discussing issues at retrospectives. 

  2. Know how to use the business' products so that they can see where and how to make improvements. User interface, workflow, and access to insightful, actionable data is so important to successful customer facing application design and business system workflows that the technologists working on them have to step into their user's shoes and learn for themselves.

  3. Individuals are hungry to learn more and take the initiative to train themselves - Sure I can get individuals and a team formal training, but that's not where it starts. A strong IT culture prefers rolling up the sleeves and experimenting first, training once they know the basics. 

  4. It's not always the Business' fault - Agile teams might blame the product owner for over promising and everyone has something critical to say about the business strategy, but strong IT teams will think through how to improve their own practices before tossing blame or being critical of other business functions.

  5. Speak openly about where they suck and need to improve is important. I want to hear about technical debt from software architects. Ops team need to look at metrics and decide where improvements can have business impact. QA needs to own up when their testing is inadequate. Most importantly, teams need need to speak up when they recognize a problem - and a problem well stated is half solved.

  6. Share information on how things work, document process, and educate colleagues on how to fix things. IT teams that horde information, over-complicate things so that business users don't understand how things work, or make it impossible for their colleagues to enhance their technical implementations are impeding organization growth and scalability.   

  7. Want to get involved understanding business priorities and challenges. They ask the product owner questions on the priorities or want to understand how their efforts are contributing to growth or efficiencies. They'll seek out multiple solutions to a problem and debate which ones provide optimal solutions. They'll listen to marching orders, but seek out to participate in the debate on what and how the business should move an agenda forward. They are strategic agile thinkers and learn how to innovate.

  8. Leverage data and ask questions - To become smarter about their own operations, they will collect meaningful data, convert them to metrics, look for trends, and prioritize improvements. They will learn to ask good questions about data, and take other steps to help become a data driven organization.

  9. There is an interest in spending time together outside of the office and having fun - My last team threw summer family picnics, and assembled everyone for pot luck Diwali celebrations. Good teams want to spend time together, celebrate wins, appreciate individual interests, and be human. 

  10. Are responsive when there are operational issues - It's one thing to have all the systems monitored, ITIL practices to handle incidents, playbooks to recover from an issue, and strong technical skills to research root causes - all things strong Ops teams work toward - but the best ones also develop a good bedside manner, are responsive to even the most difficult users, find ways to communicate using language everyone understands, and make sure business teams receive regular updates when there are problems.

And here is one bonus one - probably the most important

  1. Great IT teams refuse to fail, are willing to take on challenges and find ways to make their initiatives succeed.They are relentless to hit deadlines. They seek to keep things simple. They promote enthusiasm and heal downers and doubters. 

continue reading "Ten Ways To Improve IT Culture with Agile, DevOps, Data, and Collaboration"

Wednesday, February 18, 2015

Seize the Day! The Ops Role in DevOps Transformation

Reviewing some of my latest DevOps posts. I've covered why DevOps is important, asked the question Who Owns DevOps and answered that the CIO needs to own transformation initiatives. In my last post, I stated that agile development teams should stick to owning the practices tied to Development like automating testing and standardizing deployments so that they stay aligned and focused on initiatives that improve customer experiences, automate workflow or generate revenue

So, what is the role of the Ops team in a DevOps transformation? Can the team responsible for keeping the lights on, resolving incidents under expected SLAs, handling requests to deploy application upgrades, insuring that business systems are secure, keeping up with the latest patches and upgrades transform their practice with DevOps?

DevOps Practices is really About Agile Ops

If the CIO owns the transformation and Dev is largely providing a supporting role, then the opportunity to transform is really with Ops. Here's my extended rationale:

  • The real benefit in cloud environments is with enterprises that manage thousands of server instances and petabytes of data and where standardization and automation provide significant cost advantages. New configuration tools like Docker, Puppet, and Chef are designed to automate configuration management to these extremes - but it is Ops that should learn the technology and take on the responsibility of configuring them. Same responsibility (configuring systems) new tools.

  • Agile development teams have already aligned their effort to business driven priorities, but their frustration is when Ops or the infrastructure can't support frequent changes. Perhaps Dev needs to spin up new environments to test an upgrade or evaluate a new platform and it takes too long to configure? Maybe releases need to be scheduled more frequently but the deployment steps are too complicated? These are Operational accountabilities that might require contributions from the Dev team. Ops should collaborate with Dev to define application, configuration, and other changes that will drive the capability to release more frequently. 

  • The increase in cloud instances and applications - along with higher expected service levels from customers and stakeholders require an updated approach to monitoring applications and collecting performance data. Applications need many more monitors, real time alerts, and longer time frame performance trending. Ops teams need to invest in their skills to respond to this greater need.

  • Security is only going to get more business critical and more difficult to perform. Because Ops is in the front lines protecting security, they are in a better position to drive infrastructure and configuration standards that help simplify the disparity of assets that need secure configurations and support.

  • Server loads are increasingly more dynamic especially when many applications must handle variable loads driven by global, big data, or mobility computing needs. Ops are already versed in handling load balancers, application clusters, SANs, SDNs and other tools to dynamically scale system resources. The additional automation tools might require more "coding" but should not be outside of Ops ability to learn.
Ops should seize the day and take responsibility for these improvements and transformations. When agile development teams elect to tackle these challenges, they are doing the overall business a disservice by focusing on operational needs rather then business improvement drivers.

What about DevOps culture? Should Dev and Ops run as one team? Next post!

continue reading "Seize the Day! The Ops Role in DevOps Transformation"

Monday, February 09, 2015

Why Agile Development Should Not Own DevOps

Thanks to Ramon Monteiro @ivisualize for developing this cool visual depicting my last post on The Key to aligning Agile Dev and Stable Ops to a Successful DevOps Transformation. While agile development teams continue to iterate and improve applications for the Business, and Ops insures that systems are stable and reliable, the CIO leads the entire IT organization through a DevOps transformation.

Warning! Let Ops own Operations

Somewhere in the emergence of DevOps tools and practices some agile teams have taken the initiative and gone beyond their core responsibilities into system and operational domains. Some argue that because tools like Docker, Chef and Puppet enable software driven configuration management and scripted system deployments that it now falls under Development responsibilities to program virtual environments and automate their scalability.

Don't Cross The Streams! Developers Stay Left!

Review my last post. When I lead a group of developers, I target them to spend at least 70% of their time on development efforts that directly improve business capabilities. That means they only get 30% of their time to operational and support tasks like responding to incidents, addressing application lifecycle issues, resolving technical debt, and improving development processes. All of these are critically important, but keeping up with these responsibilities is a tall order and difficult to fulfill within their 30%.

Dev simply doesn't have the time to take on additional operational responsibilities. They should not be involved configuring servers or automating their configurations. If they are going to participate in a DevOps transformation, they should support their Ops teams by focusing on the Dev practices of DevOps such as standardizing their application builds, automating testing, or strengthening their version control practices.

Why Agile Development Teams Cross the Line?

Agile development teams by definition should be self organizing but if there is a need to take on DevOps practices they may elect to put this work on their own backlog. A product owner that senses infrastructure or operational gaps may support them. If there is a strong cultural divide with Ops, if there is weak governance on procuring infrastructure (including cloud), or if there aren't practices to insure development priorities are applied to appropriate technical functions then agile teams may cross the line and take on Ops' practices.

This isn't the answer. I doubt agile teams would like their product owners to start designing databases or developing technical standards (though some try to) because they lost confidence in their development teams. Development teams need to have a similar respect and collaboration with their Ops teams regarding operational responsibilities.

If Ops teams aren't getting it and DevOps is a strategic priority for the CIO, then he/she needs to work that out with the Ops team's management. If the CIO accepts Dev stepping in, then it might create longer term animosities or produce conflicts with fulfilling business priorities.

How can the CIO Help? 

A CIO should recognize when one team needs support and another team is overrunning them. If an Ops team is struggling with the new operational practices and tools of DevOps, the CIO needs to step in and pace the program accordingly Some options:
  • Define roles and responsibilities so Dev and Ops understand who owns what DevOps practices and where collaboration is required. 
  • Bring in a coach that adapts to the CIO's and organization's governance and culture. 
  • Recognize skill gaps and address by bringing in experts, investing in training, and providing sufficient time for practicing new skills.
  • Define reasonable scopes and goals especially on transformation initiatives.

continue reading "Why Agile Development Should Not Own DevOps"

Tuesday, February 03, 2015

The Key to aligning Agile Dev and Stable Ops to a Successful DevOps Transformation

So who does own DevOps? I left readers with this cliffhanger after my last post and my intro post on The What, Why and How of DevOps. If an organization is embarking on a DevOps transformation, how should it be led so that Dev and Ops teams are likely to collaborate more and battle less in adapting and maturing new practices?

To answer this question, I would suggest we get back to basics and review the role of Dev and Ops in an IT organization.

Dev Must Align with Business Priorities

The Development team should operate as the technology extension of the business. This means that they must be looking to provide expertise around enterprise applications and specifically on how business users can be more productive, drive data driven decisions, connect with customers, or make supply chains more efficient. It means that they have to be spending their time building or enhancing applications. When looking at how the Dev team spends its time, ideally they aught to be spending 70-80% of their time developing new capabilities and ideally only 20-30% on "support".

Therein lies a key question - what is support? I'm using Support as a catch phrase for time spent on any other non-administrative tasks that is not Development. It could be chasing an operational incident or diagnosing a defect. It could be upgrading versions of anything from an enterprise application to a javascript library used in an application.

And... it can also be anything technical to make the Development team more efficient. It can be investing time in configuring version control, automating builds, or scripting deployments. It can be time spent researching technology tools. All of this time, while very critical to an efficiently running Development program is time balanced against direct efforts to improve the business.

Many Dev teams have adopted agile development practices to better align with the business and to focus their development efforts. The successful ones will deploy frequently and push their Ops teams to support more frequent changes. But while Dev teams optimize for speed and agility, then may compromise stability and quality. When this happens, it is the Ops team that often feels the pain points of fixing what Dev released.

Operations Align with Stability and Reliability

While Dev is optimizing for speed and may compromise reliability, Operations has the opposite charter and look to insure the performance and reliability of the environment as their top priority. Their practices such as incident management and change management align with this charter and aim to protect the business from disruption. Disruption can be outages that affect customers and business operations, but also mean incidents that hamper organization productivity.

The byproduct of protecting the organization is often rigidity. Examples include infrequent time windows for change management or lengthy lead times to deploy new infrastructure. Ops teams may also suffer by only learning systems view of monitoring a system - the system is up vs. the application is performing as expected.

DevOps Practices Align Development and Operations

DevOps practices are designed to make Development more reliable and Operations more agile and nimble - effectively helping each organization with their weaknesses. For Development, practices such as automating test cases, scripting application deployments, and standardizing application builds all aim to improve the quality and reliability of hand-offs from Development into Operations. For Operations, standardizing system configurations, automating server changes, and scripting cloud operations are all tools to help automate Ops and enable them to be more agile with infrastructure. Therein lies the transformation, and more specifically a better balance between agility and stability.

So Who Leads DevOps?

The transformational CIO does. Transformation requires a senior leader to sponsor the investment, prioritize the effort, and market the wins. Transformational efforts that cover multiple practice areas need leadership to help prioritize and set the scope of the effort.

But regarding Dev and Ops collaborating through a transformation - the CIO needs to step in and redefine objectives, roles, and responsibilities. In what areas does Dev need to be more stable? To what business benefit and to what extent does Ops need to become more agile? When instituting a configuration management tool, what is Ops' primary objectives and where must Dev contribute?

The CIO needs to lead these discussions, set priorities and secure the collaboration required to make this transformation successful.

continue reading "The Key to aligning Agile Dev and Stable Ops to a Successful DevOps Transformation"

Tuesday, January 27, 2015

Who Owns DevOps???

An agile development team is humming along, maturing its practice, putting out regularly scheduled application releases, improving quality, and making business users happy. Then, something happens. Maybe the infrastructure needs an upgrade or needs to scale. Maybe there is a difficult to diagnose performance issue. Or maybe, the business and development teams are looking to deploy a new enterprise application to support a new capability.

The Development and Operations Clash

When any of these scenarios occur, the question is whether or nor the Operations teams are ready to partner with Dev on the timing, complexity, or scale of the task at hand. When there is a mismatch in culture, values, methodology, timing, skills, or operational practices, a classic Dev Ops clash arises. Finger pointing, bottlenecks, anger, or isolation may form between the IT Development and Operational teams.

It's not just business or development driven needs and issues that can drive this rift. Perhaps Dev is pushing changes and breaking operational procedures. Maybe there is a mismatch between an application and a required systems upgrade that's holding back Ops from maintaining an environment. Or maybe Dev isn't as good as they think they are in releasing defect free, secure, high reliability, applications that perform well and Ops is left holding the bag.

DevOps to the Rescue

In my last post, I covered the What, Why and How of implementing DevOps. The basic technical concept is that as you automate more of the interactions with the infrastructure from building, testing, deploying and monitoring you can remove many operational defects and better align Dev and Ops processes. From a practice point of you, the big questions are what tools and to what degree to invest developing any single DevOps practice area.

Who Implements DevOps

When I've discussed DevOps with colleagues, other CIOs, DevOps experienced practitioners, developers and with system engineers the big divide around it is who "owns" DevOps.
  • Does the DevOps starting spark come from Dev encroaching into Ops responsibilities? 
  • Is it Ops who standardizes configurations and pushes Dev to align their development and release practices? 
  • Or is it better to reorganize into a single DevOps team that collectively owns this practice and its maturity?

I'm not sure if there is a consensus best practice on this question but it certainly is one of the more heated topics around DevOps.

While all will acknowledge that DevOps requires a culture change there is some debate on who should take ownership and how to align the IT organization to achieve business and operational benefits.

I feel very strongly on this subject and will provide my perspective over the next 2-3 posts. Sorry for the cliffhanger folks. But here's a hint. Like all practices, how you go about changing to a DevOps practice depends on the goals, the issues, the opportunities, the availability of talent, and what resources are needed on other business priorities. This transformation has to fit the organization and I suspect most would benefit by documenting current processes and capturing metrics before jumping into the who, and how of DevOps. 

So before I get to some answers, start by collecting some metrics.
continue reading "Who Owns DevOps???"

Tuesday, January 20, 2015

The What, Why, and How of DevOps

I learned much about what I know today about systems operations from Bill, a head of network engineering that I hired for my SaaS company about fifteen years ago. We already built separate systems for managing dev and test from production, matured source code controls, and implemented some automation around testing and deployment. What Bill taught me is that we needed to separate developers from production environments because it required a different mindset, tools, and practices in order to keep production environments stable and scalable.

He was right. After removing access, instrumenting some basic change control practices, and improving monitoring of our systems we saw better stability and performance. We also saw better developer productivity largely because they were now more focused on development tasks and less on operations. There was some grumbling about procedures and some finger pointing when things went wrong, but as the CTO I accepted most of it since the teams behaved well most of the time and there was measurable improvements in both Dev and Ops.

Today's DevOps

Flash forward to today, and many of the tools and practices Bill instituted have matured and are core elements of a "DevOps" practice. Since many production environments are now in the cloud or virtual hosts and can grow to thousands and tens of thousands of servers, the operational focus of the practice is on automation. Specifically:

  • Configuration management tools store infrastructure configurations in a database and also help automate setup and integration.  New server, install application, add storage are all operations that can be automated.
  • Code deployment tools integrated with configuration management know what environments exist, provide visuals on applications deployed and enable automated mechanisms to deploy new application version or back out ones if there are issues. 
  • Testing tools make it easier to automate tests and configure them to run during application builds or deployments. 
  • Monitoring systems automate data capture from many disparate systems and provide tools to diagnose root cause of complex performance or stability issues. 

DevOps Culture

DevOps is also about the culture of aligning Dev and Ops teams so that they have a single mindset on improving the customer experiences. For Dev teams, that usually means standardizing platforms, following an agile development process, and participating in Ops practices. For Ops teams, it means instrumenting many of the tools identified above and targeting better stability, improved costs, or better responsiveness.

Maturing agile delivery and DevOps over time, will lead teams to support continuous delivery - the ability to push code out to Production frequently. For companies in highly competitive industries and where technology provides differentiation, the ability to make frequent application deployments can be a differentiator.

What's Not to Like?

All this sounds too good, and it might be for many organizations. First, many organizations will debate tool selection especially when operating in heterogeneous system environments. Are these new tools or replacing existing ones? In many situations, it's a mix of both and the complexity of funding, selecting, and instrumenting new tools will paralyze some organizations before they start.

Second, like agile development these practices are a lot easier to instrument at startups and small companies where the infrastructure can be built from the ground up with these tools, and DevOps practices established from the start. In larger or more established organizations,  it takes investment, time, talent and executive backing to make this happen - something I predict will be a major IT transformation in the next year

Finally, like agile development, DevOps has basic principles but it does not have a manifesto the caliber that agile development has and had from its beginnings. This means that there is significant variance to what DevOps means from one company to the next. The debate on what practices to implement, how they should be implemented, and how to manage the DevOps organizational structure will require a strategy and planning.

Aligning DevOps with Business Goals

Like all IT practices, it's important to start with business goals and measures. Perhaps the organization has to improve stability? Maybe there is significant growth and a need to manage costs and complexity while the business scales? Maybe the business is in a technically competitive industry where continuous delivery is important? Perhaps you are working in a regulated industry where quality and repeatability of the deliverable is required? These drivers should lead to an IT strategy that help develop an operational strategy and prioritize  investments.

Second, management has to be very active when considering the organizational structure and cultural aspects of this transformation. In addition to aligning with goals, changes in these areas should be architected to better align IT with the business, and not just help instrument DevOps practices.

Where to Start?

The very best place to start? I suggest taking measurements of current state before instrumenting too many changes. How stable is the production environment? How frequent are application changes being deployed? How often are applications deployed with significant defects? How often are you scaling systems to meet demand? 

Of these metrics, which ones need the most improvement to match business need? Prioritize 1-2. Now you can start thinking about DevOps.

continue reading "The What, Why, and How of DevOps"

Tuesday, January 13, 2015

Best Data Visualization and Dashboard Management Practices in Self Service BI Programs

I've seen many organizations run into difficulty defining governance and best practices managing the lifecycle of BI dashboards. Perhaps you have deployed a self service BI program - data scientists have access to tools, IT has deployed severs to publish completed dashboards, and employees are anxious to leverage visualizations and become more data driven. Job done right?

Not so fast. Flashback 10-20 years ago when business users became proficient with MS Excel, MS Access and more advanced BI solutions. Perhaps too sufficient and you have an organization of spreadsheet jockeys and a data landfill instead of a landscape. Has the organization ended up with an explosion of spreadsheets and reports? Ever have an issue where decisions were made on a report made with buggy calculations? When you open a report, do you have questions on what the columns mean, where are the sources of the data, or what is the logic behind an aggregation?
This is common and occurs because organizations fail to put in some basic governance practices around data, reports, dashboards, visualizations, and analytics. 
I stress the word basic - not overbearing, but recognize that a self-service BI program without basic data governance may provide value but will slow down before achieving its full potential.

Best Practices in Self Service BI Programs

So let me suggest some starting points.

  1. Define a life cycle - I strongly suggest organizations manage dashboards like applications. They are developed, they need to be tested, their needs to be some documentation, users need to be trained, they need to be published, feedback from users needs to be gathered, enhancements should be prioritized. This doesn't need to be as onerous as it sounds, but having these disciplines insures that dashboards developed provide enough value and worth the effort. Testing, documentation, and training insures that dashboards developed are leveraged by a audience wider and hopefully prevents the duplication of effort in creating similar or derivative dashboards.
Implications for IT
    • Data scientists need a central place to store source files; spread sheets, Tableau workbooks, Qlik application, etc. Ideally these should be stored somewhere where files can be versioned and tagged.
    • Servers should be configured to handle the equivalent of dev, test, and production.
    • IT should consult with data scientists to consider tools, templates, and storage of documentation.
  1. Documentation should focus on data flows, data definitions, calculations, aggregations, and known data quality issues. Data scientists joining the organization need to have a strong understanding of their starting points while dashboard consumers need to be able to use them and interpret the results. Organizations need to define documentation standards for these audiences and when in the lifecycle they should be updated.

  2. Define style guides covering layouts, control choices, common components, visualization types, color palettes and other design considerations so that there is consistency between dashboards developed by different data scientists.

  3. Testing should focus on insights, calculations, and usability. Do the results look reasonable? Are calculations providing expected values? Are the dashboards usable and intuitive or complex and clumsy? The lifecycle should be developed for iterative review and feedback. Dare I say agile data practices?

  4. Governance practices should target reviews of the quantity and disparity of dashboards developed. This is really important in order to avoid landfills because if everyone develops their own dashboard versions that largely perform the same analytics, the sheer quantity of them will make it more difficult for users to navigate and possibly provide conflicting results. It will also add complexity when data models or software systems need upgrading.

  5. Measure usage and impact because data visualizations and dashboards not being used should be phased out.

Evolve the Practice

Not all of this needs to be defined up front. Define these and other practices as needed.

continue reading "Best Data Visualization and Dashboard Management Practices in Self Service BI Programs"