Five DevOps Lessons from a CIO Getting in the Weeds on AWS

DevOps - Get in the weeds
I always advise CIOs and technologists to get into the weeds to figure out what's going on. So last week I spent some time getting a number of AWS services up and running. Nothing too crazy, but successful.

My first project was to set up a new domain, on AWS Route 53 to enable landing pages and vanity URLs for my upcoming book, Driving-Digital: The Leader's Guide to Business Transformation Through Technology. While it's been a few years since I last set up DNS, the basics are fairly straightforward. What I had to learn was how to properly set up S3 to act as a web server and use the redirection rules to enable a set of vanity urls. Again, not too complicated especially if you configured an Apache web server.

My second project involved setting up an Amazon RDS Postgresql database with a VPC to enable selective inbound traffic. My goal was to enable an ETL service to automate moving data from my Google Analytics account into a database and then use Tableau to join data and build visualizations. This setup took a bit longer as I had to figure out how to setup a VPC properly and then grant appropriate permissions in the database.

My third experience was working with a client looking to setup a Node.js application on Elastic Beanstalk using Travis to automate deployments.

DevOps Lessons Learned in Cloud Configurations

So here are some of the lessons learned from my short and simple exercises -

  1. You can teach an old dog new tricks - If you've configured infrastructure in a data center, then enabling similar basic capabilities in AWS largely requires learning a new vernacular and tools. Again, I was only doing very basic DNS, VPC, web server and database configurations but was able to accomplish these tasks without training classes or reading extensive manuals. When I got stuck, I Googled for answers. For CIO, if you have system administrators and engineers that largely operate tech in the data center, getting them to operate similar technologies in the cloud only requires some basic training and time to experiment.  

  2. Planning required for more complex tasks - Setting up S3 to run a barebones web server is straightforward, but it doesn't have anywhere near all the capabilities of an Apache web server. Like all infrastructure needs, it's important to know your basic configuration requirements before selecting a technology and its implementation.

  3. Debugging is still hard - When I couldn't connect my ETL service to the database, I had to go through basic debugging steps to determine what network layer was blocking the traffic and how to configure it properly.  This took several trial/errors and the tools to debug weren't immediately obvious.

  4. Online documentation is very good, but not ready for DevOps - The documentation I poured over was very good explaining how to do basic and more advanced configurations. I  was also able to find reasonable documentation on solving problems once I had already recognized the source of the issue. It was poor in describing tools or steps to debug, or providing links to best practices. For DevOps, I was hoping to see some separation of duties to determine what steps should be automated as infrastructure configuration versus what should be setup as part of deployments.

  5. Move to Infrastructure as Software quickly - Tracing through a manually configured cloud environment is only slightly easier than doing the same task in your data center infrastructure. At least the cloud is web based and most of the tools are centralized, but navigating different configurations, naming conventions, and pipelines can become very complex if you're tracing through someone else's configuration. As CIO, we all thought setting configuration standards and moving to more homogeneous architectures would address the complexity, but I suspect only a small percentage of IT organizations are successful with this approach. On the other hand, if the configuration is automated in well structured and documented code, then there are fewer places and artifacts required to learn and maintain a computing architecture.
So today's post comes to you from my time in the weeds!

continue reading "Five DevOps Lessons from a CIO Getting in the Weeds on AWS"

Driving Digital: The CIO's Playbook

Thanks to Landmark Ventures and Anthony Juliano for a great CIO Summit last week and for giving me a speaking spot for Driving Digital: The CIO's Playbook.

If you weren't there - here's what I covered

How are you executing your Digital Transformation Program

These practices and many others are described in my upcoming book, Driving Digital: The Leader's Guide to Business Transformation Through Technology that will be available August/2017. In addition to agile transformation and data science programs, I cover product management, the IT organization of the future, ideation and portfolio management, data platform architecture and DevOps. Most importantly, I cover how to align the executive team on digital transformation and on changing to a digital mindset.

I've developed a page to sign up for announcements on Driving Digital and to get access to related services that I make available.

The talk is only ten minutes. Watch below!

continue reading "Driving Digital: The CIO's Playbook"

Aligning product owners and business stakeholders to a release management strategy

Aligning to a release management strategy
I have found that the hardest part of getting a release management practice instituted is in getting the business alignment. Development teams often recognize the need to develop a release management strategy. If they are strong technically, they may look to mature their DevOps practices to support continuous deployment if it makes sense for their business. If not, they may build continuous integration and delivery capabilities, then look to implement a more traditional release and deployment strategy.

Even when product owners and stakeholders are involved in formulating the strategy, they often struggle adhering to this in practice.

It's not always their fault. Product owners have their customers to satisfy. Some customers can be very demanding, or the product owner may just be feeling the pressure for high customer satisfaction. So for the purpose of this post, I'm going to assume that there are rational business reasons for a product owner's need for flexibility in the release practice and this isn't just a product owner behaving badly. If it is, here are my suggestions on handling difficult product owners.

Release Management Anti-Patterns

What are some of the issues that misalign business need with a development team's release management practices?

  • There's a problem and I need a hotfix
  • Users are complaining and we need an improvement fast
  • There's a new marketing event scheduled and the app needs
  • We're driving a price increase and need a big feature release 
  • How can we make a big workflow change easier for users 

Let's look at these one by one and consider mitigation strategies

There's a problem and I need a hotfix

The technical solution is to make sure your branching and testing strategy supports hotfix branches. See one example. But the bigger problem is the cost to implement the hotfix especially if the product owner is prone to request many of them. First, it disrupts the development team to branch, fix, and deploy the hotfix. Second, it requires at least two rounds of testing; one when the hotfix is deployed and a second it is merged into the main development branch. Here's what I suggest
  • Make sure the product owner understands the tradeoff by reducing the commit in the current sprint to support the hotfix.
  • Illustrate and build awareness around the added testing costs
  • Measure and benchmark the behavior (# hotfixes / time-period), communicate the impact to the roadmap, and single out product owners that have higher hotfix requests.

Users are complaining. We need an improvement fast

This is more likely to happen with internal applications deployed rapidly and where users may have different workflow needs. It may also happen if release cycles are too long and users prefer seeing more incremental changes. Suggestions
  • Make sure to capture and reflect if the needs of a specific user group were missed.
  • If this happens infrequently, consider scheduling minor releases before going onto your next major release. See more specifics on major/minor releases.
  • If this happens frequently, consider shortening your release cycle.

There's a new marketing event and the app needs

This is more specific to products or customer facing applications. Ideally, marketing events should be considered when developing the overall product and release roadmap. That being said, it's fairly common for an agile marketing team to find new opportunities requiring changes to the release schedule and feature prioritization. Suggestions:
  • Make sure the marketing team is engaged when developing roadmaps and release schedules.
  • Develop a communication/collaboration process so that you are informed as early as possible when a new marketing opportunity may impact the roadmap.
  • Make it easy for product owners to update priorities in the agile backlog.
  • Develop a branching strategy that can accommodate these opportunities.

We need a big feature release

This shouldn't really be an ad hoc business decision and this business strategy needs to be reflected in the product roadmap and the release strategy. If it comes out of nowhere, it probably means that there is a shift in business strategy, or this may be a onetime event. Suggestions:
  • Make sure the team understands whether this a shift in business strategy that should require a review of the roadmap and the release strategy, or if this is a onetime event.
  • Consider having shorter and incremental "alpha" releases for internal users and longer "go-to-market" releases that are customer facing.

Can we make a big workflow change easier for users

This suggests that you have the capability to roll out incremental improvements to a subset of the user community, or, have the ability to do A/B tests when rolling out new capabilities. Suggestions:
  • If you need to roll out changes to different customers or user communities incrementally, then this capabilities has to be developed within the application. Add it to the backlog and develop a rollout strategy based on the application's authentication groups and roles.
  • There are many third party tools to support A/B testing or you could build the capability in house. If A/B testing is needed, just make sure it's developed as a horizontal capability so that it can be leveraged across multiple applications.
If you send me any other anti-patterns, I'll look to update in a future post.

continue reading "Aligning product owners and business stakeholders to a release management strategy"

Defining Your Agile Release Management Strategy

Release Management
In my last post, I asked whether continuous deployment is right for all businesses and suggested a number of business criteria and technology capabilities to help determine whether organizations should go beyond continuous integration and continuous delivery to a full continuous deployment operation. I recommended in an earlier post that release management strategies are important to agile practices, however, maturing to continuous deployment is just one, rather advanced approach, Organizations can still achieve high quality software releases and develop product delivery roadmaps with more traditional release management practices.

The Problems with Ad Hoc Release Management Practices

Many organizations start with ad hoc release practices and leave it to individual product owners with or without the agreement of their development teams to decide when applications are ready to be released, when to schedule release dates, and how frequent to schedule them. Leaving it to individuals or teams to make these decisions leads to several potential issues. First, these teams usually don't act in isolation and their schedule may create conflicts when they require participation from shared service teams. Second, the ad hoc nature makes it more difficult to forecast roadmaps since the amount of overhead tied to different release schedules may not be consistent from release to release. Third, these teams can create quality issues if there isn't some guidance set on what types of application changes are appropriate for shorter versus longer release cycles. Lastly, these teams tend to want to make "quick fixes", "patches", "support changes", or "emergency changes" frequently and without realizing that this makes it difficult to focus on more strategic application improvements.

Over the years, I've been brought into several organizations stuck in this cycle of frequent patch releases. The team makes one release, the product owner reports user complaints or production issues and demands fixes, the team complies with a patch release that only partially fixes the issues, the product owner reports additional customer feedback as defects, the team chases it with another patch release and this cycle repeats. If this continues, users get frustrated with the quality of the application, teams get the reputation of having poor practices, and product owners fail to deliver on their strategic roadmap.

Getting out of this cycle is critical and many teams can't do it easily on their own. They need someone to lead them through steps that (i) recognize the issues created by ad hoc release practices, (ii) define release principles that make sense for the business strategy and for users, (iii) take active steps to normalize their release management practices.

Defining Release Management

Here is a summary of the steps I specify with organization that want to go from ad hoc release schedules to a normalized release management practice

  1. Define a frequency of change that makes sense for the business strategy and for users. A B2C company that targets small experimental changes may want more frequent releases than a B2B company that's producing analytics for their customers. Keep in mind that any release that contains new capabilities, functional changes, workflow changes, or substantial UI changes often requires some form of testing and user training, so the release cadence should align with how easy it is to engage the user community on new capabilities.

  2. Define release types that correspond to the impact to users and technical risks. Presented below are three typical release types.

    Release TypeScopeTypical Duration
    MajorNew features and capabilities. Workflow changes. No changes to underlying systems, app infrastructure, and libraries3-6 sprints
    MinorSmall changes and fixes. No new capabilities or system changes2 sprints or less
    System UpgradeChanges to systems, app infrastructure and libraries with no or minimal functional changes1+ sprints to test, then validate impact

    You'll notice a couple of things. First, I instill testing disciplines by having releases focus on application changes (major and minor releases) versus system changes (system upgrades) which provides scope to the testing required. Second, minor releases are characterized by shorter durations that have less time to engage users on changes, so the scope of minor releases is tied to improvements that have less impact and technical risk. Third, major releases don't have undefined durations and the duration should be normalized to the business strategy and technical capabilities of the organization. Lastly, it's hard to determine the duration of a system upgrade, so I advise teams to plan at least one sprint to test the upgrade and add to the backlog the issues that need remediation. Once these are identified, the team can then project the duration for this type of upgrade.

  3. Your release starts when planning the release, not when you're ready to deploy. This is a change  the team needs to recognize. Deployment planning which goes into defining the technical steps to push an application into production is not the same thing as release planning. Release planning should start before any development starts. It should start by defining the business goals, selecting the release type, and defining at least part of the backlog for what needs to be developed. It should include defining what business activities are required pre and post the release. Once this scope is understood, then a target date can be selected for the release.

  4. Look to establish a schedule of releases. Once a team has been through at least one successful release of every type, it can then explore developing a release strategy. This strategy should factor in how many major releases the business team requires, how many system upgrades are needed to keep technologies up to date, and how best to schedule minor releases. 
Once these practices and governance are in place, then optimization leads to shorter release cycles and the capability of performing more frequent releases. I'll leave that to another post.

Further reading 

continue reading "Defining Your Agile Release Management Strategy"

Is Continuous Deployment right for your business?

It's easy for technologists to target a challenging architecture or development practice, after all, they love solving technical challenges. So as automation has become a key capability of software development, and organizations have begun to adopt DevOps practices many development teams are striving for the "holy grail" of continuous deployment.

Some product owners also love the notion of continuous deployment. Ask for a feature today, build tomorrow and deploy the same day. Seems too good to be true.

My question to you is, do you have all the technical capabilities to truly enable continuous deployments? Is your business ready to establish a competitive advantage with this capability and has the right mindset and culture to support experimentation? Are the application users ready to embrace rapid changes to the user experience and workflows? Does your organization have the size and scale of application development that warrants the investment in continuous deployment?

Continuous Integration vs. Delivery vs. Deployment

To answer these questions, let's start with the basics by reviewing the three practices of continuous integration, continuous delivery, and continuous deployment.

Most experts agree that continuous integration, the automation of integrating code, building software packages and executing automated test cases is an important baseline for development teams. Even for the smallest teams, the ability to click a button or run a command to see that the changes implemented have successfully been incorporated into the application and tested is an important efficiency. To receive immediate feedback that the code you just checked in failed to build or pass a regression test enables the developer to make fixes before he or she moves on to the next agile story. If the time to run continuous integration scripts is fast, developers are more likely to check in code frequently leading to better quality.

Organizations that support many applications and development teams can also benefit from continuous delivery. This automation picks up where continuous integration ends and automates the steps to push software builds to a destination environments. It can be a development environment, testing, staging, UAT, demo, or a production environment.

Automating delivery not only saves time and effort, it's made advantage is quality. Manual deployments, especially ones that require multiple steps that differ by environment or type of deployment are highly error prone. For those of us that survived the days of manual deployments, we'll recall nights spent chasing configuration differences between test and production environments that caused a code push to production to fail. If you're a developer, you probably have bad dreams of resolving issues when a system admin skipped a critical step in a delivery. If you're that system admin, you likely have pent up frustration from the complexity and frequency of development changes. Continuous delivery not only addresses these issues, it potentially improves the culture as there is less blame from the human errors associated with manual deployments.

The next stage is continuous deployment whereby new code is built, tested, delivered to environments, and fully deployed to production frequently. Some organizations translate continuous deployment to a production push on a frequent schedule like daily or hourly while others will deploy with every build. The first time I saw it in action I was both amazed and frightened on the level of automation achieved and the frequency of changes being deployed to the application's customers.

Is Continuous Deployment Appropriate for All Businesses?

Like I said, it's easy for development teams to want continuous delivery, but that doesn't mean it makes businesses sense. First let's consider why some businesses may want continuous delivery -

  • They are in a hyper-competitive industry where they need to deploy frequent changes to improve customer experience or test new capabilities. Many larger B2C companies fit this profile.
  • Business leaders promote a culture of experimentation and "failing fast" paving the way for development teams to try new things, promote the ones that succeed and rollback the ones that were misses.
  • They have also developed sophisticated ways to test changes using A/B tests, user experience metrics, and surveys to know that their deployment achieved the desired business outcome.
  • The have also matured code branching practices so small features changes can go out with the nightly deployment but new features requiring longer development periods can be integrated when ready.
  • Their automated testing not only runs through regression tests but also trigger performance, load, and security testing. 
  • They have a large enough development pipeline warranting the investment in advanced testing, automation, and SDLC practices.
You can see that there is technical investment and a level of process maturity to enable continuous deployment. In addition, the business strategy and culture has to warrant this process and be able to take advantage of it.

Why continuous deployment may not be right for your business

Here are some reasons why continuous delivery may not be right for your business

  • Your applications are used within business-important (critical?) workflows. 
  • Software deployments often require training users on new or changed capabilities.
  • Your application enables customers to customize the application and your testing isn't sophisticated enough to validate a sufficient number of use cases.
  • Your organization doesn't have the sophistication to take business advantage over this capability. 
  • There's no process to validate the impact or recognize changes that have less desirable outcomes.
  • Your organization doesn't subscribe to experimentation and will view any deployment that had a negative outcome as a failure. 
  • Your business stakeholders are less willing to release capabilities to customers without going through UAT (user acceptance testing).
  • Your testing practices aren't mature. Testing coverage isn't sufficient and you haven't automated testing for performance, security, and other testing needs.
  • You have a history of unstable releases that required patching or rollbacks because of technical reasons. 
  • You have few development resources, and the investment to enable continuous deployment competes with the effort required to develop new innovations or improve applications.
The figure below summarizes when continuous deployment might make sense -

Who's driving your DevOps strategy?

Continuous integration, delivery, and deployment should be part of the overall DevOps strategy and as I said in one of my earlier posts, the CIO has to set the DevOps strategy and priorities. Targeting continuous deployment shouldn't be left to developers that want a seamless, frictionless process with Ops at the potential cost of customer experience and quality. It also shouldn't be driven by a controlling product owner that wants to drive quick changes without planning for their impacts or investing in the technical practices required to perform continuous deployment.

If you're interested, contact me if you want to develop a decision matrix on whether to invest in continuous development.

continue reading "Is Continuous Deployment right for your business?"

Release management strategies for agile development teams

Is release management part of your agile practice? Are you able to tell stakeholders target dates for your next application releases? Are you tracking how often your teams hit these targets and which releases hit the targeted scope and quality?

I’ve asked this question of many teams and hear various answers. Some are practicing continuous delivery, others attempt to formalize release plans at the start of a development cycle, and others only formalize releases when they are near ready to deploy to production. The teams that are in the most dire straits are in a constant churn of patch releases because of last minute requests from product owners or because they need to fix post-deployment defects.

The Importance of Release Management

When I speak to CIO and technology leaders, one of my key messages is that they need to take on practices that mimic software companies. You may not be selling software as part of your business, but the more customer facing application development you do, the more you need practices developed and matured at software companies.

So imagine if one of your software vendors did not communicate their roadmap and schedule of major and minor releases? What if they didn’t publish the schedule of general availability of these releases and the overall impact to users when upgrading to them? What if these releases have a history of disrupting or inconveniencing users because they have frequent defects or other unanticipated impacts? Software vendors that operate with these issues are branded as unreliable, unstable, or difficult to work with and are likely to lose customers over time.

Your customers and business leaders have similar expectations. They expect predictable, reliable releases where they can manage the changes and impacts to business operations or customer experiences.

What Delivery Plan Makes Sense for your Business and Technology?

Your release strategy has to align to how impactful changes are to end users and to the business model. It is strategic decision and companies should target a methodology based on the nature of the business and industry. For example, high profile B2C, SaaS, and enterprise software companies will employ different release strategies depending on how new functionality affects customer experience, functionality, workflow, APIs, and integration.

Your release strategy also depends on technical capability. How frequent you can reliably release code into production depends on the level of automation in your testing and deployment procedures and the overall complexity of your application and data architecture.

Does continuous delivery make sense for your business? If not, how can you get agile application development teams to go from adhoc releases to more reliable release management practices. I will explore answers to these questions in my next posts.

continue reading "Release management strategies for agile development teams"

How to handle detractors to digital transformation

Handling detractors
It takes a lot of effort to align an organization to a digital strategy and digital transformation program. They are disruptive programs that aim to drive growth through new products and customer experiences and not everyone in the front office is going to want a new direction. They also require changes to the underlying business processes by advancing customer service capabilities, automation, and data intelligence. If you're a manager overseeing a large operation that's about to shrink because of automation, you're unlikely going to be a happy participant.

There's a lot of good advice out there to handle detractors in change management programs. 

Start with the champions - "My advice is to forget about the Antagonists at first and to put the initial focus on your Champions. Because if you start where you already have a strong base of support, your Champions will spread that message throughout their vast networks, building the strong platform you need." - Mark Murphy, In Change Management, Start With Champions, Not Antagonists

Respond to conflict to drive transformation - "For collaboration to occur, there needs to be conflict. Great collaboration can get heated. To an outsider, it sometimes resembles hostility or anger, but when we look more closely it is neither. Without collaboration, our ability to create transformative change is limited." - Doug Moran, Don't Mistake Cooperation for Collaboration

Leverage agile practices to drive collaboration, supporters and incremental wins - "If you want big impact from data science and big data, then think of demonstrating wins incrementally. - Isaac Sacolick, Why Agile Data Science Practices Drive Big Data Impact

Communicate and celebrate small wins - "Meaningful organizational change often takes years, yet most people lose interest in an initiative after a few weeks or months. Be prepared for this, and bolster excitement and commitment by continuously rewarding the accomplishment of shorter-term goals." - Alexandra Levit, 10 Ways NOT to Do Change Management

Change management takes preparation and time - "We’ve heard the adage that for successful end user adoption and engagement of a solution, we can’t “communicate on Monday, train on Tuesday, and go live on Wednesday.” For an implementation to be successful, particularly one that requires stakeholder behavioral change, a key component is time, both collectively and individually, for the pending changes. Each person has to make the decision to change, and ultimately go along with (adopt) a new way of working independently, and this is not an overnight process." - David Chapman, The Grass is Greener

Slowly add participants to your transformation program - "The leading analytics-driven organizations not only have more employees involved in data and analytics, but they are also more informal about involving them, and worry less about segmenting teams. This more informal approach may be the key to success." - EY, Change management for analytics success

Stay focused and win change without them - "Be secure in the knowledge that you are doing something good. Sometimes there’s nothing you can do. You can’t win them over, you can’t avoid them, you can’t laugh with them. So you have to just ignore them, and keep telling yourself that when you do achieve your goal, that will be your reward for enduring this detractor." - Leo Babauta Best 8 Ways to Deal with Detractors
continue reading "How to handle detractors to digital transformation"