Tuesday, August 25, 2015

5 Things To Do When Your Career Hits Rock Bottom

Last year at this time I hit rock bottom in my career. For those of you that know me, you know why and for others it's not important to hash out the past. I learned from the experience. So for those of you that are in a bad situation at work, in your role, or in your career I'm able to offer some advice.

Keep in mind, it's taken me a year to be able to write about it. If your in a bad situation, you're not going to have an epiphany after reading this post or any other advice oriented materials that are available. You have to climb your way out. Here are some steps:

  • Reflect on your career and recognize what you don't want, but more importantly, what you want from your employment and career. Eventually, you will find that you have options, but you want to be able to judge whether any new situation is truly better for you, or just different.

  • Learn something new. Healthy distractions are a good thing. I biked. I biked a lot. I learned about the cloud and big data. I built up a new list of startup ideas. 

  • Confide in your close friends and colleagues. I owe a big thank you to a handful of you, some of whom I could not disclose any detail on what was going on.

  • Find an alternative form of expression - I blogged more. I tweeted. I cooked. I started writing a book. My one regret was not finishing it.

  • Determine your "outs". This is a poker term, and refers to making sure you have options and only go all in (or bet big) with your chips when you can draw a big card that can improve your hand. If you're out of work or might be soon, it's critical to do the basic things to make sure you can sit at the table with the pros like network, get your LinkedIn profile updated, etc. but that's not enough. Think about what you'd do if you were solo for awhile. What services could you offer? Who could be an early customer? How might you get the word out that you were available? 

I'll be the first to admit that I wasn't very good at following this or any advice. If you've hit rock bottom,  hang in there and build your ladder.
continue reading "5 Things To Do When Your Career Hits Rock Bottom"

Monday, August 17, 2015

How to Get an Agile Product Owner to Pay for Technical Debt

Paying for Technical Debt
It's the most common question I get when discussing agile development as a CIO with a team of developers that are enthusiastic about maturing their practice. How do you get the Product Owner to prioritize fixing technical debt above new features, enhancements, special requests for customers, and improvements to internal tools? How do you get business stakeholders interested in seeing performance improvements, security testing, error logging, automated data processing, code commenting, class refactoring, automated testing, platform upgrading, and in addressing other technical debt?

Can You Handle The Truth?

The simple answer is, without proper culture, incentives or rules most Product Owners will underinvest in fixing technical debt. There are exceptions, but in performing agile transformations now in four organizations I can tell you that the pressure on Product Owners is simply too great and they will almost always drop priorities on technical debt to fulfill stakeholder needs.

The simple way to get technical debt prioritized is for the CIO to step in and establish governance requiring a significant percent of effort applied to this need. 

I usually set this at 30%, but it can be as high has 40% for complex architectures, legacy applications with significant risk, or mission critical applications. It can be a lot lower for simple applications.

My rationale is that software companies typically charge 20-30% of license costs to provide support and maintenance. That's for software organizations selling their software and the typical organization or enterprise isn't going to be as efficient, hence the 30-40% benchmark.

Prioritizing Technical Debt

Here's the process I endorse:

  1. Technical leads and their teams are responsible for itemizing technical debt stories on the backlog at the end of every sprint.
  2. Delivery heads or architects are responsible for prioritizing the technical debt and should articulate their rationale for the prioritization.
  3. Here's where there's room for a lot of collaboration:
    • The Product Owner can get some of her needs accomplished within the scope of technical debt if she can articulate business needs as part of technical debt stories.
    • The Technical Lead can also get more technical debt addressed if it is bundled in properly selected functionality driven (non-technical deb) stories.
    • Both the Product Owner and Technical Lead have to partner so that these requirements do not drastically change the estimated Size of the story.
  1. The Product Owner and Technical Lead abide by the agreed on governance principles to insure the Sprint-level Technical Debt percentage is addressed. The Sprint-level technical debt target should be 5-10% lower than the total technical debt target.
  2. The remaining technical debt should be dedicated to 1-2 Technical Debt Releases per year to address larger system or platform upgrades. 
  3. Smarter teams will prioritize more technical debt at the start of a multi-sprint release and do less of this work near the end of the release cycle when the Product Owner feels more pressure to add more functionality to the  release.
It's not perfect. It's not ideal. It does get the job done.

continue reading "How to Get an Agile Product Owner to Pay for Technical Debt"

Wednesday, August 05, 2015

Legacy Thinking versus Agile Thinking

Today, I am intrigued and bothered by "legacy thinking". Legacy thinking, is when leaders in an organization, stakeholders on an initiative, or managers of resources or funding believe that they can invest once and largely get it right the first time. We know what this looks like as an end product; legacy software applications, roads that are in disrepair, hospitals with antiquated practices, retail stores with poor customer experiences.  This is a legacy service; a product, service, or operation that lacks feedback processes and resources to make continuous improvements. These services become complex over time because the underlying knowledge of why and how something was constructed erodes.

In between legacy thinking and service and thinking is legacy funding. It's often the byproduct of the organization's culture (legacy thinking) and accounting practices that lead organizations to fund and allocate resources once around a product, project, or process improvement. As a CIO, I see it every time a project comes in with a fixed investment and resource allocation with no or little provisions to support ongoing improvements. A facilities manager feels it every time they have to make difficult judgement calls on what to repair or replace.

Agile Development vs Agile Thinking and Agile Funding

The success of agile processes to develop new products, complete projects, or run operations in everything from software development, construction, and marketing has made many organizations successful getting things done. But I'm not sure this success has fully transitioned organizations to Agile Thinking and Agile Funding. Agile thinking is driven by asking question, experimenting, and prioritizing. It requires ongoing, but not necessarily flat investment depending on the depreciation and competitive cycles of the services. I call it cyclical investment so that every few years, a product or service will need a bump up investment for a step-up improvement. Also, rather than just ROI, agile funding insures that prioritization is driven by broader KPIs and metrics.

The byproduct of agile thinking and funding leads to an agile service, largely characterized by improvements driven by customers, economics (efficiencies), and competitive factors (innovation).

Unfortunately, many organizations are transforming to agile in the opposite direction. They develop the product or service using agile practices, but the transformation in funding and culture comes later if at all.

Almost fifteen years since the agile manifesto was created. We still have work to do.

continue reading "Legacy Thinking versus Agile Thinking"

Monday, July 20, 2015

Why Business Leaders are Clueless about Data Integration

When someone says that the data integration process is automated, I suggest asking questions to clarify what they mean by automated. You'll probably conclude that the process is anything but automated let alone reliable, scalable, secure, or configurable.

To some, automation implies efficient and reliable but still including manual steps so long as they are performed quickly and easily. Others assume that if the process can be completed without IT's direct involvement then it is automated. Still others don't care whether it's automated but are angered when there is a breakage in the process or if it can't scale magically as more data is piped in. Lastly, there is an assumption that the daily process running on a volume of data will magically scale when existing data has to be reprocessed for a change in business logic or storage schema.

As hard as it is to modify software, modifying a semi-automated (in other words, partially manual) data integration can be even more daunting even if the steps are documented. For example, fixing data quality issues or addressing boundary conditions tend to be undocumented steps performed by subject matter experts.

Business Leaders are Clueless About Data Integration

So now you want to fix data integration. Take out the manual steps. Make the process more nimble, agile, reliable, etc. Why is it so hard to get business leaders on board with the investment needed in data integration?

Data processing and data integration technologies like ETL, Hadoop, Spark, Pig, Hive, IFTTT are difficult enough for technologists to fully understand and solution to the right data issues, but the jargon just frustrates business leaders. Many are clueless about the technologies (other than the over-hyped term Big Data) and are more surprised about the need to have them and invest time to build expertise in them. "Information technology" and processing data has been around a long time, so there is underlying assumption, even with Big Data, that integration is cheap and easy.

Now unless you are doing very basic, point A to B straightforward plumbing, data integration can become quite complex as new data sources are added, logic changed, and new applications developed. Data integration may start off as simple, but over time legacy data flows are difficult to understand, extend, or modify.

The complexity slows down IT, and if data and analytics is strategic to the business, it frustrates business leaders that they can't just add a new data source, modify calculations, improve on data validation, or strategically change the downstream analytics.

So now, whatever technologies IT selects and however it is presented to the business, it comes across as plumbing. All that time investment just to have a stable data flow? Analytics, visualization, application and for the most part doing anything useful with the data will cost extra?

Data Integration = Core Big Data Infrastructure

The simple answer is that data integration is a key foundational technology capability if data is strategic to the business. It's not just a technology, it is a competency. Unfortunately, ranked against other data technologies like data warehousing (in RDBMS, NoSQL, etc) and delivery (including analytics, data visualization, mobile application development, etc.), data integration capabilities are often a distant third in priority. So it's no surprise that many processes are not fully automated and that business leaders don't "get" the importance of this capability.

Knowing you have a problem is the first step to solving it... More in my next post!

continue reading "Why Business Leaders are Clueless about Data Integration"

Tuesday, July 14, 2015

Fixing your IT Legacy Before the Next Bridge Collapse

Last week we saw two significant, public system outages, one belonging to NYSE because of a "configuration issue" and the other to United Airlines because of a "router issue". This week, the news is on OPM's  data breach of 21 million users and their lack of investment and follow through on resolving security issues identified by auditors.

My first thought on seeing these failures is that CIOs have little hope in achieving 99.9%+ uptime, fifteen minute or less recovery times, and secure perimeters when the world's greatest organizations suffer from what looks like basic IT 101 issues. Though I am certain the issues are far more complex in reality and fixing them a significant challenge to their IT staff, I think about how many times CIO walked into a Board room with their head tucked in because of similar issues? A critical operational outage because of some preventable infrastructure, operational, or software issue?

But my second thought turns to our roads, bridges, highways and infrastructure. At the bottom line, the US would have to invest $3.6 trillion to bring it all up to snuff by 2020. That's some technical debt and we're lucky that there isn't infrastructure related catastrophes reported on a frequent basis. But what you may not see in the numbers but people feel on a daily basis is the negative impact poor infrastructure has on growth. Stuck in traffic because a highway needs expansion? Subway delays because of a switching issue? How about that high speed rail that would be faster, cheaper, safer and more environmentally friendly?

Over the past five years, businesses have been smarter about investing in innovation and digital capabilities. We've been enabling analytics and data driven decisions, adjusting to a user driven mobile workforce, and insuring we can personalize and improve customer experiences.

But have we attacked the IT legacy and technical debt with the same enthusiasm let alone internal celebration? Are you still running Windows 2003 even though Microsoft will end extended support for it this week? Will you upgrade that .Net application on V3.5 or that Java one on V7? Will you connect and secure your data sources, or continue to run them as inefficient data silos?

Lots of work to do, but before you spend hours reviewing, debating, and blaming what to do when you hit your next outage, I suggest spending a greater amount of energy, funding, focus and celebration on where you will attack your technical legacy. 
continue reading "Fixing your IT Legacy Before the Next Bridge Collapse"

Monday, July 06, 2015

DevQOps - Giving QA a Seat at the DevOps and Digital Transformation Table

I was surprised over the lackluster response to my last post, Why a QA Practice Is Critical to Long Term Success. After all, if more organizations are looking to achieve continuous delivery, why are we not talking about the critical role QA plays in a DevOps transformation?

If more organizations are investing in self-service BI programs, how are these new dashboards being tested before they are used in critical decision making? How are organizations validating new SaaS applications before they are deployed to users? Deploying new marketing tools? What steps are being taken to insure KPI changes are as expected and no new issues are introduced?

Might I suggest that we give QA a seat at the table.

Perhaps we should call it DevQOps

Let's review some of the key QA functions

  • QA should automate regression tests so that agile development and iterative releases can be performed efficiently and reliably

  • QA should be validating performance of the application especially when response time is critical to success and where usage growth is expected

  • QA should help minimize the work required by User "Acceptance Testing" since users are often ill equipped to test applications through multiple flows, data inputs, and boundary conditions

  • QA should lead efforts to validate security and perform other code validations

  • QA should insure the user experience is optimized for different devices and browsers

  • QA should manage business risk by itemizing, prioritizing, and developing action plans to mitigate them


Why Give QA a Seat at the Table?

Nothing is new in the list of responsibilities I listed and QA has been performing some or all of these roles in many application development practices for some time. If you're already investing in DevOps, you can't legitimately claim or achieve continuous delivery without factoring in some of these practices. My issue is that these needs are not given equal billing in a DevOps transformation and QA seems to be the middle child being squeezed out by their more aggressive Dev and Ops siblings.

What is new is that many organizations are building customer facing applications and proprietary workflows as part of their digital transformation programs. QA is often an afterthought in the budgeting process and maturing QA a distant third versus Dev and Ops practices
The mindset among many business managers is that QA is still "the thing you do after the application is developed", that developers should be writing bug-free code, and that whatever validation is required can be performed by the expected business users. They are more likely to invest in additional developer resources if they believe it will gain them functionality or speed to market, or in system engineers if it will gain them reliability or help reduce costs. The CIO and IT leaders in these programs are then more likely to invest their efforts maturing development and operational practices without learning quality assurance tools, practices and governance.

No QA = Legacy Application?

Whether you're moving to a DevOps or investing in digital transformation, I suggest leaving out or underinvesting in QA is a mistake. Ever here anyone going back and easily adding a QA practice to a legacy application? It's hard and often more expensive to do after the fact, so less likely to be done.

In fact, I would argue that a successful application without QA is the definition of a legacy application.

continue reading "DevQOps - Giving QA a Seat at the DevOps and Digital Transformation Table"

Monday, June 22, 2015

The Most Important and Often Underinvested IT Function

If you're practicing agile development without QA team members, a reasonably defined testing process, and sufficient criteria to help define "done", then at some point your development process will go off the cliff of complexity.

That's right. No QA, off the cliff you go without a parachute.

The size of your development team, the number of technologies used in the development stack, and the business criticality of the applications are all quality criticality indicators. They shape how much runway IT has before quality factors drive material business risk that can easily overwhelm the IT development and operations teams.

Even if you're a solo citizen developer on a low code platform, at some point you're going to make application changes where having defined test plans help avoid issues that might impact users, customers, data quality, security or performance. In larger scale applications, performance testing is not something you can do on demand because the first load always fails and valid testing practices need to be implemented as part of the agile development process.

Why a QA Practice Is Critical to Long Term Success

Let's look at some basic concepts that point to why QA is so critical to businesses that rely on technology
  • More functionality, more testing - As applications are developed with more features and capabilities more functions need testing. Every agile sprint increases the testing landscape, and if the discipline to define test cases and establish regression tests isn't developed in parallel, the backlog to develop them becomes too hard and long to execute on afterward. If a development team is adding functions and then breaking others, it's likely because there isn't regression testing in place to catch these issues.

  • Applications are more complex involving multi-tier architectures, multiple databases, transactions spanning multiple APIs, and computing performed on multi-zone cloud environments. If you aren't building unit performance tests along the way then identifying a bottleneck can be a lengthy, painful process when it emerges.

  • Data and Analytics Testing - The application is working, but the data is wrong. Many applications today are data driven enabling their users - internal or external to make better data driven decisions. Application testing isn't sufficient because if the data is wrong, the entire value to the end user is compromised. But testing data, validating calculations, understanding boundary conditions, and insuring aggregations or statistic calculations are valid is really really hard without a defined test strategy.

  • Security Testing - A failed application security test may be costly to fix. Worse is a security failure in production that could have been avoided with security practices entrenched into the development practice. If you're not testing for security issues, then it's unlikely the development team is applying basic security design principles into developed applications.

  • Multiple devices, browsers - Finally, today's user experiences span phones, tablets, laptops, IoT devices with different browsers and plugins. Insuring that user interfaces are functioning and that the user experience is optimized across all these modalities has never been easy, however, customers have minimal patience for mediocre or clumsy experiences.


 Why Many Organizations Underinvest in QA?

The workflow may be broken. The data may be wrong. The user interface may look broken on an Android phone. The performance may be degrading over time. There may be a security hole just waiting for someone to exploit it. There is likely loss of revenue if there is an outage.

With so many things that can go wrong, why do many enterprises and the CIOs that lead them underinvest in QA?

Next post.

continue reading "The Most Important and Often Underinvested IT Function"