The First Load Test Always Fails

Here is a typical, very simplified load testing scenario:

The tester defines one or more workflows through the application and simulates them in a tool. The tool allows setting the number of virtual users, the delay between clicks, and other parameters that determine how much load to generate. The tester determines an appropriate load based on the website's metrics or projected metrics. She then talks to the system administrator to determine what system parameters can be monitored during tests. She runs a few tests to make sure the scripts were developed properly and if all looks good, she schedules a time to do the test. She runs the tests, sees no errors, collects the metrics, produces reports and happily tells everyone of the successful test.

Is it really successful?

For those of you that are experienced load testers, you'll find all kinds of flaws in this approach. I in fact am not an experienced load tester, but I have overseen many load testing initiatives. In my experience, the one truth I have always seen about load tests is:

The first load test always fails

Why is this? One simple answer is that most teams start load testing too late in the development process. By that time, all the complexities of the infrastructure, software, and application often produce inconsistent response times and poor testing results. Sometimes the test themselves function properly but are the wrong testing strategy. Sometimes, the tester set the traffic loads too high.

At the end of the day, it doesn't matter the reason. The real question is, are your first load tests designed to find the failure points? Are you measuring the right system parameters? Do you have the appropriate level of logging to help identify the poorly performing parts of the application? Do you have the right team members standing by ready to help diagnose the issue?

Are you and your team expecting the tests to fail?

1 comment:

  1. Load Testing is not as easy as recording and playing it back. Testing needs to be planned, designed and set benchmarks.

    Fuad Mak CO Founder Test Architect, SQA Solution http://www.sqasolution.com

    ReplyDelete

Comments on this blog are moderated and we do not accept comments that have links to other websites.

Share

About Isaac Sacolick

Isaac Sacolick is President of StarCIO, a technology leadership company that guides organizations on building digital transformation core competencies. He is the author of Digital Trailblazer and the Amazon bestseller Driving Digital and speaks about agile planning, devops, data science, product management, and other digital transformation best practices. Sacolick is a recognized top social CIO, a digital transformation influencer, and has over 900 articles published at InfoWorld, CIO.com, his blog Social, Agile, and Transformation, and other sites. You can find him sharing new insights @NYIke on Twitter, his Driving Digital Standup YouTube channel, or during the Coffee with Digital Trailblazers.