Implementing Agile Load Testing

Someone recently asked me, "How do you implement load testing so early in the development process"? I thought about the question for a second before coming up with the following answer:

1) Don't just say you're going to load test, do it - If you're a team lead, manager, or qa engineer it's easy to make excuses why you can't start testing early in the development process. But there are always some low hanging opportunities even if you don't have the perfect person to execute them, the most appropriate tool set, or even reasonable areas of an application to test. Ingraining load testing into the development process and making it a discipline needs to be the first objective.

2) Identify objectives and test criteria - Early test criteria should focus on the testing methodology itself and simple validation. See my previous post on why The First Load Test Always Fails. But as you mature your process, engage stakeholders to identify performance goals (response time under load), and volume (peak traffic levels).

3) Identify your resources - If you're serious about load testing, you have to invest in the right people and tools. It takes a comprehensive set of skills to perform load tests properly. I'll post some criteria in a future post.

4) Make the QA Engineer part of the team - A co-located team is a key attribute (requirement!) of making agile work, so make sure the QA Engineer has a seat with the team. This promotes teamwork, collaboration, and enables the engineer to pick up technical details that enable him/her to craft better tests.

5) Implement load testing as an agile process - The hard part of implementing load testing during a traditional development cycle is that it's difficult for the QA Engineer to truly know what areas of an application are done and ready for testing. That's not the case in agile since the stories are identified, the time line fixed, and the acceptance tests itemized. The QA Engineer knows what's being developed, its completion date, and its scope and can build tests in parallel to the development cycle. The goal should be to implement these tests as part of the acceptance tests (ideal), or in a staging cycle when a release is going though testing and regression.

6) Today's performance issues become tomorrow's stories - The load testing life cycle is like peeling an onion; once you find today's issues and address them (the outer layer of the onion), your next tests will identify the next set of issues and performance optimization opportunities (the next layer...). This life cycle is highly complimentary to the agile life cycle. As issues and opportunities are identified via load testing, the resolutions can be documented as stories and prioritized in a future sprint/iteration.
continue reading "Implementing Agile Load Testing"

Yikes! Defects, Bad Docs -> Customer Frustration

I recently purchased a Sandisk e250 so that I can take advantage of Rhapsody On The Go's music subscription service. I opened the device and plugged it into my USB port thinking that I needed to charge up the device before I could use it. Unfortunately, my system churned and produced an error message "unrecognized device". With very little troubleshooting help, I was forced to call Sandisk for support. Their support was excellent in quickly diagnosing the issue; I needed to upgrade my Windows Media Player to the latest version. Once completed, I needed to go into Devices, drop the Sandisk and add it again. Problem solved.

Lesson 1: If you're software or device has an important software requirement, put in some bright materials in the packaging or a sticker highlighting the requirement.

My woes weren't over. I loaded in Rhapsody 4, authorized my e250, and went to copy playlists. I thought it was all working, but then saw a bunch of alerts with the following error message.

"A problem has occurred in obtaining the device's certificate".

At least there is a link to Rhapsody's knowledge bases with assistance on this issue. Unfortunately their remedy is one of those 'try everything' answers from setting dates (easy), reformatting device (upsetting), to upgrading firmware (annoying). Two hours later I had tried all seven remedies with no success. So I emailed Rhapsody about the issue, listed their url, and told them I had tried the remedies. Their response:

This issue would occur due to many reasons like your device is supported, but does not have updated firmware, incorrect date and time settings in the device, So I recommend you to follow the instructions in the link given below:

http://real.custhelp.com/cgi-bin/real.cfg/php/enduser/std_adp.php?p_faqid=5224
Ok. So the person responded clearly didn't read my email showing that I already tried everything in this remedy. I escalated a second time and didn't get much of an answer except a suggestion to call Sandisk's support.

Great, call Sandisk about a problem with Rhapsody. I had my doubts, but I called them anyway. After trying a number of things, we finally found the issue.

Root Cause

It turns out there are three different ways to reformat the e250. You can do it directly from Rhapsody's software. You can open up My Computer, click on Sansa e250, then right click on "Internal Memory" to do a format. Finally you can format the device directly on the e250.

It turns out that formatting the e250 from the device performs other steps. The format via Rhapsody and Windows Explorer didn't work, but the hard format on the device completely solved the issue.

Is this a defect? Bad documentation? Poor customer service? With the exception of Sandisk's customer service which was excellent - all the above. Lucky for them I was persistent.
continue reading "Yikes! Defects, Bad Docs -> Customer Frustration"

The First Load Test Always Fails

Here is a typical, very simplified load testing scenario:

The tester defines one or more workflows through the application and simulates them in a tool. The tool allows setting the number of virtual users, the delay between clicks, and other parameters that determine how much load to generate. The tester determines an appropriate load based on the website's metrics or projected metrics. She then talks to the system administrator to determine what system parameters can be monitored during tests. She runs a few tests to make sure the scripts were developed properly and if all looks good, she schedules a time to do the test. She runs the tests, sees no errors, collects the metrics, produces reports and happily tells everyone of the successful test.

Is it really successful?

For those of you that are experienced load testers, you'll find all kinds of flaws in this approach. I in fact am not an experienced load tester, but I have overseen many load testing initiatives. In my experience, the one truth I have always seen about load tests is:

The first load test always fails

Why is this? One simple answer is that most teams start load testing too late in the development process. By that time, all the complexities of the infrastructure, software, and application often produce inconsistent response times and poor testing results. Sometimes the test themselves function properly but are the wrong testing strategy. Sometimes, the tester set the traffic loads too high.

At the end of the day, it doesn't matter the reason. The real question is, are your first load tests designed to find the failure points? Are you measuring the right system parameters? Do you have the appropriate level of logging to help identify the poorly performing parts of the application? Do you have the right team members standing by ready to help diagnose the issue?

Are you and your team expecting the tests to fail?
continue reading "The First Load Test Always Fails"
Share

About Isaac Sacolick

Isaac Sacolick is President of StarCIO, a technology leadership company that guides organizations on building digital transformation core competencies. He is the author of Digital Trailblazer and the Amazon bestseller Driving Digital and speaks about agile planning, devops, data science, product management, and other digital transformation best practices. Sacolick is a recognized top social CIO, a digital transformation influencer, and has over 900 articles published at InfoWorld, CIO.com, his blog Social, Agile, and Transformation, and other sites. You can find him sharing new insights @NYIke on Twitter, his Driving Digital Standup YouTube channel, or during the Coffee with Digital Trailblazers.