Information Technology Dark Side

I’m beginning to believe test cycle duration is a critical factor in the risk of a project: the longer the test cycle, the higher the risk.

What exactly is a test cycle?
Here’s my definition:

test cycle(n): the shortest possible calendar time that it takes to start the test environment, prepare for a test, execute the test, collect and analyze the results, report bugs, recuperate, and start the next cycle

To illustrate how this affects risk, let’s consider some real projects I worked on.

Printing Policy Documents
I managed a project that made massive changes to a policy document printing system. It was a mainframe based batch system and the tests required looking at actual paper document printed in a centralized printing facility in another state. Here’s how the test cycle went:

1. Testers prepared polices
2. A batch job ran overnight to process the policies to the right state for testing
3. Testers made changes to policies
4. A batch job ran overnight to process those changes and generate policy documents
5. The guys in the printing facility printed the policies AFTER they finished processing all the “real” policies (sometimes took a few days)
6. The policies were shipped overnight to us
7. Testers examined thousands of pages of paper, noting defects (took at least a day)
8. Fried testers slept all day
9. And we started all over again

Best case test-cycle: 5 days

We only had a month to test these changes giving us a theoretical maximum of 4 test cycles to find and fix every bug. In reality, we only got two cycles in that month and added an extra 5 weeks to the test period to get 3 more cycles in. I know the risk was high on this project because it turned into reality, and it was directly the result of having such a long test cycle.

A day after going live, we found critical defects that forced a massive emergency fix. Groan.

TriSano
TriSano is a web-based application that I can run locally and is implemented using a lean approach, so I can pull code and test it any time I want to. Here’s how my test cycle goes:

1. Pull the code from git (1 minute)
2. Build the app (1 minute)
3. Load any data I need to use (5 minutes max)
4. Kick off a server (1 minute)
5. Pick a finished story and think about how to test it (5 minutes)
6. Do the test (50 minutes max)
7. Review notes, isolate bugs, and create bug reports (normally less than 50 minutes)
8. Walk the dog (15 minutes)

Best case test cycle: less than 2 hours

This is pretty low risk, made lower by the fact that the development cycle time is equally short (especially for bugs). In a week, I can easily do 10 test cycles, but I often do 20+.

We ship a new minor version of our product every two weeks, pretty much like clockwork and we rarely have critical defects. When we do, we can usually patch them within a day or two.

Stumble it!