I started this blog years ago when I first began to suspect that creating software in corporate IT was different from creating software in a software company. I called IT the dark side of technology, based on a belief that the fact that IT products are generally not sold on a free market and therefore is incentivized in completely different ways, generally to the detriment of their work product and the skill and engagement of IT workers themselves.
Well, eventually I found my way out. I work at a real software company, and I have for 13 months now. So, now seems like an appropriate time to answer the question I knew was coming: Is the grass really greener on the other side.
I’ve never felt skilled when it came to crafting the look and feel of an application. There were two components to my feelings of inadequacy: 1) Not understanding CSS and 2) Not being terribly artistic.
The CSS Anthology has really gotten me past #1, and as a result I’m discovering that I am better than I thought at making a web site look good.
Here’s what I like about this book:
It starts with the basics.
I think a lot of my struggles with CSS were the result of never learning the basics. The first chapter of Anthology, “Making a Start with CSS” was really very helpful to me. It filled in the gaps of what I had guessed by hacking at CSS with a much better grasp of what CSS does and how styles get assigned to elements. It was a mini-course in remedial CSS, and I really needed it.
It gets progressively more complex.
I really found the way the recipes got progressively more complex throughout the book helped me to learn effectively. Perhaps this is because I’ve always been such a CSS hack, but the learning increments from chapter to chapter were just the right size. I read the book on an airplane and found I could grok it all the way through.
It’s a cookbook with a strong index.
Want to know how to make a box with round corners? Or remove the default styling from an unordered list? Anthology answers 101 questions like this, and I’ve found the index is very useful for finding solutions for the problems I encountered as I worked on re-designing TroopTrack.com.
If you’re feeling weak with CSS, get this book. It’s really very good.
A few weeks ago I found a javascript bug that was really just a silly mistake by the developer, but it was one that could have easily gone undetected. I found it because when I run the app locally, I always run it at this url:
http://localhost:3000/trisano
This dev runs the app like this:
http://localhost:3000
His javascript made assumptions about the url that were never-false when run his way, but were immediately broken when run mine.
So, while this probably sounds like a simple thing, it’s a good,routine practice to NOT run the application you’re testing just like the developers do. I’m not talking only about the url you run it at, but consider using any other options that are available in a production deployment of your app that are not used by devs and consider including them in your testing routine.
I’m beginning to believe test cycle duration is a critical factor in the risk of a project: the longer the test cycle, the higher the risk.
What exactly is a test cycle?
Here’s my definition:
test cycle(n): the shortest possible calendar time that it takes to start the test environment, prepare for a test, execute the test, collect and analyze the results, report bugs, recuperate, and start the next cycle
To illustrate how this affects risk, let’s consider some real projects I worked on.
Printing Policy Documents
I managed a project that made massive changes to a policy document printing system. It was a mainframe based batch system and the tests required looking at actual paper document printed in a centralized printing facility in another state. Here’s how the test cycle went:
Testers prepared polices
A batch job ran overnight to process the policies to the right state for testing
Testers made changes to policies
A batch job ran overnight to process those changes and generate policy documents
The guys in the printing facility printed the policies AFTER they finished processing all the “real” policies (sometimes took a few days)
The policies were shipped overnight to us
Testers examined thousands of pages of paper, noting defects (took at least a day)
Fried testers slept all day
And we started all over again
Best case test-cycle: 5 days
We only had a month to test these changes giving us a theoretical maximum of 4 test cycles to find and fix every bug. In reality, we only got two cycles in that month and added an extra 5 weeks to the test period to get 3 more cycles in. I know the risk was high on this project because it turned into reality, and it was directly the result of having such a long test cycle.
A day after going live, we found critical defects that forced a massive emergency fix. Groan.
TriSano
TriSano is a web-based application that I can run locally and is implemented using a lean approach, so I can pull code and test it any time I want to. Here’s how my test cycle goes:
Pull the code from git (1 minute)
Build the app (1 minute)
Load any data I need to use (5 minutes max)
Kick off a server (1 minute)
Pick a finished story and think about how to test it (5 minutes)
Do the test (50 minutes max)
Review notes, isolate bugs, and create bug reports (normally less than 50 minutes)
Walk the dog (15 minutes)
Best case test cycle: less than 2 hours
This is pretty low risk, made lower by the fact that the development cycle time is equally short (especially for bugs). In a week, I can easily do 10 test cycles, but I often do 20+.
We ship a new minor version of our product every two weeks, pretty much like clockwork and we rarely have critical defects. When we do, we can usually patch them within a day or two.
In other words, if the software could behave in multiple perfectly legitimate ways (based on all the other stuff in the context of the application), then you should go with the one that is going to surprise the user the least.
I like this as a test oracle and as a tool for advocating bugs. In my environment at least, it’s easier to get iffy bugs fixed if I can provide alternative solutions that are easier to understand (i.e. less surprising to the user).
Anyway, thanks to Josh Tolley for the good idea.
PS – Since someone is bound to point this out, let me just acknowledge right away: as a rubyist I should have already been familiar with this.
Best practices are everywhere and they are very trendy. Everyone should have a list of best practices for their domain on their blog just for the google traffic. Here are my best practices for good testing:
Do not interrupt busy testers more than once a day.
Be willing and able to test anything even if it’s not finished and get meaningful results – if you can’t you need to focus on best practice #3.
Learn a new testing skill every day.
Understand and use multiple oracles to find bugs.
Understand and isolate bugs before reporting them.
Always understand the project context and how it affects the determination that something is a “bug” vs. being an unfinished feature.
Solve your own problems as much as possible.
Know and acknowledge your limitations – don’t pretend you can do security testing if all you know is a simple XSS attack that you learned on my blog.
Advocate for bug fixes without turning the rest of the team against you.
Tell managers who break best practice #1 to leave you the hell alone.
Exploring my Way to a New Career
I started getting into exploratory testing as a way of making my releases go better. I was a project manager in a big IT department and I wanted a cost-effective way of finding the bugs that mattered most as quickly as possible (I call these the “best bugs”). I learned testing so that I could teach my teams how to do it well, and I found that I could get business analysts reasonably decent at testing with about an hour of instruction and periodic coaching.
Later on I became even more disenchanted with corporate IT and project management and testing took on a different role in my career. It became my escape path. I could do it, and I could teach it, and I started to do both on the side. In my head, at least at first, I saw testing as my bridge back to development. Exploratory testing work would lead to automated testing work using Ruby/RSpec/Selenium, giving me credibility as a ruby dev, and eventually allowing me to convince some software company that I would make do as a ruby on rails developer.
Oddly enough, things didn’t really work out that way. Testing did get me out of corporate IT as I now work for a software company, but the majority of my responsibility still revolves around testing. And yet, the work I’ve done on the side with TroopTrack.com, for example, and even some on TriSano, makes me confident that I am at least respectable as a Rails developer.
So… why am I still testing? Why haven’t I moved on to my intended “final” destination?
The Perversity of the Testing Challenge
Testing is, in some ways, a perverse challenge. The job of a tester is to find and expose weaknesses in a product before they impact a user. This is a good thing, and it has an overall healthy effect for the product and it’s maker. At many levels, however, it’s a bit perverse.
Testing’s perversity is especially evident in the relationships between testers and others who work on the product. Take devs as an example – they often hate testers. Why? Because the tester’s spend all their time trying to find the ways the devs screwed up, publicize those mistakes (bug reports), and force them to fix the mistake. Imagine if your spouse did this with some aspect of your life – it wouldn’t make for a happy marriage and it doesn’t do much for relationships at work either.
Another perversity of testing is the happiness I feel when I find a bug. I get a bit excited as I try to isolate it to it’s simplest path of consistent repeatability. But… shouldn’t I be sad that a dev messed up? Shouldn’t I be a tad bit disappointed at the additional cost this mistake adds to our organization? Naturally, it’s good that the problem was found in time, but wouldn’t it be better if it never occurred at all? Of course it would, but… I’m still happy it happened.
Yet another perversity of testing is that I don’t really do anything about it. I’m not a bug hunter – I don’t find them, stalk them, and kill them (generally). I just find and stalk them. Someone else kills them, and then I check their pulse. This is a bit perverse in the sense that it’s kind of hypocritical, especially for a guy who can code. I’m more than happy to point out all the problems around me and then do nothing about them.
The last perversity of testing I’ll mention is the personal nature of my battle with devs (the fact that I call it a battle is perverse in itself – we’re on the same team). Testing software with the same devs for an extended period of time is kind of like playing chess, basketball, etc. against the same people. You learn their weaknesses and you try to exploit them. You know the types of mistakes they make and you adjust your test plan for them. When it works, there is a sense of glee at having bested them that is wholly unjustifiable. I gloat. I sit back in my $250,000 chair, stare at the screen, and gloat when I find a bug this way.
All this stuff is a bit sick and wrong, but I love it. I loooorrrrve it. It makes testing FUN. It turns it into a game full of intrigue, strategy, and execution. It motivates me to constantly improve, not so much for the good of the product and our company, but for the sheer perverse joy I find in finding bugs.
If you find yourself typing asdf over and over again as you test data entry on
a web-app, do this instead:
<script>alert("f")</script>
Paste it in every field.
If an alert pops up OR the field gets saved and then gets rendered without the
script tags, you’ve found a cross-site scripting vulnerability.
The minute you find one, your brain will power up and you’ll find more energy for testing that data input.
Please note: I’m not a hacker, nor am I a security tester. The test method I’ve described above is not enough to verify that a web site is fully secure, but it’s a good way to find vulnerabilities in your web site while you do other testing. Please don’t use it on web sites you don’t have any business testing for vulnerabilities on – you could be breaking the law.
Most of these ruts are the result of test fatigue. They are signs that your energy level is suffering. Sometimes, just recognizing you are in these ruts is enough to fix them. Other times it means you need a break. Take a walk, get a drink, write some code, read an article, etc. to re-energize yourself.
Randomly pounding the keyboard for data entry. “Asdf” for every field on a web page is a good sign you are getting worn.
Not taking good notes. Are you only recording bugs found? Not making notes of tests that seemed to work. You are tired.
Chasing esoteric bugs when you don’t have time. Are you caught up in a completely unrealistic scenario but you just can’t make yourself pull out even though you have more important test charters to work on? Work avoidance behavior!
Not bothering to isolate bugs. You did x, y, and z and the app crashed, so you filed a bug report and moved on. Does it crash with just x? Are there variants of y and z that don’t make it crash? How do they work together. If you don’t know and don’t care, you need to power up. Go stomp on a mushroom.
You can’t think of any error scenarios to test. Happy path test and done? Your brain is becoming disengaged – do something creative to get it back in gear.
Testing takes energy if you are going to do it well. Being conscious of your energy and managing it well means better software because you’ll find more bugs that are meaningful.
