Information Technology Dark Side

Struggles of a Self-Taught Coder

Information Technology Dark Side header image 2

Is there REALLY any rigor in Waterfall?

October 15th, 2007 · 8 Comments

In 1970, Dr. Winston Royce wrote the paper “that started it all” for waterfall project management in IT. His paper, “Managing The Development of Large Software Systems” outlines an incredibly rigorous process that was adopted by the United States Department of Defense. From there, this approach has spread throughout the IT world until it is so entrenched in the IT world that it is almost unavoidable.

Since Dr. Royce’s paper is frequently cited by those who promote agile project management, I felt it was important that I read it. There were several things that stood out for me, points that everyone should understand about waterfall project management:

Dr. Royce admits the waterfall approach is “risky and invites failure.” He then goes on to detail a rigorous approach to waterfall that reduces the risk of failure, but at a substantial cost.

The world of software development has completely and fundamentally changed since 1970. We don’t program on cards anymore. We have user interfaces, servers, the web, desktops, applications, etc. Not to mention, computers do so much more than calculate re-entry paths for primitive spacecraft, which is the the type of capability Dr. Royce’s projects implemented.

The waterfall approach is incredibly expensive. He claims that a true waterfall project, one with adequate rigor to be successful, must have a minimum of 1500 pages of specification documentation for a $5,000,000 software project (1970′s $’s – nearly $27,000,000 today). Those documents would be rigorously reviewed by many people. I haven’t calculated how much that might cost, but you can guess it would be a lot.

The waterfall approach doesn’t result in maintainable software. Remember the 1500 page specification for your IT project? That must be maintained throughout the life of the application if you are to have any hope of keeping the software operational.

Dr. Royce’s assertion of the value of waterfall is not based on scientific rigor of any kind, no matter how rigorous the prescribed approach is. Dr. Royce, clearly an experienced scientist, makes a fundamental error in the rigor of his approach. He states that he believes in the waterfall approach, but never cites his reasoning for this belief. He simply believes it to be true. To this day, there is still no conclusive evidence supporting the claim that IT projects should be managed this way, although most of the IT world behaves as if there is. On the contrary, there is a substantial body of research that, even with the rigor Dr. Royce prescribes, waterfall projects still are risky and invite failure.

For the last couple of years I’ve sort of ridden the fence on where I stand on project management. I’ve conceded that waterfall might have a place in the IT world in the past, that it’s a tool we should have in our toolbox, even if we don’t use it very often. While I still feel that way about agile, and other approaches like spiral, iterative, etc., I’m getting off the fence with waterfall.

The waterfall approach to managing software development projects is one of the most costly mis-diagnoses ever. It structures projects for failure. It creates a cost-change curve that is unacceptably expensive (you know, the cost of a change grows exponentially as the project progresses) and is largely responsible for the high failure rate of IT projects. Waterfall creates a project structure that is rigid and forces you to make decisions and predictions when you know the least, then abide by them no matter what realities you expose in the process.

I am off the fence. Waterfall doesn’t belong in any IT project manager’s toolbox. It is based on and littered with myth. There is not a single shred of scientific evidence, at least as far as software development is concerned, to support the idea that waterfall actually works. It is a cancer between IT professionals and the business they support. It cripples a corporation’s ability to evolve their business, to use technology to meet their goals. It is too expensive and provides too little by way of results.

Thirty-seven years ago Dr. Winston Royce answered the question of how to make waterfall “work,” and it isn’t pretty. Unfortunately, he never bothered to ask whether we should, to question the assumption that software must be developed sequentially. I’m going to cut him a little slack. It’s pretty hard to change software that was written with punch cards. But I’m not extending the same courtesy to the IT professionals who have swallowed that notion since then. If you believe in waterfall as the “best” way to develop software, the one “right” way, you are like a lawyer carrying a claw hammer in his briefcase. A hammer is useful for a carpenter, but lawyers who rely on them don’t do well. Get some real tools.

Before you get ballistic with me, I want to point out one thing. I’ve never seen a project manager who actually manages projects via pure waterfall. I’ve seen many try, but in the end they ALWAYS scrap the “holy process” and become pragmatic, doing only what it takes to deliver their project. Yet, so many of us still talk about waterfall as if it represents some utopic state of software development, as if the achievement of pure waterfall would be something like heaven, streets paved with gold and praises sung to requirements documents. That’s the belief I want to challenge, this theoretical ideal world where everything happens sequentially “because that’s the right way to do things.”

So, take a minute and ask yourself the question Dr. Royce failed to ask. Does software have to be be build sequentially? Is it REALLY like a house, in that you can’t put up walls without first having a detailed blueprint? You can’t frame without a foundation? You can’t shingle a roof if you haven’t installed joists?

I have built many software applications non-sequentially. I’ve build user interfaces without data models. I’ve built data models without requirements. I’ve made significant design changes late in the project. And I’ve done all of this cheaper, and at less risk, than would have been required using waterfall.

One more thought, and then I’ll give up on this rant. It’s too long already. Here’s my final thought, and the biggest fundamental issue I have with waterfall. I believe the obligation of every IT professional is to provide value to the business they serve, well beyond the amount of the salary they receive. Our obligation is to make our companies tougher, more competitive, smarter, and more profitable. That’s why we’re there. To make our employers better. That means we, as project managers, have an obligation to our employer to deliver valuable software at the LOWEST COST POSSIBLE. Whatever quasi-religious beliefs we have about the “right way” to deliver software should be less important than this goal.

If you enjoyed this post, make sure you subscribe to my RSS feed!
Stumble it!

Tags: Project Management

8 responses so far ↓

  • 1 Chris R. Chapman // Oct 16, 2007 at 12:52 pm

    Great post – it’s good to see developers actually go back and do some fact-checking to formulate an informed opinion on things like BDUF/waterfall.

    I have a couple of points to add to what you’ve mentioned on Royce above with respect to context:

    * Royce developed the phased delivery model to cope with regulatory requirements set out in the US DoD STD-2167 document, which was so byzantine and bureaucratic that the waterfall was the only way to cope with it;

    * A common misconception is that Royce “invented” the single-pass, phased model – actually, in his paper he recommended that the phased model _be repeated cyclically_

    * Even the DoD realized 2167 was a dud and abandoned it in 1994 with MIL-STD-498 which promotes evolutionary requirements design.

    * In 2000, the DoD released a further spec, 5000.1 and 5000.2 which emphasized using iterative/incremental software delivery.

    It’s a shame that no one bothered to RTFM 30 years ago before entrenching the waterfall into computer science to the extent that it’s difficult to undo.

    Not to say we’re going to stop trying, though…

  • 2 David Christiansen // Oct 16, 2007 at 1:00 pm

    Thanks Chris for a great comment. I learned something new from my own blog!

  • 3 Ben Simo // Oct 30, 2007 at 1:34 pm

    Great post!

    It is sad that software development philosophies and practices developed in a world of government regulation, punch cards, and very expensive computer time still have such a strong a hold on today’s commercial software development.

    Ben Simo
    http://QuestioningSoftware.com

  • 4 David Christiansen // Oct 30, 2007 at 5:33 pm

    Thanks Ben. I agree – in an industry marked by so much innovation, we seem to have barely moved at all in this area. That’s ironic.

  • 5 Followup: Rigor of Waterfall // Nov 5, 2007 at 8:49 pm

    [...] now and then I write an article that gets a lot of attention for whatever reason. My post, “Is there REALLY any rigor in waterfall,” was one of those. Sure, it wasn’t as popular as “The Rules of Office [...]

  • 6 CleverWorkarounds » Why do SharePoint Projects Fail? - Part 8 // Jun 10, 2008 at 9:03 am

    [...] post entitled "Observations on the Rigor of Waterfall", and referred to a post from David Christiansen. Both authors do a great job in systematically pulling apart some of the misconceptions of the [...]

  • 7 briefcases // Oct 30, 2009 at 9:03 am

    I am impressed, you did a great job with this post, thank you and keep coming with fantastic posts like this.

    Dave

  • 8 RenĂ© Johnsen // Feb 2, 2012 at 4:25 am

    Great post!

Leave a Comment