Agile vs. waterfall: Comparing project management cultures

We often see obvious things as unworthy of discussion. There is no reasonable product person today that seriously contemplates the possibility of applying crude waterfall to software product development. Waterfall is as dead as dead can be, and everybody agrees on that.

The problem with ignoring the obvious, though, is that we let our guard down. Things we have granted as gone can silently infiltrate our comfort zones, taking advantage of our bad habits to transform themselves and take the forefront again. Since no one questions the obvious, ghosts can haunt us without us even noticing.

From the tech bubble burst in the late 90s to the second burst we are witnessing in 2022, waterfall has evolved from pariah to successor in the form of late-stage agile.

One can only keep the dead away if they are willing to discuss the obvious. That’s what I’m writing about today. From the fundamental dichotomy between the two approaches to how waterfall has infiltrated back in our lives, let’s dissect the obvious so that we can keep our guard up.

History of waterfall vs. agile

Before we talk about the modern day, let’s briefly discuss the history of both waterfall and agile for some additional context.

Putting the “fall” in waterfall

First things first, let’s take this out of the way: waterfall project management is not inherently bad. It’s an effective and reliable tool to manage projects that can only deliver value when ready, have low uncertainty, and have clear start and end dates.

Construction, industrial engineering, and chemistry are some of the industries that take massive advantage of waterfall approaches.

I’m talking specifically about software waterfall project management. Back in the 80s and 90s that was all we had, but as time went by, we discovered better ways to deliver software.

A key factor for waterfall’s demise was the dot com bubble burst in the early 2000s. After a vertiginous fall of all financial indicators of the tech industry, it hit rock bottom in late 2002. Businesses needed a way to regain investors’ trust by spending less, delivering faster, and guaranteeing ludacris paybacks on early investments.

The rise of agile

In 2001, a group of engineers and tech consultants got together and came up with a nine-line document that would change everything. They kept working on what was left of the industry and their Manifesto not only helped businesses to stay afloat in a tempestuous market, but it also helped them bear results eventually.

The Agile Manifesto took the center stage one year later with the creation of the Scrum Alliance by, among others, two of the original signatories of the manifesto. Time went by and scrum became the first framework contender to waterfall. Encompassing the agile principles to its core, scrum broke away from a hardline approach to software development in favor of an iterative stance on value delivery.

---

---

When venture capitalists came to understand that companies with agile teams had greater chances of delivering the results they needed, scrum quickly became a requisite for tech businesses to attract investments. It was a venture capitalist in fact that took things one step further and coined the concept of the “lean startup.”

Eric Ries is a key figure in the construction of what agile came to be. His book, The Lean Startup, summarizes everything that came before it and transforms what would be just another framework into a management philosophy. Companies coached by him became the first unicorns and most businesses that embraced the lean concept saw their values skyrocket in a blink of an eye.

Product management in the present day

Entering the second decade of the 21st century, agile, scrum, and lean are ubiquitous concepts inside the tech industry. Waterfall is nothing but a distant memory of what our ancestors would use to deliver software.

It’s “hip” and highly rewarding to undergo digital transformations, and even businesses that have nothing to do with software want to embrace the best practices…that’s when the obvious becomes dangerous.

What is fundamentally different?

Before we start exploring how waterfall corrupted late-stage agile culture, let’s get our concepts straight. What does each of those approaches rely on?

A quick disclaimer, I won’t talk about frameworks, but culture. Leave the manuals on the side, this is an assessment of values that support framework exercises.

Waterfall fundamental values

Predictability

Waterfall is allergic to change. From the conception stage, you must know everything about the project and you must be able to see a clear finish line on the horizon. Waterfall projects have a triangle between scope, price, and time that, if needed to change, should happen inside price and time domains. Scope is always written in stone.

Control

Since scope is so tight, control must be exercised to the microscopic level. Managers must be absolutely on top of how things are evolving so that they can quickly react if something seems to be straying away from the original plan.

Hierarchy

From the hierarchy of processes to the hierarchy of personnel, control can only be exercised inside an environment where roles are pretty well defined and accountability is centralized around those few who have a birds-eye view of the project, usually upper management.

Structure

Finally, for hierarchy to exist, you need a structure and someone responsible for keeping it up. Specialists are beams that deliver limited by their responsibilities, while pure management figures are bolts and nuts accountable for keeping the whole thing together. The “specialists babysitting” mindset.

Agile fundamental values

Adaptability

Agile presumes that it’s impossible to control all possible outcomes beforehand and that surprises are not a matter of if, but when. The more adaptable a project can be while still delivering similar value to the user, the better.

The “responding to change over following a plan” line of the Manifesto represents this.

Horizontality

A project can only be adaptable if it lets go of the “command and control” mindset. Individuals inside the organization should have the freedom to react as they see fit at a moment’s notice, and they should not be afraid of doing so due to potential punishment or constraints.

This particular value is translated by the “individuals and interactions over processes and tools” line.

Collaboration

Where does the limit between horizontality and chaos lie? An agile system cannot exist outside of a collaborative environment. This takes into consideration the specialists, of course, but also the customers. An ever-changing environment demands that, despite any change of plans, the customer feels confident in you to always have their best interests in mind.

The “customer collaboration over contract negotiation” line of the Manifesto translates this value.

Delivery

Not to say that agile doesn’t have a structure, it does. Scrum is the most famous of them all and serves the project, not the other way around.

The objective of any agile organization is to deliver value to its customer, not to do things the right way. If things are done in an unorthodox fashion but they work, that’s perfect. The end goal matters more than the journey.

The “working software over comprehensive documentation” line of the Manifesto translates this value.

Discovery

This fifth value has no counterpart on waterfall and is not present in the Agile Manifesto. Of all of the lean contributions to agile culture, this is the most important one. Based on the fact that we are in an ever-changing reality and that no one knows better than anyone else, being able to fail and learn with that is the fastest way to identify what is worth delivering.

The first four values might be applied to projects and products alike, but it’s this fifth one that differentiates one from the other. A product is a living thing that is evolving without a clear endgame in front of it. Discovery is as important for a product as breathing is for us.

How agile is becoming waterfall

To quote Aristóteles closing argument on Metaphysics, book X part 10:

Things which differ in kind are farther apart than those which differ in form

I know Aristóteles was not what you were expecting to find here, and I won’t elongate myself on this. My point here is that despite their differences, agile and waterfall are somewhat interchangeable and, to some extent, the same. Waterfall and agile culture are different forms of managing software projects, but they are made of the same constituent concept: people managing people.

The values we covered, on the other hand, are not interchangeable. They are different in kind, they are indeed the quintessential difference between agile and waterfall. Following the scrum guide by the book, having squads, agile coaches, dailies, and meetups might make you show up as agile, but unless your values are aligned with the Manifesto, you’re just dressing waterfall as agility.

This is precisely the scenario we have been witnessing in the last few years. As more and more companies see the results of strong agile culture creating unicorns and industry juggernauts, more of them want a quick way to execute digital transformation. What happens is that they start practicing agile, but keep the waterfall values of control with a lack of flexibility and hierarchy.

Even worse, since the number of successfully transformed companies is way smaller than those who just pretend to have transitioned, more and more people have no experience with agile values, leading them to believe that doing agile with waterfall values is perfectly normal.

Product people become keepers of the scrum structure, roadmaps become unalterable scope traps, engineers are limited by being boxed under a business hierarchy, and collaboration gives space for control as managers get more and more insecure with how little agency they have over what is being delivered.

---

---

Little by little, agile culture dies by the values, while waterfall resumes industry control using the latest frameworks as hosts: SAFe, late scrum, Agile PMI…

What is the problem?

Despite what the above arguments might entail, there is no battle for the soul of the industry happening. This romantic view of right vs. wrong or good vs. evil has no space in this debate. The discussion is way more pragmatic: which set of values gets you the most money with the least headache?

The answer might be different depending on whom you ask it.

Enormous companies might find it hard to maintain a horizontal, adaptable culture since it has so many moving parts. Time and money spent on trying to replicate a startup rulebook inside your decades-old business with thousands of employees is a drain that will probably bear no results.

On the other hand, founding a disruptive digital business where all decisions must go through you first and the only valid vision is yours, probably means you are in for a disaster in the long run.

The problem is not doing one or the other, having this or that set of values. The problem is that the blurred lines do a great deal to homogenize the market, putting nimble startups and gigantic corporations inside the same bucket.

Enterprises invest a fortune in trying to become what they possibly can’t be, while fresh businesses need to grow fast before they can start to exercise proper control inside their structure. Everybody spends more money to achieve the same results.

When capital is available for cheap in the market, no big deal, but when it becomes scarce, things start to break.

It might not be a coincidence that waterfall values were dominant before both of the tech bubble bursts…

Conclusion

Waterfall vs. agile is anything but an obvious debate. There are so many nuances to it that summarizing everything in short of 2000 words is almost a sin.

Nevertheless, I hope this has been a conversation starter for you and your peers. That you can assess on your own which values your team or your company stands by and how they are affecting your job.

Waterfall culture is not wrong, but it’s context-dependent, just like agile. Large and complex mechanisms benefit from a waterfall approach, and if they want to become more agile, they should keep it a side project. Smaller and younger businesses should take things at their own pace, be horizontal, and focus on delivering value. Leave structure and control for the future, when you become large and complex yourself.

If you want to know more about how legacy companies can take leverage of agile values, I highly recommend the book Exponential Organizations by Salim Ismael.