Why Scrum Works

At work, we have been using Scrum methods for over two years. It has really been amazing how well this system works for us, at least for development. I only wish we could apply this technique more widely to our organization.

Perhaps some background would help. My employer, when I arrived here eight years ago, was one of the most disciplined software organizations I have ever seen. We produced a software release to the market every year, on time, with good quality. Every. Year. Features being cut before they were ever started, because there would be no time to finish them. It was truly amazing. 

However, this advanced waterfall process was slowly destroying our product. There were two key issues: 1) we'd never plan anything risky, so the releases were getting blander, 2) we had no flexibility should something arise, since everything was planned out. We needed to do something else.

One of the key revelations about software development is that it is, to some degree, unpredictable. Even in very disciplined organization, you reach a point where the most extensive planning process cannot pin down every possible variable. The unexpected lurks in every project -- particularly for those projects which involve something new and unique. Those very projects you need to do to stay competitive.

Classic management would dictate more precision in the planning process, more and more research up front to prevent these unexpected events. But at some point, the up-front costs outweigh the worth of the project. Shouldn't we expend our efforts on building software, rather than building perfect plans?

Scrum does not solve the unpredictability of software development. Scrum embraces that unpredictability. Scrum gives software development teams the tools to manage and control that unpredictability, so that it does not derail a project. Part of the secret is not trying to plan everything. Strictly speaking, only one sprint is planned at a time.

So, why does this work? Why does planning just 2-4 weeks at a time work better than trying to plan it all?

There are three principles that make Scrum effective:

1) Customer Focus - when writing a story, we write it from the perspective of the customer. Completing a story means delivering a concise bit of functionality to the customer. With a completed sprint, the software ought to be in a ready-to-ship state. It's actually very hard to write stories this way. In the end, this process tends to break down large features into tiny slivers of functionality. The beauty of this process is that each bit of functionality can be prioritized against the other. Instead of doing months of work to finish a feature, we're focused on the high-priority aspects every few weeks. This can result in leaving some low-priority aspects of the feature on the table, but on the whole it gets the important aspects built and finished first.

2) Narrow Scope - nothing kills a software project as bad as feature-creep. You start working, and the list of requirements gets longer as you go, until the project will never end. You can't do this with Scrum. Each sprint is fixed in time. Each team has only so many resources. The only variable is the scope. If you discover new requirements, it isn't automatically part of the work. Most likely, you write it up as a separate story to be prioritized against the other stories.

3) Vertical Delivery - the hardest aspect of Scrum is working in a vertical fashion. You can't divide stories by discipline (eg design story, coding story, QA story), you've got to do all the design, coding, testing in the same few weeks. This forces one to think very small for each story. Vertical Delivery also affects your software designs. Many software products have inter-dependent layers (eg database, business logic, user interface). You don't have the luxury of perfecting each layer before moving on to the next. Vertical Delivery means you have to do everything in each layer to deliver the bit of functionality to the customer. The big architecture in the sky is out. Instead, you must choose a more evolutionary approach, where you have just enough architecture to get the job done for the story.

While it may seem cumbersome at first, these principles are actually a very natural process. Imagine an independent software development organization of just one developer and his cat. How would he get things done? Given the very small development team, he's not going to go off for months (or years) doing major features. Instead, he'd focus on delivering something to the customer, keep the scope narrow so he doesn't get sidetracked, and deliver all the components vertically.

Once you've got those basic principles down, the real magic of Scrum happens between sprints. Each sprint, stories are getting done, and little bits of functionality are ready for the customer. The work that gets done has a major effect on the stories remaining. Maybe we've gotten enough of the value out of Feature A, and we can now focus on Feature B. Perhaps there's some new requirement that's come up, and we really need Feature C in order to compete, or comply with a new law, or support a new platform. Prioritization occurs constantly.

Scrum keeps development teams focused on the most important work, delivering working software for customers. It continues to do so even in the face of constantly changing requirements and conditions.

You couldn't plan for that.