How many of you software geeks have been through a corporate Agile transformation? Did it at first seem as though everything was going to get dramatically better? As though you’d be unleashed to really do your work, and that no longer would useless processes hold you back?
How did it turn out? Did the organization truly transform? Or did “Agile” become one more layer on top of all the other processes? Did you still have to go through bureaucracy to get the tools and training you needed, causing you to blow your sprints and deadlines, and yet it was still your fault? Were you unable to get testing effort when it was needed? Did you still have the same sets of managers and directors telling you who was on what team, instead of self-organizing to choose teams yourselves? How did you cope with missing a deadline by a single day? Did it mean blowing up a sprint, or shortcuts? What about changing requirements? Did you upset management by saying no, or by failing to re-estimate, re-prioritize, and deliver?
If any of this sounds familiar, I’d encourage you to watch this video (don’t worry, I’ll still be here): Allen Holub – The Death of Agile
What Holub gets at in the video is that the order of importance in the organization is culture first, then principles, then practices. Yet teams start by being told “here’s your Agile process” (usually Scrum), and then they’re turned loose, without a solid understanding of “why”, let alone “how much is necessary and how much can you trust me to just do my job?”
You could readily map “processes”, “principles”, “culture” to Shu-Ha-Ri as Alistair Cockburn, one of the signatories of the Manifesto for Agile Software Development, mentions in Is the failure in agile adoption due to cargo cult, shu actions, or just laziness . In practice, what happens is someone, often Agile coaches, comes in, teaches you the Shu (a process), and you use it blindly. But unless your organizational culture allows you to really improve your processes, you can get stuck there. If they encourage it, you can move up to the Halevel, understanding the principles, and finally internalizing and transcending them at the Rilevel as you master the culture itself.
But for this to all work, the corporate culture must be behind it. If the organization thinks that agility is just something that happens over in Engineering, you’ve already failed.
True story: I had already had a background in Scrum about a decade ago–which failed as we did it just in one group and we did it in the Cargo Cult manner ourselves. Later, the entire product creation organization was trained in Agile, Lean, Scrum, and SAFe by a group of certified Agile coaches/consultants. During a break I caught up with an executive and asked him how much he understood about Agile. He confessed that he didn’t know much, and told me that he didn’t really need to know it.
In other words, it was beneath him.
I had a nagging feeling that he was wrong, but couldn’t put my finger on it more than to say that Agile eschewed the hard deadlines and focused on trusting the team to do the highest priority things. He basically told me that I was wrong, and that deadlines weren’t in any way going away. (By the way, I don’t disagree with deadlines, but I’ll have to talk about that in a later post.)
He had just demonstrated the failures that Holub talks about–failures to change culture from the top down. Since going through that, I saw perhaps every failing that Holub and Cockburn discuss. And as Holub says, this isn’t just one organization. It’s happening across the industry. I have a friend at a major company (one nearly everyone has heard of) and when I began to talk with her about IDesign’s Project Design, she shrugged and replied, “we have sprints.”
Well–how do you argue with that? They have sprints! What more can you ask for? As it happens, they also have wonderful food on their corporate campus. See what Holub has to say about that.
Now I am not saying I agree wholesale with Holub. People who have trained with IDesign will take issue with a few things in his presentation, as I will go over next.
Holub says changing requirements are why you have short iterations–you do something and get it in the hands of the customer. In so doing he underscores a key failing of the Agile community–the assumption that up-front design is worthless. Martin Fowler–another signatory–discusses this at length, from the perspective of XP, back in 2004 and before the rest of the Agile movement really took off, in Is Design Dead? And in a follow-up, FlaccidScrum, he talks in terms that go back to Shu-Ha-Ri. As does Mike Sutton, in Why Scrum is designed for misuse., arguing against the rituals and for independent thought. Likewise Rachel Davies says similar in Shu-Ha-Ri Considered Harmful? I find it a shame that organizations, through layered bureaucracies, rigidity, and incompleteness in both training and trust, squash the “individuals and interactions” and “responding to change” parts of the Manifesto.
One enormous thing much of the Agile community gets wrong is to say that the failure to craft perfect requirements means that any up-front planning can be conflated with “Big Up Front Design” or “Waterfall”. This is not true. Instead the real game is about decomposing the system properly, using Volatility-Based Decomposition as I talk about here. Fowler even dismisses VBD as “very difficult”, but those of use who design this way regard such dismissiveness as more proof that people think there is a magic way around the First Law of Thermodynamics, which we like to paraphrase by saying, “there is no free lunch”, or even better, “you cannot add value without sweating”.
Yes, good design is hard. That doesn’t mean you can’t develop the mental muscles to do it well, it doesn’t mean you are doomed to go without any design, and it definitely doesn’t mean that good design is “big up front design” nor that a change in requirements will render your design irrelevant. A well-done architecture should be done fast, encapsulate volatility effectively, and even be antifragile to change.
Now FWIW, SAFe actually has a place for an architecture team, which sounds great if you believe firmly in architecture. But if you supposedly implement it without really tearing down the boundaries in the organization, how can the architects be effective? Will teams listen to the architects or will they plan sprints and cowboy code around them? When you look at the enormous diagram depicting SAFe, it may be abundantly clear that you are looking at nothing but boundaries, and more artifacts than you can shake a stapler–or computer clipboard–at.
Holub also believes that planning a project is a fallacy. Here again the IDesign culture says otherwise. People using effective Project Design techniques (which I also talk about here) typically estimate finish dates within 10%. The IDesign folks themselves operate under 3%. This is based on a combination of:
- solid architecture as input–one that can be built without running into the problems most badly- or un-designed systems encounter
- effective estimation techniques
- Proper assignment to teams and prioritization of work
- Understanding the scope of activities
- Critical path analysis
- Risk computation (yes, you can actually quantify the design risk of a project, as well as the execution complexity of the project, allowing you to understand where the sweet spot of success is, as well as the options to present to management)
- Progress reporting (surprise, timesheets are actually quite valuable here)
The upside is that in such an environment, it is actually possible to make software engineering behave like real engineering, rather than as the ugly duckling of the engineering family. And I’m being serious: other engineers scoff at the notion that software is engineering, when teams routinely go 2x-10x over budget only to produce substandard products. No mature engineering industry tolerates such things.
I can hear you saying, “But if you were building a house, it would cost extra to expand the bedroom once the siding’s already gone on!” And that’s true, but for many of you, your problems are more fundamental: in order to expand that bedroom, you will need to shift two bearing walls, affecting four adjacent rooms, re-route all the wiring not to mention plumbing in the adjacent bathtub, oh and by the way pour a completely new foundation in addition to all the things you were already talking about–because that’s the true extent of the design shortcomings of your architecture. The uncomfortable truth is that many of your systems are more disastrous than Pruitt-Igoe (the housing project, not the Philip Glass song).
So am I talking about “Big Up Front Design”? Or worse, that four-letter word “Waterfall”? First off, check your history: Waterfall was never intended to be used on large projects. And the Agile community loves to automatically use it as their whipping boy when you talk about doing any kind of planning that the Agile coaches didn’t tell you about.
Aside: at the activity (or user story) level, what you do is very similar to a small Waterfall. This is the only scale where it’s remotely appropriate.
But rest easy: as I said earlier, good design is a small-scale effort, both at the system and the project level. Which do you think costs more? Days of design? Weeks of refactoring and the ripple of regression testing (even if automated) and fixing? How about months to years of ongoing maintenance costs when you didn’t design? Do you want to be in that special Hell?
In case hearing about these failings from Holub isn’t enough, never fear: the list goes on. Andy Hunt, another signatory, repeats many of these same points in The Failure of Agile. Robert Martin, still another signatory, in The True Corruption of Agile, responds to Martin Fowler’s post and to one of Holub’s, and while he doesn’t agree wholeheartedly with Holub either, and in fact he emphasizes the value of practices, he is all for embracing better ones. Whereas Ron Jeffries, still yet another signatory, acknowledges in Scrum Fails, I’m Told, Endlessly that Scrum in particular has a lot of failures, mostly concluding that it’s not well-enough implemented or understood.
Perhaps my favorite though is Dave Thomas, another of the signatories. In the video Agile Is Dead (Long Live Agility) (as well as in this similar blog post) he explains why even the capitalization of the word Agile offends him, demonstrating how the Agile movement has been coopted by an industry of certificate-pushers. He talks in similar terms as Holub, though to Thomas’ credit, he also gives you a golden rule about making choices that enable future change: When faced with two or more alternatives that deliver roughly the same value, take the path that makes future change easier. In my view, Thomas’ point very nicely ties together both actual agility as well as considerations of volatility. And I contend that’s key to transcending rote processes, to transition from doing “Agile” to being agile.