March 19, 2006
Resilience is better than anticipation
“In preparing for battle I have always found that plans are useless, but planning is indispensable.”
— Dwight D. Eisenhower
Paul Saffo, favorite futurist of every journalist on the tech beat, famously said, “Never mistake a clear view for a short distance”. An important corollary is: being able to see the destination doesn’t necessarily mean you can see the road that gets you there. One lesson I take from this is that trying to plot a detailed map of that road can be a big waste of time.
This tome represents half a million (1993) dollars of Grade A, USDA Choice, prime Vision, happily paid for by our friends at Fujitsu. This lays it all out, in loving detail. This is the document that sold the venture capitalists on funding Electric Communities’ transition from a three-guys-who-do-cyberspace consulting partnership to a full bore Silicon Valley startup company. We had a regular business plan too (albeit one that was a little vague in the “and this is where the money comes from” part; see It’s a business, stupid), but what the VCs were really buying into was this: the utopian dream. It was exhilarating, it was brilliant, it was some of the best work I’ve ever done.
It was doomed.
It was at once too detailed and too vague. This is the nature of big complicated plans: they have lots of details (that’s what makes them big and complicated) and they leave lots out (because, the world being the complex thing that it is, no matter how much detail you give, it’s never enough to completely describe everything relevant). Plus, the more details and complexities there are, the more opportunities you have to make mistakes. As the number of elements you are juggling grows large, the probability of significant errors approaches certainty. (One notable VC who declined to invest in Electric Communities told us, “This is one of those Save The World plans. We don’t do those; they never work.” I want him for an investor the next time I try to start a company.)
Big utopian plans are doomed by their very natures, because they are always wrong. Being the huge fan of F. A. Hayek that I am, I should have figured this one out a lot sooner. I no longer believe in big plans that try to comprehensively anticipate every requirement and every contingency. Instead, I believe in resilience. Resilience is better than anticipation (a formulation for which I need to credit my friend Virginia Postrel).
It is better to have a simple plan and be resilient in its execution.
By resilience, I mean the ability to quickly and inexpensively adapt to the inevitable changes in circumstance, unforseen events, and surprising discoveries that any significant undertaking is bound to encounter. Unless what you are doing is completely trivial, the awful truth is that you must do it in an environment that is filled with uncertainty.
Most people will acknowledge the value of being prepared for unexpected external contingencies like earthquakes or downturns in the market. Fewer will take into account the broader, more diffuse, but ultimately more important phenomenon which plagues most long-term projects, namely that the nature of the world shifts between the time you start and the time you plan to be done, so that a plan that might have been ideal when you started might be disastrous by the time you finish (Freeman Dyson has some interesting things to say about this in Infinite in All Directions). Very few indeed appreciate what I think is the biggest source of uncertainty, which is that you don’t really understand exactly what to do and how to do it until you are well on your way to having already done it. It is in the doing of a thing that you end up discovering that you need to do other things you hadn’t originally counted on or that you now need to do some things differently from how you’d originally intended. Indeed, you may discover that you’ve already done some things wrong that now need to be done over or just thrown away.
You might reasonably ask how I can possibly reconcile this extreme skepticism about the value of (or, indeed, the possibility of) planning with what I mainly do for a living, which is to develop large, complex software systems. These undertakings would seem to demand exactly the kind of comprehensive, large-scale planning that I’m criticizing here. Indeed, this is how the world of software engineering has usually approached things, and they have the long history of schedule overruns, budget blowouts, and general mayhem and misery to prove it. Accepting the limitations of human rationality with respect to planning and forecasting is merely bowing to reality.
My new religion, in the realms of project planning in general and software development in particular, is what I guess I’d call “hyperaggressive incrementalism”:
Do really small steps, as small as you can manage, and do a lot of them really, really fast.
Don’t do anything you don’t have to do, and don’t do anything that you don’t have an immediate need for. In particular, don’t invest a lot of time and effort trying to preserve compatibility with things that don’t exist yet.
Don’t try too hard to anticipate your detailed long term needs because you’ll almost certainly anticipate wrong anyway. But be prepared to react quickly to customer demands and other changes in the environment.
And since one of the dangers of taking an incremental approach is that you can easily drift off course before you notice, be prepared to make sweeping course corrections quickly, to refactor everything on a dime. This means that you need to implement things in a way that facilitates changing things without breaking them.
Don’t fix warts in the next rev, fix them now, especially all the annoying little problems that you always keep meaning to get around to but which never seem to make it to the top of your todo list. Those annoying little warts are like barnacles on a ship: individually they are too small to matter, but in aggregate their drag makes it very hard to steer and costs you a fortune in fuel.
Simple is better than complicated. General is better than specialized. But simple and specialized now is better than general and complicated some day.
With respect to software in particular, adopt development tools and processes that help you be resilient, like memory safe, strongly typed, garbage collected programming languages and simple, straight-ahead application frameworks (i.e., Java very good, J2EE very bad). I also favor rigorous engineering standards, ferociously enforced by obsessive compulsive code nazis. I sometimes consider it a minor character flaw that I’m not temperamentally suited to being a whip-cracking hardass. Every team should have one.
Writing things down is good, but big complicated specifications are of a piece with big utopian plans: too ponderous to be useful, usually wrong, and rapidly obsolescent.
In general, it is better to have a clear, simple statement of the goal and a good internal compass, than to have a big, thick document that nobody ever looks at. That good internal compass is key; it’s what distinguishes a top tier executive or developer from the second and third stringers. Unfortunately, the only way I’ve found to tell whether someone is good in this respect is to work with them for several months; that’s pretty expensive.
My bias at this point is to favor productivity over predictability. It’s OK to have a big goal, possibly even a really big, visionary, utopian goal, as long as it’s just a marker on the horizon that you set your compass by. Regardless of what plans you may have (or not), a productive process will reach the goal sooner. Though predictability is elusive, productivity, in contrast, is actually achievable.