Emergent Design

Before talking about emergent design, I always feel that I should suggest going and looking at Jim Coplien's discussions of the problems with "agile" design.

The problem is, Coplien's right. So-called emergent design - that is, design generated by an atomic, bottom-up approach in incremental steps - will work only if someone is (or a few people in very tight communication are) working in conformity to an implicit but detailed and coherent overall plan, in which case it's not really emergent at all. Continuous refactoring can preserve and even improve an underlying good design, but it's a very long way to get from a heap of well-executed details to anything like an optimal system architecture.

The other problem is, we don't really have a choice. The alternative is detailed top-down design - that is, waterfall - but that requires a requirements stage which is detailed, thorough, and subtle, and no organization I've been in in the past decade and a half was really committed to good requirements. In thirty years I have never seen genuinely comprehensive requirements, even in documents which purported to be comprehensive.

Many organizations don't even provide the up-front information needed for stage-based iterative design (Plan, Do, Review, Act). My experience is rather of painfully pulling details out of business analysts as one discovers one needs them. One of the marks of this is that the requirements are actually distributed like snow through an extended series of e-mails.

The presence of actual business representatives in a scrum model can take care of some of the need for iterative feedback. It's not so great for providing an overall context required for structure.

A good developer can make it look as though good emergent design is taking place, but it's largely an illusion based on the fact that a good developer should have some sense of what is going on. You can code from the bottom up if you have a good implicit understanding of what the overall structure you are working towards is, including both a decent understanding of of the problem domain (usually the "business domain") and a good technical understanding of the sorts of issues which are likely to arise in implementation. You end up effectively filling in small concrete components of implicitly organized packages creating necessary implementations in "the right place" as you go along. (Even then, I've had the experience of gradually raising the level of some piece of code from an implementation detail up to a general utility one step at a time as its generality became progressively more apparent.)

One thing that comes to the partial rescue is that, although we don't think of it that way, actual code can be viewed as a precise way of expressing specifications. An initial cut may be inefficient and weakly organized but it's usually worlds ahead of a typical "design" specification.

The first cut of code is (in an "agile" or semi-agile context) thought of as a part of the design phase. It need not be in a prototyping language; it may make a great deal of sense to write it in the final implementation language. It might be that it needs to be the one that you throw away (in Brooks ' classic advice), or it might survive as a part of the final implementation.

The model of starting with "the simplest thing that could possibly work" can mean various things depending on what is meant by "work". It is frequently taken to mean something that would not actually work because not enough thought has gone into what is required. In many cases really following this at a high level would involve writing a template with multiple policies, where the logic in the template class is genuinely functional but the complexity has been deferred to the policies; or starting with low-level concrete classes not because one is doing bottom-up design but because one can accurately define how such a class will have to function. This latter approach is even more attractive because no one class should have too great responsibility, complexity deriving from composition.

Implementing small components can be an excellent way of stimulating thought about their larger context. When and how can something change? What are the limits of its values? What are the edge cases? Questions in petto are good ways of shedding light on larger structural questions.

One question which is asked less than it should be in code reviews is: "should this code be here?". Frequently changes are made at a highly concrete level. Even aside from questions of reuse - does this logic also exist elsewhere in the current codebase - questions of generality often arise which might indicate that a piece of functionality should be hoisted from:

1) an inner class to a standard class inside a module but private to the module; or

2) a private class within a module to a public class, or a class made available by a public interface and a factory; or

3) a class in an application to a library; or

4) a library high-up in the stack to one lower on the stack as being of more general application.

Kevlin Henney defines architecture as what makes some things easy to do and others hard to do. Emergent design tends to ignore the fact that initial decisions constrain later ones; it assumes that partial understanding can be fixed at an unknown time in the future. To be effective, following an emergent design model requires an active approach of correcting this: to be asking oneself all the time "what are the structural effects of doing this in this way?". And if the answer is "we don't have that information yet" then it means that one has to be proactive in getting that information, in documenting it and its implications, and in setting out what the fundamental assumptions of the system are. ("We are assuming that it is simple and fast to determine what a business day is." "We are assuming that a tenth-of-a-second turnaround between a message received from a client and an acknowledgement will always be adequate." "We are assuming 99.999% reliable availability of the network, but not 99.99999%." "We are required to assume that the database may not be available for short periods of time after 5 PM and before 10 PM because of end-of-day-processing and we may have to queue certain sorts of updates during that time." "We need to be able to run a build and regression test on a snapshot of the code in one hour to support CI." All of the above have implications for both physical and software architecture.)

I spent the first six or so years of my life as a developer in a position where I really did have business expertise (supporting legal publishing functions). In DDD terms, I had an excellent mental model of the problem domain. But the normal experience of a modern developer is that, unless you spend a very long time in one place, you probably won't have that sort of mental model. (It takes a year or more in a moderately complex system just to get up to speed on the general code structure, never mind the business domain. Typical turnover between employers is a handful of years. Developers who hang around frequently leave development for management.) In that context it's fortunate that many of the critical elements in mid-level design are technical and that many of the core requirements of any system get reflected in the functionality fairly early on.