Agile Specification Guidance & Support
Agile software development has gained an incredible amount of momentum, to the point that "agile" is considered to be what "good" developers follow, while formal methods that support the CMMI are associated with all that is "bad" in software development. The term "agile" has a lot to do with it, as anything that is not agile can be assumed to be some sort of antonym to agile, such as plodding, bloated, slow, cumbersome, and so forth. The resulting dichotomy has grown to the point that agile is mistakenly perceived as laissez-faire, or "do what you want to do" software development, and anything else is automatically bad because it is not agile. In particular, many developers perceive the adoption of agile software development to mean that they do not have to devote any effort to the actual design of the software. Everything is done with iterations and prototypes between the actual users and the developers. This is nice, but it doesn't work.
It doesn't work because end-users don't do well watching a demonstration of some hacked up prototype that only the developer can make work, and even then the demonstration is interspersed with pauses while the developer goes back and tweaks some code right in front of everyone so that the demonstration can proceed. In the end, the users leave with a blurry impression of the prototype mixed in with examples of how the development environment works. This works even less well for those end-users that were unable to make it to the demonstration in the first place. In the end, the end-user community has only a vague notion of what is actually going to be implemented. I have seen this happen over and over again in the name of agile software development. It doesn't work, and it's not the way agile is supposed to be.
Currently, it appears that four major methodologies have emerged as predominant under the agile umbrella. Let's take a look at each one to see whether they support the concept of a design effort:
Crystal Clear: Alistair Cockburn
The author specifically describes a release plan, use cases, design sketches, and a common object model. These are used to develop actual code, test cases, and a user manual among other things. There are a lot of books and third-party tools available to help all of this modeling and sketching before any actual coding is done.
Extreme Programming: Kent Beck, et al.
Extreme stresses the use of a "simple design" as opposed to an anticipatory design. The objective is to keep the design simple, and add flexibility through re-factoring only when and where necessary. For example, if a confirmation dialog is displayed only once in the application, it makes sense to hardcode the design as well as the implementation. Later, when it becomes necessary to sprinkle confirmation dialogs throughout the application, it makes sense to re-factor the design to include a utility confirmation dialog that accepts parameters to influence its behavior. Either way, the re-factoring happens at the design level before it happens at the code level.
Feature-Driven Development: Jeff DeLuca, Peter Coad
FDD specifically describes a hierarchy starting with a model, composed of features. Then come feature designs followed by feature builds. FDD stresses planning by feature as well as designing by feature.
Scrum: Ken Schwaber
Scrum focuses on the concept of a product backlog, which is a list of all business and technology features envisioned for the product. The product backlog is a catchall place for everything that is or will be incorporated into the application. Out of this comes a release backlog, which is a selection of features that should be in the next product release. Finally, a sprint backlog is created consisting of features to be implemented in the current (usually 30 day) sprint, along with tasks required to implement each feature. Interestingly, a "feature" can be anything from a complete object model to a data entry user interface layout to a formatting and presentation mask (such as "Money").
While almost everyone is advocating design, almost no one advocates formal documentation. For example, extreme programming recommends most of the design be held in CRC cards, with unit tests and acceptance tests acting as substitutes for additional documentation. Everywhere I look, people are avoiding documentation like the plague. This probably has something to do with the old IEEE standards that specified huge volumes of documentation that never worked, because people don't read big books after they get out of school.
However you look at the picture, all of the agile gurus have defined a design -> build approach. While the designing and building is done for the most part in small increments, it nevertheless gets done. There's a reason for this, and it's called the End User. The users are the reason the software is being developed, and it stands to reason that the software should fit their needs. Since nobody's been able to figure out how to make the Vulcan Mind Meld work yet, we're faced with a knowledge transfer problem, and knowledge transfer is something I've specialized in for a long time.
No one is able to transfer their knowledge to another in one shot. We're just not built that way. We transfer knowledge best in little bits and bites, starting with the obvious and moving to the less obvious. In academia, this is known as transferring explicit knowledge and tacit knowledge. A significant amount of explicit knowledge has to be transferred before tacit knowledge becomes meaningful to the recipient, and thus eligible for transfer. In essence, one can't understand the nuances of a subject until they understand the basic framework.
Specifications are how knowledge is transferred from the end user to the developer. Specifications establish that common framework, and then fill it in with the detailed business rules that embody the tacit knowledge of the process under automation. Without that transfer of knowledge from the user to the developer, the software won't be right. This is why the role of systems analyst is so essential to the success of a software project. Analysts elicit the knowledge from the users and build the specification. Developers use the specification to build and test the product. It's that simple.
This is why this Agile Specification site exists. All the gurus agree that design should be done at some level before the code is written. A formal documentation approach takes too long and imposes too much load for most projects. Today, we can build Web sites that do the job better, faster, and cheaper. It's hard to get started unless you've done it many times before, or you have a template and guidance to work from.