Building Software the New Flexible Way

I like the way the industry of software development has progressed in the last ten years or so in many ways, but one significant area with which I am particularly impressed is the project lifecycle. In the eighties and nineties, massive, expensive and ambitious projects rose and fell everywhere. A software project was like a juggernaught that began with an idea, grew into a grand proposal, proceeded into a giant list of requirements, and from there was expected to progress steadily and admirably through the phases of high level design, low level design, build, test and deliver all through the miraculous coordination of huge teams of people and tremendous complexity. Oh the naivety, the broken hearts, the stress, the shattered budgets and the sour relationships!

Over time many bold and smart figures in the industry have turned the activity of developing software into a much more 'human' enterprise. We have recognized that computer software is so much more malleable than the products born out of centuries of engineering like bridges and buildings. We have had the courage to allow the undecided and the unknown into the process in a way that feels much more natural to us and much less frightening overall. And I believe today's software is better for it.

The modern software process still begins with an idea of course, one that is the seed of the project. That can be anything and, in the business world at least, usually arises from need for a business refinement or new opportunity. This idea broadly sounds like "we would a like a computer system to do [such and such]". Sometimes, a totally brand new computer system small or large is to be purchased and customized and sometimes the requirement is for an existing system to be modified.  In any case, running with the idea leads to a proposal being formulated (sometimes only mentally) stating the overall aims of the project. These parts of the project remain unchanged from the past. The main changes to the project cycle begin next.

The diagram below tries to summarize what happens from here. Notice a few important things about the project flow: (1) At no point do we try to nail down all possible details of the target system before developing. We agree from the outset that the project will be completed over time through a collaborative and iterative process. And (2) Each phase of the project is more similar to the old style of project. The phase has a certain set of requirements that are met by a process of design, deliver, quality control and refined according to feedback. Unlike the past, however - and this is important - quality is built into this process in the form of test-driven development over the low level design. This benefits both the phase and the whole project by guaranteeing quality in the deliverable while facilitating ongoing reqression testing as the product grows. (3) The end of each phase signals a continue-or-not point in the project. Invoices, salaries, etc are ideally paid up until the end of an agreed and delivered phase, but the sponsor always has a) something for their money and b) control over future spending. If phases are well designed, there is also a reasonable chance of useful deliverables coming out of even an aborted project.

Diagram 1:  A very high level summary of new project structure