Dreaming in Code - One Quest for Transcendent Software

The realization came over me with full force that a good part of the remainder of my life was going to be spent in finding errors in my own programs. —Maurice Wilkes

Software fails and fails hard. Dreaming in Code documents the downfall of the well-funded Chandler project. Experienced team members were brought together, they failed and the result was discarded.

I can make it for you fast, cheap or well. Pick any two.

Chandler’s successes

Chandler had grand ideas and a new, unique approach to data where many types of data are organized into a central object store. It attempted to emulate intelligent features from products like Lotus Agenda. Agenda contained “automatic assignment” that predicted semantic meaning from email text and felt like magic. The Chandler developers wanted to use Chandler to build Chandler — “eat your own dogfood”.

Hertzfield chose languages based on the understanding that processor time is cheaper than programmer time. Python gave him a three times productivity improvement over Java, and that was already three times over C.

Kapor put “(0) User experience” before technical criteria and proposed writing software to Vitruvius’s principles of good design:

  • Firmness, sound structure and no bugs
  • Commodity, suitable for intended purpose
  • Delight, user experience is pleasurable

Chandler’s failures

The hard thing about building software is deciding what to say, not saying it —Brooks

Deciding precisely how to build Chandler’s big ideas was a core uncertainty that spread to every decision. People wanted ‘more concreteness’ but they needed to be more concrete about what needed to be more concrete. Success is often a by-product of a firm choice made and reasserted to limit a project’s scope and successful programmers thrive with constraints.

Having multiple groupware systems caused staying in the loop to eat up much of the workday. Developers drowned in emails, bug reports and wiki pages. A heroic additional glossary that acted as a central point of reference for project definitions was left unread.

Nobody should start to undertake a large project, you should start with a small trivial project, and you should never expect it to get large. If you do, you’ll just overdesign and generally think it is more important than it is likely is at that stage. Or, worse, you might be scared away by the sheer size of the work you envision. So start small and think about the details. —Linus Torvalds

The decision to build a server increased Chandler workload and subtly began to fork the OSAF organization.

Management

Management is about human beings. Its task is to make people capable of joint performance, to make their strengths effective and their weaknesses irrelevant —Peter Drucker

A track record is the only evidence that matters when evaluating a software development manager.

When Humphrey took over IBM in 1966, managers did not make plans because “they did not have time”, but “the right way to do the job was undoubtedly the fastest and cheapest way… Managers were not managing but reacting.” He realized that “as long as I permitted them to announce and ship products without plans, they would continue to do so.”

At Mozilla, Mitchell Baker said they “lost a lot of time, some of it to the technology but a lot of it to the inability to get management”, this management was the original Netscape management, connected with their key engineering forces. The important part was “just getting far enough along to have something that basically works.”

Software development

All things fall and are build again,
And those that build them again are gay.
—William Butler Yeats, “Lapis Lazuli”

When being trained to get a master of fine arts in poetry, you study the great works of poetry, but we cannot do that with software due to a lack of history and more-so due to most of it being locked away behind commercial organizations. Trainee poets study with mentors, present work for regular criticism by peers and labor over multiple revisions of the same work.

“Information systems only give value when they touch human beings“ says Jaron Lanier, but when they do the prospect of perfection dissolves. See the big picture when building a cathedral, avoid polishing each stone perfectly. Many projects are held up by sequential constraints that cannot be sped up with additional resources. It is therefore important to have great programmers as programmer productivity varies by orders of magnitudes, and these great programmers complete sequential tasks faster.

Communicating abstractions unambiguously — from programmer to machine, from programmer to programmer, and from program to user — is the single most challenging demand of software development. To prevent ambiguity, programmers must maintain variable names with changes, however Cunningham that found most do not.

Software is 100% likely to change over an expected 10 year lifetime — “You’re gonna have to change it for different hardware and software environments, change languages three times, and go through four personnel turnovers. What you have to do is fight as hard as you can to keep quality up so that ten years from now you have something left.” Only a sixth of the time in software projects is spent writing code, but a full half is spent on testing.

“Thinking things out in the order that the computer will execute them” is how programmers work, yet doing so is ultimately beyond their capability. That means “programming is a trial-and-error craft”.

Quotes and facts

  • Boehm’s Law “the later in the development process you fix a bug, the more it costs to do so”
  • Brooks’ Law “Adding manpower to a late software project makes it later”
  • Linus’s Law “Given enough eyeballs, all bugs are shallow”
  • Organic human relationships are a semi-lattice (A City Is Not A Tree, 1969)

See also