Book: Software Configuration Management Using Vesta

In a previous life I was a small part of a project called Vesta that developed a fairly radical approach to Software Configuration Management. Unfortunately, although Vesta has a lot of theoretical and practical advantages, it has proved difficult to get people to consider using it, because "not enough people are using it" (only a few hundred).

Vesta is available open-source for Linux at http://www.vestasys.org/ under the LGPL.

I have just received a copy of a new book about Vesta by its principal developers.

I hope that this book will stimulate some fresh consideration of using Vesta on projects where reliability and repeatability of software construction are important. I attach some excerpts from the Introduction and Conclusions sections, to give the flavor and perhaps entice more people into looking at the book.

Introduction

Vesta embodies the belief that reliable, incremental, consistent building is overwhelmingly important for software construction and that its absence from conventional development environments has significantly interfered with the production of large systems. Consequently, Vesta focuses on the two central challenges of large-scale software development -- versioning and building -- and offers a novel, integrated solution.

Versioning is an inevitable problem for large-scale software systems because software evolves and changes substantially over time. Major differences often exist between the source code in various shipped versions of a software product, as well as between the latest shipped version and the current sources under development, yet bugs have to be fixed in all these versions. Also, although many developers may work on the current sources at the same time, each needs the ability to test individual changes in isolation from changes made by others. Thus a powerful versioning system is essential so that developers can create, name, track, and control many versions of the sources.

Building is also a major problem. Without some form of automated support, the task of compiling or otherwise processing source files and combining them into a finished system is time-consuming, error-prone, and likely to produce inconsistent results. As a software system grows, this task becomes increasingly difficult to manage, and comprehensive automation becomes essential. Every organization with a multi-million line code base wants an automated build system that is reliable, efficient, easy-to-use, and general enough for their application. These organizations are very often dissatisfied with the build systems available to them and are forced to distort their development processes to cope with the limitations of their software-building machinery. …

Further, in contrast to most conventional SCM systems, Vesta takes the view that [versioning and building] interact, and that a proper solution integrates them so that the versioning and building facilities leverage each other's properties. …

Vesta subdivides the general problem of versioning into version management and source control. Building breaks down into system modeling and model evaluation. ...

Conventional build systems typically do not require and therefore rarely have comprehensive building instructions. Instead, they depend on the environment, which might comprise files on the developer's workstation and/or well-known server directories, to supply the unspecified pieces. This partial specification prevents repeatable builds. The first step toward achieving repeatability is to store source files and build tools immutably and immortally, as Vesta does, so that they are available when needed. The second step is to ensure that the building instructions are complete, recording precisely which versions of which source files went into a build, which versions of tools (such as the compiler) were used, which command-line switches were supplied to those tools, and all other relevant aspects of the building environment. Vesta's system models do precisely that. …

By following those instructions to the letter, the builder performs in effect a scratch build of the system. Completeness of the instructions makes the build repeatable, but for practicality it must also be incremental. Incrementality means skipping some build actions and using previously computed results instead, an optimization that risks inconsistency. To ensure that an incremental build is consistent, the Vesta builder records every dependency of every derived file on the environment in which it was built. …

Conclusions

The Vesta system's objective: to be a software configuration management system that scales to accommodate large software, is easy to use, and produces repeatable, incremental, and consistent builds. …

To summarize briefly, Vesta
● preserves source code immutably and immortally,
● supports both simple linear versioning and arbitrarily complex branching for parallel development,
● makes all versions directly accessible through the file system,
● provides very fast check-out and check-in using copy-on-write,
● supports distributed development with source replication, cross-repository check-out and check-in, and cross-realm access control,
● manages storage for source and derived files largely automatically,
● provides a flexible, general description language for the precise description and modular organization of software system construction,
● enables integration of new build tools within the description language without modification of either the tools or the Vesta system,
● builds software configurations repeatably, incrementally, and consistently, and
● runs as fast as Make for scratch builds and outperforms it for incremental ones. …

Vesta has been in daily use by a major engineering group since 1999. This group has evolved over the years, but it started out as DEC/Compaq's Araña group, which at the time it began using Vesta was a team of about 130 developers working on a large microprocessor design. The team was organized as two subgroups, one in New England and the other in California, each with its own Vesta repository. Both the chip design itself and the team's custom design software were stored in the repositories and developed using Vesta's suite of tools. The final code base consisted of about 700,000 lines. …

Overall, the Araña group found Vesta a substantial improvement over their previous build tools. Vesta's strong support for parallel source development and repeatable builds saved them considerable time (3 to 6 months in the architectural design phase alone), and the distributed developmental features provided answers to some extremely difficult problems they faced in bicoastal software and design database management. They also found Vesta's repeatability and consistency guarantees to be extremely useful for tracking down difficult bugs, a characteristic that gained in importance as they approached completion of the chip design.

After the DEC/Compaq Alpha division was transferred to Intel, the former Araña group continued to use Vesta on new projects. As of February, 2005, the number of developers using Vesta had grown to over 350, and the system being built had exceeded three million source lines. …

Collectively, the group at Intel routinely runs 20 simultaneous builds, each performing up to 10 tool invocations in parallel. Thus, 200 separate tools (mostly on distinct machines) are simultaneously competing for the repository's services. ...