RPM building system

I used in the past to build some custom RPM packages for the Linux distribution of my choice, Fedora Core. Building packages is not that difficult, it just requires a prepared environment, some skill to script how a software should be compiled and installed on the machine, and a relative patience, to restart the process from the beginning, when an unexpected error occurs during package generation, which can take a repeated long time when the software to be packaged is a big project. The following items are required:

• a compilation environnement, a C or C++ compiler, the autotools, perl, python, all *-devel packages needed to compile the software or project to be packaged;
• an RPM build tree in the homedir (~/rpm/SPECS, ~/rpm/SOURCES, ~/rpm/SRPMS, ~/rpm/RPMS, ...), because you won't compile packages as root, won't you?
• a .spec file that describe the way a tarball should be compiled, possibly patched, installed, and which installed files of will be part of the RPM package.

The drawback of this method is that building a package uses the compilation tools of your own machine, which may introduce subtle problems (for example if the packages of your machine are not up to date, or if you use a non standard compiler by default, if you have a non-standard PATH, or even if your spec file is buggy you can potentially delete files by mistake,...).

Another typical problem with this method is that it is only possible to build packages for the distribution that is running on the build machine, and not for another one. For these reasons, it is important to have a sanitized build environment, that will be independant of the host build environment.

The Fedora project provides a complete infrastructure for building RPM packages, managing the whole chain, starting from the spec file handling, to the publication of packages and their related yum metadata on a remote server, for several Fedora Core distributions simultaneously.

Mock

Mock is a chroot-ed environment, automatically configured with a minimal set of packages, and used to build RPM packages. It relies on a set of yum repositories, and can generate build environment for several distributions in parallel. The drawback of mock is the counterpart of its force : it is slow to generate the build environment, because it installs with yum and RPM a minimal linux distribution each time a package is to be build. Hopefully this stage can be optimized :

• If the build machine has a huge memory (more than 2GB), it is a very good idea to put the whole build environment in RAM, by using a tmpfs for /var/lib/mock for example.
• mock has the possibility to generate a cached version, with a tarball, of the minimal build environment. The final build environment is generated by the cached tarball, plus the supplementary packages referenced in BuildRequires: rules of the spec file, including their dependencies.
• another possibility is to use a squid web cache, and to configure it to cache downloaded packages (see fedoraproject.org wiki for details), which is more optimal than mirroring locally the whole distribution tree.

Plague

Plague is the client-server infrastructure used to build Fedora Extras packages. It defines a set of builders, whose role is to receive a .src.rpm file, to build the package using mock locally, and to return the binary packages along with the compilation log to spot errors occuring during the compilation. The more builder you have, the better your build requests are processed. Builder can generate binaries for architectures that are compatible with their host architecture.

A plague server collects build requests, queues them, and dispatches them to the builders, it also collects generated packages, and build logs. All this information is made available via a web interface.

A plague client on the package maintainer machine is a command-line tool, that communicates with the build server, and is used to submit src.rpm packages to be compiled. In this case, src.rpm packages are _not_ the result of a previous compilation, but simply the container that includes the tarball, patches and the spec file needed to compile the program (rpmbuild -bs <specfile> is used to generate this src.rpm file). The client can process the build queue, kill pending jobs, requeue them, query the builders status for example.

CVS

The last step of this infrastructure is a CVS server, used to store spec files, and patches for each package. A directory different is created for each package, subdirectories are created to handle branches for multiple distribution versions (FC-4, FC-5, FC-6, devel, ...). Spec files and patches files are versioned in these subdirectories.

• Tarball and various binary files are not commited in the CVS server, because the CVS protocol is not suited for binary files. They are uploaded and downloaded on demand on a separate server, using the HTTP protocol instead;
• the branch mecanism of CVS is not used to handle different distribution versions of a same package. Branch creation instead is handled by duplicating the devel branch directory on the CVS server directly;
• package generation, testing, changelog update, all these maintainer tasks are made easier by some common Makefile rules;
• packages version are identified by CVS tags;
• and interaction with the plague server is possible based on these CVS tags. This relies on the assumption that the plague server has a read access to the CVS server, and can itself generate the src.rpm file from the CVS tag. This mechanism uses less bandwidth, because the tarball doesn't need to be download to the package maintainer machine, and the resulting src.rpm doesn't need to be uploaded back to the build server.

Push scripts

The extras-buildsys module of the fedora CVS repository provides some handy python scripts, that collect plague-builded packages, sign them with GnuPG, puts them in a common tree, creates the repodata information for yum, generates html pages describing them, and eventually uploads the final tree onto a remote web server.

The directory layout of packages as generated by plague is specific, because is stores each package version into a specific directory, and each architecture binaries in subdirs. These directories are processed by createrepo, because their content needs to be made available to mock too.

In the mock configuration of each builder, the following repos are needed:

• fedora core
• fedora updates
• fedora extras
• local build packages in their exported tree, after the script pushed them
• local build packages in their plague directories, before the script pushed them.

Goodies : viewcvs, bonsai and lxr

• Bonsai is a web interface initially created for the mozilla project, that allows to easily display the CVS activity of a project, by querying commits on a selected time-frame, and/or on specific modules, and/or by particular commiters;
• Lxr is a CVS source code browser and indexer, also developped for the mozilla project, and intimately coupled to bonsai;
• ViewCVS is an alternate CVS source code browser.

Links

The Plague server : http://bellet.info/build

ViewCVS: http://bellet.info/viewcvs

Bonsai: http://bellet.info/bonsai/toplevel.cgi?treeid=extras

Lxr: http://bellet.info/lxr

External links

http://fedoraproject.org/wiki/Projects/Mock

http://fedoraproject.org/wiki/Projects/Plague

http://fedoraproject.org/wiki/Infrastructure/VersionControl/Current

http://fedoraproject.org/wiki/Extras/UsingCvsFaq

http://cvs.fedora.redhat.com/extras.shtml