Fedora Extras Build System System Requirements: - Python 2.3 or 2.4 - pyOpenSSL - sqlite python bindings - mock (http://fedoraproject.org/wiki/Projects/Mock/) - createrepo Getting Started ------------------------------------------ To allow users to retrieve logs and other status, you need to run an HTTP server that allows access to the result dir (the 'server_work_dir' config option). You will also need to set up the infrastructure for the yum repository that the builders connect to and retrieve the latest packages. This can be either HTTP, NFS, SMB, etc. You then need to point 'yum' to this repo in step (5) of the Builder Setup. Builder Setup: 1) In the CVS checkout directory on the builder system, execute: make DESTDIR=/ install 2) Create a user for the builder. The builder drops root privileges before running the build, but must have root to be able to do initial setup 3) Copy the client Key, Cert, and CA Cert to the /etc/plague/builder/certs directory 4) Things to modify in the builder's CONFIG.py: - Modify the 'distro' and 'repo' options to match the targets you've configured in /etc/mock/. These targets are usually in the form of "distro-target-arch-repo". 'arch' and 'target' are passed by the build system dynamically, but 'distro' and 'repo' are hardcoded in the config file. Examples are "fedora-development-i386-core" and "fedora-development-i386-extras". 5) Configure the mock target files in /etc/mock. You only need one target file for each major arch you support. For example, you don't need separate 'ia32e' or 'amd64' config files, since these just use the normal 'x86_64' config file 6) Start the client. ex: "/usr/bin/plague-builder 127.0.0.1 i386 i686" On the Server: 1) Follow the instructions at the bottom of this file titled "Configuring SSL for your Build System" 2) In the CVS checkout directory, execute: make DESTDIR=/ install 3) Copy the server Key, Cert, and CA Cert to the /etc/plague/server/certs directory 4) Copy the client authentication CA Cert to the /etc/plague/server/certs directory 5) Things to modify in the server's CONFIG.py: - Update the Key, Cert, and CA Cert, and client auth CA Cert file options to point to the files in steps 3 and 4 - Modify the 'targets' option to add/remove the arches and targets you'll be building - Modify the 'builders' option to point to the build clients you'll be using. Note the "https". - If you want to do simple SRPM builds, set the 'use_srpm_not_cvs' option to true 6) Start the server. ex: "/usr/bin/plague-server 127.0.0.1" Operation: 1) You must add a user account for any user who wishes to use the build system. This is accomplished with the 'plague-user-manager.py' tool, installed by default in /usr/bin. You add a user like this: /usr/bin/plague-user-manager.py /etc/plague/server/userdb add \ dcbw@redhat.com own_jobs kill_any_job modify_users server_admin 2) Clients then run plague-client to queue jobs. When first run, plague-client creates the ~/.plague-client.cfg file - Point the client to the server's address - Point the client to the correct certificates - Make sure you change the email address in ~/.plague-client to match that of the 'user-cert' certificate 3) To build a package, you use plague-client like so: /usr/bin/plague-client build ethtool /home/dcbw/ethtool-1.8-4.src.rpm devel 4) If the client returns "Package ethtool enqueued." then the enqueue was successful You can list your own jobs with: /usr/bin/plague-client list Builders can be listed & updated as well, see plague-client's usage message. Architectural Overview: ------------------------------------------ The build system is composed of a single build server, and multiple builders. Builders run an XMLRPC server to which the build server delivers build jobs. The build server runs an XMLRPC server to allow submission of jobs, and to relay basic status information about both builders and the build system as a whole to users. The Builder: usage: /usr/bin/plague-builder -c Currently, builders are limited to building one job at a time, though there is no restriction on running multiple builders on a single machine. They do not queue pending jobs, but will reject build requests when something is already building. The build server is expected to queue and manage jobs at this time, and serialize requests to builders. main() `- Creates: XMLRPCBuilderServer `- Creates: i386Arch, x86_64Arch, PPCArch, etc (subclasses of BuilderMock) The builder creates an XMLRPC server object (XMLRPCBuilderServer), and then processes requests in an infinite loop. The build server queries each client periodically for its status and the ID of the currently building job, if any. Each build job (BuilderMock and its architecture-specific subclasses like i386Arch) proceeds through a number of states. Build jobs are periodically given time to do work (BuilderMock.process()) by the builder controller (XMLRPCBuilderServer._process()), which is in turn periodically given time by the client's main loop. During their processing time, build jobs check their state, see if any actions have completed, and advance to the next state if needed. Communication with mock and retrieval of status from mock are done with popen2.Popen4() so that mock does not block the XMLRPC server from talking to the build server. All communication with the build server is done through SSL to ensure the identity of each party. Both the XMLRPC server and the builder's file server are SSL-enabled, and require SSL certificates and keys to operate. See later section in this document on how to configure SSL certificates for your build system. The Build Server: usage: /usr/bin/plague-server The build server runs two threads. The first, the XMLRPC server (XMLRPCBuildMaster class), accepts requests to enqueue jobs for build and stuffs them into an sqlite database which contains all job details. The second thread, the Build Master (BuildMaster class), pulls 'waiting' jobs from the database and builds them. A third top-level object that runs in the same thread as the Build Master is the BuilderManager, which keeps track of builders and their status. main() |- Creates: AuthedXMLRPCServer (system users talk to this object) |- Creates: BuilderManager |- `- Creates: Builder (one for each remote build client) |- `- Creates: ArchJob (one for each build job |- on each arch) `- Creates: BuildMaster `- Creates: PackageJob (one for each build job) The BuilderManager object serves as a central location for all tracking and status information about each build job on each arch. It creates a Builder instance for each remote builder. The Builder instance keeps track of specific jobs building on all architectures on that remote builder. PackageJobs must queue requests with the BuilderManager for the architecture specific build jobs they need. The BuilderManager will wait until a Builder object is available to build the job, create the new ArchJob on the builder, and notify the parent PackageJob of its new ArchJob. The BuilderManager has a periodic processing routine that is called from the BuildMaster thread. This processing routine calls the Builder.process() routine of each Builder instance, which in turn updates its view of the remote builder's status. Thus, the BuilderManager, through each Builder instance, knows the status and currently building job on each remote builder. PackageJobs track a single SRPM build through the entire build system. They are created from the BuildMaster thread whenever the BuildMaster finds a job entry in the sqlite database with the status of 'waiting'. PackageJobs proceed through a number of states: "initialize", "checkout", "make_srpm", "prep", "building", "finished", "addtorepo", "failed", "killed", and "needsign". Flow goes like this: initialize => checkout checkout => make_srpm make_srpm => prep prep => (queue architecture-specific job requests with the BuilderManager) building - All build jobs finished or failed? => finished - otherwise => building finished - failed jobs? => failed - otherwise => addtorepo addtorepo => repodone repodone => needsign All communication with builders is done through SSL to ensure the identity of each party. When the builder requests the SRPM to build, SSL is used. When the build server retrieves logs and RPMs from the builder, SSL is also used. This ensures that builders can be more or less trusted, or at least that some random builder is not serving you packages that might contaminate your repository. See later section in this document on how to configure SSL certificates for your build system. Configuring SSL for your Build System -------------------------------------- When you set up the build system, you essentially become a Certificate Authority. Because the build server and the build clients communicate using SSL, they need to exchange certificates to verify the others' identity. You must first createa key/cert pair for the Build System Certificate Authority, which signs both the build server's certificate, and each build client's certificate. The Certificates on the Server: config_opts['server_key_and_cert'] -> server SSL certificate and private key config_opts['ca_cert'] -> CA certificate used to sign both server and builder certificates config_opts['ui_ca_cert'] -> CA cert that signs package maintainer's certificates, used to verify connections from plague-clients are authorized The Certificates on the Builders: config_opts['builder_key_and_cert'] -> builder SSL certificate and private key config_opts['ca_cert'] -> _same_ as server's 'ca_cert', the CA certificate used to sign both server and builder certificates Setting up the Certificates: A tool called "certhelper.py" is included in the utils/ directory, or possibly as /usr/bin/plague-certhelper if installed from a package. Here we use certhelper.py for either program. 1. Create the Build System Certificate Authority key and certificate: certhelper.py ca --outdir=/etc/plague/ca_dir --name=buildsystem After entering the certificate's details, if there are no errors, you end up with a self-signed certificate in /etc/plague/ca_dir/buildsystem_ca_cert.pem, and a CA private key in /etc/plague/ca_dir/private/buildsystem_ca_key.pem. You will need buildsystem_ca_cert.pem later. 2. Create a certificate and key for the build server: certhelper.py normal --outdir=/etc/plague/server/certs --name=server \ --cadir=/etc/plague/ca_dir --caname=buildsystem After entering details for the server's certificate, you end up with a combined certificate and private key file in /etc/plague/server/certs/, server_key_and_cert.pem. Copy the buildsystem_ca_cert.pem file from step 1 into /etc/plague/server/certs as well. Update the server's config file in /etc/plague/server/CONFIG.py to match the absolute path to server_key_and_cert.pem (the 'server_key_and_cert' config option) and also the absolute path to buildsystem_ca_cert.pem (the 'ca_cert' config option). Both should be based in /etc/plague/server if you follow the directions here. IMPORTANT: make sure only the build server's user (normally root) can read server_key_and_cert.pem, since it contains the server's private key. 3. For each builder you plan to deploy, you will need to generate a certificate for that builder as well. certhelper.py normal --outdir=/etc/plague/builder/certs --name=builder1 \ --cadir=/etc/plague/ca_dir --caname=buildsystem After entering details for the builder's certificate, you end up with a single file, /etc/plague/builder/certs/builder1_key_and_cert.pem. By default, the plague-builder is set up to look for the file ".pem" where is the host name of the builder machine. You will need to rename the builder1_key_and_cert.pem file to the builder's hostname with a .pem extension. If the builder is on a separate machine, you will need to copy the certificate file to that machine along with the CA's certificate, buildsystem_ca_cert.pem, and the server's plain certificate, server_cert.pem. Update the builder's config file (normally in /etc/plague/builder/CONFIG.py) to match the absolute paths to the builder's key_and_cert file, and to the build system CA certificate, buildsystem_ca_cert.pem. IMPORTANT: make sure only root can read builder1_key_and_cert.pem, since it contains the buidler's private key. ------------------ Package Maintainer Certificates: Config options from ~/.plague-client.cfg, used by /usr/bin/plague-client: server-ca-cert -> _same_ as build server and builder's 'ca_cert' user-ca-cert -> CA cert that signed the package maintainer's 'user-cert' user-key -> package maintainer's private key, can be blank if private key and certificate are in the same file user-cert -> package maintainer's certificate, signed by 'user-ca-cert' and sent to build server to validate the plague-client's connection To allow package maintainers to connect and queue up packages, it is best to use a completely separate Certificate Authority to sign user certificates. If you use the same CA as you use for the build server and builders, any maintainer could set up a rogue builder using his/her certificate. To prevent this, create a new certificate authority for package maintainer certificates. 1. The procedure closely follows the steps for the build system certificate setup. Create the CA in the same way (using a different --outdir and --name of course). The *_ca_cert.pem file that certhelper.py spits out should be added to the build server's CONFIG.py file for the 'ui_ca_cert' config option. It should also be distributed to package maintainers, who enter this certificate in their ~/.plague-client.cfg file as the 'user-ca-cert' config option. certhelper.py ca --outdir=/etc/plague/user_ca_dir --name=users 2. To create user certificates, do the same steps as for creating builder certificates above, except MAKE SURE to enter that user's email address at the certificate information entry's "Email Address []:" prompt. This is essential for user validation and notification of build status. certhelper.py normal --outdir=/tmp/user_certs --name=user1 \ --cadir=/etc/plague/user_ca_dir --caname=users Here, certhelper.py will produce a file called user1_key_and_cert.pem in /tmp/user_certs. Send this file, along with users_ca_cert.pem and buildsystem_ca_cert.pem, to the package maintainer whose email address you entered for this certificate. The package maintainer then sets up his/her ~/.plague-client.cfg to point to the correct files: 'user-ca-cert' => users_ca_cert.pem 'server-ca-cert' => buildsystem_ca_cert.pem 'user-cert' => user1_key_and_cert.pem 'user-key' => should be blank