During Quantal I started working on a Continuous Integration (CI) environment to run commit level testing of Gnome core components on the latest development release of Ubuntu.
Today was a major milestone, for the first time with the help of Martin Pitt, all the red dots are gone, leaving only blue/green (build successfully and make check pass) or yellow dots (module builds but make check failed) This is a great step forward \o/
The current system is fetching, building and checking of 160 packages on the latest development release of Ubuntu, here is how it works.
The testing environment is an LXC container with a minimal system to run Ubuntu Raring/AMD64. The host system is running on Ubuntu Server 12.10.
In the container, I installed JHBuild from git, system dependencies, a basic webserver to serve the results outside of the container (an alternative would have been to install a Jenkins slave inside the container, but I didn’t choose this option due to communication issues between the slave and the master) and a script run from cron to drive the builds and report the results.
Jenkins has the passive role of reporting the results.
The build/test workflow is the following:
- Every 15min, the list of modules to build meta-gnome-core is fetched with jhbuild
- The revision of the modules in the list is compared to the previous run. New packages are added automatically to Jenkins.
- If a package or one of it’s dependency has changed, it is addded to the rebuild list
- The rebuild list is then processed to first build the package with jhbuild, then re-run a jhbuild with option check to run a make check if the build succeeded. JHBuild is setup to run non-interactively of course, and to force a checkout and autogen on failure.
- Logs and results are captured and published via the webserver.
- Jenkins monitors the state files published by this webserver, updates the results and collects the log files.
The whole system could have been summarized by
while :; do jhbuild build; sleep 60; done
but I wanted a bit more granularity and control over what is being build and tested. Furthermore building the whole stack every single change in glib for example was not really efficient (WebKit tends to use a couple of CPU cycles to build )
So, overall it’s pretty simple, there are still reliability issues and some failures need manual interventions, but having kicked all the red dots is encouraging.
The next steps will be to add a notification system (and send messages the right people), detailed test reports (convert gtester results to jUnit XML), rely on jhbuild’s sysdeps instead of a yet another list of static dependencies, improve the performance of reverse dependencies checking, and …
… fix the yellow dots!