Implementation of Continuous Integration for Linux Images Jérôme Belleman, Linux Support Team HEPiX Fall 2017 Implementation of CI for Linux Images 2
Context HEPiX Fall 2017 Implementation of CI for Linux Images 3
Operating System Image Production Linux Support Team make images Built with Koji E.g. accessible from OpenStack: % openstack image list +-----------------+--------------------------------+ ID Name +-----------------+--------------------------------+ 29262 d...3 c8fc9 CC7 - x86_64 [2017-04 -06] d8c28e... a9015b SLC6 - x86_64 [2017-04 -06] f1c040...0 c92c4 SLC6 - i686 [2017-04 -06] b2b7ba... e5c9c3 CC7 TEST - x86_64 [2017-04 -06]... HEPiX Fall 2017 Implementation of CI for Linux Images 4
It s Critical That... We build images quickly We build images autonomously We make sure images work flawlessly HEPiX Fall 2017 Implementation of CI for Linux Images 5
Testing Out Images Perform some manual checks: Try starting a VM Try logging into it Check Puppet reports Check cloud-init logs A list of tests HEPiX Fall 2017 Implementation of CI for Linux Images 6
We Need To... Automate test execution Perform more thorough tests Be able to manage tests HEPiX Fall 2017 Implementation of CI for Linux Images 7
Towards a Variety of Tests HEPiX Fall 2017 Implementation of CI for Linux Images 8
So Far... Koji CC7 - x86_64 [2017-04-06] SLC6 - x86_64 [2017-04-06] GitLab CI OpenStack koji image-build server create ssh./runtests.sh HEPiX Fall 2017 Implementation of CI for Linux Images 9
A Job to Build an Image Runs koji image-build Once a day check if already built today Downloads the image from Koji Uploads the image into OpenStack HEPiX Fall 2017 Implementation of CI for Linux Images 10
A Job to Create a Test VM Koji CC7 - x86_64 [2017-04-06] SLC6 - x86_64 [2017-04-06] GitLab CI OpenStack koji image-build server create ssh./runtests.sh HEPiX Fall 2017 Implementation of CI for Linux Images 11
A Test VM Purpose: A prerequisite A test in its own right Not much in Docker image before_script: Copy.repo files Install e.g. OpenStack RPMs What it does: Run openstack server create HEPiX Fall 2017 Implementation of CI for Linux Images 12
A Job to Create a Test VM Koji CC7 - x86_64 [2017-04-06] SLC6 - x86_64 [2017-04-06] GitLab CI OpenStack koji image-build server create ssh./runtests.sh HEPiX Fall 2017 Implementation of CI for Linux Images 13
CentOS Functional Tests https://github.com/centos/sig-core-t_functional Utilities (bzip2, curl, grep,... ) System bits (kernel, cron, iptables,... ) Services (MySQL, Apache,... ) Compilers (GCC, JDK,... ) File system (root files,... ) X Window... HEPiX Fall 2017 Implementation of CI for Linux Images 14
Sample Runs HEPiX Fall 2017 Implementation of CI for Linux Images 15
Jobs To Run Each Test before_script: Install SSH Keytab/private key from secret variable Wait until we can SSH SSH and install git Clone CentOS functional tests What they do: SSH and./runtests.sh $TESTNAME HEPiX Fall 2017 Implementation of CI for Linux Images 16
.gitlab-ci.yml job_ template : & job_ template script : - ssh root@ $TESTNODE "./ runtests. sh $CI_ JOB_ NAME " p_gcc : <<: * job_ template stage : development p_git : <<: * job_ template stage : development p_java -1.6.0 - openjdk : <<: * job_ template stage : development HEPiX Fall 2017 Implementation of CI for Linux Images 17
Scaling with Tests Koji CC7 - x86_64 [2017-04-06] SLC6 - x86_64 [2017-04-06] GitLab CI OpenStack koji image-build server create ssh./runtests.sh HEPiX Fall 2017 Implementation of CI for Linux Images 18
Scaling with Tests Koji CC7 - x86_64 [2017-04-06] SLC6 - x86_64 [2017-04-06] GitLab CI OpenStack koji image-build server create ssh./runtests.sh HEPiX Fall 2017 Implementation of CI for Linux Images 18
Scaling with Tests Koji CC7 - x86_64 [2017-04-06] SLC6 - x86_64 [2017-04-06] GitLab CI OpenStack koji image-build server create ssh./runtests.sh HEPiX Fall 2017 Implementation of CI for Linux Images 18
Which Granularity? A fresh VM for each CI job Problem of destructive tests A fresh VM for each test? Really? HEPiX Fall 2017 Implementation of CI for Linux Images 19
Test VM Pool Workflows...... to schedule test VMs in OpenStack project... to manage VM life cycle CI Runner VM instead? But special privileges. HEPiX Fall 2017 Implementation of CI for Linux Images 20
Interesting Tests Access to AFS, EOS, CVMFS,... RPM health collectd Attaching Cinder volumes Performance HEPiX Fall 2017 Implementation of CI for Linux Images 21
A Wider Outreach HEPiX Fall 2017 Implementation of CI for Linux Images 22
Allow End-Users to...... run their own tests...... against their own images HEPiX Fall 2017 Implementation of CI for Linux Images 23
Allow End-Users to Run Their Tests Koji CC7 - x86_64 [2017-04-06] SLC6 - x86_64 [2017-04-06] GitLab CI OpenStack koji image-build server create ssh./runtests.sh HEPiX Fall 2017 Implementation of CI for Linux Images 24
Allow End-Users to Run Their Tests Koji CC7 - x86_64 [2017-04-06] SLC6 - x86_64 [2017-04-06] User Repository GitLab CI koji image-build OpenStack Merge Request server create ssh./runtests.sh HEPiX Fall 2017 Implementation of CI for Linux Images 24
User Interface Merge requests either way From a CLI? imageci run ' CC7 - x86_64 [2017-09 -20] ' mysql Directly from their git repository? HEPiX Fall 2017 Implementation of CI for Linux Images 25
Querying Results GitLab API Clear views From CLI? imageci results ' CC7 - x86_64 [2017-09 -20] ' mysql Or simply... HEPiX Fall 2017 Implementation of CI for Linux Images 26
Sample Runs HEPiX Fall 2017 Implementation of CI for Linux Images 27
Concerns and Opportunities Jobs running for too long How many jobs can we run? Work to adapt CentOS functional tests Testing Docker images HEPiX Fall 2017 Implementation of CI for Linux Images 28
home.cern