Application of Virtualization Technologies & CernVM. Benedikt Hegner CERN

Transcription

1 Application of Virtualization Technologies & CernVM Benedikt Hegner CERN

2 Virtualization Use Cases Worker Node Virtualization Software Testing Training Platform Software Deployment }Covered today Server consolidation Cloud Computing Volunteer Computing Data Preserveration

3 Worker Node Virtualization Helps in the management of computing centres Decoupling of jobs and physical resources Eases management of batch farm resources Enables computing centres to move to new computing models more easily Examples: CERN virtual batch system, CNAF worker nodes on demand

4 Software Testing Virtual machines can cut time and money out of the software development and testing process Great opportunity to test software in a large variety of platforms Each platform can be realized by a differently configured virtual machines Easy to duplicate same environment in several virtual machines Testing installation procedures from well defined state Etc. Example: Execution Infrastructure in ETICS (spin-off of the EGEE project) Set of virtual machines that run a variety of platforms attached to an Execution Engine where Build and Test Jobs are executed on behalf of the submitting users

5 Training Platform Similar as for software testing infrastructure, virtualization helps to deploy rapidly dedicated software and workstations/servers for training Need for many nodes rapidly and typically for a rather short period of time Isolation with respect production servers Disposable workstations/servers The resources for today s hands-on are set up following this idea

6 Software Deployment Problem LHC Millions of lines of code Different packaging and software distribution models Complicated software installation/update/configuration procedure Long and slow validation and certification process Very difficult to roll out major OS upgrade (SLC4 -> SLC5) Additional constraints imposed by the grid middleware development Effectively locked on one Linux flavour Whole process is focused on middleware and not on applications How to effectively harvest multi and many core CPU power of user laptops/desktops if LHC applications cannot run in such environment? Good news: We are not the only one with such problems

7 Horizontal Integration Application Libraries Traditional model Horizontal layers Independently developed Maintained by the different groups Different lifecycle Tools Databases OS Application is deployed on top of the stack Breaks if any layer changes Needs to be certified every time when something changes Results in deployment and support nightmare Hardware

8 Vertical Integration Virtual Machine Application Libraries Tools Databases OS Application driven approach Analyzing application requirements and dependencies Adding required tools and libraries Building minimal OS Bundling all this into Virtual Machine image Virtual Machine images should be versioned just like the applications Assuring accountability to mitigate possible negative aspects of newly acquired application freedom Problem in HEP The Application is rather huge (10GB) and changes once per week per experiment Overloads the deployment infrastructure at the sites

9 The CernVM project CernVM is a R&D project started 3 years ago on Virtualization at CERN The CernVM image is an attempt to mitigate the standard difficulties of VMs (performance, image distribution, trust, contextualization, etc.) Tuned for best performance of HEP applications Single image fits all [LHC] experiments Very small is size (only 250MB) with just-enough OS Experiment software is factorized out (dedicated File System) Flexible configuration and contextualization mechanisms

10 The CernVM filesystem } Experiment software is changing frequently and we want to avoid need to frequently update, certify and redistribute VM images with every release } Only a small fraction of software release is really used } Demonstrated scalability and reliability } Now being deployed on across all Grid sites as the channel for software distributions

11 Application Software Delivery CernVM comes with the read-only file system (CernVM-FS) optimized for software distribution Very little fraction of the experiment software is actually used (~10%) Very aggressive local caching, web proxy cache (squids) Transparent file compression Integrity checks using checksums, signed file catalog Operational in off-line mode No need to install any experiment software Virtually all versions of all applications are already installed The user just needs to start using it to trigger the download CernVM-FS can be used outside the CernVM context You can use it as well on a standard Scientific Linux 5/6 installation Removes the load of software deployment at your site

12 Stratum Model + Fast and Scalable + No single point of failure - Complex hierarchy Content Distribution

13 Client-Side Fail-Over Proxies SL5 Squid, load-balancing + fail-over e. g. CVMFS_HTTP_PROXY="A B C" Mirrors Fail-over mirrors at CERN, RAL, BNL For roaming users automatic ordering based on RTT

14 Client-Side Fail-Over Proxies SL5 Squid, load-balancing + fail-over e. g. CVMFS_HTTP_PROXY="A B C" Mirrors Fail-over mirrors at CERN, RAL, BNL For roaming users automatic ordering based on RTT

15 Publishing Releases 2. Publishing is an atomic operation 1. Each experiment is given a VM to install and test their software using own installation tools

16 That s all :-)

17 Hands-On Session Install Squid to serve the software on one of the provided VMs see exercise for storage Install the CernVM-FS fuse-module on another of the provided VMs Chapter 2 of Let the installed fuse module point to the already set up proxy If time allows, try out of the box CernVM images on your local machine For more hints for a local installation at your institute