VMs at a Tier-1 site EGEE 09, 21-09-2009 Sander Klous, Nikhef
Contents Introduction Who are we? Motivation Why are we interested in VMs? What are we going to do with VMs? Status How do we approach this issue? Where do we stand? Challenges 03-09-2009 BIG Grid - Virtualization working group 2
Introduction Collaboration between NCF: national computing facilities Nikhef: national institute for subatomic physics NBIC: national bioinformatics center Participation from Philips, SARA, etc. Goal: Enables access to grid infrastructures for scientific research in the Netherlands 03-09-2009 BIG Grid - Virtualization working group 3
Motivation: Why Virtual Machines? Site perspective Resource flexibility (e.g. SL4 / SL5) Resource management Scheduling / multi-core / sandboxing User perspective Isolation from environment Identical environment on multiple sites Identical environment on local machine 03-09-2009 BIG Grid - Virtualization working group 4
Different VM classes Class 1: Site generated Virtual Machines No additional trust issues Benefits for system administration Class 2: Certified Virtual Machines Inspection and certification to establish trust Requirements for monitoring / integration Class 3: User generated Virtual Machines No trust relation Requires appropriate security measures 03-09-2009 BIG Grid - Virtualization working group 5
Typical use case Class 1 VM Resource management Torque/PBS Job queue Virtual Machine Manager VM queue Box 1 Normal WN Box 3 8 Virtual SL5 WNs Box 2 8 Virtual SL4 WNs Site infrastructure 03-09-2009 BIG Grid - Virtualization working group 6
Typical use case Class 2 VM Analysis on Virtual Machines Run minimal analysis on desktop/laptop Access to grid services Run full analysis on the grid Identical environment Identical access to grid services No interest to become system administrator Standard experiment software is sufficient 03-09-2009 BIG Grid - Virtualization working group 7
Typical use case Class 3 VM Identification and classification of GPCRs Requires very specific software set Blast 2.2.16 HMMER 2.3.2 BioPython1.50 Even non-x86 (binary) applications! Specific software for this user No common experiment software 03-09-2009 BIG Grid - Virtualization working group 8
Project status Working group: virtualization of worker nodes https://wiki.nbic.nl/index.php/biggrid_virtualisatie Kick-off meeting July 6 th 2009 System administrators, User support, management Phase 1 (3 months) Collect site and user requirements Identify other ongoing efforts in Europe First design Phase 2 (3 months) Design and implement proof of concept 03-09-2009 BIG Grid - Virtualization working group 9
Active working group topics Policies/Security issues for Class 2/3 VMs Technology study Managing Virtual Machines Distributing VM images Interfacing the VM infrastructure with the grid Identify missing functionality and alternatives Accounting and fare share, image management, authentication/authorization, etc. 03-09-2009 BIG Grid - Virtualization working group 10
The Amazon identity crisis The three most confronting questions: 1. What is the difference between a job and a VM? 2. Why can I do it at Amazon, but not at the grid? 3. What is the added value of grids over clouds? We don t want to compete with Amazon! 03-09-2009 BIG Grid - Virtualization working group 11
Policy and security issues E-science services and functionality Data integrity, confidentiality and privacy Non-repudiation of user actions System administrator point of view Trust user intentions, not their implementations Incident response more costly than certification Forensics is time consuming 03-09-2009 BIG Grid - Virtualization working group 12
Security 101 = Attack surface Compromised user space is often already enough trouble 03-09-2009 BIG Grid - Virtualization working group 13
Available policies Grid Security Policy, version 5.7a VO Portal Policy, version 1.0 (draft) Big Grid Security Policy, version 2009-025 Grid Acceptable Use Policy, version 3.1 Grid Site Operations Policy, version 1.4a LCG/EGEE Incident Handling and Response Guide, version 2.1 Grid Security Traceability and Logging Policy, version 2.0 VO-Box Security Recommendations and Questionnaire, version 0.6 (draft, not ratified) 03-09-2009 BIG Grid - Virtualization working group 14
Relevant policy statements Network security is covered by site local security policies and practices A VO Box is part of the trusted network fabric. Privileged access is limited to resource administrators Software deployed in the grid must include sufficient and relevant site central logging. 03-09-2009 BIG Grid - Virtualization working group 15
First compromise Certified package repository Base templates Certified packages Separate user disk User specific stuff Permanent storage At run time No privileged access Comparable to VO box Licenses? 03-09-2009 BIG Grid - Virtualization working group 16
Second compromise Make separate grid DMZ for Class 3 VMs Comparable to Guest networks Only outbound connectivity Detection of compromised guests Extended security monitoring Packet inspection, netflows (SNORT, nfsen) Honeypots, etc. Simple policy: one warning, you re out. Needs approval (network policy) from OST (Operations Steering Team) 03-09-2009 BIG Grid - Virtualization working group 17
TECHNOLOGY STUDY 03-09-2009 BIG Grid - Virtualization working 18
Managing VMs Resource management Torque/PBS Job queue Haizea OpenNebula VM queue Box 1 Normal WN Box 3 8 Class 2/3 VMs Box 2 8 Virtual WNs Site 03-09-2009 BIG Grid - Virtualization working group 19
Distributing VM images iscsi/lvm Box 1 Normal WN Repository (SAN) Image Image Image Image Image Class 2/3 upload solution Box 2 8 Virtual WNs Box 3 8 Class 2/3 VMs 03-09-2009 BIG Grid - Virtualization working group 20
Cached copy-on-write Repository Box 1 COW Cache Image Image VM COW VM Box 2 VM COW Image VM COW Cache 03-09-2009 BIG Grid - Virtualization working group 21
Interfacing VMs with the grid Repository (SAN) Image Image Image Image Image Class 2/3 upload solution Class 2 Class 3 discussion Grid middleware globus-job-run globus-gatekeeper globus-job-manager contact-string jm-pbs-long jm-opennebula qsub / opennebula Resource management Torque/PBS Nimbus/OCCI OpenNebula 03-09-2009 BIG Grid - Virtualization working group 22
VM contact-string User management mapping Coffee table discussion Mapping to OpenNebula users Authentication / Authorization Access to different VM images Grid middleware components involved: Cream-CE, BLAHp, glexec Execution Environment Service https://edms.cern.ch/document/1018216/1 Authorization Service Design https://edms.cern.ch/document/944192/1 Parameter passing issue 03-09-2009 BIG Grid - Virtualization working group 23
Monitoring/Performance testing 03-09-2009 BIG Grid - Virtualization working group 24
Performance Small cluster 4 dual CPU quad core machines Image server with 2 TB storage Integration with experimental testbed Existing Cream-CE / Torque Testing Network I/O, is NAT feasible? File I/O, what is the COW overhead? Realistic jobs 03-09-2009 BIG Grid - Virtualization working group 25
Other challenges Accounting, scheduling based on Fair Share Scalability! Rapidly changing landscape New projects every week New versions every month So many alternatives VMWare, SGE, Eucalyptus, Enomaly iscsi, NFS, GFS, Hadoop Monitoring and security tools 03-09-2009 BIG Grid - Virtualization working group 26
Conclusions Maintainability: no home grown scripting Each solution should be part of a product Validation procedure with each upgrade Deployment Gradually move VM functionality in production 1. Introduce VM worker nodes 2. Virtual machine endpoint in grid middleware 3. Test with a few specific Class 2/3 VMs 4. Scaling and performance tuning 03-09-2009 BIG Grid - Virtualization working group 27