Going Places with Distributed Computing

Transcription

1 Going Places with Distributed Computing ESC Grid Workshop 2007 Kate Keahey University of Chicago Argonne National Laboratory

2 There is more to Grid applications than running simple batch computations

3 Galaxies Collide on the I-WAY I-WAY (SC95) Colliding Galaxies Co-scheduling components Tightly coupled simulation distributed over NCSA, PSC, SDSC Multiple components Dark matter Gas Visualization We never managed to do this again

4 Functional MRI (fmri) Large datasets 90,000 volumes / study 100s of studies Wide range of analyses Testing, production runs Data mining Ensemble, Parameter studies

5 Moral Hazard 11/29/07, ESAC Grid Workshop Kate Keahey

6 A Different Kind of Complexity Complex experimental application codes Developed over more than 10 years, by more than 100 scientists, comprises ~2 M lines of C++ and Fortran code Require complex, customized environments Rely heavily on the right combination of compiler versions and available libraries Dynamically load external libraries depending on the task to be performed Environment validation To ensure reproducibility and result uniformity across environments

7 Critical Applications The National Fusion Collaboratory: Grids for Experimental Science 11/29/07, ESAC Grid Workshop Kate Keahey

8 Urgent Resource Requirements Applications that require resources in response to urgent needs Event tracking, simulation and prediction, etc. Unexpected

9 Combating Complexity

10 The Swift System Clean separation of logical/physical concerns Logical Concerns XDTM specification of logical data structures Concise specification of parallel programs (SwiftScript, with iteration, etc.) Rigorous provenance tracking and query: Virtual data schema & automated recording Physical Concerns Efficient execution on distributed resources: Karajan threading, Falkon provisioning, Globus interfaces, pipelining, load balancing Improved usability and productivity Demonstrated in numerous applications

11 Swift Architecture Specification Scheduling Execution Provisioning Abstract computation Execution Engine (Karajan w/ Swift Runtime) Virtual Node(s) file1 Resource Provisioners Falkon/VWS SwiftScript Compiler Virtual Data Catalog SwiftScript C C C Provenance collector C Swift runtime callouts Status reporting Provenance data launcher Provenance data launcher App F1 file2 App F2 file3 Amazon EC2 Yong Zhao, Mihael Hatigan, Ioan Raicu, Mike Wilde, Ben Clifford

12 Provisioning Resources Condor Glideins Provision resources for a Condor pool by gliding in Condor daemons as jobs through a GRAM interface MyCluster Deploys personal clusters (SGE or Condor) on Teragrid resources. Unlike Condor Glideins, each user gets their own cluster, instead of having resources added to an existing pool Falkon Deploys a daemon that is specially optimized for scheduling lightweight tasks, such as those found in workflows.

13 The Underpinnings: Resource Provisioning as Central Abstraction for Grid Computing

14 Of Jobs and Resources Job management vs resource provisioning The focus of infrastructure available today it to schedule jobs Resources are provisioned as a side-effect of job deployment We need general-purpose leases Associated with the required environment With a well-defined resource quota and availability General-purpose Adaptable to many interactions: managed by customized schedulers, log in via ssh, etc. Building on the side-effect Request a resource quota Once obtained, prepare the environment and make it available via some required method

15 Two Things to Consider Lease Terms Expressive enough so that we can define a variety of leases Best-effort, advance reservations (including immediate leases), periodic, preemptible/non-preemptible, etc. A description of the required resources: leasing CPUs, storage An environment Making lease terms explicit Standards (OGF): WS-Agreement, JSDL Enforcement Corresponding to lease terms Enforcing a large set of the desired features Enforcing them efficiently

16 Virtual Workspaces A dynamically provisioned environment Environment definition: we get exactly the (software) environment we need on demand. Resource allocation: Provision the resources the workspace needs (CPUs, memory, disk, bandwidth, availability), allowing for dynamic renegotiation to reflect changing requirements and conditions. Implementation Traditional means: publishing, automated configuration, coarse-grained enforcement Virtual Machines: encapsulated configuration and fine-grained enforcement Paper: Virtual Workspaces: Achieving Quality of Service and Quality of Life in the Grid

17 The Virtues of Virtualization App App Guest OS (Linux) App Guest OS (NetBSD) Virtual Machine Monitor (VMM) / Hypervisor Hardware App Guest OS (Windows) VM VM VM App Parallels Xen VMWare UML KVM Bring your environment with you Excellent enforcement and isolation Fast to deploy, enables short-term leasing etc. Have a performance impact but it is acceptable for most modern hypervisors Suspend/resume, migration

18 Deploying Workspaces Remotely VWS Service Workspace -Workspace metadata -Pointer to the image -Logistics information -Deployment request -CPU, memory, count, etc. Virtual Workspace implementation

19 Interacting with Workspaces The workspace service publishes information on each workspace as standard WSRF Resource Properties. Users can query those properties to find out information about their workspace (e.g. what IP the workspace was bound to) VWS Service Users can interact directly with their workspaces the same way the would with a physical machine. Trusted Computing Base (TCB)

20 Cloud Computing

21 Virtual Workspaces for STAR STAR image configuration A virtual cluster composed of an OSG head and STAR worker s Using the workspace service over EC2 to provision resources Allocations of up to 100 s Dynamically contextualized for out-of-the-box cluster

22 with thanks to Jerome Lauret and Doug Olson of the STAR project Running Running Runningjobs jobs jobs:::: Running jobs 230 VWS/EC2 Running Running Runningjobs jobs jobs:::: Running jobs 300 PDSF WSU Running Running Runningjobs jobs jobs:::: Running jobs 150 Fermi Job Completion : 11/29/07, ESAC Grid Workshop BNL File Recovery : Kate Keahey Running Runningjobs jobs::: Running jobs 50

23 with thanks to Jerome Lauret and Doug Olson of the STAR project with thanks to Jerome Lauret and Doug Olson of the STAR project Nersc PDSF EC2 (via Workspace Service) WSU Accelerated display of a workflow job state Y = job number, X = job state

24 Can I Do It at Home? Challenge: how can I provide a cloud using virtualization without disrupting the current operation of my cluster? Flying Low: the Workspace Pilot Integrates with popular LRMs (such as PBS) Implements best effort leases Glidein approach: submits a pilot program that claims a resource slot Provides administrator tools Deployed on UC TeraPort

25 Workspace Pilot in Action Level 2: provision VMs Level 1: provision raw resources VWS VM VM Xen dom0 VM VM LRM/PBS Xen dom0 Xen dom0

26 Division of Labor: Decoupling Leasing and Platform Management

27 A Word from the Expert The greatest improvements in the productive powers of labour, and the greater part of the skill, dexterity, and judgment with which it is anywhere directed, or applied, seem to have been the effects of the division of labour. (Adam Smith) 11/29/07, ESAC Grid Workshop Kate Keahey

28 What Does It Mean to Provide Resources? Environment Environment Hardware Software management packages: e.g., COD, Bcfg, Pacman (requires privilege or can deal only with upper layers of software) Hardware Virtual Appliance VMM Hardware Providing isolation via e.g. Xen, KVM, Vmware, Vserver (different levels of encapsulation)

29 Where Do Appliances Come From? Marketplaces (VMWare, EC2, Workspace ) Appliance Provider (a user, a VO, a Grid ) appliance description Good but: ease-of-use? maintenance? all those formats?

30 Where Do Appliances Come From? Marketplaces (VMWare, EC2, Workspace ) Appliance Management Software (OSFarm, rpath, CohesiveFT )) appliance description Xen VMware CDROM Appliance Provider (a user, a VO, a Grid ) Better!

31 Deploying Appliances Appliances are portable It can be reused and customized to many contexts Making the appliance contextaware: Other appliances Site-specific information (e.g. a DNS server) User/group/VO/Grid-specific information (e.g. public keys, host certs, gridmapfiles, etc.) Security issues Who do I trust to provide legitimate context information? How do I make sure that appliances adhere to my site policies? VM VM VM VM site Virtual Organization

32 Where Do Appliances Come From? Marketplaces (VMWare, EC2, Workspace ) Appliance Management Software (OSFarm, rpath, CohesiveFT )) appliance description Xen VMware CDROM appliance assertions appliance contextualization Appliance Provider (a user, a VO, a Grid )

33 Contextualization Challenge: Putting a VM in the deployment context of the Grid, site, and other VMs Assigning and sharing IP addresses, name resolution, application-level configuration, etc. Solution: Management of Common Context Common Context IP hostname pk contextualization agent Configuration-dependent provides&requires Common understanding between the image vendor and deployer Mechanisms for securely delivering the required information to images across different implementations Paper: A Scalable Approach To Deploying And Managing Appliances, TeraGrid conference 2007

34 Workspace Ecosystem Appliance Providers: Virtual Organizations, groups or individual users via OSFarm, rpath, CohesiveFT, bcfg2, etc. Resource Providers: Local clusters, Grid resource providers (TeraGrid, OSG) Commercial providers: EC2, Sun, slicehost, Provisioning a resource, not a platform Middleware: appliances --> resources manage secure appliance deployment Combining networks and storage VWS EC2

35 Making Leases Cost-Effective

36 What Can We Afford? Enforceable == cost-effective Can we afford to provide desirable features? advance reservation semantics, preemption for urgent computing, renegotiation, etc. Typically utilization is a problem: draining Preemption could help but has issues Porting to use specific software required VMs to the rescue Suspend/resume, migration Policies, features, etc. Thesis work by Borja Sotomayor

37 The Draining Problem Node 3 Node 2 Node 1 SHORT-TERM LEASE With pre-emption Node 3 Node 2 Node 1 SHORT-TERM LEASE Without pre-emption

38 Combining Leasing and Job Management Lease Requests Job Requests Best-Effort Lease Requests Lease Manager Events Execution Manager Manages the VMs (image transfers, start, stop, suspend, ( resume Manages the applications running inside the VMs VMM-enabled Worker Nodes

39 Interleaving Soft and Hard Leases Not using VMs (even with backfilling) results in a noticeable hit on runtime. In this case, the scheduler cannot readily start large parallel jobs because of the resource leases. With VMs, these can be started, and suspended before the leases start. Injected leases are short (1h-2h), very frequent (every 4 to 8 hours), large ( cluster (number of s between 1/3 and ½ of the

40 Conclusions Grid applications are demanding Complex workflows, multi-component applications, complex environment requirements, critical execution needs, etc. Higher-level languages to manage locationindependent computing Resource leasing as a central abstraction of Grid computing Enables acquisition of required resources SLAs: making relationships very explicit Short-term leasing an enabling element Environment management: level the playing field for many communities Users are voting with their feet

41 Conclusions (cntd) Towards centralized resource leasing Economies of scale Load-balancing demand Division of labor Appliance providers Resource providers Leveraging diversity of leases -> economy of scale

42 Credits Workspace team: Tim Freeman, Borja Sotomayor Guest appearances Ian Foster, Frank Siebenlist With thanks to many collaborators: Jerome Lauret (STAR, BNL), Doug Olson (STAR, LBNL), Marty Wesley (rpath), Stu Gott (rpath), Ken Van Dine (rpath), Predrag Buncic (Alice, CERN), Haavard Bjerke (CERN), Rick Bradshaw (Bcfg2, ANL), Narayan Desai (Bcfg2, ANL), Duncan Penfold-Brown (Atlas,uvic), Ian Gable (Atlas, uvic), David Grundy (Atlas, uvic), Ti Leggit (University of Chicago), Greg Cross (University of Chicago), Mike Papka (University of Chicago/ANL)