HPC learning using Cloud infrastructure Florin MANAILA IT Architect florin.manaila@ro.ibm.com Cluj-Napoca 16 March, 2010
Agenda 1. Leveraging Cloud model 2. HPC on Cloud 3. Recent projects - FutureGRID 4. Open research problems 5. Conclusions
1. Leveraging Cloud model
Cloud support beneficial to HPC Application portability Image management Simulate scaling VM migration Better resource utilization Rapid provisioning
Application portability Users maintains their own virtualized OS Linux, Windows, Solaris,... Isolation: No conflict: OS level, libraries, software,... Concurrent mix of different environments on same physical server Subnetwork: cluster of servers
Image management Capture an HPC environment Create & customize your own Vendor supplied software stack Usage Save, restore Share, publish Archive legacy environment Possible in any storage configuration LVM, SAN LUN, VMWare
Simulate scaling Develop & debug large scale execution Time consuming, system not productive Result and execution time not a factor Shared cluster: more efficient usage: don't need to tie up resource when ready, execute on dedicated cluster
VM migration Fault tolerance Detect pending failure: disk error, high temperature, communication error,... Migrate VM to healthy server Load balancing High end servers server 1 VM server 2 Requirements SAN, separate network for migration Our observation: 300-400msec down time SAN
Better resource utilization Consolidate multiple VM's on same node Multicore CPU in high end server idle time due to disk I/O, blocking send/receive Network bandwidth Applications have different communication requirement Tradeoff: Flexibility in job time More users, full system utilization
Rapid provisioning New VM in about 2-3 minutes No work by IT team: Cluster created automatically when requested, freed up automatically when done With self service portal: Change thinking about IT infrastructure Enable more experimentation Vehicle for teaching
Concerns about Cloud Overhead Virtualization cost Computation, memory, disk I/O, networking Reliable performance Resource sharing in VM's Communication capacity
2. HPC on Cloud
Typical use cases HPC environment resources dedicated (physical) applications tuned to environment batch scheduled: MOAB + xcat Public Cloud resources virtualized and shared personalized environment run continously, create/start/stop by user
Developing & running HPC applications Dedicated cluster Virtualized environment Custom scheduler Grid model Install grid management software Launch applications Map Reduce Model Install Hadoop or use ready made template Launch applications
Map Reduce, a new computation model Model developed by Google 1. map: transmute input to (key, value) pair 2. reduce: reduce set of (key, value) into one Hadoop runtime developed by Yahoo Distribution Redundant computation: fault tolerant File system for distributed data Simple distributed computing model Applicable for many applications: log processing, Web index building
Word Count Dataflow (Hairong Kuang, Yahoo)
Word Count Example (Hairong Kuang, Yahoo) Mapper Input: value: lines of text of input Output: key: word, value: 1 Reducer Input: key: word, value: set of counts Output: key: word, value: sum Launching program Defines the job Submits job to cluster
Current Hadoop projects http://wiki.apache.org/hadoop/poweredby Yahoo: More than 25,000 computers running Hadoop (100,000 CPUs) Largest cluster: 4,000 nodes (2*4cpu boxes, 4*1TB disk & 16GB RAM) Support research for Ad Systems and Web Search Scaling tests to support development of Hadoop on larger clusters
3. Recent projects: FutureGRID
FutureGrid The goal of FutureGrid is to support the research on the future of distributed, grid, and cloud computing. FutureGrid will build a robustly managed simulation environment or testbed to support the development and early use in science of new technologies at all levels of the software stack: from networking to middleware to scientific applications. The environment will mimic TeraGrid and/or general parallel and distributed systems FutureGrid is part of TeraGridand one of two experimental TeraGrid systems (other is GPU) This test-bed will succeed if it enables major advances in science and engineering through collaborative development of science applications and related software. FutureGrid is a (small 5600 core)science/computer Science Cloud but it is more accurately a virtual machine based simulation environment
Future Grid FutureGrid Hardware
FutureGrid Usage Scenarios Developers of end-user applications who want to develop new applications in cloud or grid environments, including analogs of commercial cloud environments such as Amazon or Google. Is a Science Cloud for me? Is my application secure? Developers of end-user applications who want to experiment with multiple hardware environments. Grid/Cloud middleware developers who want to evaluate new versions of middleware or new systems. Networking researchers who want to test and compare different networking solutions in support of grid and cloud applications and middleware. (Some types of networking research will likely best be done via through the GENI program.) Education as well as research Interest in performance requires that bare metal important
Education and Training Importance of experimental work in contemporary distributed systems research Needs also to be addressed in education Complement to fundamental theory FutureGrid: a testbed for experimentation and collaboration around new architectures Education and training key to: Enable new users to quickly get started Enable students to experiment with FutureGrid technologies from core to bleeding-edge Foster dissemination of FutureGrid architectures
Goals and Approach A flexible, extensible platform for hands-on, lab-oriented education on FutureGrid Focus on usability- lowering barriers to entry Plug and play, open-source Apply virtualization and social networking technologies to create educational sandboxes Virtual Grid appliances: self-contained, prepackaged execution environments Group VPNs: simple management of virtual clusters by students and educators
Background Virtual appliances Encapsulate software environment in an image Virtual disk file(s) and virtual hardware configuration Grid appliance at UF Encapsulates cluster software environments Current examples: Condor, MPI, Hadoop Homogeneous images at each node Virtual LAN connecting nodes to form a cluster Deploy within or across domains
Appliance Deployments PlanetLab overlay: ~450 nodes, 24/7 on a shared infrastructure Archer cluster ramp-up: UFL, NEU, UMN, UTA, FSU
Appliance interface VM Hardware configuration User files Domain tools Linux + IPOP+ Condor
Connecting virtual nodes The virtual LAN is a self-configuring VPN IP-over-P2P overlay network IPOP flexibility, usability in the setup and management of a virtual cluster Users can configure and manage their own VPN groups using simple interfaces GroupVPN:all-to-all connectivity within group; membership managed by group owner Web interface: Grid appliance; FutureGrid, or deploy your own (WebUI appliance ongoing work)
GroupVPN Example: Archer 1: Download appliance 2. Create/join VPN group Download config Free pre-packaged Archer Virtual appliances - run on free VMMs (VMware, VirtualBox, KVM) CMS, Wiki, YouTube: Community-contributed content: applications, datasets, tutorials Archer Global Virtual Network Archer seed resources 450 cores, 5 sites 3. Boot appliances Automatic connection to group VPN self-configuring DHCP Middleware: Condor scheduler NFS file systems
References/Contact FutureGrid http://www.futuregrid.org Grid appliances used in FutureGrid education http://www.grid-appliance.org HPC Group Europe http://www.hpc-g.eu.org
4. Open research problems
Open research related to HPC Networking latency/bandwidth reduce overhead, better sharing Scheduling gang scheduling for VM's Placement algorithm optimize for power consumption, utilization Migrate applications to new computation model: Map Reduce
Research on networking Virtualization support in advanced NIC e.g. Infiniband Shift virtualization from software to firmware Reduce latency 50% and double throughput Virtual machines can support HPC applications I/O requirement without impact
5. Conclusions
Some successful educational Cloud Google Cloud: Built by IBM HiPODS team, supported by NSF for university research University-based Cloud: Virtual Computing Lab, North Carolina State University Collaboration with IBM University Relations Apache open source Lightweight infrastructure, effective for teaching purpose
Current state Concerns and perception Virtualization overhead not significant Relatively reliable performance can be achieved Many alternatives to organize Cloud + HPC Continuing research, optimization
Challenge of HPC administration Dynamic computing power/platform request Maintenance cost increase: staff, monitoring, management, etc Flexible infrastructure request Need dynamic & flexible system to meet user faster!! Need improve computing effectiveness!! Need a shared & simple management system!!
Compelling arguments for Cloud Advantages: Application portability Image management Simulate scaling VM migration Better resource utilization Rapid provisioning Impact: Easier to use and manage for both users and administrator Interesting new research areas
Thank you! 39 Cloud Computing 6/30/2009