Reduces latency and buffer overhead. Messaging occurs at a speed close to the processors being directly connected. Less error detection

Transcription

1 Switching Operational modes: Store-and-forward: Each switch receives an entire packet before it forwards it onto the next switch - useful in a general purpose network (I.e. a LAN). usually, there is a finite buffer size so it is possible that packets will be dropped under heavy load. Also impose a larger in-switch latency. Can detect errors in the packets Worm hole routing (Also called cut-through switching): Packet is divided into small flits (flow units). Switch examines the first flit (header) which contains the destination address, sets up a circuit and forwards the flit immediately. Subsequent flits of the message are forwarded as they arrive (near wirespeed). Reduces latency and buffer overhead. Messaging occurs at a speed close to the processors being directly connected. Less error detection 1

2 Cluster Network Technologies The performance of (affordable) interconnects has dramatically increased. Cluster interconnects include, among others: Gigabit Ethernet Myrinet Quadrics InfiniBand 2

3 Interconnects in Top500 list 11/2008 3

4 Interconnects in Top500 list 11/2008 4

5 Cluster Network Technologies Gigabit Ethernet: The technology has matured and now offers very good performance at a very low cost. Latency performance is moderate - many Ethernet switches are designed for general LANs (store & forward) where latency reduction is not necessary the primary incentive (the latency is order of ms). Zero-copy OS-bypass message passing can be supported with programmable NIC and direct memory access. 5

6 Cluster Network Technologies Myrinet: using fibre optic cable Uses a fat-tree structure that can accommodate large numbers of nodes. Low latency (7-10 µsec) with a peak bandwidth of 4G bps. Provides zero-copy message passing and can offload packet processing to the NIC. Uses cut-through/worm-hole switching to reduce latency. More expensive than Ethernet (a) Twisted pair cable in Ethernet (b) Fibre optic cable 6

7 Zero copy protocol 7

8 Myrinet Software: Basic OS-Bypass Structure Applications MPI VIA Middleware UDP Host OS IP TCP OS-bypass APIs (multiple host processes) Ethernet Myrinet Myrinet Control Program (MCP) (executes in the Myrinet interface) 10/100/1000 Mb/s Mb/s 8

9 Cluster Network Technologies Quadrics: product of a strategic partnership between Quadrics & Compaq (used in ASCI/Q). Very low latency of 2-5 µsec due to fast interconnects and highly tuned software stack (MPI libraries); bandwidth is about 2Gbps Uses a fat quad-tree arrangement. 9

10 Cluster Network Technologies InfiniBand: by Intel. Basic link speed of 2.5Gb/s. Cut-through/worm-hole switches are used. Current installations are achieving latencies of less than 7 µsec, but this is expected to improve. 10

11 Cluster Workload Management Goal: maximising the delivery of resources to jobs, given job requirements and local policy restrictions Three parties Users: supplying the job requirements Administrators: describing local use policies Workload management software: monitoring the state of the cluster, scheduling the jobs and tracking the resource usage Some or all the following activities are performed Queuing Scheduling Monitoring Resource management Accounting 11

12 Queuing Job submission usually consists of two primary parts: Resource directive (e.g. the amount of memory, the number of CPUs needed) Job description (e.g. job name, the location of the required input files) Once submitted, the jobs are held in the queue until the matching resources are available 12

13 Scheduling Determining at what time a job should be put into execution on which resources There are a variety of metrics to measure scheduling performance System-oriented metrics (e.g. throughput, utilisation, average response time of all jobs) user-oriented metrics (e.g. response time of a job submitted by a user) They can contradicts each other and balance needs to be made 13

14 Monitoring providing information to administrators, users and the scheduling system on the status of jobs and resources the method of collection may differ between different workload management systems, but the general purposes are the same 14

15 Resource management Handling the details of Starting a job under the identity of the user Stopping a job Cleaning up the mess left behind after the job either completes or is aborted Removing or adding resources For the batch system, the jobs are put into execution in such a way that the users need not be present during execution For interactive systems, the users have to be present to supply arguments or information during the execution of the jobs. 15

16 Accounting Accounting for which users are using what resources for how long Collecting resource usage data (e.g. job owner, resources requested by the job, total amount of resources consumed by the job) Accounting data can be used for: Producing system usage and user usage reports Tuning the scheduling policy Calculating future resource allocations Anticipating future resource requirements by users Determining the area of improvement within the cluster 16

17 PBS PBS, Portable Batch System, is a flexible workload management and job scheduling system Originally developed at NASA Different versions of PBS OpenPBS PBSpro Torque (recommended) Three key system demons pbs_server: run in the head node; is the centre of PBS pbs_mom: run in computing nodes; actually place the job into execution pbs_sched: scheduling jobs 17

18 PBS PBS job submission script #!/bin/sh #PBS -l walltime=1:00:00 #PBS -l mem=400mb #PBS -l ncpus=4 cd ${HOME}/PBS/test mpirun -np 4 myprogram Submitting a job % qsub myscriptfile x Inquiring the status of a job % qstat Delete a job %qdel

19 Maui By Maui high-performance computing centre and other partners A job scheduler that can interact with a number of different resource managers (e.g. PBS) Maui is an external scheduler, meaning it does not include a resource manager but rather extends the capabilities of the existing resource managers the underlying resource manager continues to maintain responsibility for managing nodes and tracking jobs Maui uses the APIs of other resource managers (e.g. PBS) to obtain system information Maui controls the decisions of when, where, and how jobs will run 19

20 Schedule Policies The simplest policy: First-Come First-Served Jobs are initiated in the same order as they are submitted. Does not require prior knowledge about tasks (e.g. runtime). Problems: jobs can block other jobs from starting, despite there being no performance benefit to either user. 20

21 First-Come First-Served 21

22 Backfilling The problem with FCFS is that idle time (sum of unused processing intervals) can be significant. One improvement is to backfill. Allows a job to start if it does not delay the first job in the queue. 22

23 Backfilling 23

24 Backfilling Advantages: Utilisation is improved. Disadvantages: Information about the job execution time is required. User estimation are usually inaccurate. It is a policy decision to decide what to do if a job overruns; many administrators choose to terminate a job if it exceeds its allocated execution time otherwise some users may deliberately underestimate the job length to get an earlier job start time. 24

25 Backfilling a problem if predicted runtime is wrong: 25

26 Scheduling Policies Reservation: Increasingly user-based quality of service (QoS) is an important scheduling metric. In addition to normal scheduling, reservation services can be used to plan resource allocation. Users are able to set up a reserved block of processing capability that they are able to use at some point in the future. Task management system agrees to the reservation. Users are subsequently able to run jobs within their reservation quotient. 26