University College London. Department of Computer Science

Transcription

1 Aspects of a Processing Grid Peter T. Kirstein P.Kirsteincs.ucl.ac.uk Søren-Aksel Sørensen S.Sorensencs.ucl.ac.uk Stefano A. Street S.Streetcs.ucl.ac.uk Sheng Jiang S.Jiangcs.ucl.ac.uk University College London 1

2 Processing scenarios Batch processing Large number of small (10 s mins) jobs Typical for Bioinformatics and HEP. Resource hungry but not fussy. Large data sets. Interactive experimentation Single distributed job User controlled (haptic steering) Dynamic resource requirements Performance sensitive (latency) Visualization and steering 2

3 Cluster computing Switch Advantages Cost effective solution to resource shortage Simple security (isolated environment) Easy management. Problems Switching cost (Ethernet) Task mixing (delay vs. throughput) Resource management (time sharing, performance protection) 3

4 Cluster of clusters Large systems can be created from a hierarchy of sub-clusters. The UCL cluster uses this architecture Gigabit switch Switch Switch Switch Still growth problems (eg. power and cooling) Utilisation problems 4

5 Multi-domain clusters Internet Problems Security Load balancing Resource discovery 5

6 6

7 7

8 Application User interface via DAS Web interface UCL Web interface IC Web interface EBI RDB of annotation UCL RDB of annotation IC RDB of annotation EBI Transparent Processing System CPUs UCL-CS CPUs Biochem Resource Management under GLOBUS CPUs IC CPUs EBI 8

9 Distributed GenTHREADER Configuration User annotation submission committed via Web Form Web service filters & divides sequence file into sub jobs Web-based Submission Interface Grid Distribution Manager Globus GenTREADER GenTREADER SGE SGE GenTREADER SGE GenTREADER GenTREADER GenTREADER 9

10 eprotein current production setup JANet Front End IC SGE GT2 IPv4 UCL BioChem Front End UCL Backbone Front End UCL CS 10

11 eprotein future setup EBI Front End JANet Front End IC SGE GT3 IPv6 VPFS UCL BioChem Front End UCL Backbone Front End UCL CS 11

12 IPv6 Advantages Larger Address Space Mobility Support Built-in Security Hooks for QoS Better Auto-configuration End-to-end Addressing Deploys New Technologies 12

13 Grid over IPv6 Full availability of IPv6 will allow provision of better Grid services Mobility support in distributed networks or globalscale networks Performance potentially much better Auto-configuration for resource discovery Allow separation of addressing and routing Better group communication Better security 13

14 Demonstrator Setup Intel Linux (Red Hat ) GT3 (alpha) Java (Sun SDK 1.4.1) IP-independent - Class InetAddress used everywhere Inet4Address Inet6Address Tomcat lightweight edition 14

15 Testing GenTREADER setup GT3 GRAM services Local subnets Gigabit switch Switch Switch Switch 15

16 Interactive use Distributed systems use PVM/MPI2 Dynamic scheduling through dummies Dummy process scheduled by SGE. Extra resource communicated to application. Dummy replaced by application process. Priorities used to favour interactive processes Cluster wide process control. 16