Everything you always wanted to know about SAGA*

Transcription

1 Everything you always wanted to know about SAGA* *but were afraid to ask Thilo Kielmann Shantenu Jha, Andre Merzky 2006 Open Grid Forum

2 Why SAGA? Why are there so few grid applications out there? Is there a simple, stable, integrated and uniform high-level programming interface that provides the most common grid programming abstractions? Need to hide underlying complexities, varying semantics, heterogenities and changes from application program(er) Measure(s) of success: Does SAGA enable quick development of new grid applications? Does it enable greater functionality using less code? 2

3 Copy a File: Globus GASS int copy_file (char const* source, char const* target) { globus_url_t source_url; globus_io_handle_t dest_io_handle; globus_ftp_client_operationattr_t source_ftp_attr; globus_result_t result; globus_gass_transfer_requestattr_t source_gass_attr; globus_gass_copy_attr_t source_gass_copy_attr; globus_gass_copy_handle_t gass_copy_handle; globus_gass_copy_handleattr_t gass_copy_handleattr; globus_ftp_client_handleattr_t ftp_handleattr; globus_io_attr_t io_attr; int output_file = -1; if ( globus_url_parse (source_url, &source_url)!= GLOBUS_SUCCESS ) { printf ("can not parse source_url \"%s\"\n", source_url); return (-1); } if ( source_url.scheme_type!= GLOBUS_URL_SCHEME_GSIFTP && source_url.scheme_type!= GLOBUS_URL_SCHEME_FTP && source_url.scheme_type!= GLOBUS_URL_SCHEME_HTTP && source_url.scheme_type!= GLOBUS_URL_SCHEME_HTTPS ) { printf ("can not copy from %s - wrong prot\n", source_url); return (-1); } globus_gass_copy_handleattr_init (&gass_copy_handleattr); globus_gass_copy_attr_init (&source_gass_copy_attr); if (source_url.scheme_type == GLOBUS_URL_SCHEME_GSIFTP source_url.scheme_type == GLOBUS_URL_SCHEME_FTP ) { globus_ftp_client_operationattr_init (&source_ftp_attr); globus_gass_copy_attr_set_ftp (&source_gass_copy_attr, &source_ftp_attr); } else { globus_gass_transfer_requestattr_init (&source_gass_attr, source_url.scheme); globus_gass_copy_attr_set_gass(&source_gass_copy_attr, &source_gass_attr); } output_file = globus_libc_open ((char*) target, O_WRONLY O_TRUNC O_CREAT, S_IRUSR S_IWUSR S_IRGRP S_IWGRP); if ( output_file == -1 ) { printf ("could not open the file \"%s\"\n", target); return (-1); } /* convert stdout to be a globus_io_handle */ if ( globus_io_file_posix_convert (output_file, 0, &dest_io_handle)!= GLOBUS_SUCCESS) { printf ("Error converting the file handle\n"); return (-1); } globus_ftp_client_handleattr_init (&ftp_handleattr); globus_io_fileattr_init (&io_attr); globus_gass_copy_attr_set_io (&source_gass_copy_attr, &io_attr); &io_attr); globus_gass_copy_handleattr_set_ftp_attr (&gass_copy_handleattr, &ftp_handleattr); globus_gass_copy_handle_init (&gass_copy_handle, &gass_copy_handleattr); 3 result = globus_gass_copy_register_url_to_handle ( &gass_copy_handle, (char*)source_url, &source_gass_copy_attr, &dest_io_handle, my_callback, NULL); if ( result!= GLOBUS_SUCCESS ) { printf ("error: %s\n", globus_object_printable_to_string (globus_error_get (result))); return (-1); } globus_url_destroy (&source_url); return (0); }

4 Copy a File: SAGA #include <string> #include <saga/saga.hpp> void copy_file(std::string source_url, std::string target_url) { try { saga::file f(source_url); f.copy(target_url); } catch (saga::exception const &e) { std::cerr << e.what() << std::endl; } } 4

5 What SAGA is Simple API for Grid Applications: For grid-aware applications: Dealing with the Grid explicitly High-level (= application-level) abstractions Hides details of underlying middleware(s) 5

6 What SAGA is NOT SAGA does NOT hide the grid: It still exposes that resources (like files or jobs) can be remote But it hides all those details that you never wanted to deal with... SAGA is NOT a service (management) interface 6

7 SAGA is client-side software SAGA isn't middleware 7

8 SAGA: In action A SAGA engine can talk to many middlewares at the same time, with dynamic selection/loading 8

9 The SAGA Landscape 9

10 The SAGA Interface Hierarchy Let us first have a quick tour around the API. Then, we will look at specific (small) examples. 10

11 Look&Feel: SAGA object The common root for all SAGA classes. 11

12 Errors and Exceptions SAGA defines a hierarchy of exceptions (and allows implementations to fill in specific details) 12

13 Session, Context, Permissions Only needed if you wish to handle multiple credentials. Otherwise, your default context is used. 13

14 Attributes Key-value pairs, e.g. for resource descriptions 14

15 Application Monitoring/Steering Metric defines application-level data structure(s) that can be monitored and modified (steered). 15

16 Asynchronous Operations, Tasks Most calls can be synchronous, asynchronous, or tasks (need explicit start.) 16

17 Task states 17

18 Jobs Jobs are submitted to run somewhere in the grid. 18

19 Job states Jobs: Tasks: 19

20 Jobs job_service uses job_description to create a job job_description attributes are based on JSDL State model is based on BES job_self represents the SAGA application 20

21 Files, Directories, Name Spaces Both for physical and replicated ( logical ) files 21

22 Streams Simple, data streaming end points. 22

23 Remote Procedure Call A rendering of GridRPC 23

24 Is SAGA Simple? Well, it depends: It is certainly not simple to implement; lots of the pain using the middleware goes into the SAGA engine and adaptors. But it is simple to use (see next slides): Look&Feel vs. Functional Packages Somewhat like MPI: most users only need a very small subset of calls 24

25 File Management saga::directory dir ("any://remote.host.net//data/"); if ( dir.exists ("a") && dir.is_file ("a") ) { dir.copy ("a", "b", Overwrite); } list <string> names = dir.find ("*-[123].txt"); saga::directory tmp = dir.open_dir ("tmp/", Create); saga::file file = dir.open ("tmp/data.txt"); 25

26 Job Submission saga::job_description jd; saga::job_service js ("gram://remote.host.net"); saga::job j = js.create_job (jd); j.run (); cout << "Job State: " << j.get_state () << endl; j.wait (); cout << "Retval " << j.get_attribute ("ExitCode") << endl; 26

27 Jobs (cont.) saga::job j = js.create_job (jd); j.run (); j.suspend (); j.resume (); j.checkpoint (); j.migrate (jd); j.signal (SIGUSR1); j.cancel (); 27

28 Job Migration saga::job self = js.get_self (); self.checkpoint (); self.migrate (jd); self.signal (SIGUSR1); self.cancel (); 28

29 Checking all my Jobs vector<string> ids = js.list (); while ( ids.size () ) { string id = ids[ids.size () - 1]; saga::job j = js.get_job (id); cout << id << " : " << j.get_state () << endl; } ids.pop_back (); 29

30 JSDL-based Job Descriptions saga::job_description jd; jd.set_attribute ("Executable", "/bin/tail"); jd.set_attribute ("Arguments", "-n, 20, -f, all.log"); jd.set_attribute ("Environment", "TMPDIR=/tmp/"); jd.set_attribute ("WorkingDirectory", "data/"); jd.set_attribute ("FileTransfer", "last.log >> all.log"); jd.set_attribute ("Cleanup", "False"); 30

31 SAGA Job Description Attributes 31

32 Name Spaces Management of entities in name spaces Files, replicas, information, resources, steering parameters, checkpoints... Manages hierarchy (mkdir, cd, ls,...) Managed entries are opaque (copy, move, delete...) 32

33 Files Implements name space interface POSIX oriented: read, write, seek Grid (not so simple) optimizations: Scattered I/O Pattern based I/O Extended I/O (a la GridFTP) 33

34 Replicas Implements name space interface, adds name space entries O/REP oriented: list, add, remove replicas, manage meta data Grid optimizations are hidden (replica placement, consistency,...) 34

35 Name Spaces saga::ns_dir dir ("gridftp://remote.host.net//data/"); if ( dir.is_entry ("a") &&! dir.is_dir ("a") ) { dir.copy ("a", "../b"); dir.link ("../b", "a", Overwrite); } list <string> names = dir.find ("*-{123}.text."); saga::ns_dir tmp = dir.open_dir ("tmp/", DeReference); saga::ns_entry entry = dir.open ("tmp/data.txt"); entry.copy ("data.bak", Overwrite); 35

36 Files saga::file f ("gridftp://remote.host.net/data/data.bin"); char buf[100]; if ( f.get_size () >= 223 ) { int pos = f.seek (123, Current); int len = f.read (saga::buffer (buf), 100); } 36

37 Tasks: sync and async operations saga::file file ("gsiftp://remote.host.net/data/data.bin"); // normal, synchronous file.copy ("data.bak"); // async versions saga::task t1 = file.copy <saga::task::sync> ("data.bak.1"); saga::task t2 = file.copy <saga::task::async> ("data.bak.2"); saga::task t3 = file.copy <saga::task::task> ("data.bak.3"); // t1: Done // t2: Running // t3: New 37

38 Tasks: getting results saga::file file ("gsiftp://remote.host.net/data/data.bin"); // normal, synchronous ssize_t size_0 = file.get_size (); // async versions saga::task <ssize_t> t1 = file.get_size <saga::task::sync> (); saga::task <ssize_t> t2 = file.get_size <saga::task::async> (); saga::task <ssize_t> t3 = file.get_size <saga::task::task> (); // wait... ssize_t size_1 = t1.get_result (); ssize_t size_2 = t2.get_result (); ssize_t size_3 = t3.get_result (); 38

39 Upcoming API Extensions MessageBus Structured data transfer, also many-to-many Service Discovery Based on GLUE schema Adverts Persistent storage of application-level data Checkpointing/Recovery Based on GridCPR 39

40 Implementations (late) 2007 LSU/VUA: C++ engine Local, GT4/preWS, OMII-UK GridSAM, XtreemOS LSU: C++ light Local, GT4/preWS, OMII-UK GridSAM LSU: C, Perl, Python, Fortran Wrappers for C++ engine 40

41 Implementations (late) 2007 VUA: Java engine Local, GT2/3/4, ssh, OMII-UK GridSAM, XtreemOS, PBS, SGE, glite(?) (many via Java-GAT) DEISA: Java library DEISA (Unicore) files and jobs NAREGI: Java library NAREGI services 41

42 Other Implementations UVa: Genesis-II adaptors for Java IN2P3 (Lyon) JSAGA (interoperability) Others we are not aware of? 42