Logic Networks on the Grid: Handling 15 Million Jobs Jan Bot, Delft Bioinformatics Lab 07-06-10 Delft University of Technology Challenge the future
Overview Explanation of the application Challenges for the grid Custom grid solution design & implementation More challenges (aka problems) Adding a desktop cluster Errors and statistics Discussion 2
But first... Does anybody not know what these are: Life Science Grid Grid middleware ToPoS ORM (Object Relational Mapper) 3
The application: overview Input data: ~100 mouse tumors 4
Grid pipeline Prepare inputs: prepare data for future grid runs Multiple parameter settings are tested, output of these tests contains the 'real' data Choose best parameter settings, it should still be feasible to do at least 100 permutations Do permutations, 10 permutations per run 5
Run properties Number of jobs per run is fairly large: 24 * 6228 = 149472 Run time is, due to the optimization algorithm, unpredictable: jobs can take anywhere between 2 seconds and 14 hours Outputs are small, both for the real runs and for the permutations 6
Middleware problems Scheduling: This amount of jobs cannot be scheduled using the normal (glite) middleware Overhead of scheduling could out-weight the run time Bookkeeping No method of tracking this amount of jobs Output handling: No grid resource can store large amounts of small files (dcache is not an option) Other solutions (such as ToPoS) are slow when retrieving output 7
Scheduling jobs with ToPoS ToPoS takes care of the first two categories of problems but presents some new challenges: ToPoS does not scale beyond 10.000 jobs per pool No client software which facilitates spreading the tokens over multiple pools 8
Python ToPoS clients To deal with the limitations of ToPoS two clients were implemented: Grid client: uses the most basic Python httplib module can fetch, lock and delete tokens has a generator to transparently handle tokens in multiple pools Local client: uses the more advanced UrlLib2 module can create and delete pools, spread tokens over multiple pools, delete all locks in a pool, gather ToPoS statistics, etc. 9
Dealing with the outputs Outputs: are small and well defined why not just flush them to a database? Proposed solution: Python as language SQLAlchemy as ORM XML-RPC as communication channel MySQL (for now) as database 10
Client design send results Bash script start python Python script Matlab (MCR) call Matlab Set environment variables Fetch input data Make binaries executable Load modules Start python script Loop: Fetch token from ToPoS Call Matlab (MCR) Parse output & send to result server Perform algorithm 11
Application design ORM DB App XML-RPC Clients Design overview Listen loop: listen for incoming calls XML-RPC server Client Fetch token Do work Upload output main DB DB code Flush loop: flush once every minute Thread model 12
Implementation & the weakest link Implemented in Python Hosted on a P4 in a broom closet in our department On power failure: everything collapses (but that's not very likely, right?) 13
Getting ready to run: data replication Getting the data from one (remote) site is expensive Use data replication across all sites to minimize external traffic and divide the load over multiple SRMs Data replication can be done easily with the V-Browser Manual approach: Register file: lcg-cr -l lfn:///grid/lsgrid/jridder/mgtest/mg_perm2_5_datapack.zip MG_Perm2_5_Datapack.zip Replicate file (in this example to nikhef): lcg-rep --vo lsgrid -d tbn18.nikhef.nl srm://gb-setud.ewi.tudelft.nl/dpm/ewi.tudelft.nl/home/lsgrid/generated/2010-05-26/file006bff9b-49ef-46bd80cd-5b8110171557 On a WN, retrieve a local copy: DATAPACK=lfn:/grid/lsgrid/jridder/MGtest/MG_Perm2_5_Datapack.zip echo $VO_LSGRID_DEFAULT_SE TDATA=`lcg-lr --vo lsgrid $DATAPACK grep $VO_LSGRID_DEFAULT_SE` lcg-cp --verbose $TDATA $DATAPACK 14
Adding a desktop cluster Practical (student) pcs are not doing anything at night Use these computers to increase computation power Compute at night & in weekends Our scenario (using ToPoS and an external output server) is ideal for testing such a cluster Use condor to manage the work 15
Desktop cluster locations Two locations Drebbelweg: 250 practical pcs Mekelweg: 50 100 pcs distributed throughout the building Different locations means different vlans: use two condor queues 16
Problems during run Many jobs seemed to quit prematurely while most of them ran fine Errors could be traced back to Deimos and Nikhef The middleware doesn't really provide statistics to the enduser Output files cannot always be retrieved 17
Gathering statistics Add run information (e.g. start & end times) to the job-output Add an additional XML-RPC method to capture error information Uploading error info is easy: Use return status of external program Use Pythons internal error handling capabilities All error messages (of the entire job) are located in one text file 18
Job Running times (1) 19
Job running times (2) One permutation run (10 permutations) takes: 415140369 seconds 115316.77 hours 4804.87 days 13.16 years Now, repeat 9 times (yes, that's a century) 20
Work done per site 21
Nikhef and Deimos mortality 22
Gathering error info Gathering error information on the grid is prone to error Again, work around the middleware: Implement additional XML-RPC call to gather error information 23
Error & Fix Jobs failed due to one error: Could not access the MCR component cache" Fix: export MCR_CACHE_ROOT=$( mktemp -d ) basically tells the MCR to store all temporary information in a new tmpdir Will be included in the next POC environment 24
Mortality after fix 25
Discussion We can schedule millions of jobs and capture their outputs on the grid, it just takes a custom solution Other fields (such as pattern recognition) can benefit from this solution Is their similar work being done? If not, can we design and implement a generic solution which does the same? 26
Thanks Jeroen de Ridder Roeland van Ochten Marcel Reinders Jeroen Engelberts Pieter van Beek Evert Lammerts Jan Just Keijzer 27
Life Science Grid Site CPUs SARA 2000 NIKHEF 5000 Philips 1500 RUG 160 Erasmus 32 Keygene 32 TU Delft 32 RUG 32 AMS 32 NKI 16 AMC 16 LUMC 16 WUR 16 UU 16 kun 16 Total 8900 28
Grid Middleware The glue (or spaghetti) that unifies job management across clusters Middleware gina: condor keygene: pbs TU Delft: lsf... RUG: SGE Different sites with different job scheduling applications Heterogeneous compute resources 29
ToPoS: Token Pool Server Add work Submit jobs 1. Get token 2. Do work: Translate token Call function Upload output 3. Delete token Fetch work ToPoS: a pilot job framework. Pilot job: one job / thread which keeps running until all the work has been done. A 'token' represents one unit of work. Tokens can be locked to prevent other jobs from doing the same work twice. Why is ToPoS needed: Problems with grid middleware Inability to deal with large amounts of jobs Failing jobs Job accounting Etc. 30
ORM: Object-Relational Mapper Mapper for persistent storage of objects into a database Saves you from having to write any DB code yourself Examples: Python: SQL Alchemy, Storm Java: Hybernate, Cayenne Ruby: ActiveRecord 31
Why not Molgenis? Familiar with Python, which already has all the tools to make this Design XML-ify generate rol-out to cumbersome 32