GENIUS: Grid Enabled Neurosurgical Imaging Using Simulation

Transcription

1 GENIUS: Grid Enabled Neurosurgical Imaging Using Simulation Peter Coveney, Steven Manos & Marco Mazzeo Centre for Computational Science University College London

2 Introduction The GENIUS project aims to model large scale patient specific cerebral blood flow in clinically relevant time frames Objectives: To study cerebral blood flow using patient-specific image-based models. To provide insights into the cerebral blood flow & anomalies. To develop tools and policies by means of which users can better exploit the ability to reserve and co-reserve HPC resources. To develop interfaces which permit users to easily deploy and monitor simulations across multiple computational resources. To visualize and steer the results of distributed simulations in real time

3 HemeLB Efficient fluid solver for modelling brain bloodflow called HemeLB: Uses the lattice-boltzmann method Efficient algorithms for sparse geometries Topology-aware graph growing partitioning technique Optimized inter- and intra-machine communications Full checkpoint capabilities.

4 Getting the Patient Specific Data Data is generated by MRA scanners at the National Hospital for Neurosurgery and Neurology trilinear interpolation pixels x 100 slices, res: mm x 0.8 mm Our graphical-editing tool x 682 cubic voxels, res: mm

5 reconstruction and BC set-up; fluid sites, inlet and outlet sites in red, black and green respectively; 4.7M sites stationary von Mises stress flow field obtained with our ray tracer HemeLB The complete simulation with IO, convergence criteria at each time step, for 3062 time steps, takes about 90 seconds on HPCx with 64 processors; We can investigate complete cerebral systems with much higher resolution.

6 Ray Tracing A ray tracer has been incorporated into HemeLB core to efficiently monitor simulation results in real time Data transmission and pre/post processing are avoided at every frame Only updated image is sent to the client

7 US/UK Grid Infrastructure Used NGS HPCx UK NGS Leeds Manchester TeraGrid Oxford RAL LONI LSU HSC La Tech ULM The GENIUS project makes use of infrastructure provided by LONI, TeraGrid and NGS, connected by dedicated switched optical light paths Lightpaths culminate at National Hospital, London NSU Alex SU ULL McNeese LSUSLU UNO LSU HSC Tulane

8 Single-site performance HPCx Bigben Cray XT3 Hector Cray XT4

9 Cross-site Runs with MPI-g GENIUS has been designed to run across multiple machines using MPI-g Some problems won t fit on a single machine, and require the RAM/processors of multiple machines on the grid. MPI-g allows for jobs to be turned around faster by using small numbers of processors on several machines - essential for clinician HemeLB performs well on cross site runs, and makes use of overlapping communication in MPI-g

10 Cross-site MPIg performance LONI AIX p5 frames, 32 cores per site over 1, 2 and 3 sites

11 HemeLB/MPI-g Requires Co-Allocation We can reserve multiple resources for specified time periods Co-allocation is useful for meta-computing jobs like HemeLB, viz and for workflow applications. We use HARC - Highly Available Robust Co-scheduler (developed by Jon Maclaren at LSU). Slide courtesy Jon Maclaren

12 What can I co-allocate? In principle, anything that has a scheduler supporting user-made reservations: compute Viz networks. Just need a HARC Resource Manager (RM) that can talk to the scheduler HARC is very extensible HARC deployed on LONI, NGS & some TG sites Slide courtesy Jon Maclaren

13 HARC HARC provides a secure co-allocation service Multiple Acceptors are used Works well provided a majority of Acceptors stay alive Paxos Commit keeps everything in sync Gives the (distributed) service high availability Deployment of 7 acceptors --> Mean Time To Failure ~ years Transport-level security using X.509 certificates HARC is a good platform on which to build portals/other services XML over HTTPS - simpler than SOAP services Easy to interoperate with Very easy to use with the Java Client API

14 GUR Co-scheduler Grid Universal Remote GUR is the production coscheduling service at SDSC and NCSA. Service allows users to create synchronized reservations (constrained by earliest start time, latest end time, duration and number of nodes). Policies control the effect of user reservations on the normal job queue.

15 Real Time Visualisation and Steering A way to let HemeLB know the parameters to be steered --> we use the RealityGrid steering system to steer the input data on the fly. One aim is to do all this for distributed (crosssite) simulations For medical applications, need may be urgent

16

17 SPRUCE Special PRiority and Urgent Computing Environment Applications with dynamic data and result deadlines are being deployed Late results are useless Wildfire path prediction Storm/Flood prediction Patient specific medical treatment Some jobs need priority access Right-of- Way Token

18 SPRUCE Right-of-Way Tokens Event 2 Automated Trigger 1 First Responder SPRUCE Gateway / Web Services Right-of-Way Token Human Trigger Right-of-Way Token

19 Application Hosting Environment Need to utilize resources from globally distributed grids Administratively distinct Running different middleware stacks Wrestling with middleware can't be a limiting step for scientists Need tools to hide complexity of underlying grids

20 Application Hosting Environment Applications are stateful WSRF services Lightweight hosting environment for running applications on grid resources and on local resources Community model: expert user installs AHE, shares applications with others Simple clients with very limited dependencies

21 AHE Concepts Applications not jobs Application could consist of a coupled model, parameter sweep, steerable application, or a single executable AHE supports single site jobs, multisite MPIg jobs, and single and multisite steerable jobs We use application to denote a higher level concept than a job In AHE terminology, an application may require running multiple jobs Architecturally, the AHE is a portal, where the interface is a rich client, not a web browser Of course, AHE services can be used behind a Web portal, if you like

22 PDA/Cellphone Visualisation and Steering RealityGrid: a tool for modelling and simulating very large or complex condensed matter structures Human Factors issues key Large-scale applications, scheduling when resources become available, around the clock computation AHE application launching built in to client User interaction/workflo w: not necessarily 9:00am 5:00pm, from the desk environment

23 AHE - HARC integration Users can co-reserve resources from the AHE GUI client using HARC When submitting a job, users can either run their jobs in normal queues, or use one of their reservations AHE passes reservation through to the Globus GRAM AHE client uses HARC Java client API to manage reservations

24 Creating HARC reservations in the AHE

25 AHE ReG & WS GRAM Integration For steerable applications, AHE server starts up a ReG Steering Web Service (SWS) for the application, & sets the end point reference in the job s environment AHE can submit jobs to GridSAM (described in JSDL) or WS GRAM (described in JDD) by applying a XSL Transform to the job specific WS- ResourceProperties document AHE --> WS GRAM can launch single or multisite MPIg jobs

26 Conclusions We have developed computationally efficient tools to manipulate, simulate and visualize patient specific vascular systems Goal is to better comprehend blood flow behavior of normal and anomalous cerebral systems and provide patient specific models for clinical guidance in surgical operations Application Hosting Environment extended to launch MPI-g cross site runs, integrate ReG steering and visualisation, as well as making HARC cross site reservations

27 LONI Staff NGS Staff TeraGrid Staff Luis Fazendeiro Daniel Scott Owain Kenway Dr. Radhika Saksena Dr. Simon Clifford Prof. Frank Smith Dr. Nick Ovenden Dr. Brian Toonen Prof. Nicholas Karonis Dr. Stefan Brew Prof. David Hawkes Kate McLeish Dr. Jon Maclaren Dr. Shantenu Jha Rob Haines Dr. Robin Pinning Acknowledgments