The Earth System Grid: A Visualisation Solution Gary Strand
Introduction
Acknowledgments PI s Ian Foster (ANL) Don Middleton (NCAR) Dean Williams (LLNL) ESG Development Team Veronika Nefedova (ANL) Ann Chervenak (ISI/USC) Carl Kesselman (ISI/USC) David Bernholdt (ORNL) Kasidit Chanchio (ORNL) Line Pouchard (ORNL) Alex Sim (LBNL) Arie Shoshani (LBNL) Bob Drach (LLNL) Dave Brown (NCAR) Gary Strand (NCAR) Jose Garcia (NCAR) Luca Cinquini (NCAR) Peter Fox (NCAR)
Current Practices o Scientist (or others) wants a visualisation o Visualisation person gets appropriate data after verifying with data manager as to the name, location, total size, etc. o Data moved to local machine that has visualisation tools o Visualization created on local machine o Hopefully, someone remembers to archive the visualisation
Simple Vis Example
Problems in the process o What if the data cannot be found (e.g. we have 1.2 million files, 73 TB of data), or the data manager is unavailable? o What if there isn t enough disk space or sufficient other resources? o What if a better visualisation tool is located elsewhere? o What if the visualisation should be shared? o What if the visualisation is lost? o ESG is part of the answers to these questions
What is ESG? LBNL: Climate storage facility ANL: Computational grids, & grid-based applications LLNL: Model diagnostics & inter-comparison USC/ISI: Computational grids, & grid-based applications NCAR: Climate change predication and scenarios LANL: Next generation coupled models & computing ORNL: Climate storage & computational resources
ESG Architecture LBNL HPSS! High Performance! Storage System! disk! ANL SRM! Storage Resource! Management! gridftp! server! NCAR opendapg! server! gridftp! Striped! server! CAS! Community Authorization Services! Tomcat servlet engine! MyProxy! server! disk! LLNL MCS client! RLS client! MyProxy client! CAS client! SRM! Storage Resource! Management! gridftp! server! GRAM! gatekeeper! gridftp! gridftp! server! ORNL ISI MCS! Metadata Cataloguing Services! SOAP" SRM! Storage Resource! Management! gridftp! gridftp! server! SRM! Storage Resource! Management! RLS! Replica Location Services! RMI" disk! MSS! Mass Storage System! disk! HPSS! High Performance! Storage System!
Solutions o What happens when data cannot be found, or the data manager is unavailable? Metadata catalogue service (MCS) Replica location service (RLS)
MCS and RLS and Metadata Services ESG CLIENTS API! & USER INTERFACES! PUBLISHING! ANALYSIS & VISUALIZATION! SEARCH & DISCOVERY! ADMINISTRATION! BROWSING & DISPLAY! METADATA! EXTRACTION! METADATA! ANNOTATION! HIGH LEVEL METADATA SERVICES! METADATA & DATA! REGISTRATION! METADATA! BROWSING! METADATA! QUERY! METADATA! AGGREGATION! METADATA! VALIDATION! METADATA! DISPLAY! METADATA! DISCOVERY! METADATA ACCESS! (update, insert, delete, query)! CORE METADATA SERVICES! SERVICE TRANSLATION! LIBRARY! METADATA HOLDINGS! Data &! Metadata! Catalog! Dublin Core! Database! mirror! Dublin Core! XML Files! COARDS! Database! COMMENTS! XML Files!
Solutions (contd.) o What if there isn t enough disk space or sufficient other resources? Hierarchical Resource Manager (HRM)
HRM
Solutions (contd.) o What What if a better visualization tool is located elsewhere? Distributed visualization
CDAT Example of an ESG Script Access The next-generation language, Python, is used to access the Earth System Grid (ESG) at LLNL Import cdms, vcs db = cdms.open( ldap://localhost:389/database=demo,ou=pcmdi,o=llnl,c=us ) f = db.open( ncep_reanalysis_mo ) ds = f( ts ) x=vcs.init( ) x.plot(ds)
CDAT: Example of an ESG GUI Client Access
Solutions (contd.) o What if the visualization should be shared? Access Grid plus Visualisation Tool
Collaborative Environments Science Portals + AccessGrid: University of Michigan (Knoop, Hardin) Vegetation & Ecosystem Mapping Program (VEMAP) NCAR/SCD VETS/KEG Argonne National Labs
Conclusions " Visualisation can require as many services and resources as the initial computation " Many sites do not offer sufficient resources for the visualisations earth sciences require " ESG provides, and will provide, the tools that enable visualisation on a grander scale
Conclusions (contd.) " ESG tools enable better data access, better data knowledge, and the processes of collaboration for the needs of investigating, visualising, and learning