ORIGAMI AND GIPS: RUNNING A HYPERSPECTRAL SOUNDER PROCESSING SYSTEM ON A LIGHTWEIGHT ON-DEMAND DISTRIBUTED COMPUTING FRAMEWORK

Transcription

1 ORIGAMI AND GIPS: RUNNING A HYPERSPECTRAL SOUNDER PROCESSING SYSTEM ON A LIGHTWEIGHT ON-DEMAND DISTRIBUTED COMPUTING FRAMEWORK Maciej Smuga-Otto, Raymond Garcia, Graeme Martin, Bruce Flynn, Robert Knuteson Space Science and Engineering Center, University of Wisconsin Madison 1224 W. Dayton Street, Madison WI Abstract Upcoming meteorological satellite instruments will produce an order of magnitude or more data than the current generation, and will require large scale testing of processing and product generation algorithms. These requirements put a serious strain on existing computing resources, and require the inventive use of emerging clustering and storage technologies. The UW SSEC has developed a demonstration application nicknamed Origami, which provides integrated search, on-demand distributed processing of data pulled from a Storage Area Network (SAN) device, and visualization tieins. This demonstration is being expanded in various directions, including a facility to provide a scientist-friendly front end for an imaging interferometer processing system, called GIPS, also developed at SSEC originally in support of the GIFTS instrument. Where Origami succeeds at effectively integrating diverse computing resources such as clusters, storage systems and desktop visualization applications, GIPS provides the core processing capability to handle raw hyperspectral data. We present this combination of lightweight web interface with high-throughput processing as a tractable approach for handling validation of products expected from the future crop of meteorological satellite instruments. INTRODUCTION The scientific workflow need we are addressing involves the retrieval, processing and visualization of large volumes of data, often on the order of Terabytes, associated with hyperspectral sounder-imager instruments. These data are usually stored on separate network-accessible volumes, and require the processing power of a modern compute cluster to perform even basic calibrations. Regardless of where they come from and how they are processed, the products from the data will end up on a researcher s desktop to be further studied within some visualization environment. These three pieces, processing pool, storage system and local workstation, are the hardware components that must be successfully tied together. Helping attain this goal are various commonly available commodity software packages for visualization, queuing, data storage management and grid resource management. What is missing is a way to glue all these software and hardware resources together in a way that is specifically useful to the working atmospheric scientist using satellite instrumentation data. While there are certain commercial science workflow management and scientific programming packages that may technically satisfy this need, they are often closed solutions that carry considerable dependency limitations. GIFTS PROCESSING Processing raw data from the Geosynchronous Imaging Fourier Transform Spectrometer (GIFTS) instrument was the principal motivation for this work. Initially the GIFTS Information Processing System (GIPS) was meant to cover various aspects of distributed computing as well as hosting the GIFTS algorithms. Eventually, it became clear that a successful deployment would require reducing the scope of the project to its fundamental requirement of providing an environment for the algorithms.

2 GIPS was therefore retooled to handle provisioning of the algorithms with appropriate data and metadata. This form of GIPS is suitable for running on a single compute node, and the need arose to find a distributed computing solution to match. Figure 1: At the heart of GIPS is an algorithm pipeline that produces calibrated radiances from raw imaging interferometer data and is a candidate heavyweight ground processing task ORIGAMI Origami is the outcome of a parallel effort in distributed computing at the SSEC - it is an experiment in rapid prototyping of distributed science workflows with lightweight scripts. It began as an attempt to tie together computational, storage and visualization resources that had previously been separately available to scientists. For a typical algorithm research and development task, the interested scientist had to obtain and coordinate these resources manually, usually having to rediscover a solution to problems of resource management and connectivity that have already been solved many times in the past. The Origami project attempted to glue together these separate resources, and thereby solve the common problems encountered in a typical meteorological algorithm development workflow. On the compute farm side, Origami consists of a daemon that takes work orders from the user, and launches the appropriate executables on available compute nodes, using matched data sourced from the storage component in our test case, a Storage Area Network (SAN). This functionality is similar in many respects to existing queuing systems, except that the metadata for tying computations to data resources is also maintained in the process, and can easily be tracked by the scientist.

3 Figure 2: Origami: Workflow for processing and visualizing large volumes of data The user interface aspect of Origami originally initially consisted of a web application that communicated using XML Remote Procedure Calls with the back-end daemon to interactively create work orders. This module was eventually split into two pieces one of which is a middleware web service that communicates using pure XML with an arbitrary web client. The other half of the user s view of an Origami workflow is the ability to easily gather together data output from the distributed computation into a visualization environment on the user s desktop. Certain emerging visualization environments, such as Unidata s Integrated Data Viewer (IDV) and the SSEC s McIDAS V, have the capacity for integrating various third-party modules. One current project is focusing on integrating the complete Origami user interface into the McIDAS V application, thus exposing the entire workflow to the end-user from a single application. INTEGRATING GIPS INTO ORIGAMI: THE XML ALGORITHM INTERFACE The relevant aspect about Origami for the scope of this presentation is that it can be extended to serve as a glue technology between GIPS running on individual nodes, and other components of a true distributed computing system. To make GIPS compatible with an external distribution and workflow manager such as Origami, we found that it was beneficial to convert all algorithm interfaces to a common XML format. This XML interface file would be converted to an API-level interface for compiling into GIPS, and would also be used by the Origami engine to perform workflow and scheduling decisions. The XML format is currently a simple attribute-based one, with a tag for each algorithm that contains sub-tags specifying that algorithm s inputs and outputs, as well as selecting from a standard list of other external dependencies, such as an algorithm s need for numerical model data or for a radiative transfer model. While this format is likely to change, it is described using the XML Schema Description (XSD) standard, thus enabling format changes to easily propagate through the code that uses this interface description. An initial set of tools was developed in the Python scripting language and the SAX event-based parsing model for XML, following our philosophy of rapid development.

4 Figure 3: Integration of Origami and GIPS drove the creation of an XML interface for algorithms hosted by the system This XML interface style has also been applied to GEOCAT and LEOCAT, two other satellite data processing frameworks being developed at the SSEC. These frameworks utilize a subtly different XML schema, but share many of the tools used to leverage the approach. CONCLUSION A dominant theme in this work has been the an emerging model of our intended science workflow system that uses maximally decoupled components, to allow for build versus acquire decisions to be made independently for each component. We initially intended to build an integrated framework that handled most of the resource management, but discovered that queuing systems and more generally grid toolkits already provided the bulk of that functionality. Instead, Origami has been evolving in the direction of managing domain-specific metadata, and coordinating the various existing externally built components so that they satisfy the workflow for processing and visualizing the meteorological satellite data. Figure 4: Component level architecture of the distributed processing system.

5 REFERENCES Smuga-Otto M., R. Garcia, R. Knuteson, G. Martin, B. Flynn, D. Hackel (2006) Integrating High- Throughput Parallel Processing Fraemwork and Storage Area Network Concepts into a Prototype Interactive Scientific Visualization Environment for Hyperspectral Data. Eos Trans. AGU, 87(52), Fall Meet. Suppl., Abstract A21D-0858 Garcia, R. K.; Ackerman, S. A.; Antonelli, P.; Dedecker, R. G.; Dutcher, S.; Howell, H. B.; Huang, H.L.; Knuteson, R. O.; Olson, E. R.; Revercomb, H. E.; Smuga-Otto, M. J. and Tobin, D. A prototype for the GIFTS information processing system. International Conference on Interactive Information and Processing Systems (IIPS) for Meteorology, Oceanography, and Hydrology, 21st, San Diego, CA, 8-13 January 2005 (preprints). American Meteorological Society, Boston, MA, 2005, Paper P1.45. Wallnau K., S. A. Hissam, R. C. Seacord, Building systems from commerical components, Addison- Wesley Longman Publishing Co., Inc., Boston, MA, 2001