Background mission of the HESSI satellite solar flares purpose of the HEDC. Challenges data volume / data flow derived data types research topics

Transcription

1 Background mission of the HESSI satellite solar flares purpose of the HEDC Etzard Stolte IKS Group Challenges data volume / data flow derived data types research topics Solutions system architecture current GUI / StreamCorder Progressive Visual Previews Progressive Analysis Previews Primary objective explore the basic physics of particle acceleration and energy release in solar flares simultaneous, high resolution imaging and spectroscopy of solar flares from 2 kev X-rays to 20 MeV all observations are send to earth Small Explorer (SMEX) mission class Launch launch vehicle: Pegasus launch date: July 4, 2000 Orbit: circular, 600 km altitude. 38 degrees inclination of the equator Field of view: full sun (1 degree) observation lifetime: 3 years 1

2 What are solar flares? a solar flare is an enormous explosion in the solar atmosphere, involving sudden bursts of particle acceleration, plasma heating and bulk mass motion they are believed to be caused by the magnetic fields around solar spots in the suns corona the occurrence rate follows an 11-year cycle What will HESSI measure? frequency, location and evolution of photon emissions during solar flares What will be done with the photon observations? create small QuickLooks of e.g. photon acceleration create large images of e.g. sun spectra visualize photon flux of thousands of flares Scientific Data Warehouse Provide high volume access and data analysis to astro-physicists at ETH and the rest of the world Manage at least 1 TB of primary data (photon counts) and 2 TB of secondary data (scientific visualizations) Objective 1: Data Management Build an Oracle database capable of handling the expected data- and user- load Support analyses that involve GB of data and days of computing in a distributed environment Objective 2: Optimizations -> topic of this talk Develop data-/application specific methods to improve system performance Support online trial/error data analysis Provide central client user interface 2

3 Univ. of California, Berkeley (HESSI) Satellite assembly Analysis routines.. NASA, Goddard/Washington (HESSI) Imaging, Flare Theory flight control center... ETH (HEDC) Prof. Benz, Astronomy (Solar Physics) Prof. Gross, Computer Science (analysis environment) Prof. Alonso, Computer Science (data management) World Berkeley clean/calibrate burn on CD send data to ETH HEDC send summaries to ETH Mauna Loa Solar Obs. Kanzelhöhe Big Bear Solar Obs. NOAA SpaceEnvr. many others 3

4 Event 1, nbins = Xoffset = Yoffset = Flux = g=0.050 niter = 70 Event 2, nbins= Coll=6 9, transp= Chi2_n= Raw Data [0.1-50MB] / segment - photon counts, telemetry - start 1.2 TB / 80% - 2 yrs. 1.2 TB / 30% Meta Data [ MB] / tuple - event attributes, comments - start 0.1 GB / <1% - 2 yrs. 0.5 GB / <1% Visual Data [ MB] / file - Images, QuickLooks - start 0.3 TB / 20 % - 2 yrs. 2.7 TB / 70% Telemetry Fits (50 MB/file, 1TB Total) HLE cache (non solar,?) Pre-Binned Data (< flare size) HLE fits (solar) (flare size, 250 GB) Ancillary Data cache (?) Aspect Solution cache (?) HEDC- DBMS Oracle 8(i) Derived Data Products (250 GB) Catalogues + HEDC (1 GB) 4

5 Data Management effective, automatic data placement partitioning schemes, replication types optimal storage hardware optimal data segment size Precision/Performance tradeoff progressive low-resolution analyses/previews generalize resolution estimate for a given analysis method image version control Stream Management client raw- and visual-data caching integrity guarantees of asynchronous analyses Data Quilt : reconcile overlapping segments of different resolution QoS guarantees, scheduling, load balancing and parallel execution user (browser/java clients) astro server (IDL) analysis middleware (Java, Apache) Oracle 8i (PL/SQL, Java stored procedures) JRE 5

6 Client Tier Web Browser TCP/IP StreamCorder DM StreamCorder DM TCP/IP Web Server RMI Registry Middle Tier PL DM TCP/IP JDBC Server Tier Analysis Server File System DBMS 20% (0.2k) 1: % (91k) 1:4 Original (370k) 1:1 Compressors, such as JPEG, Wavelet and Fractals offer 100% signal reconstruction High compression ratios for lossy compression Progressive decompression, with the most important information first -> Image quality during download is a function of time HEDC will use progressive visual previews, as Only one encoded image file has to be managed Every transferred byte during the user query/display cycle is reused After an interrupt, transmission can resume (-> internet users) 6

7 30% (0.03MB) 1:70 50% (0.1 MB) 1:21 Original (2.1 MB) 1:1 Experiments have shown, that Some analyses, such as spectrograms, work correctly with low resolution data -> analysis precision is a function of time The HEDC will use progressive analysis previews, as Processing is greatly accelerated All transferred raw data can be re-used by remote users or the processing subsystem to increase result precision successively -> online trial/error analysis over long time spans becomes possible Catalogue - collection of HLEs - user defined - overlapping High Level Event - e.g. flare period - various analysis/related derived data products - attributes Analysis - e.g. QuickLook, Image - standard attributes - free attributes 7

8 HEDC FrontEnd Manage Data / Streams Monitor system state Consistent user interface GUI Modular design Resource Management Disk, CPU, Network Database Connections Single/multiple server Local caching StreamCorder Modules: Cordlets Browse tuple (edit, query, etc.) Browse/View spectra Manage Jobs Link to analysis Corder Cordlet DM HEDC Cordlet Cordlet 8