DATA FUSION, DE-NOISING, AND FILTERING TO PRODUCE CLOUD-FREE TEMPORAL COMPOSITES USING PARALLEL TEMPORAL MAP ALGEBRA Bijay Shrestha Dr. Charles O Hara Preeti Mali GeoResources Institute Mississippi State University
OUTLINE Introduction Map Algebra Temporal Map Algebra Vegetation Indices Parallel Temporal Map Algebra Results Conclusions
Motivation Undesired cloud cover in satellite images Need for cloud-free satellite observation Making use of high temporal resolution of satellite data using cross-platform fusion Moderate Resolution Imaging Spectroradiometer (MODIS) Aqua and MODIS Terra Rule based cross-platform fusion of MODIS Aqua and MODIS Terra Using Parallel Temporal Map Algebra
Map Algebra An approach to raster data handling which treats spatial data layers as variables which may be combined using mathematical operators. Source: Geographic Information Systems and Cartographic Modeling, Tomlin
Map Algebra (Contd.) Local Operation 0 1 2 0 1 2 0 2 4 3 4 5 + 3 4 5 = 6 8 10 6 7 8 6 7 8 12 14 16 Arithmetic, Relational, Bitwise, Boolean, Logical, Accumulative and Assignment Operators can be used for Map Algebra. 3 x 3 Focal Neighborhood 1 2 3 4 5 1 2 3 4 5
Temporal Map Algebra Temporal extension to conventional map algebra. Treats time series of imagery as three dimensional data set XY dimension represent Earth s surface Z dimension represents time Y Y X Image n Z Time Image 1 X
Temporal Map Algebra (Contd.) Temporal Map Algebra Local Function 0 1 2 0 1 2 0 1 2 5 6 7 5 6 7 3 4 5 10 11 12 10 11 12 6 7 8 + 0 1 2 0 1 2 27 28 29 5 6 7 5 6 7 30 31 32 10 11 12 10 11 12 33 34 35 0 1 2 0 1 2 27 29 31 5 6 7 5 6 7 33 35 37 10 11 12 10 11 12 39 41 43 Temporal Map Algebra 3x3x3 Focal Neighborhood 1 0 1 2 3 4 0 5 1 6 2 7 3 8 4 9 2 5 1011 1011 6 7 121314 8 9 3 Row 14 1011 1516 1516 12 171819 1314 4 10 11 12131419 15 1516 2122 2122 16 171819 17 232425 1819 5 25 21 2122 22 232425 23 2425 1 2 3 4 5 1 2345 Time Column Source: Analyzing time series satellite imagery using temporal map algebra, J.Mennis and R. Viger
Vegetation Indices Dimensionless, radiometric measures that function as indicators green vegetation. Normalized Difference Vegetation Index (NDVI) NDVI = NIR NIR + Re Re d d Healthy, chlorophyll-based vegetation strongly reflects near-infrared wavelengths and reflects relatively weakly in the visible red. Range [-1, +1] NDVI is highly useful in vegetation studies.
Parallel Processing Decomposing a large process into small processes that can be solved simultaneously to provide faster execution time. Need for intensive computing to integrate and process large datasets. Many spatial programs are inherently parallel. Parallel processing can provide leap in performance.
Parallel Processing (Contd.) Requires large volume of satellite data Input Temporal Data Sets NDVI cube Image Quality cube Geo-location Angle cube
Performance Metrics Serial Execution Time (T S ) Parallel Execution Time (T P ) Speedup S= T S / T P Efficiency E = Speedup/p * p represents the number of processors used in parallel processing
Tools Used Matlab MatlabMPI LibTIFF
MODIS Datasets 6 MODIS datasets used MYD09GQK - 250m surface reflectance MODO9GQK data MYDMGGAD - 1 Km Geo-location angles MODMGGAD data MYD09GST - 1 Km surface reflectance MOD09GST quality data MYD - Aqua MOD - Terra
Methodology The following methodology was used Selection Collection Preprocessing Execution Analysis
Pre-processing Surface reflectance day 1 NDVI day 1 Surface reflectance day 2 Surface reflectance day N NDVI day 2 NDVI day N Aqua and Terra NDVI cubes Surf.Refl. Quality day 1 Surf. Refl. Quality day 2 Surf. Refl. Quality day N QMask day 1 QMask day 2 QMask day N Aqua and Terra Quality Mask cube Input to Fusion Algorithm Geolocation Angles day 1 view zenith angle day 1 Geolocation Angles day 1 Geolocation Angles day N view zenith angle day 2 view zenith angle day N Aqua and Terra View zenith angle cube
Fusion For day from 1 to 5 if angle is less than 48 AND mask is Land select pixel else get next pixel If no pixel is selected select pixel with highest NDVI
Example September 13-17 Aqua MODIS Terra MODIS Fused MODIS
Parallel Processing (Contd.) Temporal Cubes Block Distribution & Processing 1 2 n Fused Product
Experimental Results (Contd.) The evaluation of parallel Temporal Map Algebra consists of following parts A measurement on the performance of serial processes The performance profiling of the parallel processes to fuse Aqua-Terra datasets
Experimental Results (Contd.) For the evaluation compositing was performed on 32-days MODIS data cubes using fusion algorithm The experiments were conducted on the EMPIRE cluster at HPC 2 at Mississippi State University. Initializing processes on different machines in cluster using MatlabMPI. Each Process Reads the cubes stored in Network File Server Performs computations Stores results in common storage Takes Readings Metrics are computed
Experimental Results (Contd.) Serial Run-Time 32 days Aqua-Terra Fusion Dimensions - 3148 4000 x 32 NDVI, Quality, Angular Temporal Cubes Input Size - 5384.5 MB Output Size - 2692.25 MB Serial Time - 58.98 hours
Experimental Results (Contd.) Parallel Run-times With Centralized Data Run-Times for Different Number of Processors 70 60 50 Hours 40 30 20 10 0 0 8 16 24 32 40 48 56 64 72 Number of Processors
Experimental Results (Contd.) Speedup and Efficiency SpeedUP Efficiency 30 1.2 25 1 Speedup 20 15 10 Efficiency 0.8 0.6 0.4 5 0.2 0 0 8 16 24 32 40 48 56 64 72 Number Of Processors 0 0 8 16 24 32 40 48 56 64 72 Number of Processors
Conclusion Further quantitative analysis needs to be performed to evaluate the fused Aqua-Terra product. The execution performances were analyzed using various metrics. The accuracy of the results, however, is affected by various factors such as network bandwidth, network traffic, processor speed, and network/cluster architecture. The use of Matlab and MatlabMPI as the programming environment has added some overhead to the total serial and parallel times for the computation. These overheads can be reduced by programming in C or C++ that have lower overhead. Temporal Map Algebra can be used to solve more complex spatio-temporal analysis problems on large datasets with many dimensions.
Proposed future Work Qualitative analysis of the cross-platform fused images Exploration of the use of TMA to solve newer spatiotemporal analysis problems with higher complexity with many dimensions. Achieve better performance by using software systems that have low overhead. Rapid prototyping/development capability
THANK YOU bijay@gri.msstate.edu cgohara@gri.msstate.edu preeti@gri.msstate.edu