Window Query and Analysis on Massive Spatio-Temporal Data

Transcription

1 Available online a ScienceDirec IERI Procedia 10 (2014 ) Inernaional Conference on Fuure Informaion Engineering Window Query and Analysis on Massive Spaio-Temporal Daa Huan Wang, Junhui Deng, Guodong Yuan* Deparmen of Compuer Science and Technology, Tsinghua Universiy, Beijing,100084, China Absrac Along wih he expansion of compuer-based climae simulaions, efficien visualizaion and analysis of massive climae daa are becoming more imporan han ever. In his paper, we ry o explore he facors behide climae changes by combining window query and ime-varying daa mining echniques. Wih consan query ime and accepable sorage cos, he algorihms presened suppor various queries on 3d ime-varying daases, such as average, min, and max value. A new ime-varying daa analysis algorihm is given, which is especially suiable for analyzing big daa. All hese algorihms have been implemened on and inegraed ino a visual analysis sysem, wih iled-lcd ulra-resoluion display. Experimenal resuls on several daases from pracical applicaions are presened The Auhors. Published by Elsevier B.V. This is an open access aricle under he CC BY-NC-ND license Published by Elsevier B.V. (hp://creaivecommons.org/licenses/by-nc-nd/3.0/). Selecion and peer review under responsibiliy of Informaion Engineering Research Insiue Selecion and peer review under responsibiliy of Informaion Engineering Research Insiue Keywords: Window Query; Visual Analyics; Spaio-Temporal Daa Mining; Visualizaion; Earh Sysem Model; 1. Inroducion In recen years, wih he fas growing of compuaional capabiliies and sorage capaciies, much more scienific daa has been generaed by advanced observaion insrumens and simulaion experimens, such as he global ocean real-ime observing sysem. We will ake he earh sysem model (ESM) as an example. The * Corresponding auhor. Tel.: address: whhu168@163.com. This paper is suppored by he Naional High Technology Research and Developmen Program of China (863 Program) under gran No. 2010AA The Auhors. Published by Elsevier B.V. This is an open access aricle under he CC BY-NC-ND license (hp://creaivecommons.org/licenses/by-nc-nd/3.0/). Selecion and peer review under responsibiliy of Informaion Engineering Research Insiue doi: /j.ieri

2 Huan Wang e al. / IERI Procedia 10 ( 2014 ) ESMs are ses of equaions describing processes wihin and beween he amosphere, ocean, sea-ice and he erresrial and marine biosphere [1]. According o he simulaion of earh climae and environmenal changes, scieniss can ge a beer undersanding on he ecological environmen and furhermore inerpre he earh evoluion mechanism in more deails. Big daa of size up o TB is everywhere in earh science. Typically, even a single scan of such a daase will cos dozens of minues. Tha is why he visualizaion echniques and ools are so imporan for scieniss [2]. On he oher hand, however, he way human beings inend o inerpre daa is much differen from ha of compuers. Wih higher-level visual and percepual capabiliies, for example, we can find paerns from images and animaions more quickly and precisely. In fac, climae visualizaion has been an imporan par of VISC for he las wo decades. Usually, 2D scalar daa can be visualized using algorihms such as color mapping, while 3D scalar daa can be visualized using algorihms such as isosurfaces, conouring exracions, volume rendering, ec.. A number of visualizaion oolkis have also been developed, such as VTK [3] and visi [4]. In recen years, visualizaion echniques are moving fas in he direcion of large-scale, ineracive, and real-ime. Visual analyics (or, visual daa mining) is a combinaion of visualizaion and daa mining echniques. Wong e al. gave a lis of op challenges in exreme-scale visual analyics [5]. In his paper, we presen a comprehensive visual analysis plaform for massive spaio-emporal daa based on a combinaion of window query echniques and ime-varying daa mining mehods. 2. Background 2.1. Basic Conceps The problem of range query is one of he core problems in compuaional geomery. Suppose is a sysem consising of several subses of a d-dimensional Euclidean space R d. P is a poin se consising of n elemens in R d. The problem of range query can be defined as: for any given region R, o design efficien algorihms ha can find ou wha elemens belong o R (fig. 1(a)). If P and R are given a he same ime, he problem can be ransformed ino a problem of solving P R. We can make a one by one es on he elemens of P, which is obviously ime-consuming [6]. s ini,, i j1 1 i 2 j s s ni1 ini, 1, j i2 1 i2 1 i2, ni1, Fig. 1. (a) Lef: an example for range query; (b) Righ: he compuaion of sparse able in 1d siuaion In real applicaions, P is ofen given in advance and relaively unchangeable, which means i can be solved by a more efficien algorihm. In his siuaion, i is absraced ino a precise mahemaical model. For any given poin p P, a weigh wp ( ) S is assigned. I can be proved ha ( S, ) is a commuaive semigroup [6]. Then his problem is mahemaically expressed o compue he sum of all he weighs of p belonging o P R.

3 140 Huan Wang e al. / IERI Procedia 10 ( 2014 ) Relaed Work A number of algorihms have been invened for range queries such as Quad-Tree, B-Tree, KD-Tree, Prioriy Search Tree, Skip Liss, ec. [7, 8, and 9]. Taking he range min/max query (RMQ) as an example, he laes research resuls have shown ha he RMQ problem can be solved in On ( ), O(1) [10]. Through consrucing a Caresian ree in linear ime, i is ransformed ino a Lowes Common Ancesor (LCA) problem [11], which can be solved in On ( ), O(1). However, his resul canno be direcly exended o spaioemporal daa eiher because of he inconsisency of ree srucure for each ime slice or heir complexiy for implemenaion. This paper gives an efficien algorihm using sparse able echniques in On ( log n), O(1), which makes a good balance boh in space and ime complexiy. 3. Window Query For convenience, only he formulae of 2d spaio-emporal daa query are given and he 3d siuaion can be easily exended. Suppose he number of rows, columns and ime slices of a 2d daase V are RCT,, and define N R* C. The lef-op corner subscrip of a query window W is (1, i j 1), and he righ-down is (2, i j 2). I mus be said ha because he daa is vas, usually up o hundreds of gigabyes, he oupu daa canno be oally handled in he memory. Therefore efficien cache sraegies and sorage echniques mus be carefully designed. Usually, he daa is divided ino secions according o he ime dimension for opimal performance Average Query We can easily consruc an average query algorihm using he pigeonhole principle. I is composed of wo sages: he preprocessing sage and he query execuion sage. Firsly, considering a subse of V from (0,0) o (, i j) a ime, si, j, is he sum of all he poins locaed in i. Therefore, we have: s v s + s -s. (1) ij,, ij,, i 1, j, ij, 1, i-1, j-1, Wih a lier observaion, we can use he dynamic programming mehod o compue s i, j,, which raverses he whole daase only once. Therefore, boh he space and ime complexiy are ( N* T). Secondly, following he similar principle, he average value avg in he window W is given by: si2, j2, -si1, j2, -si2, j1, si 1, j1, avg. (2) ( i2i11)*( j2 j11) Obviously, he ime complexiy is ( T ) Min/Max Query Because of he similariy of range max and min query, below we will only discuss he range max query. Firsly, define a sparse able s i, j, ni, nj,, wih he size N* log R* log C * T, where s i, j, ni, nj, is he max ni nj value in he subse saring a ( i, j ) having lengh (2,2 ) a ime. I is obvious ha we can ge a recurrence algorihm as follows, and fig.1 (b) shows how i works in 1d siuaion: s max s, s, s, s. (3) i, j, ni, nj, i, j, ni-1, nj1, ni1 nj1 ni1 nj1 i2, j, ni1, nj1, i, j2, ni1, nj1, i2, j2, ni1, nj-1, Wih he iniial condiion si, j,0,0, vi, j,, he preprocessing funcion can be compued using dynamic programming echniques. Therefore, boh he space and ime complexiy are ( N*log N* T). Secondly, choose wo subses ha enirely cover he area[1, i i2]*[ j1, j 2], and hen find he max value u beween hem. Therefore, we have 0.5*( i1 i2) i1 2 i 1 i2, so is for u j. Le ui log( i2 i1 1) and uj log( j2 j1 1). The query process only needs ( T ) ime complexiy and he formula is given by:

4 Huan Wang e al. / IERI Procedia 10 ( 2014 ) Time-varying Daa Analysis max max s, s, s, s. (4) i1, j1, ui, uj, ui uj ui uj i22 1, j1, ui, uj, i1, j 2-2 1, ui, uj, i22 1, j 2-2 1, ui, uj, Due o he ubiquious naure of ime-varying daa, a wide variey of daa mining mehods have been applied by differen research communiies like sock marke researchers, signal processing engineers, and corporae income predicors [12]. Neverheless, heir scope is ypically limied. In our applicaion, we need o design a mehod ha can no only analyze he changing rends of local daa, bu also easily observe he global rends. The Growh Marix mehod is a very good soluion o his problem. I was firs proposed by Daniel A. Keim in financial analysis in 2006 [13]. Differen from he char echniques, i ranslaes 2d informaion ino 3d, which could display much more hidden informaion and make beer use of human beings sensiiviy o color. Fig. 2. (a) Lef: visual analysis for he min query resuls; i shows he global air emperaure from 1850 o 2048; he black region circled in purple indicaes he emperaure flucuaes wildly nearby; (b) Middle: a complee view of he visual analysis sysem; (c) Righ: visualizaion for he GAMIL daase For a ime series S wih size T, firsly generae a 2d riangular layou L, wih he horizonal axis represening he sar ime i, he verical axis represening he end ime j, and he color a Li (, j) represening he value of GM (, i j ), which is a funcion of Si () and S( j ). This basic echnique can no only visualize he inernal effec a differen scales (days, weeks, monhs, years), bu also he inra effec of differen ime series. As shown in fig. 2 below, he consruced riangle marix GM is mapped o he image layou L linearly. Therefore, he color L(, i j) of every bi on L represens he changing rends of S. Generally, GM (, i j) S()- i S( j) is chosen. 5. Implemenaion Our sysem is implemened in C++ and Java Apple. The sysem consiss of 3 layers: resource and infrasrucure layer, business layer, and displayer layer, as shown in fig. 3. In he business layer, he Daa Conversion model basically eliminaes daa noise and unifies forma (usually NETCDF [14]). The query parameers such as he locaion and size informaion of he query window are packed and ransferred o he Query Engine model on he server. The query engine is responsible for exracing daa from disks and feeding back he ime series o he clien, which can be responded in (1) ime. In he display layer, adminisraors can inerac wih he server using command lines and users can visi he clien using browsers. Users can also use iled-lcd display in order o ge ulra-resoluion images.

5 142 Huan Wang e al. / IERI Procedia 10 ( 2014 ) Fig. 3. The visual analysis sysem archiecure for massive spaio-emporal daa Ulra-resoluion images and visualizaions are imporan for he earh scienific researchers. When higher resoluion is used, he suble weaher phenomena and fine-grained deails are easier o be observed. For example, he 1 laiude by 1 longiude resoluion generaes 64,800 daa poins, while he 0.25 laiude by 0.25 longiude resoluion generaes 1,036,800 daa poins, which mean 16 imes amoun of compuaion. Our iled-lcd displays are buil enirely from common commodiy pars, which help us use he lowes price-perpixel echnology. As shown in fig. 4, he display environmen is chosen as 4*6 iled layous, wih each ile 1440*960 resoluions. Fig. 4. The server clusers and iled-lcd displays used in he visual analysis sysem 6. Experimens In he experimens, o verify he correcness of he sysem, we firsly generaed a group of sine wave daa and creaed hundreds of query recangles for each daase randomly. We also esed he GAMIL daases from he Insiue of Amospheric Physics o verify he 3d spaio-emporal daases, wih 362*196*30 size and 120 ime slices, as shown in fig. 2 (b) and (c). For he performance analysis, we esed on he Linux operaion sysem, 4 Inel(R) Core(TM) i7 CPUs wih each 8 cores, 2.67GHz, 20GB memory and 100 Mbi/s wihin he local area nework environmen. We used he bias correced downward long wave radiaion flux daases (dlwrf) simulaed by Princeon Universiy Hydro Climaology Group, wih 360*180*1 size and 2920*5 ime slices (3-hourly oupu from 2003 o 2007), as shown in fig. 2(a). We found he query speed was almos real-ime. The query window could be arbirarily dragged, almos wihou any delay, laency or jier. 7. Conclusion Daa-driven visual analysis is a promising new echnology in scienific research in recen years. In his paper, we designed and implemened a comprehensive oolki for massive spaio-emporal daa analysis using window query and Growh Marix mehods. In he experimens, we used he large screen echnology o beer

6 Huan Wang e al. / IERI Procedia 10 ( 2014 ) display he ulra-resoluion climae simulaion daa. I urned ou ha our cos-effecive large-screen iled- LCD soluion for ulra-resoluion visualizaion offered a high-performance display. In he fuure, we will expand he sysem o much more saisical variables like medians, anomalies. Mulivariae visual analysis for differen areas, variables and daases should be suppored in he fuure. We also expec o explore how o beer design range query algorihms on massive spaio-emporal climae daa, no only limied o window query. Acknowledgemens The auhors are exremely graeful o Liang Zhang, Huiming Zhao for sysem developmen and he Cener of Earh Sysem Science of Tsinghua Universiy for daa suppor. This research was suppored by he Naional High Technology Research and Developmen Program of China (863 Program) No. 2010AA References [1] Bin Wang. A ypical ype of high performance compuaion: earh sysem modeling: earh sysem simulaion [J]. Physics, 2009, 38(08): 0-0. [2] Keim D, Andrienko G, Fekee J D, e al. Visual analyics: definiion, process, and challenges [M]. Springer Berlin Heidelberg, [3] Schroeder W, Marin K, Lorensen B. The Visualizaion oolki: an objec-oriened approach o 3D graphics 1998[J] [4] "VisI Web Sie", Lawrence Livermore Naional Laboraory, hp:// [5] Wong P C, Shen H W, Johnson C R, e al. The op 10 challenges in exreme-scale visual analyics [J]. IEEE compuer graphics and applicaions, 2012, 32(4): 63. [6] Bo li. Algorihm research on geomeric range query [D]. Harbin Universiy of Science and Technology, [7] Shi Q, JaJa J F. Efficien echniques for range search queries on earh science daa [C]//Scienific and Saisical Daabase Managemen, Proceedings. 14h Inernaional Conference on. IEEE, 2002: [8] Same H. Foundaions of mulidimensional and meric daa srucures [M]. Morgan Kaufmann, [9] Moun D M, Park E. A dynamic daa srucure for approximae range searching [C]//Proceedings of he weny-sixh annual symposium on Compuaional geomery. ACM, 2010: [10] Fischer J, Heun V. Theoreical and pracical improvemens on he RMQ-problem, wih applicaions o LCA and LCE [C]//Combinaorial Paern Maching. Springer Berlin Heidelberg, 2006: [11] Bender M A, Farach-Colon M. The level ancesor problem simplified [J]. Theoreical Compuer Science, 2004, 321(1): [12] Lin J, Keogh E, Lonardi S, e al. Visually mining and monioring massive ime series [C]//Proceedings of he enh ACM SIGKDD inernaional conference on Knowledge discovery and daa mining. ACM, 2004: [13] Keim D A, Niezschmann T, Schelwies N, e al. A specral visualizaion sysem for analyzing financial ime series daa [C]//EuroVis. 2006: [14] Rew R, Davis G. NeCDF: an inerface for scienific daa access [J]. Compuer Graphics and Applicaions, IEEE, 1990, 10(4):