GeoDec: A Framework to Effectively Visualize and Query Geospatial Data for Decision-Making

Transcription

1 GeoDec: A Framework to Effectively Visualize and Query Geospatial Data for Decision-Making Cyrus Shahabi, Farnoush Banaei Kashani, Ali Khoshgozaran and Songhua Xing Information Laboratory (InfoLab) Computer Science Department University of Southern California Los Angeles, CA {shahabi,banaeika,jafkhosh,sxing}@usc.edu Abstract In this paper, we introduce GeoDec, an end-to-end system that enables geospatial decision-making by virtualizing the real-world geolocations. With GeoDec, first the geolocation of interest is rapidly and realistically simulated and all relevant geospatial data are accurately fused and embedded in the virtualized model. Subsequently, user can interactively formulate her abstract decision-making queries in terms of a wide range of fundamental spatiotemporal queries supported by GeoDec and evaluate the queries in order to verify her decisions in the virtual world prior to executing the decisions in real world. GeoDec blends a variety of techniques developed in the fields of databases, artificial intelligence, computer graphics and computer vision into an integrated three-tier architecture. We elaborate on various components of this architecture, which include an extensive, multimodal and dynamic data tier, an efficient, expressive and extensible query-interface tier and an immersive, flexible and effective presentation tier. I. INTRODUCTION Geospatial decision-making is a critical requirement and enabling component of numerous geospatial applications such as urban planning, emergency response, military intelligence, simulator training and serious gaming. With the abundance of the available geospatial data today (e.g., satellite and aerial imagery, maps, vector data, point data), arguably the most effective approach for geospatial decision-making at/about a geolocation is by virtualization of the real-world geolocation. By realistic modeling and simulation of a geolocation of interest, seamless embedding and fusion of a host of geotagged and timestamped real datasets associated with the geolocation and finally efficient execution and presentation of a basic (but expressive) set of spatiotemporal queries on top of the information-rich virtual geolocation, the decision-maker can immerse in and navigate the geolocation and intuitively formulate and evaluate various abstract decision-making queries in terms of the basic queries. For example, through this threestep virtualization process (i.e., simulation/modeling, data embedding/fusion and querying and query presentation), one can simulate a city such as New York City, embed the road network data, building point data, elevation data and microclimate data in the virtual city and use a combination of range queries, nearest neighbor queries, line of sight queries, etc., to evaluate decision-making queries such as if the Freedom Tower is built at Ground Zero, how will it affect the wind-vector field of the South Manhattan?. Using such an approach, decision-makers can effectively investigate and verify their decisions in an equivalent virtual geolocation prior to executing the decisions in real world. We assume and argue that such a virtualization-based decision-making technology must satisfy the R-A-I-S-E requirements in order to be effective: Realistic simulation, Accurate information fusion, Interactive query and access, Scalable infrastructure and Efficient time-to-build. Considering R-A-I-S-E as our design objectives, we have designed and developed an end-to-end virtualization-based geospatial decision-making system, dubbed GeoDec (Geospatial Decision-making). GeoDec blends a variety of techniques developed independently in the fields of databases, artificial intelligence, computer graphics and computer vision into a three-tier architecture. At the data tier (bottom tier), GeoDec hosts and maintains a large set of scalable data sources of various types including 1) a spatiotemporal database consisting of mostly static data such as 3D building models, terrain data, vector data, satellite imagery and raster maps, 2) an information mediator which provides a uniform interface to access the dynamic web sources containing geospatial data such as weather data and traffic data and 3) a stream server that serves real-time data streams such as video and audio. GeoDec provides a user-friendly wizard for users to efficiently simulate the geolocation of their interest in a semi-automatic fashion by constructing accurate and realistic 3D building models with texture and incorporating them into the data tier; hence, realistic simulation and efficient time-to-build. Moreover, at the data tier various automatic data fusion techniques are used to accurately combine the data sources relevant to the geolocation (e.g., by aligning vector data with satellite imagery) to ensure correct embedding of the data in the simulated geolocation. Once the geolocation is simulated and enriched with data, users can use the query evaluation and query presentation capabilities of the query-interface tier (middle tier) and presentation tier (top tier) of GeoDec, respectively, to interactively query the virtual geolocation. At the query-interface tier, GeoDec supports a wide range of basic (but sophisticated) spatiotemporal queries such as continuous range queries, shortest path queries, continuous neatest neighbor (knn) queries, trajectory queries, continuous line of sight queries and event

2 queries, which can be combined and applied to various fusions of data modalities. For example, with our running example the decision-maker can issue a continuous knn trajectory query to find the nearest strong air-flow areas as she travels through the streets of the virtual Manhattan. Moreover, the query-interface tier defines and implements a standard set of web-services interfaces through which different visualization components at the presentation tier can uniformly access and query the data. This ensures data independency and facilitates extensibility and scalability by allowing various public and/or proprietary visualization clients to be used as the GeoDec client. At the presentation tier, GeoDec client(s) are used to expose the virtual geolocation to the user and to allow her to formulate her queries and view the results of her queries once executed by GeoDec. We have ported a number of common public visualization clients such as Google Earth, Google Maps and Microsoft Virtual Earth clients on top of GeoDec to serve as GeoDec client. While these clients are designed for effective visualization of the virtual geolocation and the query result, they have limited capabilities for query formulation and only support a small set of simple canned queries (see Section II for more details). On the other hand, our proprietary GeoDec client supports both effective visualization and extensible query formulation. It adopts an optimal code and data placement scheme to minimize the data communication overhead between the server and the client, while placing sufficient data at the client side to allow for effective in-place query update. With this client, in addition to the standard interfaces, users can use a glove-based interface and/or 3D navigation devices to interact with the system. With such easy to use interfaces users are immersed in the virtual geolocation, which in turn enables intuitive decision-making. A preliminary version of this work appeared in a 4-page conference paper [27], where we introduced a limited version of GeoDec available at the time of the publication. Since then GeoDec has significantly evolved to an end-to-end system with an extended three-tier architecture including numerous new components, subsuming the original version. In particular, in this paper we cover the newly added features to the query and the presentation tiers: we extended the query-interface tier to support complex spatiotemporal queries, namely, line of sight queries, event queries, POI queries and parcel queries and we enhanced the presentation tier by introducing the level-ofdetail road network visualization, terrain modeling capabilities, support for 3D interface devices and modeling complex objects such as trees. The remainder of this paper is organized as follows. In Section II, we discuss various possibilities in placement of the code and data between the server and the client and explain the hybrid approach adopted by GeoDec. Section III describes the three-tier architecture and the components of GeoDec in detail. In Sections IV through VI, we elaborate on the techniques we used to implement the three-step virtualization process, namely, geolocation modeling, data fusion and query processing, respectively. Section VII describes two geospatial decision-making scenarios enabled by GeoDec. Finally, Section VIII covers the related work and Section IX concludes the paper. II. DATABASE-AWARE DISPLAYS VERSUS DISPLAY-AWARE DATABASES Before getting into details regarding GeoDec s features and capabilities, it is important to study its unique design vision and where it places GeoDec in the realm of geospatial applications with regards to GeoDec s view of scalability and interactivity. Recently, there has been a rapid increase in the availability of geospatial data, and this growth has motivated the launch of many geospatial services. With these services, a remote user issues a query asking for the visual representation (e.g., map) of a limited geographical neighborhood. Subsequently, the geospatial data pertaining to the queried area (e.g., segments of roads) must be retrieved from a database, transferred and then displayed on the user s graphic display. In particular, the geometric representation of this data must be eventually mapped to a limited number of pixels on the user s display device. Depending on where and how this mapping is performed, we identify two different approaches adapted by today s geospatial services: database-aware display and display-aware database. With the database-aware display approach, the server sends the relevant spatial data together with the corresponding metadata to the displaying client which then employs its local graphic facilities to render each and every geometric object. With this approach, the mapping is applied locally based on the resolution in which the data is being displayed. This potentially slow and exhaustive local rendering of data suits system architectures with a high communication bandwidth such as a single-user PC-based system that hosts both the data server and the client. Many professional GIS applications such as ESRI s ArcGIS Desktop products such as ArcInfo, ArcView and ArcEditor implement this approach [3]. To avoid the transfer of the bulky spatial data to the displaying client and to expedite the rendering task at the client side, online geospatial services such as Google Maps [6], Yahoo! Maps [10] and Microsoft Live Local [8] have adapted a different approach. In this display-aware database approach, the server renders the requested data as a raster image with the knowledge of the client s display resolution. Only this image, which is a visual representation of the data, is then transmitted to the displaying client. Hence, the client cannot access the retrieved data for further processing. That is, any consecutive query on the retrieved data (e.g., nearest neighbor query or asking for the name of a restaurant) in the client results in another handshake with the server which hosts the original dataset. This approach results in quicker data transfer and is appropriate for web-based systems where many users access data servers over limited bandwidth, mainly for visualization purposes. Hence, both of these approaches are appropriate for a specific application and a certain system architecture. A system implementing the database-aware display approach assumes that the client requires the entire data in its most detailed form. However, utilizing the display-aware database approach in an online service presumes that the client only requires a nice visual representation of the data. We believe that many of the next generation geospatial applications may require

3 Fig. 1. The GeoDec architecture a combination of both and hence hybrid or middle-ground approaches are required. In this article, we, for the first time, raise this issue and propose to utilize a hybrid approach for effective presentation of geospatial data and querying it. While no general solution can serve the requirements of all geospatial applications, we strongly believe that the effective presentation must be customized based on both the application requirements and the underlying architecture. The reason we devised this hybrid approach is because while GeoDec s three-tier architecture separates the server that hosts the data from the clients that display the data, the GeoDec s users need more than just the visualization of data and hence the transfer of a relatively larger dataset is preferable to the transfer of a simple raster format. Consequently, some user queries and interactions can be directly supported at the client side with no further data exchange with the server. At the same time, the architecture cannot handle a full-blown transfer and rendering of the entire dataset of the geographical area being analyzed. In the following section, we provide some more details regarding the three-tiers of the GeoDec system, each layer s functionalities and how they interact. III. GEODEC ARCHITECTURE The three-tier architecture has been extensively implemented in many database driven applications [17], [18]. Compared to the simpler two-tier client/server architecture (where the server level usually refers to the DBMS software responsible for handling data storage on disk pages and the client level handles the user interface), the three-tier architecture adds a middle layer sometimes called the application layer between the client and server. GeoDec adopts a three-tier architecture depicted in Figure 1. This design makes the interface independent of the inherent data model and facilitates scalability by allowing several visualization components to specify queries and receive the results back in a uniform language hiding the source of information. In this section, we study each layer and show how the interaction between these layers makes GeoDec a scalable tool for geospatial decisionmaking. A. Negaah-the Querying and Visualization Interface One of the main design goals of GeoDec is the provision of an immersive environment that enables users to interact with GeoDec and perform a wide range of spatio-temporal queries intuitively, in order to facilitate the process of decisionmaking. The key advantage of having a three-tier architecture is to allow several interfaces interact with our spatio-temporal database server (Section III-C) thus expanding user s experience working with a wide range of interfaces customized for users specific data and query needs. We have integrated several well-known geospatial interfaces such as Google Earth, Google Maps and Microsoft Virtual Earth into GeoDec and have also developed Negaah, our proprietary interface that provides a wide range of spatio-temporal queries not supported by typical web-based geospatial and mapping applications. Integrating multiple interfaces into the GeoDec s top layer allows us to overlay users query results on top of satellite or raster maps provided by Google or Microsoft servers and provide users with a well-known and easy to use environment to view their query results. Furthermore, users enjoy other features of such applications while interacting with query results generated by the GeoDec s engine. Although these interfaces are powered by highly scalable servers and a huge variety of geospatial data sources, they do not offer enough flexibility to implement the wide range of spatial and temporal queries desirable for a geospatial decision-making system like GeoDec. In other words, these interfaces can be used as powerful visualization tools but are not very suitable as querying interfaces. For example, although it is very intuitive for a user to get directions, users are not able to click on a building, road or most other geographic objects of interest and query the server for information related to those objects (e.g., the road length, speed limit and number of lanes or the area of each house and other information about the property). In order to fill this gap we developed our own interface Negaah, which offers significantly more customized queries and more intuitive interaction to users. Negaah is a visualization interface for GeoDec that allows users to navigate and interactively query the 3D environment in real-time. GeoDec provides users with greater querying flexibility by allowing them query geospatial data based on a user-selected area and time interval. The user can selectively query and display different layers of information, and move forward or backward in time. Negaah also supports more sophisticated queries such as calculating the line of sight along a path between two points considering constraints imposed by the shape of the land surface or other 3D objects or can find the K-nearest objects to a point taking into account the land surface information. Examples of data sources that Negaah can query include 3D models, vector data (e.g., roads, parcels), traffic information, elevation data, point data (such as building names) and moving objects/trajectories (such as public transportation routes). Sections VI and VII offer more in-depth details regarding GeoDec s querying capabilities. As Negaah is GeoDec s main visualization interface, we have integrated all querying and visualization functionalities GeoDec can offer in Negaah. A subset of these features are available on other interfaces based on their level of support for the necessary interactions and querying. Depending on certain limitations imposed by off-the-shelf products, it is clear that some GeoDec features cannot be implemented with these interfaces. For instance, Google Maps or Microsoft Virtual

4 Fig. 2. Negaah s class hierarchy. Earth do not allow users to specify a bounding box for their spatial queries or use a time slider to move back and forth in time while viewing the results of a temporal query. Negaah is developed using Microsoft.NET C# framework. All rendering and visualization modules are also implemented using OpenGL. Figure 2 describes the hierarchy of Negaah s components and how they interact with each other. In this hierarchy, the Graphical User Interface implements all the windows forms and components (e.g., buttons and sliders). The Interaction Component provides all the functionalities to enable user interaction with Negaah, through traditional input devices (e.g., keyboard and mouse) or some novel and intuitive devices (e.g., 3D Space Explorer from 3D Connexion [1] and Data Gloves from Fifth Dimension Technologies [4]). The OpenGL Rendering Component handles all the computer graphics rendering issues, which include OpenGL initialization, lighting, shading and camera setup. The Camera object is used to set up the position, orientation and the viewpoint from which the scene is viewed. Camera is defined as a static class since we do not need multiple instances of camera objects. User interactions with Negaah result in various spatiotemporal queries. These queries are sent to our middleware layer (Section III-B) by the Query Scheduler in the form of web-service requests. The Query Scheduler is designed as an abstract class with a virtual query function implemented by all of its derived classes (e.g., Buildings Query Scheduler, Moving Objects Query Scheduler ). Each specific query scheduler is dedicated to querying a certain object data type through its Query Units. The main responsibility of each query unit is the visualization of a certain data object in a query result. For instance, upon querying buildings in a region, every building query unit is in charge of drawing the geometry of a building as well as its texture mapping. unified format. When users interact with any of the supported GUI s, different queries are sent to Jooya in the form of webservice requests. Usually, Jooya formulates the query results into an XML file for data exchange and checks client s credentials before sending a query, hence improving the database security. Depending on the user s query type, Jooya sends a query to our spatio-temporal Database Manager (Darya) through low-level SQL commands via ODBC connection, to Prometheus [31], our information mediator (which provides a uniform query interface to a set of web sources that contain information about a geographic location) or to our video server called Yima (which is a scalable real-time streaming server) [29]. Examples of queries sent to Darya are queries for infrequently updated or bulky data such as road network data, video metadata, and 3D building models. On the other hand, Jooya uses Prometheus to query highly dynamic data such as live traffic information obtained via a variety of web sources or uses Yima to request for stored surveillance video. Knowing the position of all cameras, all surveillance videos are indexed based on their spatial and temporal information. Figure 3 shows part of an XML response Jooya generates to answer a vector data query where each road segment is represented with a placemark tag. For each placemark, the XML file includes several important attributes such as the end point coordinates and the road type, where different colors are assigned to different types of roads. Jooya s XML result conforms to Google Earth s KML schema, hence enabling any visualization layer that can support KML to sit on top of Jooya for its integrated query and access needs. Jooya, however, would need to include a customized query-interface blade for each new GUI. This enables Jooya to act as an interface for web-based geospatial GUIs such as Google Earth as well. B. Jooya-the Middleware Layer As discussed in Section III-A, one of the key features of GeoDec is allowing several independently developed graphical user interfaces to query our spatio-temporal database layer. In order to create a conceptually simple yet effective way of allowing such interchangeability, we developed Jooya, our middleware layer which enables various GUI s to interact with our database. Jooya offers a universal way of specifying the type of query (e.g., range, shortest path, nearest neighbor, etc.), as well as its parameters and retrieves the results back in a Fig. 3. Part of an XML response of a vector data query returned from Jooya

5 C. Darya-the Spatio-temporal Database Server The back-end of GeoDec termed Darya is the system s spatiotemporal database engine running Oracle 10g. This module is in charge of managing spatio-temporal data stored in a database, a task that includes data modeling, storage and retrieval (querying). Darya stores a variety of data sources such as road network information (i.e., vector data), 3D building models, terrain data, gazetteer data, satellite and aerial imagery and raster maps. The main advantage of Darya is enabling fast and efficient access to multi-dimensional datasets that change infrequently. One of the key features of Darya is the fact that the majority of its data is tagged by both time and space thus allowing a wide range of spatial, temporal and spatio-temporal queries to be efficiently supported by GeoDec. For instance, all buildings are tagged by their valid time interval (i.e., date of creation and date of demolition, if any). Therefore, users can select an area of interest as well as a time duration and use Negaah s time slider to move back and forth in time and see the changes in the querying region and within the given time frame. Table I gives the schema of the buildings table stored in Darya. The GEOM attribute is used to stored each building as an Oracle Spatial geometry type and define spatial index structures for it. The FROM DATE and TO DATE attributes also add the temporal information to the buildings regarding their construction and demolishing time. These spatial and temporal attributes allow us to efficiently perform spatiotemporal queries on buildings. In the database layer, our main Name Data Type Size Not NULL ID NUMBER TRUE NAME VARCHAR2 32 FALSE COMP HGT NUMBER FALSE ROOF HGT NUMBER FALSE FROM DATE TIMESTAMP 6 FALSE TO DATE TIMESTAMP 6 FALSE DESCRIPTION VARCHAR2 128 FALSE TEXTURE URL VARCHAR2 128 FALSE GEOM MDSYS.SDO GEOMETRY FALSE TABLE I SCHEMA OF THE BUILDINGS TABLE IN DARYA focus is on storage and querying spatial data such as bulky vector data (e.g., road networks). Mainly, Darya embeds a multi-resolution vector data compression technique [21] which effectively compresses the result of query windows, taking into account the client s display resolution. In addition, we have also studied and implemented several query optimization and efficient indexing techniques for spatial databases which include Voronoi Based KNN Search in Network Databases [22] and the Spatial Skyline Queries [30]. We plan to incorporate these new techniques in Darya to support more complex queries in near future. IV. CONSTRUCTING A VIRTUAL SPACE As a sophisticated querying and visualization tool, Negaah is capable of modeling 3D buildings, terrains and more complex objects such as trees. In this section, we first discuss how a virtual space is modeled in GeoDec and then detail how terrain or other complex objects such as trees or cars are added to the model. Modeling a virtual space in GeoDec consists of the following four phases. Background image construction: during the first phase, a large satellite image is mapped on the ground as the background. This calibrated image servers as the building block for the second phase of the modeling process. 3D object construction and indexing: in this phase, 3D building components of the virtual space are constructed from the background image using a semi-manual technique and are added to the model [24]. Terrain modeling: during this phase, the flat model constructed is enhanced with terrain information to create a more realistic model as well as enabling certain queries in GeoDec to use the terrain elevation data. Texture mapping: during the last modeling phase, high resolution texture images are added to the model in order to create realistic objects in space. We now discuss the above phases in more detail and briefly describe the techniques used in each phase. A. Modeling Buildings GeoDec utilizes a semi-automatic software previously and independently developed [24] to model highly detailed 3D structures and buildings. Using two images taken from the same area with different angels, this software automatically calculates the correct height of a building by asking the user to identify a few corners of a building with mouse clicks giving users the option of working with a single image at a time. At the same time, by using basic information such as one side and one corner of objects identified by the user in a semi-automatic process, highly complex polygonal roof shaped buildings can be modeled (see Figure 7). These models are originally created using the VRML (Virtual Reality Modeling Language) format, a standard file format for representing 3D interactive vector graphics. However, in order to facilitate internal file exchange and querying, we have converted our models to XML files. Once the 3D models are constructed, they are stored and indexed in our database using Oracle s spatial indexing features. The key reason behind storing the 3D models in the database is to enable efficient spatial querying of the objects in Negaah. As we discuss in Section VI, GeoDec supports a variety of sophisticated spatial queries that cannot be evaluated efficiently unless objects are indexed using efficient spatial index structures such as R-trees. Storing the objects in our spatial database also allows us to efficiently retrieve a subset of objects relevant to a query. For example, while interacting with a virtual space, GeoDec users can query all buildings that are located inside a rectangular region selected by the user. The query is efficiently evaluated using the underlying spatial index structure to retrieve only the objects that are spatially related to user s query (e.g., those located inside the region or intersect with the region s boundary). B. Modeling the Terrain Modeling the terrain differs from modeling a building due to the irregularity of the land surface. A terrain could be defined

6 Fig. 4. TIN model of Yosemite National Park in Negaah Fig. 5. Modeling a tree as a 2D surface in 3D space where a height value is assigned to every point in the terrain. Strictly speaking, it is not possible to model a terrain thoroughly unless we measure the height of every point on that terrain. Therefore, the common approach is to approximate the height of most of the points by interpolation based on the sampled points actually measured. Triangular Irregular Network (TIN) is the most popular model to construct a polyhedral terrain for this purpose. TIN is generated from DEM (Digital Elevation Model) of sampled ground positions with 3D coordinates at regularly spaced intervals [23]. Based on these samples, TIN constructs the surface triangles by connecting the points as non-overlapping triangles, usually known as Delaunay Triangulation [12]. In most cases, the data size represented by a TIN is very large: A region of 10km by 10km may contain millions of triangular facets gathered at 10- meter sampling interval and a sub-sampling process is often needed. In order to create an ideal platform for visualizing the terrain data, we have imported three DEM data sets of Eagle Peak area in Wyoming, Bearhead area in Washington and the Yosemite National Park in California to GeoDec. Figure 4 illustrates the TIN model constructed for a mountain area in the Yosemite National Park. Supporting spatial queries such as K-Nearest Neighbors (KNN) on a land surface created using the above process is very challenging due to the fact that computing the shortest path becomes very costly. In order to avoid such huge costs, we have built two complementary index schemas known as Tight Surface Index(TSI) and Loose Surface Index(LSI) as well as a data structure called Surface Index Tree to enable efficient KNN retrieval and prune the search space [28]. Although the actual surface KNN query is not supported on GeoDec yet, we plan to incorporate this functionality into GeoDec for real applications in the near future. C. Modeling More Complex Objects There are other types of objects present in a geographic area that cannot be modeled by the process described above. Objects like cars, trees and flags usually require a different modeling process due to the irregularity or complexity of their shapes. In case of trees, the texture information is also very complex. As a proof of concept we have integrated several models of trees in Negaah. Similar to the method that has been adopted in many 3D computer games, the tree modeling is performed using textures with alpha channels (transparency). Modeling of such a base object is simple, usually as 4 or 6 sided cylinder. As illustrated in Figure 5, we first get two Fig. 6. Video fusion identical texture images with alpha channels. These images can be generated from photos taken from the trees by carefully extracting the object s texture from the image. Next, we map these two textures onto two perpendicular crossed planes and set the transparency value to zero. This allows objects located behind the trees to be partially visible if they are occluded by a tree. D. Adding Textures Textures enrich the visual complexity of a scene with elements that are not captured in the static 3D geometry of a model. Textures capture both the static and dynamic appearance of the scene surfaces to make the 3D world appear realistic. Adding textures to our 3D models is the result of the work independently done by Neumann et al. in an Augmented Virtual Environment system [26]. Starting from the 3D building models, imagery is projected onto these models and aligned or calibrated based on 3D and 2D landmarks and correspondences. Static textures are captured by still or video images. The acquisition method and texture mapping process employs a base texture buffer and image warping transformations. Dynamic textures are captured as live video streams. We dynamically fuse video textures from network cameras on a 3D model to reveal the spatial and temporal relationships between the video streams. Dynamic textures should originate from fixed cameras and are mapped directly onto the faces of buildings or the ground. However, most of the time, due to the perspective projection transformation inside the camera, the videos may not fit into our models correctly. Therefore, some manual one-time preprocessing is needed to analyze each video and select some fixed points in the video as control points (usually located on the sides or corners of a building or the ground). These control points are then positioned according to their corresponding points in our models. Finally, the video is distorted to fit our models. The more videos we have, the more realistic dynamic texture will represent the environment.

7 The dynamic textures used in GeoDec can be generated from recorded videos or from live camera feeds. Figure 6 illustrates how five different surveillance camera feeds are fused together and texture mapped onto the 3D model. E. Manipulating GeoDec Models Integrating the above phases into a new model construction wizard in GeoDec allows users to conveniently model a new geographic place and navigate through the environment by going through the phases discussed in the beginning of this section. Once a virtual model of a geographic space is constructed, users can save the model for future access to enrich the model incrementally. For instance, upon the availability of more complex building models or higher resolution satellite imagery, the model can be opened, modified and stored again with the new changes. This simple process allows GeoDec to act as a tool to model a geographic space efficiently. Once the necessary data sources such as 3D models, terrain data, road network data, etc. are available for a region, a new model can be created and saved in GeoDec supporting every feature available in GeoDec from previously constructed models. Utilizing the process described in Section IV, we have constructed detailed virtual representations of the University of Southern California, main campus as well as an area in downtown Los Angeles. model. Our vector-to-imagery conflation technique exploits a combination of the knowledge of the road network with image processing techniques. We first find road intersection points from the road vector dataset. For each intersection point, we then perform image processing in a localized area to find the corresponding intersection point in the satellite image. Finally, we compute the transformation matrix from the two sets of intersection points to align the vector data with the imagery. The running time is dramatically lower than traditional image processing techniques due to the limited image processing used. Furthermore, exploiting the road direction information improves both the accuracy and efficiency of detecting edges in the image. To integrate maps with satellite imagery, we utilize common vector datasets as glue. We first identify road intersections on imagery by the technique described above, and then we also detect the road intersections on maps [15]. Finally, we apply a geospatial point pattern matching algorithm to find matches between the two point sets. Now that we have a set of matched control point pairs, we partition the map into small triangles, and utilize the local transformation matrix to transform the map piece by piece. By doing so, we find the location (i.e., the geocoordinate) of the map and align the map with satellite imagery. Our proposed approach facilitates the close integration of vector datasets, imagery and maps, thus allowing the creation of intelligent images that combine the visual appeal and accuracy of imagery with the detailed attribution information often contained in diverse maps. V. GEOSPATIAL DATA FUSION Creating a realistic model of a geographic space allows users to be immersed in a realistic environment. However, enabling decision-making requires enriching the model with various layers of geospatial data such as vector data (e.g., road network information, parcel data) and raster imagery. In this section, we briefly study our approaches for integrating these heterogeneous (and sometimes inconsistent) data sources into our models. A. Road network and map fusion In order to generate a useful visualization of integrated geospatial data (i.e., vector data, satellite imagery, parcel data and raster maps), a simple superimposition of various geospatial sources is not sufficient to align the sources with each other for the following reasons: (i) For many raster maps, the geocoordinates of the maps are unspecified (ii) For other geospatial data with specified geocoordinates, the projections and transformations used to produce the maps, orthorectified imagery or vector data are unknown. The current commercial tools to solve this problem require heavy user interventions. It is simply too slow and tedious to fully exploit the rich sources of information available in geospatial datasets. Towards this end, we have developed a set of techniques for automatically aligning maps and road vector data on orthorectified imagery [13] [15]. In order to allow integration of the aligned data with the 3D model described in Section IV-A, the maps and road vector data are aligned to one of the satellite images used to construct the 3D B. Data Integration and Efficient Geospatial Querying In order to efficiently and accurately integrate a wide variety of information about a geographic location, GeoDec utilizes a geographic data integration system called Prometheus [32]. Prometheus organizes the available information sources in a domain hierarchy containing well-known domain concepts, such as, satellite image, map, or vector data. Moreover, Prometheus also models different types of integration operations and their effects. Examples of the integration operations include Overlay and Align. The Overlay operation may result in two information layers not aligning with each other while the Align operation described in Section V-A improves the alignment between two data layers. When GeoDec receives a request to retrieve the data for a geographic location, Jooya sends a request to the data integration system to obtain different types of information (i.e. point data, vector data, raster maps, and satellite imagery). Prometheus determines the relevant sources for the requested data, retrieves relevant data from the sources, performs necessary alignment or integration operations, and returns the integration information to Jooya which is then sent back to Negaah to be displayed on the 3D model. VI. QUERYING WITH GEODEC One of the most important features of GeoDec is its querying capability, which differentiates it from many other geospatial visualization applications. In this section, we discuss some of the queries currently integrated within GeoDec.

8 Fig. 7. The 3D buildings (left) augmented with texture and background image (right) Fig. 8. GeoDec s querying features As most client side user interface programs, Negaah initializes and sends several requests to the middle layer based on users interaction. Each request in GeoDec is formed as a spatio-temporal query over different collections of data. Several criteria could be used to classify these query types. The first criterion is type of the object data set (similar to several data Layers in Google Earth). Based on this criterion, Negaah allows querying a large set of object types such as 3D buildings, point data (textual information associated with buildings such as the name of the offices in that building), parcels, points of interest (e.g., restaurants and hospitals ), live traffic, road network, photos, videos and moving objects (such as public transportation trams). Sometimes, one query result concerning a particular object data is of interest while in other scenarios, users are more interested in querying a group of heterogeneous objects simultaneously. For example, a query about parcels, which returns descriptions and information about several contiguous area of land bounded by polygons is interesting to people in urban planning or the real estate business. However, a traveler may be concerned about queries on traffic, road network and shortest path together to arrange his/her itinerary. To facilitate the rapid, effective and meaningful analysis of this integrated information, a good visualization is essential to the design of Negaah. With our design, the screen is divided into a query panel and a visualization panel as shown in Figure 8. In order to query Negaah, users interact with the query panel while the query results are displayed in the visualization panel. Furthermore, some query results might need special visualization effects, like the line of sight query. With this query, given the perspective from the query point, the region of interest is color coded into the visible areas on the flat ground (painted green) and on the building faces (painted blue) as illustrated in Figure 9. Queries supported by GeoDec can be divided into the following four categories based on how they are expressed by users. 1) Range Query 2) Point Query 3) K-Nearest Neighbor Query or KNN 4) Trajectory Based Query A range query is a standard spatial query which returns all the objects inside the query range, both the window and circle queries are supported in Negaah. Examples of range queries supported in GeoDec are querying the system for the information associated to each building, the road information or live traffic within a user-specified bounding box. A point query allows the user to specify a point as the query origin and query for any information in the vicinity of the query point. An example of a point query is choosing a point on a building and querying the system for any information related to the building (e.g., building name or building address). The line of sight query is another example of a point query where Negaah returns the color classification of the area based on their visibility from the querying point. The KNN query is utilized to find the K objects that have the nearest distance to a given query point, usually in Euclidean spaces. However, some unconventional and variants of KNN query, such as continuous KNN query in the spatial networks or road network based KNN queries, are also supported in GeoDec. For instance, users can search for the three closest points of interest (e.g., restaurants or financial institutions) to a query point. The trajectory based query is one of the novel query modes that is of growing interest to the database community. For the trajectory based queries, two different variants are considered namely moving object queries and user defined trajectory queries. In moving object queries, the user query generates a

9 Fig. 9. Two snapshots from the line of sight query result set which is in the form of a trajectory showing object movements for a particular duration of time. With the user defined trajectory queries, users actually specify their queries by drawing a trajectory. The former type is usually used to track a moving object such as a campus tram continuously updating its position via GPS signals over a period of time. User defined trajectory based queries, on the other hand, consider the trajectory or path as an input for a query. The query results are then updated depending on the query point s position on the trajectory. For example, a user creates a path from his/her home to school by drawing it on Negaah and querying the system for all the nearby points of interest along this path or the traffic and incident information near the same trajectory in the past hour. As the user moves along the trajectory, the query result set is automatically updated. As a geospatial decision-making tool, GeoDec allows users to visualize, interact and query a wide range of objects present in a geographical space such as buildings, moving objects and photos. GeoDec allows users to choose from any of the previously discussed query modes to query an object, resulting in various combinations of object queries. Taking a simple object type of photo as an example, a photo could be queried in several query modes. A range query (circle) will return all the photos within a certain radius from the query point. A nearest neighbor query, returns only one photo that is closest to the query point. For a moving object query, one could track a path made up from photos uploaded continuously from a GPS equipped mobile phone. Finally, for a user defined trajectory query, the user gets all the nearby photos along a specific user drawn path. Aside from the variety of spatial query modes discussed above, GeoDec allows users to add a temporal dimension to their queries in order to filter the results within a certain time interval. Utilizing a time slider, users can then move back and forth in time to change the query result based on objects temporal information. For instance, users can trace the sequence of pictures taken in a region within a specified time frame. The wide range of queries supported by GeoDec enables users to combine the information provided by a group of different queries to create more sophisticated decision-making scenarios. In the following section, we discuss how GeoDec enables high-level decision-making by combining the queries discussed above. We use two intuitive scenarios to discuss how GeoDec facilitates the process of decision-making. VII. EXAMPLE GEOSPATIAL DECISION-MAKING SCENARIOS Our main objective in constructing GeoDec is enabling geospatial decision-making. In this section, we discuss two examples that illustrate how GeoDec is used as a decisionmaking tool. We first discuss the concept of GeoDec s event queries that exploit dynamism in the underlying geographic domain using corresponding geospatial data (spatiotemporal data). We show how GeoDec integrates a heterogeneous set of data sources and queries to model various events taking place in a geographic place. Next, we discuss how GeoDec has been used in a real-world scenario as part of a collaborative research with the University of California at Los Angeles. A. Querying Events In a dynamic geographic domain, requesting information from the system about an event is transformed to a set of queries on the geospatial datasets related to a single event or multiple related events happening in sequential or arbitrary order. For example, querying about a criminal suspect getting on a bus in a certain time period is transformed to spatiotemporal query on all tram trajectories and corresponding video feeds in a given time and space. Note that an event is a much more abstract concept compared to any of the queries that GeoDec supports. However, the key idea here is to blend several query scenarios together to transfer some information to the user that was not possibly captured by any single query currently supported in GeoDec. The following scenario represents how GeoDec can be used in providing as much information as possible about an event taking place in a location modeled by GeoDec. The tram service and public transportation department at the University of Southern California tracks its trams locations every 2 seconds from 7:30AM to midnight every academic year for a fleet of 47 vehicles. With GeoDec, we use a wrapper tool to retrieve this massive data from its corresponding online source and store it in Darya for querying purposes.

10 Fig. 11. Three-tier architecture of the parallel systems Fig D view of the tram trajectory For our scenario, suppose the public safety office of the university knows that after an incident took place, a criminal suspect has got on a bus with the tram ID 709 at a time between 4:45PM and 5:00PM on December 6th, The officers would like to verify whether the criminal suspect matches the description of their most wanted man. This scenario requires Negaah to display the trajectory of a tram of interest for a user-specified date and time period. Furthermore, GeoDec performs a spatio-temporal join with all surveillance videos stored in the system to find the relevant footage possibly capturing the passenger and tram activities for the given tram trajectory. The result of such a query is illustrated in Figure 10 where the path shows the trajectory of the tram specified by its tram ID in the requested time frame. Using a time slider, users can move back and forth in time to find the exact location of the tram at any given time and also to check whether video surveillance is available for any location in the vicinity of the current tram location. These videos are displayed as clickable icons along the trajectory allowing users to watch a video footage that might have captured the incident by clicking on a camera icon. Finding the relevant video footage is a very efficient process given the underlying spatio-temporal index structure utilized by GeoDec. Note that an abstract notion such as a crime is broken into a set spatiotemporal queries on Darya so that different aspects of the event are captured. Providing users with the results of such a query can significantly increase the chance of obtaining information on the incident and can drastically reduce the total number of suspects to look for. The result of such a query is then rendered by Negaah as a set of connected line segments associated with time attributes showing the tram path (trajectory) in the given time interval. Several icons are added to the query result notifying the user that surveillance video is available for the given location that might have captured the event during the given time frame. B. User Participatory Decision-Making The Urban Sensing Project [9], at the University of California Los Angeles is dedicated to develop cultural and technological approaches by collecting data from mobile sensors. Furthermore, publishing and sharing such sensor data become much easier and more immediate due to ubiquity of wireless access. In the Urban Sensing project, users participate in obtaining information about a geographic environment by using their cell phones to take images, capture video or measure the noise level (using noise sensors connected to their phones) for a particular location. In collaboration with Urban Sensing project, we explore GeoDec s powerful geospatial querying capability to utilize the spatially and temporally tagged data (e.g., images) gathered by private and municipal monitoring of different geographic places to enable geospatial decisionmaking as well as creating an information rich environment. 1) System Architecture: The urban sensing project itself consists of another three-tier architecture parallel to our GeoDec s architecture discussed in Section III. Programs running on a large number of mobile sensors constitute part of the client interface, and a web server is deployed as a middle layer and the database layer termed Sensorbase, stores all the raw data collected by cell phones. We term the middle layer the Collection Server. Any interaction between the two systems happen at the middleware layer between Jooya and the collection server as depicted in Figure 11. With this new system, the clients include both Negaah and mobile sensors, and the middle layer consists of Jooya and the collection server communicating in order to interact with Darya and the Sensorbase database servers. For the integration of the two systems, we faced an interoperability challenge. Sensorbase runs on mysql, a nonspatial DBMS and hence indexing the objects based on their spatial information was not supported by Sensorbase. In order to resolve this issue, upon the arrival of a new query, Jooya polls Sensorbase to update Darya based on the recent objects additions to Sensorbase. During this process, after Jooya has passed the identity check and been granted an authorization by the collection server, it retrieves new sensor data from Sensorbase and translates its spatially tagged attributes to an appropriate format for insertion and indexing in Darya. While the above scenario resembles a pull operation, we could alternatively use a push operation to instantly update Darya upon the arrival of any new photo. However, we did not take this approach due to some security concerns. 2) Querying User Participatory Data: One of the most dominant data types captured by Urban Sensing users is image datatype. These images are tagged both spatially and temporally and hence contain longitude, latitude, elevation, time taken and sensor identity attributes. All these different types of data are submitted by GPS-equipped mobile phones (through