Lecture 5: Spatial Data Management in GIS Dr. Bo Wu lsbowu@polyu.edu.hk Department of Land Surveying & Geo-Informatics The Hong Kong Polytechnic University Contents 1. Learning outcomes 2. From files to database 3. Definition of spatial database 4. Evolution of spatial database 5. Fundamental database elements 6. Spatial database design 7. ESRI GeoDatabase 8. Issues for spatial database creation 2011/10/12 2
Learning Outcomes By the end of this lecture you should be able to: Outline why databases are important in GIS Explain how the relational database model works Describe how to set up a relational database Know the procedure of database design Explain how to create a database in a GIS List important considerations for GIS databases Know the basic structure of ESRI GeoDatabase 2011/10/12 3 When do we store data? During and at the end of a session of data acquisition, editing, updating, or processing Communication (e.g. satellite-ground) Data exchange Spatial Database Archiving What do we store in a database? Anything important for GIS: spatial, attribute, and topological information Where do we store data? Mass media: disks, tapes, CD-ROMs, and optical discs Working media: hard discs, memory, and radio waves Maintenance and security HOW do we store data? Files Database Data Store Issues 2011/10/12 4
Data Store in Files File File is a collection of organized records of information A record has usually a record number and record content ASCII Files Use ASCII characters to represent information ASCII (American Standard Code for Information Interchange) characters: 0, 1,..., 9, A,..., Z, a,..., z, +, -,..., @,... Binary Files Organize information according to bits and a combination of bits, and are in a computer-readable format Record # Record Content 1 x 1,y 1 2 x 2,y 2 3 x 3,y 3...... 2011/10/12 5 Examples of Data Files Binary File ASCII File 2011/10/12 6
From Files to Database Example - Assuming there are ten cities with different elevation: H 1 =4m, H 2 =1004m, H 3 =820m, H 4 =640m,..., H 10 =20m A simple list file 4 1004 820 640... 20 An ordered sequential file 4 20 640 820... 1004 Direct indexed files Low elevation (L): 1 ~ 200m Medium elevation (M): 201 ~ 700m High elevation (H): 701 ~ 2000m Easy to generate and edit Slow to search specific data Search became relatively easy Not easy to update This is actually a small database Easy to search Easy to update 2011/10/12 7 Why We Need Database in GIS The problems with the traditional approach to data management Data redundancy (the unnecessary repetition or duplication of data) High maintenance costs Difficulties in moving from one system to another Data-sharing difficulties Lack of security and standards Lack of coherent and integrated management of data A different version of the visitor s details may be stored in each of the separate systems 2011/10/12 11
What Is a Spatial Database Database: is a large collection of data in a computer system, organized so that it can be expanded, updated, and retrieved rapidly for various uses. It could be a file or a set of files. Spatial database: stores GEOREFERENCED data. For example, buildings with their locations, bank account holders with addresses. 2011/10/12 12 Generally Accepted Definition of Spatial Database Is a spatially enabled DBMS (Database Management System) system, with additional capability to handle spatial data It offers Spatial Data Types (SDTs) in its physical data representation (data model) It supports special query language, e.g. SQL (Structured Query Language), for efficient manipulation of spatial data and geometric operations It supports efficient spatial indexing and effective algorithms for handling spatial joins 2011/10/12 13
A Simple Relational Database Data are organized in a series of tables, each of which contains records for one entity. Tables are linked by common data known as keys Queries are possible on individual tables or on groups of tables 2011/10/12 14 Early Spatial Database: Loosely Coupled Approach Characteristics A relational DBMS or some components of it for descriptive data A specific module for spatial data management Usually proprietary Examples: ArcInfo (ESRI), TiGRis (Intergraph) Drawbacks: The coexistence of heterogeneous data model, which implies difficulties in modeling, use and integration Since proprietary, hard to integrate and interoperate DB Files 2011/10/12 17
Current Spatial Database: Extended DBMS Characteristics: New spatial data types (point, line, polygons) are handled as base alphanumeric types The query language SQL is extended to manipulate spatial data and descriptive data Many other DBMS functions, eg. spatial indexing, query optimization are adapted so as to handle geospatial data efficiently Being Open in data model and architecture Examples: ArcSDE (ESRI), Oracle Spatial Spatial & alphanumeric types 2011/10/12 18 Entity Fundamental Database Elements An entity is a phenomenon of interest in reality that is not further subdivided into phenomena of the same kind eg. city, building Entities in reality modeled in a spatial database as Objects Object An object is a digital representation of all or part of an entity Entity Types An entity type is any grouping of similar phenomena that should eventually get represented and stored in a identical way. (e.g. streets, buildings, rivers, vegetation 1st step in DB design is the selection and definition of entity types to be included 2nd step of DB design is to choose an appropriate method of spatial representation for each entity type 2011/10/12 19
Fundamental Database Elements (cont'd) Spatial Object Type The digital representation of entity types in a spatial DB Classification is based on the spatial dimensions: 0,1,2,3D Point (0-D), line (1-D), area/polygon (2-D), volume (3-D) Object Class An object class is the set of objects which represent the set of same entities Attributes An attribute is a characteristic of an entity Usually non-spatial, can be stored as alphanumeric values Are presented as columns in attribute tables in the DB Attributes Values The actual value of the attribute that has been recorded in the DB An entity type is almost always labeled and known by attributes Are usually presented as cells in attribute tables 2011/10/12 20 Fundamental Database Elements (cont'd) Data Type The attribute of a variable, field or column in a table that determines the kind of data it can store Common data types include character, integer, decimal, single, double and string Layers Spatial objects can be grouped into layers, also called themes One layer may represent a single entity type or a group of conceptually related entity types Behavior A set of rules define how an objects can be edited and drawn Relational Join An operation by which two tables are related through a common field, known as a key Spatial Join A type of table join operation in which fields from one layer s attribute table are appended to another layer s attribute table based on the relative locations of the features in the two layers 2011/10/12 21
Fundamental Database Elements (cont'd) Database Model Is a conceptual description of a DB defining entity type and associated attributes Georelational Data Model A spatial data model that represent spatial features as an interrelated set of spatial and attribute data The georelational data model is the fundamental data modal used in ArcInfo Geodatabase Data Model An object-oriented data model introduced by ESRI that represents spatial features and attributes as object and the relationships between objects A geodatabase can store objects, such as feature classes, feature dataset, non-spatial tables, and relationship classes 2011/10/12 22 Database Design Considerations Almost all entities in reality have a 3-D spatial character, but not all dimensions may be needed Highway pavement actually has a depth which might be important, but is not as important as the width, which is not as important as its length Representation should be based on the types of process that the application may ultimately utilized Vector vs. Raster Map scale of the source data is important in constraining the level of detail represented in a DB On a 1:20000 map individual building are not visible 2011/10/12 24
Database Design Considerations (cont d) What storage media to use? How large is the database? How much can be stored online? what access speed is required for what parts of the database? How should the database be laid out on the various media? What growth should be allowed for in acquiring storage devices? How will the database change over time? Will new attributes be added? Will the number of features stored increase? How should the data be partitioned - both geographically and thematically? Is source data partitioned? Will products be partitioned? 2011/10/12 25 Database Design Considerations (cont d) What security is needed? Who should be able to create - new attributes, new objects? Who should be able to edit and update? Should the database be distributed or centralized? If distributed, how will it be partitioned between hosts? How should the database be documented? How should database creation be scheduled? Where will the data come from? Who determines product priorities? Who is responsible for scheduling data availability? 2011/10/12 26
Spatial Database Design 1 - Conceptual Conceptual design Software and hardware independent Describes and defines entities and spatial objects Identifies how entities will be represented in the database Selection of spatial objects types points, lines, polygons, raster cells Should highway segments be explicitly linked in the DB? Should a building be represented as an area or a point? 2011/10/12 27 Spatial Database Design 2- Logical Logical design Software specific but hardware independent Translation of the conceptual model into the data model of GIS Determined by database management system 2011/10/12 28
Spatial Database Design 3 - Physical Physical design Both hardware and software specific Related to issues of file structure, memory size and access requirements 2011/10/12 29 Spatial Database Design Physical Design Logical Design Conceptual Design 2011/10/12 30
Bad Aspects in Database Design Omitted data No update potential Inappropriate representation of entities Lack of integration between various parts of the database Unsupported applications 2011/10/12 31 General Steps in Spatial Database Design the ESRI Geodatabase Approach 2011/10/12 32
Step 1: Model the User s View Things to do: Identify organizational functions Identify the data required to support the functions Organize the data into logical sets of common features 2011/10/12 33 An Example: System Function Diagram for LSIS 2011/10/12 34
Step 2: Define Entities and Relationships Things to do: Identify and describe entities that you model Identity and describe the relationships among these entities Document the entities and relationships with diagrams (UML, Data Flow Diagrams) 2011/10/12 35 An Example: System Entity Descriptions for LSIS 2011/10/12 36
An Example: E-R Diagram for LSIS E-R Diagram: Entity Relationship Diagram 2011/10/12 37 Step 3: Select Geographic Representations 2011/10/12 38
Select Geographic Representations Things to be considered: The feature might be represented on a map The shape of the feature might be significant in performing geographic analysis (e.g., tracing water pipe network) Accessing one feature from another by relationships Features will have different representations at different map scales Textual attributes will be displayed on the screen or map products (e.g., labels, annotations) Point, Line, Polygons, Surface, Images..??? 2011/10/12 39 An Example: Geographic Representation in LSIS Same entity, different symbol for different map scales 2011/10/12 40
Step 4: Matching Entities to GeoDB Data Model 2011/10/12 41 Matching Entities to GeoDB Data Model Things to do: Software specific (data model supported by the s/w) To develop an efficient and effective database schema Determine the appropriate geodatabase representation for entities Ensure that complex feature classes are supported 2011/10/12 42
Step 5: Organize Spatial Database Structure Things to do: Assign entities to feature classes and subtypes (subclass) Group related sets of features into geometric networks or planar topologies Organize feature classes and datasets into geodatabase 2011/10/12 43 An Example: Spatial Database Structure 2011/10/12 44
ESRI s Geodatabase Core ESRI ArcGIS data model Set of ArcObjects components in ArcGIS for accessing data A physical store of geographic data 2011/10/12 45 Geodatabase Data Management Personal Geodatabase Single user editing Stored in MS Access Size limit of 2 GB ArcGIS File Geodatabase (9.2) 1 TB per table Reduced storage requirements ArcSDE Geodatabase Enterprise Supports multiuser editing via versioning Requires ArcEditor or ArcInfo to edit Personal Geodatabase File Geodatabase ArcSDE ArcSDE Geodatabase Oracle SQL Server DB2 Informix 2011/10/12 46
Create a Geodatabase Define schema in ArcCatalog Define feature classes, datasets, relationships, etc Import and convert data from other formats Shapefile Coverage CAD Raster Copy and Paste Use an ESRI Data Model Industry specific data models available Copy geodatabase template GDB 2011/10/12 47 Introducing Versioning Effective functionality in the storage, retrieval and query of historical data. New data item -- EDITED FEATURE is proposed to be added to each feature of the map layers. Feature change log has been identified, such as User ID, Geo-reference number, Map ID, Map Feature ID and others. Versioning function in ArcInfo 8 can be used as an efficient tool for Recording historical change of a certain types of land feature, such as land parcel Periodic data archiving (e.g., once each week) can achieved overall database Data volume can be one of the major concerns 2011/10/12 48
Key Hardware Parameters Data Volume of a GIS Databases for GIS applications range from a few megabytes (a small resource management project) to hundreds Gbytes Case A small raster-based project: IDRISI, 100 by 200 cells 50 layers A mid-sized vector-based project: National Forest in ARC/INFO A national archival database Spatial database imagery of Landsat accumulated to 1989 imagery Hong Kong (3,000 sheets), BMS 1:1000, Arc/Info16 layers Hong Kong CIS (3200 maps): 1:1000, 33 layers, Arc/Info Data Volume 10 Mbytes 300 Mbytes many hundreds of Gbytes order 101 3 bytes 4 Gbytes 3 Gbytes 2011/10/12 49 Key Hardware Parameters (cont d) Access speed On-line": data which can be accessed Archival media: Magnetic tape CD-ROM Network configuration Should database be centralized or distributed? All departments share one common database, or parts of the database exist on different workstations in an integrated network 2011/10/12 50
A Centralized Database Architectural To enhance overall system performance To solve data integrity, versioning control and data concurrency To integrate various GIS sub-systems To apply client-server and other technologies for handling data communication and management between Information Centre and sub-offices Office-1 Office-2 Office-19 Office-20 Server HQ DB Information Centre 2011/10/12 51 Scheduling Database Creation Database creation: is a time-consuming and expensive operation which must be phased over several years of operation To know the complexity of data on each input source document to forecast data input workload 2011/10/12 52
Database Scheduling Issues Determine the order of datasets input, must rank products based on Perceived benefit Cost of necessary input To know the payoffs between producing a single tile of a new product producing further tiles of an existing product Priorities under the constraint of data input capacity is a delicate operation for the Database Manager 2011/10/12 53 Review Further readings Geodatabase, ESRI, (http://www.esri.com/software/arcgis/geodatabase/index.html) David Arctur, 2004, Designing Geodatabases: Case Studies in GIS Data Modeling, Esri Press, ISBN-13: 978-1589480216, 393p. Summarization of the main ideas presented in this lecture: Questions? 2011/10/12 54