Introduction to Geographic Information Science. Updates. Last Lecture. Geography 4103 / Database Management

Transcription

1 Geography 4103 / 5103 Introduction to Geographic Information Science Database Management Updates Last Lecture We tried to explore the term spatial model by looking at definitions, taxonomies and examples An understanding of the methods we use (analysis tools), appropriate data models and of the problem we face (modeling) are central Deriving meaningful representations of events, occurrences or processes by making use of the power of spatial analysis Modelbuilder: How do you like it? 1

2 Today s Outline We will look into Database Management Systems (DBMS) Exploring what Databases and their elements are and what DBMS means Types of DBMS How attribute data & feature info is managed and stored Operations on relational DBMS (Relational Operators) to manipulate and query/select data Spatial data Learning Objectives Database Management Systems (DBMS) Databases and their elements Types of DBMS Relational Operators to manipulate and query/select data Spatial data Databases and DBMS It s all about our data (attribute information & feature information ) A database is a collection of data files that is structured (organized). A database management system (DBMS) is a specialized computer program used to organize & manipulate (manage) the database (data storage, editing, and retrieval). Oracle, Access, Postgres 2

3 DBMS & GIS - Often huge tables - Require maintenance (change, add, delete) to store data properties and their relationships - Must serve different people/applications for queries - Protection from corrupting/deleting & access restrictions - Geodatabases as a more complex type Logical vs. Physical Structures Logical structure = database design (schema) Logical specification of attributes and relationships Conceptual model of items, mappings, cardinality Entity- relation diagram / notation (UML) Physical structure = database implementation Many possible implementations of any schema Depends on intended db use requirements Speed access, frequent update Flexible relationships Protect data security Logical Structure (Schema) Bolstad s (2005) forest trails database Entity sets hold attributes* Relationships hold mappings Use these for joining tables Cardinality (1-N, M-N) defines nature and direction of the relationship Review joins-and-relates Recreation Activity Entity sets hold features, too M Features 3

4 cardinality cardinality Attention: 1:M and M:N Use relate instead of join cardinality If join: only first element in shp files gdb: relationships for all mappings M:1 Landuse features (M) & descriptions (1) Spatial Joins Physical Structures/ Database Models particular way of conceptually organizing multiple data files in a database (implementation) Flat File: text files Hierarchical: parent-child Network: nodes & links Relational: tables related via keys Hybrid/ Object-oriented Hierarchical and network database models have generally been replaced by the relational data model. Flat File Data in a text formatted file (row/column format). Initial stage format Advantages: Transparent, easily transportable Disadvantages: Little structure, few error safeguards, no ability to cross-reference or link among entries 4

5 Hierarchical DBMS Root entity & tree (e.g. ArcCatalog, Windows Explorer) and parentchildren relationships Simple, hard to capture complex relationships, slow searches Redundancies exist (updates!) - duplicates forests trails features redundant Network DBMS Eliminate redundancy - permit multiple parents for each child forests trails features Disadvantages: difficult to implement difficult to update difficult to validate activity Advantages: fast search flexible relationships no duplicates Hierarchic and Network DBMS In Practice Redundant items Not an error (no way to avoid) No redundant nodes, but errors in relations point 4 not part of edge f point 5 should be part 5

6 Physical Structures/ Database Models particular way of conceptually organizing multiple data files in a database Flat File: text files Hierarchical: parent-child Network: nodes & links Relational: tables related via keys Hybrid/ Object-oriented Hierarchical and network database models have generally been replaced by the relational data model. Relational DBMS Introduced by E.F. Codd (1968) Mathematician at IBM, same time as Mandelbrot Most frequently encountered DBMS in GIS Flexible Wide range of data types Simple to implement, modify and understand Bernhardsen: simple table structure permitted development of SQL Sometimes retrieval is slow (so optimize tables) Use fewer columns, fewer joins Use relationship classes instead of joins Table: Data organized in rows and columns Record (rows/tuples): a set of tuples represents logical entities (e.g. road, lake, land use polygon) Field (column/item): The attribute (property) of the logical entity Index/key: Attribute(s) used to identify, organize, or order records in a database (needed for relational algebra or joins; see below) Record (or tuple) Terminology Field (or attribute/item) ID AREA Perim Class Code a 11z a 119f integer domain real domain (float/double) alphanumeric domain (a string) 6

7 Original Flat Files Minimal structure Field types/domains specified (possible values) Advantages: Minimum structure, easy programming, flexible Disadvantages: Can be slow due to lack of structure Relational Database Relational Keys Any unique field can be a key. Keys can span multiple columns What is unique? Forests: forest-id or forest_name Recreational features: feature or description 7

8 Primary and Foreign Keys Primary key index to a table Foreign key index contained in the table that is possibly nonunique, but which serves as primary key in another table (can be used for Joins) Why join these tables? What could it tell you? Primary Key Foreign Key Every table must have a primary key (what is primary key for Trails table?) Primary Key Relational Database Joining Tables Rule: each row holds a unique combination of values, before and after join Ideally, isolate key in as few fields as possible db is simpler, smaller disk volume, faster query This is partly why ArcGIS uses a # field forces primary key into a single column, in absence of other keys Relational Database Result of the Join What is the primary key in the Joined Table? What foreign keys do you see in Joined Table? 8

9 Which fields can be primary keys? What is the primary key for this table? Sometimes we need multiple fields to form a key e.g. Parcel-ID and Own-ID Relational Join (1:n) Resulting Joined table: 9

10 Sorting ordering by attribute values Simple sort ascending AREA Name AREA class Type Emily, Lake 52, Limnetic zone Emily, Lake 58, Limnetic zone 60, Shallow lakes 64, Shallow lakes 70, Shallow lakes Long Lake 88, Limnetic zone 143, Littoral zone Sleepy Eye Lake 170, Littoral zone Mud Lake 193, Shallow lakes Goldsmith Lake 201, Littoral zone Emily, Lake 336, Littoral zone 349, Limnetic zone 384, Littoral zone Emily, Lake 420, Limnetic zone Savidge Lake 479, Littoral zone Emily, Lake 545, Limnetic zone Dog Lake 635, Littoral zone Duck Lake 1,126, Limnetic zone Wita Lake 1,354, Littoral zone 1,418, Limnetic zone Ballantyne Lake 1,428, Limnetic zone Washington, Lake 1,914, Limnetic zone Limnetic zone Compound sort ascending Type, then descending AREA within Type Name AREA class Type 4,040, Limnetic zone 1,937, Limnetic zone Washington, 1,914, Limnetic zone Lake Ballantyne Lake 1,428, Limnetic zone 1,418, Limnetic zone Duck Lake 1,126, Limnetic zone Emily, Lake 545, Limnetic zone Emily, Lake 420, Limnetic zone 349, Limnetic zone Long Lake 88, Limnetic zone Emily, Lake 58, Limnetic zone Emily, Lake 52, Limnetic zone Dog Lake 635, Littoral zone Wita Lake 1,354, Littoral zone Savidge Lake 479, Littoral zone 384, Littoral zone Emily, Lake 336, Littoral zone Goldsmith Lake 201, Littoral zone Sleepy Eye Lake 170, Littoral zone 143, Littoral zone Mud Lake 193, Shallow lakes 70, Shallow lakes Shallow lakes Constraints on relational implementation Rules for implementing tables appear to be fast and loose. In fact, two kinds of constraints allow flexibility yet preserve logical consistency. Constraint 1 limit the number of legal operations on relational tables (Relational Algebra) Constraint 2 Balance the amount of redundancy (Normal Forms) Constraint 1 limit # operations Codd s relational algebra (only 8 operations) Combine or split tables Select rows or columns Expand tables (add rows or columns) Everything you do in Arc that geoprocesses tables is accomplished by these 8 operations. 10

11 The Eight Operators (after Bolstad, 2005) Select (rows) by attribute Select specific column(s) Select size >= big Simple or compound restricts (using logical operators AND, OR, NOT) Recall comment about fewer columns makes a simpler db; better speed, smaller disk volume (vertical subsetting) Show SQL The Eight Operators (after Bolstad, 2005) Combine all possible unique rows in two tables Combines all unique rows of one table with all unique rows of another table (crosstabulating) Often used in queries with All based on a condition Find all types (m,n,r) associated with size = 1 and size =2 in 3 rd table Returns list of types Find structure types that are located in 2 different Hazard zones (1 and 2) out of a table that summarizes all relationships The Eight Operators (after Bolstad, 2005) Combine tables to return records found in one or both Combine tables to return records found in both No duplicates; same set of attributes in input tables same set of attributes in input tables 11

12 The Eight Operators (after Bolstad, 2005) Return rows in first but not second table order matters! Similar to Erase Match candidate keys to expand attributes Sequential joins are possible What is the join field above? Summary We tried to explore Database Management Systems (DBMS) Databases for organizing and manipulating data Relational databases are most common Attribute data is managed and stored and tables can be linked based on keys Operations on relational DBMS (Relational Operators) are very important concepts for DBMS Spatial data are special cases and often special structures are required to manage them Constraints on relational implementation Rules for implementing tables appear to be fast and loose. In fact, two kinds of constraints allow flexibility yet preserve logical consistency. Constraint 1 limit the number of legal operations on relational tables (Relational Algebra) Constraint 2 Balance the amount of redundancy (Normal Forms) 12

13 Pitfalls of Relational Tables Relational tables have many advantages, but if improperly structured, they may suffer from: - Poor performance - Inconsistency - Redundancy - Difficult maintenance This occurs when concepts of Normal Forms in relational tables are violated. Constraint 2 limit redundancy (do this with indexing keys & dependencies) Dependencies needed to make relational DBMS work. Dependency means that one column predetermines another. Dependency Redundancy (they complement and balance each other) Too much bulky database, slower performance Not enough can t find all the info in the table easily, and difficult to join when added information needed Simple (Functional) Dependency Dfn: knowing one field in a row determines what the value in another field would be. Example: Student Database Knowing a Buff One number determines student name Knowing name determines major (even Undeclared) Knowing major determines College (A&S, ENG) Functional dependencies are good (they re simple) Transitive dependencies are bad Transitive: sequence of simple dependencies in one table. Bad because too much redundancy creates complex primary and foreign indexing keys (again, bulky, slow, and possibly contradictory) 13

14 How to resolve Constraint #2? Normalization insures indexing keys provide just the right amount of dependency in single table. Just the right amount means that edits can be made in just one table and propagated through the rest of the DBMS using table relationships (and joins). And database edits cannot easily corrupt the data (goal is to free the database of modification anomalies). How to resolve Constraint #2? Normalize in stages, called Normal Forms Each form inserts or eliminates dependencies Codd proposed six in a sequence three added later First three needed for GIS When all three are in place, relational database contains only simple dependences A normalized database is suitable for general purpose queries, meaning special cases in the database should not require different query formulation than general cases. Normal Forms 1 st normal form: Atomic columns and cell values every cell contains only one attribute value, and no repeat columns appear in any single table 2 nd normal form: establish simple dependencies attributes that do not make up the primary key are functionally dependent only on the primary key Split tables to remove duplicate rows 3 rd normal form: eliminate transitive dependencies Split tables to remove dependent rows and columns Six additional normal forms can be established, but GIS uses only these three 14

15 Establish 1 st Normal Form BuffOne Student Name Major Dept College Sally Jones Interface Dsn CSI ENG Bob Willis Policy, Human Geog ENVS, GEOG A&S Phys Geog, GEOG, Kathy Dunn Policy ENVS A&S Hal Smith GIS GEOG A&S Carl Tomlin Analysis, GIS ENVS, GEOG A&S Problem? Cells can have only one value (thus queries need to recognize and isolate one major from list of possibly multiple majors queries become more difficult than they need to be) Establish 1 st Normal Form 1st Normal Form? Not yet! No cell contains more than one value, and now primary key is BuffOne or Student Name; BUT multiple columns persist BuffOne Student Name Major Dept College Major Dept College Sally Jones Interface Dsn CSI ENG Bob Willis Policy ENVS A&S Human Geog GEOG A&S Kathy Dunn Phys Geog GEOG A&S Policy ENVS A&S Hal Smith GIS GEOG A&S Carl Tomlin Analysis ENVS A&S GIS GEOG A&S Establish 1 st Normal Form Done! Every cell has only one value, and no duplicate columns. BuffOne Student Name Major Dept College Sally Jones Interface Dsn CSI ENG Bob Willis Policy ENVS A&S Kathy Dunn Phys Geog GEOG A&S Hal Smith GIS GEOG A&S Carl Tomlin Analysis ENVS A&S Bob Willis Human Geog GEOG A&S Kathy Dunn Policy ENVS A&S Carl Tomlin GIS GEOG A&S But what is primary key now? (note Bob Willis, Kathy Dunn Carl Tomlin have double majors) 15

16 Establish 2nd Normal Form 2nd Normal Form Remove duplicate rows to establish primary keys for simple dependencies; every non-key field depends only on the primary key. Students Majors BuffOne Student Name Major1 ID Major 2 ID Sally Jones Null Bob Willis Kathy Dunn Hal Smith Null Carl Tomlin Major ID Major Dept ID Dept College Interface Design 134 CSI ENG Policy 378 ENVS A&S Analysis 378 ENVS A&S Phys Geog 260 GEOG A&S Human Geog 260 GEOG A&S GIS 260 GEOG A&S Done! Establish 3 rd Normal Form 3 rd Normal Form -- Eliminate transitive dependencies Major ID Dept ID and Dept ID College BuffOne Student Name Major1 ID Major 2 ID Students Sally Jones Null Bob Willis Kathy Dunn Hal Smith Null Carl Tomlin Major ID Major Dept ID Departments Interface Design 134 Dept ID Dept College Policy CSI ENG Analysis ENVS A&S Phys Geog GEOG A&S Human Geog GIS 260 Majors Done! In 3 rd Normal Form Tables have distinct sets of rows and columns Primary keys are unique identifiers within tables, and in each table, they span as few columns as possible. Items appearing in multiple tables keep same ID throughout All dependencies are simple; no transitive dependencies exist in any single table 16

17 Hybrid DBMS In GIS often hybrid database designs are used Coordinate data in specialized structures (fast retrieval) *object-related grouping, *indexing, listing as well as *pointers to link between geographic features and attributes Topology explicitly stored in an indexing table (using lists and pointers to keep information about adjacency, ) Attribute data in relational databases Attributes and features stored in same type DBMS Relational Arc Geodatabase Integrated DBMS Item b) is the Management Base; c) is Data Base Where is the Analytic Base? Geodatabases stored in Integrated DBMS Object-Oriented DBMS Items are objects, encapsulate data in frames Classes have properties = behaviors = methods Subclasses inherit properties Objects pass messages E.g. GIS SmallWorld, ArcObjects (learn a bit about this in GIS 2) Has evolved into agent-based modeling, dynamic modeling and mobility tracking) 17