CS Database Systems Spring 2018

Transcription

1 CS Database Systems Spring 2018 Assignment 2: Data Migration Due on Tuesday, May 1st, start of class Problem Background: The United States Geological Survey creates and maintains data for generating topographic gazetteer maps of the United States to show geographic features such as lakes, dams, television towers, cities, and so forth. The USGS has made this information available in pipe-delimited text files so users can generate their own maps and create queries about geographic features. The data is in a flat-file format. Your task is to design a normalized database to store the data, migrate the data from the text file to your database, and then create a series of queries to test the extracted data. Process Step 1: Obtain the raw data For this problem, you will use the data for the state of Wyoming. We will be using a specific datafile (from 2011) for this assignment, so that everyone is dealing with the same data and results. The web site describes the contents of the file. The website contains files for every state in case you were interested in more than just Wyoming. However, please use the file I provided and do not download the data files from the website directly. Step 2: Create the Data Model Examine the data and the description information on the Web site, derive the data relationships, and create a crows-foot ER model for a database to manage this data. (TIP: one way to make it easier to examine the data is to import it into an Excel spreadsheet.) Think about normalizing the data to third normal form. You will need to examine the data carefully to find repeating groups and understand its structure! You might need to create dependency diagrams to confirm your normal form. Create surrogate keys for entities that do not have unique primary key values. Use the entity and attribute naming conventions discussed in class. Create the ER model using Visio. Name the file MigrationER.vsdx (visio). Step 3: Create the DDLs Write an Oracle SQL script that creates the normalized database tables for the tables that your data model identifies, the sequences to generate primary keys for each table that contains a surrogate primary key and foreign keys. Include commands at the beginning of the script to drop tables and sequences before recreating them. Name this file MigrationDDL.sql. Step 4: Create a Java/JDBC program to Migrate the Data Create a Java package named edu.uwec.cs.yourloginname.migration that contains. All you Java code for this assignment will go in this package. Save your code in a file named Migration.jar. Step4 - Task 1: Read the file data into a master table

2 To optimize throughput performance in a data migration operation, you should move the data from the file system to the database as efficiently as possible. After the data is in the database, you can then change its normalization format. Read the contents of the pipe-delimited WY_Features_ txt text file into a single master database table that contains all of the file fields. TIP: When working with large data sets (note this dataset has over 25k lines), it is a good idea to create and debug your procedures using a small file. Making a smaller file containing the first five records or so from the larger data file is a good start. When you are satisfied that your package is working correctly with the small file, go ahead and migrate the large data file. TIP: Usually, downloaded data contains bad records with missing data values, inconsistent values, and so forth. This file is no different. Make sure your code can deal with any issues that come up. TIP: Even though bulk prepared statements give the most speed for JDBC, it might be a good idea to do your initial debugging with normal statements so you can get better feedback as the Oracle JDBC driver doesn t provide printing of bound prepared statements. TIP: Don't forget to commit your inserts! If you try to view the data from a second script window in SQL Developer I discovered it sometimes (all the time not sure) acts like it is a different session and won't show the records you insert until you commit from where you run your script. COMMIT is the SQL command to do this. JDBC should be auto-committing on each update, but there are been cases where it gets turned off. Step4 - Task 2: Migrate the data to the normalized table design Write procedures to migrate the data from the master data table into the normalized database tables you created earlier. The data must migrate in a reasonable amount of time. This implies use of PreparedStatements (in bulk mode) if necessary. TIP: You should reuse JDBC Statements for multiple queries for efficiency, but remember that when a new query is issued on a Statement it will invalidate the last ResultSet. This implies you may need a couple of Statements. TIP: You should use Oracle sequences to pick the surrogate primary keys. This implies that you should never hardcode those surrogate keys, as either primary or foreign when updating. You will need to reverse lookup the primary keys when inserting foreign keys. Make sure the reverse lookup fields you are using select a unique row. Step5: Test your migrated database Test your migrated database by creating a sql script containing SQL queries to retrieve and display the following information. Save this in a file named MigrationTest.sql. (I included the results so you can confirm that your data is correct.) 1. Count the total number of features in Wyoming (don't include other states Tip: I inserted features even when their county entry was null and with unknown or 0 location entries (they did have a state_numeric and state_alpha_code entry) if you decided to discard these entries your number will be 7 less than mine). 28, Display the name and elevation of the feature in Wyoming with the highest elevation. Gannett Peak 13, Display the names and elevations (in feet) of all features in Wyoming with elevations over 13,000 feet above sea level.

3 Gannett Peak Grand Teton Fremont Peak Turret Peak Doublet Peak Mount Helen Mount Sacagawea Dinwoody Peak Jackson Peak Bastion Peak Mount Febbas Mount Woodrow Wilson Mount Warren Sunbeam Peak Flagstone Peak Pedestal Peak Downs Mountain Rampart Peak The Sphinx Cloud Peak Absaroka Range American Legion Peak Desolation Peak Wind River Peak Miriam Peak Split Mountain Northwest Peak Chimney Rock Henderson Peak Klondike Peak Mount Koven Upper Fremont Glacier Twin Peaks Gooseneck Pinnacle Bobs Towers Display the feature name, elevation in feet, feature_lat_dms, and feature_long_dms of all features containing the search text Devil s Tower. Devils Tower National Monument N W Devils Tower N W Devils Tower Post Office N W Devils Tower N W 5. Retrieve the names of the Wyoming counties with the largest and fewest numbers of features, as well as their corresponding number of features. I'll allow using several queries to get this data (you don't have to do it all in one query add comments to your file explaining how you are accomplishing this with your queries). Fremont 2998 Hot Springs 478

4 6. Retrieve the county names and feature names of all features whose name contains the text post office. Make your search non-case-sensitive. Albany Albany Post Office (historical) Albany Gramm Post Office (historical) Albany Holmes Post Office (historical) Campbell Peckenpaugh Post Office (historical) Carbon Arlington Post Office (historical) Carbon Dexterville Post Office (historical) Carbon Elwood Post Office (historical) Carbon Victoria Post Office (historical) Converse McKinley Post Office Crook Devils Tower Post Office Crook Hay Creek Post Office (historical) Fremont Diamond G Ranch Post Office (historical) Fremont South Pass City Post Office (historical) Fremont Vonnie Post Office (historical) Johnson Hazelton Post Office (historical) Johnson Saddlestring Post Office (historical) Laramie Meriden Post Office Lincoln Alpine Post Office (historical) Natrona Bishop Post Office (historical) Natrona Goose Egg Post Office (historical) Natrona Hells Half Acre Post Office (historical) Natrona Raderville Post Office (historical) Park Kirwin Post Office (historical) Park Pahaska Post Office (historical) Sublette Elkhorn Post Office (historical) Teton Colter Bay Post Office (historical) Teton Jenny Lake Post Office (historical) Teton Lake Outlet Post Office Teton Zenith Post Office (historical) Weston Newcastle Post Office

5 The files you should turn in for this assignment are: MigrationER.vsd MigrationDDL.sql Migration.jar MigrationTest.sql Please zip these together, named yourloginname(s)migration.zip and place in the W drive drop box for the class. You are to also turn in a hard-copy cover page in class on the due date. You may work with a partner on this assignment if you wish.