Lab Assignment 2. CIS 612 Dr. Sunnie S. Chung

Transcription

1 CIS 612 Dr. Sunnie S. Chung Lab Assignment 2 1. Creating a User Defined Type (UDT) 2. Text Processing to Create a Table Valued Function 3. Visualization of Data in Mongo DB in JSON Geo Location Data Type with GIS API In modern web applications in Big Data Processing/ Big Data Analytic systems, Text Data Processing usually requires two or three Phases of Big Data Processing: Phase 1: One of the most common data preprocessing of Big Data is parsing/processing unstructured/semi structured text data such as Web Server Logs or Texts in HTML/XML/JSON to extract information you need to store in a structured data format such as CSV/TSV/JSON/XML file format. Creating an Inverted Index or Word Count from webpages/electronic books is an example of text processing. Phase 2: Once extracted information is stored as a structured file in CSV/TSV or Key Value Store (JSON), the second phase of data processing is to create permanent Database Tables or Key Value Store Collections in Parallel Database Systems that will be used for further complex analysis such as Information Retrieval or Data Analytic processes. Most of the existing advanced SQL database systems allow to create databases tables from complex structured objects such as in a CSV/JSON file format using ORM database programming features like UDT, UDF, or Table Valued Function in common host languages. Object Relational Mapping (ORM) for Phase 2: Relational Database Features with Creating User Defined Data Type (UDT) and Table Valued Function in a SQL Server from objects like a CSV file that is generated in Phase 1.

2 In this Lab2, you are exploring such ORM features in RDBMS by creating your own Data Type and Table Function for Phase 2 processing in a SQL Server from objects like a CSV file that is generated at the end of Phase 1. For this Lab2, you can use any language of your choice with any RDBMS Server that supports UDF, UDT, Table Function with any external programming language. Some suggestions would be: 1. C++/C#/Java with ODBC/JDBC in MS.NET Framework with MS SQL Server 2. Java/JDBC with Oracle RDBMS Server Part 1: Although most of database servers already have a built-in Geo Point Type, we want to study how to create complex data types or built-in data types in a Relational database server by implementing your own data type through a simple example. 1. Creating User Defined Type PointType and Table Function Dept_GeoLocations that Uses PointType 1. Create User Defined Type named PointType as follow: a) Write the body of UDT PointType in your choice of a language like C# or Java Structure: x // x coordinate y // y coordinate ToString //x:y Methods: Parse an input string x:y, Get each member / Set each Member, Check IsNull Example: if (s.isnull) { return Null; } b) Create CLR UDT PointType to be able to be used in the following two scripts below for the lab output of Part 1. c) Test your UDT with the following two scripts:

3 Example use of PointType: This is a possible test script that uses the UDT you created. Test Script1: CREATE TABLE dbo.tablepoints ( ID int IDENTITY(1,1) PRIMARY KEY, Pnt PointType) INSERT INTO dbo.tablepoints (Pnt) VALUES (CONVERT(PointType, '3:4')) INSERT INTO dbo.tablepoints (Pnt) VALUES (CONVERT(PointType, '-1:5')) INSERT INTO dbo.tablepoints (Pnt) VALUES (CAST ('1:99' AS PointType)) SELECT ID, Pnt.ToString() as StringPoint, Pnt.X as X, Pnt.Y as Y FROM dbo.tablepoints Test Script2: CREATE TABLE Locations (grid_location dbo.pointtype) INSERT INTO Locations (grid_location) VALUES ('3:2') INSERT INTO Locations (grid_location) VALUES ('-1:1') INSERT INTO Locations (grid_location) VALUES ('-1:-1') INSERT INTO Locations (grid_location) VALUES ('-8:-9') INSERT INTO Locations (grid_location) VALUES ('4:-9') view the data SELECT * FROM dbo.locations SELECT grid_location.x AS "X", grid_location.y AS "Y",

4 grid_location.tostring() AS "Point" FROM Locations update some rows UPDATE dbo.locations SET grid_location.x = 5 WHERE grid_location.y < 0 SELECT grid_location.x AS "X", grid_location.y AS "Y", grid_location.tostring() AS "Point" FROM Locations;

5 Part 2: Write Table Function/CLR Table Valued Function or Stored Procedure that processes (parse) a text file to creates an term count index file (in CSV/TSV) to create an Index Table named TermIndex in a SQL database server. Use the given text file ( MasterAllUnionAddress.txt on the class website) for input that is a combined text of all the US President Union Addresses. The Output of Part 2 is an index table named TermIndex with the following table schema: TermIndex (termid, term, count) Useful Resources: Creating UDT function Registering UDT function Other References 2. Lab Outputs of Part 1 and Part 2 - For the Lab Output for 1) UDT: Submit Screen Shots of each result of two given test scripts. - For the Lab Output for 2) Table Function: Submit all the source codes and Screen captures showing the content of your table created from your script with Select * From TermIndex; NOTE!! Each screen capture of the script outputs should show your server name, your query in the script, and the corresponding result of the query together in ONE window to prove all the outputs are generated in your server.

6 Part3: There are well known GIS APIs to visualize such complex data type: ArcGIS, Google Map, Leaflet API or QGIS. Use of JSON Geo Data Type Implement a visualization map using any of Map API for GPS location data retrieved from MongoDB (The data set is given on the class webpage) ecastingchallenge#data Open Source GIS API QGIS API: Leaflet API: Other Professional GIS API ArcGIS API: Google Map API You can use any language of your choice for this. Submit your report with your codes and the output - visualized map.