COMP33111: Tutorial/lab exercise 2

Transcription

1 COMP33111: Tutorial/lab exercise 2 Part 1: Data cleaning, profiling and warehousing Note: use lecture slides and additional materials (see Blackboard and COMP33111 web page). 1. Explain why legacy data might be difficult to integrate. Provide an example. 2. What is the role of data profiling? Explain what a data profiling report should have. Explain the steps in data cleansing. 3. Explain how approximate joins can help in data cleansing at the instance level. 4. Some DW practitioners class data errors in the ETL process into four in- categories: incomplete data; incorrect data; incomprehensible data; inconsistent data. Provide examples for each category. Consider 5. How could we resolve various types of conflicts in data transformation/mapping? (see Section 3.3) 6. Explain why data warehouses are time-variant? 7. What are the main challenges in building a data warehouse? Explain and discuss them using an example. Explain the main architectural components in a data warehouse. 8. Explain the nine steps in modelling, constructing and managing a data warehouse. 9. Describe the main clues to identify measures and dimensions while designing a data-warehouse. Give an example using the star schema. 10. A videotape company has stores in several regions. They would like to track profit information across different departments (Video Sales and Video Rentals) and regions (East, West, Central) in different years (e.g and 2012). Design an appropriate data warehouse schema using the star multi-dimensional model and discuss the fact and dimension tables you would need. Would you need/recommend a showflake schema? Explain your views.

2 Part 2: Data exploration and integration with WEKA 1 The first step in a data integration and/or analytics project is getting to know your data. In this tutorial, you will examine three data sets using the Weka framework. It provides a large number of machine learning algorithms and visualisations useful for exploratory data mining. Task A: Install and check that WEKA is running WEKA should be installed on all CS computers. If you need your own copy, download it from On start, you will see the Weka GUI chooser screen appear on your desktop. Select Explorer. The main interface for WEKA will appear as shown below 1 Tailored after: Lab 1: Getting To Know Your Data (MSCS 228: Data Mining) by Dr Craig A. Struble, Marquette University, and other resources.

3 Task B: Using WEKA to understand your data (in ARFF format) This task is intended to provide an introduction to Weka with exploration of data sets that are ARFF formatted. ARFF is the data format for Weka, so no data transformation is necessary. B1. ARFF limits the attribute types it supports in its data files. Using the WEKA documentation, discuss what are the attribute types ARFF supports? B2. Download the two data sets (labor.arff and contact-lenses.arff) from the course web site. Examine the format used in ARFF files. Start Weka and load contact-lenses.arff. Notice that the file contains 24 instances where each instance represents an individual who either wears soft contact lenses, hard contact lenses, or no contact lenses (regular glasses). The five attributes are listed on the left age, spectacle-prescrip, astigmatism, tear-prod-rate, contact-lenses. Notice that Weka provides summary information for each attribute - browse each of the attributes in the data file. For example, on the right, values for the attribute age can be seen. Nominal indicates that the age attribute is not numeric. There are eight instances of young individuals, eight pre-presbyopic and eight prebyopic instances. The coloured chart indicates the distribution of the instances relative to the age attribute within each class. Explore other attributes in detail.

4 B3. Select the Visualization tab in Weka. The Visualization tab provides a scatter plot with two data attributes as the axes. You can change which attributes are along the axes using the drop down menus in the top portion of the visualizer or by clicking on the plots in the right portion of the visualizer. Explore different scatter plots. B4. Repeat the task with the labor.arff file. The data in labor.arff contains two classes, bad and good, along with other attributes. Looking at the values for the different attributes, select three attributes that might be good predictors for the class (i.e. if you knew the value for that attribute, you could guess pretty well whether the class for the same instance was good or bad). Explain why you chose the three attributes you chose. Task C: Exploring your csv data towards a data warehousing model A company has two branches that sell different types of products via Web or through catalogue sale. You are given 2 datasets (download them from the course web site): Catalog_Orders.txt and Web_Orders.txt. Each of the rows in the datasets contains seven attributes, representing: ID of the transaction, INVOICE number, DATE, CATALOG (describing the type of product), internal code (PCODE), QTY and customer number. You also have a file Products.txt that describes the products. C1. Using Weka, explore each of the data sets i.e. their attribute values. Write a brief report that includes: data types, length and value ranges, data variance, uniqueness (provide simple statistical analyses); distribution of key attribute values, relations between pairs or small numbers of attributes (if any); are there any typical string patterns (e.g. phone numbers or dates); are there any specific properties of significant sub-populations within the files? C2. Consider data quality in the three data sets: are there any illegal values; any misspellings; is the data complete (does it cover all the cases required), are there any missing values? Does it contain errors and if there are errors how common are they, how are they represented, where do they occur? Consider, in particular the values in the CATALOG attribute. C3. Use your favourite programming environment to write a program to clean the data sets. Provide a report that documents what has been done. C4. What are the challenges in integrating these datasets in a single data warehouse (consider, for example, DATE)? How would you resolve them? Integrate the datasets in a single file. Repeat the exploration as in tasks C1 and C2. C5. Propose a data warehouse model for this company. What could be measures? What are the potential dimensions (consider Products.txt)? Which data model you would use (star/snowflake)? Draw a schema diagram that explains your model.

5 Task D: Transforming Data into ARFF In this task, a raw data is first transformed into a formatted ARFF file. Once transformed, the exploration steps are repeated as above. As part of this process, you need to identify the different attribute types as those are required by the ARFF format. Note that the ARFF format is more limited in the kinds of data attributes. When using a system with limited data types, it may be necessary to transform the raw data values in order for the tool to work properly. Keep this in mind as you work through the steps below. D1. Visit and read the information about the Teacher Pay by States data set. D2. Identify the attributes of the data. Record the attributes and the type of attribute for the data. D3. Select, download and save the raw data set in a file. D4. Convert the raw data set into CSV (comma separated value) format. One easy way to do this is to load it into Excel and use Save As.... You can also write a program to do this. When you do this conversion, it is considered good practice to replace any nominal values represented by numbers with their textual representation. This will make the data easier to interpret as you summarize and mine it. D5. Edit the CSV file, and add the ARFF header information to the file. This involves creating line, line per attribute, to signify the start of data. It is also considered good practice to add comments at the top of the file describing where you obtained this data set, what its summary characteristics are, etc. A comment in the ARFF format is started with the percent character % and continues until the end of the line. D6. Load your ARFF file into Weka and repeat the steps you performed in task B. You may run into errors as you load your ARFF file - look at for tips how to solve the problem. D7. Repeat the analysis of the data set as you did in tasks B and C. In addition, there exists one data point that appears to be a clear outlier. What point is this? (Try to use Weka to identify this point). Do you see any natural groupings from the teacher pay vs. spending per student dataset? Do the natural groups correspond to the three regions in the dataset?