Assignment 1. Question 1: Brock Wilcox CS

Transcription

1 Assignment 1 Brock Wilcox wilcox6@uiuc.edu CS Question 1: In the introduction chapter, we have introduced different ways to perform data mining: (1) using a data mining language to write data mining functions in a commercial data mining system, e.g., Microsoft SQL Server Analysis Manager or Oracle Data Mining, (2) develop your own customized data mining functions from scratch for particular applications, and (3) write functions for invisible data mining. 1. Present examples for each way of building data mining functions. 2. Discuss the pros and cons of each way to perform data mining. 3. Suppose you are hired by a web-retail company, such as Amazon.com, discuss three data mining functions you would build for the company so that customers at home can do "invisible data mining". (1.1, 1.2) Data mining using a commercial data mining system involves buying an already built system, from Oracle for example. Using the provided tools, you'd import your data and use the provided software applications for performing data mining tasks. This technique is good because there is lots of documentation, and you might be able to find someone already trained to use the technology. You may, however, run into issues where it is difficult to customize the solution. Constructing customized data mining tools for a particular application involves going through a complete software cycle designing, implementing, testing, and maintaining the data mining software. It is likely that this would be implemented on top of a more generic database system, such as mysql or Oracle, instead of constructing customized storage mechanisms. The major pro is that you can completely control the implementation and the major con is that it may take significant effort to develop and you must maintain it forever. An example of invisble data mining is seller ratings on a website, constructed by taking the mean of all feedback from that seller. This can be useful for buyers or other users of the results of such invisible data mining, leading them to being better informed and perhaps lead to further purchases. A con is that without visiblity into how these numbers are constructed, the consumer of the information can be easily mislead. (1.3) Three of the customer-oriented invisible data mining functions that I would construct would be: Related products, based on category and/or historically grouped purchases Seller sales feedback, based on number of returns for example Product price compared to other products, based on mean and quartile rank

2 Question 2: Suppose a student collected the price and weight of 20 products in a shop with the following result price $5.89 $49.59 $59.98 $159 $17.99 $56.99 $82.75 $ $31 $125.5 weight price $4.5 $22 $52.9 $61 $33.5 $328 $128 $ $229 $189.4 weight Calculate the mean, Q1, median, Q3, and standard deviation of price and weight; 2. Draw the boxplots for price and weight; 3. Draw scatter plot and Q-Q plot based on these two variables; 4. Normalize the two variables based on the min-max normalization (min = 1, max = 10); 5. Normalize the two variables based on the z-score normalization; 6. Calculate the Pearson correlation coefficient. Are these two variables positively or negatively correlated? 7. Take the price of the above 20 products, partition them into four bins by each of the following methods equal-width partitioning, and equal-depth (equal-frequency) partitioning (2.1) Price Weight Mean $ Q1 $ Median $ Q3 $ Std Dev $ (2.2) Boxplots for price and weight:

3 (2.3) Scatterplot and Q-Q plot for price and weight: (2.4 and 2.5) (2.4) (2.4) (2.5) (2.5) price weight min-max norm pricemin-max norm weight1-10 price 1-10 weight z-norm price z-norm weigh (2.6) The correlation of these is 0.54, which means that these are positively correlated. Since this is more than 0.5 these have a medium-to-high level of correlation. (2.7) Price - Equal Depth Bin 1: 4.5, 5.89, 17.99, 22, 31 Bin 2: 33.5, 49.59, 52.9, 56.99, Bin 3: 61, 82.75, 125.5, 128, Bin 4: , 159, 189.4, 229, 328

4 Price - Equal Width Bin 1: 4.5, 5.89, 17.99, 22, 31, 33.5, 49.59, 52.9, 56.99, 59.98, 61, Bin 2: 125.5, 128, , , 159 Bin 3: 189.4, 229 Bin 4: 328 Weight - Equal Depth Bin 1: 1.4, 1.5, 2.2, 2.7, 3.2 Bin 2: 3.9, 4.1, 4.1, 4.6, 4.8 Bin 3: 4.9, 5.3, 5.5, 5.8, 6.2 Bin 4: 8.9, 11.6, 18, 22.9, 38.2 Weight - Equal Width Bin 1: 1.4, 1.5, 2.2, 2.7, 3.2, 3.9, 4.1, 4.1, 4.6, 4.8, 4.9, 5.3, 5.5, 5.8, 6.2, 8.9 Bin 2: 11.6, 18 Bin 3: 22.9 Bin 4: 38.2

5 Question 3: Design a data warehouse for Walmart-like chain store to analyze the sales transactions. Suppose the data warehouse consisting of the following dimensions: product, store, sales, and time; and a set of measures you would like to define. (3.1) 1. Draw a star-schema, based on your consideration of power and convenience of analysis of the warehouse 2. Suppose you start from the top (all-summary) of the multi-dimensional hierarchy, what are the concrete OLAP operations (drilling, slicing, etc.) you need to find the following: average daily profit of each prodcut in the Toys department in January find which store has the highest monthly increase of sales of bread among all the stores in Illinois 3. Suppose we want to present the standard-deviation of salesby by item category, location and week, and freely drilling up and down in multidimensional space, describe how this measure can be computed efficiently 4. Median and rank are two holistic measures. Discuss how to develop an efficient (maybe approximate) methods to compute these two measures in a multi-dimensional space. product dimension table product_key description model_number color type sale fact table product_key store_key time_key sale_price quantity_sold store dimension table store_key store_name state zip time dimension table time_key day month quarter year (3.2) The average daily profit of each prodcut in the Toys department in January 2009 can be calculated by: Roll up blah blah To find which store has the highest monthly increase of sales of bread among all the stores in Illinois: Roll up

6 (3.5) blah 5. Suppose we want to present the standard-deviation of sales by by item category, location and week, and freely drilling up and down in multidimensional space, describe how this measure can be computed efficiently 6. Median and rank are two holistic measures. Discuss how to develop an efficient (maybe approximate) methods to compute these two measures in a multi-dimensional space.

7 Question 4. Suppose a company would like to design a data warehouse that may facilitate the analysis of moving vehicles in an online analytical processing manner. The company registers huge amounts of auto movement data in the format of (Auto_ID, location, speed, time). Each Auto_ID represents one vehicle associated with information, such as vehicle category, driver_category, etc., and each location could be associated with a street in a city. You may assume a street map is available for the city. 1. Design such a data warehouse that may facilitate effective on-line analytical processing in multidimensional space. 2. The movement data may contain noise. Discuss how you can develop a method that may automatically disocver some data records are likely erroneously registered in the data repository. 3. The movement data may be sparse. Discuss how you can develop a method that may construct reliable data warehouse despite of the sparsity of data. 4. If one wants to drive from A to B starting at a particular time, discuss how a system may use the data in this warehosue to work out a fast route for the driver.