DATA MINING II - 1DL460

Transcription

1 DATA MINING II - 1DL460 Spring 2016 A second course in data mining Kjell Orsborn Uppsala Database Laboratory Department of Information Technology, Uppsala University, Uppsala, Sweden 25/02/16 1

2 Personell Kjell Orsborn, lecturer, examiner: phone: , room: 116, ITC building 19 Tore Risch, lecturer phone , room: 137, ITC building 19 Sobhan Badiozamany, course assistant, phone , room: 112, ITC building 19 25/02/16 2

3 Preliminary course contents Lecture topics: Course intro - overview of topics in data mining 2 Data mining in science and engineering Web mining Search engines Adv. association analysis Spatial & temporal data mining Sequential association analysis Cluster validation Advanced clustering methods: Chamelon, Cure Birch (SNN, Rock, Jarvis-Patrick) large scale clustering methods Stream data mining Privacy preserving data mining Outlier detection Additional topics if time: More on large scale data mining Visual data exploration Invited guest lectures 25/02/16 3

4 Course contents continued Assignments: Assignment 1 Web mining HITS / PageRank Assignment 2 Implementation of Association Rule Mining Assignment 3: Implementation of scalable K-means Applying generic indexing in data mining Review and presentation of tools for data mining Large scale data mining tools Stream data mining tools 25/02/16 4

5 Examination Written examination grade 3, 4 and 5 Assignments all assignments should be passed with a passing grade 25/02/16 5

6 Introduction to Data Mining II (Tan, Steinbach, Kumar ch. 1) Kjell Orsborn Department of Information Technology Uppsala University, Uppsala, Sweden 25/02/16 6

7 Data Mining The process of extracting valid, previously unknown, comprehensible, and actionable information from large databases and using it to make crucial business decisions, (Simoudis, 1996). Involves the analysis of data and the use of software techniques for finding hidden and unexpected patterns and relationships in sets of data; in contrast to information and knowledge that are already intuitive. Patterns and relationships are identified by examining the underlying rules and features in the data. Tends to work from the data up and most accurate results normally require large volumes of data to deliver reliable conclusions. Data mining can provide huge paybacks for companies who have made a significant investment in data warehousing. 25/02/16 7

8 Historic view of data mining Han et al, /02/16 8

9 The data mining process Data cleaning (to remove noise and inconsistent data) Data integration (where multiple data sources may be combined) Data selection (where data relevant to the analysis task are retrieved from the database) Data transformation (where data are transformed or consolidated into forms appropriate for mining by performing summary or aggregation operations) Data mining (an essential process where intelligent methods are applied in order to extract data patterns) Pattern evaluation (to identify the truly interesting patterns representing knowledge based on some interestingness measures) Knowledge presentation (where visualization and knowledge representation techniques are used to present the mined knowledge to the user) Cleaning & Integration Selection & Transformation Data Warehouse Data Mining Evaluation & Presentation 1 Knowledge Patterns Database Database Database File File File 25/02/16 9

10 Why data mining? "There was 5 exabytes of information created between the dawn of civilization through 2003," Schmidt said, "but that much information is now created every 2 days, and the pace is increasing... People aren't ready for the technology revolution that's going to happen to them... (Eric Schmidt, Google) 25/02/16 10

11 The information explosion The world s information is doubling every two years. In 2011 the world will create a staggering 1.8 Zettabytes. By 2020 the world will generate 50 times the amount of information and 75 times the number of "information containers" (files) while IT staff to manage it will grow less than 1.5 times. [ref. IDC/EMC 2011] 25/02/16 11

12 Why data mining? The explosive growth of data: from terabytes, through petabytes, to exabytes Data collection from automated data collection tools, database systems, web, e-commerce, transactions, stocks, remote sensing, bioinformatics, scientific simulation, computerized society, news, digital cameras, Human analysts may take weeks to discover useful information Much 4,000,000 of the data is never analyzed at all Total new disk (TB) since ,500,000 3,000,000 2,500,000 The Data Gap From: R. Grossman, C. Kamath, 2,000,000 1,500,000 V. Kumar, Data Mining for Scientific and Engineering Applications 1,000, ,000 Number of analysts /02/16 12

13 Why mine data? traditional commercial viewpoint Lots of data is being collected and warehoused Web data, e-commerce purchases at department/ grocery stores Bank & credit card transactions Computers have become cheaper and more powerful Competitive pressure is strong Provide better, customized services for an edge (e.g. in Customer Relationship Management) 25/02/16 13

14 Why mine data? scientific and engineering viewpoint Data collected and stored at enormous speeds (GB/hour) remote sensors on a satellite telescopes scanning the skies microarrays generating gene expression data scientific simulations generating terabytes of data Traditional techniques infeasible for raw data Data mining may help scientists and engineers in classifying and segmenting data in hypothesis formation 25/02/16 14

15 Bioinformatics Climate GIS Energy Manufacturing Transportation Telecom Astronomy Scientific and engineering data mining 25/02/16 15

16 Data mining in industrial engineering Large ongoing industrial initiatives to develop tools and techniques for analyzing, controlling and improving industrial processes. Instrumentation of machines and equipment Sensors, data streams, streaming analysis Analysis of large amounts of data to improve industrial processes Industrial version of Internet of things (IoT) Industrial Internet (USA et al) Industry 4.0 (Germany and EU) Made in China /02/16 16

17 Why not traditional data analysis? Data growth challenges and opportunities include the 3 V s, i.e. increasing volume (amount of data), velocity (speed of data in and out), and variety (range of data types and sources). Tremendous amounts of data Algorithms must be highly scalable to handle large datasets of terabytes of data and beyond High-dimensionality of data Micro-array may have tens of thousands of dimensions High complexity of data Data streams and sensor data Spatial, spatiotemporal, multimedia, text and Web data Time-series data, temporal data, sequence data Software programs, scientific simulations Structure data, graphs, social networks and multi-linked data Heterogeneous databases and legacy databases New and sophisticated applications High-level interface to data query language support for data analysis Breaking the Chains: On Declarative Data Analysis and Data Independence in the Big Data Era, Volker Markl VLDB keynote, Hangzhou, China, /02/16 17

18 Data mining tasks Prediction methods Use some variables to predict unknown or future values of other variables. Description methods Find human-interpretable patterns that describe the data. From [Fayyad, et.al.] Advances in Knowledge Discovery and Data Mining, /02/16 18

19 Classification - definition Given a collection of records (training set) Each record contains a set of attributes, one of the attributes is the class. Find a model for class attribute as a function of the values of other attributes. Goal: previously unseen records should be assigned a class as accurately as possible. A test set is used to determine the accuracy of the model. Usually, the given data set is divided into training and test sets, with training set used to build the model and test set used to validate it. 25/02/16 19

20 Clustering - definition Given a set of data points, each having a set of attributes, and a similarity measure among them, find clusters such that Data points in one cluster are more similar to one another. Data points in separate clusters are less similar to one another. Similarity Measures: Euclidean distance if attributes are continuous. Other problem-specific measures. 25/02/16 20

21 Association rule discovery - definition Given a set of records each of which contain some number of items from a given collection; Produce dependency rules which will predict occurrence of an item based on occurrences of other items. TID Items 1 Bread, Coke, Milk 2 Beer, Bread 3 Beer, Coke, Diaper, Milk 4 Beer, Bread, Diaper, Milk 5 Coke, Diaper, Milk Rules Discovered: {Milk} --> {Coke} {Diaper, Milk} --> {Beer} 25/02/16 21

22 Sequential pattern discovery definition Given is a set of objects, with each object associated with its own timeline of events, find rules that predict strong sequential dependencies among different events. (A B) (C) (D E) Rules are formed by first disovering patterns. Event occurrences in the patterns are governed by timing constraints. (A B) (C) (D E) <= xg >ng <= ws <= ms 25/02/16 22

23 Outlier analysis - deviation or anomaly detection Detect significant deviations from normal behavior Applications: Credit Card Fraud Detection Network Intrusion Detection Typical network traffic at University level may reach over 100 million connections per day 25/02/16 23

24 Challenges of data mining Scalability Dimensionality Complex and heterogeneous data Data quality Data ownership and distribution Privacy preservation Streaming data Data preparation High-level access to data 25/02/16 24