Knowledge Discovery in Databases. Databases. date name surname street city account no. payment balance

Transcription

1 Databases date name surname street city account no. payment balance Jan Novak Dlouha 5 Praha Jan Novak Dlouha 5 Praha Jan Novak Dlouha 5 Praha Karel Nemec Podolska 4 Praha Karel Nemec Podolska 4 Praha Jan Novak Dlouha 5 Praha Karel Nemec Podolska 4 Praha Flat file client id_client name surname street city... transaction id_transaction id_account date payment balance... account id_account id_client... Relational database Querying: QBE vs. SQL SELECT client.name, client.surname, client.street, client.city, account. id_client, transaction.balance FROM client, account, transaction WHERE client.id_client = account.id_account; AND transaction.id_account = account.id_account; AND transaction.balance < 100; GROUP BY client.city P. Berka, /14

2 Decision support using databases 1. Executive Information Systems managerial information systems designed for fast access to information user friendly interface but less flexible 2. On-Line Analytical Processing multidimensional concept of data storage and manipulation (DATA CUBE), intuitive data manipulation, work with data collected from heterogeneous data sources data conversion is necessary, use of analytical methods statistical summaries, what-if analysis, Client/Server architecture, support for multi-user access, OLAP results stored separately from the source data, dynamic manipulation with sparse matrices, missing values processing, unlimited number of dimensions and aggregation levels. P. Berka, /14

3 product date region sales city Database structure basketid date product city sales screws Praha nuts Praha screws Brno nails Brno screws Praha nails Praha screws Kladno 35 Table SALES Praha Brno Kladno screws nuts nails screws nuts nails screws nuts nails Sparse matrix P. Berka, /14

4 sales aggregation products aggregation cities aggregation regions Data manipulation: slice and dice roll up vs. drill down Microsoft Data Analyzer P. Berka, /14

5 Implementation: hypercube multicube true OLAP vs. ROLAP User interface OLAP engine MOLAP ROLAP SQL engine summarized data granular data MOLAP vs. ROLAP physical implementation of the system: star schema, snowflake schema. P. Berka, /14

6 dimension store fact table dimension time STORE KEY data o prodejně město ID okresu data o okresu ID regionu data o regionu úroveň (level) STORE KEY PRODUCT KEY PERIOD KEY cena počet dimension product PRODUCT KEY data o produktu značka výrobce úroveň (level) PERIOD KEY data o období rok čtvrtletí měsíc den Star dimension store STORE KEY ID okresu ID regionu data o prodejně město ID okresu data o okresu ID regionu data o regionu úroveň (level) fact table store STORE KEY PRODUCT KEY PERIOD KEY cena počet data o okresu ID regionu data o regionu fact table district ID okresu PRODUCT KEY PERIOD KEY cena počet fact table region ID regionu PRODUCT KEY PERIOD KEY cena počet Snowflake P. Berka, /14

7 OLAP functionality reachable by classic technologies Microsoft Access Microsoft Excel P. Berka, /14

8 3. Data warehouse subject oriented, integrated, time variant, nonvolatile data repository used for decision support silně sumarizovaná data m e t a d a t a středně sumarizovaná data současná detailní data starší detailní data 1. vrstva produkční databáze 2. vrstva ddddd Data Warehouse Data 3. vrstva Data Mart P. Berka, /14

9 4. Business Intelligence computerized tools and techniques used to collect, integrate, analyze, interpret and present (business) data and information. 4 main components: data warehouse business analytics (querying, reporting, statistical analyses, data mining) business performance management user interface (presentations) Major Components of BI (Turban a kol., 2007) P. Berka, /14

10 Databases meet Data Mining 1. Query languages for KDD Mine Rule (Boulicaut, 1998) - association rules MINE RULE Priklad AS SELECT DISTINCT 1..n produkt AS BODY, 1..1 produkt AS HEAD, SUPPORT, CONFIDENCE FROM Prodej WHERE BODY.město = HEAD.město AND BODY.datum = HEAD.datum EXTRACTING RULES WITH SUPPORT: 0.1, CONFIDENCE: 0.5 MSQL (Imielinski, Virmani, 1999) association rules, records Emp(Id,Age,Sex,Salary,Position,Car) GetRules (Emp) into R where support > 0.1 and confidence > 0.9 SelectRules (R) where body has {Age=*), (Sex=*)} and body is {(Car=*)} MSQL - rules Select * from Emp where violates all (GetRules (Emp) where body is {(Age=*)} and head is {(Salary=*)} and confidence > 0.3) MSQL - exceptions P. Berka, /14

11 DMQL (Han et al., 1996) different rules Find association rules related to average_grading, birth_place, address from student where major = computer_science and birth_place = Canada with support threshold = 0.05 with confidence threshold = 0.7 DMQL association rules Find classification rules for computer_science_students according to average_grading related to birth_place, address from student where major = computer_science and birth_place = Canada DMQL classification rules Find discriminant rule for cs_grads with status = graduate in contrast to cs_undergrads with status = undergraduate related to average_grading, birth_place, address from student where major = computer_science and birth_place = Canada DMQL discrimination rules 2. API standards SQL/MM Data Mining OLE DB for Data Mining P. Berka, /14

12 3. Extending databse systems with data mining ( in-database data mining) MicroSoft SQL Server Decision trees, association rules, naive bayes classifier, neuronal nets, text mining, sequence clustering, time series - uses OLE DB DM and PMML to describe tasks and models) and BI Development studio as interface Oracle Data Mining tools for classification, regression, anomaly detection, association mining, clustering, feature extraction, attribute ranking P. Berka, /14

13 New Trend Big Data Big Data are data that cannot be handled using standard data base systems and standard data analysis tools. Big Data sources: sensors, surveillance systems, mobile phones, GPS devices, RFID readers, social networks, computer networks, web logs, scientific data Big Data Characteristics (4 V s): Volume: the size of Big Data goes beyond standard data storage and manipulation techniques Velocity: Big Data is often available in real time Variety: Big Data contains not only structured data (e.g. in tabular or relational form) but also texts, images, audio or video Veracity: the quality and reliability of Big Data can vary P. Berka, /14

14 Challenges for collecting, storing and manipulating Big Data New forms of data storage: file systems (e.g. Google File System), NoSQL databases (data that cannot be expressed using tabular relations of relational databases) New forms of computation: parallel computing, distributed computing, grid computing, cloud computing Example solution: Apache Hadoop Hadoop distributed file system Map/Reduce: divide and conquer approach to break-down intractable problem into tractable sub-problems P. Berka, /14