Databases 2 (VU) ( )

Transcription

1 Databases 2 (VU) ( ) Denis Helic KMI, TU Graz Oct 5, 2015 Denis Helic (KMI, TU Graz) Dbase2 Oct 5, / 33

2 Lecturer Name: Denis Helic Office: IWM (Know-Center), Inffeldgasse 13, 5th Floor, Room 070 Office hours: Tuesday from 12 til 13 Phone: / Denis Helic (KMI, TU Graz) Dbase2 Oct 5, / 33

3 Outline 1 Welcome and Introduction 2 Course Organization 3 Motivation 4 Course Highlights Denis Helic (KMI, TU Graz) Dbase2 Oct 5, / 33

4 Welcome and Introduction KTI B Introduction to Knowledge Technologies Databases B M Multimedia Information Systems I M Knowledge Discovery and Data Mining I (Relational data) (Web systems) (Theory and basics) Semantic Technologies (Ontologies) M B Web Science and Web Technology (Web data) M Knowledge Discovery and Data Mining II (Applications) Network Science (Networks and analysis) M M Multimedia Information Systems II (Visualizations) Structured Data Unstructured Data Data Analysis Sensors & User Models M Science 2.0 M Evaluation Methodology M (Sensor data) (Science and Social Media) Applications (User studies) + Projects, Bachelor Thesis, Master Projects, Master Thesis, PhD Thesis Denis Helic (KMI, TU Graz) Dbase2 Oct 5, / 33

5 Welcome and Introduction Course context Databases 2 (VU) ( ) Obligatory course in Computer Science, Software Development & Business Elective course in Telematics Denis Helic (KMI, TU Graz) Dbase2 Oct 5, / 33

6 Welcome and Introduction Goals of the course To learn about the fundamentals of NoSQL systems To learn about semi-structured and unstructured data To learn about Map-Reduce and graph databases To practically implement a Map-Reduce program Denis Helic (KMI, TU Graz) Dbase2 Oct 5, / 33

7 Welcome and Introduction Goals of the course Student goals: to pass the examination Bonus goal for all: to have fun! Denis Helic (KMI, TU Graz) Dbase2 Oct 5, / 33

8 Course Organization Course Calendar (VU/ ) : Course organization : Introduction to NoSQL : Submission system / Python Tutorial Denis Helic (KMI, TU Graz) Dbase2 Oct 5, / 33

9 Course Organization Course Calendar (VU/ ) : Map-Reduce : Map-Reduce (cont.) : Map-Reduce (cont.) : Map-Reduce (cont.) Denis Helic (KMI, TU Graz) Dbase2 Oct 5, / 33

10 Course Organization Course Calendar (VU/ ) : Discussion of student projects : Discussion of student projects (cont.) : Examination Denis Helic (KMI, TU Graz) Dbase2 Oct 5, / 33

11 Course Organization Course Logistics Course website: http: //kmi.tugraz.at/staff/denis/courses/dbase2/index.html Slides will be made available on the course website Additional readings, references, links, etc. also on the website Denis Helic (KMI, TU Graz) Dbase2 Oct 5, / 33

12 Course Organization Grading One programming project with a Map-Reduce framework You will need to analyze a Web access log Task: filter the file and calculate a distribution of visited URLs We will measure your accuracy Denis Helic (KMI, TU Graz) Dbase2 Oct 5, / 33

13 Course Organization Grading F 1 = 2 p = r = p r p + r tp tp + fp tp tp + fn Denis Helic (KMI, TU Graz) Dbase2 Oct 5, / 33

14 Course Organization Grading Example Correct result includes 40 URLs. Your results has 60 URLs with 20 correct URLs and 40 wrong URLs. tp = 20, fp = 40, fn = 20 p = 1 3, r = 1 2, F 1 = 2 5, x = 16 Denis Helic (KMI, TU Graz) Dbase2 Oct 5, / 33

15 Course Organization Grading F 1 is a number between 0 and 1 Total number of points for the example x = 40 F 1 Final examination 2 questions with 20 points each Total example and examination is 80 points Denis Helic (KMI, TU Graz) Dbase2 Oct 5, / 33

16 Course Organization Grading You have to reach at least 10 points for example! You have to reach at least 10 points for the final exam! You have to reach at least 41 points combined to be positive! Denis Helic (KMI, TU Graz) Dbase2 Oct 5, / 33

17 Course Organization Grading 0-40 points: points: points: points: points: 1 Denis Helic (KMI, TU Graz) Dbase2 Oct 5, / 33

18 Course Organization Dates SVN project created We will check if the SVN project is created and we will a remainder if necessary If after that the SVN project is not there you will not be able to participate in the course student tasks handed out (via SVN) test tasks with solutions handed out Denis Helic (KMI, TU Graz) Dbase2 Oct 5, / 33

19 Course Organization Dates submission date (soft deadline) 24h delay is allowed 1h delay results in 1 point subtracted Denis Helic (KMI, TU Graz) Dbase2 Oct 5, / 33

20 Course Organization Questions? Raise them now! Ask after the lecture Visit me in the office hours Send me an As a side note: you should(!) interrupt me immediately and ask any question you might have during the lecture Denis Helic (KMI, TU Graz) Dbase2 Oct 5, / 33

21 Motivation Database systems Systems for managing data in different forms Create, Retrieve (Search, Browse, etc.), Update, Delete, etc. CRUD operations Persistent storage With a special purpose Denis Helic (KMI, TU Graz) Dbase2 Oct 5, / 33

22 Motivation Database systems TUGOnline TUGOnline is a database system for managing data about students, e.g. which courses, which marks,... Also management of teachers, courses, and other administration tasks. When a student passes an examination then a sequence of operations is executed, e.g. create an examination, update data, insert mark, store, etc. Denis Helic (KMI, TU Graz) Dbase2 Oct 5, / 33

23 Motivation Database systems Various kinds of database systems Depending on the nature of data that is managed: Structured data clear separation of structure and content Relational Databases Unstructured, or semi-structured information e.g. text documents Document Management Systems If semi-structured information comes in the form of different media (text, video, images, etc.) Multimedia Systems Denis Helic (KMI, TU Graz) Dbase2 Oct 5, / 33

24 Motivation Database systems Wikipedia A typical example of a document management system is Wikipedia. Semi-structured data exist when there is a certain structure but this structure differs from one data record to another whereas in the case of structured data each data record has always one and the same structure. An example of semi-structured data records are articles. Each article can have title, subtitle, one or more authors, abstract, one or more chapters and subchapters. Each chapter can have one or more paragraphs, might include figures, and so on. The exact number and nesting of these structural elements is however different for different articles. Also, in Wikipedia we have links which connect articles with each other, and thus, the data records are interlinked into a network. Denis Helic (KMI, TU Graz) Dbase2 Oct 5, / 33

25 Motivation Networks 6 How to search in a small world Pajek Figure 2: HP Labs communication (light grey lines) mapped onto the organizational hierarchy (black lines). Note that communication tends to cling to the formal organizational chart. Figure : Social network of HP Labs constructed out of communication. From: How to search a social network, Adamic, with one another. The h-distance, used to navigate the network, is computed as follows: individuals have h-distance one to their manager and to everyone they share a manager with. Distances are then recursively assigned, so that each individual Denis Helic (KMI, TUhas Graz) h-distance 2 to their first neighbor s Dbase2 neighbors, and h-distance 3 to their second Oct 5, / 33

26 Motivation Networks Figure : Network of pages and hyperlinks on a Website. From: Networks, Mark Newman, Denis Helic (KMI, TU Graz) Dbase2 Oct 5, / 33

27 Motivation Database systems Taking into account the networking environment Standalone database systems no network Distributed database system network is needed Denis Helic (KMI, TU Graz) Dbase2 Oct 5, / 33

28 Motivation Database systems In this course we discuss semi-structured data E.g. networked data records such as Wikipedia E.g. Web access logs Also, we discuss Distributed Database Systems Denis Helic (KMI, TU Graz) Dbase2 Oct 5, / 33

29 Motivation How much data is being produced? Study at Berkley: projects/how-much-info-2003/ The World Wide Web contains about 170 terabytes of information on its surface This is seventeen times the size of the Library of Congress print collections Instant messaging generates five billion messages a day This is 274 Terabytes a year Denis Helic (KMI, TU Graz) Dbase2 Oct 5, / 33

30 Motivation How much data is being produced? generates about 400,000 terabytes of new information each year worldwide P2P file exchange on the Internet is growing rapidly That was 2003, what do we have today? In 1993: G transferred over the Internet per year Denis Helic (KMI, TU Graz) Dbase2 Oct 5, / 33

31 Motivation How much data is being produced? generates about 400,000 terabytes of new information each year worldwide P2P file exchange on the Internet is growing rapidly That was 2003, what do we have today? In 1993: G transferred over the Internet per year In 2008: G transferred over the Internet in a second Denis Helic (KMI, TU Graz) Dbase2 Oct 5, / 33

32 Motivation How much data is being produced? Data, data everywhere: Denis Helic (KMI, TU Graz) Dbase2 Oct 5, / 33

33 Motivation How much data is being produced? We need new database technology to manipulate these huge amounts of data It should be very fast We will need to pass on e.g. consistency or simplicity NoSQL technologies try to address this problem Denis Helic (KMI, TU Graz) Dbase2 Oct 5, / 33

34 Course Highlights Databases 2: some highlights NoSQL Map-Reduce Graph Databases Python Denis Helic (KMI, TU Graz) Dbase2 Oct 5, / 33