Web Science (VU) ( )

Transcription

1 Web Science (VU) ( ) Elisabeth Lex ISDS, TU Graz March 5, 2018 Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

2 Lecturer Name: Elisabeth Lex Office: ISDS, Inffeldgasse 13, 5th Floor, Room 072 Office hours: By appointment Phone: / Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

3 Lecturer Name: Denis Helic Office: ISDS, Petersgasse 116, Room 26 Office hours: Tuesday from 12 til 13 Phone: / Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

4 Language Lectures in English Communication in German/English If in German: please informally (Du)! Examination: German/English Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

5 Outline 1 Welcome and Introduction 2 Course Organization 3 Motivation 4 Course Overview 5 The Schelling Model 6 Course Calender Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

6 Welcome and Introduction Course context Web Science (VU) ( ) Obligatory course Bachelor Software Development and Business (6th Semester: the old study plan) Obligatory course Bachelor Computer Science (6th Semester: the new study plan) Elective course in Master Computer Science/Software Development and Business if not already taken in Bachelor Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

7 Welcome and Introduction Goals of the course The goals of this course are: (i) To discuss the Web as a half technical half social system (ii) To understand that the further development of the Web requires a combined scientific and engineering approach Science: analyze the Web as an object of a scientific inquiry and learn something new Engineering: use the new knowledge and improve the algorithms (iii) To learn about the basic scientific methodology for the Web: network theory and analysis, data mining, social processes on the Web (iv) To learn about python as analysis tool Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

8 Welcome and Introduction Goals of the course Student goals: to pass the examination Bonus goal for all: to have fun! Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

9 Course Organization Course logistics Course website: socialcomputing/courses/webscience/ Slides will be made available on the course website Additional readings, references, links, etc. also on the website Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

10 Course Organization Grading Programming project in python with a provided code skeleton Handout via web page & lecture: , Submission: Visualize evolution, plot histograms of interesting quantities Final exam ( exact time & location will be announced in lecture & newsgroup) Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

11 Course Organization Grading Project: 30 points Final examination 5 written questions for a total of 50 points Total for the project and examination is 80 points Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

12 Course Organization Grading You have to reach at least 16 points for the project! You have to reach at least 25 points for the final exam! You have to reach at least 41 points combined to be positive! Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

13 Course Organization Grading 0-40 points: points: points: points: points: 1 Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

14 Course Organization Dates Until : register for the course ( :59) If you do not register you can not obtain the grade for this course Create your SVN project in TUGOnline as early as possible Special naming scheme: see submission slides Add tutor as readers If your SVN project is not created you will not be able to participate in the course Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

15 Course Organization Dates : the student project handed out (via Web page) : submission date (hard deadline) The tutor will check out your submission from SVN at the hard deadline No SVN project or tutor can not check out or nothing submitted = 0 points Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

16 Course Organization Course policy Plagiarism: By submitting home assignments, you agree that your work will be processed by plagiarism tools. You are allowed to discuss home assignments with colleagues You are not allowed to jointly work on the assignments, copy solutions or share code. The software will check first, then study assistants If they suspect a case of plagiarism you will be asked for your comment In 99% of cases this already clarifies the situation In the remaining 1% of cases Denis or me will be involved, after that the Dean of the study Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

17 Course Organization Questions? Raise them now (+1 +1) Ask after the lecture (+1) Visit us in the office hours (+1) Send us an (-1) As a side note: you should(!) interrupt me immediately ( ) and ask any question you might have during the lecture Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

18 Motivation The Web The fastest growth of any technology in the human history Time to reach 50 million people Telephone 75 years Radio 35 years TV 13 years The Web 4 years Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

19 Motivation The Web Figure: Jakob Nielsen, 100 Million Web Sites, Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

20 Motivation The Web The size of the Web Visible Web and The Deep Web (behind passwords) The size of the Web 1000 billions blogspot.com/2008/07/we-knew-web-was-big.html Indexed pages 50 billions Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

21 Motivation The Web Figure: is-the-growth-of-the-web-slowing-down-or-just-taking-a-breather/ Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

22 Motivation Web Science Web as an object of scientific inquiry [...] As the Web has grown in complexity and the number and types of interactions that take place have ballooned, it remains the case that we know more about some complex natural phenomena (the obvious example is the human genome) than we do about this particular engineered one. A Framework for Web Science by T. Berners-Lee and others Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

23 Motivation Web Science Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

24 Motivation Web Science Web as a sociotechnical system The web is half social and half technical; you can t separate the social and the technical T. Berners-Lee Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

25 Motivation Web Science What does it mean? In the beginning there were only we: software engineers We had the full control over our system, its behavior and its functionality But we implemented the algorithms that work with the data generated by the users E.g. PageRank: links created by the users E.g. Recommendations: user interactions Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

26 Motivation Web Science We lost the control over the system! What you can find on the Web depends on the algorithm and what other users are looking for What you can buy on Amazon depends on the algorithm and what other users have already bought What you watch on Netflix depends on the algorithm and what other users have already watched How can we as engineers gain the control over the system again? Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

27 Motivation Web Science Observe: e.g. collect access logs on a recommender site Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

28 Motivation Web Science Observe: e.g. collect access logs on a recommender site Measure: e.g. how many users buy which products Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

29 Motivation Web Science Observe: e.g. collect access logs on a recommender site Measure: e.g. how many users buy which products Quantify: e.g. similarity between users Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

30 Motivation Web Science Observe: e.g. collect access logs on a recommender site Measure: e.g. how many users buy which products Quantify: e.g. similarity between users Make a model: e.g. users with similar interests buy similar products Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

31 Motivation Web Science Observe: e.g. collect access logs on a recommender site Measure: e.g. how many users buy which products Quantify: e.g. similarity between users Make a model: e.g. users with similar interests buy similar products Predict with the model and validate: e.g. implement and evaluate Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

32 Motivation Web Science Observe: e.g. collect access logs on a recommender site Measure: e.g. how many users buy which products Quantify: e.g. similarity between users Make a model: e.g. users with similar interests buy similar products Predict with the model and validate: e.g. implement and evaluate Apply: engineering approach to implementing the model in the software Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

33 Motivation Web Science Observe: e.g. collect access logs on a recommender site Measure: e.g. how many users buy which products Quantify: e.g. similarity between users Make a model: e.g. users with similar interests buy similar products Predict with the model and validate: e.g. implement and evaluate Apply: engineering approach to implementing the model in the software All steps together: scientific methodology (Web Science) Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

34 Course Overview Course Topics World Wide Web: a short history What is network theory? Why it is relevant for the Web? How do networks evolve? How can we model dynamics of social influence and aggregating beliefs on the Web? How do we search in networks? Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

35 Course Overview How many of you know... 6 degrees of separation (Small-world phenomenon) Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

36 Course Overview How many of you know... 6 degrees of separation (Small-world phenomenon) Degree distribution Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

37 Course Overview How many of you know... 6 degrees of separation (Small-world phenomenon) Degree distribution Power-law networks Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

38 Course Overview How many of you know... 6 degrees of separation (Small-world phenomenon) Degree distribution Power-law networks The meaning of PageRank Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

39 Course Overview How many of you know... 6 degrees of separation (Small-world phenomenon) Degree distribution Power-law networks The meaning of PageRank Agent-based modeling Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

40 Course Overview The beginnings of the Web : Ted Nelson and Douglas Engelbart created model for linked content: Hypertext and Hypermedia 1973/74: Vint Cerf: father of the Internet, and others put together protocols and technical specifications and coin the term Internet Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

41 Course Overview The Internet and the Web Internet Web Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

42 Course Overview The beginnings of the Web : A scientist from CERN wanted to share information with scientists from CERN and other universities: proposal for World Wide Web by Tim Berners-Lee Hypertext HyperText is a way to link and access information of various kinds as a web of nodes in which the user can browse at will. It provides a single user-interface to large classes of information (reports, notes, databases, computer documentation and on-line help). We propose a simple scheme incorporating servers already available at CERN... A program which provides access to the hypertext world we call a browser... Tim Berners-Lee, R. Cailliau. 12 November 1990, CERN Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

43 Course Overview The beginnings of the Web... Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

44 Course Overview The beginnings of the Web... URL: a unique address of a Web resource (page, image,...) HTTP: a stateless protocol built on the top of TCP/IP for data exchange HTML: a simple markup language for Web documents <a href=...>: an element for linking other documents on the Web Simple, scalable, flexible: reasons for the huge success Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

45 Course Overview Web Science Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

46 Course Overview Web Science Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

47 Course Overview Science Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

48 Course Overview Theories on the Web... A few examples of assertions: Every page on the web can be reached by following less than 10 links. (True/False/Depends?) A wikipedia page contains, on average, 0.03 false facts (True/False/Depends?) 1%-4% of users express their search queries in the form of goals such as increase adsense revenue (True/False/Depends?) The average number of words per search query is more than 3 (True/False/Depends?) Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

49 Course Overview Theories on the Web... Can these statements be easily validated? Can they lead to good/interesting theories? What constitutes good theories? Clarity, simplicity Predictive and explanatory power Applicability and utility Testability and falsifiability Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

50 Course Overview Networks A significant part of this course will focus on network theory Graph theory vs. Network theory Graph theory focuses on mathematics and analytical approaches (small graphs) Network theory focuses on networks that can be observed in the real world Large empirical networks, statistical approaches There are many different forms of networks available on the Web Can you name a few of them? Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

51 Course Overview Networks P Figure: Social network of HP Labs constructed out of communication. From: How to search a social network, Adamic, Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

52 Course Overview Networks Figure: Network of pages and hyperlinks on a Website. From: Networks, Mark Newman, Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

53 Course Overview Networks On the course Web site there are links to many resources The content of this course is mainly based on the textbook: Networks, Crowds, and Markets: Reasoning About a Highly Connected World, by David Easley and Jon Kleinberg, 2010 It is a free online book! Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

54 Course Overview Agents Agents, interactions, dynamics Individual agents follow simple rules Interaction is guided by simple rules However, evolution may result in a very complex system Emergent properties Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

55 Course Overview Game of Life Invented by John Horton Conway Rules 1 Any live cell with fewer than two live neighbours dies, as if caused by underpopulation 2 Any live cell with two or three live neighbours lives on to the next generation 3 Any live cell with more than three live neighbours dies, as if by overpopulation 4 Any dead cell with exactly three live neighbours becomes a live cell, as if by reproduction Demo: Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

56 The Schelling Model The Schelling Model A small preference for a specific kind of neighbors lead to total segregation Shows how global patterns (spatial segregation) can arise from local preferences (homophily) A simple robust mechanism leads to segregation Segregation achieved even if no one explicitly aims for it Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

57 The Schelling Model How does the Schelling Model work? (1/2) Assume a population of individuals (aka agents) of type X or O Types represent immutable characteristics (e.g., age) Two populations are initially placed into random locations of a neighborhood grid After placing all agents, each cell is either occupied by an agent or empty The neighbor relationships among the cells can be represented very simply as a graph: cells are the nodes, edges are inserted between two cells that are neighbors on the grid Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

58 The Schelling Model How does the Schelling Model work? (2/2) Now, determine if each agent is satisfied with its current location Agent is satisfied if is surrounded by at least t of its own type of neighboring agents Threshold t applies to all agents in the model (in reality everyone might have a different threshold they are satisfied with) The higher t, the higher the likelihood that agents will not be satisfied with their current location Example: For example, if t = 3, agent X is satisfied if at least 3 of its neighbors are also X If fewer than 3 are X, then the agent is not satisfied, and it will want to change its location in the grid Any algorithm can be used to choose new location (e.g., random selection, nearest available location, 1 row at a time) Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

59 The Schelling Model Example (a) Initial stage (b) After one round Figure: Left image: all dissatisfied agents have an asterisk next to them. Right image: shows new configuration after all dissatisfied agents have been moved to unoccupied cells (1 row at a time) where they are satisfied. May cause other agents to become unsatisfied, then new round of movement begins Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

60 The Schelling Model Netlogo Example of the Schelling Model Go to File/Model Library/Social Science/Segregation Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

61 The Schelling Model Observations from Schelling s Model Spatial segregation takes place even though no individual agent actively seeks it Segregation doesn t happen due to built-in model agents that are willing to be in the minority Ideally, all agents are carefully arranged in an integrated pattern However, from random start hard for agents to find such integrated patterns At more general level, Schelling model is an example of how fixed characteristics (e.g., ethnicity) can become highly correlated with mutable characteristics E.g. decision where to live, which over time conforms to similarities in agents immutable types, producing segregation Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

62 Course Calender Course calendar : Introduction and Motivation : Networks I : Networks II (Random Graphs) : Small World Phenomenon I : Small World Phenomenon II Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

63 Course Calender Course calendar : Power Laws and Preferential Attachment I : Power Laws and Preferential Attachment II : Power Laws and Preferential Attachment III : Network Dynamics I (Information Cascades) Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

64 Course Calender Course calendar : Network Dynamics II (Agent-based Modeling) : Network Dynamics III (Opinion Dynamics) : Information Networks I : Information Networks II (PageRank) : Exam (exact time + location to be announced) Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56

65 Course Calender Questions? Raise them now (+1 +1) Ask after the lecture (+1) Visit us in the office hours (+1) Send us an (-1) As a side note: you should(!) interrupt me immediately ( ) and ask any question you might have during the lecture Elisabeth Lex (ISDS, TU Graz) WebSci March 5, / 56