Web 2.0 Lecture 1: Motivation and Course Overview

Transcription

1 Web 2.0 Lecture 1: Motivation and Course Overview doc. Ing. Tomáš Vitvar, Leopold-Franzens Universität Innsbruck and Czech Technical University in Prague Faculty of Information Technologies Software and Web Engineering Last modified: 24/02/ :36:55 Hellos Tomáš Vitvar Software and Web Engineering chair, CTU FIT Researcher, developer, software architect Have worked in Ireland, Austria, Germany, Czech rep. Work for ProgrammableWeb.com Member of W3C Advisory Committee EU research projects leader and evaluator Jaroslav Kuchař Ph.D. student at CTU FEL research interests in Web data mining, Web analytics, Web technologies Lecture 1: Motivation and Course Overview, CTU Summer Semester 2

2 Overview Motivation Scope, Requirements, Learnings Your project, basic concepts Example project overview Assessment, Resources, Communication Lecture 1: Motivation and Course Overview, CTU Summer Semester 3 Big Picture Web 2.0 is... beyond PHP, HTML and AJAX! Lecture 1: Motivation and Course Overview, CTU Summer Semester 4

3 Motivation in Brief The Web is for users users not only consume but also produce (prosumers) improved user experience The Web is a huge database users generate content (explicit and implicit knowledge) large portion of this database is social data The Web is Social obvious stuff: Facebook, Twitter user profiles and social links (public, private) less obvious stuff: Wikipedia, DBLP,... basis for personalized services, targeted delivery of products, commercials, etc. Lecture 1: Motivation and Course Overview, CTU Summer Semester 5 Overview Motivation Scope, Requirements, Learnings Your project, basic concepts Example project overview Assessment, Resources, Communication Lecture 1: Motivation and Course Overview, CTU Summer Semester 6

4 Scope Web 2.0 principles Read/Write Web, Programmable Web, Social Web,... Data on the Web metadata, microformats, microdata accessing the data, protocols Social Network Analysis social network structures and behavior visualization Highly dependable back-end technologies cloud-based distributed computing, databases (Cassandra, FlockDB) Lecture 1: Motivation and Course Overview, CTU Summer Semester 7 Requirements in General Software engineering know to model requirements, UML programming in general Java all examples in Java! Java 5+ syntax Web architecture URI, XML, HTTP, RESTful services Algorithms some knowledge of the graph theory Lecture 1: Motivation and Course Overview, CTU Summer Semester 8

5 What You Will Learn in Lectures Concepts and theory ~ 40% principles that drive Web 2.0 technological developments metrics and algorithms Technology ~ 40% state of the art Web 2.0 technology data, protocols, frameworks Examples ~ 20% toy examples to demonstrate concepts, theory and technologies Lecture 1: Motivation and Course Overview, CTU Summer Semester 9 Organization of Lectures 12 Lectures 2 lectures per every second week Wednesdays Czech: T9:155, 14:30-17:45 (2 lectures) Dates 1. 23/02, 2 lectures 2. 09/03, 2 lectures 3. 20/03, 2 lectures 4. 06/04, 2 lectures 5. 20/04, 2 lectures 6. 04/05, 2 lectures Lecture 1: Motivation and Course Overview, CTU Summer Semester 10

6 What You Will Train in Practicals Iterative development of a system for social data analysis assignment every second week development cycle according to the schedule Design requirements, algorithms, data models Implementation a data capture and analysis service algorithms for social data analysis visualisation of the data using Gephi GAE infrastructure services (tasks), if you use GAE we do not care about the platform but we do care about your implementation! Lecture 1: Motivation and Course Overview, CTU Summer Semester 11 Organization of Practicals Work in teams of 3 people Practicals every second week Number of sessions: 6-7, 5 major tasks 1. Design of your system, endpoint registration 2. Data capture implementation, mashups development 3. Data export, visualization 4. Data analysis 5. TBD All textual/design diagrams results in the wiki All (nicely commented!) code results in the svn Lecture 1: Motivation and Course Overview, CTU Summer Semester 12

7 Overview Motivation Scope, Requirements, Learnings Your project, basic concepts Example project overview Assessment, Resources, Communication Lecture 1: Motivation and Course Overview, CTU Summer Semester 13 Development Platform Google App Engine (optional) Free available daily quota for: CPU time (6 CPU hours) data storage (1GB) data transfer (1GB in and 1GB out) your service will have to be accessible somewhere if you do not use GAE RESTful service endpoint In GAE you can use RESTEasy Or implement HTTP Servlets Eclipse/NetBeans IDE Dev environment for Java Google App Engine convenient plugins Lecture 1: Motivation and Course Overview, CTU Summer Semester 14

8 Your Conceptual Architecture Lecture 1: Motivation and Course Overview, CTU Summer Semester 15 Example Data Views: Wikipedia Lecture 1: Motivation and Course Overview, CTU Summer Semester 16

9 Other Examples Facebook/Twitter: social networks social links: people in a social network of a FB app IMDB or Wikipedia: actors in movies social links: two actors appear in the same movie Mailing lists: people who talk to each other social links: two people appear in the same message Blogs/Forums: people who discuss social links: two people discuss the same blog post Flickr: people taking pictures social links: two people own a picture with the same tag LinkedIn: people in organizations social links: two people work in the same company Lecture 1: Motivation and Course Overview, CTU Summer Semester 17 Things to keep in mind Size of the data Limit the dataset to fit your computing resources should fit all data Resource quotas example in GAE 1GB of storage, 1GB of in and out, 6 CPU hours if you exceed daily quotas, you should divide your tasks Methods for data capture Complete Capture: a complete dataset as one snapshot repeat regularly in intervals when you need to access historical data example: your social network Incremental Capture: capture data incrementally you only capture the last data (no history) example: posts in a forum Optimal data structure nodes and edges with timestamps Lecture 1: Motivation snapshots and Course as Overview, time-views CTU Summer on Semester your 2010/2011, data@tomasvitvar 18

10 Complete Capture When you can access historical data you do not have an access to changes (increments) Twitter social links, Facebook friends, pool of URLs of a site, etc. often can access through API (be aware of API call limits) Twitter [API calls/hour]: 350 (auth), 150 (anonymous) No API? screen scraping example: crawl and scrape a pool of URLs with data Lecture 1: Motivation and Course Overview, CTU Summer Semester 19 Complete Capture: Sleeping Determine the data capture interval Big enough to capture changes in data ideally diff in snapshots should be ~30% Small enough to fit your computing resources constraints A simple method to adjust the interval dynamically: // initial interval; for example 1 day (in minutes, 60*24) interval = 1440; // read the first snapshot last_snapshot = read(); while (true) { sleep(interval); data = read(); diff = calcdiff(last_snapshot, data); // if diff > 0.3 interval will decrease, if diff < 0.3 interval will increase // let's not be so aggressive in interval changes... interval += interval * (0.3 - diff); last_snapshot = data; } // set the minimum interval so that we do not overload our service if (interval < 120) interval = 120; Lecture 1: Motivation and Course Overview, CTU Summer Semester 20

11 Incremental Capture When you can access changes in data polling: using an atom/rss feed polling interval can be between minutes (depends) pushing: using a streaming API (e.g., Twitter status updates) No API? Screen scraping example: list of items from a RSS give you HTML pages you scrape Lecture 1: Motivation and Course Overview, CTU Summer Semester 21 Overview Motivation Scope, Requirements, Learnings Your project, basic concepts Example project overview Assessment, Resources, Communication Lecture 1: Motivation and Course Overview, CTU Summer Semester 22

12 People discussing in a forum Identify Data Source Date source: University forum (CTU FIT) We get all forum pages from the forum's RSS feed Define data structure Data structure entities, persistence Classes and methods Design methods/operations Reading data (using screen scraping) Graph analysis API expose metadata and data through API RESTful architecture: resources, URLs, etc. Lecture 1: Motivation and Course Overview, CTU Summer Semester 23 Data Structure: Timestamped Graph Lecture 1: Motivation and Course Overview, CTU Summer Semester 24

13 Reading Data using Screen Scraping polling: run every minutes (as a cron job) classes: Graph, Edge, Node, Snapshot public void read(graph graph) { // fetch all pages' url from the RSS feed (each page corresponds to a topic) List urls = RssParser.getUrls (" // create the screen scraping object, org.htmlcleaner.htmlcleaner HtmlCleaner cleaner = new HtmlCleaner(); // create a new snapshot with the current time as the starting time Snapshot snapshot = new Snapshot(); for (String url : urls) { // fetch the page at the url (this will invoke GET on the url) TagNode html = cleaner.clean(url, "utf-8"); // use the title as a caption for an edge Edge edge = graph.addedge((tagnode) html.evaluatexpath("//title")[0].gettext().tostring(), snapshot); // the page contains all people's names discussing the topic (use them as names Object[] names = html.evaluatexpath("//form[@id='quickmodform']//td[@rowspan]/b for (Object name : names) graph.addnode(edge, ((TagNode)node).getText().toString(), snapshot); } // be polite, sleep for 30 seconds between fetching the pages Thread.sleep(30000); // mark the end of the snapshot snapshot.end(); } Lecture 1: Motivation and Course Overview, CTU Summer Semester 25 Register your service Registration API URI: Methods: PUT, GET Parameters: team-id id of your team PUT to register/update your endpoint send: <?xml version="1.0" encoding="utf-8"?> <endpoint> <!-- base uri (without version) --> <endpointurl> </endpoint> To use curl where endpoint.xml contains the above data, run: curl -X PUT -v -H "Content-Type: application/xml" GET to retrieve the information about your endpoint: curl Lecture 1: Motivation and Course Overview, CTU Summer Semester 26

14 API v1 Expose metadata about your dataset URI: base_uri/{version}/metadata Method: GET Parameters: version API version, v1 is the first one <?xml version="1.0" encoding="utf-8"?> <metadata> <!-- short name and description of your dataset --> <name>ctu FIT Forum</name> <description>people discussing in a forum</description> <!-- date the dataset was last updated (last snapshot) --> <lastupdate> t20:26:00.501z</lastupdate> <!-- interval in minutes you are applying to update the dataset (or polling interval <updateinterval>10080</updateinterval> <!-- list of all authors in your team and a city where you come from --> <authors> <author name="name1" city="city2" /> <author name="name2" city="city1" /> </authors> </metadata> Lecture 1: Motivation and Course Overview, CTU Summer Semester 27 Visuzalization Your Data Expose your data through your API in GEXF format Visual Analysis in Gephi Lecture 1: Motivation and Course Overview, CTU Summer Semester 28

15 Overview Motivation Scope, Requirements, Learnings Your project, basic concepts Example project overview Assessment, Resources, Communication Lecture 1: Motivation and Course Overview, CTU Summer Semester 29 Assessment Practicals Presence is mandatory but you can miss up to 1 practical without sending regrets Every task will give you 6 points max Activity in practicals will give you 10 points max covers your performance Total, to pass: Final exam Mandatory written test: 3 exercises, ~1 hour each will give you 20 points max total F2F discussion: only if Final score: points max discussion can adjust your points freely Lecture 1: Motivation and Course Overview, CTU Summer Semester 30

16 Assessment Final Presentation Final presentations Selected teams to present their project (~3 teams) last lecture (04/05) You may get extra points for the exam in exceptional cases no written exam for you We may chose you or you can volunteer deadline: by mid of April projects related to the university will have a preference Scope of your presentation minutes + discussion ~10 minutes dataset and data structure analysis and interoperation of results application for the real life Lecture 1: Motivation and Course Overview, CTU Summer Semester 31 Assessment Final Marks Mark Points In words A výborně B velmi dobře C dobře D uspokojivě E dostatečně F 49 0 nedostatečně Source: Everything good and bad will count practicals, coding, (pro-)activity, passiveness, hacking, lectures, exam, cheating,... Lecture 1: Motivation and Course Overview, CTU Summer Semester 32

17 Resources Online sources EDUX Your project (svn) online bookshelf slides /slides/lecture{x}.(html pdf) where X is the lecture number Books G. Vossen, S. Hagermann: Unleashing Web 2.0: from concepts to creativity, Elsevier/Morgan Kaufmann, 2007, ISBN ( Web 2.0 Concepts) D. Easley, J. Kleinberg: Networks, Crowds and Markets: Reasoning About a Highly Connected World, Cambridge University Press, 2010, ISBN ( Social Network Analysis) R. Steinmetz, K. Wehrle: Peer-to-Peer Systems and Applications, Springer, 2005, ISBN ( Dependable Back-end technologies) Other Many sources on the Web, to be listed throughout the course Lecture 1: Motivation and Course Overview, CTU Summer Semester 33 About Slides HTML5 Tested: Google Chrome, Safari (latest versions) keys: left, right, P print mode, B browser mode, S slideshow mode Lecture 1: Motivation and Course Overview, CTU Summer Semester 34

18 Communication Language Text: English (slides, tweets, posts, instructions, etc.) choose English or Czech for your own contributions to the wiki Voice: Czech Channels limit s in general technology news related to web engineering courses it's a reading for you, make sure you follow this Hashtags: #w2e_w20ann (W20-)related organizational announcements Direct ask me anything directly, if not of interest to public me at tomas@vitvar.com, follow me on Lecture 1: Motivation and Course Overview, CTU Summer Semester 35 Questions? Lecture 1: Motivation and Course Overview, CTU Summer Semester 36