Extraction of Segments from Web 2.0 Pages

Size: px

Start display at page:

Download "Extraction of Segments from Web 2.0 Pages"

Helena Mathews
6 years ago
Views:

Extraction of Segments from Web 2.0 Pages URL Genre Detection Page Segmentation Segment Classification Output Format httc Hessian Telemedia Technology Competence-Center e.v - www.httc.de Dipl. Inform.

1 Extraction of Segments from Web 2.0 Pages URL Genre Detection Page Segmentation Segment Classification Output Format httc Hessian Telemedia Technology Competence-Center e.v - Dipl. Inform. Renato Dominguez Garcia Renato.Dominguez.Garcia@KOM.tu-darmstadt.de Tel KOM - Multimedia Communications Lab Prof. Dr.-Ing. Ralf Steinmetz (Director) Dept. of Electrical Engineering and Information Technology Dept. of Computer Science (adjunct Professor) TUD Technische Universität Darmstadt Merckstr. 25, D Darmstadt, Germany Tel , Fax RDG Web2.0_Extraction ppt author(s) of these slides 2008 including research results of the research network KOM and TU Darmstadt otherwise as specified at the respective slide 13. Oktober 2008

2 KOM Research Program Ubiquitous Communication Mobile Networking Peer-to-Peer Networking E-Learning IT Architectures E-Business & E-Finance Communication Services & IP Telephony Workflows Network Mechanisms Quality of Service, Dependability & Security Filtering of irrelevant Information Knowledge Media Application Areas Fundamentals Research Areas KOM Multimedia Communications Lab 2

Motivation The influence of the internet for the US president election has grown since 2004 (+11%) [1] Primary information source about the US president election for young Americans (18 29 years old)

Automatically detection of blogs, wikis and forums may be useful Information extraction from small segments is easier than from large web pages Genre Detection and Information extraction can be used

3 Motivation The influence of the internet for the US president election has grown since 2004 (+11%) [1] Primary information source about the US president election for young Americans (18 29 years old) is the internet (42%) [1] Each third American read blogs [2] PR professionals recognize importance of blogs [3] Information extraction from blogs can help to understand the public opinion Automatically detection of blogs, wikis and forums may be useful Information extraction from small segments is easier than from large web pages Genre Detection and Information extraction can be used in other fields: Community Mining, Improvement of search results vs Source: Source: Topic N o of comments Author Date Content [1] [2] DER SPIEGEL 30/2008 of , Page 94 [3] KOM Multimedia Communications Lab 3

4 Our Approach: Genre Detection URL Genre Detection Page Segmentation Segment Classification Output Format Genre Detection 6 Genres Blogs (Start pages, post pages) Wikis Forums (Start pages, thread pages) Others Based on the structure of web pages (Patterns) Machine Learning Techniques Support Vector Machines 336 Features Corpus: ~ Web pages Evaluation 1345 Instances (1000 Blogs/Wikis/Forums) 87,5 % Correctly classified instances Number of detected patterns Number of outer patterns Ratio of patterned vs unpatterned Code Length of patterns Offset before first pattern starts Depth of patterns a b c d e f <-- classified as a = Blog_Page b = Blog_Post c = Wiki_Page d = Forum_Page e = Forum_Thread f = Outlier KOM Multimedia Communications Lab 4

threads share a similar structure Set of links Many long

5 Our Features: Examples Blog s start page Wiki-page Forum s start page title All posts share a similar structure patterns All threads share a similar structure Set of links Many long patterns Few patterns Many short patterns KOM Multimedia Communications Lab 5

Our Approach: Segmentation URL Genre Detection Page Segmentation Segment Classification Output Format Page segmentation (Four steps) Pre-processing (cleaning HTML) Segmentation based on the

6 Our Approach: Segmentation URL Genre Detection Page Segmentation Segment Classification Output Format Page segmentation (Four steps) Pre-processing (cleaning HTML) Segmentation based on the hierarchical structure of web Pages Visual-based segmentation Filtering based on heuristics Segment classification Machine Learning Techniques Random Forest 139 Features Corpus: ~ 500 instances Evaluation Genres (blog posts, comments, others) 97,2 % correct classified instances Source: Source: KOM Multimedia Communications Lab 6

Router Virtualization as an Enabler for Future Internet Multimedia Applications

Router Virtualization as an Enabler for Future Internet Multimedia Applications httc Hessian Telemedia Technology Competence-Center e.v - www.httc.de Osama Abboud, M.Sc Prof. Dr. Ing Ralf Steinmetz Vorname.Nachname@KOM.tu-darmstadt.de