Outline... Goals of the survey... Web mining. Web Content Mining: Definition. Web mining. Goals of the survey Tools Web content mining

Transcription

1 Outline... Web Content Mining Goals of the survey Tools Web content mining Václav Snášel, Miloš Kudělka VSB-Technical University of Ostrava Czech Republic 2 Goals of the survey... Classification of recent approaches. Web page segmentation. Genre detection. Table extraction. Opinion, News, and Discussion extraction. Product details and Technical features extraction Web mining Web content mining describes the discovery of useful information from Web contents. The goal of Web content mining is to improve finding information or filtering information for the users. Web structure mining tries to discover the model underlying the link structures of the Web. This model can be used to categorize Web pages and can be useful to generate the relationship between Web sites. Web usage mining tries to make sense of the data generated by the Web surfer's sessions or behaviors. Web usage mining mines the data derived from the interactions of the users. 3 4 Web mining Web Content Mining: Definition Web mining - the application of data mining techniques to extract knowledge from Web content, structure, and usage. Web Content Mining Text Image Audio Web Mining Web Structure Mining Hyperlinks Document Structured Web Usage Mining Web Server Logs Application Level Logs Application Server Logs Web Content Mining is the process of extracting useful information from the contents of Web documents. It may consist of text, images, audio, video, or structured records such as lists and tables. Web Content mining refers to the overall process of discovering potentially useful and previously unknown information or knowledge from the Web data. Vedio Structured Records 5 6

2 Web usage mining Web usage mining, which aims to discover interesting and frequent user access patterns from web usage data, can be used to model past web access behavior of users. The acquired model can then be used for analyzing and predicting the future user access behavior. In Semantic Web environment, user access behavior models can be shared as ontology. A Web page is like a family house Each of its sections has its significance, determined by the function which it serves. Every section can be named so that everybody imagines the same thing under that name. Three tasks for a blindfolded person: what sections the building contains the purpose of the building furnishings of individual sections 7 8 Task for web mining Usability principles are a good foundation Web usability received renewed attention as many early e-commerce Web sites started failing in 2000 (Wikipedia). User Centered Design - corresponds to what users are used to and does not make the user change their way of working. In which way does the visual organization of the Web pages help to lead the visual exploration for information retrieval? 9 10 Usability principles are a good foundation Eye-tracking conclusion: It must be compatible with the set of the designer's intentions. It must be compatible with the set of the user's potentials Gestalt principles can provide a theoretical base... Proximity: If things are close together viewers will associate them with one another. Similarity: Similar elements tend to be perceived as a group. Continuity: Our eyes want to see continuous lines and curves formed by the alignment of smaller elements. Closure: Elements are not completely enclosed in a space. If enough information is provided, elements tend to be perceived as a group

3 Opportunities and Challenges Web offers an unprecedented opportunity and challenge to data mining The amount of information on the Web is huge, and easily accessible. The coverage of Web information is very wide and diverse. One can find information about almost anything. Information/data of almost all types exist on the Web, e.g., structured tables, texts, multimedia data, etc. Much of the Web information is semi-structured due to the nested structure of HTML code. Much of the Web information is linked. There are hyperlinks among pages within a site, and across different sites Opportunities and Challenges The Web is noisy. A Web page typically contains a mixture of many kinds of information, e.g., main contents, advertisements, navigation panels, copyright notices, etc. Above all, the Web is a virtual society. It is not only about data, Tools information and services, but also about interactions among people, organizations and automatic systems, i.e., communities. The Web is also about services. Many Web sites and pages enable people to perform operations with input parameters, i.e., they provide services. The Web is dynamic. Information on the Web changes constantly. Keeping up with the changes and monitoring the changes are important issues Tools Vector Space Model Latent semantic indexing Formal Concept Analysis Patterns, Patterns language Collaborative search Vector space model 17 18

4 Vector space model Documents are represented as column vectors of term weights in a term-bydocument matrix A. Vector space model In another words. Documents are represented as linear combination of terms. A = d 1 d 2 d 3 d m t t t t n d 1 = Σ w 1i t i d 2 = Σ w 2i t i. d n = Σ w ni t i Vector SpaceModel Vector SpaceModel t1 t2 t3 q dj d d d d d d d q t1 t2 t3 q dj d d d d d d d q Similarity in vector model Similarity in vector model Similarity between two documents or a document and a query is usually calculated as normalized scalar product of their vectors (cosine measure). j dj Θ q i 23 24

5 Vector SpaceModel wij > 0 whenever ki dj wiq >= 0 associated with the pair (ki,q) vec(dj) = (w1j, w2j,..., wtj) vec(q) = (w1q, w2q,..., wtq) To each term ki is associated a unitary vector vec(i) The unitary vectors vec(i) and vec(j) are assumed to be orthonormal (i.e., index terms are assumed to occur independently within the documents) The t unitary vectors vec(i) form an orthonormal basis for a t- dimensional space In this space, queries and documents are represented as weighted vectors 25 Let, Vector SpaceModel N be the total number of docs in the collection ni be the number of docs which contain ki freq(i,j) raw frequency of ki within dj A normalized tf factor is given by f(i,j) = freq(i,j) / max(freq(l,j)) where the maximum is computed over all terms which occur within the document dj The idf factor is computed as idf(i) = log (N/ni) the log is used to make the values of tf and idf comparable. It can also be interpreted as the amount of information associated with the term ki. 26 Vector SpaceModel Vector SpaceModel The best term-weighting schemes use weights which are give by wij = f(i,j) * log(n/ni) the strategy is called a tf-idf weighting scheme For the query term weights, a suggestion is wiq = (0.5 + [0.5 * freq(i,q) / max(freq(l,q)]) * log(n/ni) The vector model with tf-idf weights is a good ranking strategy with general collections The vector model is usually as good as the known ranking alternatives. It is also simple and fast to compute. How to compute the weights wij and wiq? A good weight must take into account two effects: quantification of intra-document contents (similarity) tf factor, the term frequency within a document quantification of inter-documents separation (dissimilarity) idf factor, the inverse document frequency wij = tf(i,j) * idf(i) Vector SpaceModel Advantages: term-weighting improves quality of the answer set partial matching allows retrieval of docs that approximate the query conditions cosine ranking formula sorts documents according to degree of similarity to the query Disadvantages: assumes independence of index terms; not clear that this is bad though Latent semantic indexing 29 30

6 Latent semantic indexing LSI k-reduced singular decomposition of the term-by-document matrix Latent semantics hidden connections between both terms and documents determined on documents content Document matrix D k = Σ k V kt (or D k = V kt ) Term matrix T k = U k Σ k (or T k = U k ) Query in r. dimension q k = U k T q (or q k = Σ k -1 U kt q) Latent semantic indexing In another words. Documents are represented as linear combination of meta terms. d 1 = Σ w 1i m i d 2 = Σ w 2i m i. d n = Σ w ni m i Retrieval in LSI Similarity between two documents or a document and a query is usually calculated as normalized scalar product of their vectors of meta term. 33 Singular value decomposition For an m n matrix A of rank r there exists a factorization (Singular Value Decomposition = SVD) as follows: σ i = λ i A = UΣV T m m m n V is n n The columns of U are orthogonal eigenvectors of AA T. The columns of V are orthogonal eigenvectors of A T A. Eigenvalues λ 1 λ r of AA T are the eigenvalues of A T A. ( σ... ) Σ = diag 1 σ r 34 Singular value decomposition Semi-Discrete Decomposition (SDD) Terms Dokuments A k (n x m) = U k s 1s2 * * Σ k (n x k) (k x k) s k V k T (k x m) Defined as A A k = X k D k Y K T. Each coordinate of X k and Y k is constrained to have entries from the set ϕ = { 1, 0, 1} The matrix D k is a diagonal matrix with coordinates 0. Optimal choice of (x i, d i, y i ) for a given k can be determined using greedy algorithm, based on the residual R k = A A k 1 (where A 0 is a zero matrix). Although we speak about rank-k SDD, it is a sum of rank-1 matrices

7 Latent semantic indexing 1/3 What is meta term? Meta term is linear combination of terms. In ideal case each meta term will identify one topic in a way which can be done automatically. Meta term 1/1 SVD m 1 (20): state ( ) with ( ) would ( ) scored ( ) but ( ) had ( ) will ( ) have ( ) city ( ) are ( ) they ( ) was ( ) her ( ) game ( ) you ( ) she ( ) that ( ) his ( ) said ( ) points ( ) Meta term 1/2 SVD m 3 (20): points (0.4706) scored (0.3439) game (0.2461) lead (0.1741) rebounds (0.1381) league (0.1371) half (0.1047) team (0.1029) quarter (0.0982) play (0.0923) coach (0.0866) victory (0.0864) led (0.0856) season (0.0848) games (0.0802) second (0.0800) conference (0.0798) basketball (0.0778) point (0.0709) seconds (0.0708) Result: The topic in these articles is possibly sport Meta term bring some questions What does the meta term mean? Is there some interpretation of meta terms? What do the term weights in meta term represent? How to derive correct terms from higher-rank SVD concepts? SDD Meta terms Unlike SVD, the term weights in reduced space are -1, 0, +1, with distinct positive and negative meta terms Since the decomposition is a sum of rank-1 matrices, we do not need to determine how to derive meta terms from higher-level LSI concepts Meta term 1/1 - SDD m 4 (13): +putt +stroke +shoot +par +pga +golf +tournament +bogey +tour +round +hole +birdie +nicklaus Result: The topic in these articles is golf 41 42

8 Meta term 1/2 - SDD m 6 (5): +party +soviet +gorbachev +communist +union Result: The topic is USSR m 28 (11): +crash +engine +aircraft +plane +airport +pilot +air +passenger +flight +fly +airline Result: The topic is airline accidents WordNet Thesaurus, synonym dictionary Sets of synonyms (synsets) for nouns, verbs, adjectives and adverbs are stored in ontology several hierarchies have been created We have special generalization/specialization hierarchy for nouns and verbs (hypernym/hyponym) Mapping meta terms to synsets 1/1 Mapping meta terms to synsets 1/2 F : M ontolog y M set of meta terms Syn set of synset Syn 45 Every term stored in WordNet is assigned to one or more synsets, in noun, verb, adverb or adjective category. The corresponding synsets are ordered by term frequency in corpus used for WordNet creation. Synsets have a text description of their meaning Example: noun task 1. (18) undertaking, project, task, labor -- (any piece of work that is undertaken or attempted; "he prepared for great undertakings") 2. (15) job, task, chore -- (a specific piece of work required to be done as a duty or for a specific fee; "estimates of the city's loss on that job ranged as high as a million dollars"; "the job of repairing the engine took several hours"; "the endless task of classifying the samples"; "the farmer's morning chores") 46 Mapping meta terms to synsets 1/3 We convert the document vector in term space to synset space by taking all nonzero term weights and adding them to all corresponding synsets. Before addition, the weights may be multiplied by synset importance. Mapping meta terms to synsets 1/3 How to measure quality of mapping meta terms to synsets? What is the worst, good, the best? task (1.2) undertaking (0.05) undertaking, project, task, labor job, task, chore undertaking (the trade of a funeral director) 47 48

9 Mapping meta terms to synsets 1/3 How to measure quality of mapping meta terms to synsets? Levels above given synset Obtain more general concepts which may contain terms we would like to retrieve. We should not take too many levels, otherwise the precision will be poor. What is the worst, good, the best? Qualitative measures mapping F by Precision (P) and Recall (R) representative head of state President of the United States Prime Minister, PM, premier emissary 49 Bush 50 Experimental data Los Angeles Times articles from TREC collection with ca 57,000 terms Princeton WordNet 2.0 and its SDK was used, direct access to WordNet structures SDD was calculated with C-based version of SDDPack 51 Most important SDD meta terms Meta term Terms 4 putt stroke shoot par pga golf tournament bogey tour round hole birdie nicklaus 6 party soviet gorbachev communist union 15 government income interest exchange mortgage net treasury oil bond economic fiscal profit loan tax loss lower cent decline dollar trade price rate investor yen period fall trader quarter economy economist rise deficit increase revenue corp york total shares earnings federal 1989 inflation recession firm share sale index investment growth gain sell analyst billion higher market stock 52 Calculated meta terms Meta term 22 horse race Terms 24 network nbc cable broadcast television channel abc cbs game 28 crash engine aircraft plane airport pilot air passenger flight fly airline 33 labor union 36 german east germany 47 yard team league player coach game 63 muslim islamic islam rushdie Ontology Definition: to add semantic annotation to web documents so that they can be easily understand by human and read by machines for further inferences Ontology Learning: semi-automatic extraction of semantics from the web to create an ontology. Mapping and Merging Ontologies: to merge different ontologies and build a new domain specific ontology Instance Learning: automatic or semi-automatic methods to extract information from web-related documents, either to help in annotating new documents or to extract additional information from existing unstructured or partially structured documents

10 Creating an Ontology Ontology is a conceptualization of domain into human understandable but machine readable formats. A quadruple of entities, attributes, relationships and axioms. Steps in creating an ontology for the data : determining the scope of the ontology reusing existing Ontologies enumerating all the concepts needed defining the taxonomy defining the properties defining facets of the concepts defining instances Are normally Performed by Ontology Engineer Can be performed semi-automatically LSI x Ontology -- Conclusion Better results than with SVD-based LSI Faster calculation than BFA-based approach Some meta terms contain only synonyms or parts of word phrases Manual classification is straightforward WordNet is not much suitable since similar words have different hypernyms. Even other similarity axes do not bring better results. On a different (e.g. domain-dependent) ontology, or classification could yield better results Formal Concept Analysis Formal Concept Analysis Formal concept analysis (FCA) has been introduced by R. Wille. Knowledge acquisition by methods of formal concept analysis. In E. Diday, ed. Data Analysis, Learning Symbolic and Numeric Knowledge. Nova Science Publishers, pp , New York, and applied in many quite different realms like psychology, sociology, anthropology 57 B. Ganter and R. Wille. Formal Concept Analysis, Mathematical Foundation. Springer, Heidelberg, Formal Concept Analysis Formal Concept Analysis A concept lattice is an ordered hierarchical structure of formal concepts that are defined by a binary relation between an object set and an attribute set. Discovering sensible groupings of objects that have common attributes in a certain context C = (O, A, I) Concept is a maximal set of objects (extent) sharing a set of attributes (intent) (X O, Y A) so that X = τ(y) = {o O a Y: (o, a) I} and Y = σ(x) = {a A o X: (o, a) I} 59 60

11 Reducing size of concept lattice Concept Lattice top c 6 c 5 c 4 c 3 c 0 c 1 c 2 bot Query navigation FCA-Merge: method top c 5 c 6 c4 c 5 O1 c 3 c1 O1 c 3 c 0 1st step 2nd step 3rd step Cluster v Concept Analysis Multiple partitionings Clustering does not show all possibilities Items in multiple groups Features and clusters Origin of cluster decision is lost Concept more efficient computationally Clustering needs more filtering Patterns, Patterns language 65 66

12 Patterns in Architecture Does this room makes you feel happy? Why? Light (direction) Proportions Symmetry Furniture And more Patterns - LIGHT ON TWO SIDES OF EVERY ROOM Architecture, Design Patterns, When they have a choice, people will always gravitate to those rooms which have light on two sides, and leave the rooms which are lit only from one side unused and empty. (Alexander et al., 1977 pattern 159) Patterns - LIGHT ON TWO SIDES OF EVERY ROOM Patterns The solution is then included: Locate each room so that it has outdoor space outside it on at least two sides, and then place windows in these outdoor walls so that natural light falls into every room from more than one direction. (Alexander et al., 1977 pattern 159) 69 Architecture, Design Patterns, In essence, patterns are structural and behavioral features that improve the applicability of software architecture, a user interface, a Web site or something another in some domain. J. Tidwell, Designing Interfaces: Patterns for Effective Interaction Design, O'Reilly Media, Inc., What is a Design Pattern? A description of a recurrent problem and of the core of possible solutions. In Short, a solution for a typical problem Why do we need Patterns? Reusing design knowledge Problems are not always unique. Reusing existing experience might be useful. Patterns give us hints to where to look for problems. Establish common terminology Easier to say, "We need a Facade here. Provide a higher level prospective Frees us from dealing with the details too early In short, it s a reference 71 72

13 History of Design Patterns Structure of a design pattern* Christopher Alexander The Timeless Way of Building A Pattern Language: Towns, Buildings, Construction Gang of Four (GoF) Design Patterns: Elements of Reusable Object-Oriented Software Many Authors Architecture Object Oriented Software Design Other Areas: HCI, Organizational Behavior, Education, Concurent Programming Pattern Name and Classification Intent a Short statement about what the pattern does Motivation A scenario that illustrates where the pattern would be useful Applicability Situations where the pattern can be used *According to GoF Structure of a design pattern Structure A graphical representation of the pattern Participants The classes and objects participating in the pattern Collaborations How to do the participants interact to carry out their responsibilities? Consequences What are the pros and cons of using the pattern? Implementation Hints and techniques for implementing the pattern 75 Patterns Architecture, Design Patterns, In essence, patterns are structural and behavioral features that improve the applicability of software architecture, a user interface, a Web site or something another in some domain. J. Tidwell, Designing Interfaces: Patterns for Effective Interaction Design, O'Reilly Media, Inc., Pattern Pattern There are catalogs of patterns. For example: Tidwell, Designing Interfaces: Patterns for Effective Interaction Design. O'Reilly Media, Inc., For pattern description we use the structure originated by Kent Beck Title appropriate pattern name Problem: A single brief sentence describing the problem which pattern solves. Context: A list of situations where the pattern occurs. Forces: A list of details which influence the pattern identification. We are focusing especially on features useful for automatic detection. Solution: Description of the solution with examples

14 Patterns - Catalogue Patterns - Catalogue Pattern (Toy) Example <?xml version="1.0" encoding="utf-8"?> - <PATTERN> <ID>0</ID> <NAME>Information about price</name> <PROXIMITY>8</PROXIMITY> <BASE_WEIGHT>1</BASE_WEIGHT> <PROMINENCE_WEIGHT>1</PROMINENCE_WEIGHT> <COMPOSITE_WEIGHT>2</COMPOSITE_WEIGHT> <RECURRENT_WEIGHT>0,25</RECURRENT_WEIGHT> <TEXTUAL_WEIGHT>0</TEXTUAL_WEIGHT> <SYNERGY_WEIGHT>2</SYNERGY_WEIGHT> - <PRIMARY_KEYWORDS> <WORD>EU</WORD> <WORD>Dollar</WORD> <WORD>Price</WORD> </PRIMARY_KEYWORDS> - <SECONDARY_KEYWORDS> <WORD>Price</WORD> <WORD>Prices</WORD> <WORD>monetary value</word> <WORD> guarantee </WORD> <WORD> warranty </WORD> <WORD> guaranty </WORD> <WORD> goods </WORD> <WORD> commodity </WORD> </SECONDARY_KEYWORDS> - <PRIMARY_ONTOLOGIES> <WORD><price_token></WORD> </PRIMARY_ONTOLOGIES> - <SECONDARY_ONTOLOGIES> <WORD><percentage_token></WORD> </SECONDARY_ONTOLOGIES> </PATTERN> 81 Gestalt principles Visual systems usually implement the four basic principles: Proximity - Similar information are close. Similarity Similar things have silmilar meanin. Continuity- Each information follow one by one. Closure Related information are grouping. 82 Patterns - Gestalt principles Following the Gestalt principles we can suppose a page pattern as a group of characteristic technical elements (whose are based on GUI patterns such as lists, tables, continuous texts) and group of domain specific elements for the domain we are involved in (typical keywords related to given pattern and other entities such as the price, date, percent etc.). The key aspect of the pattern manifestation is that the introduced elements are close to each other. Collaborative search 83 84

15 Collaborative search Motivation Query similarity measures Comparing query similarity measures: an experiment Web Search The Web is fast growing and quickly changing dynamic environment An ultimate documentary database with extreme number of users and documents The data is omnipresent but the information must be retrieved Efficient search becomes key ability in networked environments Traditional (consensual) search paradigms are failing to keep up with the growth of Web and increasing number of users Document (data) centric design of search services implements some principles more than 30 years old New approaches are sought Solutions? Personalization Web Search as an Information Retrieval Activity Boolean Information Retrieval Model Web search engines are sophisticated information retrieval systems forged to the needs of World Wide Web An Information Retrieval System (IRS) is a software tool for data representation, storage and information search An IRS provides IRS two main functions: data storage information retrieval (to satisfy users information need) An IR model is a formal background defining internal document representation, query language and document query matching mechanism Among the oldest but till nowadays widely used information retrieval models Based on set theory, Boolean logic and exact document query match principle The documents are represented as sets of indexed terms and search expressions are implemented as Boolean logic formulas composed of search terms and standard Boolean operators AND, OR and NOT Very appealing Extended Boolean IR model (fuzzy), most powerfull query language Collaborative Search Community based adaptive web search Leverage of the search results by reusing and exploiting similar search sessions Similar search sessions are reused to enrich the result set The identification of similar search sessions is key part of collaborative search technology Query similarity Result set similarity User feedback similarity Query Similarity Measures Query similarity metrics are usually based on the similarity of search queries as fulltext expressions. Term based similarity term overlay, Levenshtein (edit) distance Query tree similarity (Cordón et.al.) Levenshtein distance extended to evaluate similarity of query syntax trees Boolean expression similarity (T. Radecki) Boolean query similarity measure based on Jaccard s coefficient Equivalent S* independent on result set has been defined 89 90

16 Boolean Expression Similarity S* Experiment Data: an extract from the Reuters Corpus Volume 1 (RCV1) A and (not B or C) ((a ( NOT b NOT c)) OR ((a ( NOT b c)) OR (a (b c)))) 91 a Jaccard s coefficiet b Levenshtein distance c Levenshtein tree distance d S* 92 Collaborative search Collaborative search is a promising search improvement method With Jaccard s coefficient as an objective metrics of query similarity, S* can be used as its result independent equivalent Collaborative search can provide personalized and safe (i.e. anonymous) service Web content mining Web content mining algorithm is like a blindfolded person... Algorithms for the detection of page type (Genre detection). Algorithms for the detection of page parts (on a domain dependent or domain independent level). Algorithms for the extraction of information content (Web information extraction). Visual layout based Web page analysis... The trend is evolving towards visual layout based Web page analysis... A Web page is represented by various individuals formats (VIPS, MDR, m-tree, zone-tree,...). The purpose is to find data records (or sub Web pages with a useful content). The aim can be a comparison of two Web pages or sub Web pages

17 Genre detection methods The goal of Genre detection methods is to assign the Web page to a known type... Methods are based on existing (manually identified) classifications. In traditional genre classification, one page belongs to a single genre. There is a need of multi genre classification schemes. Known approaches are focused on home pages, e-shopping, academic Web pages, news, and blogs. 97 Tables Tables are an important element for structuring related data... Domain independent Named Web object Tables are analyzed along four aspects: Physical - a description in terms of inter-cell relative location Structural - the topology of cells as an indicator of their navigational relationship Functional - the purpose of areas of the tables in terms of data access Semantic - the meaning of text in the table and the relationship between the interpretation of cell content 98 Opinion extraction Opinion extraction is about how to summarize customer opinions on product features... Domain dependent Named Web objects The main source for analysis: Opinions of customers on product Web pages Discussions on thematic forums Individual reviews in the form of articles Product details Product details and features usually contain a picture, product name, price information... Domain dependent Named Web object The main source for analysis is a Product page How to extract information and save into a database and then use it How to extract product technical features (the aim is to be able to compare similar products) DynamicMining Motivation Current methods of Web content mining focus on analyzing static web sites and cannot deal with constantly changing web sites, such as news sites. Dynamic Mining propose a method for mining online news sites. This method applies dynamic schemes for exploring these web sites and extracting news reports, and uses domain independent statistical analysis for trend analysis. The overall method is an application of web mining that goes beyond straightforward news analysis, trying to understand current society interests and to measure the social importance of ongoing events. We want to buy mobile phone

18 Motivation Motivation Object-level Information Extraction A Web object is constructed by collecting related data records extracted from multiple Web sources. The sources for holding object information could be HTML pages, documents put on the Web (e.g. PDF, PS, Word, and other formats.), and deep contents hidden in Web databases. (In previuos Figure.) Motivation There is already extensive research to explore algorithms for extraction of objects from Web sources. Object Identification and Integration Each extracted instance of a Web object needs to be mapped to a real world object and stored into the Web data warehouse. To do so, we need techniques to integrate information about the same object and disambiguate different objects. Motivation Web object retrieval After information extraction and integration, we should provide retrieval mechanism to satisfy users information needs. Basically, the retrieval should be conducted at the object level, which means that the extracted objects should be indexed, ranked and clustered against user queries Algorithm 1. For proximity we defined method how to measure closeness (distance) between entities in searched text segments. 2. For similarity we defined method for measuring similarity of two searched text segments (for Discussion we are able to identify repetition of replies). We work with comparison of trees representing text segments. 3. For continuity we defined method how to find out whether two or more found text segments make together instance of pattern. We assume that two or more little-similar text segments (trees of entities from one pattern) match together. 4. For closure we defined a method for computation of weight of one single searched text segment. In essence we used two criteria. We rated shape of the segment tree (particularly ratio of height and entity count) and quantity of all words and paragraphs in text segment. On the overall computation of weight also the proximity rate participates. 107 Algorithm membership computation FOR each page entity in all page entities IF page entity is pattern entity THEN IF does not exist snippet to add page entity to THEN create new snippet in list of snippets END IF add page entity to snippet END IF END FOR FOR each snippet in list of snippets compute proximity of snippet compute closure of snippet compute value(proximity, closure) of snippet IF value is not good enough THEN remove snippet from list of snippets END IF END FOR compute similarity of list of snippets compute continuity of list of snippets compute value(similarity, continuity) of pattern RETURN value 108

19 Experiments Experiments - Re-ranking We collected 31,738 various web pages which we got from the Google search engine using queries on products. After the analysis we discovered that on the 11,038 web pages there was not any extracted patterns. There were more than 200 searches of products tested (cellular phones, computers, components and peripheries, electronics, sport equipment, cosmetics, books, CDs, DVDs, etc.). 5 4,5 4 3,5 3 2,5 2 1,5 1 0, Standard Patterns Experiments - Retrieval Accuracy Experiments - Re-ranking relevant pages retrieved intopt returns RA = T Pattern Extraction Implementation In our experiment we were searching web pages in sets of thirty using very precise query. The query contained product identification (ex. Nokia 9300) and group of six words from the pattern dictionary connected in OR relation for making query more accurate. From the searched pages our algorithm extracted nine patterns (Price Information, Purchasing Possibility, Special Offer, Annuity Selling, Product information, Discussion, Review, Sign on possibility, Advertising). For evaluation of each pattern we used seven criterions. Each criterion was rated using threedegree scale. In all it is expressed using 21 Boolean values

20 Pattern Extraction In our experiment we were searching web pages in sets of thirty using very precise query. The query contained product identification (ex. Nokia 9300) and group of six words from the pattern dictionary connected in OR relation for making query more accurate. From the searched pages our algorithm extracted nine patterns (Price Information, Purchasing Possibility, Special Offer, Annuity Selling, Product information, Discussion, Review, Sign on possibility, Advertising). For evaluation of each pattern we used seven criterions. Each criterion was rated using three-degree scale. In all it is expressed using 21 Boolean values. SOM web pages from selling product domain SOM web pages from selling product domain Vision The crucial aspect of our approach is that we do not need to analyze page s HTML code. Our algorithm is based on analysis of plain text of the page. For page evaluation we do not use any meta-information about page (such as title, hyperlinks, meta-tags and so on). We also confirmed that key characteristics of web patterns are independent of language environment. We tested our method in English and Czech language environment. The only thing we had to do was to change patterns dictionaries Pattrio: Inspired by Patterns and Objects... Web design patterns and patterns languages Named Web object as a Web design pattern projection Catalog of Named Web objects Detection of Named Web objects Use of Named Web Objects Design patterns and pattern languages "Each pattern describes a problem which occurs over and over again in our environment, and then describes the core of the solution to that problem, in such a way that you can use this solution a million times over, without ever doing it the same way twice" Christopher Alexander Patterns are usually related to each other and they occur in groups. They can be worked with similarly as with a dictionary, because each pattern has its name, which characterizes its use

21 Patterns are intended for developers and they do not contain technical details... Design pattern is a text description about how to solve an existing problem. Technical details are important for the recognition by the user (and by the algorithm). "The page contains the Price information, the Purchase possibility and the Special offer. There are also Technical features and the Discussion at the bottom..." A different description has to be used (Pattrio catalog). The Named object is a projection of a Web design pattern (or Genre) to a concrete part of Web page Discussion pattern Problem: How can a discussion about a certain topic be held? How can a summary of comments and opinions be displayed? Context: Social field, community sites, blogs, etc. Discussions about products and service sales. Review discussions. News article discussion. Forces: A page fragment with a headline and repeating segments containing individual comments. Keywords to labeling discussion on the page (discussion, forum, re, author, ). Keywords to labeling people (first names, nicknames). Date and time. There may be a form to enter a new comment. Segments with the discussion contributions are similar to the mentioned elements view in form. Solution: Usually, an implementation using a table layout with an indentation for replies (or similar technology leading to the same-looking result) is used. The Discussion is often together with the Login. If Discussion is on a product Web page there are usually Purchase possibility and Price information. The Discussion can be alone on the page. In another case there is also the Something to read. In different domains the Discussion can be displayed with Review, News, etc Design patterns Words: {Main: re, reply, discussion, forum, author, question, answer, thread, contribution, subject, sent; Complementary: date, name, post, topic} Data types: {Main: date, time, first name; Complementary:} Technical elements: {link, label, input, table} Rules: {proximity: 16; closure: normal; similarity: high; continuity: low} A detection algorithm is based on Gestalt principles... A set of elements (entities) that are characteristic for the Named Web object (words, data types, technical elements). A set of partial algorithms whose results are the extracted data types. Algorithms for the evaluation of rules (proximity, closure, similarity, continuity) and relations. Associations: {Contains: Short Paragraphs; Uses: Date per Paragraph; Complements: Review and Comments}

22 seller information <paragraph token> feedback score <numeric_token> <paragraph token> positive feedback <percentage_token> <paragraph token> member since <date_token> in united states <paragraph token> read feedback comments <paragraph token> add to favorite sellers <paragraph token> view seller other items seller information <paragraph token> feedback score <numeric_token> positive feedback <percentage_token> <paragraph token> member since <date token> The accuracy of Pattrio method is about 80%... Named Web objects can provide a simple description for SERP The user sees as similar the pages with similar features... Is the Web a geographical network? The Named Web object can be understood as a feature of the page. The Web page can be represented as a vector of a defined dimension (24). The vector-space model can be used. The similarity of two pages can then be interpreted as the similarity of vectors representing these pages (cosine measure). Jack Goldsmith, Tim Wu, Who Controls the Internet: Illusion of a Borderless World. Oxford University Press, Focuses on state responses to the Internet s challenge to national sovereignty. The main argument is that national governments, through coercion and control over local intermediaries, still exert regulatory control in the realm of the Internet. Thus, Goldsmith and Wu question the popular notion that the Internet is erasing national boundaries and rendering geography obsolete. Declaration of Cyberspace Independence

23 Conclusions This tutorial introduced several topics of Web content mining: Structured data extraction Sentiment classification, analysis and summarization of consumer reviews Information integration and schema matching Knowledge synthesis Template detection and page segmentation The coverage is by no means exhaustive. Research is only beginning. A lot to be done References at: