Web Analysis in 4 Easy Steps. Rosaria Silipo, Bernd Wiswedel and Tobias Kötter

Transcription

1 Web Analysis in 4 Easy Steps Rosaria Silipo, Bernd Wiswedel and Tobias Kötter

2 KNIME Forum Analysis

3 KNIME Forum Analysis Steps: 1. Get data into KNIME 2. Extract simple statistics (how many posts, response time, response length) 3. Classify posts and detect topic shifts 4. Identify content and users

4 Step 1: Get Data Step 2: Simple Statistics Step 3: Classify Posts Step 4: Content & Users Get Data into KNIME

5 Step 1 Get Data into KNIME Two alternatives: Connect to underlying database, read content Use database nodes Crawl the web page, parse html Use XML parser & Palladian s html retriever nodes

6 Step 1 Get Data into KNIME Two alternatives: Connect to underlying database, read content Use database nodes Crawl the web page, parse html Use XML parser & Palladian s html retriever nodes

7 Step 1 - Extract Data from Database Doable but complicated: 7+ tables need to be read, prepared and joined

8 Step 1 Get Data into KNIME Two alternatives: Connect to underlying database, read content Use database nodes Crawl the web page, parse html Use XML parser & Palladian s html retriever nodes

9 Step 1 Structure of Forum Several Categories, KNIME General, KNIME Reporting, Palladian, (~20 in total)

10 Step 1 Structure of Forum Discussion threads on several sub-pages

11 Step 1 Structure of Forum Each thread consists of an initial post and a variable number of comments

12 Step 1 Crawler Flow

13 Step 1 Crawler Flow

14 Step 1 Crawler Flow

15 Step 1 Crawler Flow

16 Step 1 Crawler Flow

17 Step 1 Crawler Flow

18 Step 1 Structure of Forum Discussion threads on several sub-pages

19 Step 1 Crawler Flow

20 Step 1 Crawler Flow

21 Step 1 Crawler Flow Input for all subsequent workflows!

22 Step 1: Get Data Step 2: Simple Statistics Step 3: Classify Posts Step 4: Content & Users Extract simple Statistics (how many posts, response time, response length)

23 Step 2 Simple Statistics

24 Step 2 Simple Statistics Input table from crawler workflow

25 Step 2 Simple Statistics Meta nodes perform simple preprocessing, e.g. average number of active users per month

26 Step 2 Simple Statistics Many different reporting nodes with different statistics. Reporting extension to generate PDF, DOC,

27 Step 2 Simple Statistics

28 Step 2 Simple Statistics Number of active users per year An active user is an user with at least one comment or one post in that year.

29 Step 2 Simple Statistics Number of posts per year Numbers are just posts (new discussion threads), not comments

30 Step 2 Simple Statistics Number of posts per month and year Big increase early Coincidentally, Simon Richards (richards99) joined

31 Step 2 Simple Statistics Who comments/answers on posts?

32 Step 2 Simple Statistics Response time

33 Step 2 Simple Statistics Number of comments per post

34 Step 1: Get Data Step 2: Compute Statistics Step 3: Classify Posts Step 4: Content & Users Classify Posts and Detect Topic Shifts

35 Step 3 Classify Posts Use text mining to classify forum post into categories such as io, manipulation, mining, No training set available (mis-)use KNIME node description See evolution of discussion topics over the years

36 Step 3 Classify Posts Want to classify forum post (only first post, no comments)

37 Step 3 Classify Posts using KNIME node description text as labeled training set

38 Step 3 Classify Posts Reads node descriptions from xml dumps (generated with KNIME command line tool) Uses forum data input file and prepares with text mining tools

39 Step 3 Classify Posts Unzips an archive with all xml files into temp location

40 Step 3 Classify Posts XML files read with loop and preprocessed (header and footer removed)

41 Step 3 Classify Posts Description is converted into KNIME text document, from which (stemmed) terms are extracted

42 Step 3 Classify Posts

43 Step 3 Classify Posts Training data extracted. Learning attributes are keyword occurrences; target is document category

44 Step 3 Classify Posts Verify model by splitting data into train/test. Using random forest classifier to address high dimensionality of small (and sparse) data set Training data extracted. Learning attributes are keyword occurrences; target is document category

45 Step 3 Classify Posts continuing with main input branch (Input table from crawler workflow)

46 Step 3 Classify Posts Preprocessing similar to before, extracting date, author, title,

47 Step 3 Classify Posts Extracting attribute table using the keywords from the node description (training) data.

48 Step 3 Classify Posts Remainder of the workflow ranks the prediction and prepares for the report.

49 Step 3 Classify Posts Hot topics have always been manipulation and mining tasks that KNIME is very good at. Note also increase of flowcontrol over the years and low r traffic (separate forum category, not part of this data set)

50 Step 1: Get Data Step 2: Simple Statistics Step 3: Classify Posts Step 4: Content & Users Identify Content and Users

51 Sept 4 Content & Users Look at individual categories (KNIME General, Developer, Reporting, ) Learn what is discussed See who is contributing

52 Sept 4 Content & Users Input are all discussions in one forum category

53 Sept 4 Content & Users Output is a multi page report with tag cloud and user connection graph Combines KNIME s text and network mining extensions

54 Sept 4 Content & Users

55 Sept 4 Content & Users Input table from crawler workflow

56 Sept 4 Content & Users Main loop over all ~20 categories

57 Sept 4 Content & Users General statistics per category User network analysis Text analytics

58 Sept 4 Content & Users Text analysis: Forum posts converted to documents and tagged (persons, node names, node categories)

59 Sept 4 Content & Users Terms fed into tag cloud, colors represent persons ( kilian ), nodes ( bow creator ), node categories ( xml ),

60 Sept 4 Content & Users Network analysis: User connections (content ignored)

61 Sept 4 Content & Users Network analysis: Ignore topics, only look at user relationships. Network nodes represent users, connections represent (directed) relationships between users

62 Sept 4 Content & Users Network analysis: User graph, visualized with standard KNIME graph viewer

63 Sept 4 Content & Users Data collected and send to reporting extension

64 Sept 4 Content & Users Multi page pdf output for different forum categories

65 Sept 4 Content & Users Text Mining forum category

66 Sept 4 Content & Users RDKit (community chemistry extension)

67 Sept 4 Content & Users KNIME Users not dominated by any particular users

68 KNIME Forum Analysis Learn something about the KNIME forum: Steps: 1. Get data into KNIME 2. Extract simple statistics (how many posts, response time, response length) 3. Classify posts and detect topic shifts 4. Identify content and users

69 Reviewing all workflows All workflows rely on the same input data Requires re-run of Crawler workflow and updating parameters in analysis flow

70 What do all the flows have in common?

71 They all require the Crawler data

72 Reviewing all workflows All workflows rely on the same input data Requires re-run of Crawler workflow and updating parameters in analysis flow Better: Use meta node and share it between all instances

73 Improve - Create and Share Meta Node

74 Improve - Create and Share Meta Node

75 Improve - Create and Share Meta Node

76 Improve - Create and Share Meta Node

77 Now use it in all the analysis flows

78 Improve - Meta Node instead of File Reader

79 Improve - Meta Node instead of File Reader

80 Improve - Meta Node instead of File Reader

81 Nice but now all workflows fetch the data each time they execute! Let s add a cache option.

82 Improve - Add Caching Option Quickform Node defining a switch: -Get data from web or -use cached file (lives on server)

83 Improve - Add Caching Option

84 Improve - Add Caching Option

85 Improve - Add Caching Option

86 Summary KNIME User Community is healthy and growing Community developed extensions a vital part of the KNIME experience White paper available for download on the KNIME web page Workflows available for download on the Public Example Server 050_Applications->05007_ForumAnalysis