Ching-Yung Lin, Ph.D. Adjunct Professor, Dept. of Electrical Engineering and Computer Science. September 21, 2017

Transcription

1 E6893 Big Data Analytics Lecture 3: Big Data Storage and Analytics Ching-Yung Lin, Ph.D. Adjunct Professor, Dept. of Electrical Engineering and Computer Science September 21, E6893 Big Data Analytics Lecture 3: Storage and Processing

2 Demos Hive, HBase, and Spark by TA 2 E6893 Big Data Analytics Lecture 3: Storage and Processing

3 Analytics 1 Recommendation 3 E6893 Big Data Analytics Lecture 3: Storage and Processing

4 Key Components of Mahout 4 E6893 Big Data Analytics Lecture 3: Big Data Analytics

5 Mahout reference book 5

6 Setting Up Mahout Step 1: Java JVM and IDEs (e.g., Eclipse) Step 2: Maven Step 3: Mahout Eclipse 6

7 Motivation Information Overload News Books, Journals, Research papers Movies Books Electronics! Provide users with the information they need at the right time 7

8 Recommender Systems: General Idea User profile (info about the user) comparison Set of items Recommendation e.g. news articles, books, music, movies, products, Sample applications p E-commerce Product recommender - Amazon p Enterprise Activity Intelligence Domain expert finder, p Digital Libraries Pages/documents/books recommendation p Personal Assistance Museum guidance, Algorithms p Content-based Filtering p Collaborative Filtering p Hybrid Filtering (combination of the above two) 8

9 Content-based Filtering (CBF) Recommending items based on the content and properties Preference: and Matching Item Stream sports Tech World World Health Health Recommending 9

10 Collaborative Filtering (CF) Leveraging opinions of like-minded users Given Recommend People with similar tastes 10

11 Exploiting Dynamic Patterns for Recommender Systems Dynamic nature from both items and users Items expire over time with types of Short-term Long-term Users intentions of Updating breaking information Looking for long-term items Users interests evolve over time User adoption patterns Some users are earlier adopters 11

12 Recommender Inputs Solid lines: positively related Dashed lines: negatively related Input Data: User, Item, Rating 12

13 User-based Recommendation Scenario I gettofail.com 13

14 User-based Recommendation Scenario II 14

15 User-based Recommendation Scenario III 15

16 User-based Recommendation Algorithms 16

17 Example Recommender Code via Mahout 17

18 Process and output of the example Recommendation for Person 1: Item 104 > Item 106 Item 107 is not favored 18

19 User Similarity Measurements Pearson Correlation Similarity Euclidean Distance Similarity Cosine Measure Similarity Spearman Correlation Similarity Tanimoto Coefficient Similarity (Jaccard coefficient) Log-Likelihood Similarity 19

20 Pearson Correlation Similarity Data: missing data 20

21 On Pearson Similarity Three problems with the Pearson Similarity: 1. Not take into account of the number of items in which two users preferences overlap. (e.g., 2 overlap items ==> 1, more items may not be better.) 2. If two users overlap on only one item, no correlation can be computed. 3. The correlation is undefined if either series of preference values are identical. Adding Weighting.WEIGHTED as 2nd parameter of the constructor can cause the resulting correlation to be pushed towards 1.0, or -1.0, depending on how many points are used. 21

22 Euclidean Distance Similarity Similarity = 1 / ( 1 + d ) 22

23 Cosine Similarity Cosine similarity and Pearson similarity get the same results if data are normalized (mean == 0). 23

24 Spearman Correlation Similarity Example for ties Pearson value on the relative ranks 24

25 Caching User Similarity Spearman Correlation Similarity is time consuming. Need to use Caching ==> remember s user-user similarity which was previously computed. 25

26 Tanimoto (Jaccard) Coefficient Similarity Discard preference values 26 Tanimoto similarity is the same as Jaccard similarity. But, Tanimoto distance is not the same as Jaccard distance.

27 Log-Likelihood Similarity Asses how unlikely it is that the overlap between the two users is just due to chance. 27

28 Performance measurements Using GroupLens data ( 10 million rating MovieLens dataset. Spearnman: 0.8 Tanimoto: 0.82 Log-Likelihood: 0.73 Euclidean: 0.75 Pearson (weighted): 0.77 Pearson:

29 Performance measurements 10 nearest neighbors: nearest neighbors: nearest neighbors: % of training; 5% of testing 29

30 Selecting the number of neighbors Based on number of neighbors Based on a fixed threshold, e.g., 0.7 or

31 Item-based recommendation 31

32 Item-based recommendation algorithm 32

33 Code and Performance of Item-Based Recommendation performance 33

34 Slope-One Recommender 34

35 Slope-One Algorithm Difference values from the example Slope-One got a result of near 0.65 on the GroupLens data 35

36 Other recommenders SVD recommender number of features number of training step lambda: factor for regularization SVD method got 0.69 on the GroupLens data 36

37 Linear Interpolation Item-based recommender SVD method got 0.76 on the GroupLens data 37

38 Cluster-based Recommendation 38

39 Other Recommenders not in Mahout Groups (SDM 06) A 3 rd party Knowledge Repository: 30K users and 20K documents. Study the most active 697 users who have at least 20 download in a year. Results: beyond Collaborative Filtering: (1) Collaborative + Content Filtering (53% improvement); (2) CBDR: Collaborative + Content Filtering + Graph Community Analytics (259% accuracy improvement over collaborative filtering) CB DR CB DR CB DR 39

40 In E-Commerce: Rogers Diffusion of Innovations Theory Innovators Early adopters Early majority Late majority Laggards Users adoption patterns: Some users tend to adopt innovations earlier than others! Information virtually flows from early adopters to late adopters 40

41 Recommendation Driven by Information Flow Innovator adopt Early adopter Innovator adopt?? Early adopter Late majority Early majority Late majority? Early majority Laggard Laggard adopt People with similar tastes People with similar tastes Influence is not symmetric! 41

42 Scheme Overview p Leverage the uneven influence Dataset EABIF Information Propagation Model Application: Personalized Recommendation EABIF: Early Adoption based Information Flow Network 42

43 EABIF (1) -- Markov Chain p p A Markov chain has two components A network structure where each node is called a state A transition probability of traversing a link given that the chain is in a state (P: transition probability matrix) A stationary distribution is a probability distribution q such that q = qp how likely you will stay at one node i P ij j p Application -- PageRank [Brin and Page 98] Assumption: A link from page A to page B is a recommendation of page B by the author of A A B Quality of a page is related to p Number of pages linking to it C p The quality of pages linking to it Assume the web is a Markov chain p PageRank = stationary distribution of this Markov chain 43

44 EABIF (2) p Early Adoption Matrix (EAB) Count how many items one user accesses earlier than the other pairwise comparison p Markov Chain Model Normalize EAB to a transition matrix F of a Markov chain Adjustment F to guarantee the existence of stationary distribution of the Markov chain p p Make the matrix stochastic F ij = 1 Make the Markov chain irreducible j F 0, 1 i, j N ij 44

45 Information Propagation Models 1. Summation of various propagation steps ( ) ( ( ) ( )) 2 m! F = F + F + + F if m m v r 1 u 2. Direct summation F if ( d ) ( ( 1 )) N F I F = I F r 2 r 3 N: number of the nodes 3. Exponential weighted summation F # 1 1 $ if (exp) = & F 2! F N 1! F ' ( ) ( ) ( ) 2 ( ) ( N 1 β β! β ) )(!' exp( β ) 45

46 Topic-Sensitive Early Adoption Based Information Flow (TEABIF) Network p Adoption is typically category specific An early adopter of fashion may not be an early adopter of technology Application: Personalized Recommendation LDA TOPIC 1 TOPIC 2 TOPIC 1 TOPIC 2 TOPIC 3 TOPIC 3 46

47 Experimental Setup ER dataset 2004 Apr. to 2005 Apr. as training data 2005 May to 2005 Jul. as test data 1033 users, 586 documents Process Construct information flow network based on the training data Trigger earliest users to start the process Predict who will be also interested in these documents Evaluation Precision & Recall 47

48 Experimental Results --Recommendation Quality Precision Precision Precision Comparison (Number of T riggered Users = 1, Propagation Steps = 1) No. of retrieved users Precision Comparison (Number of T riggered Users = 2, Propagation Steps = 1) CF EABIF TEABIF CF EABIF TEABIF Recall Recall Recall Comparison (Number of T riggered Users = 1, Propagation Steps = 1) CF EABIF TEABIF No. of retrieved users Recall Comparison (Number of T riggered Users = 2, Propagation Steps = 1) CF EABIF TEABIF No. of retrieved users No. of retrieved users p Comparing to Collaborative Filtering (CF), precision: EABIF is 91.0% better, TEABIF is 108.5% better Recall: EABIF is 87.1% better, TEABIF is 112.8% better 48

49 Experimental Results -- Propagation Performance Ratio (x100%) Precision Improvement Comparison (Number of triggered users = 1, Baseline: CF) m = 1 m = 2 m = 3 m = 4 m = 5 sum exp(β= 1) exp(β= 1.5) exp(β= 2) exp(β= 3) exp(β= 4) exp(β= 5) EABIF TEABIF exp(β= 8) exp(β= 16) Ratio (x100%) Recall Improvement Comparison (Number of triggered users = 1, Baseline: CF) EABIF TEABIF m = 1 m = 2 m = 3 m = 4 m = 5 sum exp(β= 1) exp(β= 1.5) exp(β= 2) exp(β= 3) exp(β= 4) exp(β= 5) exp(β= 8) exp(β= 16) Ratio (x100%) Precision Improvement Comparison (Number of triggered users = 2, Baseline: CF) m = 1 m = 2 m = 3 m = 4 m = 5 sum exp(β= 1) exp(β= 1.5) exp(β= 2) exp(β= 3) exp(β= 4) exp(β= 5) EABIF TEABIF exp(β= 8) exp(β= 16) Ratio (x100%) Recall Improvement Comparison (Number of triggered users = 2, Baseline: CF) m = 1 m = 2 m = 3 m = 4 m = 5 sum exp(β= 1) exp(β= 1.5) exp(β= 2) exp(β= 3) exp(β= 4) exp(β= 5) EABIF TEABIF exp(β= 8) exp(β= 16) p TEABIF with exponential weighted summation ( β = 3 ) p achieves the best performance: p improves 108.5% on precision and 116.9% on recall comparing to CF 49

50 Summary Exploit dynamic patterns including Leverage dynamic patterns from both documents and users perspective Analyzing documents accessing types Predicting documents expiration date Detecting users intentions Identifying users interests evolving over time CTC model Ranking the documents adaptively Time-sensitive Adaboost Utilize users adoption patterns Information virtually flows from early adopters to late adopters Experimental results demonstrate Dynamic factors are important for recommendations 50

51 Selected Publications X. Song, C.-Y. Lin, B. L. Tseng, and M.-T. Sun, Modeling Evolutionary Behaviors for Community-based Dynamic Recommendation, SIAM Conf. on Data Mining, Bethesda, MD, Apr X. Song, C.-Y. Lin, B. L. Tseng, and M.-T. Sun, Personalized Recommendation Driven by Information Flow, ACM SIGIR, Aug X. Song, C.-Y. Lin, B. L. Tseng and M.-T. Sun, Modeling and Predicting Personal Information Dissemination Behavior, ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Aug X. Song, B. L. Tseng, C.-Y. Lin, and M.-T. Sun, "ExpertiseNet: Relational and Evolutionary Expert Modeling," International Conference on User Modeling, Edinburgh, UK, Jul ,

52 Distributed Item-based Recommender 52

53 Distributed recommender get co-occurrence matrix Data: 53

54 Multiply the co-occurrence matrix with user preference The highest is 103 (101, 104, 105, 107 have been purchased by user 3) 54

55 Translating to MapReduce: generating user vectors 55

56 Translating to MapReduce: calculating co-occurrence 56

57 Translating to MapReduce: matrix multiplication 57

58 Translating to MapReduce: partial products 58

59 Translating to MapReduce: partial product II 59

60 Running Recommender on MapReduce and HDFS 60

61 Questions? 61 E6893 Big Data Analytics Lecture 3: Storage and Processing

62 E6893 Big Data Analytics: Appendix references 62

63 Part I: Pig installation and Demo Pig is a platform for analyzing large data sets that consists of a highlevel language for expressing data analysis programs, coupled with infrastructure for evaluating these programs. 63

64 1. Installation of Pig: setup.html Download pig and run following sentence: export PATH=/<my-path-to-pig>/pign.n.n/bin:$PATH 64

65 65

66 2. Running pig in local mode: pig -x local movies = LOAD '/Users/Rich/ Documents/Courses/Fall2014/ BigData/Pig/movies_data.csv' USING PigStorage(',') as (id,name,year,rating,duration); 66 DUMP movies

67 Filter: List the movies that were released between 1950 and 1960 movies_1950_1960 = FILTER movies BY (float)year>1949 and (float)year<1961; store movies_1950_1960 into '/Users/Rich/ Desktop/Demo/movies_1950_1960'; 67

68 Foreach Generate: List movie names and their duration (in minutes) movies_name_duration = foreach movies generate name, (float)duration/3600; store movies_name_duration into '/Users/Rich/ Desktop/Demo/movies_name_duration'; 68

69 Order: List all movies in descending order of year movies_year_sort =order movies by year desc; store movies_year_sort into '/Users/Rich/ Desktop/Demo/movies_year_sort'; 69

70 3. Running pig with HDFS: pig Run HDFS first: ssh localhost cd /usr/local/cellar/hadoop/2.5.0 sbin/start-dfs.sh 70

71 3. Upload file to HDFS: Make a directory: bin/hdfs dfs -mkdir /PigSource Upload a file: bin/hdfs dfs -put /Users/Rich/Documents/Courses/ Spring2014/CloudComputing/HW/MINI5/ movies_data.csv /PigSource 71

72 4. Run pig in grunt with HDFS: pig movies = LOAD '/PigSource/ movies_data.csv' USING PigStorage(',') as (id,name,year,rating,duration); DUMP movies 72

73 5. Run.pig file with HDFS: pig pig /Users/Rich/Documents/Courses/ Fall2014/BigData/Pig/run1.pig 73

74 Part II: Hbase installation and Demo HBase is an open source, non-relational, distributed database modeled after Google's Big Table and written in Java. It runs on top of HDFS, and can serve as input and output for MapReduce jobs run in Hadoop. Access through Java API and Pig. 74

75 Hbase Configuration: Download and configure Hbase by following the instruction online. To run Hbase, under your Hbase path: $bin/start-hbase.sh Enter shell $bin/hbase shell Also visit localhost:60010 to check Hbase webui 75

76 Hbase Command: Create: hbase>create table, cf hbase>put table', r1', cf:c1', 'value1 hbase>scan table Also we can do count, get, delete and drop. etc much like other DB systems. 76

77 Part III: Hive installation and Demo 77

78 Simple demo for Hive Install steps: Mac: you can first install brew $ brew install Hive Linux: cd ~/Downloads wget hive /apache-hive bin.tar.gz cp apache-hive bin.tar.gz ~ cd ~ tar zxvf apache-hive bin.tar.gz mv apache-hive bin hive cd hive cd bin./hive 78

79 Simple demo for Hive Step 1. start Hadoop, create a file under /user/ yourname hadoop fs -mkdir test Step 2. put your dataset into hdfs hadoop fs -put (your dataset path) test hadoop fs -ls test 79

80 Simple demo for Hive Step 3. create the table (SQL) in Hive shell hive Step 4. store the data into table load data inpath '/user/yourname/test' into table client; You can see your data has already been put into the table 80

81 Simple demo for Hive 1. select * from client where name='henry'; 2. select avg(age) from client where gender='male'; 81

82 Simple demo for Hive Use JDBC(JAVA) to access data in Hive Server Other choice: Python, PHP etc. 82

83 Simple demo for Hive Open the Hive (Starting Hive Thrift Server) serverhive --service hiveserver -p URL: jdbc:hive://localhost:10000/default Result in the console 83

84 Simple demo for Hive Example. Jetty Server with Jersey, restful web service Create simple API (Http request, XML/JSON format output) 84

85 Simple demo for Hive Example. Jetty Server with Jersey, restful web service 85