Web Recommendation Using Classification & MapReduce Framework

Transcription

1 Web Recommendation Using Classification & MapReduce Framework MBICT, New Vallabh Vidyanagar, Anand, Gujarat, India G.H.Patel College of Engineering & Technology, Vallabh Vidyanagar, Anand, Gujarat, India A B S T R A C T Web usage mining (WUM) is key process to extract data from web. WUM provides usage pattern & user navigation for website. Using WUM organization easily extracts frequent access patterns & also improves the web design. Web recommendation is further aspect to improve usage patterns & give specific pages to unique users. For web recommendation can be done using classification& clustering techniques. This paper is implementation of data cleaning process of web log data using WUM Process. Also this paper is about comparison of propose model with hidden markov model classification technique. Keywords: Web Log File; Web Usage Mining (WUM); Data Pre-Processing; Hadoop Clustering; Web Recommendation; Markov Model. I. Introduction Web Usage Mining: Concept-Web Usage Mining provides usage patterns of user. Web usage mining process is best way to find user patterns for research or analysis. The main difference between WUM and other two techniques is, web content mining provides content for relevant information & web structure mining provides data s structure &classification of data while WUM provides data with specific navigation patterns using web log file. WUM is useful in recommendation algorithm as well in prediction of web pages. II. WUM Process Main three steps: 1) Pre-Processing: Pre-processing the data as per requirement of analysis. Second task is extraction of fields from web log file. 2) Pattern Discovery: Using Pre-processed data by applying different mining algorithms, it is easy to get patterns. 3) Pattern Analysis: Discovery of patterns will give so many patterns, also might be possible that found so many redundant patterns. So pattern analysis will analyze the patterns & give specific patterns of data analysis. Data Pre-processing is an important activity for the complete web usage mining processes and vital in deciding the quality of patterns. [3] Pattern Discovery will give different usage patterns & Pattern Analysis will give specific pattern of user. Data pre-processing means to pre-process the original data for better & well formatted data set. In this process the irrelevant data will be discarded so that output data will be small in size as compared to original data. For example if there is a web log file & there are so many fields, so system has to pre-process that log file. That pre-processed file will be input for next step. Format of web log file: Here explained the sample web log entry of web log file of G. H. Patel College of Engineering & Technology Example: :31: GET /default.aspx Wget/1.12+ (linux-gnu) Web log file has around 14 types of fields; each has different attribute & value. But as pre-processed output, need only some fields. 191 Figure 1: Data Pre-processing

2 Field Name Field Description Example Date The date that the activity occurred Time The time that the activity occurred 1:31:16 s-ip: The IP address of the server cs-method The action the client was trying to perform (for GET example, a GET method) cs-uri-stem The resource accessed; for example, Default.htm /default.aspx cs-uri-query The query, if any, the client was trying to perform - s-port The port number the client is connected to 8 cs-username The name of the authenticated user who accessed - your server. This does not include anonymous users, who are represented by a hyphen (-) c-ip The IP address of the client cs(user-agent) The browser used on the client Wget/1.12+(linux-gnu) sc-status The status of the action, in HTTP or FTP terms 2 Sc-substatus is the sub status e.g. for a HTTP status it would be the 19 part Sc-win32-status The status of the action, in terms used by Microsoft Windows timetaken The duration of time, in milliseconds, that the action consumed Table 1: Web Log File Fields Data pre-processing has following steps: A. Data Cleaning Data will be cleaned based on uri pattern. Data cleaning provides data as per user patterns. This process will discard subpages & download pages from web log file, also delete the additional entries. Cleaning process is important to compress the data & also provide to reduce data redundancy. B. User Identification Based on user ip system can easily found the most visited users. User identification provides specific & unique users. A log file contains more than thousands entries, so system has to check each and every user s entries. For that system has to get different users & their IP address. User s identification is, to identify who access web site and which pages are accessed [14]. C. User Session Identification Based on time taken field easily count the number of session for single user. Also system can find particular session to get data from every page. D. Path Completion The aim of the path completion is to acquire complete user access path by filling up the missing page references [3]. It means how the user goes to next link or some other link. So if there is two possibilities then best path will be based on requirement of user. That means path completion means how user ended his/her actions on that web site. III. Classification: Hidden Markov Model Recommendation systems are tools that suggest related pages or resources to web surfers. They may be simple or sophisticated tools to assist clients maneuver through a web site. The concept of Markov models is used for modeling user Web navigation data, as they are compact, simple to understand, expressive, and based on a well-established theory. In a Web site with a large number of Web pages, users often have navigational questions, such as, where am I? Where have I been? And where can I go? In this paper, by checking the Web user s navigation in a Web site as a Hidden Markov chain, we can build a Markov model for web page recommendation based on past users visit behavior recorded in the Web log file. We assume that the pages to be visited by a user in the 192

3 future are determined by his/her current position and/or visiting history in the Web site. We construct a link graph from the Web log file, which consists of nodes representing Web pages, links representing hyperlinks, and weights on the links representing the numbers of traversals on the hyperlinks. By viewing the weights on the links as past users implicit feedback of their preferences in the hyperlinks, we can use the link graph to calculate a transition probability matrix containing one-step transition probabilities in the Markov model. Let, P= {p 1,p 2,p 3, p n ) be a set of pages in a web site. S: user session including a sequence of pages visited by the user. Then probability (p i S) is the probability that a user visit page p i next. Markov chain model can be defined by the tuple - <S, A, Lamda> S is corresponds to the state space; A is a N x N matrix representing transition probabilities from one state to another as a directed graph. [Transition Matrix] Page views, P = {p 1, p 2,, p n } m user transactions, T = {t 1,t 2,,t m } Lamda is the initial probability distribution of the states in S. PageId S Freq Table 2: Page Frequency Figure 2: Web Navigation T /11 9/11 2 7/7 3 7/16 7/ Table 3: Navigation Table Try and recommend the next link the user will choose to the maximum probability link was followed based on transition matrix table. 193

4 IV. Proposed Work: Recommendation Using MapReduce Framework Figure 3: Proposed System This system is for web recommendation using MapReduce Framework. The initial stage is data pre-processing in WUM. The pre-processed data is input for apriori algorithm in MapReduce Framework. The Output will specific user with its frequently visited pages as web page recommendation. Apriori in MapReduce Framework Single Node Working Apriori in single node is depending on mapper & reducer. Mapper will count the support count and reducer will provide the frequent item sets.. Figure 4: Apriori in MapReduce Framework [15] V. Experiment Result The Markov Model Classification is done on windows system using Netbeans & Wampserver. While Apriori is done on linux system using hadoop distributed framework. Figure 5: Pre- Processed Data 194

5 Figure 6: Markov Model Figure 7: Markov Recommendation Result when frequency >3 VI. Comparison & Evaluation After Pre-processing of log file, using Markov model & aprioi in MapReduce Framework give optimized analysis based on output of both. In distributed approach it will be easy to get web recommendations. The comparison is based on main three parameters: Accuracy, Precision & Coverage. Accuracy Measure is defined as degree to which the recommended list produced by agent engine produces matches with the actual navigation. [9] Coverage measure is defined as the ability of agent engine to produce all page views that are most likely visited by the user. [9] Precision measures the number of correct relevant recommendation to the total recommendations. [11] Here Comparison result is based on page hits & the web recommendation result. Dataset in terms of number of page hits or page visiting entries, recommendation in terms of grades 3,5,7,9 in terms of respectively very poor, poor, good & excellent. X-axis: Recommendation Result Grades Y-axis: Page Hits Accuracy Precision Coverage Figure 8: Web Recommendations in Markov Model Accuracy Precision Figure 9: Web Recommendations in MapReduce Framework Using Apriori 195

6 The comparison shows that at markov model will provide recommendation based on hits (frequency of page) while apriori in mapreduce provide recommendation based on specific user & page. Another comparison is that normal approach gives more accuracy & precision at higher level, while in distributed approach, it starts with 1 page hits for very poor recommendation result. So this mean if data set is larger than it will be efficient to use distributed approach instead of normal approach. [13] VII. Conclusion & Future Work Based on markov model implementation output will be not accurate means precision is low, while distributed approach is more efficient than normal classification approach. Hadoop distributed mapreduce approach is better for large datasets & it provides better accuracy as well as faster access, so it means efficiency will be more compare to normal approach to generate web recommendations. Future work can be extending by multimode apriori in mapreduce distributed approach. Distributed approach gives more rules in terms of prediction, so coverage will be more as large datasets executed in distributed clusters. So it will enhance the accuracy for getting the better recommendation of web pages. VIII. References [1] Wichian Premchaiswadi, Walisa Romsaiyud, Extracting Weblog of Siam University for Learning User Behavior on MapReduce, Intelligent and Advanced Systems (ICIAS), 4th International Conference on June 212, (Volume: 1) Pages: [2] Kathleen Hodgkinson, Abdelmounaam Rezgui, SAFAL: A MapReduce Spatio-temporal Analyzer for UNAVCO FTP Logs, Computational Science and Engineering (CSE), IEEE 16 th Conference on December 213, Pages: [3] K. Sudheer Reddy M. Kantha Reddy V. Sitaramulu, An Effective Data Pre-processing Method for Web Usage Mining, Information Communication and Embedded Systems (ICICES), International Conference on February 213, Pages: 7-1 [4] V.V.R. Maheswara Rao, Dr. V. ValliKumari, An Efficient Hybrid Predictive Model to Analyze the Visiting Characteristics of Web User using Web Usage Mining, Advances in Recent Technologies in Communication and Computing (ARTCom), International Conference on October-21, Pages: [5] Theint Theint Aye, Web Log Cleaning for Mining of Web Usage Patterns, Computer Research and Development (ICCRD), 3 rd International Conference on March 211, (Volume: 2) Pages: [6] Nayana Mariya Varghese, Jomina John, Cluster Optimization for Enhanced Web Usage Mining using Fuzzy Logic, Information and Communication Technologies (WICT), World Congress on October-November 212, Pages: [7] Husna Sarirah Husin, News Recommendation Based on Web Usage and Web Content Mining, Data Engineering Workshops (ICDEW), IEEE 29 th International Conference on April 213, Pages: [8] Chunzhi Wang, Zhou zheng, Zhuang Yang, The Research of Recommendation System Based on Hadoop Cloud Platform, Computer Science & Education (ICCSE), 9th International Conference on August 214, Pages: [9] Ravi Bhushan, Dr. Rajender Nath, Automatic Recommendation of Web Pages for Online Users Using Web Usage Mining, Computing Sciences (ICCS), International Conference on September 212, Pages: [1] Xia Min-jie, Zhang Jin-ge, Research on Personalized Recommendation System for e-commerce based on Web Log Mining and User Browsing Behaviors, Computer Application and System Modelling (ICCASM), International Conference on October 21, (Volume: 12) Pages: [11] Yahya AlMurtadha, Md. Nasir Bin Sulaiman, Norwati Mustapha and Nur Izura Udzir, IPACT: Improved Web Page Recommendation System Using Profile Aggregation Based On Clustering of Transactions, American Journal of Applied Sciences 8 (3): , 211 [12] Zhongyun Ying, Zhurong Zhou*, Fengjiao Han and Guofeng Zhu, Research on Personalized Web Page Recommendation Algorithm Based on User Context and Collaborative Filtering, Software Engineering and Service Science (ICSESS), 4th IEEE International Conference on May 213, Pages: [13]Tejas M. Modi, Bhargesh Patel, Priyanka Panchal, A Survey: Web Recommendation Using Different Approaches of Web Mining Techniques, International Journal of Advance Foundation and Research in Computer (IJAFRC)Volume 2, Special Issue (NCRTIT 215), January 215. ISSN [14]Ms. Priyanka S. Panchal, Prof. UrmiAgravat, Finding Frequent Users Access Patterns from web server log files using Data Preprocessing, International Journal of Data Warehousing & Mining, ISSN: June-213 Volume 3, Issue 2 196

7 [15] Shravanth Oruganti, Qin Ding, Nasseh Tabrizi, Exploring HADOOP as a Platform for Distributed Associative Rule Mining, The Fifth International Conference on Future Computational Technologies and Applications 213, Pages: [16] HDFS: [17]Map/Reduce: 197