A Survey on Preprocessing Techniques in Web Usage Mining

Transcription

1 COMP 630H A Survey on Preprocessing Techniques in Web Usage Mining Ke Yiping Student ID: keyiping@ust.hk Computer Science Department The Hong Kong University of Science and Technology Dec 2003

2 Abstract The World Wide Web (WWW) continues to grow at an overwhelming rate in both the sheer volume of traffic and the size and complexity of Web sites. Therefore, it becomes more and more necessary, but difficult to get useful information from Web data, in order to understand and better serve the needs of Web-based applications. As a result, the Web usage mining has become a hot research topic, which combines two of the prominent research areas comprising the data mining and the World Wide Web. Web usage mining consists of three phases, namely preprocessing, pattern discovery, and pattern analysis. For the survey, I focus on the first phase data preprocessing, which is essential and must be performed prior to applying data mining algorithms to the data sources. An overview of data preprocessing techniques aiming at identifying unique users, user sessions and transactions is presented in this survey. Key words: Web Usage Mining, Web Log Mining, Data Preprocessing. 1. Introduction and Background Web Usage Mining is that part of Web Mining which deals with the extraction of useful knowledge from the secondary data derived from the interactions of the users while interacting with the Web. The Web usage data includes the data from Web server access logs, proxy server logs, browser logs, user profiles, registration data, user sessions or transactions, cookies, user queries, bookmark data, mouse clicks and scrolls, and any other data as the results of interactions. The scope of Web usage mining is local, which means that the scope of Web usage mining spans an individual Web site. Usage Mining tools [2,4,5,6] discover and predict user behavior, in order to help the designer to improve the web site, to attract visitors, or to give regular users a personalized and adaptive service. Table 1 gives the explanations of the Web Usage Mining in terms of view of data, main data, representation, method, mining steps and applications. As is true for typical data mining applications, the issue of data preprocessing plays a fundamental role in the whole mining process. As pointed out by [22], this phase is critical for the success of pattern discovery: errors committed in this phase may make the data useless for further analysis. In Web usage analysis, data preprocessing includes tasks of repairing erroneous data and treating missing values [1, 23]. The preprocessing of web usage data, which is mainly web logs, is usually complex and time consuming. It can take up to 80% of the time spend analyzing the data. Tasks for performing preprocessing of Web Usage Mining in [1] involve data - 2 -

3 cleaning, user identification, session identification, path completion, session reconstruction, transaction identification and formatting, etc. And some other tasks, such as conceptual hierarchies construction and classification of URLs, as stated in [2, 3], may be invoked in certain cases. The aim of preprocessing is to offer structural, reliable and integrated data source to next one, pattern discovery. The typical problem is distinguishing among unique users, user sessions, transactions, etc. View of Data Main Data Representation Method Mining Steps Applications Table 1: Web Usage Mining - Interactivity - Server logs - Browser logs - Relational table - Graph - Machine Learning - Statistical - (Modified) association rules - Data preprocessing - Pattern discovery - Pattern analysis - Site construction, adaptation, and management - Marketing - User modeling The rest of this survey will present the tasks and techniques of Web usage mining preprocessing in details at section 2. Section 3 describes some problems in preprocessing. And then come to the conclusion in section Preprocessing Tasks and Techniques 2.1 Data Cleaning This step is to remove all the data useless for data analyzing and mining e.g.: requests for graphical page content (e.g., jpg and gif images); requests for any other file which might be included into a web page; or even navigation sessions performed by robots and web spiders. The quality of the final results strongly depends on cleaning process. Appropriate cleaning of the data set has profound effects on the performance of web usage mining. The discovered associations or reported statistics are only useful if the data represented in the server log gives an accurate picture of the user accesses to the Web site. The procedures of general data cleaning are as follows: Firstly, it should remove entries that have status of error or failure. It s to - 3 -

4 remove the noisy data from the data set. To accomplish it is quite easy. Secondly, some access records generated by automatic search engine agent should be identified and removed from the access log. Primarily, it should identify log entries created by so-called crawlers or spiders that are used widely in Web search engine tools. Such data offer nothing to the analyzing of user navigation behaviors. Many crawlers voluntarily declare themselves in agent field of access log, so a simple string match during the data cleaning phase can strip off a significant amount of agent traffic. In addition, to exclude these accesses, [2] employs several heuristic methods that are based on indicators of non-human behavior. These indicators are: (i) The repeated request for the same URL from the same host. (ii) A time interval between requests too short to apprehend the contents of a page. (iii) A series of requests from one host all of those referred URLs are empty. The referred URL of a request is empty if the URL was typed in, requested using a bookmark, or requested using a script. The last task of data cleaning, which is also disputable, is whether it needs to remove log entries covering image, audio and video files. In some system, such as WebMiner, all such entries are removed, while in WebLogMiner, such data are held. Generally, WUM tools applied to user navigation patterns intend to remove all such entries, because they do minor influence on the analysis of users navigation behaviors even such entries are generated by clicking of users. Typical web log cleaning methodologies mainly aim at removing image and picture files with extensions GIF and JPG (if analysis does not involve image examining). The analysis concerns the investigation of media/multimedia files but there are masses of other irrelevant files, which stay untouched during all the analysis process or even have a negative impact on the analysis. These can be internal web administrator actions, special purpose files, etc. The complexity of many web sites can influence the data cleaning process and impact the final results as well. In contrast to the typical data cleaning techniques, [9] introduces a novel advanced cleaning technique. At the case stated in [9], the frame pages do not have characteristic file extension (e.g., like pictures extensions jpg, gif). After applying the typical data cleaning techniques, the results were not meaningful to the client. Therefore, in the paper, the author proposes technique consist of two frame recognition stages: retrieving HTML code from web server and the filtering. Improved filtering removes pages with no links from other pages. Using this methodology, cleaning covers not just frame pages but all irrelevant pages and leaves only essential pages for further analysis and experimental studies. While requests for graphical contents and files are easy to eliminate, robots and web spiders navigation patterns must be explicitly identified. This is usually done by referring - 4 -

5 to the remote hostname, by referring to the user agent, or by checking the access to the robots.txt file. However, some robots actually send a false user agent in HTTP request. In these cases, [20] proposes an approach to separate robot sessions from actual users sessions using standard classification techniques. Highly accurate models to achieve this goal can be obtained after three requests using a small set of access features computed from the Web server logs. A heuristic based on navigational behavior can also be used to achieve this goal (see [24]). 2.2 User Identification This task is greatly complicated by the existence of local caches, corporate firewalls, and proxy servers. The data recorded by a Web server are not sufficient for distinguishing among different users and for distinguishing among multiple visits of the same person. The standard Common Logfile Format (CLF), as well as Extended CLF (ECLF), only records host or proxy IP from which requests originate, so different visitors sharing the same host or one proxy server can not be distinguished. Methods such as cookies, user registration or the remote agent, a cookie-like mechanism used in [18] make the identification of a visitor possible. The shortcomings of such methods are that they rely on user s cooperation, but user often denies providing such cooperation due to privacy concerns or thinking it is troublesome. In [1], the author provides some heuristics for user identification. The first heuristic states two accesses having the same IP but different browser (versions) or operation system, which are both recorded in agent field, are originated from two different users. The rational behind this heuristic is that a user, when navigating the web site, rarely employs more than one browser, much more than one OS. But this method will render confusion when a visitor actually does like that. The second heuristic states that when a web page requested is not reachable by a hyperlink from any previously visited pages, there is another user with the same IP address. But such method introduces the similar confusion when ser types URL directly or uses bookmark to reach pages not connected via links. Another main obstacle to user identification is the use of proxy servers. Use of a machine name to uniquely identify users can result in several users being erroneously grouped together as one user. An algorithm presented in [25] checks to see if each incoming request is reachable from the pages already visited. If a page is requested that is not directly linked to the previous pages, multiple users are assumed to exist on the same machine. In [17], user session lengths determined automatically based on navigation patterns are used to identify users. Other heuristics involve using a combination of IP address, machine name, browser agent, and temporal information to identify users [26]

6 2.3 User Session Definition User session here is the set of consecutive pages visited by a single user at a defined duration. For logs that span long periods of time, it is very likely that users will visit the Web site more than once. The goal of session identification is to divide the page accesses of each user into individual sessions. Similar to user identification, Cookie and Session mechanisms both can be used in the session identification, yet same problems also exit. To define user session, two criteria are usually considered: 1. Upper limit of the session duration as a whole; 2. Upper limit on the time spent visiting a page. Generally the second method is more practical and has been used in WUM and WebMiner. It is achieved through a timeout, where if the time between page requests exceeds a certain limit, it is assumed that the user is starting a new session. Many commercial products use 30 minutes as a default timeout, and [27] established a timeout of 25.5 minutes based on empirical data. Once a site log has been analyzed and usage statistics obtained, a timeout that is appropriate for the specific Web site can be fed back into the session identification algorithm. 2.4 Path Completion In order to reliably identify unique user session, it should determine if there are important accesses that are not recorded in the access log. The reason why causes such matter is mainly because of the presence of Cache. Mechanisms such as local caches and proxy servers can severely distort the overall picture of user traversals through a Web site. If user clicks backward to visit a page that has had a copy stored in Cache, browser will get the page directly from Cache. Such a page view will never be trailed in access log, thus causing the problem of incomplete path, which need mending. Current methods to try to overcome this problem include the use of cookies, cache busting, and explicit user registration. As detailed in [26], none of these methods are without serious drawbacks. Cookies can be deleted by the user, cache busting defeats the speed advantage that caching was created to provide and can be disabled, and user registration is voluntary and users often provide false information. Methods similar to those used for user identification can be user for path completion. To accomplish this task needs to refer to referrer log and site topology, along with temporal information to infer missing references. Of the referred URL of a requesting page does not exactly match the last direct page requested, it means that the requested path is not complete. Furthermore, if the referred page URL is in the user s recent request history, we can assume that the user has clicked the backward button to visit page. But if the referred page is not in the history, it means that a new user session begins, just as we have stated above. We can mend the incomplete path using heuristics provided by referrer and site topology

7 2.5 User Session Reconstruction This step is to reconstructing the faithful users navigation path within the identified sessions. In particular, errors in the reconstruction of sessions and incomplete tracing of users activities in a site can easily result in invalid patterns and wrong conclusions. By using additional information about the web site structure is still possible to reconstruct a consistent path by means of heuristics. Heuristic methods for session reconstruction must fulfill two tasks: First, all activities performed by the same physical person should be grouped together. Second, all activities belonging to the same visit should be placed into the same group. Knowledge about a user s identity is not necessary to fulfill these tasks. However, a mechanism for distinguishing among different users is indeed needed. For the achievement of session reconstruction, there are several proactive and reactive strategies. Proactive strategies include user authentication, the activation of cookies that are installed at the user s work area and can thus attach a unique identifier to her requests, as well as the replacement of a site s static pages with dynamically generated pages that are uniquely associated with the browser that invokes them. These strategies aim at an unambiguous association of each request with an individual before or during the individual s interaction with the Web site. Reactive strategies exploit background knowledge on user navigational behavior to assess whether requests registered by the Web server can belong to the same individual, and whether these requests were performed during the same or subsequent visits of the individual to the site. These strategies attempt to associate requests to individuals after the interaction with the Web site, based on the existing, incomplete records. [7] evaluates the performance of heuristics employed to reconstruct sessions from the server log data. Such heuristics are called to partition activities first by user and then by visit of the user in the site, where user identification mechanisms, such as cookies, may or may not be available. 2.6 Transaction Identification Before any mining is done on Web usage data, sequences of page references must be grouped into logical units representing Web transactions. A transaction differs from a user session in that the size of a transaction can range from a single page reference to all of the page references in a user session, depending on the criteria used to identify transactions. Unlike traditional domains for data mining, such as point of sale databases, there is no convenient method of clustering page references into transactions smaller than an entire user session. This problem has been addressed in [28] and [17]. How to define transaction depends on what kind of knowledge we want to mine. In - 7 -

8 WUM [2, 3], user session mentioned above as user session based on duration and transaction here as user session based on structure. As WUM aims at analyzing navigation patterns of users, so it is reasonable to take session as transaction. However, WebMiner [1, 17] concentrates on association rule and sequential pattern mining, so it tends to divide user session into transaction, which is set of semantically related pages. In order to divide user session into transaction, WebMiner classifies pages in a session into auxiliary pages and content pages. Auxiliary pages are those that are just to facilitate the browsing of a user while searching for information. Content pages are those that user are of interest and that they really want to reach. Using the concept of auxiliary and content page references, there are two ways to define transactions. The first would be to define a transaction as all the auxiliary references up to and including each content reference for a given user, which is a so-called auxiliary-content transaction as all of the transaction. The second method would be to define a transaction as all of the content references for a given user, called content-only transactions. Based on the two definitions, WebMiner employs two methods to identify transaction: one is reference length; the other is maximal forward reference. It also uses time window as a benchmark to evaluate the two methods [1, 17, 16]. 2.7 Formatting Once the appropriate preprocessing steps have been applied to the server log, a final preparation module can be used to properly format the sessions or transactions for the type of data mining to be accomplished. For example, since temporal information is not needed for the mining of association rules, a final association rule preparation module would strip out the time for each reference, and do any other formatting of the data necessary for the specific data mining algorithm to be used. [29] stores data extracted from web logs into a relational database using a click fact schema, so as to provide better support to log querying finalized to frequent pattern mining. [30] introduces a method based on signature tree to index log stored in databases for efficient pattern queries. A tree structure, WAP-tree, is also introduced in [51] to register access sequence to web pages. The author also does the optimization of this structure to exploit the sequence mining algorithm. 3. Problems in Preprocessing Clearly improved data quality can improve the quality of any analysis on it. Data quality is a source of major difficulties for Web usage mining, and particularly for the sessionizing problem. The solution of a dedicated server recording all activities of each user individually was put forward as a tested and successful - 8 -

9 solution. In the absence of such a server, cookies and scripts can be used to distinguish among unique users. However, they are not always feasible or popular, due to privacy considerations. A problem in the Web domain is the inherent conflict between the analysis needs of the analysts (who want more detailed usage data collected), and the privacy needs of users (who want as little data collected as possible). In addition caching and proxy servers can make reconstructing reliable sessions difficult. 4. Conclusion In this survey, I address a typical sequence of tasks of Web usage mining preprocessing. The techniques and methods used in each individual task are also presented in great details. One important thing I want to point out is that each task relies heavily on each other. In practical application, some of the tasks are carried out together and do not distinguish with each other very clearly. Moreover, for specific mining applications, the procedures of preprocessing may be of little variation. As for the application of Web personalization [14], the preprocessing steps may include data selection, cleaning and transformation and the identification of users and user sessions. Acknowledgments I would like to express my gratefulness to Dr. Wilfred Ng for guiding this survey. I also thank Lu An for the interesting discussions. References [1] Robert Cooley, Bamshad Mobasher, and Jaideep Srivastava. Data Preparation for Mining World Wide Web Browsing Patterns (1999), Knowledge and Information Systems V1(1). [2] B. Berendt, M. Spiliopoulou. Analyzing navigation behavior in Web sites integrating multiple information systems. VLDBJournal, Special Issue on Databases and the Web 9, 1 (2000), [3] M. Spiliopoulou. Web usage mining for Web site evaluation. Communications of the - 9 -

10 ACM Volume 43, Number 8 (2000), Pages [4] R. Cooley, M. Deshpande, J. Srivastava, and P.N. Tan. Web usage mining: Discovery and applications of usage patterns from web data. ACM SIGKDD Explorations, 1(2), January [5] Kdnuggets. Software for web mining. [6] M. Spiliopoulou and L.C. Faulstich. WUM: a Web Utilization Miner. In Proceedings of the EDBT Workshop WebDB98, volume 1590 of LNCS, pages , [7] B. Berendt, B. Mobasher, M. Spiliopoulou, and M. Nakagawa. A framework for the evaluation of session reconstruction heuristics in web usage analysis. INFORMS Journal of Computing, 15(2), [8] L.C. Faulstich, et al. WUM: A Tool for Web Utilization Analysis. In EDBT Workshop WebDB , Spain. [9] Z. Pabarskaite. Implementing Advanced Cleaning and End-User Interpretability Technologies in Web Log Mining. In Proceedings of the 24 th International Conference ITI [10] A. Abraham and Xiaozhe Wang. i-miner: A web usage mining framework using hierarchical intelligent systems. In The IEEE International Conference on Fuzzy Systems FUZZ-IEEE'03, [11] T. Ohmori, Y. Tsutatani, M. Hoshi. A Novel Datacube Model Supporting Interactive Web-log Mining. In Proceedings First International Symposium. pp Cyber Worlds, [12] R. Kosala, H. Blockeel. Web Mining Research: A Survey. SIGKDD Explorations: Newsletter of the Special Interest Group (SIG) on Knowledge Discovery & Data Mining, ACM, 2(1), [13] B. Masand and M. Spiliopoulou. Webkdd-99: Workshop on web usage analysis and user profiling. SIGKDD Explorations, 1(2), [14] M. Baglioni, U. Ferrara, A. Romei, S. Ruggieri, and F. Turini. Preprocessing and Mining Web Log Data for Web Personalization. The 8th Italian Conf. on Artificial Intelligence : Vol of LNCS, September

11 [15] R. Cooley, B. Mobasher, and J. Srivastava. Web Mining: Information and Pattern Discovery on the World Wide Web. In Proceedings of the 9th IEEE International Conference on Tools With Artificial Intelligence (ICTAI 97), Newport Beach, CA, [16] R. Cooley, B. mobasher, and J. Srivastava. Data Preparation for Mining World Wide Web Browsing Patterns. Knowledge and Information Systems, Volume 1, No. 1, [17] Robert Cooley, Bamshad Mobasher, and Jaideep Srivastava, Grouping Web Page References into Transactions for Mining World Wide Web Browsing Patterns (1997), in Proceedings of the 1997 IEEE Knowledge and Data Engineering Exchange Workshop (KDEX-97), November [18] C. Shahabi, A. M. Zarkesh, J. Adibi, V. Shah. Knowledge discovery from users Web-page navigation. Research Issues in Data Engineering, Proceedings. Seventh International Workshop on, 7-8 April 1997, Page(s): [19] Federico Michele Facca and Pier Luca Lanzi. Recent Developments in Web Usage Mining Research. Data Warehousing and Knowledge Discovery: 5th International Conference, DaWaK 2003, Pages(s): [20] Pang-Ning Tan, and Vipin Kumar. Modeling of Web Robot Navigational Patterns. In WebKDD 2000 Web Mining for E-commerce Challenges and Opportnities, Second International Workshop, August [21] Ron Kohavi, et al. Lessons and Challenges from Mining Retail E-Commerce Data. Journal of Machine Learning, [22] D. Pyle, Data Preparation for Data Mining. Morgan Kaufmann Publishers Inc., Dan Francisco, CA. [23] R. Cooley, P. Tan, J. Srivastava Discovery of interesting usage patterns from Web data. B. Masand, M. Spiliopoulou, eds. Advances in Web Usage Analysis and User profiling. LNAI 1836, Springer, Berlin, Germany [24] Pang-Ning Tan, and Vipin Kumar. Discovery of web robot sessions based on their navigational patterns. Data Mining and Knowledge Discovery, 6(1): 9-35, [25] P. Pirolli, J. Pitkow, and R. Rao. Silk from a sow s ear: Extracting usable structures from the web. In Proceedings of 1996 Conference on Human Factors in Computing

12 Systems (CHI-96), Vancouver, British Columbia, Canada, [26] J. Pitkow. In search of reliable usage data on the www. In Sixth International World Wide Web Conference, pages , Santa Clara, CA, [27] L. Catledge, J. Pitkow. Characterizing browsing behaviors on the World Wide Web, Computer Networks and ISDN Systems 27(6), 1995, pp [28] M.S. Chen, J.S. Park, and P.S. Yu. Data mining for path traversal patterns in a web environment. In Proceedings of the 16 th International Conference on Distributed Computing Systems, pages , [29] Jesper Andersen, Anders Giversen, Allan H. Jensen, Rune S. Larsen, Torben Bach Pedersen, and Janne Skyt. Analyzing clickstreams using subsessions. In International Workshop on Data Warehousing and OLAP (DOLAP 2000), [30] Alexandros Nanopoulos, Maciej Zakrzewicz, Tadeusz Morzy, and Yannis Manolopoulos. Indexing web access-logs for pattern queries. In Fourth ACM CIKM International Workshop on Web Information and Data Management (WIDM 02), [31] Jian Pei, Jiawei Han, Behzad Mortazavi-asl, and Hua Zhu. Mining access patterns efficiently from web logs. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pages ,