Key words: Wikipedia - a non-profit encyclopedia project available online and written collaboratively by volunteers from around the world

Transcription

1 Jacob Wenger Programming Paradigms Thursday, May 12, 2011 Six Degrees of Wikipedia Abstract: I created an application which allows you to find the shortest paths between any two Wikipedia articles. Paths, in this sense, follow along the hyperlinks found on every Wikipedia page. Data about Wikipedia was found online in the form of queries returning every row in various tables from the Wikipedia database. I downloaded these files, parsed them, sorted them, and stored them in a local SQLite database. I created a simple, intuitive front-end graphical user interface (GUI) using PyQt. The GUI requires the user to provide only the starting and ending article names and click a large button to find the paths between them. Additional features which determine if the current query is an actual article and which provide suggested article names are also provided by querying the local database. Aspects of functional programming (e.g. list comprehensions) and regular expressions (e.g. command-line utility sed and querying the database for suggested articles) were used in this project. Although Python was the predominant language used, a shell script which retrieved and parsed the online data was also created. Key words: Wikipedia - a non-profit encyclopedia project available online and written collaboratively by volunteers from around the world Page a Wikipedia article Link a hyperlink on a page which leads to another page Redirect a page which does not exist but redirects to an existing page; for example, Airplane is a redirect to Fixed-wing aircraft and Gandhi redirects to Mohandas Karamchand Gandhi Breadth first search a graph search algorithm that begins at the root node and explores all of its neighboring nodes; then, for each of those nodes, it explores their unexplored neighbors; this process continues until the end node is found or until all nodes are explored SQLite - an embedded relational database management system contained in a relatively small C programming library; it is essentially a less-powerful version of an SQL database PyQt a Python binding of the cross platform graphical user interface (GUI) toolkit Qt

2 Introduction: The articles which comprise Wikipedia [5][6] represent, in some sense, the collective knowledge of humanity. In only ten short years since its founding in 2001 Wikipedia has already amassed over 3 million articles, a number which is rising every single day. I was interested in seeing how closely this knowledge is tied together. Concepts such as six degrees of separation, which states that every person is connected to every other person through only six familial links, further piqued my interest. As a result, I decided to create an application called Six Degrees of Wikipedia (SDOW for short) which determines the shortest path from one Wikipedia article to another by following the hyperlinks found on the Wikipedia pages themselves. To my surprise, this problem had not previously been sufficiently solved. Through my research, I discovered that only one person, namely Stephen Dolan, had even attempted to complete a similar project [3]. However, his implementation was simple, buggy, and sometimes incorrect (as I can now prove) and thus was not satisfactory. Besides him, I saw no other mention of anyone who even attempted to tackle the problem of finding the shortest path between two Wikipedia articles. This was an exciting discovery since it meant I was the first person to provide a suitable solution to problem I posed. This project began by obtaining data on Wikipedia articles from an online dump provided by Wikipedia itself [1]. This information was parsed and stored in a local SQLite database. A breadth-first searching algorithm was implemented in Python and made use of the local database to find the paths between two Wikipedia articles. Finally, a graphical user interface (GUI) was created in PyQt to make the experience for users quick and intuitive. Usage examples: To open the GUI for Six Degrees of Wikipedia, simply type the following command: python./sixdegreesofwikipedia.py The GUI is very simple and intuitive. There are two line edits which represent the starting and ending Wikipedia article names. There is also a button titled Find Wikipedia Links! which finds the paths between the two articles when pressed. Finally, there is a search log at the bottom which keeps track of the recent search history. The GUI is shown on the following page.

3 If you wanted to find the shortest path from the Wikipedia articles Search-based application and Wikipedia, you can start by typing Search-based application into the starting article line edit. As you type, you will notice that the color of the text changes. The text will be one of three colors: (1) green for when it corresponds to an actual Wikipedia article, (2) yellow for when it corresponds to a Wikipedia redirect, and (3) red in every other case. Another feature you will notice is suggested search terms, similar to those found in Google. The suggestions are filtered in order to provide you with the pages you most likely care about. The combination of these two features provides a powerful user interface which tells the user if the article they are searching for exists and saves them time by auto-completing their query through suggestions. When Search is typed in the image below, it is colored yellow to signify it is a redirect. In addition, suggested articles, including Search-based application, are given.

4 Once both the starting and ending articles are specified, the Find Wikipedia Links! button can be pressed. If either of the article names does not actually correspond to a real Wikipedia article (i.e. they are red), the search log will alert you to this error. If either of them are redirects (i.e. they are yellow), the search log will tell you to which article they redirect. If both articles exist and neither of them are redirects (i.e. they are green), the search for the shortest path between them will begin. Most searches take between five to ten seconds. A message which tells you how many paths were found between the two articles and of what degree the paths are will be written to the search log. This will be followed by lists of paths, each element of which is separated by a pipe. The results will look like those seen in the image below. Approach and methods: A lot went into the making Six Degrees of Wikipedia. First, the Wikipedia data needed to be obtained. It came from an online dump of the Wikipedia database [1]. I downloaded gzipped files which contained a query of all the rows in three tables: pages, links, and redirects. I also found a file containing the hit counts of every Wikipedia article [2]. These large files were unzipped, parsed using command line utilities like grep, awk, and sed, sorted, and placed into a local SQLite database through the use of Python scripts. A shell script named getdata.sh was run as an SGE job on opteron.crc.nd.edu and took over six hours to run. Next, a searching algorithm needed to be written to find the paths between any two Wikipedia

5 pages. I modeled Wikipedia as a directed graph, where each article was a node and each link between two articles was an edge. In order to find the shortest path between two nodes, a breadth first search (BFS) was the best option. The search began at the node represented by the starting article. All of its neighbors (i.e. the articles to which it links to directly) were found and if any of them were the node representing the ending article, the BFS finished. Otherwise, the unexplored neighbors of the starting node s neighbors were found and again checked to see if they were the end node. The BFS continues until the end node is reached or all the nodes are explored. This procedure guarantees that the shortest path between the two nodes would be found if it exists. However, it is quite slow because even after only a few iterations, millions of nodes are being searched. To speed up this process I implemented a bi-directional BFS. This essentially did two separate BFS, one from the starting node and one from the ending node. The two BFS switch back and forth after each one completes one level of the search. Whenever these two BFS overlapped (i.e. contained the same node), the entire process finished. This sped up things considerably. One final enhancement I made was implementing an intelligent bidirectional BFS. By intelligent, I mean that the algorithm always chose to explore the next level of neighbors of the BFS that had the fewest neighbors. This was an improvement over the plain bi-directional BFS and again resulted in a good speedup. The Python script breadthfirstsearch.py contains the BFS algorithm. The BFS algorithm made use of the local databases I created. First, I used the pages table to convert the article names provided in the GUI to article IDs. Next, I ran the BFS from the starting article ID to the ending article ID suing the links and redirects databases. This returned a list of paths of IDs, which were converted to article names again using the pages database. Although the GUI is rather plain, it contains a lot of functionality. I created a modified QLineEdit class which changes colors according to its current text and which includes a dynamically changing QCompleter which provides search suggestions. The color is determined by querying the redirects and pages databases and returns nearly instantaneous results. The suggestions are found by using a regular expression to query the pages database and are filtered by using the page hit counts database. Finally, the Find Wikipedia Links! button is a simple QButton which calls the BFS when it is pressed and the search log is a simple QTextBrowser.

6 Discussion and future work: I accomplished the goal I set out to achieve. My Six Degrees Of Wikipedia application properly finds the shortest path between any two Wikipedia articles. However, there are still many way I could improve and expand SDOW. My main concern is speeding up the BFS algorithm. The best solution to his problem would most likely be parallelizing the algorithm using MPI or Work Queue. Another area for improvement is suggested articles when a user is typing in the article name. Providing suggestions which are more relevant through the use of more rigorous filters would be a nice improvement. Also, my program does not accept article names with special characters (e.g. é, ü, ñ, etc.) so integration with these is another area for improvement. Conversion to a fully functioning web-based application which allows you to visualize the paths between articles, as show in the image below, is already underway. However, much progress can still be made. Finally, an analysis of the results (e.g. finding strongly connected components or determining the average distance between any two nodes through an all-pair analysis) could yield interesting results on the relatedness of human knowledge. References: [1] Database dump progress. Wikipedia. Accessed November Available at wikimedia.org/ backup-index.html. [2] Index of /wikistats/. Domas Mituzas. Accessed 2 May Available wikistats/. [3] Six Degrees of Wikipedia. Stephen Dolan. Accessed November Available at ~mu/wiki/. [4] The top 500 sites on the web. Accessed 4 May Available at topsites.

7 [5] Wikipedia. Wikipedia. Accessed 4 May Available at Wikipedia. [6] Wikipedia: The Free Encyclopedia. Wikipedia. Accessed November Available at Acknowledgements: First, I would like to thank my brother, Aaron Wenger, for his help when this project was in its infancy. This began as a collaboration between the two of us and would not have gotten off the ground without him. Stephen Dolan should also be thanked for the information I obtained from the description he wrote about how he completed a project similar to SDOW. Although he never actually responded to my s he still gave me some helpful pointers. I would also like to thank Chad Heise for his help coming up with this project idea and being a tester throughout the evolution of the code. Finally, thanks goes out to RJ Nowling for showing me how to submit jobs to SGE. Authors: Jacob Wenger is a junior majoring in computer science and the third in a line of Wenger s (preceded by his brother 05 and sister 10) who have attended the University of Notre Dame. When he is not coding the night away, he enjoys watching and attending sporting events, playing basketball and football out on the quad, going out to trivia with his friends, and watching movies. Jacob has spent the past two summers in a research setting and studying abroad in Alcoy, Spain, but will spend next summer out in Seattle working for Microsoft. Upon graduation, Jacob will most likely work in industry for a few years and eventually attend graduate school to pursue a degree in higher education.