An Intelligent Plural Auto Recognition Search Engine. Brian King Senior, Electrical & Computer Engineering Department

Transcription

1 An Intelligent Plural Auto Recognition Search Engine Brian King Senior, Electrical & Computer Engineering Department Siripong Malasri Professor and Dean School of Engineering Christian Brothers University Memphis, TN ABSTRACT Today, there are over tens of millions of web sites on the Internet. The two methods that are currently available for finding information on the web are search engines or subject guides. Search engines are a better choice when looking for specific information. Most search engines do keyword searches against a database. Presently, keyword searches use exact match approaches. Plural is not considered as part of a search for a singular keyword. Therefore, important information could be bypassed by current search engines. In this paper, a search engine was designed and developed to add the plural of a singular word as a keyword for the search. A rule -based system containing plural rules including exceptions was developed to identify the plural of a given singular word. Once the search keywords are established, the search engine randomly activates a commercial search engine and selects the URL at the top of the list. It scans and records the number of matches. The search engines then use the depth-first search approach to follow links based on the number of hits. When a user-specified stopping criterion is met, the search engine generates a report to the user and the user can visit those web pages identified by the engine. INTRODUCTION Searching the Internet is an activity that many people perform as a part of their routine tasks. This activity has become more and more cumbersome with the addition of information to the World Wide Web on a daily basis. This paper presents a search engine based on artificial intelligence concepts. There are two basic search techniques that are used to solve AI problems: breadth-first search and depth-first search. These search techniques are present in all types of AI problems, but are especially useful in network problems. Figure 1 shows the algorithm for the breadth-first search (Bigus, 1998). Create a queue and add the first Node to it. Loop: If the queue is empty, quit. Remove the first Node from the queue. If the Node contains the goal state, then exit with the Node as the solution. For each child of the current Node: Add the new state to the back of the queue. Figure 1 Breadth-First Search Algorithm 1

2 Basically, this algorithm implies that this method searches one level at a time in its entirety. The algorithm will end as soon as the answer is discovered or when the entire network has been searched. The problem with this technique is that it is not optimal and it grows exponentially with respect to time and computer memory consumption. The other method is depth-first search. Figure 2 shows the algorithm for this technique. Create a queue and add the first Node to it. Loop: If the queue is empty, stop. Remove the first Node from the queue. If the Node contains the goal state, then exit with the node as the solution. For each child of the current Node: Add the new state to the front of the queue. Figure 2 Depth First Search Algorithm Unlike the breadth-first search method, the depth-first search method does not search one layer at a time. Instead, it searches one entire path to its end and then backtracks up the network and searches down an untested path. This method has lower memory requirements than the breadth-first method, but is not complete or optimal (Bigus, 1998). This work utilizes the depth-first search method. A method for the learning and reasoning system of an agent must be chosen before the agent can be constructed. Three of the most common methods are rule-based systems, neural networks, and decision trees. The most popular form of knowledge representation used in AI applications is if-then rules, hence is used in this work. DESIGN & IMPLEMENTATION The first step in the design process was to have a clear understanding of what the final prototype should look like and how it would accomp lish each of it s goals. Figure 3 shows the flowchart for this design. Figure 3 Software Design Flow Chart 2

3 This project encompasses building a java application that searches the Internet for a user-specified keyword and it s plural. There are many reasons that Java was the chosen language for this project. Java is an architecture-neutral language, implying that it is operating system independent. Also, the programs are cross-platform applications (portable). Another reason that Java was chosen was due to the built-in functions that allow for web access, file transfer, and additional security features. The program starts on one page and searches that page for the keyword along with any links from that page. The initial page is obtained from one of the major search engines (Yahoo, Infoseek, etc.). This gives the program hundreds of thousands of possible results from which it gives the user the best matches. Then it advances to the first link and scans it for keywords and links. It continues on this process until the user has been given the number of matches that they specified. If the program reaches a site that does not contain the keyword at least once, the program then ignores all of the links from that page. The next section of the paper shows how certain sections of the code were written. The first step is retrieving the keyword from the user. Figure 4 shows the code for getting the keyword from the user. if (args.length!=1) { System.out.println ("Usage: java searchia <keyword>"); System.exit (10); String keyword = args[0].substring (0); Figure 4 Keyword Retrieval The if statement checks to ensure that the user has entered a keyword. If the user forgot to enter the keyword, the program quits and shows the user the correct method for running the program. The program takes the arguments from the command prompt and stores them in a string named keyword. Now that the program has the keyword, it takes this word and uses the English rules for forming plurals (Appendix A) and the exceptions (Appendix B) and creates the plural to match the given keyword. Figure 5 shows the code for handling the exceptions. if (keyword.equals("man")) { keyword1 = "men"; Figure 5 Plural Exceptions This code utilizes the built in equals () feature of the String class. This function is true if the strings that are referenced are equal. The two strings are identical only if their length is the same and if each character is identical. Since each character must be the same, this function is case-sensitive. Thus, it is important that the keyword be converted to lowercase to ensure the words will match. Once all of the exceptions have been checked, each of the rules must be systematically checked. Figure 6 shows the code for the rules. 3

4 else if (keyword.charat(i) == 'o') { keyword.getchars(0, i, keyarray, 0); String key = String.copyValueOf (keyarray); key += "es"; keyword1 = key; else if (keyword.charat(--i) == 'o' && keyword.charat(++i) == 'n') { --i; keyword.getchars(0, i, keyarray, 0); String key = String.copyValueOf (keyarray); key += "a"; keyword1 = key; Figure 6 Plural Rules This section of code uses the charat function. This function allows the programmer to choose any character contained within the string. The length of the string is determined from the command int keylength = keyword.length ();. Once the characters at the given position have been tested and deemed a match, another function getchars is used to eliminate the necessary portion of the original keyword. The problem is that now another function must be used to convert this array of characters back into a string. Once the new string is formed, it is combined with the appropriate ending and renamed keyword1. The next step in the process is to obtain the starting point for the program. Figure 7 shows the code for the random number generator and the search engine selector. int rand; Random x = new Random (); rand = x.nextint(7); switch(rand) { case 1: wp = " + "p=" + keyword+keyword1; System.out.println("Search results provided courtesy of Yahoo"); break; Figure 7 Engine Selector This portion of the code randomly chooses a number between 0 and 6. This number is then used to determine which search engine will be used as the starting point of the program. Visiting each site discovered the method used to address the search engine and noted the URL that was displayed after a search was activated. The final portion of the code deals with keyword matching and link identification. Figure 8 shows a sample of this section of code. 4

5 URL url = new URL(wp); BufferedReader in = new BufferedReader ( new InputStreamReader ( url.openstream())); String buffer; String keystr = keyword; while (!(null==(buffer=in.readline()))){ buffer = buffer.tolowercase(); if (buffer.startswith("<a href=")) { ++ucount; int index =-1; index = buffer.indexof(keystr); while(index >= 0) { ++k; index += keystr.length(); index = buffer.indexof(keystr, index); Figure 8 Identification This section of code uses built-in features to input an html file and reads the file line by line until the end of the file has been reached. At the same time, the code converts the input to all lower cases to simplify the work of matching the links and keywords. Also, a built -in index method is used to search for the keywords. The startswith () method is used to identify the links. All links in an html file begin with <a href=. The only other feature of the project is implementing a breadth-first search using the obtained links. This search will continue until ten levels have been searched or until the program runs out of links to travel. CONCLUSIONS This paper explains how to construct an intelligent agent that will aid the user by acting as an interface agent. This agent, in its simplest form, filters out the garbage and tries to give the user only the information that is most relevant to their requested search. While the search engine met the design criteria set forth by this project, there are many ways that it could be further improved to provide an even more efficient and reliable way to search the Internet. One method would be to allow the program to find all of the forms of the provided keyword, not just the plurals. Another method would be to increase the duties of the agent. That is, allow the agent to suggest sites, not solely on a requested item, but also in conjunction with websites that have been viewed in the past. Another method would be to incorporate a neural network or a genetic algorithm into the agent in order to make the program more efficient and productive. REFERENCES Bigus, Joseph, and Jennifer Bigus. Constructing Intelligent Agents with Java, Wiley Computer Publishing, Jennings, N.R. Applications of Intelligent Agents, University of London, 09 September

6 APPENDIX A PLURAL EXCEPTIONS Word Man Foot Child Person Tooth Mouse Woman Goose Sheep Deer Fish Datum Species Alumna Memorandum Larva Moose Series Medium Plural Men Feet Children People Teeth Mice Women Geese Sheep Deer Fish Data Species Alumnae Memoranda Larvae Moose Series Media APPENDIX B PLURAL RULES Ends With s,x,,sh,z fe f o us is on most others How to form the plural add es change f to v and add s change f to v and add es add es change us to i change is to es change on to a add s 6