Search Engine Survey. May 17, The World Wide Web (WWW) has become a huge information source whose content is increasing

Size: px

Start display at page:

Download "Search Engine Survey. May 17, The World Wide Web (WWW) has become a huge information source whose content is increasing"

Stephen Mosley
5 years ago
Views:

1 Search Engine Survey by Wei Tang May 17, Introduction The World Wide Web (WWW) has become a huge information source whose content is increasing and changing every day. To nd something in this huge data base is like nding a needle in a haystack. Then there comes the help of search engines. Since the emergence of the Internet and the WWW, people have been thinking of ways to ease the search of information. Search engines are considered to be the most successful thing on the Net since its creation. A little history The grandfather of all search engines was Archie [?], created in 1990 by Alan Emtage, a student at McGill University in Montreal. As early as 1990, there was no WWW, the primary method of storing and retrieving les was via the File Transfer Protocol (FTP). Initially, anyone who wanted to share a le had to set up an FTP server in order to make the le available to others, allowing others to retrieve it. But many important les were still scattered on various FTP servers. The le distribution was by passing the word around on the Internet, normally through exchanges and discussion groups. Archie changed all this. Archie's gatherer searches FTP sites across the Internet and indexed all of the les it found. Its regular expression matcher provided users with access to its database. Then came Veronica. It was created as a type of searching device similar to Archie but for Gopher les. Matthew Gray's World Wide Web Wanderer was the rst robot on the web and was designed to track the Web's growth. It soon amended its way because early versions of the software ran rampant through the Net and caused a noticeable netwide performance degradation. But the Robot technique introduced by Wanderer were widely adopted in later search engines development. By December 1993, three search engines powered by robots had made their debut: JumpStation, the World Wide Web Worm, and the Repository- Based Software Engineering (RBSE) spider. Due to lacking the intelligence to understand what it was that they were indexing, early spiders really suered. That was why some searchable directories came about. The pioneer was EINet Galaxy, now know as the Tradewave Galaxy. The Galaxy went online in January It contained Gopher and Telnet search features in addition to the web-searching features. Galaxy is a true directory in the sense that it lists only URLs that have been submitted to it, and all categorization and review of the submitted URLs is done by hand. This results in higher-quality pages and more relevant searches, but 1

2 far fewer pages to search through. In April 1994, two Stanford Ph.D. candidates, David Filo and Jerry Yang created the fantastic Yahoo! [?]. Yahoo! was considered a searchable directory rather than a search engine because the entries were entered and categorized manually. But since Yahoo! has automated some aspects of the gathering and classication process, it blurred the distinction between search engine and directory. In early 1994, WebCrawler came into being with its very unique feature: allowing the user to search the full text of entire documents, changing the way previous robots worked: storing the title and the URL, and the rst 100 or so words of a document 1. The most important point about WebCrawler is that it was the rst full-text search engine on the Internet. Several competitors emerged within a year of WebCrawler's debut: Lycos, Infoseek, and Open- Text. Lycos was out of the labs at Carnegie Mellon University during the July of Infoseek was initially just another search engine. It borrowed conceptually from Yahoo! and Lycos, not really innovating in any particular way. Infoseek's user-friendly interface and the numerous additional services (such as UPS tracking, News, a directory, and the like) got it commendation, but it was Infoseek's strategic deal with Netscape in December 1995 that brought it to the forefront in the search engine camp. Infoseek convinced Netscape to have its engine pop up as the default when people hit the Net Search button on the Netscape browser (Prior to this, Yahoo! was Netscape's default search service, and now it is Excite.). Because of Netscape's undoubtable dominance in the browser market by then, Infoseek gained the popularity. AltaVista was a latecomer to the scene. It had its online debut in December Nonetheless, it had a number of innovative features that quickly catapulted it to the top. One of the most impressive features is its speed. Run on a cluster of Alpha stations, AltaVista can handle millions of hits per day without slowing down its processing a bit. Today, AltaVista has grown to be one of the largest index engines. Main memory indexing and cluster servers have made this to happen. On May 20th, 1996, Inktomi Corporation [?] was formed, and HotBot was unleashed to the world (HotBot is powered by Inktomi, so it Lycos now.). The Inktomi engine was quickly licensed to Wired magazines web site, HotWired. This site's popularity accounted for much of the initial fervor over HotBot. Although the youngest among all the major search engines, HotBot is probably the most powerful of the search engines, with a robot that can supposedly index 10 million pages per day. Northern Light Technology, was founded \by a seasoned management team of librarians, businessto-business, and Internet professionals who recognized the need to ll the gap left by search engines and private research services. Northern Light introduced its research engine in August of 1997" [?]. Northern Light claimed to have the largest search engine database. It now partners with the U.S. Commerce Department's National Technical Information Service (NTIS) for the "one-stop shopping" service: gov.search [?]. This survey is organized as follows: in Section??, I will give a brief classication of dierent search services; Section?? introduces some basic concepts about search engines and surveys several major search engines, including AltaVista [?], HotBot [?], Infoseek [?], Lycos [?], Excite [?], and Northern Light [?]; Section?? presents a summary of this report. 1 In 1997, Excite bought out WebCrawler, and now AOL is using an Excite derivative as the engine behind its own NetFind. 2

3 2 Classication of search services As discussed in Section??, there are various avors of search services on the Web. Generally, they can be classied into three categories: dictionaries, search engines and meta engines. Directory (or Catalog) One of the most commonly misunderstood topics is the dierence between directories and search engines. Directories, unlike search engines, are not automated. Site URLs, titles and descriptions must be submitted to them manually for best results. Usually, each directory presents the user with a number of categories under which the listing can be placed. In some cases, the directory will determine which category the site will be placed into, either automatically or through a manual review by an editorial sta. The best example of a directory is Yahoo! [?]. They are placed in easily navigable category folders and sub-folders. When a search is performed on a directory, the results that are generated originate directly from the listing information that was organized under a certain topic in the dictionary. Another example is LookSmart [?]. Search engines There are relatively fewer search engines than directories on the Web today because of the technical diculties of building a real search engine. The real search engines are \robot-based" because they employ programs called \robots" or \spiders" that are constantly roaming the web indexing everything they nd. Search engine databases, when compared with that of directories are huge. There are two reasons for this: 1. search engines do not review sites that are submitted to decide if their content is relevant - everything that meets its programmed requirements is accepted; 2. search engines will index multiple pages from one web site, whereas directories will usually only allow you one (sometimes two) listings per domain name. Currently the most popular robot-based search engines on the Web are: AltaVista, HotBot, Excite, Lycos, Infoseek, Webcrawler, Northern Light, Planetsearch, and GoTo.com. In the rest of this report, I refer to these robot-based search engines \general purpose search engines". Meta engines Another category of online search services is \meta engines", which forward search queries to many of the major search engines at once. The interface is usually uniform, hiding the distinct features from each used search engine. And the results are normally reformatted by the meta engines. MetaCrawler [?]. is an example in this category. Another interesting site is Askjeeves [?], which answers questions in natural English language, such as \what is the meaning of search engine" and \where do I nd Java networking books". Interestingly, Infoseek has a standalone product called \Infoseek Express" that is a meta engine and runs as a plug-in to both Netscape and Internet Explorer browsers. 3

Figure 1: How search engines work (from http://www.ipoinc.

4 Figure 1: How search engines work (from 3 General purpose search engines In this section, I will survey several major general purpose search engines and compare them based on certain criteria. Some basic concepts and architectures are also discussed. 3.1 How do they work? Normally, a search engine has three main components: 1. a software robot to roam around the Web; 2. everything the robot nds out goes into the second part: a giant index; 3. search engine software is the third part of a search engine. This is the program that sifts through the millions of pages recorded in the index to nd matches to a search and rank them in order of what it believes is most relevant. Figure?? shows a general picture of how search engines work. 3.2 How do they dier? Not all search engines are created equal. There are some important criteria that we can use to distinguish them. Here, I present the dierences among six major search engines: AltaVista, HotBot, Infoseek, Lycos, Excite, and Northern Light. Size Size matters. Although it is very unlikely that any of the existing search engines can cover 4

Figure 2: Search engine sizes (as of May 1, 1999, from http://www.searchenginewatch.com) the whole Web, all the search engines are trying to index as many web pages as possible. Figure?

5 Figure 2: Search engine sizes (as of May 1, 1999, from the whole Web, all the search engines are trying to index as many web pages as possible. Figure?? shows a recent report on the index sizes of major search engines. Figure?? shows the trend of size increases for each of them over time. Index scope Among the six search engines, all of them provide search facilities on the Web, Newsgroups, and Addresses (Business, Residential, ). Only Northern Light provides FTP search. Indexing method In terms of indexing methods, they can be based on full text, keyword, or even natural language. All of the six major search engines use full text indices. Only Lycos and Excite support natural language indexing method. Search Logic 5

$com) Search Engine AltaVista Hotbot Infoseek Lycos Excite Search Logic Default is \OR"; advanced searching supports full boolean terms: OR, AND, NOT, NEAR; using upper case letters forces an exact$

6 Figure 3: Search engine sizes over time (from Search Engine AltaVista Hotbot Infoseek Lycos Excite Search Logic Default is \OR"; advanced searching supports full boolean terms: OR, AND, NOT, NEAR; using upper case letters forces an exact match, lower case will search both upper and lower case; uses \*" to truncate Default is \AND"; boolean searching supports AND, OR, and NOT; advanced searching supports \must", \should" and \must not" semantics; uses \*" to truncate Default is \OR", allows variations of AND and NOT; searches are case sensitive Default is \OR"; advanced searching supports AND, OR, NOT and rich relationship operators: ADJ, NEAR, FAR and BEFORE Default is \AND", supports OR and AND NOT; recognizes upper case letters on word beginnings as names; terms are automatically searched as word prexes or roots Northern Light Default is \AND", allows OR and NOT; supports both \%" and \*" as truncation symbols; boolean terms OR and NOT may be inserted between words or phrases in quotes In addition, all these search engines support \+" as including words and \-" as excluding words, as well as double quotes \" to enclose a phrase. 6

7 Power features Search Engine AltaVista Hotbot Infoseek Lycos Excite Northern Light Power feature supports searching in 25 languages (including Asian languages) and translation between 5 Romanized languages; renes on date supports searching in 9 Romanized languages; renes on date, le extension, location/domain, page depth and word stemming (grammatic variations); searches within results renes on domain and location; searches within results supports searching in 15 Romanized languages; searches within results; features powerful MP3 searching supports searching in 9 languages (including Asian languages); renes on domain and location; has adult content lter; searches similar pages supports searching in 5 Romanized languages; renes on date, web source type, subject and location; categorizes results as \custom search folders" to further rene results, very impressive Ranking algorithm There are several schemes that the search engines use to determine a page's relevance ranking: 1. Location of keywords Words occurring near the top of the page are usually given a greater relevance score by the search engines. Also, some search engines give priority to pages on which the keywords (the words you're searching for) appear in the title of the page, or in the URL (the web address). 2. Frequency of Keywords The number of times a word appears on the page will boost that page forward in the ranking of pages provided by the Search Engine. But don't abuse it, because most of the search engines have anti-spam techniques to identify whether a page is \spamming". If so, they may reject to index the page, or degrade the relevance ranking. 3. Popularity Some Search Engines also give extra points to pages that have other pages linked to them. Excite, Infoseek, Lycos, Google, and Webcrawler consider the \popularity" of a site. 4. Meta Tags Meta Tags are description and content keywords created by the site designer and embedded in the source code. These Meta Tags are considered by Hotbot and Infoseek, but ignored by Excite and AltaVista. 5. Paid listings GoTo.com was the rst one using paid listings. Then AltaVista followed its lead and 7

8 introduced paid listings in April, 1999, hoping that could improve relevancy better than any algorithm. 3.3 What is the problem? Although search engines are great and they have made search on the Net a lot easier than before, there are still some unsolved questions. There are too many of them Nobody really knows how many search engines are out there on the Internet. I only named a few major ones in this report. There are perhaps hundreds of specialized search engines and meta engines available on the Net. No one will try to learn how to use all the search engines. It is most likely that one is prone to stick to one (probably the rst one ever used) particular search engine and do not use others at all (unless it is really necessary). There is no standard Every search engine has its own robot and indexing algorithm. There have been eorts trying to bring all the major players together and settle a interchangeable search engine standard [?]. But it was only adopted by a few vendors. If there were a standard, document accuracy, vendor independence and engine functionality would have improved a lot more. Result duplication Every search engine suers the duplicates in the search results. This is so because of the nature of web pages (documents link to each other). It's also because elimination of duplicates in the database tends to be a tedious work to do. As the main goal of search engines is the speed, duplicates are overlooked. Query expressiveness All search engines use keyword-based search, because it is easy both for the users and for the software vendors. Some search engines use enhanced boolean expressions. But they are still based on keywords. Due to this limitation, we won't expect the same accuracy as we do in database SQL query results. Multimedia search Most of the existing search engines are searching text documents. There have not been general-purpose content-based multimedia search engines on the Internet, although there are some prototype systems [?]. Multimedia applications, especially video on-demand applications are already emerging (such as CNN's videoselect). There will be a need to search audio, image, and video contents on the Net. 4 Summary As discussed in previous sections, search engines are so widely used and they have provided us much convenience. Along with the growth of the Internet, I would also expect the growth of the 8

9 search engine market. While the general search engines continue to enhance their performance, there will be more specialized search engines emerging on the Internet or enterprise intranets. With speed and size still the main challenges with the search engines, relevance is becoming more and more important. AI techniques should be used in the development of search engines to provide more intelligent services. Some new trends include changing the \pull" nature of existing search engines to be \push-enabled". If SQL-like search languages are to be used, there will be a big challenge for distributed query optimization, routing, and scheduling also. References [1] A history of search engines. [2] Archie. [3] Askjeeves. [4] AltaVista. [5] Excite. [6] Hotbot. [7] Infoseek. [8] Inktomi. [9] LookSmart. [10] Lycos. [11] Metacrawler. [12] Northern Light. [13] One-stop shopping. [14] Search Engine Showdown. [15] Search Engine Watch. [16] L. Gravano, C.K. Chang, H. Garcia-Molina, C. Lagoze, A. Paepcke STARTS: Stanford Protocol Proposal for Internet Meta-Searching In Proceedings of the 1997 ACM SIGMOD International Conference On Management of Data, May [17] WebSEEk. [18] Yahoo!. 9

Directory Search Engines Searching the Yahoo Directory

Directory Search Engines Searching the Yahoo Directory Searching on the WWW Directory Oriented Search Engines Often looking for some specific information WWW has a growing collection of Search Engines to aid in locating information The Search Engines return