Crafting a Lightweight Search Engine

Size: px

Start display at page:

Download "Crafting a Lightweight Search Engine"

Myrtle Crawford
6 years ago
Views:

1 Crafting a Lightweight Search Engine Feng-Jen Yang Department of Mathematics and Information Sciences University of North Texas at Dallas Dallas, TX Abstract Web-based search is commonly perceived as a desirable functionality of web sites. Although there are several open source search engines that can be tailored and embedded in a web site for free, those engines tend to be general and may not be easy to adapt them into an efficient search engine within a particular application domain. In this paper, I presented an easy but practical alternative that can be followed by readers as a template to build their own lightweight search engines with only a few programming. Keywords: 1 Introduction Lightweight Search Engine, Web-Based Query. As more and more information is digitized and made available on the web, nowadays, online search is popularly used and commonly perceived as an essential functionality of a web site. Although there are several open source search engines, such as ASPSeek [1], Lucene [2], Namazu [3], mnogosearch [4], and WebGlimpse [5], available for web programmers to customize and fit into their own business operations, most of those free search engines are originated from general purposes and may not be easy to be reformed into special purpose search engines [6, 7]. Besides, embedding a big volume component into a small scope of operational web site may compromise the search performance due to the big consumption of CUP time. Another downside of being an open source search engine follower is the limitation on free technical support. Although some voluntary participants are willing to share their know-how through mailing list or online conferencing, most of the advanced consultants are still by payment. To this end, embedding an open source engine into a special purposed web site may not be the best choice despite the openness of its source code. With these concerns in mind, this hands-on project is crafted to provide an easy alternative for those in need of a web-based search engine but embedding an open source engine is not a practical practice. Instead of spending a great deal of time striving to understand a huge open source search engine and then tailor it to fit into a specific domain. We can actually perform a very similar functionality by crafting a lightweight search engine from the scratch that requires only a few programming. As an illustrative implementation of this lightweight search engine, I hypothetically confined the search domain into online books lookup and have the search engine perform partial-matched searches instead of exact-matched search to allow some fuzziness during the compare operations. For the purpose of quick prototyping, I am using only three technologies that most of the programmers are familiar with, namely the Microsoft Access, the Active Server Page (ASP) and the HyperText Markup Language (HTML). To ensure that most of the readers can follow and try out this implementation, the rest of this paper is written in an instructional and stepwise manner. 1.1 The Coherence of Search Operations An effective search engine is not only efficient in the lookup for items but, more importantly, also able to cope with human s partial, fuzzy, or incomplete memory about the keywords and other search criteria. Personally, I perceived this as the coherence between general users and the search operations. This expectation can be met by allowing users to perform searches based on their partial or fuzzy memory about the data they are looking for. Technically, this can be achieved by allowing partial and incomplete keywords to be used as search criteria [8]. Indeed, expecting users to spell out complete and correct keywords for the items they are looking for is neither necessary nor practical in real life, since most of the human memory can only be retained for a short term. Fuzziness and uncertainty caused by human s short term memory should be considered and incorporated into the design and implementation of search operations. 1.2 The Replacement for Web Crawler Unlike a typical search engine that is counting on a web crawler to glean and index information

automatically from the entire World Wide Web. For smaller domains of search, it is not necessary to be overwhelmed by world-wide information.

In this project, I adopted Microsoft Access as the platform for creating and maintaining the database that is running at the backend to support the web-based searches. 2.

For simplification, the data contents are minimized on purpose to have only one table that can be created by using the integrated development environment of Microsoft Access in the following steps: 1.

Within the same wizard, set the No field with the following properties: 3. Within the same wizard, set the Title field with the following properties: 4.

$accdb and save it to the folder at: 1 C:\inetpub\wwwroot\search Although the above database is very simplified, it does reflect the stereotype of data collections for the purpose of online search.$

2 automatically from the entire World Wide Web. For smaller domains of search, it is not necessary to be overwhelmed by world-wide information. Instead, it could be more efficient, if the crawler is replaced by a supportive database which can be maintained in a regular manner of database design and administration. In this project, I adopted Microsoft Access as the platform for creating and maintaining the database that is running at the backend to support the web-based searches. 2. The Supportive Database In this demonstrative implementation, a relational database is created and executed at the backend to support the web-based query operations. For simplification, the data contents are minimized on purpose to have only one table that can be created by using the integrated development environment of Microsoft Access in the following steps: 1. Start from a blank database and use the table design wizard to create a table named Book with the following schema, in which the No field is chosen to be the primary key: 2. Within the same wizard, set the No field with the following properties: 3. Within the same wizard, set the Title field with the following properties: 4. Within the same wizard, set the Author field with the following properties: 5. Open the Book table and enter the following hypothetical data: 6. Name the database as BookDB.accdb and save it to the folder at: 1 C:\inetpub\wwwroot\search Although the above database is very simplified, it does reflect the stereotype of data collections for the purpose of online search. The number of columns as well as the number of tables can be expended as needed. The whole database can also be further refined by performing a certain level of normalizations on the schema. 3. The User Interface Since the supportive database is hidden from general users but running at the backend, the search engine must provide a friendly frontend operational interface that allows users to specify their search criteria for their target data. The engine can then go on to look for those items that are partially matched to these criteria. In the context of text-based search, the search criteria are usually represented by a combination of keywords entered from users. To work with human s fuzzy and uncertain memory about their intended data, this search engine relaxes the restriction on search 1 To use a Windows PC as the web server, the Internet Information Server (IIS) component of Windows must be installed. After installing IIS, the inetpub folder and its subfolder wwwroot are created automatically. The programs have to create the search folder as a subfolder of wwwroot and save the database at this location.

3 criteria from a combination of exact keywords to a combination of patterns, i.e., a combination of partial keywords. The user interface of this query engine is shown in Figure 1. Below the user prompt, there is an input form in which two textboxes are arranged to take search criteria from users and two command buttons are used to either submit the search or reset the form. The source code of this interface is listed in Figure 2. This HTML file is named as index.htm and saved at the same location where the backend database is located, i.e., C:\inetpub\wwwroot\search. 4 The Pattern Matched Search Since human memory tends to be fuzzy, uncertain, or even partial. A real coherent search engine should be friendly enough to take these memory-caused factors into consideration [9, 10, 11]. To ensure a good alignment between the user s fuzzy memory and the engine s actual search operations, the following SQL syntax is applied to perform queries based on patternmatched comparison in which the LIKE operator is used in the WHERE clause to search for a specified pattern in the given column. The % character is used to define wildcards both before and after the pattern: SELECT * FROM table WHERE column LIKE %pattern% In this manner, a search criterion can be formed by incorporating both certain and uncertain memories from a user. As a result, only those rows of data containing the designated pattern in the given column are extracted. On the other hand, what are before and after the designated pattern in the given column are not concerned. The source code of these search operations is listed in Figure 3. This ASP program is name as search.asp and saved at the same location where the backend database and frontend user interface are located, i.e., C:\inetpubb\wwwroot\search. Figure 1. The User Interface <html> <h3>please enter your search criteria:</h3> <form action="search.asp" method="post" name="fromcriteria"> <table width="200"> <tr> <td>title:</td> <td><input name="txttitle" type="text" size="60" maxlength="60"></td> </tr> <tr> <td>author:</td> <td><input name="txtauthor" type="text" size="60" maxlength="60"></td> </tr> <tr> <td><input type="submit" value="submit"></td> <td><input type="reset" value="reset"></td> </tr> </table> </form> </html> Figure 2. The HTML Code of User Interface

4 <html> <% dim stream 'the input stream dim conn 'the connection object dim results 'the query result dim no, title, author 'the table data in a row 'create the input stream stream = "Provider=Microsoft.ACE.OLEDB.12.0;Data Source=" & Server.MapPath("BookDB.accdb") 'attach the input stream to the database set conn = server.createobject("adodb.connection") conn.open stream 'get search criteria (keywords) from user keytitle = request.form("txttitle") keyauthor = request.form("txtauthor") 'run the query and keep the result sql = "select * from Book where Title like '%" & keytitle & "%' And Author like '%" & keyauthor & "%'" set results = conn.execute(sql) %> <h3>the Search Result:</h3> <table border="1"> <tr><th>no</th><th>title</th><th>author</th></tr> <% 'display the results in a table while (not results.eof) 'get table data no = results("no") title = results("title") author = results("author") 'write a row response.write("<tr><td>" & no & "</td><td>" & title & "</td><td>" & author & "</td></tr>") 'go on to the next row results.movenext wend 'close the database conn.close Set conn = nothing set results = nothing %> </table> </html> Figure 3. The ASP Code of Search Operations

5. Sample Executions After complete afore mentioned implementations, this search engine is ready to operate at the follow web address: http://localhost/search

To look for all books having intro in the Title filed and having an in the Author field: 3. To look for all books having adv in the Title filed: 4.

To look for all books having rose in the Author filed: 6 Conclusion The modernization of communication technologies has turned the web into a global platform

5 5. Sample Executions After complete afore mentioned implementations, this search engine is ready to operate at the follow web address: It can be operated in a manner similar to the operations of most web-based search operations. Some sample executions are illustrated as follows: 1. To look for all books having intro in the Title filed and having an in the Author field: 3. To look for all books having adv in the Title filed: 4. The entire database can be browsed by leaving both criteria blank: 2. To look for all books having rose in the Author filed: 6 Conclusion The modernization of communication technologies has turned the web into a global platform where people can work across time and space. This digital revolution is making web-based search operations more vitally demanded than ever. As a result, an effective search engine is no longer just a facilitating feature, but, more realistically, it has become a core function.

6 In some way, the effectiveness of web search operations is empowered by the swiftness of response time as well as the easiness of operations. Although web programmers can tailor and embed open source search engines into their search domains, the customization from general purposes into special purposes may be time consuming and not a practical choice. This project demonstrated an easier alternative that most of the readers can follow to craft their own lightweight search engines to suit their own search domains. With this self-defined search engine, users can flexibly turn their fuzzy memories about the intended data into combinational fuzzy search criteria. By relaxing or tensing the fuzziness of search, users can easily extend up or narrow down the search results until the target data is located. This approach is practical, flexible and, more importantly, it requires only a few programming. [9] Jonas S Karlsson, J. and Kersten, M O- Storage: A Self Organizing Multi-Attribute Storage Technique for Very Large Main Memories, Proceedings of the Australasian Database Conference (ADC '00), City, State, pp [10] Ford, N., Miller, D. and Moss, N Web Search Strategies and Human Individual Differences: Cognitive and Demographic Factors, Internet attitudes, and approaches. Journal of the American Society for Information Science and Technology, Volume 56, Issue 7, pp [11] Ford, N., Miller, D. and Moss, N Web Search Strategies and Human Individual Differences: A Combined Analysis. Journal of the American Society for Information Science and Technology, Volume 56, Issue 7, pp References [1] The ASPSeek open source search engine is available at [2] The Lucene open source search engine is available at [3] The Namazu open source search engine is available at [4] The mnogosearch open source search engine is available at [5] The WebGlimpse open source search engine is available at [6] Micarelli, A., Gasparetti, F., Sciarrone, F. and Gauch S Personalized Search on The World Wide Web. Brusilovsky, P., Kobsa, A. and Nejdl, W. (Eds.), The Adaptive Web: Methods and Strategies of Web Personalization, Heidelberg, Germany: Springer, pp [7] Burke, R Hybrid Web Recommender Systems. Brusilovsky, P., Kobsa, A. and Nejdl, W. (Eds.), The Adaptive Web: Methods and Strategies of Web Personalization, Heidelberg, Germany: Springer, pp [8] Zou, L., Chen, L. and Özsu, M Distance- Join: Pattern Match Query in a Large Graph Database, Proceedings of the 35th International Conference on Very Large Data Bases (VLDB09), Lyon, France, pp

Microsoft Access Illustrated. Unit B: Building and Using Queries

Microsoft Access Illustrated. Unit B: Building and Using Queries Microsoft Access 2010- Illustrated Unit B: Building and Using Queries Objectives Use the Query Wizard Work with data in a query Use Query Design View Sort and find data (continued) Microsoft Office 2010-Illustrated