AN ARCHITECTURE FOR FAST AJAX ENABLED WEB FORMS. Bryon Chan

Transcription

1 AN ARCHITECTURE FOR FAST AJAX ENABLED WEB FORMS Bryon Chan Department of Electrical and Computer Engineering University of Auckland, Auckland, New Zealand Abstract This project aims to improve the usability of online forms by creating a framework that will facilitate text field input validation before the form is submitted. This is achieved by creating a lightweight combo box component that can be updated dynamically by the server using a database as the data store. A key characteristic of the advanced combo box is the ability to dynamically change the menu according to text input. The emerging web technology AJAX will be used as the technology behind this. This particular implementation will include full text search functionality of the list and also auto-complete for the text field. The combo box must integrate easily with current online form systems and require minimal setup and maintenance by a web developer. At the backend, a search system was implemented that enabled fast information retrieval style searches to be performed on database data. Using the architecture presented, a Google Suggest-like application can be implemented. 1. Introduction HTML forms are used throughout the computing world to collect information in a structured way. Not only are they used throughout the World Wide Web, but also by many enterprise systems which use browser user interfaces as the primary mode of access to the system. Many of these systems require a higher degree of correctness in the information collected than can be achieved by simply relying on the user to fill in the form correctly. Therefore UI designers use a number of tactics to guide a user to enter valid information, e.g. flashing a red label beside a text field to indicate an error, popping up a tool tip describing the error that was made. Web developers however are limited in their options, because the traditional submit response model of HTTP means that a user has to submit the form before the error is detected. This usually means that the user will have to wait for the new page to load, return to where he made the mistake, fix it and submit again. Most Internet users would have experienced the frustration or at least inconvenience of having to interact with a form in this way. Intuitively, immediate validation and feedback of user input would be a better solution. This paper presents a system that enables dynamic (i.e. pre-submit) validation of user input. The system will make use of the XMLHTTPRequest object in JavaScript, which are the two main technologies used in AJAX, which means that the only browser requirements are support for XMLHTTPRequest and that JavaScript is turned on. This means that over 80% of users will meet the requirements without having to download plugins, or applets, etc. A key deliverable for this project is the advanced combo box. A combo box combines a text field and a dropdown menu and functionally subsumes the INPUT and SELECT components. Other HTML form components such as TEXTAREA and RADIO buttons were out of the scope of the project, because the focus was single-line text input fields. The advanced combo box will enable dynamic validation through AJAX, i.e. validate as the user types, and also allow non-trivial validation by connecting to the server. The responsiveness and therefore usability of the combo box was a major focus of the project. To meet the client s (Peace Software) particular needs, the project team decided to focus on search field input validation, which would be used in Peace s customer search module. Search field input validation can be described as giving a user feedback about their query, e.g. possible completions, similar terms, spelling mistakes and number of documents that match, before the user actually conducts the search. This project focuses on returning possible completions. A recent example of this type of search field input validation is Google Suggest, which gives users feedback on their query, while they are typing. Google Suggest is described in more detail in Section 2. The evolution of this project over it s course has led to a clear separation of it s components. That is, the outputs of this project can be separated into two parts: which we call the interface and the architecture. The interface is the combo box component written in JavaScript that uses the XMLHTTPRequest object to communicate with the server. This enables page updates to occur without needing to refresh the entire page, which is a significant usability improvement over the traditional submit-response interaction style. The architecture is a database search system, which enables data from a database to be searched using an IR style keyword search. These parts are combined to make a database search system with a combo box interface,

2 which essentially searches as you type. However, the reason for highlighting this separation is to highlight the reusability and modularity that has been built into these parts. For example, taken on it s own, the architecture is a fully functional database search system. At the same time, the combo box is a fully modularised website development component that can be used and extended independently of the search system. For example, the combo box can be used to deliver error messages as part of a form field validation system that does not use a database as a backend, as shown in Fig Example Applications A search system based on the architecture presented could be used in several applications to improve performance and usability Customer search One example is to use it in customer search applications. The search system would be used in place of the existing customer search form. This application of the system would enable a customer service representative to locate customer data faster. This is the intended application that led to interest in the project from Peace Software Google Suggest Google Suggest 1 presents a standard Google search interface with the addition of an auto-completing function. The auto-complete returns a list of suggested completions of what the user has typed, while they are typing. Additionally, the suggestions are ordered by overall popularity of those queries. The architecture presented allows such an application to be built using a database as the data store, combining the capabilities of a database system with fast IR search speeds and an interactive interface. Figure 2 Google Suggest 3. Outline of paper Section 4 describes related work in the fields of web form field validation, web combo boxes and keyword searching in databases. Section 5 discusses the results of the investigation into appropriate search technologies to conduct fast prefix searching. Section 6 describes the architecture. Section 7 describes the interface. Section 8 describes the background of the technologies used, including AJAX, information retrieval and ETL. Finally, conclusions are presented in Section Related Work This section describes the related work done in the three main fields of interest: form field validation, web combo boxes and database keyword searching Form field validation There are two ways of validating user input, from the client side, commonly by using JavaScript [1, 2], or from the server side, which occurs after submission. JavaScript gives fast, dynamic but trivial validation. For example, checking that an address only has symbol. Server side validation is usually slower, requiring a post back and is less usable for a user (e.g. having to backtrack to where he made a mistake after the error page is returned). In addition to this, client side validation can be easily circumvented and hence requires back-up server side validation. In any case, server side validation is needed for users who choose to switch off JavaScript. Given the current problem of slow server side validation, this project provides a form field validation system that provides immediate feedback and support to the user based on data in a database. Figure 1 Using the combo box to deliver error messages. 1 Google Suggest -

3 4.2. Web combo boxes JavaScript combo boxes A combo box is a combination of a text field and a dropdown menu. This gives users the freedom of inputting text themselves or selecting from a static list of pre-defined options from the dropdown menu. There is no native HTML combo box, however web developers have used JavaScript to couple an INPUT element and a SELECT element to achieve the same functionality AJAX Enabled combo boxes The most advanced example of an existing AJAX enabled combo box is the Web ComboBox 2 made by EBA. Key features of the Web ComboBox include autocomplete, paging, events and data binding for easy development. The advanced combo box that was developed for this project is different because the focus was on the backend search speed Database keyword searching Recent work has been done by researchers who recognize that the need for knowledge of the database schema and knowledge of the querying language represent a barrier to any user that has an interest in the data, but without the expertise to access it. The traditional solution is to implement search forms. However for a large number of relations, this can be cumbersome and expensive to implement. Systems such as DISCOVER [3], DBXplorer [4] and BANKS [5] enables keyword searches over an entire collection of tables. They share a similar approach: given a set of keywords, find the set of tuples from the tables where one or more keywords exist. These tuples become the nodes of a graph and primary to foreign key relationships between the tables (where the tuples were located) become the edges, so that all subgraphs that contain a set of nodes where all the keywords are present is a search hit. [6] presents improvements to DISCOVER s relevance ranking and performance by exploiting the IR characteristic that generally only the most relevant matches need to be processed. These approaches avoid creating redundancy in the database and performance is not the main concern as in our approach. Several databases include a special full text indexing option, so that textual data can be searched in an IR style. Examples are Oracle Text [7], MySQL 1 JavaScript combo boxes EBA Web ComboBox V3 - webdevelopment/codeandcomponents/ebawebcombov3/defaul t.htm FULLTEXT, SQL Server Full Text Indexing [8]. IR functionality includes stop words, relevance rankings, phrase searching and proximity searching. These built in full text indexes have the advantage of being closely integrated to the data. In fact, queries are specified using extensions to SQL. [9] makes use of these built in index engines to index data offline to enable fast keyword searches in a system called EKSO. The approach is similar to the one described in this paper, however our approach takes processing outside of the database and handles the synchronisation required, whereas EKSO uses the database vendor s integrated text search tool. We discuss the advantages of our approach in section Search Technology To successfully implement the search field functionality of the advanced combo box, an appropriate search technology must be found. This section describes the evaluation of search technologies, which enables search speeds that are sufficiently fast to facilitate search field input validation. To be more specific, the problem is that of prefix searching, i.e. given an input string: find all words that start with the input string. Two candidate data structures were shortlisted, these were: inverted indexes and tries Inverted indexes An inverted index is very similar to the index that one can find in the back of books. In such indexes, words that are determined to be important are listed in alphabetical order with the page number where the words are found. Instead of writing new code to implement an inverted index, Lucene, a suitable open source library was used. Lucene is a full featured searching and indexing library hosted by Apache. Lucene implements the prefix-searching feature as a pre-processing stage that rewrites the query to include all the expansions of the prefix. E.g. pat* will expand to patrick OR patricia OR patterson, etc. In a previous project, it was observed that prefix searches in Lucene was particularly slow, not meeting the performance requirement for that project. To improve this performance a prefix-indexing technique was used. For example, to index the word AJAX : instead of just indexing AJAX, the prefixes A, AJ, AJA were indexed as well. The results of a preliminary experiment show that a performance gain is achieved by a space-time trade-off, i.e. the result is a larger index, but faster search times Tries The idea of using tries came from initial work done on suffix trees. Suffix trees [10] store every suffix (i.e. a substring starting from any position of the string to the

4 end of the string) of a string in a tree. Search complexity is proportional to the length of the query. Tree building is proportional to the length of the string. However, use of a suffix tree in our application would be inefficient as we assume the user to search from the beginning of the search term. E.g. when searching for christopher, a user would type from the beginning of the word, not in the middle, i.e. chris.., not tophe... The need to reduce the inefficiency of storing every suffix lead the research towards a data structure called a trie. A trie is a tree structure where each node is a prefix or a word and each edge is a letter. Therefore every word in a subtree has the same prefix. Fig. 3 illustrates this. This option was initially considered because a tree structure is intuitively well matched to prefix searching. Imagine each letter typed as traversing from the root down the tree, narrowing the search space down logarithmically. Inverted indexes on the other hand, treat each search as a new search through the index. Tries are fast data structures, however the obstacle to efficient implementations is the large space requirement. Techniques exist to increase space efficiency, including ternary search trees [11] and PATRICIA trees [12]. The prototype developed used ternary search trees to get acceptable space efficiency; however using a PATRICIA tree could have saved more space. individual searches, but produced unacceptably high search times when put under load Chosen search technology: Lucene Lucene was chosen for two main reasons: First, superior search performance in a multithreaded environment. Second, there are many functional requirements that would take too long to implement on the trie prototype that has already been implemented in Lucene, including thread safety, delete operations and sorting/ranking of results. Time (ms) Trie versus Lucene performance (10 search threads, "people", 20 searches/second) Trie Lucene Search Figure 4 Lucene vs. Trie search performance Figure 3 A trie containing the words ajax, arch, arc and art Lucene vs. Trie performance A trie-based search prototype was developed to compare against Lucene as the search implementation for the search system. Lucene started with some high search times on initialisation, but stayed below the desired 100ms threshold for the rest of the searches. The trie prototype performed faster than Lucene with 6. Architecture One of the challenges of implementing database systems is maintaining satisfactory query performance while tending to other database issues at the same time, e.g. transaction management, data integrity, security. A tactic frequently used where performance is needed, is to trade space with time, e.g. by combining fields from different tables into a single table to avoid joins. Another tactic is to perform some query processing in advance, e.g. by using database indexes to pre-sort on certain fields. Lucene s inverted index combines both tactics by creating a full text index that exists outside of the DBMS, which allows search performance that is largely unhindered by other database operations. This solution allows speeds that facilitate search field input validation. The development of Lucene s prefix searching independently of the database illustrates the advantage of implementing an IR access layer on top of a database that can be tuned for specific requirements without needing to consider possible effects on the operational database. The transformation of structured data into flat text for the purpose of searching may seem counterintuitive: consider the query nelson as opposed to

5 firstname= nelson. The user s knowledge that nelson is a first name cannot be expressed in an IR query. In an ideal world, searches on structured data are best handled by a structured query language. However, this structure can come at the cost of performance. Consider any query that requires a join between multiple tables. As datasets get larger and searches increase in frequency, the load imposed on the database becomes a risk not just to the performance of that query, but also to the overall performance of the database. Another reason such an approach would seem unreasonable is due to the problems associated with synchronizing an external index to data in a database. We discuss ways to manage synchronization that is event driven, as opposed to a periodic scanning method, such as those employed by IR crawlers. The system architecture is shown in Fig. 5. Simply put, the back-end is responsible for creating and updating the full text index and the front-end is responsible for allowing multiple users to search the index in an IR-style. The back-end is comprised of the database and an Extract, Transform and Load 1 layer (ETL), which has an IR indexing service. The front end is made up of the index structure created by the indexing service and a search application that searches over it Indexing Service Figure 6 Address relations The Indexing Service is a full text indexer that initially builds and subsequently synchronises the Full Text Index to the database. The Indexing Service is implemented as a transformation process in the ETL layer. Aside from adding records to the index, this service also has the responsibility of transforming the normalised data into a flat form suitable for IR indexing. For example, given a set of relations containing address data as shown in Fig. 6. The Indexing Service can be configured to extract the information, by using a SELECT query as shown in Fig. 7. Each row in the result set is added to the index as a document, which is analogous to GOOGLE crawling and indexing a webpage on the Internet. Each document has a corresponding record in the result set, so the key is also indexed. Using the same Internet analogy, this is the URL, which enables searchers to actually navigate to the original data. Figure 5 Architecture for fast database search 6.1. Back-end architecture The back-end architecture proposed is an Extract, Transform and Load (ETL) solution. ETL is a wellknown data warehousing technology. By integrating ETL into our architecture, we are leveraging existing ETL tools and ideas in order to enhance our architecture. ETL extracts information from the database, then transforms the information so that it can be loaded into the destination, in this case the index. ETL is designed to be able to extract data from multiple databases, which could be using different systems, e.g. Oracle, SQL Server. 1 The ETA on ETL www2.cio.com/analyst/report2218.htm Figure 7 Transformation of a record into a document. Indexing the entire desired dataset is a time and resource consuming process. To minimise this, the

6 Indexing Service should be able to index incrementally, i.e. adding or removing a document from the index without rebuilding the entire index again, some indexing tools allow two indexes to be merged, allowing the current index to be merged with an index of subsequent changes at a smaller cost than an entire rebuild. This is one of the reasons Lucene was chosen ahead of other full text indexers Synchronisation The major issue of implementing external indexes of database data is how to handle updates to the source data. A solution is needed that keeps the data in the index identical to the database data, without using too many database resources. For example, consider a customer whose details had just been inserted into the database, and who soon after makes a call to the call centre. The call centre personnel need access to this customer s records immediately. The information that is required by the Indexing Service to reflect the changes to the index are: For an insert, The values of the new record. For an update or a delete, The key, as described in the previous subsection. The procedure that updates the index simply deletes the document that matches the key and re-indexes the document. Therefore, the Indexing Service handles updating the index and the only thing the backend is responsible for is passing the required information on, whenever a change to data is made. Another important requirement is that synchronisation does not negatively impact on normal database operation. Three distinct ways of achieving this are identified. These are logs, triggers and by using events and listeners in the data access code. Database logs contain information about all updates to a database. Most DBMSs already have logs for backup and recovery and for auditing purposes. The disadvantage of using logs is that they are not accessible immediately after a database change and they are database specific. Triggers are database event driven procedures that are often used to enforce business rules and maintain data integrity in DBMSs. Triggers are configured to execute whenever an insert, delete or update operation is done on a table. A trigger has access to all the information needed to update the index. Finally, using events and listeners in code. This approach is exemplified by the use of Hibernate. Hibernate maps relational database tables onto Java objects. In other words, the database becomes just a persistence layer. In this case, the information needed to update the index can be obtained as part of the object oriented code, whenever a customer object is being inserted, or deleted or updated Front-end The Search Engine allows users to search the data using an IR style query. This means the user can include any piece of relevant information into a single field without having to qualify it with the type of the information. The advanced combo box was implemented as the front end to this search system. This allowed real time, search-as-you-type access to the database. This is described in more detail in section Advantages of external indexes The architecture allows multiple Search Engines to be deployed. This means that search load can be spread out onto as many Search Engines as are needed to meet performance requirements. Also, multiple Search Engines can be deployed at locations geographically separate from the central data repository, e.g. call centres. This would further improve search times by eliminating delays caused by network latency. Lastly, a significant load is taken off the database, allowing the rest of the system to speed up. Figure 8 Implementation of the architecture using open source tools and applied to a typical system architecture 6.6. Implementation in a typical architecture A typical architecture is shown in Fig. 8 with the implementation of our search system using open source software. The initial creation of the index is handled by

7 Clover. Once the Index is built, the Hibernate objects will fire an event when data is changed. The event contains information such as what type of change and which document was changed. Clover uses this event as the input for an index-updating transformation, which goes on to update the Lucene index. While the update is occurring, Lucene allows users to still search on the same index. However, for changes to be visible, users will need to get a new IndexSearcher object, this is a small inconvenience as several IndexSearchers are instantiated and destroyed over the course of a search, e.g. every time a user refreshes the search page. Fig. 8 also shows the separation of the Search Engine from the database. One of the advantages of this is the ability to modify and extend the Search Engine without worrying about negative consequences to other systems. In this case, the Search Engine has been extended to include search-as-you-type interface, i.e. the advanced combo box. 7. Interface This section describes the advanced combo box, which is the interface component of the project. The solution we present uses data from a database to evaluate a query. The most common use of this would be to autocomplete queries, e.g. if a user types an, the system would return possible completions like andrew, andrea, annie, etc. A combo box is an input component that allows the user to fill in a text field either by typing in directly or by choosing an option from the menu. There is currently no native combo box component in HTML, however there are many combo boxes that use JavaScript to combine the INPUT and SELECT components to achieve the same functionality. The menu is static, i.e. it s contents are hardcoded in the page source. The advanced combo box is a web component written in JavaScript. It is AJAX enabled, which gives it the ability to dynamically update the information that it presents, i.e. without requiring a page reload. This improves interaction by allowing behind the scenes calls to the server. The subsequent updating of specific page content is far more unobtrusive and more usable than an entire page reload. The features of the advanced combo box include: Results displayed in a database like table Auto-complete Hit highlighting Number of results Paging Mouse and keyboard navigation Client side cache of search results Time out mechanism to minimise server calls Figure 9 Features of the advanced combo box AJAX 8. Background of Technologies AJAX stands for Asynchronous JavaScript and XML. It is a description of a web development technique that makes use of the XMLHTTPRequest object to improve user interaction on a web page. Applications that have been built on AJAX and contributed to the current interest in AJAX are Gmail and Google Suggest. Fig. 10 shows the way AJAX works. The user causes an event on the web page, for example, a key press or a button press, which executes the Javascript code that gets an XMLHTTPRequest object and sends an HTTP request to a server object. The HTTP request contains the parameters of the request, e.g. in the case of the advanced combo box, the query is sent. The server object then returns data, which doesn t necessarily have to be XML, which is processed using JavaScript and changes are made to the original web page as necessary. The important thing to note is the server does not respond with a new page that the browser displays, instead the server returns data, that is used to change the original page. AJAX has some disadvantages, primarily lack of support from browsers. However, browser makers recognise AJAX as an important technology those who have not already supporting AJAX are doing so in future versions of their software. Figure 10 How AJAX works

8 8.2. Information Retrieval The goal of information retrieval is to enable people to find information that they are looking for. The information may be stored on a page in a book or on a page in a website. In turn the book may be in a library and the website may be on the World Wide Web. If the library was well organised and relatively small, a good librarian could find all the books that were relevant to your spoken query. But on a larger scale, vast quantities of information can become searchable only by harnessing the power of computers Apache Lucene Lucene 1 is part of the Apache Open Source project. It is a Java library for full text searching functionality. This refers to the fact that Lucene only provides functionality to index Lucene Documents. When the index has been built, it provides an API that returns references to matching Documents. Lucene has an inbuilt query syntax that allows many features including Boolean search, wildcard searching, fuzzy searching, range searching, proximity searching etc. It is up to the user to convert their information into Lucene Documents and display the results ETL The backend is based on an ETL application. ETL is a data integration technology. It is a response to the fact that valuable data can be located in different sources and combining the data into one destination, will facilitate data analysis. Therefore ETL tools are designed to extract, clean and prepare data from multiple sources, then load them into the destination usually a database. Two ETL tools, Clover.ETL and Enhydra Octopus were evaluated. Clover was chosen because of the quality of coding compared to Octopus. This made it easier to understand and therefore extend. 9. Conclusions In this paper we presented an architecture for providing search-as-you-type access to database data. This was achieved by having a fast database search system based on information retrieval technology. The interface built on top of this system, the advanced combo box, made use of the speed to provide real time feedback to users, in the form of a dynamically updated menu showing the results. This represents a significant usability improvement to current database search forms, which is our main aim. The advanced combo box contributes to the more general problem of form field validation by providing a convenient way of receiving data from the input field and displaying the server response in the menu. The technology behind the advanced combo box, AJAX is likely to remain popular and become more widely accepted in the future. Acknowledgements The student would like to thank the project partner Jason Flaherty and the supervisor Dr. Gerald Weber for their parts in making this project successful. Also, thank you to the second supervisor Ms. Su Tang for her feedback during the project. 10. References [1] W. Bontrager. JavaScript Form Input Validation and Correction. Retrieved May 6, 2005 from ipt/javascript-form-validation-correction /page1.html, [2] Form Validation. Retrieved May 6, 2005 from dation.html, [3] V. Hristidis and Y. Papakonstantinou. DISCOVER: Keyword search in relational databases. In Proc. of VLDB, [4] S. Agrawal, S. Chaudhuri, and G. Das. DBXplorer: A system for keyword-based search over relational databases. In Proc. of ICDE, [5] Bhalotia, A. Hulgeri, C. Nakhe, S. Chakrabarti, and S. Sudarshan. Keyword searching and browsing in databases using BANKS. In Proc. of ICDE, [6] V. Hristidis, L. Gravano, and Y. Papakonstantinou. Efficient IR-style keyword search over relational Databases. In Proc. of VLDB, [7] P. Dixon. Basics of Oracle Text retrieval. IEEE Data Engineering Bulletin, 24(4), [8] J. Hamilton and T. Nayak. Microsoft SQL Server full-text search. IEEE Data Engineering Bulletin, 24(4), [9] Q. Su, J. Widom. Indexing relational database content offline for efficient keyword-based search. In Proc. of IDEAS, [10] P. Weiner. Linear Pattern Matching Algorithms. Proc. 14th IEEE Annual Symp. on Switching and Automata Theory, pp1-11, [11] J. Bentley and R. Sedgewick. Fast algorithms for sorting and searching strings. In Eighth Annual ACM-SIAM Symposium on Discrete Algorithms (1997), SIAM Pres, [12] D. R. Morrison. PATRICIA - Practical Algorithm to Retrieve Information Coded in Alphanumeric. Jrnl. of the ACM, 15(4) pp , Oct Apache Lucene -

9 Department of Electrical & Computer Engineering Part 4 Project Report 2005 An Architecture for Fast AJAX enabled Web Forms Final Report Author: Bryon Chan Date: 11 th September 2005 Project Partner: Jason Flaherty Supervisor: Dr. Gerald Weber

10 Declaration of Originality This report is my own unaided work and was not copied from nor written in collaboration with any other person. Signature: