Web site Image database. Web site Video database. Web server. Meta-server Meta-search Agent. Meta-DB. Video query. Text query. Web client.

Transcription

1 (Published in WebNet 97: World Conference of the WWW, Internet and Intranet, Toronto, Canada, Octobor, 1997) WebView: A Multimedia Database Resource Integration and Search System over Web Deepak Murthy and Aidong Zhang Department of Computer Science State University of New York at Bualo Bualo, NY Abstract In recent years, the Internet and the Web have enabled us to distribute multimedia data such that it can be shared by people across the world in a transparent manner. The information presentation capabilities of a Web browser make it a powerful tool to achieve this objective. This involves the creation of individual multimedia repositories at s as well as search agents at the Web server which co-ordinate between clients and remote servers. However accessing multimedia information over the Web poses several challenges relating to the implementation of these systems in client-server environments when searching for images and other forms of data. In this paper, we describe the design and implementation of a system which achieves the above objective in an ecient manner. 1 Introduction Many applications involve accessing remote multimedia databases for the purpose of retrieving information which could be in the form of photographs, x-rays, scanned articles or satellite pictures. The access to such databases could be initiated by a client machine running a browser (such as Netscape or the Internet explorer) and allowing a typical user to issue image queries. These queries are processed by the browser and sent to the Web server which then selects the relevant target multimedia database site(s) and forwards the query to the database(s). The database involved searches for possible matches to the posed query and sends back the results to the Web server to be forwarded to the client. The selection of the most relevant database sites is based on the similarity of the query to the data in the sites. We have built such a system using distributed and object-oriented dynamic computing. Specically, we have used Java, a powerful platform independent language combining features of several object-oriented languages, to develop such a system. 1.1 Related Work The transformation of a user query into a form suitable for querying remote image databases is an interesting problem. Content-based querying involves more computation than text-based index lookup. Currently, no methods exist that allow Web-based access to image databases using content-based search. The closest methods that arrive are those that allow Web users to use pre-designed image databases by directly connecting to the specic site using the URL. In recent years, image database designs emphasize totally automatic methods of database creation as well as search for queries within sub-regions of the image rather than the entire image itself. While no site locator methods exist for handling image queries, methods exist for handling text databases where statistical approaches are used to record the frequency of occurrence of text keywords from known sites to construct an index of relevant sites for directing a query [GGMT94, GGM95]. The closest commercial systems being developed to access multiple remotely located multimedia databases are modeled 1

2 after traditional search engines such as Lycos, Alta Vista, etc., in that they use the context of the image description in the HTML pages at s to extract images. Also, most of the site selection work has been focused towards handling text information. For example, Web search engines such as those described above, currently create Web indices in their search engines by periodically scanning potential s and using the text information in their resident HTML pages. These systems employ statistical approaches to record the frequency of occurrence of text keywords from known sites to construct an index of relevant sites for directing a query [BDH + 94]. The above mentioned systems have not addressed the issue of integrating various multimedia database resources over the Web and supporting exible and transparent multimedia information search in such an environment. In designing such an integrated system, crucial issues like how the capability of each database (termed meta-data) can be described to enable a Web server to perform an intelligent selection of relevant databases in response to user queries need to be addressed. 1.2 CGI-based v/s Java-based Database Access A typical CGI-based database application on the Web consists of 3 components: a client running a Web browser, a HTTP server with a CGI program and a database server. A database query is initiated by sending a user request (using HTML forms) from the to the HTTP server. Upon receiving the user request, the HTTP server invokes the CGI program to assemble user input data into database specic query statements and sends them to the database server for processing. The query results are returned by the database server to the CGI program and then passed back to the through the HTTP server. The motive behind having a CGI program is to provide a generic interface between a HTTP server and user-dened server applications. There are some common problems associated with this approach which can be solved if a Java applet is used to perform the client actions. We will now discuss some of these problems here: Problem of Bottlenecks - Communication between a client and a database must always go through the HTTP server which becomes a bottleneck if there are a large number of users accessing the HTTP server simultaneously. For every request submitted by a or every response delivered by the database server, the HTTP server has to convert data from/to a HTML document thereby adding a lot of overhead to query processing. A Java applet can make its own network connection using sockets. Hence, it is possible to access databases from an applet directly, without going through the HTTP server, at the same time providing session-oriented communication. Lack of Eciency - For every query submitted to the database server through CGI, the server must perform the same logon and logout procedure even for subsequent queries submitted by the same user. This process consumes a lot of time and resources, especially when a large number of clients are accessing the database at the same time. Unlike the CGI-based approach, an applet can keep a connection open as long as it is alive and the user is in session thereby supporting interactive queries and complex database transactions. Lack of User Access Control - Each database query must be sent by a along with the necessary user ID and password. This is not a secure practice since request data is usually sent by the s to the HTTP servers in ASCII format. The user ID and passwords are embedded in the CGI program to access the database and control of user access is handled by the HTTP server. It is desirable to use existing user access control functions provided by the database server so that the same security and access privileges can be enforced across the network. Since there is no need to go through the HTTP server in the Java-based approach, none of the access control restrictions imposed by the HTTP server apply, thereby enabling user access control to be handled by the security mechanism of the database server. Lack of Presentation Graphics - A HTML document is static with no dynamic presentation graphic capabilities. Even though it is possible to make presentation graphics through generating 2

3 images in GIF or other graphic formats, this method is slow and time consuming because all the images have to be generated by a CGI program on the HTTP server side. A Java applet can handle sophisticated presentation graphics, thereby utilizing the full strength of distributed computing in an object-oriented environment. 1.3 Outline of this paper This paper will focus on the design and integration of multimedia databases for use in a Web-based environment. We will present the implementation strategies to use the rich information presentation capabilities of Web browsers for users to access information from various remote multimedia databases. A central metaserver which sits in the Web server is designed to support such an integration. The meta-server includes a meta-database and a meta-search agent. The meta-database organizes the meta-data of data housed at each remote multimedia database and their query capabilities. The meta-search agent controls the distribution of user queries to relevant database sites through the site server. 2 System Design and Architecture In this section, we present the design and architecture of our Web-based multimedia data retrieval system. The architecture of the system (termed WebView) is shown in Figure 1. The three main components are multimedia database systems at s, a meta-server consisting of a meta-search agent and a metadatabase at the Web server, as well as a set of Web applications at s. Using the Web as a medium to access multimedia sites involves accepting a user query in an acceptable form, selecting the appropriate sites, processing the query at the sites and presenting the results back to the user. Image database Text database Video database Image+text database Web server Meta-server Meta-search Agent Meta-DB Image query Text query Video query Text+image query Figure 1: WebView: the Web-based multimedia data management and retrieval system architecture. A Java database client can be created by constructing wrapper classes of vendor-specic client library functions and compiling the wrapper classes as a dynamic linked library. But due to security reasons, only stand-alone Java applications are able to connect to dynamic linked libraries. A Java applet on the other hand does not have permission to access any les on the local le system. The only way for an applet to access a remote server is through Java's socket API. There are two methods to construct a Java applet as a client for accessing a remote database server: Standard 2-tier architecture - involves only the client and the remote database server. All the client functions are implemented in Java. This requires extensive programming eort because a Java client must be implemented at the protocol level for vendor-specic databases. Improved 3-tier architecture - involves implementing a stand-alone Java server (a Java application running in the Web server) which is used as a gateway. The gateway passes the result and response 3

4 WEB CLIENT [Java applet] GATEWAY [Java] [Application] DATABASE [O2, Oracle,..] Figure 2: Java client-server model. messages between the applet and the remote database server as shown in Figure 2. This is more ecient in that the gateway server can be created as a stand-alone Java application with native client library functions wrapped in Java classes. It communicates as a server with the Java applet at one end while accessing the database server at the other end. We shall follow this approach since this is more exible and portable. Using Java, a new strategy to develop and maintain client/server applications can be established. Instead of installing client packages on users' computers and upgrading them for every new release, a user can download the client software in the form of a Java applet from the Web. 2.1 Web Clients and the User Interface The is a typical user using a standard browser to invoke an HTML document which brings up a Java applet. The applet has the user interface implemented using the rich graphical features supported by the Java API and is used to obtain the user query. The user query is an image or a portion of an image and is accepted from the user interactively in that the user is allowed to display as well as select relevant portions of the image. The image query is then forwarded to the Web server which further routes it to the relevant database site(s). The nal query results (names of the retrieved images) along with the names of the corresponding databases are displayed to the user and the image desired by the user (selected interactively) can be directly retrieved from the specic database using an associated CGI script. 2.2 The Meta-server The meta-database records information needed for database site selection. The meta-database is present in the Web server and organizes the information about remote database sites based on the type of queries they support and the types of media data they house. Given a set of databases at various sites, an initial meta-database is constructed from pre-dened templates (or icons) returned by the individual databases. Such templates contain meta-data about the data in the database, including the type of media data housed, expected query form, specialized algorithms supported and statistical data associated with each multimedia template. Data such as monetary cost and latency of database sites can also be stored to enable early pruning of costly sites. These templates can be periodically updated by the meta-search agent and relayed to component databases. The initial categorization of databases in the meta-database is used to direct queries to relevant sites. A record of recently returned responses and the associated queries is maintained at the server to avoid redundant searches. The meta-database is organized into a hierarchical structure. Database sites are grouped according to template (or icon) classes/subclasses, including keywords, texture pattern, color pattern, and shape pattern. All groupings are based on the degree of similarity among images. The information needed for group assignment is supplied at the time the database registers with the meta-search agent. A database site can belong to one or more template group. Probability data for each database site with respect to templates is also supplied at the time of database registration. Whenever a user query comes in, the meta-server must selectively forward it to the relevant databases (those having the highest potential to nd images matching the query) for ecient searching. In order to achieve the above objective, templates consisting of sample icons are created to represent dierent classes of images corresponding to the dierent databases. The information regarding these templates is stored in the meta-database. When a query comes in, the meta-server runs a local search to come up with matched templates. It then calculates the potential for each remote database by using the statistical information of 4

5 the database and combining it with the similarity between the query and matched templates. This potential is used as a weight to rank the databases so that the top N potential databases can receive the query. The remote servers housing these databases then search for the query and return the matching images to the user. The functionality of the meta-server is split into two modules. The rst module, register module is responsible for collecting and updating the information about the remote databases. The second module, selection module is responsible for making the decisions based on the information in the meta-database and is also responsible for routing the query to the remote databases. The register module - The purpose of the register module is to collect and update information about the remote databases. It takes two conguration les as input. The template conguration le species the location of the templates and the location where other template specic information should be stored. The database conguration le contains location data (including host and port specications) of the remote databases. At startup, the register module reads the information from the conguration les and sends the image data of each template to all the remote databases. The remote databases search for images matching these templates and return statistical data corresponding to the templates. The register module then stores the data in a separate le for each template. This constitutes the meta-database. The selection module - Reecting the hierarchical meta-data organization of database sites in the meta-database, the search process begins by comparing a user query with the templates. One or more query class categories may be derived by selecting the closest templates. The mapping of each user query to templates is determined by a set of rules. This may require nding a semantic equivalence between the user's description and the template indices for the meta-database or features extracted from the query data pattern (for visual queries). The meta-search agent then makes a decision on the relevant database sites to be searched. If the initial selection does not suciently prune the database sites, the meta-search agent can perform a further selection based on user feedback before distributing to relevant database sites. Also, if no template can be found for the query data pattern, the meta-server has the option of creating new templates. Under this scenario, all databases will be asked to provide information pertaining to the new templates. This module also takes two conguration les similar to those in the register module as inputs. The actual meta-database as described in the register module above is designed in the form of a hash table which contains an entry corresponding to each template. This entry includes information like the number of matches, mean, variance and other statistical data which is gathered by a register-server program deploying data corresponding to each template data to all the database servers (indicated by the conguration les) at system startup time and also at regular intervals. Database information is also stored in other tables. Every database server has an information server program running which returns the statistical data corresponding to the templates. The meta-server stores and updates this information in the hash table. Upon receiving a user query, a local search is performed to nd the matching templates, which are then used to index into the hash table for information about the templates at various database sites. This information (along with the similarity of the query to the matching template) helps the meta-server in ranking the databases based on their potential and the query is then forwarded to the top N ranked databases. Both register and selection modules support dynamic conguration. Databases and templates can be added and modied dynamically and their new status indicated by the conguration les. A description of the meta-database and selection approaches can be found in [CMZSM97]. 2.3 Other Design Issues System Issues - The environment is multi-threaded in order to support and follow up on queries from multiple users. Each client is handled from startup time to termination by a separate thread with the parent synchronizing between them. Java's socket API is used to provide connectivity between the applet, Web server and remote servers. 5

6 User Interaction - The list of selected database sites and matching image names is sent back to the user allowing him/her to select the desired database/image. The desired images are fetched directly from the remote database by converting the address to a URL and retrieving it over the Web. The user is actively involved in the decision making process thereby resulting in better results. Image Retrieval - The traditional way of retrieving images from databases is to assign text annotations to image data. This method is insucient to unambiguously describe the image content and is also time consuming considering the task of manually tagging the images. Hence we retrieve images based on the content thereby resulting in qualitatively better matches. Such a technique has been developed on normal le systems and is being integrated with databases like O2 and Oracle. An Information server is a program which runs on the remote server and generates information pertaining to each template (like number of matching images, mean, variance, etc.,) which is used to create the meta-database. A Database server is a program running on the remote image database. This program performs content based image retrieval, taking the query icon image data passed by the application running on the Web server, computing its feature vectors and using them to search the database to retrieve the matching images. Resource handling - Once the session is complete, the applet sends a connection teardown message to the Web server. This request is also forwarded to the remote database servers thereby initiating cleaning operations which involve closing down the sockets between the applet client-web server and Web server-database server, and also removing redundant information relating to the retrieved images thereby freeing valuable resources. 3 Conclusion We have presented a design approach for the integration of multimedia databases located at remote sites. This approach includes the creation of multimedia databases, the meta-database, and the meta-search agent. A prototype system to support Web-based multimedia information retrieval has been implemented using Java. Preliminary experiments have yielded satisfactory results. The meta-database must be dynamically updated to prevent the redundancy of meta-data recorded earlier due to frequent updations to databases. Renement of an existing meta-database in response to component database updates is a dicult task. Since it is impractical to require the databases to report their status every time they are updated, the renement of the meta-database can only be based on careful evaluation and validation of the query results. This has proven to be non-trivial, since we do not know whether the results are indeed relevant to the query without user input. We will pursue this research as part of the future work involved. The current system can be extended to include more multimedia databases containing dierent kinds of data thereby creating a network of multimedia information systems transparently accessible from anywhere in the world. References [BDH + 94] M. Bowman, P. Danzig, D. Hardy, U. Manber, and M. Scwartz. Harvest: A scalable, customizable discovery and access system. Technical Report CU-CS732-94, Department of Computer Science, University of Colorado-Boulder, [CMZSM97] W. Chang, D. Murthy, A. Zhang, and T. Syeda-Mahmood. Metadatabase and Search Agent for Multimedia Database Access over Internet. In the Fourth IEEE International Conference on Multimedia Computing and Systems (ICMCS'97), Ottawa, Canada, June [GGM95] L. Gavarno and H. Garcia-Molina. Generalizing Gloss to Vector-Space Databases and Broker Hierachies. In Proceedings of the 21st International Conference on Very Large Data Bases, pages 78{89,

7 [GGMT94] L. Gavarno, H. Garcia-Molina, and A. Tomasic. The Eectiveness of Gloss for the Text Database Discovery Problems. In Proceedings of the ACM SIGMOD International Conference on Management of Data, Minneapolis, May