Efficient Content-Based Indexing of Large Image Databases

Transcription

1 Efficient Content-Based Indexing of Large Image Databases ESSAM A. EL-KWAE University of Nort Carolina at Carlotte and MANSUR R. KABUKA University of Miami Large image databases ave emerged in various applications in recent years. A prime requisite of tese databases is te means by wic teir contents can be indexed and retrieved. A multilevel signature file called te Two Signature Multi-Level Signature File (SMLSF) is introduced as an efficient access structure for large image databases. Te SMLSF encodes image information into binary signatures and creates a tree structure tat can be efficiently searced to satisfy a user s query. Two types of signatures are generated. Type I signatures are used at all tree levels except te leaf level and are based only on te domain objects included in te image. Type II signatures, on te oter and, are stored at te leaf level and are based on te included domain objects and teir spatial relationsips. Te SMLSF was compared analytically to existing signature file tecniques. Te SMLSF significantly reduces te storage requirements; te index structure can answer more queries; and te SMLSF performance significantly improves over current tecniques. Bot storage reduction and performance improvement increase wit te number of objects per image and te number of images in te database. For an example large image databases, a storage reduction of 78% may be acieved wile te performance improvement may reac 98%. Categories and Subject Descriptors: H.3.3 [Information Storage and Retrieval]: Information Searc and Retrieval Query formulation; Query formulation General Terms: Algoritms, Documentation Additional Key Words and Prases: Content analysis and indexing, document managing, image databases, index generation, multimedia databases Autors addresses: E. A. El-Kwae, Department of Computer Science, University of Nort Carolina at Carlotte, 90 University City Boulevard, Carlotte, NC 83; eelkwae@uncc.edu; M. R. Kabuka, Department of Electrical and Computer Engineering, University of Miami, 5 Memorial Drive, Room 406, Coral Gables, FL 3346; kabuka@cmiami.med.miami.edu. Permission to make digital / ard copy of part or all of tis work for personal or classroom use is granted witout fee provided tat te copies are not made or distributed for profit or commercial advantage, te copyrigt notice, te title of te publication, and its date appear, and notice is given tat copying is by permission of te ACM, Inc. To copy oterwise, to republis, to post on servers, or to redistribute to lists, requires prior specific permission and / or a fee. 000 ACM /00/ $5.00 ACM Transactions on Information Systems, Vol. 8, No., April 000, Pages 7 0.

2 7 E. A. El-Kwae and M. R. Kabuka. INTRODUCTION Tere as been an increasing interest in multimedia tecnology in recent years. In particular, image and video databases received enormous attention. Te application areas based on images include, but are not limited to, Medical Imaging, Medical Information Systems, Document Image Processing, Office Information Systems, Remote Sensing, Management of Eart Resources, Geograpic Information Systems (GIS), Cartograpic Modeling, Mapping and Land Information Systems, Robotics, Interactive Computer- Aided Design, and Computer-Aided Manufacturing Systems. Tose applications involve uge amounts of data. For example, te TIGER database from te United States Bureau of Census is 9GB. NASA s Eart-observing system will generate TB of satellite data everyday once in full operation, and medical image databases are usually on te order of terabytes. Tus, for fast query processing, efficient algoritms for image indexing are required to accommodate tese enormous databases. Indexing image databases is based on image content, a major difference from conventional databases. Indexing may be based on different criteria suc as color, texture, sape, motion, volume, semantic constraints, or spatial constraints. Eakins classified image queries into tree levels tat range from te igly concrete to te very abstract [Eakins 996]. Level, te lowest level, comprises retrieval by primitive features suc as texture, color, and sape. Systems supporting level retrieval usually rely on automatic extraction and matcing of primitive features. Experimental and even a few commercial systems now exist wic can provide content-based retrieval on te basis of primitive features suc as color or sape suc as te QBIC system [Flickner et al. 995] and MIT s Potobook [Pentland et al. 994]. Level comprises retrieval by derived attributes involving logical inference about te objects depicted in te image. Tis level is of more general applicability tan level. However, te main block of progress at tis level is te process of interpreting an image in order to generate a sufficiently ric set of logical features for retrieval. Given te improved mecanisms for building up te knowledge bases of image feature interpretations, it sould be possible to construct systems tat offer level image retrieval witin restricted but nontrivial domains suc as potograps of sporting events. Level 3 comprises retrieval by abstract attributes, involving a ig degree of abstract and possibly subjective reasoning about te meaning and purpose of te objects or te scenes depicted. At tis level, problems so far proved insurmountable. Te prospects for viable image retrieval at tis level seem vanisingly small. Most existing image retrieval systems use low-level features for image retrieval. Te recent success of image-understanding approaces in various domains suggests te transition to te next level, wic is retrieval by identified objects. Tis level extends te query capabilities of an image retrieval system to support iger-level queries. Te focus of tis researc

3 Efficient Content-Based Indexing of Large Image Databases 73 Query Fig.. Searcable Index Candidate Image Set Similarity Algoritm A block diagram for spatial similarity retrieval in image databases. Ranked Set of Images is on indexing images based on te identified objects approac corresponding to Level in Eakin s classification. One of te most important metods for discriminating among te objects in an image database is te perception of te objects and te spatial relations tat exist between tem in an image. Te spatial data embedded in images sould be preserved in te logical image representation so tat users can easily retrieve, visualize, and manipulate images in te image database based on te images spatial content. Spatial similarity retrieval is a two-step process as sown in Figure. Te first step is to searc an index to retrieve te candidate set of images. Tis step acts as a filter to avoid unnecessary searces (false alarms). Te candidate set is ten submitted to te similarity algoritm for te images to be ranked according to teir degree of similarity to te query image. Calculating te spatial similarity of a query image to all database images as proibitive cost especially in large image databases. An efficient indexing mecanism sould minimize te number of false alarms tat ave to be tested by te similarity algoritm. Te index structure sould ave te following features: Te index structure sould be capable of efficiently answering bot general and specific queries suc as Find all images including a given set of objects and specific queries suc as Find all images including a set of objects and satisfying te given spatial constraints. Te index structure sould be dynamic to allow for online insertion and deletion of images. Te searc mecanism sould filter out nonrelevant images efficiently. On te oter and, it sould not discard any relevant images (i.e., avoid true dismissals). Te index structure sould be efficient in terms of te performance and te storage space required. Te main contribution of tis article is introducing te Two Signature Multi-Level Signature File (SMLSF), an image indexing tecnique tat improves te querying speed and te storage requirements of current indexing tecniques. Te SMLF is an extension of te Multi Level Signature File (MLSF), previously proposed for text indexing [Lee et al. 995]. Te SMLSF introduces te concept of using two different signa-

4 74 E. A. El-Kwae and M. R. Kabuka tures in te same indexing structure. Using two signatures allows te SMLSF to respond to bot general and specific queries and is te main factor for improving te indexing performance. To quantify te improvement in storage requirements and speed, analytical comparison of te SMLSF and te two-level signature file (LSF), currently used for image indexing, is given. Te rest of tis article is organized as follows. In Section, spatial relations are defined, and a background of current spatial similarity algoritms is given. In Section 3, image representation tecniques are discussed. Several existing indexing tecniques suc as R-Trees, inverted lists, weigted center of mass, as table indexing, iconic indexing, ordered triplets, and two-level signature files are surveyed in Section 4. Te limitations of current indexing tecniques are also igligted. Te SMLSF signature file is introduced in Section 5. In Section 6, te analytical comparison of te SMLSF and te LSF is given bot for storage requirements and searc performance, followed by te conclusions in Section 7.. IMAGE RETRIEVAL BY SPATIAL SIMILARITY Retrieval by spatial similarity (RSS) deals wit a class of queries tat is based on spatial relationsips among te domain objects. RSS query engines select and rank images from te database according to te degree wit wic tey satisfy te spatial relationsips specified by te query image. Image databases tat support spatial queries provide a ig-level object-oriented searc rater tan searc based on te low-level image primitives of objects. In addition, tese databases can provide visualization and iconic indexing capabilities [Lee et al. 989]. Spatial relations may be classified into directional relations and topological relations. Te frequently used directional relations are nort, sout, east, and west. Tese relations are called strict directional relations [Li et al. 996a] (Figure (a)). Some researcers add eigt more directional relations. Four of tese norteast, nortwest, souteast, and soutwest are called mixed directional relations (Figure (a)). Te oter four, left, rigt, above, and below, are called positional directional relations [Li et al. 996a]. On te oter and, some researcers specify te directional relation between two objects as te slope of te line joining teir centroids [Gudivada 995a; 995b; Gudivada and Jung 995; Gudivada and Ragavan 995] (Figure (b)). Egenofer and Franzasa [99] and Egenofer and Franzasa [995] point out tat tere are eigt fundamental topological relations tat can old between two planar regions. Tese topological relations are disjoint, contains, inside, meets, equals, covers, covered-by, and overlaps (Figure 3). Current RSS algoritms may be classified into symbolic projection metods, geometric metods, spatial reasoning metods, and grap-matcing metods. Symbolic projection metods are based on a D image representation called te D-String [Cang et al. 987]. Te D-String transforms

5 Efficient Content-Based Indexing of Large Image Databases 75 NW N NE q W E SW S SE (a) Strict and positional directional relations (b) Slope directional relations Fig.. Directional relations. Contains Overlaps Disjoint Meets Inside Fig. 3. Topological relations. Covers Covered By Equals te problem of pictorial information retrieval into a problem of D subsequence matcing. Various extensions of te D-Strings suc as te D-G String [Cang and Jungert 99], te D-C String [Lee and Hsu 990; 99], and te D-C String [Huang and Jean 994] ave been proposed to deal wit situations of overlapping objects wit complex sapes. Geometric metods use te inerent geometric features in te image for image representation suc as tose based on te Weigted Center of Mass [Hou et al. 99] and te R-Strings [Gudivada 995b]. Grap-matcing metods represent te domain objects included in an image and teir spatial relationsips by a grap called te Spatial Orientation Grap (SOG). Te nodes of tat grap represent te domain objects wile te edges carry te spatial relation between object pairs. Examples of tese metods are SIM R [Gudivada and Ragavan 995] wic uses Hamiltonian Cycles to represent te order of te list of objects in te image. Spatial similarity is quantified in terms of te number as well as te extent to wic te edges of te SOG of te database image conform to te corresponding edges of te SOG of te query image. Spatial reasoning metods define rule bases from wic new reltionsips can be deduced using a given set of relationsips [Li et al. 996a]. 3. IMAGE REPRESENTATION Various forms exist for image representation in image databases. Te Pysical or te raw images are stored at te pixel level, wile te logical representations abstract te contents of te pysical image. For example, image color istogram is an appropriate logical feature to process color similarity queries. For spatial similarity queries, various logical represen-

6 76 E. A. El-Kwae and M. R. Kabuka tations exist suc as D-Strings [Cang et al. 987], R-Strings [Gudivada 995b], spatial orientation graps [Gudivada and Ragavan 995] and te symbolic image [Gudivada and Jung 995]. Te International Organization for Standardization (ISO) as begun to clarify te scope and objectives of a potential multimedia content description interface standard, known as MPEG-7. Te standard is intended to facilitate te future development of audiovisual content-based searc engines [Carlson 997]. Te symbolic image is a logical representation of te pysical image obtained by associating a unique name wit eac of te domain objects identified in te image. Image segmentation and imagine-understanding tecniques can be used to identify and label te objects in an image. Altoug computationally expensive, te process of image understanding is performed only once at te time of te image insertion into te database. Image-understanding tecniques may be classified into manual, uman or knowledge base assisted, and application-specific tecniques. A definition and samples of eac tecnique are given below. In manual tecniques, a user extracts text annotations to describe te image. Tese annotations are stored in a conventional text database. Manual tecniques are inerently time consuming and tedious, and teir effectiveness is often limited. Generating text annotations is sometimes very difficult or not possible as in te case of describing texture. In addition, te resulting annotations are usually not consistent among different users or even for te same user in different settings. Human-assisted object recognition utilizes uman input and feedback to identify objects in an image. If a user labels a piece of an image as a cair, ten a feature representation or model of tat part of te image can be used to propagate te label to oter visually similar regions. Te user can ten browse troug te images in te database and select patces from one or more images as positive examples. Based on te selection of patces, te system selects te best models to represent te object and propagates te object label troug oter regions in te database tat sould ave te same label based on te models. Falsely labeled patces can be removed by selecting tem as negative patces. Tis approac as been developed into a system wic assists a user in annotating large sets of images based on texture properties of tese images [Picard and Minke 994]. Knowledge-base-assisted tecniques utilize te knowledge about te expected contents in an image. A knowledge base approac for image understanding was introduced [El-Kwae and Kabuka 996]. Tis approac integrates unsupervised image segmentation, image labeling, and a taskspecific knowledge base to extract useful information from images. Te knowledge base cecks for consistency of te obtained labels. If any inconsistencies are found, it generates corrective feedback to improve on te clustering process. Tis cycle of segmentation, labeling, and consistency cecking by te knowledge base is repeated until consistent labels are assigned. Te knowledge base is built wit an expert user s assistance

7 Efficient Content-Based Indexing of Large Image Databases 77 during te early stages of te system. Examples of images are introduced, and te user adds, deletes, or modifies te knowledge base until satisfactory results are obtained on te example images. Te ypotesis-and-test paradigm is anoter knowledge-based tecnique based on ypotesis generation and verification. A model database includes a model for eac object in te domain of discourse. A data-driven indexing mecanism for model retrieval (ypotesis generation), as well as a standard model-driven approac to ypotesis verification, was suggested [Grosky and Merotra 990]. In tat prototype system, te model of eac object is in te form of a two-level ierarcical description. Te top level is te entire object, wile eac leaf is a portion of te object boundary. Wen a particular boundary segment is found in te image, various top-level objects are ypotesized. An application-specific indexing and content-based retrieval of captioned images was suggested [Sriari 995]. A system, called Piction, to identify uman faces in segmented newspaper potograps based on te information contained in te associated caption was implemented. Anoter application-specific object extraction tecnique is based on object color. Wile it is not practical to develop a general object extraction system based on color, it is still possible to define a limited set of assumptions for common objects suc as uman faces, sunny skies, te sea, forests, and lawns. Te objects for wic color extraction works best are tose for wic te ue falls into a narrow caracteristic range despite variations in illumination and potograpic conditions in different scenes. For example, for te skin color, te ue range value is 0.7 to., wile te croma value is greater tan 0 [Furt et al. 995]. A knowledge-based approac for labeling D Magnetic Resonance (MR) brain images was introduced [Li et al. 996b]. Te approac consists of two components, a boolean neural network (BNN) clustering algoritm and a constraint satisfying boolean neural network (CSBNN) labeling algoritm. Te CSBNN uses a knowledge base tat contains information on te image feature space and te tissue models (Figure 4(a)). An algoritm for te segmentation of brain MR images was introduced in Tsai et al. [995]. Tis algoritm was applied to a brain image, and te result is sown in Figure 4(b). Te cerebrum region is extracted using a single tresold, computed directly from te istogram, and a sequence of morpological operations. Te CSF regions are detected from T weigted images by adaptive tresolding, and te ventricular and extraventricular regions are identified. Te brain matter is furter classified into gray and wite matter from te PD images using a low-level knowledge-based segmentation rule. Finally, a ceck for any abnormal signal intensities is made. Tis includes detection of any lesions or abnormal ventricles. In general, completely automatic image-understanding systems are not feasible except in limited domains. Tis explains te large body of researc on uman-assisted and knowledge-base-assisted image-understanding scemes, wic provide a feasible alternative to te image-understanding problem witin restricted but nontrivial domains.

8 78 E. A. El-Kwae and M. R. Kabuka Fig. 4. (a) An example for knowledge-based brain MR segmentation. (b) An example for automatic brain MR segmentation. Cerebrum region (A), abnormal signal intensity (B), and ventricle (C), Extraventricle (D). Figure 4(a) is reprinted from Li, X., Bide, S., and Kabuka, M., Labeling of MR Brain Images Using Boolean Neural Network, IEEE Trans. Med. Imaging 5, (996), pp IEEE. For te rest of tis article, te image database is assumed to consist of a pysical database wic stores te original images and a logical database, wic stores logical image representations. Queries are resolved troug te logical database, and only tose pysical images tat are part of te query response are actually retrieved. For te logical representation, an image I is assumed to consist of a set of n domain objects. Eac object O i is represented by a logical representation R i. Different representations for R i will be discussed. Tus, an image is defined as I O i,r i, i n. 4. RELATED WORK TO IMAGE INDEXING FOR RETRIEVAL BY SPATIAL SIMILARITY Indexing is crucial for efficient access to all sorts of information. Indexing addresses te issue of ow te information sould be organized so tat queries can be resolved efficiently and relevant portions of te data quickly extracted. In an image database environment consisting of tousands or millions of images, it is not practical to compute te similarity of a query image wit every database image. For example, if te query and database images ave no common objects, tere is no need to compute te similarity. Te index sould not sacrifice accuracy for speed, wic means tat it sould not drop any relevant images (true disposals). On te oter and, it sould try to minimize te number of irrelevant images returned (false alarms). Most current spatial similarity retrieval algoritms focus on assessing similarity between two images and ignore te indexing problem. Tis led to te existence of efficient similarity assessment algoritms witout te accompanying indexing mecanisms. Witout an efficient indexing mecanism, searc as to be done sequentially, wic incurs proibitive cost in large image databases. Several indexing tecniques ave already been proposed, and tey are briefly discussed in tis section.

9 Efficient Content-Based Indexing of Large Image Databases 79 Te R-Trees, proposed by Guttman [984], are widely used for spatial and multidimensional databases. Te R-Tree is a ierarcical data structure derived from te B-Tree. Arbitrary geometric objects can be andled by an R-Tree by representing eac object by its minimum bounding rectangle (MBR). Oter algoritms suc as te R -Trees [Sellis et al. 987], te R*-Trees [Beckmann et al. 990], and te TV-Tree [Lin et al. 994] improve te quality of te tree. If te data rectangles are given a priori, preprocessing can be applied to pack te data to acieve iger utilization. Several packing algoritms exist including te Hilbert Packed R-Tree [Kamel and Faloutsos 993] and te STR [Leutenegger et al. 997]. Te inverted file is a simple indexing tecnique tat as been extensively used for text indexing [Frakas and Yates 99]. Guidivada [Gudivada 995b; Gudivada and Jung 995] used inverted lists to avoid sequential searc of te image database. Te inverted file is a sorted list of domain object names. Eac location in tis list points to a set of images in wic te domain object corresponding to te index occurs. Given a query image Iq wit n objects, te set of images pointed to by te entries in te index corresponding to te domain objects in te query is located. Te intersection of suc lists, S, is a list of images including all domain objects in te query. Only te images in S need to be evaluated for spatial similarity wit te query. Tis metod is simple and avoids true dismissals, but te size of te list associated wit eac domain object grows rapidly as te number of images increases, wic increases te required storage and te cost of merging lists during query processing. In addition, te inverted file can only answer te general query Find all images in wic te query objects are present. Tus, te inverted file will suffer from a large number of false drops if spatial constraints are included as part of te query. A content-based indexing tecnique based on te weigted center of mass (WCOM) was introduced [Hou et al. 99]. An image is transformed to a point in multidimensional space by extracting image features from te image and inserting tem into a feature vector. Since te ig dimensionality slows down te retrieval process, only 4 objects, eac represented by a feature vector of 3 parameters, are selected from eac image. Tese objects are called significant objects and are defined as tose objects wic are larger and closer to te image center of mass. Since tis sceme excludes some of te objects in te image, it migt lead to true dismissals, wic is not acceptable in most image database applications. Anoter tecnique for indexing D-strings as been proposed [Cang and Lee 99]. Indexing is based on all te object pairs included in an image. For eac pair of objects o i and o j, an ordered triplet is created o i,o j,r ij were r ij is te spatial relationsip between te two objects. Eac triplet is assigned an address and entered into a as table. Eac pair of objects in te query image acts as a separate query and is used to retrieve te set of images stored at te corresponding as table address. Te intersection of te retrieved sets constitutes te candidate result set. Tis sceme requires all te images to be known in advance. A preprocessing step is

10 80 E. A. El-Kwae and M. R. Kabuka f f Level (block) signatures Level (record) signatures Access Pointers Signature Generation Logical Images Fig. 5. Level Superimposed Coding Two-level signature filter (LSF). required to derive a perfect as function wic guarantees tat no two pairs of objects are mapped to te same address in te as table unless tey ave te same properties. However, if new images are inserted to te database, te as function ceases to be perfect. Anoter approac for D-Strings indexing based on groups of two or more objects was introduced [Petrakis and Orpanoudakis 993; Petrakis 993]. An image is decomposed into groups of objects called image subsets. All image subsets from up to a specified size K max are produced. Te number of image subsets becomes very large especially wen K max 5 and n 0 were n is te number of objects per image, wic renders tis metod unsuitable for large image databases. A separate as table is created for image subsets. For image retrieval, two type of queries are considered, direct-access queries wit objects q K max and indirect-access queries for wic q K max. For indirect queries, te original query is decomposed into smaller queries of K max objects eac. Te answer set eac of tese queries is derived, and teir intersection is obtained. Tis algoritm was tested on a small simulated image database of 000 images containing between 4 and 0 objects eac. Results sow tat te retrieval response time is slower, in some cases, tan tat of a sequential searc. Tis is in addition to te significant storage overead. Te LSF (Figure 5) indexing metod was introduced [Lee and San 990] for images indexed by D-Strings. Te LSF uses a two-level signature file to represent te D-Strings in te spatial database. A signature file is a filtering mecanism tat quickly eliminates most of te images tat are irrelevant to te given query. Signature files ave been widely

11 Efficient Content-Based Indexing of Large Image Databases 8 Objects: Butterfly (B), House (H), Luggage (L) and Artist (A) Fig. 6. Object Signatures: (B): (H): (L): (A): Image Signature: Queries Signature Result ) House Matc ) Desk No Matc 3) House and Artist Matc 4) Car False Drop Signature generation and comparison based on superimposed coding. employed in information retrieval systems for bot formatted and unformatted data [Faloutsos 985; Frakes and Yates 99; Faloutsos and Cristodoulakis 984; Roberts 979; Davis and Kamamoanarao 983; Lee et al. 995]. Recently, te signature file tecnique was applied to image databases [Lee and San 990; Tseng et al. 994]. Image signatures may be obtained in a number of ways [Faloutsos 985]. Te most common metod is superimposed coding in wic eac object (or object pair) in an image is ased into a word signature. An image signature is generated by superimposing (ORing) all its individual signatures. To resolve a query, te query signature is generated and matced (ANDed) against image signatures. Figure 6 is an example for generating an image signature from object signatures. Te equations used to calculate te signature weigt and widt are obtained so tat te false drop probability is minimized and are proved in Roberts [979] and Davis and Kamamoanarao [983]. Let m and w be te lengt and weigt (number of ones) of a signature respectively. It is assumed tat bits are randomly cosen and tat eac of te possible signatures is equally likely to be cosen wen an image signature is generated. Te probability P a, a is te probability tat a bit positions are set to one in a signature superimposed from a signatures eac of weigt w and lengt m. If a is sufficiently large and w m, te following approximation of P a, a can be obtained: P a, a m m w a If w x is te weigt of a signature superimposed from x word signatures, ten te following equation sows te relationsip between P a, a and w x : a

12 8 E. A. El-Kwae and M. R. Kabuka w x mp x, m m w x Te false drop probability of a signature file P f can be represented by P s, w. Tis is actually te false drop probability for an unsuccessful searc of a single-term query. However, it accurately approximates te false drop probability of a successful searc [Faloutsos and Cristodoulakis 984]. By minimizing te false drop probability, te following relationsip can be obtained: m w s 0.5 ten P f 0.5 w Given P f, s, and te above relationsip, w and m ave te following values: w ln ln and m P ln sln P In te LSF, eac pairwise spatial relationsip, contained in a D- String, is represented by a spatial string. Eac D-String is associated wit a record (leaf) signature of m bits and a block (root) signature of m bits. Te record signature of a D-String is generated by extracting te spatial relations between object pairs in te image. A record term signature, wit exactly w bits equal to, is generated for eac object pair using as functions. In general, w as functions are used to determine te distinct positions of te bits. Te record signatures of all te object pairs in te image are ten superimposed (ORed) togeter to generate te image record signature. Te block signatures of all object pairs in all images included in te block are superimposed to generate te block signature of te block of images. In general, m is larger tan m, since muc more information is to be represented in a block signature tan in a record signature. Eac block signature as an associated pointer to te first record signature of images in tat block. Eac record signature as an associated pointer to te corresponding logical image representation wic in turn as a pointer to te location of te pysical image. It was previously proven tat te storage requirement for te single-level signature file (SLSF), te LSF, and te MLSF indexing metods is te same if tey all ave te same global false drop probability [Lee et al. 995]. Tis is because adding new levels to te signature file allows te local false drop probability at eac level to increase wile maintaining te same global false drop probability. Using a larger false drop probability allows te signatures at eac level to be sorter tan tat of a single-level signature file. Te savings at te first level of signatures are compensated for at te iger levels because te number of items to be encoded for eac

13 Efficient Content-Based Indexing of Large Image Databases 83 signature increases for te iger levels. It was also sown tat te required storage per level is constant [Lee et al. 995]. For eac level up te ierarcy, te lengt of a signature increases b times, wile te number of signatures in te level decreases /b times. Tus, te required storage for eac level remains constant. Te LSF was proved not to be optimal in terms of performance [Lee et al. 995]. Altoug te LSF, and te MLSF require te same storage, te number of bits compared for processing a query was sown to be muc less in case of te MLSF tan in te LSF. Increasing te number of levels in te signature tree was sown to improve te performance. Tus, te optimum blocking factor is b, i.e., eac node in te MLSF includes exactly signatures and te MLSF becomes a binary tree. From tis discussion, it may be concluded tat using te MLSF for indexing is expected to improve over te LSF in terms of te number of bits compared per query. Anoter image retrieval algoritm for D-Strings called te bit-sliced two-level signature file (BSLSF) was introduced [Tseng et al. 994] to furter improve te performance of image retrieval over te LSF metod at te expense of insertion cost. Te BSLSF applies te concept of bit-transposed files to speed up te filtering process. In tis metod, te record and te block signatures are generated in a similar way to LSF. However, te block and te record signatures are not stored sequentially but in bit slices. In tis case, a query signature wit w q number of s, requires tat only w q slices to be examined rater tan all te block signatures. Tat is, only w q n bits will be examined rater tan m n in te LSF case. Te S-tree is a multilevel signature file tecnique tat as been investigated in te literature [Deppisc 986]. One problem wit tis metod is tat iger-level signatures are created by superimposing signatures at lower levels. As more signatures are included, te bit density of te signatures will increase, rendering te metod useless due to a large number of false alarms. 5. THE SMLSF TECHNIQUE Te SMLSF tecnique (Figure 7), introduced in tis article, is an extension of te MLSF tecnique. A multilevel signature file is a forest of b-ary trees wit every node, except leaf nodes, in te structure aving b cild nodes. Te number of levels in te structure is. Some assumptions are made to simplify te analysis of te SMLSF. Te trees are assumed to be complete b-ary trees. Tus, te relationsip between n, b, and is n b. Local parameters representing te value of some global parameter p at level i are denoted p i. For instance, at te it level, te local false drop probability is represented by p f i. To furter simplify te analysis, it is assumed tat te local false drop probability is te same for every level. Te relationsip between te global and local false drop probabilities is

14 84 E. A. El-Kwae and M. R. Kabuka Fig. 7. Two Signature Multi-Level Signature Filter( SMLSF). p f p i f p j f bp f i j p i f p f /. f p i Te bit density problem of te S-tree does not exist in SMLSF, since signatures at iger levels ave longer lengts and are generated independently from tose at lower levels. In te SMLSF tecnique, two types of signatures are used. Type I signatures are based only on te objects included in te image, wile Type II signatures are based on te included objects in addition to teir spatial relationsips. If an image I includes x objects, tere exists y x x object pairs. A certain spatial relation relates eac of tese pairs. Tus, a Type I signature will encode te image by superimposing x signatures, wile a Type II signature encodes an image by superimposing x x signatures. Te signature widt tat minimizes te false drop probability sould be equal to m ln sln P were s is te number of distinct items to be encoded per image. Tus, decreasing te number of items to be encoded leads to a sorter signature. Tis means tat a Type I signatures uses a sorter signature tan tat of a

15 Efficient Content-Based Indexing of Large Image Databases 85 Fig. 8. Storage reduction of Type I and Type II signatures. Type II signature. Te storage requirement of te two types of signatures is as follows: m I ln xln P m II ln x x ln P Ten, te ratio between m II and m I can be calculated as m II ln x x ln P m I ln xln P m II m I x. Tis means tat eac Type II signature requires additional storage over tat required by Type I signature wenever x 3. For an image wit 3 or more objects, Type I signatures will be ceaper tan Type II signatures (Figure 9). In fact, Type II signatures require double te storage of Type I signatures for images wit 5 objects and 9.5 times te storge required by Type I signatures for images wit 0 objects. For large image databases, a substantial reduction in storage requirement may be acieved if Type I signatures are used for image indexing. In addition to storage reduction, performance improvement is expected wen sorter signatures are used. From Figure 8, a single Type I signature comparison is 9.5 times ceaper tan a single Type II signature comparison for images wit 0 objects. On te oter and, Type II signatures carry

16 86 E. A. El-Kwae and M. R. Kabuka more information tan Type I signatures, since tey encode te spatial relationsips in te image in addition to te included objects. For tis reason, Type II signatures can answer a more specific query tan tat of a Type I. Assume tat an image database IDB is encoded twice, once using Type I signatures IDB I and te second time using Type II signatures IDB II. Wen a spatial query is submitted, IDB I will return all te images including te given objects in te query image regardless of weter tose images satisfy te given spatial constraints. IDB II will return all te images including te given objects and satisfying te spatial constraints in te submitted query. Te images returned by IDB I will be a superset of tose returned by IDB II. Assume tat te images returned by IDB I but not returned by IDB II are called I DIFF. Te images included in I DIFF are similar but not identical to te query image. Since queries in an image database often require te retrieval of similar images in addition to te exact images, Type II signatures are not adequate, if used alone, to retrieve similar images. Current metods for spatial similarity indexing [Lee and San 990; Tseng et al. 994 ] use only Type II signatures for indexing. On te oter and, if only Type I signatures are used, a large number of false alarms may be generated in I DIFF tat may ave a low similarity wit te query image. In te SMLSF, bot types of signatures are used for image encoding. Tis way, exact queries may be answered using Type II signatures, wile general similarity queries may be answered using Type I signatures. 5. Index Creation in te SMLSF In order to create te signature tree in te SMLSF, independent signatures are created for eac image, one for eac level in te tree. At te leaf level, signatures are of Type II, wile at iger levels, signatures are of Type I. For leaf-level signatures, eac pairwise spatial relationsip in te image is represented by a specific ID. For example, if te two objects participating in te pairwise relationsip are A and B, suc tat te ID of A is less tan tat of B, ten te spatial relationsip between A and B, termed S AB,is calculated. Te object pair is represented by te spatial string ABS AB. Te leaf signature of widt m and weigt w is ten generated for tis object pair using w as functions. Eac as function selects a single bit out of te m bits in te signature. Te image signature is created by superimposing (ORing) all te object pair signatures of te image. For example, assume tat an image includes 4 objects (A, B, C, D suc tat teir ID order is A, B, C, D). Te spatial strings between object pairs are sown in Table I. Assume tat m 8 bits and w bits; ten a possible set of signatures for tose spatial strings is also sown in Table I. Superimposing all tose signatures results in an image signature of 000. Tis signature is used to index te image at te leaf level.

17 Efficient Content-Based Indexing of Large Image Databases 87 Table I. Spatial Strings in te Example Image and Teir Leaf Signatures Spatial String Leaf Signature AB AC AD BC BD CD For all oter levels in te signature tree, generating te image signature involves only te objects included in te image. For an image wit x objects at level i, x signatures are generated. For eac object, te object ID is used as a key to w i as functions to generate a signature of widt m i were w i and m i are te signature weigt and widt at level i. Te object signatures are ten superimposed to generate te image signature at level i. Signatures of images in te same block are ten superimposed to generate te block signature. For example, if b (i.e., tere are two images per block), image signatures are superimposed to generate te block signature at te level; 4 image signatures are superimposed to generate te block signature at te level, and so on. In general, i image signatures are superimposed to generate te block signature at level i. Associated wit eac nonleaf block signature is a pointer to te signatures at te next lower level. A pointer to te logical image representation is associated wit eac leaf signature wic in turn points to te pysical image. An example of a SMLSF for an image database of eigt images is sown in Figure 9. Te tree as tree levels wit te following signature widts: m 6, m, and m3 8. For eac image, tree signatures are created s, s, and s 3, were te s signatures are stored at te leaf level. Images at level two are created by superimposing te s signatures of two images wile te root signatures are generated by superimposing te s 3 signatures of four images. Te steps needed to create te SMLSF index can be summarized as follows: () Given te values of n (number of images in te database), p f (global false drop probability), and x (number of objects per image), determine te value of (number of levels). Ten, for eac level, determine te values of p i f (local false drop probability), m i (signature lengt), and w i (signature weigt) as described above. () For eac image in te database, create te leaf-level signature as follows: For eac object pair in te image, create a spatial string tat represents te two objects and teir spatial relation. Ten, generate a signature representing tat spatial string.

18 88 E. A. El-Kwae and M. R. Kabuka 0000 LI PI LI PI LI3 PI LI4 PI LI5 PI LI6 PI LI7 PI LI8 PI8 Level (Root) Level Level 3 (Leaf) Logical Images Fig. 9. SMLSF for an example image database of eigt images. Pysical Images Superimpose te signatures of all te spatial strings of te image to create te leaf image signature (Type II Signature, of lengt m, and weigt w ). Add a pointer from te leaf signature to te logical image representation. (3) For eac image in te database, create signatures for te remaining levels as follows: For eac object in te image, create te signature for tat object at eac level. Superimpose te object signatures at level i to create te image signature (Type I Signatures). (6) For eac level starting from te root (level ) to te level rigt above te leaf (level ), superimpose b i image signatures to create level i signature (of lengt m i and weigt w i ) and adjust pointers from one level to anoter.

19 Efficient Content-Based Indexing of Large Image Databases Query Processing in te SMLSF Two types of queries can be answered by te SMLSF indexing mecanism. Te first is te specific query Given a query image, find all te images wic include te set of objects in tis image suc tat tose images satisfy all te spatial relations in te query image. Te second is te general query Given a query image wit a given set of objects, find all te images wic include tis set of objects. To process a query image q submitted to te SMLSF, signatures labeled q, q,.., q are generated for te query image by using te same procedure described above for generating image signatures. Starting at te root, te query signature q is ANDed to all te root signatures. If te result is not equal to q, ten it can be certain tat tere are no images underneat te root tat satisfy te query, i.e., unsuccessful searc. If te result is equal to q (i.e., all te bits in te q signature are also contained in te root signature), tere is a cance tat some images underneat tis signature satisfy te query. In tis case, searc resumes wit te signatures at te second level. Again, te query signature q is ANDed to all signatures at tis level. If te result is not equal to q, ten it can be certain tat tere are no images underneat tis block to satisfy te query. Oterwise, te subtree underneat tat block as to be searced, and so on. Tis procedure is repeated for all qualifying blocks If te query is of Type I, te searc procedure is repeated until te leaf level is reaced. All te leaf signatures wic matc te q signature are query candidates. Due to te information loss in representing images by signatures, a leaf signature tat matces te query may not actually include te set of objects and te spatial relations in te query image. Tis is called a false drop, and it occurs wit a probability called te false drop probability. Te logical image representation pointed to by te query candidates are ten passed to te Retrieval by Spatial Similarity (RSS) algoritm. Te RSS algoritm performs a detailed ceck on te images to exclude false drops and rank te oter candidates based on teir degree of similarity to te query image. If te query is of Type II, te searc procedure is repeated until te st level (te level rigt above te leaf level) is reaced. Tis is because tis level is te last level wic uses Type II signatures. All signatures at tis level matcing te q signature are query candidates. Te false drop probability for tis query is sligtly iger tan tat of Type I queries because a smaller number of levels is used to locate te query. Te logical images under all query candidates are ten passed to te RSS algoritm, wic performs a detailed ceck to exclude false drops and rank te oter candidates based on teir degree of similarity to te query image. Te false drop probability of Type I queries is p f wile tat for Type II queries is

20 90 E. A. El-Kwae and M. R. Kabuka p f p f i p f /. Since p i f, ten p i f / p i f. Tus, te false drop probability of a Type II query is iger tan tat of a Type I query. Te difference between te two probabilities decreases by increasing te number of levels in te signature tree. 6. COMPARISON OF THE SMLSF TO EXISTING TECHNIQUES To compare te SMLSF and te LSF [Lee and San 990] indexing metods for large image databases, two criteria are used: te amount of storage required and te total number of bits compared during query processing. 6. Storage Reduction Ratio Te storage reduction relation (SRR) is used to measure te ratio of te storage required by te SMLSF to tat required by te LSF [Lee and San 990]. SRR is calculated as te difference between te storage required by bot metods normalized by te storage required by te LSF: SRR M LSF M SMLSF M LSF THEOREM. Te SRR of te SMLSF compared to te LSF may be calculated as follows: SRR x 3 x PROOF. A summary of te proof is given ere, wile te detailed proof is given in Appendix B. e Te storage required by te SMLSF is equal to te sum of te storage required at eac of te levels. Since signatures used at te leaf level are Type II signatures and since tose used at all oter levels are Type I signatures, te storage required is divided into two parts: M SMLSF b i m i b i m i nm n b Te signature widt m ln s i ln P.

21 Efficient Content-Based Indexing of Large Image Databases 9 For te SMLSF tecnique, s y at te leaf level (were y x x ), and s i x at all remaining levels. ln n ln yln p M SMLSF i b n ln n ln xb i ln p i y x ln p i yln p i yln p i Since n ln yln p i is equal to nm and ten M SMLSF nm n ln ln p i y y x ln p, i ln p f ln p i M SMLSF nm n ln y x 3 x ln M SMLSF M LSF n ln Te SRR can be calculated as follows: SRR n ln x ln p i x x 3 x x 3 M LSF, p i ln p ln p. ln p

22 9 E. A. El-Kwae and M. R. Kabuka SRR x x 3 x From te above equation, it can be seen tat te storage reduction relation (SRR) depends on te number of objects in te image x and on te number of levels in te signature tree. If te number of objects per image is greater tan tree objects, te SRR will be positive, indicating tat indeed tere will be storage reduction. Tis is expected because if an image as exactly two objects, tere is only one object pair. In tis case, only one relationsip is used to create te signature in te first level, wile in te oter levels two objects signatures ave to be superimposed to create te image signature. From te SRR equation, it can be seen tat increasing te number of levels in te tree leads to an increase in te storage reduction. Since log b n, te maximum value of is acieved wen te packing factor b is equal to. Coosing b equal to is also te optimum value to decrease te number of signature searces [Lee et al. 995]. In image databases including more tan two objects per image, a storage reduction will always be acieved wen te SMLSF is used. Various practical applications include multiple objects per image. Some examples include Te MR brain segmentation described above. A framework tat supports quantitative (nonsymbolic) representation and comparison of images based on spatial arrangements of typed objects was introduced [Del Bimbo et al. 998]. Te framework was designed specifically for te retrieval witin a library of digital images reproducing renaissance paintings. In te large majority of cases, 0 to 30 objects were used to provide a detailed description of te contents of a painting. A query mecanism for a database of images and CAD models was introduced [Hildebrandt and Tang 996]. A prototype system for querying sip data, using symbolic descriptions of component objects of te sip and teir relative locations, was developed. A query consists of a set of pairwise spatial relationsips required in te symbolic image. A large set of objects was used for composing a spatial query. Examples of tose objects include crane, derrick, funnel, gun, elipad, mast, mortar, runway, radar, superstructure, and missile. 6. Computation Reduction Ratio Te performance of te SMLSF is compared to te LSF based on te number of bits to be retrieved and compared to resolve a given query. Te measure used for comparison is te computation reduction ratio (CRR) defined as te difference between te number of bits compared during query processing for te SMLSF and te LSF normalized by te number of bits compared for te LSF. Te CRR is calculated using te following equation:

23 Efficient Content-Based Indexing of Large Image Databases 93 CRR B LSF B SMLSF B LSF To process a query, signatures are retrieved from te signature file and compared to te query signature. Te retrieved signatures eiter satisfy te query and sould be retrieved by any indexing sceme tat does not allow true dismissals, or te retrieved signatures are irrelevant signatures tat are only retrieved because of false drops. Te number of bits compared during an unsuccessful searc is a good indicator of te amount of effort done by an indexing mecanism on irrelevant data. 6.. Performance of te LSF metod. In te following equations, subscripts and indicate te parameter value at te root and te leaf levels respectively. For example, n denotes te number of signatures at te root level wile n denotes te number of signatures at te leaf level indexed by a signature at te root level. A block of images at te leaf level is searced only wen its parent signature satisfies te query. Tus, te relationsip between te global false drop probability P f and te local false f f drop probabilities P and P can be expressed as P f f P.P f. Tis is because a signature at te root level is generated, independent of signatures at te leaf level [Lee et al. 995]. For an unsuccessful searc (i.e., t 0), te following equations can be derived [Davis and Kamamoanarao 983]: n n n s n s s s w w ln ln P ln ln P and and m ln s ln P m ln s ln P w q, m w m q w q, m w q m Te effective values of w and m for a single-level signature file aving te same false drop probability as tat of te two-level signature metod can be expressed as [Lee et al. 995]: w ln ln P w w

24 94 E. A. El-Kwae and M. R. Kabuka M m n m n nm Tus, a LSF file wit a global false drop probability equal to tat of SLSF requires te same storage as tat of te single-level metod [Lee et al. 995]. Altoug te false drop probability at a certain level is iger tan tat of a single-level signature file, a false drop introduced at te root level is eliminated at te leaf level, since te signatures in te two levels are generated independently. In general, te equations used to obtain tis result are based on te false drop probabilities of unsuccessful searces. For successful searces t 0, te storage overead of te two-level metod is greater tan te single-level metod for te same false drop probability. THEOREM. Te number of bits matced during an unsuccessful searc in te LSF is equal to B n m n 0.5 wq, n m n m nm 0.5 wq,. PROOF. A summary of te proof is given ere, wile te detailed proof is given in Appendix C. e Te number of signatures searced in te LSF is A A A wic is equal to te number of signatures searced at te root and te leaf levels respectively. Te total cost of searcing te LSF is equal to te total number of bits cecked, at bot levels, during a searc. Tis cost can be calculated at eac level as te number of signatures searced multiplied by te widt of signatures at tat level, i.e., B B B were B i A i m i. During a searc, all te signatures at te root level will be searced, i.e., A n and B n m. For te leaf level, only tose blocks pointed to by signatures from te root-level matcing te given query are searced. Signatures matced at te root level may be due to true drops (denoted A T, ) or false drops (denoted A F, ). For eac of tose matcing signatures, n signatures from te leaf level will be searced. Tus, for te LSF metod, B can be estimated as follows: Te probability of a node in te root level to be selected due to true drop is equal to [Lee et al. 995] A T, n n. Te probability of a node at te root level being selected due to false drop is equal to [Lee et al. 995] A F, n n 0.5 wq,. From te previous equations, te number of bits compared in case of te LSF is equal to

25 Efficient Content-Based Indexing of Large Image Databases 95 B n m n n n t n 0.5 n wq, m. In case of unsuccessful searc (i.e., t 0), te number of bits compared is equal to B n m n 0.5 wq, n m n m nm 0.5 wq,. 6.. Performance of te SMLSF Metod. Te SMLSF coding sceme creates a tree tat consists of levels of signatures. Signatures at te leaf level (level ) are superimposed codes generated directly from object pairs in te images. Signatures at iger levels (levels up to, wic is te root) can be considered as being generated from a very large image block consisting of approximately n x objects (were x is te number of objects in te image). Terefore, te optimal signature lengt is different at eac level. For te SMLSF metod, B can be estimated as follows: THEOREM 3. Te number of bits matced during an unsuccessful searc in te SMLSF is equal to B b i Z k m i. k PROOF. A summary of te proof is given ere, wile te detailed proof is given in Appendix D. e Te total number of signatures searced is equal to te number of signatures searced at eac level: A A i For te first (root) level, all b signatures are searced. For iger levels, te number of signatures searced is equal to te number of signatures searced at te previous level, bot due to true drop A T, and to false drop A F,, multiplied by te blocking factor b. A i b i A T, A F, b i Te probability of a node at level i to be selected due to true drop is equal to i A T, i b t. b

26 96 E. A. El-Kwae and M. R. Kabuka Fig. 0. Comparison of storage required for te SLSF, LSF, and te SMLSF metods. Te probability of a node in te root level to be selected due to false drop is equal to A F, i b b 0.5 wq, i i A i A T, i 0.5 wq, i i. Let z i 0.5 wq, i ; ten for i, te number of signatures searced at level i will be equal to A i A T, A F, b ba z ba T, z. Tis means tat te number of signatures searced at level i can be estimated by te following recurrence relation: A i ba z ba T, z Te initial conditions for te previous recurrence equation are already known, since A B. Assume tat A T, 0, z 0. By inspection, te number of signatures searced at a level i can be estimated as follows: A i b i z k b i j A T, j z j k j k j B b i z k b i j A T, j z j z k m i k j k j z k

27 Efficient Content-Based Indexing of Large Image Databases 97 Fig.. Te SRR of te SMLSF over te LSF. Fig.. Comparison of te bits compared during query processing for te SLSF, LSF, and te SMLSF metods. In case of unsuccessful searc t 0, te value of A T, i 0, ten, B b i z k m i. k In order to compare te performance of te SLSF, LSF, and te SMLSF, te following parameters were used: n 30 images; x was

28 98 E. A. El-Kwae and M. R. Kabuka Table II. Storage Requirement of te SLSF, LSF, and te SMLSF Metods X SLSF LSF SMLSF SRR varied from 3 to 0 objects per image; and p f n 9.3*0 0. Te image size was assumed to be (i.e., 64KB). Tus, te image database size was equal to 64TB. Te criteria used for comparison were M, te storage required by eac of te 3 metods and B, te amount of data compared in case of an unsuccessful searc t 0, in addition to te SRR and te CRR defined above. Figures 0 and sow te results of comparing te 3 metods in terms of te M and B. Figure displays te storage reduction ratio (SRR) acieved wen te SMLSF metod is used compared to te SLSF and te LSF metods. Note tat te storage required by bot te SLSF and te LSF is te same. It can be seen tat te SMLSF storage reduction increases by increasing te number of image objects. Te SRR starts at 0% for x 3 objects per image and increases to reac 78% for x 0 objects per image (Table II). Figure displays te computation reduction ratio (CRR) acieved wen te SMLSF metod is used compared to te SLSF and te LSF tecniques. Figure 3 displays te CRR acieved wen te SMLSF metod is used compared to te SLSF and te LSF metods. Te comparison is made for an unsuccessful searc t 0. Again, it can be seen tat te SMLSF computation reduction increases by increasing te number of image objects. Te CRR starts at 86% for x 3 objects per image and increases to reac 98% for x 0 objects per image (Table III). Te SMLSF query processing is performed in steps: during te first step S, te query criteria are based on te included objects, wile in te second step S, te query criteria include te given spatial constraints in addition to te included objects. If te number of images to be returned by

29 Efficient Content-Based Indexing of Large Image Databases 99 Table III. Bits Compared During Query Processing for te SLSF, LSF, and te SMLSF Metods X SLSF LSF SMLSF CRR S is larger tan tat to be returned by S, tere must be some images in te database wic include te required objects but do not satisfy te given spatial constraints. Te SMLSF still touces te signatures leading to tose images wile traversing from te root level to te level above te leaf level. In order to measure te number of additional images tat may be touced wile still aving a comparable performance wit te LSF, te same parameters were used, and it was assumed tat no images would be returned by S t 0. Te number of images returned by S was increased until te number of bits compared was more tan tat of te LSF. For x 3, 5000 additional images may be touced by te SMLSF, wile te number of bits compared is still less tan tat of te LSF. Tis number increases to 800,000 for x 4 and 5 for x 5. For images wit x 5, te number of bits compared by te LSF for t 0 is larger tan te wole index size of te SMLSF, as can be seen from Tables II and III. Even if te SMLSF searces all images, its performance will still be better tan tat of te LSF. 7. CONCLUSIONS To avoid exaustive searc in large image databases, a multilevel signature file indexing tecnique called te Two Signature Multi-Level Signature File (SMLSF) is introduced. Tis tecnique does not cause any true dismissals, and te signature parameters may be selected to minimize te false drop probability. Two types of signatures are generated for eac image: one is stored at te leaf of te signature tree and is based on te included domain objects and teir spatial relationsips, and te oter is

30 00 E. A. El-Kwae and M. R. Kabuka Fig. 3. Te CRR of te SMLSF over te LSF. used at all te oter levels of te signature tree and is based only on te domain objects included in te image. Te analytical comparison of te SMLSF and te LSF is given, bot for storage requirement and searc performance. Te SMLSF acieves a large storage reduction over te LSF metods. A closed-form formula for te storage reduction ratio was derived and was found to be a function of te number of objects per image and, to a less extent, on te number of images in te database. Te storage reduction increases linearly wit te number of objects in te image and logaritmically wit te number of images in te database. For example, a storage reduction of about 78% can be acieved for an image database of GB images and 0 objects per image. Te SMLSF can answer bot general and specific queries, wile te LSF can only answer specific queries. An example of a specific query is Given a query image, find all images wic include te set of objects and satisfy te spatial relations in te query image. An example of a general query is Given a query image wit a set of objects, find all images wic include te set of objects in te query image. Te performance of te SMLSF, measured by te number of signatures and te number of bits compared during query processing, is substantially better tan te LSF. For example, a computation reduction of about 98% can be acieved for an image database of GB images and 0 objects per image. APPENDIX A. LIST OF SYMBOLS A number of signatures searced (at all levels).