RETRIEVAL OF TRADE MARK IMAGES BY SHAPE FEATURE - THE ARTISAN PROJECT

Transcription

1 RETRIEVAL OF TRADE MARK IMAGES BY SHAPE FEATURE - THE ARTISAN PROJECT J P Eakins *, J M Boardman and K Shields Introduction As more and more applications in both the commercial and scientific field make routine use of pictorial data, it has become increasingly apparent that the problems involved in organizing image collections for efficient storage and retrieval are far from trivial. Hence interest in research into image content retrieval has grown rapidly over the last few years 1. The field has provided a rich variety of research problems, including image encoding, storage, compression, transmission, display, and shape description and matching for image retrieval. Many different approaches have been taken, some focusing on highly specific topics such as improved patternmatching algorithms, others taking a much broader view, and attempting to study the issues involved by developing and evaluating prototype systems. Work on image retrieval at Northumbria University falls mainly into this second category. Our long-term aim is to investigate the principles on which workable shape retrieval systems can be constructed. Our approach has been essentially pragmatic - to develop a series of steadily more sophisticated prototypes capable of handing an increasingly broad range of image types and retrieval paradigms. Our first project, the development and evaluation of SAFARI (Shape Analysis For Automatic Retrieval of Images), a prototype shape retrieval system for two-dimensional engineering drawings of simple machined parts, demonstrated that we could achieve excellent retrieval performance within a restricted class of images 2. The aim of our current ARTISAN (Automatic Retrieval of Trademark Images by Shape ANalysis) project is twofold: to develop and evaluate a prototype shape retrieval system for trade mark images consisting of abstract geometric designs, and to assess the feasibility of generalizing the image retrieval approach developed in our earlier SAFARI system to a wider range of image types. It is hoped that the prototype system developed will provide a basis from which the Patent Office Trade Marks Registry can develop an improved image retrieval system for their own use. The trademark image retrieval problem The UK Patent Office has been responsible since 1876 for registering all UK trademarks. Their registry now holds over current trademarks, around 40% of which contain some form of image data. Trademarks are an important part of a company s industrial property, and it is a crucial responsibility of the Trade Marks Registry to ensure that all new trademarks registered are sufficiently distinctive to avoid confusion with existing marks. Since its inception, the Trade Marks Registry has classified trademark images using an elaborate system of manually-assigned codes. These codes form the basis of their TRIMS image retrieval system, which allows users to specify Boolean combinations of category codes and display all images meeting the search criteria. Many trademark images are intended to depict animate or inanimate objects, such as cats, stars, or flowers, and TRIMS works well in such cases. However, a sizeable fraction (now numbering well over ) are made up of abstract geometric designs with little or no representational meaning. Current indexing practice is to code for the presence of recognizable geometric shapes such as circles, triangles, or squares. This provides a partial solution to the problem, but since there are now several thousand images in each category, Registry staff attempting to establish the novelty of a trademark based on an abstract design are faced with an almost unmanageable task. There is thus a need for a system which provides reliable (and if possible automatic) indexing and retrieval for this class of images. * Department of Computing, University of Northumbria at Newcastle, Newcastle upon Tyne NE1 8ST, United Kingdom

2 Fig. 1 illustrates some typical abstract trademark images. Most consist of several distinct components. Virtually all are monochrome, and few contain textured areas. Retrieval of such images thus has to rely purely on shape matching. This may appear to simplify the similarity matching problem, but the converse is in fact true. Image colour and texture have proved to be excellent sources of robust indexing features, and systems such as IBM's QBIC (Query By Image Content) can yield quite impressive colour retrieval results 3. Shape retrieval has proved a considerably greater challenge, despite considerable research into the topic 4. It appears that few, if any, of the shape feature measures in current use are accurate predictors of human judgements of shape similarity 5 a b c d Fig 1. Some typical trademark images. Crown copyright reserved. Hence the success of the ARTISAN project depends on finding a solution to the difficult problem of shape similarity matching of multi-component images. Our approach to this problem has been to draw on some of the results of human visual cognition studies, particularly those of Biederman 6. Following the principles of Gestalt psychology 7, we hypothesise that image elements that are perceived as groups should be explicitly represented as such. In Fig 1b, for example, the curved bars at the top could be regarded as twelve individual bars, four wedgeshaped blocks, three concentric circular arcs, or - perhaps most plausibly - as one thick circular arc. In Fig 1c, the four B-shaped blocks clearly have some significance in determining the shape characteristics of the image, as do the eight D-shapes. We consider that the best way to treat these images is to declare such component groups as families, and represent them explicitly as image elements. Extrapolating Biederman s findings slightly, we propose that image components which have recognizable boundaries should be grouped into boundary families 8 where they meet one or more of the following conditions: 1. Boundaries are in close physical proximity; 2. Significant lengths of their boundaries are collinear or parallel; 3. Significant lengths of their boundaries are derived from concentric arcs; 4. Boundaries exhibit some degree of symmetry or shape similarity. Our current implementation of this concept is discussed below. Architecture of the ARTISAN system The overall functionality required from ARTISAN is similar to that of our earlier SAFARI system: to accept images in an appropriate standard format; to build up a database of stored image descriptions from these images; to extract retrieval features from these descriptions; to allow formulation of visual queries; to provide efficient and effective matching of query and stored images; to display query results in an appropriate format. However, the images which ARTISAN needs to handle are much less constrained than those in SAFARI 9. They do not necessarily have a single outer boundary defining their overall shape, enclosed regions may contain or overlap other regions, region boundaries may be impossible to describe using simple mathematical expressions, and features perceived by the eye may be implied rather than explicitly represented in the original

3 image (such as the S-shaped void in Fig 1d above). Above all, since input images are supplied as bitmaps resulting from scanning under variable conditions, they may contain elements of noise or other forms of distortion. The present ARTISAN system does not resolve all the problems resulting from this wider range of image types. In particular, it makes no attempt to recognize the majority of implied shape features. However, it does aim to build up what we believe to be a sufficiently rich and robust description of the salient features of each stored image to permit effective similarity retrieval. The current system consists of the following modules: a) Extraction of region boundaries from bitmap images and approximation by straight-line and circulararc segments. This module aims to identify regions of interest within each image, and characterize each region by an approximation of its boundary which is both sufficiently faithful to capture its major shape elements and sufficiently flexible to support subsequent processing. It generates a representation based on line segments which can be interpreted either as straight lines or circular arcs. This choice is governed largely by the need to test for alternative Gestalt interpretations of each boundary, which requires us to detect collinear or co-curvilinear line segments in distant regions of the image. Input images are first converted to uncompressed TIFF format. Each image is then scanned with a Laplacian kernel to produce a single, closed boundary for each discrete homogeneous region in the image. These boundaries are initially represented as chains of adjacent pixel coordinates, and then converted to a straight-line and circular-arc approximation using a technique based on that of Rosin and West 10. b) Reprocessing of boundary representations to remove anomalies caused by noise in the original image. The boundary segment representations generated by the above module are inevitably susceptible to noise in the original image. The presence of spurious segments can distort calculations of the characteristic shape measures used for feature indexing. This module therefore eliminates such segments as far as possible by invoking a set of boundary redrawing rules to remove or reclassify them. Our approach is based on the Gestalt principles outlined above, and uses a rule set based on the redrawing rules developed for SAFARI, widened to take account of the effect of collinear or co-curvilinear segments in other regions within the image. An example of this process is shown in Fig 2. Fig 2a - Image with extraneous short boundary segments Fig 2b. Image from Fig 2a after application of redrawing rules. Note that all short lines have been removed c) Grouping of region boundaries into families. This module groups boundaries into families which potentially mirror human image perception, as outlined above. It uses clustering techniques to group distinct image regions into two separate classes of family. Proximal families are identified by clustering region boundaries on the basis of proximity, parallelism and concentricity scores, and shape families by clustering boundaries on the basis of shape similarity. Since we expect the eye to perceive these two types of family in different ways, they are treated quite differently in subsequent processing. An example of proximal family formation is shown in Fig 3.

4 1 2 Fig 3. Boundaries from the image shown in Fig 2, grouped by ARTISAN into two families on the basis of proximity and parallelism. Fig 4. Envelopes fitted to the boundary families illustrated in Fig. 3 d) Construction of envelopes for proximal boundary families. This module constructs an envelope for every proximal family identified in the previous step, using a novel techniques which preserves concavities in the family s constituent boundaries (Fig 4). e) Extraction and storage of global shape features. This module derives a set of shape features from the image at three different levels: the entire image, each proximal family, and each individual boundary. We are still experimenting with alternative sets of shape features and ways of associating them with image components. At present, we compute four shape features (aspect ratio (p 1 + p 2 ) / C, circularity 4πA / P 2, transparency A / H, and relative area (A / L)) directly from each boundary envelope, and five further measures (complexity 10 - (7 / n), right-angleness r / n, sharpness Σ max(0, 1 - (2 θ i - π / π) 2 ) / n, straightness S / P and directedness M / P) from individual boundaries within each family, where A = area of polygon enclosed by segment boundary C = length of longest boundary chord H = area of polygonal convex hull of boundary L = area of boundary with largest area of all image boundaries M = total length of straight-line segments parallel to mode direction of straight-line segments, within a specified tolerance n = number of sides of polygon enclosed by segment boundary P = perimeter of polygon enclosed by segment boundary p 1, p 2 = greatest perpendicular distances from longest chord to boundary, in each half-space either side of line through longest chord θ i = discontinuity angle between (i-1) th and i th boundary segment r = number of discontinuity angles equal to a right-angle within a specified tolerance S = total length of straight-line segments f) Database query. This module allows the user to select a query image and run-time search parameters, extracts appropriate shape features from the query image, computes appropriate similarity scores between query and stored images by shape feature matching, and displays the most similar retrieved images on the screen. The similarity matching algorithm used is an adaptation of the matching algorithm used for our earlier SAFARI system. Retrieval results and evaluation So far, a small database of around 200 images and their associated shape files has been constructed using the techniques described above, so that queries may be performed. Until a detailed evaluation is performed, it is impossible to draw firm conclusions about the system s retrieval performance. However, initial results are encouraging.

5 Specimen retrieval results from the present version of ARTISAN are shown in Fig 5. A query image is processed in the same way as the existing images in the database to extract its shape features. It is then possible to match query and stored images, using a variety of run-time retrieval options as shown in Fig 5(a). After this, the program proceeds to access every shape file in the database so that it may be compared with the query. Finally all the images are displayed as shown in Fig 5(b), ranked in order of similarity to the query. Fig 5a. An ARTISAN query screen, showing a query image (top left) and run-time retrieval options. Fig 5b. Retrieval results from the query illustrated in Fig 5a, showing the ten most similar retrieved shapes in similarity order. The query shape is also included for comparison purposes Perhaps the most important part of our project is the evaluation of ARTISAN s retrieval effectiveness. Thanks to the co-operation of the UK Patent Office, we have the opportunity to compare the performance of our system with that of experienced trademark examiners. We shall be making use of this opportunity in two ways. Firstly, we aim to get informal feedback on the effectiveness of our initial prototype by putting to it a number of past trademark image queries, and asking trademark examiners to comment on the system s similarity rankings. As a result of this feedback, we will modify our feature set and shape matching routines in order to optimize system performance as far as we can. This improved prototype will then be used to build a database of over abstract trademark images from the Registry. Secondly, we have agreed a protocol with the Patent Office for a formal evaluation experiment, in which a set of queries are put to the test database above, using both ARTISAN and the Patent Office s existing TRIMS system. A panel of trademark examiners will judge which stored trademark images (if any) are confusingly similar to each query. These judgements will then be used to assess the retrieval effectiveness of each system. We expect these evaluation experiments to be sufficiently comprehensive to permit reliable judgements about the overall validity of our approach. We also hope to be able to draw conclusions about the retrieval effectiveness of alternative matching techniques and types of shape feature. Conclusions The ARTISAN shape retrieval system discussed in this paper combines a number of conventional features with aspects which we believe to be novel, particularly in the application of ideas drawn from Gestalt psychology. Until the system has been independently evaluated, it is too early to tell whether our approach to shape analysis and matching will yield reliable retrieval results. However, we believe that the approach can

6 provide some worthwhile insights into the image retrieval problem, and has considerable potential for development. Acknowledgements Thanks are due to the British Library and the UK Patent Office for their financial support. References 1. Gudivada, V N & Raghavan, V V, eds (1995) "Content-based image retrieval systems" IEEE Computer, 28(9), Eakins, J P (1993) "Design criteria for a shape retrieval system" Computers in Industry 21, Flickner, M et al (1995) "Query by image and video content: the QBIC system" IEEE Computer, 28(9), Niblack, W et al (1993) "The QBIC project: querying images by color, texture and shape" IBM Research Report RJ Scassellati, B et al (1994) "Retrieving images by 2-D shape: a comparison of computation methods with human perceptual judgements" in Storage and Retrieval for Image and Video Databases II (Niblack, W R & Jain, R C, eds), Proc SPIE 2185, pp Biederman, I (1987) "Recognition-by-components: a theory of human image understanding" Psychological Review 94(2), Lowe, D G (1985) Perceptual organization and visual recognition Kluwer, Boston 8 Eakins J P, Shields K, & Boardman J M (1996) "ARTISAN - a shape retrieval system based on boundary family indexing" in Storage and Retrieval for Image and Video Databases IV, (Sethi, I K & Jain, R C, eds), Proc SPIE 2670, pp Eakins, J P (1994) "Retrieval of trade mark images by shape feature" Proc First International Conference on Electronic Library and Visual Information System Research, de Montfort University, Milton Keynes, Rosin, P L & West, G A W (1989) Segmentation of edges into lines and arcs Image and Vision Computing 7(2),