FVS: A Content-Based Retrieval Library. Simon Goodall

Transcription

1 FVS: A Content-Based Retrieval Library Simon Goodall June 28, 2006

2 Contents 1 Acknowledgements 5 2 Introduction 6 3 FVS Introduction Supported file types Dependencies Platform support MySQL Considerations Linux Considerations Win 32 Considerations Architecture Adding new algorithms Adding new file formats Using FVS On the command line MySQL Module fvs generatefv fvs comparefv fvs normalise fvs update norm FVS Library The 2-D Algorithms The CCV Matching Module Module Features Module Properties Module Parameters Technical Description Example Results The Colour Histogram Matching Module Module Features Module Properties

3 4.2.3 Module Parameters Technical Description Example Results The Lab Colour Histogram Matching Module Module Features Module Properties Module Parameters Technical Description The Colour Picker Matching Module Module Features Module Properties Module Parameters Technical Description Example Results The Mono-Histogram Matching Module Module Features Module Properties Module Parameters Technical Description Example Results The Hu Moments Matching Module Module Features Module Properties Module Parameters Technical Description The MCCV Matching Module Module Features Module Properties Module Parameters Technical Description Example Results The Multi-Scale Mono-Histogram Matching Module Module Features Module Properties Module Parameters Technical Description Example Results The Pyramid Wavelet Transform Matching Module Module Features Module Properties Module Parameters Technical Description Example Results The Query by Fax Matching Module Module Features Module Properties

4 Module Parameters Technical Description Example Results The MPWT Matching Module Module Features Module Properties Module Parameters Technical Description The 3-D Algorithms The Area Volume Matching Module Module Features Module Properties Module Parameters Technical Description Example Results The Cord Histogram Matching Modules Module Features Module Properties Module Parameters Cord Hist Cord Hist Cord Hist Cord Histogram Technical Description Example Results The ShapeD2 Matching Module Module Features Module Properties Module Parameters Technical Description Example Results The EGI Matching Module Module Features Module Properties Module Parameters Technical Description Example Results The 3D Hough Matching Module Module Features Module Properties Module Parameters Technical Description Example Results

5 6 Distance Metrics and Normalisation Minkowski Norms Histogram Intersection Bhattacharyya Distance Kullback-Leibler Divergence Chi-Squared Quadratic Distance Normalisation

6 Chapter 1 Acknowledgements The images in this document have come from various sources which have been noted where they are used. Many of the images for the 2-D algorithms come from the Victoria and Albert Museum. For the 3-D algorithms, images come from both the SCULPTEUR database (not fully public) and from the Princeton Shape Benchmark princeton.edu/benchmark/. Copyright for these images remains with their respective owners. 5

7 Chapter 2 Introduction This document provides the manual to FVS, the 2-D image and 3-D object processing library in the developed for the SCULPTEUR project. This document begins with a description of the FVS library itself. This is then followed by a description of each of the 2-D and 3-D algorithms in FVS, including any optional parameters they can take. Finally a description of the distance metrics in FVS is presented along with a description of the score normalisation technique. Publications relating to the 3-D work in SCULPTEUR include [9, 8, 3, 2]. 6

8 Chapter 3 FVS 3.1 Introduction FVS is the name of the library providing the 2-D image and 3-D object processing functions. FVS is based upon the FVG library produced as part of the ARTISTE project [12]. The main differences between FVS and FVG are that FVS supports 3-D objects and it also provides a MySQL [1] module that allows generation and comparison of feature vectors stored directly in a database table. FVS provides the ability to generate feature vectors describing features of 2-D and 3-D objects, comparison between feature vectors, and a thumbnail generator to provide a small view of a query object. FVS is a library with a command line interface and MySQL module to provide the interfaces. 3.2 Supported file types Technically, FVS can read 2-D images in any format supported by ImageMagick, an image processing library used by VIPS [6] to load images. In the distributed binaries however, only JPEG [10] and TIFF [18] file formats are supported due to the complexity of compiling the necessary dependencies on all target platforms. FVS supports three 3-D file formats. These are VRML 97 [5] (both compressed and uncompressed), OFF, and TRI. It must be noted here that VRML 1.0 is not a supported 3-D file type. 3.3 Dependencies FVS has two main dependencies; these are VIPS for 2-D image loading and some processing operations, and Cyber X3D [11] for parsing VRML 97 objects. These two libraries however have their own dependencies. 7

9 Due to more recent additions to the VIPS library that add a pthread dependency in VIPS 7.10.x, VIPS 7.8.x is the best version to use with the MySQL module. 3.4 Platform support FVS is supported on both Linux platforms and Win 32 platforms. It has also been run on Mac OS-X. Providing the dependencies can be met, FVS should work on additional platforms. The feature vector I/O code ensures that data is stored in an endian independent manner which should allow FVS to run on Big Endian architectures MySQL Considerations MySQL can dynamically load modules to add functionality. The new functionality can be accessed through a normal SQL based query. There are several caveats to producing a module. The first is that the module should be compiled statically, that is so that all the module code, and that of its dependencies are compiled into a single file. It might be possible to get a module to dynamically load its dependency libraries, however this would require the correct versions to be available on the system, and have MySQL configured so that it can find them. Initial experiments in distributing FVS like this proved problematic. The second caveat is that as MySQL is a C based application, certain C++ features cannot be used. The main feature is C++ exceptions, which cause MySQL to crash as soon as an exception is thrown. The FVG library used the C++ version of VIPS which used exceptions for error handling. FVS was re-written to use the C version of VIPS to solve this problem. The third caveat is that the module needs to be thread safe, and cannot be compiled with pthreads. This is especially important as VIPS makes use of threads to improve its performance, and several of its dependency libraries make use of pthreads Linux Considerations The last couple of years have seen many advances in the GNU C++ compiler [16], gcc. Unfortunately, these advances have broken compatibility between the various versions of the compiler. Typically this means that programs compiled on the more recent versions of gcc will not work on systems hosting older versions of gcc. However programs compiled on the older gcc versions will work on systems hosting the newer versions Win 32 Considerations As most of the software is written in Linux, FVS is cross-compiled into a Win 32 DLL. This requires that all dependencies can be cross-compiled. Not every 8

10 library is in a format that easily allows this (or is able to be cross-compiled), and as such has led to the small number of 2-D image formats supported by FVS. 3.5 Architecture FVS is written as a library and has two front-ends to provide access to its functionality. The main front-end is the MySQL module used in the core of the SCULPTEUR system. The other front-end is a command line tool Adding new algorithms Adding a new algorithm is fairly straight forward. However there are several files that need to be edited to first compile the new algorithm code, and secondly tell fvs that it is available. FVS uses the autoconf [14] and automake [15] for its build system. Adding a new algorithm requires editing the existing configure.in and Makefile.am files, and creating a new one for the algorithm. The first stage to creating a new algorithm is to take an existing algortihm (such as CCV for a new 2D algorithm, or ShapeD2 for a new 3D algorithm) and copy the all the code files and the Makefile.am into the new algorithm directory. Typically the structure is as follows; fvs-version/fvs/algname/makefile.am fvs-version/fvs/algname/fa_algname.cpp fvs-version/fvs/algname/fa_algname.h fvs-version/fvs/algname/fv_algname.cpp fvs-version/fvs/algname/fv_algname.h Code files prefixed with FV denotes the code files for the class that handles the storeage of the feature vector and its reading/writing functions. Code files prefixed with FA denotes the code files for the class that handles the generation and comparison of the feature vectors. These files need to exist and have the appropriate defininations. Additional code files can be used if required. Once copied and appropriately renamed, Makefile.am needs to be updated for the new file names, and the class definitions need to be updated. The code should also be changed to use the algorithm code. Then fvs-version/fvs/makefile.am needs to have the new algorithm directory added to the list of subdirs, specified on the line beginning with SUBDIRS = ColourHistogram.... The.la file created for the algorithm also needs to be added to the libfvs 1 0 la LIBADD declaration. Finally, configure.in needs to add the new Makefile to its list of generated files. Add fvs/algname/makefile to the end of the list in the AC OUTPUT block. This completes the addition of the algorithm to the build system. To tell FVS about the new algorithm, the header files need to be added to the list 9

11 of includes in fvs/fv.h and a REGISTER FV(AlgName) needs to be added at the bottom of the file. Finally, fvs/ifipids.h defines the ID and name for each algorithm. Add two #define statement in the form #define AlgName Name AlgName and #define AlgName ID XXXX where AlgName is the name of the algorithm (no spaces and the first character must be a letter) and XXXX is a unique ID number. Once completed, the new algorithm should now be integrated into FVS. The existing algorithm implementations should be used as examples on working with the internal functions Adding new file formats Adding new 2-D file formats to FVS requires making the appropriate changes to Vips and is not an issue for FVS. Adding new 3-D file formats however is a direct issue for FVS. FVS has an internal 3-D representation that each of the file format loaders needs to be able to produce. This format stores all of the index, vertex, normal and texture information. Any transformations need to be applied to the data before passing to FVS as they cannot be stored directly in the representation. 3.6 Using FVS FVS can be used on the command line, as a library, or as a MySQL module On the command line FVS has two main modes of functionality. These are generate and compare. To generate a feature vector, FVS can be invoked with the following command; fvs -f <algorithm name> [-m] -g <input file name> [-o <output file name>][--ascii] where <algorithm name> is to be substitued with the algorithm name or ID, <input file name> is the path to the input 2-D image or 3-D object. Optionally, <output file name> specified the filename of the resulting feature vector. By default this will be <input file name>.<algorithm name>[.ascii].ifo. The ascii option specifies to create an ASCII based feature vector, rather than a binary version. This is denoted by the [.ascii] part of the default feature vector name. [-m] is a flag to enable the multiscale wrapper for 2-D algorithms. To compare two feature vectors, FVS can be invoked as follows; fvs -f <algorithm name> [-m] -c <query file name> <reference file name> 10

12 where <algorithm name> is to be substitued with the algorithm name or ID, <query file name> is the path to the query feature vector and <reference file name> is the path to the reference feature vector. [-m] is a flag to enable the multiscale wrapper for 2-D algorithms. Additionally, both these commands can take a string of options to pass to the algorithm. These can either be integer values, real values or string values. Appending integer i will pass an integer with the value i to the algorithm. Similarly use real r for real numbers and string str for string values. See the algorithm descriptions for details on what parameters to pass to the algorithms. Please note that the ordering of parameters is important. Finally, normalisation can be used by using one of several additional arguments. To initialise a normalisation blob, call fvs as follows; fvs --init-norm <norm file name> where <norm file name> is the file name for the resulting normalisation blob. By appending integer i and real r the number of bins and the maximum score value can be specified respectively. To update a normalisation blob, perform a comparison as normal, and use the update-norm argument as follows; fvs -f <algorithm name> [-m] -c <query file name> <reference file name> --update-norm <norm file name> where <norm file name> is the file name of the normalisation blob. To normalise a score, perform a comparison as normal and use the normalisescore argument as follows; fvs -f <algorithm name> [-m] -c <query file name> <reference file name> --normalise-score <norm file name> where <norm file name> is the file name of the normalisation blob MySQL Module The module has four functions. Below is the standard interface to the functions. Additionally both the fvs generatefv and fvs comparefv functions can take a list of optional arguments specified after the required options. See the details of each algorithm for what options are available fvs generatefv This function generates a feature vector of the specified type for the object in the given filename. If the multiscale wapper flag is set to 1, then the wrapper module is used. Returns STRING: The generated feature vector blob, or NULL on error. 11

13 Funtion parameters: Type What Description STRING / IN- Feature Vector This is the name or ID of the feature TEGER Name / ID vector type. INTEGER Multiscale The flag indicates whether to use the wrapper flag multiscale wrapper. (0 for false, 1 for true). STRING Input file name This is the filename of the file to generate the feature vector from fvs comparefv This function takes two feature vector blobs and returns a dissimilarity score. Returns REAL: The dissimilarity score. If score is less than 0, an error occurred. Error Code Reason -1.0 General Error during comparison One or both fv s are NULL pointers -5.0 Failed to create FeatureAPI object Error loading memoryblob into fv object Error loading memoryblob into fv Feature vector has zero length Function parameters: Type What Description STRING / IN- Feature Vector This is the name or ID of the feature TEGER Name / ID vector type. INTEGER Multiscale The flag indicates whether to use the wrapper flag multiscale wrapper. (0 for false, 1 for true). STRING Query FV Blob The query feature vector blob. STRING Reference FV The reference feature vector blob. Blob fvs normalise This function normalises a score according to the specified normalisation blob. Returns REAL: The normalised score. Error Code Reason -1.0 Normalisation Blob is NULL -2.0 Normalisation Blob has no length -3.0 There was an error parsing the blob Function Parameters: Type What Description STRING Normalisation Blob This is the blob containing the normalisation data REAL Score The score to normalise 12

14 fvs update norm This function adds a score to the normalisation blob. If the input blob is NULL, then a new blob is created. Returns STRING: The updated blob. Returns NULL on error. Type What Description STRING Normalisation Blob This is the blob containing the normalisation data REAL Score The score to add to the blob FVS Library All of the functionality of FVS is contained within the FVS library. The FVS command line and FVS MySQL Module are front-ends to this library. Please refer to these for examples in using the FVS library. 13

15 Chapter 4 The 2-D Algorithms 4.1 The CCV Matching Module Algorithm By: Stephen Chan Name: CCV ID: Module Features The CCV module [13] allows retrieval of similar images based on the general colour distribution of the image, with discrimination between colours in images which are homogeneous to some sizable area. The module is good for retrieval of images based on a known query image where the retrieved images are required to have a similar mix of colours. It is not suitable for retrieval of images based on a query image which is only part of a complete image Module Properties Module Speed: Module Accuracy: Fast Medium Module Parameters Generate: Compare: This module has no generation parameters. This module has no comparison parameters Technical Description The CCV stands for Colour Coherence Vector. A coherent region of colours in an image is a region of colour which is larger than some threshold. This module retrieves images which have similar distributions of coherent colours. 14

16 A histogram of 64 bins (4x4x4) is generated for both coherent and incoherent colours and these are matched separately. As for Colour-Histogram, 64 bins has been chosen as a trade-off between accuracy and speed. The total size of the feature vector for a CCV is therefore 64x2 integer numbers (512 bytes per feature). Coherence and incoherence are arbitrarily defined as greater and less then 5% of the total image area, respectively. This means if a pixel is part of a region which is less than 5% of the total image area it is added to the incoherent histogram within the CCV. If it is greater than 5% of the total image area it is added to the coherent histogram within the CCV. Here is an example: The chessboard image is 50% black and 50% white, arranged into 64 squares, where 32 are white and 32 are black. Each square constitutes a region each of which is 1/64th of the total image area. This is approximately 1.5% of the image area. They are therefore considered incoherent and stored into the incoherent histogram, leaving the coherent histogram empty. The second image contains the same amount of black and white (50% of each), however, the two regions are coherent and are therefore stored into the coherent histogram, leaving the incoherent histogram empty. When each coherent and incoherent histogram is matched against each other, these two images will have a large distance, as they should indeed do, unlike a standard histogram function which would give a distance of zero implying they were identical Example Results The following is an example query giving what would be considered good results. Note: The matching is based only upon the general colour distribution of the image, with discrimination between colours in images which are homogeneous to some sizable area. 15

17 Query Images courtesy of Victoria and Albert Museum In the above example, the query image is contained within the database. It is therefore expected to be found in first place, as it is. This is a good test of the integrity of the algorithm. Also, all the retrieved results contain areas of contiguous colour similar to that of the query image. Note that the contiguous white background will alter the match, because it would become a coherent region within the vector. The CCV is a whole-image query algorithm, and cannot be used to find sub-images, because it is likely that the sub-image will have a different colour distribution. An image which is a sub-image of a database image may be used as the query but the parent from which it was derived is unlikely to be found. The query below shows an example of using an image which is a sub-image of an image in the database as the query. Notice the results all have a similar colour distribution (white/yellow), but it did not find the image from which the query image was taken. 16

18 Query Images courtesy of Victoria and Albert Museum Used on its own, you should definitely not expect the CCV algorithm to be able to find specific instances of objects (e.g. chairs, pots, etc). However, used with a metadata search to locate similar types of object, this algorithm could locate those of a similar colour. 4.2 The Colour Histogram Matching Module Algorithm By: David Dupplaw Name: ColourHistogram ID: Module Features The Colour Histogram module [17] allows retrieval of similar images based on the general colour distribution of the image. The module is good for retrieval of images based on a known query image, if the retrieved images are required to have a similar mix of colours. It is not suitable for retrieval of images based on a query image which is only part of a complete image Module Properties Module Speed: Module Accuracy: Fast Low 17

19 4.2.3 Module Parameters Generate: Compare: This module takes 3 positive integer parameters specifying the number of bits for the reg, green and blue colour channel respectively. This module has no comparison parameters Technical Description This histogram matching algorithm simply uses the frequency of occurrence of each colour of the histogram within the image. The more of a particular colour an image contains the higher its frequency will be within the histogram. The histogram is made with 64 bins, a compromise between speed (it takes longer to match more bins), and accuracy (the less bins the less discriminating the results would be - i.e. images which are less similar would have lower distances). Before matching histogram are normalised so that the area underneath the histogram is unity (normalised by the number of pixels in the image). This means that the size of the image will not affect the match (other than any expected aliasing). Here is an example: Vase Images courtesy of Victoria and Albert Museum Note in the second example, how the background is dominant in the image. Because the colours of the pots are spread fairly evenly over the bins, and the background is pre-dominantly one colour, and therefore one bin of the histogram, once the histogram is normalised the background dominates the vector. This is 18

20 the main drawback of histogram matching: the background information is all included within the feature and it cannot be ignored. In the first example, the background is more evenly distributed over bins (due to the shading) and the object groups into a few bins (due to its flat colour distribution). Therefore the background does not dominate the histogram so much Example Results The following is an example query giving what would be considered good results. Note: The matching is based only upon the general colour of the images. Query Images courtesy of Victoria and Albert Museum Because the histogram is such a basic algorithm, there are no cases where it should give unexpected results. However, notice that the background (which is effectively noise in this context) affects the match, so that the results all contain large amounts of white, and some green and browns. Used on its own you should definitely not expect the ColourHistogram algorithm to be able to find specific instances of objects (e.g. chairs, pots, etc). However, when used with a metadata search to locate similar types of object, this algorithm could locate those of a similar colour. 19

21 4.3 The Lab Colour Histogram Matching Module Algorithm By: David Dupplaw Name: LabColourHistogram ID: Module Features The module allows retrieval of similar images based on the general LAB colour distribution of the image. The module is good for retrieval of images based on a known query image, if the retrieved images are required to have a similar mix of colours. It is not suitable for retrieval of images based on a query image which is only part of a complete image Module Properties Module Speed: Module Accuracy: Fast Low Module Parameters Generate: Compare: This module takes three integer values specifying the number of bits for the L, A and B colour channels respectively. This module has no comparison parameters Technical Description This histogram matching algorithm simply uses the frequency of occurrence of each LAB colour of the histogram within the image. The more of a particular colour an image contains the higher its frequency will be within the histogram. The histogram is made with 64 bins, a compromise between speed (it takes longer to match more bins), and accuracy (the less bins the less discriminating the results would be - i.e. images which are less similar would have lower distances). Before matching histogram are normalised so that the area underneath the histogram is unity (normalised by the number of pixels in the image). This means that the size of the image will not affect the match (other than any expected aliasing). 4.4 The Colour Picker Matching Module Algorithm By: David Dupplaw Name: ColourMatch ID:

22 4.4.1 Module Features The module allows retrieval of images based on the images containing certain amounts of colours. The colours need not be sizable regions, or be within the same context within images (a property which histograms impose). The results are ordered based on how similar the amounts of colours in the image are compared to the colour selection. The module is good for retrieval of images based on a single or multiple colour queries Module Properties Module Speed: Module Accuracy: Very Fast High Module Parameters Generate: Compare: Not applicable - No generate function This module takes a real value setting the fuzziness of the comparison. It can also take three integer values specifying an RGB colour. If both sets of parameters are specified, then fuzziness must be first Technical Description The colour picker uses histogram features generated by the ColourHistogram or LabColourHistogram module. It matches single or multiple colours in the histogram in a similar way to the colour histogram matching code, except that it matches only those colours in the query histogram. A histogram (of any size) is generated for each object in the database. Currently we use a 64 bin RGB colour histogram (4x4x4) or a 216 bin Lab colour histogram (6x6x6). When the colour picker is invoked, an appropriate histogram is generated from the selected colours. The values in the histogram represent the values selected. Figure 4.1: Red colour selection, gives a histogram with a single red bin populated 21

23 Figure 4.2: A two colour selection gives a histogram with two bins populated Figure 4.3: The amounts of selected colour are reflected in the histogram The matching matches each populated bin in the query histogram with the matching bin in the match histogram. Because the idea is to find images containing these colours, it is pragmatic to disregard, or give a large score, to those images which do not contain those colours. This requires a predicate for colour inclusion. The predicate is based on a percentage value of the given amount of colour (a fuzziness rating). For example, if the fuzziness rating was 50%, a bin in the match histogram would be considered a result if the value fell within +/- 50% of the value of the query histogram s bin. By default the rating is near to 100%, so that if a bin contains any amount of the given colour it will be considered a match, although results will be ordered based on how similar the amounts are to the query. The fuzziness should not be 100% else a zero amount of the colour will match. If the match histogram s bin value falls outside this range (or is zero) then the match is given a maximum score (disregarded as a match because it does not contain all the colours specified) and no more matching takes place for this feature Example Results The following is an example query giving what would be considered good results. Note: The matching is based upon the inclusion of the selected colour. 22

24 Query Images courtesy of Victoria and Albert Museum Query Images courtesy of Victoria and Albert Museum Note that many of the images contain colour charts, and this could affect the match for obvious reasons (see result 6 of first match, and result 5 of second 23

25 match). 4.5 The Mono-Histogram Matching Module Algorithm By: Stephen Chan and David Dupplaw Name: HistogramMono ID: Module Features The module allows retrieval of similar images based on the general distribution of brightnes in the image. It is really only suitable for monochromatic images, such as monochrome photography or radiography. A monochrome histogram contains the frequency (i.e. number of pixels) of each brightness level (of which there are 64 in this module) that occurs within the image. The module is good for retrieval of images based on a known query image if the retrieved images are required to have a similar overall brightness distribution. It is not suitable for retrieval of images based on a query image which is only part of a complete image Module Properties Module Speed: Module Accuracy: Fast Low Module Parameters Generate: Compare: This module takes a single integer specifying the number of histogram bins. This module has no comparison parameters Technical Description This monochrome histogram matching algorithm simply uses the frequency of occurrence of each level of brightness of the histogram within the image. The more of a particular brightness an image contains the higher its frequency will be within the histogram. Colour images in the database are converted to monochrome for matching with this algorithm, by converting RGB values to monochrome with (R+G+B)/3. The histogram is made with 64 bins, a compromise between speed (it takes longer to match more bins), and accuracy (the less bins the less discriminating the results would be - i.e. images which are less similar would have lower distances). Before matching histograms are normalised so that the area underneath the histogram is unity (normalised by the number of pixels in the image). This 24

26 means that the size of the image will not affect the match (other than any expected aliasing). Here is an example: Images courtesy of Victoria and Albert Museum As with the ColourHistogram the background has an effect on the histogram, and therefore on the matching. In the second example the dark background is increasing the frequencies of the dark bins in the histogram. The reason a monochromatic histogram is required, is that the colour histogram is not discriminating enough for monochromatic images. A 64 bin colour histogram has only 4 bins dedicated to grey-scale values. This means that most grey-scale images would look similar in a colour histogram. The histograms below show an image in both RGB and monochrome histogram space. 25

27 Images courtesy of Victoria and Albert Museum Example Results The following is an example query giving what would be considered good results. Note: The matching is based only upon the general distribution of brightness in the image. Query Images courtesy of Victoria and Albert Museum The monochrome histogram is specifically designed for matching with monochrome images. It will also work with colour images, however. The necessity for having both colour and monochrome histogram is that the colour histogram, being de- 26

28 signed for colour, is very inaccurate for grey-level images. The monochrome histogram algorithm is around 13 times more discriminating between monochrome images than the colour histogram is between monochrome images, because it contains 13 times more grey-levels. The above example shows a large monochromatic image being used as a query within a small 1000 set database containing both black and white and colour images. Because the scope of the query was not limited to monochrome images, the retrieval found both colour and monochrome images. Notice that the monochrome images it found belonged to the same painting. This will not necessarily happen in all datasets, as it depends on the deviation of the query from the other images. Used on its own you should definitely not expect the monochrome histogram algorithm to be able to find specific instances 4.6 The Hu Moments Matching Module Algorithm By: M Faizal A Fauzi Name: HuMoments ID: Module Features The Hu-Moments are a set of seven rotation, scale and translation invariant moments Module Properties Module Speed: Module Accuracy: Fast <accuracy> Module Parameters Generate: Compare: This module has no generation parameters. This module has no comparison parameters Technical Description This module first converts an image into a black and white representation setting the threshold at the average image intensity. The moments are then applied to the converted image. 4.7 The MCCV Matching Module Algorithm By: Stephen Chan Name: MCCV ID:

29 4.7.1 Module Features The MCCV module [4] allows retrieval of similar images based on the general colour distribution of the image and sub-images, with discrimination between colours in images which are homogeneous to some sizable area. The module is good for retrieval of images based on a known query image if the retrieved images are required to contain an area with a similar mix of colours to the query image. This means it is also suitable for retrieval of images based on a query image which is only part of a complete image. This module divides database images into a hierarchy of small patches and the CCV is calculated for each patch. The query is compared with all patches so that sub-images may be located in the parent image Module Properties Module Speed: Module Accuracy: Slow Very-High Module Parameters Generate: Compare: This module has no generation parameters. This module has no comparison parameters Technical Description The detail finder finds sub-images by dividing the query and the database image into a number of tiles over a number of resolutions calculating the CCV for each of the tiles. Figure 4.4: Multiscale Pyramid Structure The highest resolution image is converted into 64x64 tiles, overlapping by 32 pixels in each dimension. The image resolution is halved and, again, divided into 64x64 tiles (of which there will be 4 times less). The lowest resolution is of 64x64 pixels and 1 single tile. 28

30 For each tile a CCV is created and stored, so that the final feature vector is a set of CCV feature vectors, one for each tile. Both the query image and the database image are converted into a pyramid structure, and then each of the tiles in the query image are compared against each of the features for the tiles in the database image using the CCV matching algorithm, as described on the CCV Help Page. The query is converted to a pyramid to facilitate the database image being a subimage of the query image (double sub-image detection). An alternative is to assume the query is a subimage of the database image only, and perform a CCV match of the whole query image against each of the tiles in the database image Example Results The following is an example query giving what would be considered good results. Note: The matching is based only upon the general colour layout of the images and subimages. Query Images courtesy of Victoria and Albert Museum In the above example, all the retrieved results contain areas of contiguous skin like colour, and areas of contiguous dark-brown in a similar way to the query. Also note, that the query image, despite its diminutive size, has been found in the original image from which is was taken. Because the MCCV is a multi-scale version of the CCV, you can expect all the same types of properties and shortcomings. Used on its own you should definitely not expect the MCCV algorithm to be able to find specific instances 29

31 of objects (e.g. chairs, pots, etc). However, when used with a metadata search to locate similar objects, this algorithm could locate those of a similar colour. 4.8 The Multi-Scale Mono-Histogram Matching Module Algorithm By: Stephen Chan and David Dupplaw Name: MHistogramMono ID: Module Features The module allows retrieval of similar images based on the general distribution of brightness in the image and sub-images. It is only suitable for monochromatic images, such as black and white scans from IR-reflectance, or similar. Colour images are converted to monochrome before comparison using this module. This module divides database images into a hierarchy of small patches and the monochrome histogram is calculated for each patch. The query is compared with all patches so that sub-images may be located in the parent image. The module is good for retrieval of images based on a known query image if the retrieved images are required to contain an area with a similar distribution of brightness to the query image. This means it is also suitable for retrieval of images based on a query image which is only part of a complete image Module Properties Module Speed: Module Accuracy: Slow High Module Parameters Generate: Compare: This module has no generation parameters. This module has no comparison parameters Technical Description The detail finder finds sub-images by dividing the query and the database image into a number of tiles over a number of resolutions calculating the Mono histogram for each of the tiles. 30

32 Figure 4.5: Multiscale Pyramid Structure The highest resolution image is converted into 64x64 tiles, overlapping by 32 pixels in each dimension. The image resolution is halved and, again, divided into 64x64 tiles (of which there will be 4 times less). The lowest resolution is of 64x64 pixels and 1 single tile. For each tile a Mono histogram is created and stored, so that the final feature vector is a set of Mono histogram feature vectors, one for each tile. Both the query image and the database image are converted into a pyramid structure, and then each of the tiles in the query image are compared against each of the features for the tiles in the database image using the Mono histogram matching algorithm, as described on the Mono histogram Help Page. The query is converted to a pyramid to facilitate the database image being a subimage of the query image (double sub-image detection). An alternative is to assume the query is a subimage of the database image only, and perform a Mono histogram match of the whole query image against each of the tiles in the database image Example Results The following is an example query which would be considered good results. Note: The matching is based only upon the general brightness levels of the images and subimages. 31

33 Query Images courtesy of Victoria and Albert Museum The multi-scale monochrome histogram has the advantage over the multiscale colour histogram in that the results can appear in both grey-level and colour images. The multi-scale colour histogram will only work with colour images. The results above show this, as the correct sub-image and sub-image location is found in the monochrome image, and also in the colour image. Because the multi-scale monochrome histogram is a multi-scale version of the regular monochrome histogram, you can expect all the same types of properties and shortcomings. Used on its own you should definitely not expect the multiscale monochrome histogram algorithm to be able to find specific instances of objects (e.g. chairs, pots, etc). However, when used with a metadata search to find similar objects, this algorithm could locate those of a similar brightness distribution. 4.9 The Pyramid Wavelet Transform Matching Module Algorithm By: M Faizal A Fauzi Name: PWTretrieval ID: Module Features The PWT module [7] allows retrieval of similar images based on the general texture distribution of the image where image texture here, is concerned with 32

34 the repeating patterns throughout the whole image. The module is good for retrieval of images based on a known query image if the retrieved images are required to have a similar texture. It is not suitable for retrieval of images based on a query image which is only part of a complete image Module Properties Module Speed: Module Accuracy: Very Fast Medium Module Parameters Generate: Compare: This function has no generation parameters. This function has no comparison parameters Technical Description The PWT decomposes an image based on a wavelet transform, which can be thought of as similar to a Fourier transform, which transforms the iamge domain into a frequency domain. The frequency components of the image are analysed and a number of descriptors generated which represent the amounts of a discrete number of frequencies in the image. (a) Decomposition in image domain (b) Decomposition in frequency domain Images are resized to 512x512 to perform this decomposition, which yields 22 frequency descriptors for an image. This makes the matching very fast. The comparison is achieved using standard Euclidean distance measure Example Results The following is an example query giving what would be considered good results. 33

35 Note: The matching is based only on the texture, i.e. repeating patterns, in the whole image. Query Images courtesy of Victoria and Albert Museum The above results show the type of results you could expect to get from the PWT. The dataset contains a set of fabrics, and the similarity between the repeating pattern of the query fabric shows up in the results. You should not expect the PWT to be able to find small amounts of texture, in an image, because it is not multi-scale. So a pot with the surface of a certain texture, may not find other pots in a dataset with the same surface due to the other information (such as backgrounds) in the image causing errors in the matching. 34

36 Query Images courtesy of Victoria and Albert Museum Used on its own you should definitely not expect the PWT algorithm to be able to find specific instances of objects (e.g. chairs, pots, etc). However, when used with a metadata search to locate similar types of object, this algorithm could locate those of a similar texture The Query by Fax Matching Module Algorithm By: M Faizal A Fauzi Name: QBF ID: Module Features This module attempts retrieval when the query image is a low quality monochrome image, for example a facsimile of a painting which is in the database. The module is good for retrieval of images based on a known query image if the retrieved images are required to have a similar layout of dark and light pixels. It is not suitable for retrieval of images based on a query image which is only part of a complete image Module Properties Module Speed: Module Accuracy: Very Fast Low 35

37 Module Parameters Generate: Compare: This module takes an integer value specifying the type of feature to generate. A value of 1 specifies to generate a feature vector to be used as a reference. A value of 2 specifies to generate a feature vector to be used as a query (this is the default mode of operation). This module has no comparison parameters Technical Description The query by fax is based upon a set of PWT measures of the image at various threshold levels of a monochrome instance of the image. A QbF feature vector consists of 99 PWT features at various levels of threshold (between 1% black and 99% black) of the image. Figure 4.6: Query By Fax: Database Images converted to 99 PWT levels on left, query image on right. Matching is performed by a simple step process: Make query fax image binary (it almost is anyway, but this enforces it) Detect what percentage of the image is black/white. Select from the set of PWT vectors for a database image the vector which is of the same percentage black/white. 36

38 Perform a PWT match, as described on the PWT help page Example Results The following is an example query giving what would be considered good results. Note: The matching is based upon the layout of dark and light pixels. Query Images courtesy of Victoria and Albert Museum 4.11 The MPWT Matching Module Algorithm By: Simon Goodall Name: MPWT ID: Module Features The module allows retrieval of similar images based on the general texture distribution of the image and sub-images, where image texture here, is concerned with the repeating patterns throughout the image. The module is good for retrieval of images based on a known query image if the retrieved images are required to have a similar texture. This module divides database images into a hierarchy of small patches and the PWT is calculated for each patch. The query is compared with all patches so that sub-images may be located in the parent image. 37

39 Module Properties Module Speed: Module Accuracy: Slow <accuracy> Module Parameters Generate: Compare: This module has no generation parameters. This module has no comparison parameters Technical Description The detail finder finds sub-images by dividing the query and the database image into a number of tiles over a number of resolutions calculating the PWT for each of the tiles. Figure 4.7: Multiscale Pyramid Structure The highest resolution image is converted into 64x64 tiles, overlapping by 32 pixels in each dimension. The image resolution is halved and, again, divided into 64x64 tiles (of which there will be 4 times less). The lowest resolution is of 64x64 pixels and 1 single tile. For each tile a PWT is created and stored, so that the final feature vector is a set of PWT feature vectors, one for each tile. Both the query image and the database image are converted into a pyramid structure, and then each of the tiles in the query image are compared against each of the features for the tiles in the database image using the PWT matching algorithm, as described on the PWT Help Page. The query is converted to a pyramid to facilitate the database image being a subimage of the query image (double sub-image detection). An alternative is to assume the query is a subimage of the database image only, and perform a PWT match of the whole query image against each of the tiles in the database image. 38

40 Chapter 5 The 3-D Algorithms 5.1 The Area Volume Matching Module Algorithm By: Tony Tung Name: AreaVolume ID: Module Features The AreaVolume module finds objects with a similar area to volume ratio. This module only gives a very simple comparison between objects and should only be used as a rough estimator of similarity Module Properties Module Speed: Module Accuracy: Fast Low Module Parameters Generate: Compare: This module has no generation parameters. This module takes an integer value determining which component to compare. A value of 0 compares surface area, a value of 1 compares the volume, value of 2 compares the ratio Area V olume, and a value of 3 (default) compares the ratio 36 π. V olume2 Area 3. This module also takes a string parameter specifying which distance metric to use (default is cityblock ) and a real value as an optional argument to the distance metric Technical Description Area is obtained by summing the area of each triangle of the mesh describing the surface of the object. 39

41 Volume is the sum of the tetrahedral formed by the oriented triangles of the mesh and the centre of mass of the object. Volume and area are fast to compute but strongly depend on the mesh quality: the 3D model needs to be closed and manifold. We proposed the following formula for the volume-area calculation: Ratio = 36 π volume2 area 3 This ratio has no dimension and Ratio=1 for the sphere (surface is minimal) Example Results Here is an example query: Query Images from the Princeton Shape Benchmark Figure 5.1: Area Volume Ratio (Princeton Dataset) As can be seen, the area volume ratio finds the query object as the best match, however the other results are quite dissimilar in the princeton dataset (Figure 5.1). 40

42 Models are from the SCULPTEUR dataset Figure 5.2: Area Volume Ratio (SCULPTEUR Dataset) The Area Volume ratio performs much better on the SCULPTEUR dataset than the Princeton dataset (Figure 5.2). 5.2 The Cord Histogram Matching Modules Algorithm By: Tony Tung Name: CordHist1, CordHist2, CordHist3, CordHistogram ID: 1072, 1076, 1078, Module Features The Cord Histogram modules allow retrieval of similar shaped 3D objects based upon various attribute distributions of Cords. These attributes are Cord length, and the angles between the first and second principal axis. These modules are good for non-complex objects. They are sensitive to mesh connectivity changes Module Properties Module Speed: Module Accuracy: Fast Low-Medium 41

43 5.2.3 Module Parameters Cord Hist 1 Generate: Compare: Cord Hist 2 Generate: Compare: Cord Hist 3 Generate: Compare: Cord Histogram Generate: Compare: This module takes an integer parameter specifying the number of histogram bins (default is 16). This module takes a string parameter specifying which distance metric to use (default is euclidean ) and a real value as an optional argument to the distance metric. This module takes an integer parameter specifying the number of histogram bins (default is 16). This module takes a string parameter specifying which distance metric to use (default is euclidean ) and a real value as an optional argument to the distance metric. This module takes an integer parameter specifying the number of histogram bins (default is 16). This module takes a string parameter specifying which distance metric to use (default is euclidean ) and a real value as an optional argument to the distance metric. This module takes three integer parameters specifying the number of histogram bins for CordHist1, CordHist2 and CordHist3 respectively (defaults to 16 for each). This module takes a string parameter specifying which distance metric to use (default is euclidean ) and a real value as an optional argument to the distance metric Technical Description Feature vectors are histograms of the cords calculated over the entire mesh triangles. A Cord is defined as the vector between the centre of mass of an object to the centre of mass of a triangle on its surface. The Cord Histogram 1 algorithm makes a histogram of the lengths of these cord for every triangle. Cord Histogram 2 makes a histogram based on the angle between a cord and the first principal axis for every triangle. Cord Histogram 3, does likewise, but against the second principal axis. The principal axis are calculated by using Principal Component Analysis. This process is also used to help determine a common pose for comparison to other objects. This technique is not discriminating for complex object, and not invariant to mesh connectivity changes. 42

44 5.2.5 Example Results Here are some example results for each of the Cord Histogram variants. Query Images from the Princeton Shape Benchmark Figure 5.3: Cord Histogram 1 (Princeton Dataset) For the Cord Hist 1, we can see that it can find several other similar objects, but it also finds the snake object as the second best match which would not be expected (Figure 5.3). Models are from the SCULPTEUR dataset Figure 5.4: Cord Histogram 1 (SCULPTEUR Dataset) The Cord Hist 1 does reasonable well on the SCULPTEUR dataset, however there are a couple of unexpected results (Figure 5.4). 43

45 Query Images from the Princeton Shape Benchmark Figure 5.5: Cord Histogram 2 (Princeton Dataset) The Cord Hist 2 does slightly better with all the objects being similarly shaped, if not all from the same class of objects (Figure 5.5). Models are from the SCULPTEUR dataset Figure 5.6: Cord Histogram 2 (SCULPTEUR Dataset) With the SCULPTEUR dataset, the Cord Hist 2 retrieves a full set of fairly similar objects (Figure 5.6). 44

46 Query Images from the Princeton Shape Benchmark Figure 5.7: Cord Histogram 3 (Princeton Dataset) The Cord Hist 3 algorithm produces similar results. The human model is similar to a plane model due to the arms sticking out like wings. The vase object is not expected in these results. Models are from the SCULPTEUR dataset Figure 5.8: Cord Histogram 3 (SCULPTEUR Dataset) The Cord Hist 3 does not do as well as the Cord Hist 2 on the SCULPTEUR dataset (Figure 5.8). 45

47 Query Images from the Princeton Shape Benchmark Figure 5.9: Combined Cord Histogram (Princeton Dataset) The combined cord histogram gives the best results out of the different variants. There are a few human models and a fish model, however they are similar in their general shape to that of the query object (Figure 5.9). Models are from the SCULPTEUR dataset Figure 5.10: Combined Cord Histogram (SCULPTEUR Dataset) Against the SCULPTEUR dataset, the combined cord histogram manages to achieve a full set of vases (Figure 5.10). 46

48 5.3 The ShapeD2 Matching Module Algorithm By: Tony Tung Name: ShapeD2 ID: Module Features This module allows retrieval of similar shaped 3D objects based upon the distribution of the distance between random points on the surface of the object. This module may be used for queries on non-complex objects Module Properties Module Speed: Module Accuracy: Fast Medium Module Parameters Generate: Compare: This module takes an integer parameter specifying the number of samples to use (default is ) and a second integer parameter specifying the number histogram bins (default is 64). This module also takes a string parameter specifying which distance metric to use (default is euclidean ) and a real value as an optional argument to the distance metric Technical Description The Shape Distribution D2 approach proposed by Princeton University is a probabilistic method. The principal advantages are the robustness to noise on the mesh and the geometric invariance properties of the descriptor. Feature vectors are histogram of distance between points randomly chosen over the surface of the object with a uniform surface repartition. This method is fast to compute and permits one to describe the main shape of a 3D object but not the details. It may be used for queries of non complex shape matching and primitive shapes (sphere, torus,...) Example Results Here is an example query. 47

49 Query Images from the Princeton Shape Benchmark Figure 5.11: Shape D2 (Princeton Dataset) As can be seen this algorithm retrieves similarly shaped objects in the Princeton Dataset. The two tree objects seem a little out of place at first, however comparing them to result 6, the similarity can be seen. Models are from the SCULPTEUR dataset Figure 5.12: Shape D2 (SCULPTEUR Dataset) The ShapeD2 does not seem to perform as well on the SCULPTEUR dataset, and the only really similar object is in fourth place. 48

50 5.4 The EGI Matching Module Algorithm By: Tony Tung Name: EGIOct, EGISphere ID: 1088, Module Features This module allows retrieval of similar 3D objects based upon the Complex EGI (Extended Gaussian Images) method. This module may used for queries on complex objects Module Properties Module Speed: Module Accuracy: Fast Medium Module Parameters Generate: Compare: This module has no generation parameters. This module also takes a string parameter specifying which distance metric to use (default is euclidean ) and a real value as an optional argument to the distance metric Technical Description The idea of the Extended Gaussian Image (EGI) is to map the 3D mesh triangles orientation and/or area on a Gaussian sphere. This is a well-known technique which permits one to deduce useful information such as symmetry properties, cord lengths, etc. This approach and its variants can be used for retrieval of primitive shapes. The disadvantages of this approach are that it is not stable to mesh connectivity changes, it requires a pose estimation and has a spherical representation (with a sur-representation around the two poles) Example Results Example queries are now presented for both the EGI Oct and EGI Sphere. 49

51 Query Images from the Princeton Shape Benchmark Figure 5.13: EGI Oct (Princeton Dataset) The EGI Oct algorithm retrieves a couple of similar objects, however it also has a few not so similar objects. This is a less reliable method. Models are from the SCULPTEUR dataset Figure 5.14: EGI Oct (SCULPTEUR Dataset) The EGI Oct performs much better on the SCULPTEUR Dataset with only one unexpected result, however it is still generally the right shape. 50

52 Query Images from the Princeton Shape Benchmark Figure 5.15: EGI Sphere (Princeton Dataset) The EGI Sphere algorithm performs slightly better, with only the microscope object being particularlu out of place due its shape. Models are from the SCULPTEUR dataset Figure 5.16: EGI Sphere (SCULPTEUR Dataset) The EGI Sphere method performs well on the SCULPTEUR Dataset will no unexpected results. 51

53 5.5 The 3D Hough Matching Module Algorithm By: Tony Tung Name: HoughOct, HoughSphere ID: 1092, Module Features This module allows retrieval of similar 3D objects based upon the 3D Hough descriptor, which describes the triangle normal orientation. This module may used for queries on complex objects. Meshes should have unified normals Module Properties Module Speed: Module Accuracy: Fast High Module Parameters Generate: Compare: This module has no generation parameters. This module also takes a string parameter specifying which distance metric to use (default is euclidean ) and a real value as an optional argument to the distance metric Technical Description The 3D Hough transform produces feature vectors which consist of 3D histograms of the parameters of the polar representation of the 3D mesh triangles considered as planar surfaces. These parameters are weighted with the area and the orientation of the associated triangles. This approach requires a PCA pre-processing. It shows good results on the MPEG-7 database, and is stable to variations in the 3D mesh representation. 52

54 5.5.5 Example Results Query Images from the Princeton Shape Benchmark Figure 5.17: Hough Oct (Princeton Dataset) The Hough Oct algorithm (Figure 5.17) has a lot of quite dissimilar objects in this example. Models are from the SCULPTEUR dataset Figure 5.18: Hough Oct (SCULPTEUR Dataset) The Hough Oct algorithm (Figure 5.18) performs much better on the SCULP- TEUR dataset with all objects retrieved being similarly shaped. 53

55 Query Images from the Princeton Shape Benchmark Figure 5.19: Hough Sphere (Princeton Dataset) The Hough Sphere (Figure 5.19) algorithm has a lot of quite dissimilar objects in this example. Models are from the SCULPTEUR dataset Figure 5.20: Hough Sphere (SCULPTEUR Dataset) The Hough sphere (Figure 5.20) performs slightly better with the SCULP- TEUR dataset, however it still has several unexpected results. 54