Comparative Evaluation of Color-Based Video Signatures in the Presence of Various Distortion Types

Size: px

Start display at page:

Download "Comparative Evaluation of Color-Based Video Signatures in the Presence of Various Distortion Types"

Bethany Sutton
5 years ago
Views:

1 Coparative Evaluation of Color-Based Video Signatures in the Presence of Various Distortion Types Aritz Sánchez de la Fuente, Patrick Ndjiki-Nya, Karsten Sühring, Tobias Hinz, Karsten üller, and Thoas Wiegand Abstract The robustness of color-based signatures in the presence of a selection of representative distortions is investigated. Considered are five signatures that have been developed and evaluated within a new odular fraework. Two signatures presented in this work are directly derived fro histogras gathered fro video fraes. The other three signatures are based on teporal inforation by coputing difference histogras between adjacent fraes. In order to obtain objective and reproducible results, the evaluations are conducted based on several randoly assebled test sets. These test sets are extracted fro a video repository that contains a wide range of broadcast content including docuentaries, sports, news, ovies, etc. Overall, the experiental results show the adequacy of color-histogra-based signatures for video fingerprinting applications and indicate which type of signature should be preferred in the presence of certain distortions. Keywords color histogras, robust hashing, video retrieval, video signature I. INTRODUCTION HE aount of accessible ultiedia inforation is ever Tincreasing. For the anageent of large-scale ultiedia data, efficient organization of databases is required. One efficient anageent tool is video retrieval, which can also be used for a variety of other applications, such as content advertising, detection of duplicated video sequences and autoatic detection of copyright violations. In video retrieval, one or ore videos are selected as query (query-by-exaple) and a list of siilar videos fro the database is returned in response. Recent approaches often use content-based video properties (e.g. color or texture features) A. Sánchez de la Fuente is with the Iage Processing Departent, Fraunhofer Institute for Telecounications Heinrich-Hertz-Institut, Berlin, Einsteinufer 37, D-0587 Gerany (phone: +49-(0) ; fax: +34- (0) ; e-ail: aritz.sanchez@hhi.fraunhofer.de). P. Ndjiki-Nya, K. Sühring, T. Hinz, and K. üller are with the Iage Processing Departent, Fraunhofer Institute for Telecounications Heinrich-Hertz-Institut, Berlin, Einsteinufer 37, D-043 Gerany (e-ail: {ndjiki,suehring,hinz,kueller}@hhi.de). T. Wiegand is the Head of the Iage Processing Departent, Fraunhofer Institute for Telecounications Heinrich-Hertz-Institut, Berlin, Einsteinufer 37, D-043 Gerany, and Head of the Iage Counication Chair, Departent of Telecounication Systes, School of Electrical Engineering and Coputer Sciences, Technical University of Berlin, Berlin, Einsteinufer 7, D-0587 Gerany (e-ail: wiegand@hhi.fraunhofer.de). for atching. ost frequently, a robust or perceptual video hash or a video signature is used for video identification. First approaches developed a cryptographic hash [], which is bitsensitive. Other approaches ap the visual inforation onto a fixed-length hash code that reains practically unaltered in presence of distortions of the original signal []. Hence, robust hashing allows slightly altered (e.g. daaged) videos to be also recognized as being siilar to the original unaltered video. Radhakrishnan and Bauer [3] propose a robust signature based on projections of cropped coarse difference-iages between adjacent fraes onto randoly generated atrices. The output hash for each difference-iage is of fixed length. A quantization using two reconstruction levels is afterwards applied using the edian of the hash-eleents as threshold. The final signature is the concatenation of all binary vectors. This description is highly robust against distortions introduced by video copression, spatial and brightness scaling, whereas being sensitive to other geoetric and frae-rate conversions. The binarization step described above is also used by Coskun and Sankur []. They deterine and apply one-bit quantization to the ost significant coefficients of 3D DCT for a given video signal. In addition to blurring and noise robustness, this hash code is also robust to distortions introduced by operations including video copression as well as contrast and brightness anipulation. In this paper, the robustness of color-based signatures to specific types of distortions is evaluated. The ain advantage of these signatures is the very low coplexity required for their extraction. For soe distortion types this ight be at the cost of retrieval perforance. For other distortion types, colorbased signatures ight offer a very good efficiencycoplexity trade-off. The signatures are evaluated in a new, odular fraework that allows easy plug-in of new data and signatures. The reainder of the paper is organized as follows. In section II, the fraework and the proposed signatures are described. Section III suarizes the siilarity easures used for each type of signature. The experiental results are presented in section IV. Finally, the conclusions are drawn and future research is discussed. 538

2 A. Overview II. SIGNATURE FRAEWORK The new fraework pursues three objectives: the coplete video retrieval process with all corresponding phases; a separate operation of the individual phases; and a odular syste in which data and signatures can be easily added or reoved. The first proble is addressed by covering a large nuber of aspects a fingerprinting task ight involve: wide variety of data, including altered and daaged videos, feature extraction and atching, and bencharking of the syste. As depicted in Fig., video processing is divided into five independent odules. The first stage as depicted in Fig. generates test sets of a predefined nuber of rando clips extracted fro the video repository (further explained in section IV.A). The distorted versions of all the query clips are added to each test set during the second phase (see section IV.B). Next, the signatures are extracted fro the query videos as well as fro the sequences contained in the test sets. atching of the query clips with the test sets is then conducted. Finally, the atching results are analyzed (see section IV.D) to assess the efficiency of the signatures. B. Integrated Signatures Five signatures have been ipleented in this work. They all derive fro the color histogras of the sequence fraes (see Fig. ). The histogras are 56-bin wide and the HSV color-space is used for their generation. As explained in [4], the reliability of color inforation to characterize visual content akes it one of the ost widely used features in retrieval tasks. The siple representation provided by color histogras and the ease of their coparison (e.g. Euclidean distance) is the other ain reason for the chosen feature in the present descriptions. Such descriptions do not properly represent grayscale content, which is therefore out of scope. This paper investigates the robustness of color-based signatures to typical video distortions. Test sets distorted sequences Video Repository. rando test sets generation. distortion generation 3. descriptor generation 4. atching 5. evaluation Fig. Operation chain of the proposed fraework descriptors Descriptor Repository ) The SCean Signature The first signature is calculated as the noralized color histogra teporally averaged along the sequence fraes. The average population p of the -th bin on the final histogra is calculated as follows: p N j p,,,...,, WH N j () where is the total nuber of bins. The population j p of the -th bin of the j-th frae (j =,,,N) of the sequence is averaged, i.e. sued up and noralized by N, W (width) and H (height) (). Hence, the description is a decial vector of 56 eleents and is, referred to as SCean. This signature needs a single vector to represent the overall color content of the whole sequence. This coputational saving ay iply soe detriental loss of detail. Dissiilar videos are expected to have different color content, hence producing different descriptions. However, such a signature ay be highly sensitive to color degradations. ) The SCDiffHist Signature The SCDiffHist signature characterizes the change in color between fraes and is inspired by the color-shift signature described in [4], where the signature consists of the scalar distances between noralized color histogras of adjacent fraes. For the SCDiffHist signature, the absolute unnoralized difference histogra, d j, between neighboring fraes j and (j+) is gathered. j j j d p p,,,...,, j,,..., N () The noralized ean histogra of these difference histogras, d j, is obtained as a decial description with 56 eleents. N j d d WH N j,,,..., Note that the largest possible distance between two histogras is now twice the diensions of the frae. For this reason, the noralization factor is doubled (WH). 3) The SCCop Signature The first- and second-order oents of the difference histogras are used for the third signature. Two different criteria are considered for the calculation of oents. On the one hand, the average c tie, x, ctie, y of the centers of ass of every difference histogra (4) and their standard deviations tie, x, tie, y (5) are estiated, both in x-and y-axis. N j (3) c tie, x c j, x, (4) N / N,,, tie x c j tie x c j x, (5) N 539

3 where c j,x denotes the coordinate in x-direction of the center of ass of the j-th difference histogra, and N refers to the nuber of fraes of the sequence. b j h(j) Fig. Color histogra h(j) of frae j On the other hand, the center of ass c spatial, x, cspatial, y and the standard deviations spatial, x, spatial, y in the ean difference histogra are calculated (6), (7), and (8). Here, the y coordinate of the centers of ass deterined is coincident for both ethods c tie, y cspatial, y c y. The y oents are noralized by W and H. c spatial p p p, x, (6) cspatial, y, (7) WH /, spatial x, c spatial x, (8) where corresponds to the bin nuber of the ean difference histogra, and denotes the nuber of coefficients of a histogra. All the equations for the y coordinate are siilarly obtained with adjusted (4), (5), and (8), with the additional noralization factor WH. The generated signature SCCop has only 7 decial eleents, as in (9): y c, c, c,,,, ] (9) [ y tie, x spatial, x tie, y tie, x spatial, y spatial, x SCCop inherits the potential weaknesses of SCean and SCDiffHist. Despite the advantages it features in ters of its copactness, this signature describes the video signal very coarsely. 4) The SCHistBin Signature The signature SCHistBin is a binary variation of the SCDiffHist signature. It is inspired by [], [3], where the purpose is to obtain a higher degree of robustness. Here, the unnoralized ean difference histogra is quantized with two reconstruction levels, using the edian of the bin populations as threshold. Any bin-value higher than the edian is set to. The others are set to 0. Binarization is expected to yield higher robustness to hoogeneous color changes in the teporal direction copared to SCDiffHist. A higher degree of robustness is also reported in [], [3]. 5) The SCeanHaar Signature The last signature corresponds to the definition of PEG- 7 s GoF/GoP descriptor [5]. The unnoralized ean color histogra is wavelet-transfored, using the Haar Transfor, as defined in the PEG-7 specification [6]. In this paper, it is referred to as SCeanHaar. III. SIILARITY EASURES For the SCean, SCDiffHist, and SCeanHaar signatures, the nor (Euclidean distance) is used as siilarity easure, D, (0) between two signatures a and b. / D a i i b i (0) For the SCCop signature, the siilarity between two sequences (a,b) is easured via area coparison. Fig. 3 Intersection areas of teporal oent rectangles (analog for the spatial oents) As depicted in Fig. 3, the center of ass and the standard deviations define the center and the argins of a rectangle. The intersection of the areas described by the rectangles of two signatures is calculated and noralized with respect to the largest area (), both for the teporal area.tie and spatial area.spatial oents. The ean of area. tie and area. spatial is subtracted fro, so that a perfect atch has a distance D 0 (). Sa Sb area ax( Sa, Sb ) () area. tie area. spatial D () 540

As SCHistBin is a binary signature, the noralized Haing distance (3) is used for siilarity estiation. That is, the nuber of deviating bits is noralized by the length of the signatures. D A.

The data is organized in shots or tracks. The latter are defined as a sequence of fraes that corresponds to an uninterrupted caera operation [4].

The query clips represent a wide range of visual content (Fig. 4): docuentaries, sports, news, real/aniated fils, advertiseents, and talk shows, aong others.

For the retrieval experient, each query shot has distorted versions. The ground-truth size is thus always 3. However, there ight be other visually siilar sequences that are not listed as groundtruth.

This is the ajor otivation for drawing rando subsets fro the video repository for evaluation. Reliability of the results is achieved by repeating the subset evaluation sufficiently often.

Tested distortions Soe exeplary visual alterations affecting color content are tested.

aplitude 5.5 (of a axiu of 55) and period 50 fraes; and color inversion. The latter is chosen, in order to show the liitations of the proposed signatures.

4 As SCHistBin is a binary signature, the noralized Haing distance (3) is used for siilarity estiation. That is, the nuber of deviating bits is noralized by the length of the signatures. D A. Data set ai bi, i,, a a, b b IV. EVALUATION i i (3) The developed signatures are tested on a large variety of video content, encoded in H.64/PEG4-AVC forat at QCIF (76 pixels x 44 pixels) resolution. The data is organized in shots or tracks. The latter are defined as a sequence of fraes that corresponds to an uninterrupted caera operation [4]. 00 query shots are atched against 0 randoly generated test sets. The query tracks were cleared beforehand, i.e. trivial sequences were reoved (e.g. too short clips, static videos). The query clips represent a wide range of visual content (Fig. 4): docuentaries, sports, news, real/aniated fils, advertiseents, and talk shows, aong others. Each test set contains 00 sequences, aong which 00 are distorted versions of the query tracks. A ground-truth set consists of a query track and distorted versions of it. For the retrieval experient, each query shot has distorted versions. The ground-truth size is thus always 3. However, there ight be other visually siilar sequences that are not listed as groundtruth. This proble could be addressed by annotating the content of the video database accordingly. However, this is ipractical due to its size: tracks. This is the ajor otivation for drawing rando subsets fro the video repository for evaluation. Reliability of the results is achieved by repeating the subset evaluation sufficiently often. The nuber of test sets used is expected to provide the required generality of the results, independently of the analyzed data. a) b) c) d) e) f) Fig. 4 Exaple query shots B. Tested distortions Soe exeplary visual alterations affecting color content are tested. They are: additive white Gaussian noise (AWGN), with zero ean and a standard deviation of 50; salt & pepper noise (ulticolor), with a 5% probability of occurrence; an additive cosine in tie of aplitude 5.5 (of a axiu of 55) and period 50 fraes; and color inversion. The latter is chosen, in order to show the liitations of the proposed signatures. All distorted video signals are encoded with three different quantization steps: 6, 40, and 48. Exaple iages are shown in Fig. 5. a) Original b) Gaussian noise c) Salt & pepper d) Cosine distortion e) Color inversion Fig. 5 Snapshots of evaluated distortions C. Efficiency easure As a easure of efficiency, the average retrieval rate (ARR) [5] of each signature is considered. The paraeter is calculated along the 0 test sets and for every query track. This value can be expressed as the ratio between the nuber of correctly recovered sequences and the total nuber of sequences stored in the ground-truth sets: L ARR ARR i i, L NQ # correct. atches ARRi NQ q NG q q, (4) where L denotes the nuber of test sets considered (0 in this case), ARR i refers to the average retrieval rate for every query shot and for the i-th test set, NQ is the nuber of query tracks used (00), and NG(q) expresses the nuber of sequences that belong to the ground-truth set of query q (in the present experient, NGq, q, as the query is ignored). The threshold K(q) represents the highest rank in the list of results that a ground-truth sequence of query q can achieve and is defined in [5] as K q 4 NGq, axngq, in q. (5) Every ground-truth sequence with a rank higher than K(q) is classified as issed, else it is a true positive atch. For this 54

experient, K(q) = 4. D. Experiental results Fig. 6 and Fig. 7 show the retrieval rates for each distortion type and each quantization step (QP).

Fro the results, it is obvious that SCCop always has very low retrieval rates (below 30%). This is caused by a high loss of description accuracy.

typical distortions, where color inversion is not considered. The results lie between 70% and 75% for SCean, SCDHistBin, and SCeanHaar. SCCop is the worst colorbased signature.

5 experient, K(q) = 4. D. Experiental results Fig. 6 and Fig. 7 show the retrieval rates for each distortion type and each quantization step (QP). The standard deviation of the ARR along the test sets is very low (under 0.46%) and is, therefore, not displayed. Fro the results, it is obvious that SCCop always has very low retrieval rates (below 30%). This is caused by a high loss of description accuracy. The highest rates are achieved by SCeanHaar and SCean, closely followed by SCHistBin. SCean shows a hoogeneous response to the different types of distortion. typical distortions, where color inversion is not considered. The results lie between 70% and 75% for SCean, SCDHistBin, and SCeanHaar. SCCop is the worst colorbased signature. Although it is the only description that includes teporal inforation, it shows too low discriination capability. Overall, the ost efficient signature appears to be SCean. ost of the signatures proposed are appropriate for identifying distorted video signals of identical content in applications, where the considered set of distortions occurs. E. Size of the signatures The size of the descriptions is independent of the length of the sequences and of the diensions of the iage, as listed in TABLE I for each of the signature types. TABLE I SIGNATURE SIZE IN BYTES SCean SCDiffHist SCCop SCHistBin SCean- Haar Fig. 6 ARR for different distortion classes Fig. 7 ARR for different Quantization Paraeters (QP) Fig. 8 Overall ARR without color inversion For all investigated signatures, the retrieval rates are very siilar for different quantizations. Hence, all color-based signatures are relatively invariant to coding quantization. Aong the tested distortions, the highest ARR scores are obtained for Gaussian noise for alost every signature. Here, SCDiffHist and SCeanHaar are very robust. For salt & pepper noise, SCean signature achieves over 80% with the other signatures achieving lower ARR values. For cosine distortion, SCeanHaar achieves alost 85% ARR followed by the other signatures. Not surprisingly, color inversion leads to very bad scores, as it contains copleentary color distribution. This attack ight be overcoe if iagedifference were perfored previous to histogra generation. Hence, Fig. 8 displays the overall retrieval rates for the three Considering that the average length of the sequences tested is 4. s and the required disk space for the coded bitstrea of a single track is around 00 kbytes, the values of the table can be considered quite sall. Three signatures are about 50 ties saller than the original video and two signatures around 700 and 3000 ties saller. F. Coplexity of the signature extraction and atching For a sequence of N fraes, width W and height H, approxiate coputational coplexity orders of the signature extraction and atching are suarized in TABLE II. The ter WH refers to the -bin histogra generation fro the decoded fraes. On the one hand, for every signature the extraction ties highly depend on W, H, N, and. On the other hand, the atching is always constant for SCCop and depends only on when any of the other four signature types are atched. TABLE II COPUTATIONAL COPLEXITY OF EXTRACTION AND ATCHING Signature Extraction atching SCean O(WH + N) O() SCDiffHist O(WH + N) O() SCCop O(WH + N) O() SCHistBin O(WH + N) O() SCeanHaar O(WH + N) O() TABLE III lists the signature extraction and atching speeds (on a CPU of.66 GHz and 8 GB of RA, with QCIF resolution, = 56 bins, and a frae rate of 5 fps). The results show a very siilar tie-efficiency during the extraction for every signature type, which is validated by the coplexity orders given by TABLE II. Due to the slightly higher tie consuption introduced by the Haar Transfor, only SCeanHaar extracts less than 3 seconds of video per second, whereas the other four types of fingerprints reach a speed over 4.4 and under 4.8 seconds of video per second. 54

6 Regarding the atching stage, the signatures copared by the Euclidean distance are the slowest, closely followed by SCHistBin (Haing distance), and the fastest atches occur for SCCop (alost twice as fast as the slowest atching fingerprints). [6] A. Yaada,. Pickering, S. Jeannin, L. Cieplinski, J.R. Oh, and. Ki, PEG-7 Visual Part of experientation odel Version 0.0, ISO/IEC JTC/SC9/WG/N4063, Singapore, arch 00. TABLE III COPUTATIONAL SPEED OF EXTRACTION AND ATCHING Signature Extraction seconds of video / second atching pairs of signatures / second SCean SCDiffHist SCCop SCHistBin SCeanHaar V. CONCLUSIONS AND FUTURE WORK Five signatures based on color histogras have been tested in the presence of various distortion types. The results show that liiting the descriptions to histogra oents causes very high loss of description accuracy, leading to inefficient video retrieval. The use of difference histogras does not see to iprove the robustness of the signatures in coparison to the original histogras. However, the difference histogra perfors better, when binarization of the bin population is operated. Size reduction and lower atching coplexity are additionally achieved. The ean color histogra shows siilar results for different distortions. Therefore, this signature is considered as the steadiest and ost satisfactory overall. Another advantage of all signatures is their copactness. Using a nuber of rando subsets of the database, content-independence of the evaluation results is achieved. Future work will focus on the iproveent of the presented signatures as well as on further investigations on coplexity reduction. Furtherore, other daage classes (e.g. geoetric attacks, severe alterations on color content, or distortions in onochroe sequences) will be investigated. For such distortions, soe other feature or a ore robust version of the presented signatures ay be necessary. REFERENCES [] B. Preneel, R. Govaerts, and J. Vandewalle, "Cryptographic hash functions", Proceedings of the 3rd Syposiu on State and Progress of Research in Cryptography, W. Wolfowicz (ed.), Fondazione Ugo Bordoni, pp. 6-7, 993. [] B. Coskun, and B. Sankur, Robust Video Hash Extraction, EUSIPCO 004, pp , Vienna, April 004. [3] R. Radhakrishnan, and C. Bauer, Content-Based Video Signatures based on Projections of Difference Iages, IEEE 9th Workshop on ultiedia Signal Processing, pp , Greece, -3 Oct [4] T.C. Hoad, and J. Zobel, Detection of Video Sequences Using Copact Signatures, AC Transactions on Inforation Systes (TOIS), Volue 4, Issue, pp. -50, January 006. [5] B.S. anjunath, P. Salebier, and T. Sikora, Introduction to PEG-7: ultiedia Content Description Interface, John Wiley & Sons, Inc., New York, NY,

Novel Image Representation and Description Technique using Density Histogram of Feature Points

Novel Image Representation and Description Technique using Density Histogram of Feature Points Novel Iage Representation and Description Technique using Density Histogra of Feature Points Keneilwe ZUVA Departent of Coputer Science, University of Botswana, P/Bag 00704 UB, Gaborone, Botswana and Tranos