A New Concept of Security Camera Monitoring. With Privacy Protection by Masking Moving. Objects

Transcription

1 A New Concept of Security Camera Monitoring With Privacy Protection by Masking Moving Objects Kenichi YABUTA, Hitoshi KITAZAWA, and Toshihisa TANAKA Department of Electrical and Electronic Engineering Tokyo University of Agriculture and Technology Naka-cho, Koganei-shi, Tokyo, Japan y @gc.tuat.ac.jp, kitazawa@cc.tuat.ac.jp, tanakat@cc.tuat.ac.jp Abstract. We present a novel framework for encoding images obtained by a security monitoring camera with protecting the privacy of moving objects in the images. We are motivated by the fact that although security monitoring cameras can deter crimes, they may infringe the privacy of those who and objects which are recorded by the cameras. Moving objects, whose privacy should be protected, in an input image (recorded by a monitoring camera) are encrypted and hidden in a JPEG bitstream. Therefore, a normal JPEG viewer generates a masked image, where the moving objects are unrecognizable or completely invisible. Only a special viewer with a password can reconstruct the original recording. Data hiding is achieved by watermarking and encrypting with the advanced encryption standard (AES). We illustrate a concept of our framework and an algorithm of the encoder and the special viewer. Moreover, we

2 2 Kenichi YABUTA, Hitoshi KITAZAWA, and Toshihisa TANAKA show an implementation example. Keyword: privacy protection, security camera, watermarking, JPEG encoding 1 Introduction Recently, a large number of security monitoring cameras are set up to deter and investigate crimes. With the increase of these monitoring cameras, how to protect the privacy of recorded objects such as people, cars, and so on is becoming a major problem. Monitoring cameras record not only criminals but also general people, who may not be aware that they are monitored. Moreover, the recorded images can probably be distributed without any permission by these monitored people. Therefore, we should establish a framework for security monitoring which considers the privacy protection. A simple way to do this is to deteriorate the quality of moving objects. Based on this idea, a non-reversible method which decreases the resolution of objects whose privacy should be protected has been proposed in [1] and [2]. These strategy can make moving objects unrecognizable and still keep the nature of a moving object, that is, we can distinguish whether the moving object is a man, a car, or something else. However, the non-reversible processing loses details of the objects such as human faces, number plates of cars, etc. This implies that the reliability of monitoring cameras for security purpose can be reduced by this deterioration. In other words, if a face image of a criminal, for example, is even

3 Security Camera Privacy Protection 3 slightly destroyed, we may not be able to specify this criminal. In [3], a concept of the reversible method has been proposed. This method can display images with privacies protected or unprotected. However, this system needs a special system when the system reconstructs images as well as it displays images in which the privacy is protected. In this paper, therefore, we propose a new framework for encoding images recorded by security monitoring cameras. In this framework, an encoder generates a bitstream which gives a masked image when a normal JPEG [9] viewer is used for decoding, and a special viewer is needed to reconstruct an input image. In the masked image decoded by a normal viewer, moving objects in an input image are scrambled or erased, so that the privacy of the moving objects can be protected. A special viewer with a password decrypts the moving objects embedded in the bitstream and then reconstructs the input image. Therefore, even if recorded images taken by monitoring cameras are distributed, the objects are unrecognizable, as long as normal JPEG viewers are used. The rest of this paper is organized as what follows. In Section 2, we clarify requirements which security monitoring cameras satisfy. In Sections 3 and 4, we propose algorithms which implement our new concept. In Section 3, we describe a method of masking and in Section 4, we show how to embed moving objects in a JPEG bitstream by using watermarking [8]. In Section 5, we show experimantal results and discuss about the results. Section 6 concludes our work and mentions open problems.

4 4 Kenichi YABUTA, Hitoshi KITAZAWA, and Toshihisa TANAKA Radiance I = 0.30R G B (1) 0 Average MEN = lp (l) (2) Dispersion VAR = (l MEN) 2 P (l) (3) Privacy Protection in an Image Obtained by Security Monitoring We can classify images into two regions, moving object regions and background regions. Moving object regions are defined as regions including moving objects extracted from the input image by the moving object extraction process [4], [5], [6], [7]. Background regions are defined as regions except the moving object regions. People and cars passing by are extracted as moving objects. If the moving object regions are made invisible or unrecognizable, their privacy can be protected. Moreover, the input image with the moving objects should be reconstructed when it is needed for certain reasons, such as, crime investigation. Although we should preserve the quality of the moving object regions as well as possible, we can reduce the amount of information of the background, because we are more interested in the moving object regions than the background regions. Based on these discussions, we state the requirements of the privacy protection in the fixed monitoring camera system as follows.

5 Security Camera Privacy Protection 5 1. Masked images should be displayed by normal viewers for compressed images, such as JPEG viewers. 2. Moving objects in masked images should be invisible or unrecognizable. 3. Original input images should be reconstructed by a special viewer with a decodeing password. A decoded images should be reconstructed as close to an input image as possible. 4. An encoder should generate only one JPEG bitstream. A normal and a special viewers can decode a masked and a reconstructed images, respectively only from this bitstream. 3 Masking Method 3.1 Algorithm Overview Figure 1 shows the flow of a masking method proposed in this paper. We assume that the size of an input image obtained by a monitoring camera is W H. A JPEG bitstream of a masked image is generated by moving object extraction followed by encoding. The bitstream can be decoded by either a normal or a special JPEG viewer. The former viewer only displays a masked image, where the privacy of moving objects are protected, and the latter viewer reconstructs an input image. In the following, each part is described. Moving object extraction For extracting moving objects, we can use the background subtraction methods proposed in [4], [5], [6], or [7], for example. In these methods, a background image is produced by a sequence of input images and

6 6 Kenichi YABUTA, Hitoshi KITAZAWA, and Toshihisa TANAKA moving objects are extracted by taking the difference between an input image and the background image, and then the moving objects are detected. The pixels extracted as a part of the moving object are grouped into connected regions, wich are called moving object regions. The moving object regions are decided whether they need masking process or not, according to the region size which is defined by the number of pixels belonging to that region. Assume that there are K moving object regions in an image. Let Np(i)(i = 1, 2,..., K) be the number of pixels in the ith moving object region. If the ith moving object region satisfies Np(i) T h (i = 1, 2,..., K), (4) where T h is a threshold, then this region will be masked, and is referred to as a region to be masked. We introduce a binary image which indicates pixels in the masked regions. Let M ij (i = 1, 2,..., W, j = 1, 2,..., H) be a value of the (i, j)th pixel of this binary image. Then, M ij is defined as 1 if the (i, j)th pixel in the image belongs to region to be masked. M ij = 0 otherwise. Encoding Recall that in the proposed scheme, the privacy of the moving object regions are protected by scrambling/erasing. This step needs three data, that is, an input image, a background image, and a binary image M ij defined above. First, we produce a scrambled/erased image where moving objects are invisible or unrecognizable. Since the special viewer must reconstruct the input image,

7 Security Camera Privacy Protection 7 information of the moving object regions should be embedded in the masked image. This embedded data should be also encrypted for privacy protection. Then, this masked image with embedded data is encoded by a JPEG codec. This part will be more fully explained in Section 4. Decoding by normal viewers If we use normal JPEG viewers such as web browsers, they display masked images. Since a masked image is generated such that moving objects are invisible, privacies of these objects are protected. Reconstruction of the input image To reconstruct the masked objects, a special viewer with a decoding password is required. The moving objects are decrypted with the password and reconstructed. As a result, a reconstructed image is obtained. 3.2 How to Make Objects Invisible We propose two schemes for masking that make moving objects invisible. They are listed below. Scrambling By the scrambling, the pixels in a moving object region are randomly permutated. As a result, we may not understand what the moving objects are. Although this method can conceal a part of the object, for example a number plate of a car, and a face, some parts of information, such as color, can be recognized. Erasing By the erasing, the moving object images are replaced by the corresponding background images. Therefore, the moving objects become invis-

8 8 Kenichi YABUTA, Hitoshi KITAZAWA, and Toshihisa TANAKA ible. However, in many cases we want to keep a brief shape of an object for some purposes: counting cars and pedestrians, for example. Therefore, in this method we draw peripheral curves of moving objects. This will be illustrated later. Masked images generated by each method show what is happening, not who is there. We can choose one of the two methods depending on an objective of the system. If a part of information of moving objects, such as color, is desirable to be seen by general users, the scrambling is recommended. If all details of moving objects except the shape should be hidden, we should use the erasing. It should be noted that for reconstruction of an input image including moving objects, information of the moving objects should be embedded in the bitstream generated by the encoder. We will address this problem in the following section. 4 Embedding Moving Objects With Watermarking This section describes masking methods and how to embed data of moving objects in a masked image by using digital watermarking [8]. Figure 2 shows the simple flow of masking methods with watermarking. First, moving objects are extracted from an input image. The extracted moving objects are compressed by the JPEG and are encrypted by AES with password. Moving object regions in an input image are hidden by scrambling/erasing. This masked image is transformed by the discrete cosine transform (DCT) and the data of objects that have been encrypted are embedded in DCT coefficients of the masked image. In the following, we describe details of the proposed method.

9 Security Camera Privacy Protection Scrambling/Erasing Moving Object Regions Scrambling In this method, all pixels in a moving object region are randomly permutated. Assume that there are N pixels in a moving object region. Let R(i)(i = 1, 2,..., N) be a generated random number and P (i)(i = 1, 2,..., N) is the ith pixel in this region. Then, the ith replaced pixel is obtained by the permutation defined as P (i) = P (R(i)). (5) Erasing In this method, moving objects are made invisible. The pixels in the moving object regions in the input image are replaced by the pixels at the same coordinates in the background image. To illustrate the shape of a moving object, the edge of the moving object is explicitly drawn. We use the binary image (a set of M ij ) to specify edges as follows. The pixel of coordinate (i, j) is regarded as a pixel on the edge if M ij = 1 and one or over of four pixels in circumference are M ij = 0. In other words, when we define N ij as the sum of pixels of the binary image at the upper, the lower, the right, and the left pixels of M ij that is, N ij = M i,j 1 + M i,j+1 + M i+1,j + M i 1,j. If M ij satisfies 0 < M ij N ij < 4, (6) then the pixel of coordinate (i, j) is judged as a pixel on the edge.

10 10 Kenichi YABUTA, Hitoshi KITAZAWA, and Toshihisa TANAKA 4.2 Embedding Moving Objects With Watermarking Encryption of a Moving Object (Fig. 2,(2)) First, an input image is divided into minimum coded units (MCUs). An MCU is a minimum unit of JPEG compression, and the size of an MCU is 8 8. If an MCU includes a moving object region, the MCU is compressed by the JPEG. A bitstream generated by JPEG is stored in an N-byte-length array denoted by MO[i](i = 1,..., N). It should be noted that the bitsream consists of only Huffman codes; therefore, a JPEG header is removed. MO[i] is encrypted by AES [10] and it become MO[i](i = 1,..., Ñ) AES is a block encryption method and the block size is 16 bytes, Ñ is a smallest multiple of 16 not less than N. Watermarking the Encrypted Moving Object (Fig. 2,(4)) We embed M O[i] in the scrambling/erasing image with watermarking. First, we obtain DCT coefficients of the scrambling/erasing image by using a JPEG technique. We use DCT coefficients of middle frequencies for watermarking, because any changes of low coefficients are very noticeable and the changes of high coefficients lead to great changes when the JPEG decoder decodes quantized data. Compared with the low or high frequency DCT coefficients, a small perturbation of a middle frequency DCT coefficient does not visually affect the reconstructed image. Therefore, we propose to embed the moving object data in the middle coefficients indicated in Fig. 3. The least signifivcation bits (LSBs) of quantized DCT coefficients are replaced by MO[i] one bit by one bit. For example, if MO[i] = 0x5A = (2)

11 Security Camera Privacy Protection 11 Table 1. Grouping of DCT coefficients QDCT Z k(i) in each MCU 1 i 8 not embedded 9 i 16 Position 1 17 i 24 Position 2 25 i 32 Position 3 33 i 40 Position 4 41 i 48 Position 5 49 i 56 not embedded 57 i 64 not embedded Table 2. Adaptive embedding position selection by data size Size of embedding data [bytes] Position to be embedded 1 to to to to to to to to to 5 and if the DCT coefficients at the embedding position are 2, 5, 0, 8, 10, 1, 4, and 3, that DCT coefficients become 2, 5, 0, 9, 11, 0, 5, and 2. Let QDCT Z k(i)(i = 1, 2,..., 64) be a quantized and zigzag scanned DCT coefficients. The QDCT Z k(i) are grouped into 8 parts as shown in Table 1. The embedding position is decided according to the number of bytes to be embedded. If, the size of input image is , then the number of MCUs is ( )/ = The embedding position is dynamically selected by the size of data to be embedded as defined in Table 2. Remarks on the Use of Watermarking for Embedding the Object Data The proposed watermarking in masking methods can embed more bits than

12 12 Kenichi YABUTA, Hitoshi KITAZAWA, and Toshihisa TANAKA ordinary watermarking methods do. When watermarking is used in detecting illegal copy or embedding copyright, it should be robust to manipulation of image such as expansion, reduction, and color change. On the other hand, in the proposed masking application, it does not matter in terms of privacy protection if a masked image is attacked and can not be reconstructed. Moreover, the background region in an input image does not include important information. Therefore, the watermarking in this masking application can embed a large size of data, as mentioned in the previous section. 4.3 Decrypting and Reconstruction First, the encrypted data array the masked image. Then, M O[i] are extracted from DCT coefficients of MO[i] are decrypted by AES. Then, the moving object data is reconstructed with the JPEG decoder. Finally, the reconstructed moving object image is overwritten at the position where there was the moving object in the input image. 5 Experimental Results Experimental results of masking methods using scrambling/erasing are shown. We apply the proposed methods to a sequence of moving images with 169 frames. In this experiment, the image size acquired from a fixed monitor camera is pixels. The frame rate is 5 frames/sec. Figure 4 shows the 131st frame in this sequence whose size is KB.

13 Security Camera Privacy Protection 13 Table 3. The comparison of the masking method PSNR [db] File Size [KB] Processing Time [sec] Scrambling (192%) Erasing (163%) Figures 5(a) and 5(b) show scrambled and reconstructed images, respectively. It can be observed in Fig. 5(a) that pixels in the moving object region are scrambled and the moving object is no longer recognized. However, as seen in Fig. 5(a) and 5(b), blocking artifacts are visible in the background region due to watermarking. As shown in Fig. 5(b), the moving object region is reconstructed. In Fig. 5(c), the masked image with an erased object is shown. The moving object is replaced by the background image and it becomes invisible. The reconstructed image is shown in Fig. 5(d). By the erasing, we can hide such information that scrambling can not hide. In the case of scrambling, we can roughly estimate the moving object from the shape and a dominant color, though the face of the moving object can not be recognized. Table 3 shows comparison of peak signal to noise ratios (PSNRs), the file sizes and the processing times of the scrambleing and the erasing methods. PSNR is defined as P SNR = 10 log MSE, (7) where MSE is the mean square error between the original input and reconstructed images. The scrambling and the erasing give similar PSNRs. They are less than 30 db in the whole images. However, when we compare PSNRs of only

14 14 Kenichi YABUTA, Hitoshi KITAZAWA, and Toshihisa TANAKA the moving object images in the input and reconstructed images, the PSNR of both images are an identical value and it is db. For subjective comparison, the original and reconstructed object images are illustrated in Fig.6. This result shows that the PSNR of the moving object region is higher than that of the whole image. The background image is highly deteriorated by watermarking, however the moving object region gives higher PSNR. This result suggests that each method sacrifices the image quality of the background region to that of the moving object regions, which is desirable because the moving object regions include more important information than its background region. As seen in Table 3, the file size increases by 92% compared to the input image in the scrambling, and by 63% in the erasing. Efficiency of the JPEG compression is reduced because the watermarking decreases zero run length in DCT coefficients. The file size in the scrambling is greater than that of the erasing, because in the scrambling, the adjacent pixels in a moving object region largely differ from each other, which is caused by the interchange of pixels. The processing time in Table 3is the CPU time required for the encoding process using Pentium4 2.6GHz. It does not include the moving object extraction time. 6 Conclusion We have presented a novel concept for a security monitoring camera with protecting the privacy of moving objects in recorded images. By using the proposed encoder for security camera recordings, a normal JPEG viewer shows only the

15 Security Camera Privacy Protection 15 masked images, where moving objects are scrambled or erased. Therefore, we need a special viewer with a password to reconstruct an input image. As a result, the privacy of moving objects are strongly protected. The following problems would be still open. First, we need to decrease the processing time for real-time recording and encoding. Second, experimental results in this paper have shown that the sizes of masked images are grater than those of input images. More efficient compression is necessary. Moreover, flexible selection of DCT coefficients which is used for watermarking would be effective for decreasing the size of output bitstream as well as improving the subjective quality of the reconstructed image. This research was partially supported by the Ministry of Education, Science, Sports and Culture, Grant-in-Aid for Scientific Research (C), , References 1. I. Kitahara, K. Kogure, and N. Hagita, Stealth Vision for Protecting Privacy, Proc. of 17th International Conference on Pattern Recognition (ICPR 2004), Vol.4, pp , (2004) 2. J. Wickramasuriya, M. Alhazzazi, M. Datt, S. Mehrotra and N. Venkatasubramanian Privacy-Protecting Video Surveillance, SPIE International Symposium on Electronic Imaging (Real-Time Imaging IX), San Jose, CA, Jan A. Senior, S. Pankanti, A. Hampapur, L. Brown, Y. Tian and A. Ekin, Blinkering Surveillance: Enabling Video Privacy through Computer Vision, IBM Technical Paper, RC22886 (W ), August 28, 2003,

16 16 Kenichi YABUTA, Hitoshi KITAZAWA, and Toshihisa TANAKA 4. C. Staffer and W. E. L. Grimson, Adaptive Background Mixture Models for Real- Time Tracking, CVPR99, p. 2246, Fort Colins, CO, June, A. Lipton, H. Fujiyoshi, and R. S. Patil, Moving Target Detection and Classification from Real-Time Video, Proceeding of IEEE WACV98, November W. E. L. Grimson, C. Stauffer, R. Romano, and L. Lee, Using Adaptive Tracking to Classify and Monitor Activities in a Site, IEEE Proc, Computer Vision and Pattern Recognition, pp , R. T. Collins, A. J. Lipton, and T. Kanade, A System for Video Surveillance and Monitoring, MU-RI-TR-00-12, Robotics Institute, Carnegie Mellon University, May, S. Katzenbeisser, F. A. P. Petitcolas, F. Petticolas, Information Hiding Techniques for Steganography and Digital Watermarking, Artech House Publishers, W. B. Pennebacker, and J. L. Mitchell, JPEG Still Data Compression Standard, Van Nostrand Reinhold, Announcing the ADVANCED ENCRYPTION STANDARD (AES), November 26, 2001,

17 Security Camera Privacy Protection 17 W W H H Background image A fixed monitoring camera Coding password Random number generation Moving object Original image W Moving object extraction H Pixel to be masked M ij =1 Pixel not to be masked Binary image M ij =0 Encoding Generate the scrambled/erased image Embedding moving objects Decoding by normal viewer Masked Image Reconstruction of the Special viewer input image JPEG bitstream Random number generation Decoding password Reconstruction process Reconstructed Image Fig. 1. Flow of Masking process.

18 18 Kenichi YABUTA, Hitoshi KITAZAWA, and Toshihisa TANAKA Input image Masked image (3) Scrambling/ erasing (1)Extracting moving objects Scrambling DCT Erasing Huffman coding Moving object (2)JPEG compression and encryption JPEG compression (4) Watermarking of encrypted moving objects Encryption bitstream Fig. 2. Flow of masking with watermarking. Low frequency :coefficient not used :coefficient used High frequency Fig. 3. Position of DCT coefficients for watermarking.

19 Security Camera Privacy Protection 19 Fig. 4. Input image (the 131st frame) (a) Scrambling. (b) Reconstructed. (c) Erasing. (d) Reconstructed. Fig. 5. Masked and reconstructed images generated by two methods.

20 20 Kenichi YABUTA, Hitoshi KITAZAWA, and Toshihisa TANAKA Input image Reconstructed image Fig. 6. Images of only the moving object regions.