Shinpuhkan2014: A Multi-Camera Pedestrian Dataset for Tracking People across Multiple Cameras

Transcription

1 Shinpuhkan204: A Multi-Camera Pedestrian Dataset for Tracking People across Multiple Cameras Yasutomo Kawanishi Yang Wu Masayuki Mukunoki Michihiko Minoh In this paper, we present a public dataset for tracking people across multiple cameras. This dataset consists of more than 22,000 images of 24 people which are captured by 6 cameras installed in a shopping mall Shinpuh-kan. All images are manually cropped and resized to pixels, grouped into tracklets and added annotation. The number of tracklets of each person is 86. This dataset contains multiple tracklets in different directions for each person within a camera. To show the difficulty of the dataset, we evaluate it with some state-of-the-art methods. Keywords: Dataset, Multi-Camera, Re-identification. Introduction Many surveillance cameras are installed in our everyday environments for security and safety. Because of the large number of cameras, it is difficult to check all videos of the cameras manually. Therefore, automated surveillance system is needed. Tracking multiple people across non-overlapping cameras is basic component of automated surveillance system. Basically, people tracking across cameras is realized by comparing a person in one camera with each person in the other cameras which are called Person Re-identification. Various methods of Person Re-identification across non-overlapping cameras are developed in recent years. Especially, re-identification methods which utilize multiple images for each person collected by inter-camera tracking are of current interest. People tracking over not only two cameras pair but multiple camera network are also studied. In computer vision, benchmarking in standardized public dataset is often performed to compare a proposed method with related works.various techniques are developed through friendly competition with other methods. Owing to the privacy issue, we do not have datasets which containing many people legally captured by many cameras, even though some pedestrian datasets are published and highly used. Moreover, published datasets often have a limited number of cameras or samples because adding annotation to a large number of videos is a demanding task. The larger number of cameras, people, tracklets for each person, and images for each tracklet is required by researchers on people tracking across mul- ACCMS, Kyoto University Yoshida Nihonmatsu-cho, Sakyo-ku, Kyoto, , JAPAN kawanishi@mm.media.kyoto-u.ac.jp tiple cameras. In this paper, we present a novel manually annotated dataset which contains multiple tracklets for each person captured by multiple overlapping/nonoverlapping cameras in surveillance scenarios. 2. Related works Several datasets have been proposed for person reidentification purpose. First of all, we briefly survey the existing datasets. One of the most frequently utilized dataset is VIPeR (Viewpoint Invariant Pedestrian Recognition) dataset (6). The dataset contains 2 images for each subjects. 632 subjects are in the dataset. The images are normalized to pixels. This dataset is known as one of the most difficult dataset due to very large inter-camera variations such as illumination and viewpoint. Because this dataset is limited to a single image of each subject from 2 view points, it is not suitable for multiple shot or sequence based person re-identification methods. ETHZ dataset (4) is known as a dataset for pedestrian detection. Schwartz et al. (0) added annotations for person re-identification purpose onto the ETHZ dataset. This dataset has 3 different sequences. The first one consists of,000 frames with 83 subjects, the second one consists of 45 frames with 35 subjects, and the last one consists of 354 frames with 28 subjects. In this dataset, each subject is captured only with a single camera. Adding annotations to the 2008 i-lids Multiple- Camera Tracking Scenario (MCTS) (8) dataset, Zheng et al. published i-lids dataset (4) for person reidentification. This dataset consists 476 images for 9 different subjects. 4 images from non-overlapping cameras are available for each subject. The images are normalized to pixels. Just like the previous datasets, this dataset also contains a limited number of images for each subject.

2 The following two datasets have been published in the same paper written by Bak et al. (). ilids-ma dataset contains 40 subjects captured by two cameras. 46 frames for each subject are annotated manually and totally 3,680 images are in the dataset. Instead of the manual annotation, in ilids-aa dataset, images are segmented and merged into tracklets automatically for simulating more practical situations. This dataset consists of 0754 images of 00 different subjects. The annotation is performed by HOG-based human detector and tracker. Because of the automatic detector is not perfect accuracy, the annotations is noisy. Miss annotations make the dataset more challenging. 3DPeS dataset (2) is also used for tracking and multiple shot person re-identification. This dataset is composed by 6 calibrated cameras. This dataset consists of 200,000 images of 200 subjects. The number of each subject s images is quite enough, however, only 2 cameras on average are used for each subject. Therefore, it is suitable for inter-camera person re-identification but not for people tracking across multiple cameras. SAIVT-SoftBio dataset (3) is a dataset composed by 8 cameras. Videos are captured by the cameras at 25 fps and resolution of In the dataset, 50 subjects are moving in a building environment and captured by almost cameras. The dataset contains 400 frames per subject on average. Dana36 dataset (9) consists of more than 23,000 images captured by 36 cameras. The cameras are installed in a building and its outdoor environment. 27 cameras observe the subjects in the outdoor environment and the rest 9 cameras observe them in the building. Videos of the cameras have various resolutions and quality. Subjects in the dataset are 5 persons and 9 vehicles. 5 persons walk following a scenario. Acquiring videos in a real environment is sometimes difficult owing to privacy problems. Adding annotation manually to a huge amount of video frames is also a heavy task. Therefore, a simulator () is proposed to generate such videos by 3D model simulation in a virtual environment. A Python implementation of the simulator is openly available on the web. We can generate arbitrary numbers of images by running the simulation. However, the dataset currently has only 5 person models. Modifying the appearance textures is needed if we want to use more persons. 3. Details of the Shinpuhkan Dataset In this section, we describe the details of our new dataset. 3. Camera Configuration Videos of the dataset are captured at a courtyard shopping mall, Shinpuh-kan. Corridors in the mall face to the courtyard and some camera views get sunlight. We adopt not professional-use but consumer-use surveillance cameras shipped by AXIS Communications. We installed 34 cameras in the mall and capturing videos 0 hours every day. We selectively use 6 cameras for this dataset. Each camera view is illustrated as a red pie slice shape in Fig.. To capture videos clearly from morning till night, Fig. 2. Video frames of all cameras. Digit on each frame denotes the camera ID. Fig. 3. The subjects in the dataset. auto gain and auto white balance are enabled. Their frame rates are variable, which are around 0 fps. There are two types of camera resolutions. One is and the other is There are various illumination conditions from dark to well-lighted. We have various quality of images because image quality is depend on the illumination condition. All of camera views are shown in Fig.2. Resolution of the camera view of camera ID 0 to are pixels and camera ID 2 to 6 are pixels. 3.2 Pedestrians and Scenario The number of persons in the dataset is 24. Sample images of all of the persons are shown in Fig.3. We recruited most of them in our university, so they are university students around 20 years old. For this dataset, we acquired videos at early morning before open the shopping mall to avoid people except the subjects are captured by the cameras. We defined a scenario (Fig.4) and forced all subjects to follow it. They walked along the path in Fig.4 twice. Once a per-

3 Shinpuhkan204: A Multi-Camera Pedestrian Dataset for Tracking People across Multiple Cameras F 2F 3F Fig.. Camera locations in the shopping mall. The camera views are illustrated as a red pie slice shape. Green arrows indicates stairs, direction of an arrow means the upper floor start/goal F F F 3 8 Fig. 4. The scenario of this dataset. All subjects move along with the arrow from the start to the goal. Digits indicated beside arrows denote the order of the path which subjects move along with. Table. The numbers of tracklets for each person within cameras. cameraid Cam0 Cam02 Cam03 Cam04 #Tracklets/person cameraid Cam05 Cam06 Cam07 Cam08 #Tracklets/person cameraid Cam09 Cam0 Cam Cam2 #Tracklets/person cameraid Cam3 Cam4 Cam5 Cam6 #Tracklets/person son walks along with the path, he/she is captured by the cameras 43 times in different directions. Therefore, the dataset consists of 86 tracklets for each person. That means persons appeared 5.4 times on average to a camera view. The numbers of tracklets for each person in each camera is shown in Table. 3.3 Images and Annotations A tracklet consists of a sequence of images cropped by bounding boxes of a person from the time he/she comes into a camera view to the time he/she goes out. Each bounding box for each image is annotated manually which fits the person. The sizes of bounding boxes are from to We removed frames that the person are occluded by others or occluding objects such as columns, walls and fences. We also removed too dark frames. Basically, we selected more than 8 frames for each tracklet. In a small number of tracklets, exceptionally, we could select only 3 image because of long-term occlusion. After cropping images, we resize them to pixels. For different aspect images, resized images are filled with extrapolated pixels using their border pixels. The cropped images are saved in JPEG format. The image file names are consists of 3 digits of person ID, 2 digits of camera ID, 2 digits of tracklet ID for each person within a camera, and 3 digits of image ID within a tracklet ( jpg, jpg,..., jpg). With the filenames, we can identically specify all images with person ID, camera ID, tracklet ID and image ID. The dataset also contains an annotation file of CSV format. It includes filenames, observation times and positions of persons for each file. 3.4 Privacy, Consent and Limitations All subjects agreed with our privacy policy for video recording. This dataset do not contain other persons who do not agree with the terms. We limit the use of images in the dataset only for research purposes. We also restrict redistribution of the dataset. If you want to evaluate your method on this dataset, you need to agree with our license statement. 4. Evaluation To show the utility and difficulty of the proposed dataset, we performe evaluation with person reidentification methods. 4. Configuration of the Evaluation For each tracklet for each camera, we used a tracklet in the camera as a query and the other tracklets in the other

4 Recognition percentage Recognition percentage CMC on the Shinpuhkan204 dataset MPD CRFS CRNP Rank Fig. 5. Experimental results. Recognition rate on each camera MPD CRFS CRNP Camera ID Fig. 6. Comparison between the three methods on recognition rate for each camera. cameras as a gallery set. On the dataset, we compared following three different person re-identification methods, which are set-based comparison methods: Minimum Point Distance (MPD) (5), which is just calculate the minimum distance between two features in different tracklets; Correntropy of Robust Feature Selection (CRFS) (7), which reduce dimensionality by selecting features; and Collaboratively Regularized Nearest Points (CRNP) (2), which is an extension of Regularized Nearest Points (RNP) (3) to set-to-sets distance optimization. All of them use densely sampled color histograms (DCHs) based feature vector for an image as same as in the paper (2). They deal with the features extracted from images in a tracklet as a set. For CRFS, we used the recommended parameters in the paper. For CRNP, because the dataset has a much larger number of images in a gallery set than the number of images in a query, we use λ =, λ 2 = 45, γ =, and γ 2 = Results Results are shown using Cumulative Matching Characteristic (CMC) curves of the rank top 20 results in Fig.5. The graph shows the average recognition percentages of all queries for each method. For further evaluation, we show the difference of difficulties between different cameras. The comparison results are shown in Fig.6. For all cameras, CRFS and CRNP perform better than MPD. At Camera 6, recognition rate of MPD is much lower than the other methods. It is because the camera observes the widest area, so persons in the camera view are relatively small and features in tracklets correspond to the camera are quite different from other features in the other camera views. At Camera 4, recognition rates of all the methods are lowest in all cameras. It can be considered that the camera view is also wide, moreover, its illumination condition is much different from other camera views (see Fig. 2). Therefore, searching a person with a query from these cameras is relatively difficult. 5. Conclusion In this paper, we presented a public dataset for tracking people across multiple cameras and described the details of the dataset. One of our future work is extensive analysis on the dataset. For inter-camera people tracking, compatibility of neighboring two cameras severely affects on tracking difficulty level. We want to evaluate the difficulties of person matching between all pairs of cameras in the dataset. Another future work is expanding this dataset further. There are two directions; one is growing dataset size and the other is annotating additional information. For the first direction, we have many unannotated videos which can be published. We will publish the 2nd version of this dataset which contains much more subjects. For the other direction, calibration data of the cameras and spatio-temporal information between cameras can be annotated. Acknowledgment This work was supported by R&D Program for Implementation of Anti-Crime and Anti-Terrorism Technologies for a Safe and Secure Society, Funds for integrated promotion of social system reform and research and development of the Ministry of Education, Culture, Sports, Science and Technology, the Japanese Government. References ( ) Slawomir Bak, Etienne Corvee, Francois Bremond, and Monique Thonnat. Boosted human re-identification using Riemannian manifolds. Image and Vision Computing, August 20. ( 2 ) Davide Baltieri, Roberto Vezzani, and Rita Cucchiara. 3dpes: 3d people dataset for surveillance and forensics. In Proceedings of the st International ACM Workshop on Multimedia access to 3D Human Objects, pages 59 64, Scottsdale, Arizona, USA, November 20. ( 3 ) Alina Bialkowski, Simon Denman, Patrick Lucey, Sridha Sridharan, and Clinton B. Fookes. A database for person re-identification in multi-camera surveillance networks. In Digital Image Computing : Techniques and Applications (DICTA 202), pages 8, Esplanade Hotel, Fremantle, WA, 202. IEEE. ( 4 ) A. Ess, B. Leibe, and L. Van Gool. Depth and appearance for mobile scene analysis. In Computer Vision, ICCV IEEE th International Conference on, pages 8, 2007.

5 Shinpuhkan204: A Multi-Camera Pedestrian Dataset for Tracking People across Multiple Cameras ( 5 ) M Farenzena, L Bazzani, A Perina, V Murino, and M Cristani. Person Re-Identification by Symmetry-Driven Accumulation of Local Features. In Proc. of CVPR, pages , 200. ( 6 ) D Gray, S Brennan, and H Tao. Evaluating Appearance Models for Recognition, Reacquisition, and Tracking. In Proc. of PETS, volume 3, pages 4 49, ( 7 ) Ran He, Tieniu Tan, Liang Wang, and Wei-Shi Zheng. l2, regularized correntropy for robust feature selection. In Computer Vision and Pattern Recognition (CVPR), 202 IEEE Conference on, pages , 202. ( 8 ) Home Office. ilids datasets. imagery-library-for-intelligent-detection-systems. ( 9 ) Janez Per, Vildana Sulic Kenk, Rok Mandeljc, Matej Kristan, and Stanislav Kovacic. Dana36: A multi-camera image dataset for object identification in surveillance scenarios. In 203 0th IEEE International Conference on Advanced Video and Signal Based Surveillance, volume 0, pages 64 69, Los Alamitos, CA, USA, 202. IEEE Computer Society. (0) W.R. Schwartz and L.S. Davis. Learning discriminative appearance-based models using partial least squares. In Computer Graphics and Image Processing (SIBGRAPI), 2009 XXII Brazilian Symposium on, pages , () W. Starzyk, A. Domurad, and F.Z. Qureshi. A virtual vision simulator for camera networks research. In Computer and Robot Vision (CRV), 202 Ninth Conference on, pages , 202. (2) Yang Wu, Michihiko Minoh, and Masayuki Mukunoki. Collaboratively regularized nearest points for set based recognition. In In Proc. of The 24th British Machine Vision Conference (BMVC), 203. (3) Meng Yang, Pengfei Zhu, Luc Van Gool, and Lei Zhang. Face recognition based on regularized nearest points between image sets. In 203 0th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), volume 0, pages 7, Los Alamitos, CA, USA, 203. IEEE Computer Society. (4) Wei-Shi Zheng, Shaogang Gong, and Tao Xiang. Associating groups of people. In Proc. BMVC, pages , doi:0.5244/c