Video Retrieval. Yoshinobu Hotta FUJITSU LABORATORIES LTD. Copyright 2012 FUJITSU LABORATORIES LTD.

Transcription

1 Video Retrieval Yoshinobu Hotta FUJITSU LABORATORIES LTD.

2 Who am I? My research areas : Image processing Document image processing Japanese OCR (Optical character recognition) Text extraction from natural scene images Video analysis today s topics Gesture user interface Text extraction Gesture user interface 1

3 Outline 0. Introduction 1. Caption extraction and recognition 2. Digest video generation 3. Visual user interface 4. Summary 2

4 0. Introduction Importance of video retrieval By 2013, more than 25 percent of the content that workers see in a day will be dominated by pictures, video or audio (Gartner 2008) Current status : - Many video portals : YouTube, Google videos, Veoh.. - Many legacy videos : TV programs, Surveillance videos, E-learning videos,.. - Wide spread of smart phones Video retrieval is needed in various scenes 3

5 Applications in a wide variety of businesses Broadcast Manufacturing Security 爆笑シアター Editing Broadcast Maintenance manual Weld verification Accident inspection Interrogation video Healthcare Delivery Share Agriculture Video analysis/ retrieval Surgery video Informed consent Production Surveillance Growth observation Damage control Retail distribution Education Public service On sale! Shoplifting Announcement effect Educational videos Museum signage Proceedings video Illegal dumping 4

6 Video Retrieval System Scene retrieval from various aspects (metadata) such as object, caption, color, shape, and sound テロップ Object Caption Color Shape Sound Scene retrieval Visual user interface Digest Video Object Caption (Telop) Metadata DB Color Shape Sound Metadata annotation Video DB 5

7 1. Caption extraction and recognition What is a caption? Character strings which appears in a frame Video Database ビストロ Search : Retrieved result 6

8 Our target - Various types of captions used in Japanese TV program Uniform character color Multiple character color Uniform Background Complicated background Method A, B Method C Exception1 : movable caption (ex. End credits of movies) Exception2 : Gradation Target of our method 7

9 Outline of caption extraction / recognition Caption extraction Extract caption region candidates from static part of the video Input : Video stream Vertical caption Caption recognition time Recognize characters inside caption region Holizontal caption (It is expressed as a rectangular parallelepiped since caption regions are shown at same place over a period of time. Extracted caption region レトロ車両 (utilized as video index) 8 Copyright 2012 FUJITSU LIMITED

10 Whole flow Frame extraction Caption region extraction Character extraction Character recognition Caption text Video Character extraction A character string image Character recognition 提供富士通研究所 It is important to extract character string from complicated background Recognized text 9

11 Problems of conventional method (1)A method which utilizes uniform character color Effective for single colored caption Not effective for multi-colored caption 富士通研究所 OK 提供研空斯 NG (2)A method which utilizes uniform background color Not effective in complicated background OK NG Character extraction with multiple-color from complicated background is not easy 10

12 Problems of conventional method (2) (3) A method which utilizes color separation and select single color plane Noise removal Character recognition 提供 研空斯 Color Separation 富士通 庶侠官 ± 遍研究所 Selected ( し A recognition result of single color plane include only part of correct texts. 11

13 Multi-colored caption recognition (proposed) Decompose an input color image and extract character region candidate Recognition reliability and continuity of rectangles and colors are evaluated by Dynamic programming Inputted character string Blue Color Separation 提 99 供 81 研 36 空 90 斯 59 Green 富 98 士 99 通 90 White Result: 提供富士通研究所 庶 34 侠 41 官 76 ± 81 遍 83 研 89 究 87 所 90 12

14 Recognition result Result: 気になる費用は Result: 問題ここはどこ 13

15 2. Digest Video Generation Automatic generation of digest videos from TV program, movie, surveillance / homemade videos etc. Application Preview Preview START START Sports program Digest Video Generation Homemade videos 14

16 Scene analysis for digest video generation Specific scenes in a video are detected by analyzing video/sound Highlight detection Examples of scene analysis Analyze the highlight of sound(a shout of joy) of sports program and detect a score scene Camera motion estimation Analyze camera motion such as zoom-in, zoom-out and hand-shake in video and then detect the motion change scene Volume of sound Zoom in Zoom out time Speech/Sound detection Analyze sound features and detect a conversation scene or music scene Detect only song scene from music show 15

17 How to utilize sound feature Origina l video Ideal detection Score scene time Features Volume of sound + Removal of announcer s voice + Sudden sound change Extracte d value 16

18 Example1: Sports video retrieval Extracting various kinds of scenes for digest video creation Caption features for score, sound features for a shout of joy, image features for ground color are integrated Caption Integration Highlight scenes Sound feature Image feature Highlight scene extraction 17

19 Example2: Illegal dumping scene retrieval What is Illegal dumping? Someone illegally dumps waste. Illegal dumping is a social issue. Officials in local governments check surveillance videos to retrieve illegal dumping scenes. Image feature (Frame difference for human detection) Integration Highlight scenes Image feature (Frame difference for waste detection 18

20 3. Visual User Interface Easier video search with various GUIs Slide view Turn over one after another like the shuffle of cards Color/shape based view Arrange similar color or shape images closely Spiral view Simultaneous reproduction of images on a thumbnail 19

21 Demo of visual user interface Reduce retrieval cost by combination of keyword search and visual user interface 映像の UI に求められるもの 一覧性 20

22 4. Summary Several video retrieval technologies were introduced. Caption extraction and recognition Digest video generation Visual user interface In next 5-10 years Various types of metadata such as peoples name, object name (ex. Landmarks) can be added with progress of object recognition Web information are dynamically linked with contents of video ex. Peoples face->facebook, Landmark -> Wikipedia Restaurant -> Reputation site.. 21

23 22