Cloud-based Large Scale Video Analysis

Transcription

1 Cloud-based Large Scale Video Analysis Marcos Nieto Principal Researcher Vicomtech-IK4 Joachim Kreikemeier Manager V-Drive Valeo Schalter und Sensoren GmbH

2 INDEX 1. Cloud-LSVA project 2. ADAS validation 3. Architecture 4. Annotation

3 SECTION 1 Cloud-LSVA project

4 Cloud-LSVA Context Advanced Driver Assistance Systems (ADAS) market will continue to grow in the next decades. ADAS validation is a key success factor to overcome normative, ethical and user acceptance barriers

5 Cloud-LSVA Context A significant number of ADAS functions are based on vision systems: Obstacle detection Vulnerable road user detection Lane keeping These systems that preserve and protect road users take on even more importance with the arrival of the new EuroNCAP protocols.

6 Cloud-LSVA The problem Cloud-LSVA will create Big Data technologies to address the open problem of a lack of software tools, and hardware platforms, to annotate very large scale video datasets in the context of ADAS and Digital Cartography Today, to test camera-based ADAS, test vehicles are equipped with recording equipment and gather thousands of hours of driving Captured data (video) needs to be manually annotated to provide the ground truth

7 Storage Size Storage Size Cloud-LSVA goal Data hard to exploit Measurement data Annotations Ground truth Measurement data Annotations Ground truth

8 Cloud-LSVA project & consortium Annotation Infrastructure Maps Cloud Middleare Research Cloud-LSVA - Cloud Large Scale Video Analysis Co-ordinator: Vicomtech-IK4 Duration: 36M ADAS Spec. HW Sim. & Val. Research and Innovation Action H2020-ICT16-Big Data Research Outcome A Big Data technologies Web Page: Security & Data Privacy End Users & Stakeholders

9 SECTION 2 ADAS Validation Joachim Kreikemeier (Valeo Schalter und Sensoren GmbH)

10 How to test a Product The classical way to validate a product Number of different scenarios is tremendous We need new strategies to test ADAS functions

11 Common Platform for Development & Validation Collect Huge amount of data (up to 10 TB per day/car) Privacy has to be taken under account, follow national regulations Annotate Manual effort is huge Word-wide scope Collect data all across the globe and exchange data Validate XiL based on catalogues

12 Record and analyse data world-wide Field Tests (preliminary) R&D Sites

13 Scene annotation: pedestrian crossing Source: Cloud-LSVA D1.1

14 Recordings on a test-track and/or open road Sensors and time-stamp (in a scene) Source: Cloud-LSVA D1.1 Source: Cloud-LSVA D1.1

15 SECTION 3 Architecture

16 Cloud-LSVA Use case 10 X Sensorized & processing vehicle Recordings Data 10 PB x day 3D Point Cloud Data Video Recordings Text Data CAN Bus Localization GNSS Data Big Data Automatic Processing Information Consumers

17 Cloud-LSVA Architecture REST API Architecture Conceptual view

18 Web server Annotation engine Search engine Upload engine Analytics engine Docker app #1 Docker app #1 Docker app #2 Cloud-LSVA - Deployment Bare Metal with Hypervisor Docker-compose Core modules Docker-in-Docker On demand computation Core modules (Docker-compose) Dynamic modules Docker-in-Docker (DIND) Docker engine Guest OS (VM) Hypervisor Server - HW Docker (nvidia-docker) Docker VM (CentOS) VMWare VSphere CLSVA BM1

19 Courtesy: Intempora Data - RTMaps Sensors Vision, RADAR, LiDAR, GPS, Maps, IMU, V2X Actuators Motor, Wheel, Brake,Database, V2X Input DATA PROCESSING Output

20 Metadata VCD (Video Content Description) A VCD is metadata format and C++ library for real-time description of Objects, Actions, Contexts and Relations of recordings object : { id : 0, bbox : [37, 45, 625, 100], type : #Car } object : { id : 0, bbox : [31, 42, 610, 120], type : #Car } object : { id : 1, bbox : [31, 42, 610, 120], type : #Car } VCD library structure VCD entries can be stored in document-oriented databases and serialized (e.g. JSON) for transmission

21 SECTION 4 Annotation

22 Annotation - Types Types: Different annotation types (bbox, polygon, events, points, relations ) for different applications (image, video, 3D content, multi-sensor) Multi-sensor and 2D-3D annotations

23 Annotation - Scale Manual Manual + On-demand Automated + Manual revision Optimized Manual + Automated -Fully manual -Annotate each frame -Low speed -Manual GUI -On-demand analytics -Medium speed -Automation first -Manual GUI to validate -High speed -Learn best combination of manual and automated process -Very high speed

24 State-of-the-art Deep Learning detection algorithms SoA: SSD, YOLO, Faster-RCNN Multi-class, individual objects GPU: ~30-175ms/frame; CPU: ~8-10s/frame Regular and fish-eye Deep Learning Object detection

25 Deep Learning Object detection Beyond bounding-boxes Lane detection (multi-lane, curvature analysis) Face analysis (fatigue detection, gaze estimation)

26 Deep Learning Object detection Beyond bounding-boxes Lane detection (multi-lane, curvature analysis) Face analysis (fatigue detection, gaze estimation)

27 Deep Learning Semantic segmentation Scene understanding Autonomous driving, robot vision and understanding, medical applications, computational photography, annotation, image search engines, human-machine interaction Deep Learning There are two men riding a bike in front of a building on the road. There is also a car.

28 March 2017 June 2017 Deep Learning Semantic segmentation Re-training: upgrade trained model With more data With more/less clases vegetatio n pole sky car bus building sidewalk traffic sign vegetatio n pole sky car bus building sidewalk Model transfer Across datasets Real - synthetic road ego vehicle road ego vehicle Training with 2,500 images, 11 classes Adding 500 images with 13 classes car person traffic sign car traffic light person car dynamic buildin g pole sky person pole sidewalk road Example: unseen image (Synthia dataset) Model 1-11 classes Model 2-13 classes

29 Instance-based Segmentation State-of-the-art Deep Learning Instance-based segmentation and classification algorithms FCIS, SharpMask Individual objects Regular & fish-eye Object-agnostic, optional multiclass classification GPU: ~ ms/frame

30 3D Processing From point clouds to 3D objects -Combined Lidar-video (RTMaps) -Ground plane estimation -2D image-based detections -Back-projection to 3D and point clustering

31 Manual revision Web-based platform Web-based platform for loading automatic annotations Human revision Tools for rapid annotation (interpolation, tracking, merging, etc.) Bounding boxes, pixel-wise, events, 3D Integrated interface for annotators and admins. Connected to the DL services HTML-5, progressive video download

32 Manual revision Web-based 3D annotation interface 3D reconstruction (using odometry) and 3D static object labeling

33 Annotation performance 2D shapes Fully manual: 1 min recording -> 100 min annotation Semi-automatic (Beta): 1 min recording -> 10 min annotation Goal 2018: 1 min recording - > <1 min annotation Semantic annotation (pixel-wise) Fully manual: 1 frame -> min annotation Semi-autoamtic (Beta): 1 frame -> 5 min annotation Goal 2018: 1 frame -> < 1 min annotation Manual -Fully manual -Annotate each frame -Low speed Manual + On-demand -Manual GUI -On-demand analytics -Medium speed Cloud-LSVA Beta prototype, 2017 Automated + Manual revision -Automation first -Manual GUI to validate -High speed Optimized Manual + Automated -Learn best combination of manual and automated process -Very high speed 8 x Tesla P100M 16GBTCSP100M-16GB-PB 1 x Dual 10GbE Network card M7059F77A-X x Xeon 20-Core E5-2698v4 2,2Ghz50M 2 x SSD 240GB Up to 40 Inference DL processes simultaneously SSD, model VOC0712_512x512: FCIS, model fcis-coco: 37 ms / frame (real-time < 40 ms) 300 ms / frame

34 Thanks for your attention! Marcos Nieto - mnieto@vicomtech.org

35 THANK YOU