MULTIPLE DRONE CINEMATOGRAPHY I. Mademlis 1, I. Pitas 1, A. Messina 2

Transcription

1 MULTIPLE DRONE CINEMATOGRAPHY I. Mademlis 1, I. Pitas 1, A. Messina 2 1 Aristotle University of Thessaloniki 2 Rai - Centre for Research and Technological Innovation

2

3 MULTIDRONE in a nutshell H2020 ICT Research and Innovation Actions on multiple-actor systems 8 partners Univ. of Thessaloniki, University of Bristol, THALES, RAI, Deutsche Welle, Istituto Superior Tecnico, Univ. of Seville, Alerion

4 Main Objectives Improved multiple drone decisional autonomy, robustness and safety; Innovative, safe and fast multiple drone active perception and AV shooting Application and demonstration in three media production scenarios

5 UAV Cinematography UAV cinematography is mainly derived from traditional ground and aerial cinematography, but must also take into account UAV-specific limitations, capabilities, properties. A visual vocabulary of UAV cinematography can be defined, consisting in: Shot types Combinations of framing shot types + UAV / camera motion types. 2009, Jim Zuckerman

6 UAV Cinematography Composition principles: Central Composition. Rule of Thirds. Lighting rules. Depth-of-Field / Focus settings. Need to define a standardized UAV shot type taxonomy

7 UAV Cinematography Actual drone footage and related articles / guidelines were used to this end: A total of 8 framing (static) shot types and 26 UAV / camera motion types suitable for UAV media production have been identified. Camera motion types were clustered into groups according to their characteristics. Visually pleasing combinations of framing shot types and camera motion types were identified.

8 Framing Shot Types Framing Shot Types are more or less those of traditional cinematography. Most are defined based on the percentage of the video frame width / height covered by the single target / subject. FRAMING SHOT TYPE Percentage of frame width/height covered by target Extreme Long Shot (ELS) <5% Very Long Shot (VLS) 5-20% Long Shot (LS) 20-40% Medium Shot (MS) 40-60% Medium Close Up (MCU) 60-75% Close Up (CU) >75%

9 Framing Shot Types A couple of Framing Shot Types deal with two or more subjects / targets: 2 Shot / 3 Shot: 2/3 subjects appear on frame, equally visible (typically LS or MS). Over the Shoulder (OTS): Adapted from traditional cinematography OTS. Main target fully visible, secondary target visible at the video frame edge.

10 Framing Shot Types Example UAV shot types when shooting boat targets from the side. Extreme Long Shot Long Shot Medium Close Up Two Shot

11 UAV / Camera Motion Types UAV / camera motion types can be considered as either scene-oriented or target-oriented. Four groups of UAV / camera motion types were defined. Static shots (6). No UAV motion, target may or may not be present: Static Shot (SS) Static Shot of Still Target (SSST) Static Shot of Moving Target (SSMT) Static Aerial Pan (SAP) Static Aerial Tilt (SAT).

12 UAV / Camera Motion Types Dynamic shots (6). Moving UAV, no target: Moving Aerial Pan or Tilt (MAP, MAT) Pedestal / Elevator Shot (PS) Bird s Eye Shot (BIRD) Moving Bird s Eye Shot (MOVBIRD) Survey Shot (SURVEY) Fly-Through (FLYTHROUGH).

13 UAV / Camera Motion Types Target Tracking shots (11). UAV motion depends on target motion: Moving Aerial Pan with Moving Target (MAPMT) Moving Aerial Tilt with Moving Target (MATMT) Lateral Tracking Shot (LTS) Vertical Tracking Shot (VTS) Orbit (ORBIT) Fly-Over (FLYOVER) Fly-By (FLYBY).

14 UAV / Camera Motion Types Target Tracking shots (continued) Chase/Follow Shot (CHASE) Descent (DESCENT) Descent-Over (DESCENTOVER) Ascent (ASCENT)

15 UAV / Camera Motion Types Lateral Tracking Shot (LTS) Camera stays focused on the moving target. UAV flies sideways / in parallel to the target, matching its speed. Source: Youtube: "5 Drone Moves Every Flier Should Know",

16 UAV / Camera Motion Types Chase/Follow (CHASE) Camera stays focused on the moving target. UAV follows / leads the target from behind / from the front, matching its speed. Source: Youtube: "Drone footage of Cycle- Racing", 4LC9ig

17 UAV / Camera Motion Types Orbit (ORBIT) Camera gimbal is slowly rotating, so as to keep the still or moving target properly framed. UAV circles around the target while following its trajectory (if any). Source: Youtube: "Drone Chasing Horses", uw8py9qmw

18 UAV / Camera Motion Types Static Aerial Tilt (SAT) UAV hovers. Camera gimbal rotates slowly around the pitch axis in order to capture the scene context. Source: Youtube: "5 Drone Moves Every Flier Should Know", =1hz-lkx4o6c

19 UAV / Camera Motion Types Moving Bird s Eye Shot (MOVBIRD) Camera remains stable facing vertically down. UAV is slowly flying parallel to the terrain with constant velocity. Source: Youtube: "How to film amazing aerials with your drone DroneFilmSchool", ch?v=zmqcsdj7vbe

20 UAV / Camera Motion Types Survey Shot Camera remains stable facing ahead or backwards. UAV is slowly flying parallel to the terrain with constant velocity. Source: Youtube: "Manor House Stables - Drone Film - Horses", ch?v=56qonxc28dw copyright owner: M7 Aerial

21 UAV / Camera Motion Types Dynamic Target shots (3): a target exists, but UAV trajectory also depends on other factors Constrained Lateral Tracking Shot (CONLTS) Pedestal/Elevator with Target (PST) Reveal Shot (REVEAL) Constrained Lateral Tracking Shot (CONLTS) Camera remains stable, focused on the moving target. UAV follows the target but it is constrained to move onto a pre-defined flight plane vertical to the ground plane. Useful in sports, e.g. football.

22 Multiple UAV Motion Types Motion types involving 2 or more drones in orchestrated motion can be also considered.

23 Target detection

24 Target detection

25 Target detection Target/object examples: athletes, boats, biclycles.

26 Target detection

27 Object detection Single view object detection Deep learning (CNN) object detection. Light weight CNNs for object detection. Multiple view object detection.

28 Object detection Object detection = classification + localization: Find what is in a picture as well as where it is.

29 Object detection Input: an image. Output: bounding boxes containing depicted objects. Each image contains a different number of objects (outputs). Typical approach: train a specialized classifier and deploy in sliding-window style to detect all object of that class. Very inefficient, quite ineffective. Goal: combine classification and localization into a single architecture for multiple, multiclass object detection.

30 Object detection with CNNs Deep Learning (DL) approach: train a classifier on, say, 1000 classes of ILSVRC. OverFeat (2013) was one of the first DL approaches to object detection. Its convolutional method made multi-scale sliding window efficient. Based on AlexNet architecture.

31 Object detection with CNNs Overfeat: Object detection at increasing image resolutions Sermanet, Pierre, et al. "Overfeat: Integrated recognition, localization and detection using convolutional networks." International Conference on Learning Representations (ICLR2014), CBLS, April

32 Object detection with CNNs Impact of Deep Learning. Pascal VOC (object detection)

33 Imaging for drone safety Human crowd detection for safe autonomous drones Emergency landing site detection.

34 Human crowd detection for safe autonomous drones Detect where crowd exists. Comply with legislation. Detect emergency landing points. Provide heatmaps of the estimated probability of crowd presence in each location.

35 Human crowd detection for safe autonomous drones

36 Human crowd detection for safe autonomous drones Limited previous efforts on crowd detection, using computer vision techniques. Crowded scenes are considered in related research works involving crowds, e.g.,: crowd understanding, crowd counting, human detection and tracking in crowds.

37 Human crowd detection for drone flight safety using CNNs In [1], a method utilizing Convolutional Neural Networks (CNNs) for crowd detection is proposed. Two approaches: transforming a pre-trained CNN to a fast, fully-convolutional network, devising a two-loss-training model, enhancing the separability of the crowd and non-crowd classes. [1] Tzelepi, Maria, and Anastasios Tefas, "Human Crowd Detection for Drone Flight Safety Using Convolutional Neural Networks." in European Signal Processing Conference (EUSIPCO), Kos, Greece, 2017.

38 Human crowd detection for drone flight safety using CNNs Provide lightweight models, as imposed by the computational restrictions of the application. Effectively distinguish between crowded and non-crowded scenes. Provide crowd heatmaps to semantically enhance flight maps by defining no-fly zones.

39 Human crowd heatmaps

40 Human crowd heatmaps

41

42 Production Planning Schema XML Schema

43 Conclusions MULTIDRONE is aiming at defining and developing innovative multi-actor autonomous system for UAV-based media production Many challenges, here we focused on two Drone cinematography principles for media production planning Target detection for automated tracking using DL

44

45