Synchronised multi-room media playback and distributed live media processing and mixing

Transcription

1 Synchronised multi-room media playback and distributed live media processing and mixing LCA 2016, Geelong 3 February 2016 Sebastian Dröge <sebastian@centricular.com> 1

2 Introduction 2

3 Who? Long-term GStreamer core developer and maintainer since 10 years now Did the last few releases and worked on every part until now Co-founder of Ltd Consultancy offering services around GStreamer, graphics and multimedia related free software 3

4 What? How to synchronize media between multiple devices over the network, and How GStreamer helps here 4

5 GStreamer? What is it? AKA: Very short introduction! for more details 5

6 GStreamer Free software, pipeline based multimedia framework 6

7 GStreamer (cont'd) Goals Flexible and extensible design based on simple concepts Easy to integrate with other software (in both directions) Stable API 7

8 GStreamer (cont'd) 8

9 GStreamer (cont'd) Community-driven free software, LGPL v2.1+ Cross-platform Stable GObject based, object-oriented C API/ABI 9

10 GStreamer (cont'd) Bindings for many languages 250+ plugins for various features available Batteries included! 10

11 GStreamer (cont'd) Not a media player or playback library Not a codec and protocol library Not a transcoding tool Not a streaming server But can be (and is) used to implement all that 11

12 Scenarios Or: Why do we talk about this at all? 12

13 Synchronised multi-room playback 13

14 Video Walls CC BY 2.0 CES LG video wall By Doug Kline 14

15 Live mixing and recording CC-BY-2.0, Nayu Kim 15

16 Clocks What time is it? How long is a second? 16

17 GStreamer Clocks 17

18 A man with a watch knows what time it is. A man with two watches is never sure. Segal's Law 18

19 Clocks No two clocks in the universe run at the same rate or show the same time Need to declare a master clock Relate clocks to each other 19

20 Multiple clocks? pipeline system clock audio source audio filters/etc src video source audio-sink src sink video filters/etc src video-sink src sink 20 sink sink

21 Relating clocks to each other 21

22 Network Clocks Need to share a clock with multiple devices Idea: Same as with multiple local clocks Observations of remote clock times Main problem: network delay and jitter You all used this: automatic clock setting on your computer / phone / etc via NTP 22

23 GStreamer Net, NTPv4 & PTP Clock NetClock has existed since around 2005 Custom protocol similar to NTPv4 NTPv4 (RFC4330) client clock since 1.6 PTP (IEEE ) client clock since

24 NTPv4 24

25 PTP IEEE

26 NTPv4 / GStreamer Net Clock Simple, well-supported protocol Works well over the Internet ms accuracy GStreamer Net clock server runs on unprivileged port 26

27 PTP Clock Local, wired networks Part of many IP broadcasting standards μs to ns accuracy Hardware support Low overhead Automatic clock selection 27

28 Media Transport How do we get the media from A to B? 28

29 RTSP/RTP HTTP-style request/response protocol Media setup via RTSP/SDP Media streaming via RTP RTP has clearly defined timing semantics 29

30 GStreamer RTSP/RTP support GStreamer RTSP server Write a server in 6 SLOC RTSP source included Works the same as file input All customizable for our needs Flexible full-featured RTP stack 30

31 Synchronisation When to show this frame? 31

32 RTP Packets with timestamps and sequence numbers Per-stream synchronisation Lossy channels: detect packet loss and packet reordering Transmitted in real-time Mappings for every major codec and metadata schemes 32

33 RTP Synchronisation Estimate remote sender clock Use packet timestamps Not useful for inter-device or inter-stream synchronisation Handled in GStreamer's rtpjitterbuffer 33

34 RTP Synchronisation Real Time Clock Skew Estimation over Network Delays (TR ) Dominique Fober, Yann Orlarey, Stephane Letz 34

35 RTCP Bidirectional communication Feedback, requests, statistics Timing information for inter-stream synchronisation Handled in GStreamer's rtpbin/rtpsession 35

36 RTCP Sender Reports Stream A Stream B 5 23: : : : : : :10.16 RTCP 3 49:21.52 RTCP 9 23: :21.56 RTCP SR: 23: :00.10 RTCP SR: 49: :

37 RTCP Synchronisation Also allows inter-device synchronisation If devices use a common NTP clock Easy to use in GStreamer See

38 RTP Header Extension RTCP only every now and then RTCP tricky on the Internet Why not put NTP RTP mapping into packets? RFC

39 RFC7273 Solves the remaining problem works out of the box Signalling for Media clock Clock RTP offset Half-finished patch 39

40 Open Issue: Receiver Latency Receiver A Clock: 12:01.00 Clock: 12:01.20 Clock: 12:01.25 Source Decoder Sink src sink src sink Receiver B Clock: 12:01.00 Clock: 12:01.10 Clock: 12:01.12 Source Decoder Sink src sink src sink 40

41 Open Issue: Capture Time? Sender Time? Pipeline Clock: 12:01.00 RTP: 23:44.02 Clock: 12:01.20 RTP: 23:44.02 Clock: 12:01.25 RTP: 23:44.02 Source Other things Sink src src sink 41 sink

42 Interoperability GStreamer-only solution? No! 42

43 RTCP Synchronisation Need to agree on a common clock Works with 3rd party implementations Completely based on RTP RFCs, nothing else Capture vs. send time usually undefined 43

44 Ravenna / AES-67 IP based audio broadcasting standard Raw audio via RTP PTP clock RFC7273 for media clock distribution Supported by many professional broadcasting applications 44

45 SMPTE 2022 / 2059 IP based video transport standard Remote, real-time video production RTP for audio / video SDI over IP PTP clock FEC 45

46 Thanks! Any questions? Feel free to write me a mail sebastian@centricular.com 46