What's new in GStreamer

Transcription

1 What's new in GStreamer GUADEC 2015, Göteborg 7 August 2015 Tim Müller <tim@centricular.com> Sebastian Dröge <sebastian@centricular.com>

2 Introduction

3 Who? Long-term GStreamer core developers and maintainers since 2005/6 Founders of Ltd Consultancy offering services around GStreamer, graphics and multimedia related software

4 What is this all about? What is GStreamer? What happened in the last year?

5 GStreamer? What is it?

6 GStreamer Pipeline-based multimedia framework Basic building blocks: elements connected via pads Set of libraries with abstract API Plugins for specific features Often wrapping other libraries, e.g. libav/ffmpeg, OpenCV, various codec libraries

7 GStreamer (cont'd) Open Source, LGPL v2.1+ Proprietary applications and plugins allowed Cross-platform Linux, embedded Linux, Windows, OSX, ios, Android, Binary SDKs for Windows, OSX, ios, Android x86, ARM, PowerPC, SPARC, Stable GObject based C API/ABI Bindings for many languages Python, C++, JavaScript,.NET,

8 GStreamer (cont'd) Not a media player or playback library Not a codec and protocol library Not a transcoding tool Not a streaming server But can be (and is) used to implement all that

9 GStreamer (cont'd) Goals Flexible and extensible design Easy to integrate with other software (in both directions) Large, active developer and user community Large commercial ecosystem Users Contributors Consultancies

10 What happened in the last year?

11 Releases 1.4 released, latest bugfix release As always with binaries for all non-linux platforms most likely to finish the release series 1.6 really soon now! Join us after the core days for bug triaging, bugfixing and getting everything release-ready

12 Better UI toolkit integration GTK+ video sink / widget (>= 3.16 with OpenGL) Works very well with CSS styling, including transparency and overlays! No need for Clutter anymore See Matthew's talk today at 16:30 in Asperö Qt QML video sink / QML element (OpenGL) X11 and Wayland so for Should be portable to all platforms, including ios and Android OS X CoreAnimation video sink / CALayer (OpenGL) Should be portable to ios with some minor effort

13 GTK+ video sink / widget details How does it work? You create a GStreamer video sink as usual gtksink or gtkglsink gtkglsink should be placed into a glsinkbin ( sink property) if no other GL operations should be done. glsinkbin should be used as the real video sink If using playbin, set the gtksink or glsinkbin on the video-sink property gtk(gl)sink has a widget property that gives you a GtkWidget Place the widget somewhere in your UI and handle it like any other widget Anything received by the sink will be automatically rendered in your UI into the widget

14 GTK+ video sink / widget details Bonus: CSS styling, overlaying, transparency, etc. are supported Hardware decoders that can output GL things will be zerocopy (e.g. VAAPI)!

15 GstPlayer High-level playback API started last GUADEC Planned to be merged into GStreamer soonish (for 1.8) For all your media playback needs All (?) features needed by Totem are implemented Should reduce Totem's BaconVideoWidget code by 90+% together with the GTK+ video sink Example GTK+, Android, ios and command-line applications included Qt QML application WIP

16 GstPlayer

17 Chrome / Chromium GStreamer backend released by Samsung R&D ackend HTML5 video/audio tag support Uses GstPlayer, less than 10k SLOC Hardware accelerated OpenGL rendering Planned to be merged into Chromium upstream

18 WebKit Lots of improvements to the GStreamer backend OpenGL accelerated rendering Media Source Extensions and Encrypted Media Extensions (ugh!) support Improved WebAudio support WebRTC integration started with OpenWebRTC

19 WebRTC New W3C standard, basically Skype for the browser. It's going to be big! Kurento WebRTC media server Includes extractable GStreamer element for integrating WebRTC sessions into a pipeline: WebRTCEndPoint OpenWebRTC simple WebRTC API Released by Ericsson R&D last year All requires applications integration with Web services, simple nowadays with the GNOME platform libsoup now has WebSockets support json-glib FTW Lots of related improvements to GStreamer's RTP stack

20 WebRTC Looking for an application to write? Interested in writing a native GNOME application for Firefox Hello but don't know how? Talk to me, it would be great showcase and fun to implement!

21 pinos aka PulseVideo API & service for accessing multimedia streams Cameras, other devices, streams, Supports compressed streams, also audio Multiple users of the same device Give access to your camera to your sandboxed application See Alexander Larsson's talk this morning at 10:00 DBus API for control fd passing between processes to share media data

22 The 3 rd Dimension OpenGL support had major improvements and refactoring Easier integration into other libraries and custom OpenGL code GTK+! Qt! SDL! CoreAnimation! Chrome! WebKit! Wayland support See Matthew's talk today at 16:30 in Asperö Support for stereoscopic 3D video

23 Quality Assurance Continuous integration with Jenkins Builds for all supported platforms Unit / integration tests, gst-validate on Linux Major improvements and new tests for gst-validate Also run by CI server Tests higher-level GStreamer scenarios and runs over a media file library GstHarness Unit test helper for element behaviour tests, including (real-time) timing Moar unit tests

24 Assorted RTSP server RECORD support to send media from a client to a server Live audio/video mixing support ios/osx zerocopy decoding, Android WIP Improved HTTP Adaptive Streaming support (DASH, HLS) splitmuxsink for writing standalone MP4/WebM/etc in separate files

25 Assorted New GStreamer Editing Services backend No more GNonLin Expect PiTiVi to work even better Switch from libav back to ffmpeg New trick-mode API flags: keyframe-only, no audio, etc. Non-GL Wayland video sink v4l2 improvements dmabuf and renegotiation support

26 Bugzilla & GIT stats Bugzilla: / bugs, Δ +454 bugs 1638 open bugs, 902 non-enhancement bugs Last year / -1823, Δ +313 bugs Ratio changed from 1.17 to 1.23 We need to do something about that! If you're interested in working on GStreamer, now would be a good time to start. Many open tasks, many for every level of experience

27 Bugzilla & GIT stats 220 contributors in the last 12 months, 50 with more than 12 commits in that period +2% more contributors than in the previous year 6521 commits in the last 12 months -11% compared to previous year new lines of code (+5%) Excludes comments and whitespace

28 The Future

29 Development Tools Tracing subsystem, better debugging facilities Live pipeline debugging application (GSoC project) See Marcin's talk today some time after 17:30 during the interns lightning talks Debug log viewer Continuous integration Automated QA and builds More usage of new QA tools, more tests gst-validate, GstHarness

30 Development Resources More convenience APIs Merge GstPlayer GstPlayer is there, what else do people need? Move to Phabricator Bug tracking, code review, wiki New website? More tutorials?

31 Features Improved trick mode support Incl. HTTP adaptive streaming DASH/HLS streaming server And generic file segmenting More cross-platform improvements Windows support is lacking behind! Android zerocopy decoding

32 Features Decodebin3 better support for reconfiguration and dynamic scenarios Better APIs and support for metadata extraction Merge various 3 rd party plugins, move plugins from -bad to -good Android camera source, Kurento WebRTC endpoint, SCTP camerabin (cheese!)

33 Thanks! Any questions? Find more information at Join us at the GStreamer Conference 8-9 Oct 2015 Dublin, Ireland