Creating rich WebRTC Applications with Kurento

Transcription

1 Creating rich WebRTC Applications with Kurento Tech Talks JdeRobot 28 March 2017 Universidad Rey Juan Carlos Fuenlabrada, Madrid Miguel París

2 Who I am Miguel París Education Software Engineer Telematic Systems Master's Work experience Intern GSyC research group URJC - Researcher Kurento Architect and Real Time area responsible 2016-Now Twilio - WebRTC Media Server senior engineer 2 Contact mparisdiaz@gmail.com

3 Overview Kurento Philosophy What's Kurento? Kurento Architecture Kurento API WebRTC Kurento & WebRTC Enjoying Kurento (demos & code) 3

4 Kurento Philosophy (I) Meaning 4

5 Kurento Philosophy (II) Multimedia infraestructure for the Future Internet Media is here Media got there Transport es s r tu ure c ru uct t s r t a r s a nf r i f in al t n tio erne i a d In t r T re u t Fu Media is Enrich here Events Media got there Augment Adapt Analyze Transform Store Sensors Context 5

6 Kurento Philosophy (III) The problem: Complexity 6

7 What's Kurento? (I) The equation to face the complexity Future Internet Multimedia Infrastruct ure Simple Developm ent APIs 7

8 What's Kurento? (II) Multimedia infrastructure Interoperable media exchange (multiplatform/multiprotocol) WebRTC, RTP, HTTP (video tag), etc. Process media (Computer vision, augmented reality, media indexing, etc.) Media and metadata recording and recovery Transform and adapt media (H.264, H.263, VP8, Ogg, and others) Media routing and mixing Etc. APIs REST API JavaScript API Java API Is distributed through a flexible FOSS license Apache 2.0 8

9 Kurento Architecture (I) Kurento Media Server (KMS) the nucleus of Kurento KMS is a middleware for media streams Receives the stream Process the stream Issues the stream Media Source Send Receive Analyze Augment Enrich KMS 9 Transform Transcode Record Process Replicate Media Sink

10 Kurento Architecture (II) The API for accecisng KMS Applications define the processing of streams getting through KMS Java JavaScript Media API Media API Media Source Send Receive Analyze Augment Enrich KMS 10 REST API Transform Transcode Record Process Replicate Media Sink

11 Kurento API (I) Easy abstractions Media Element Media pipeline Provides a specific media Chain of media elements functionality implementing the desired media logic (topology). Send/receive media Process media Transform media The Media API provides the capability of creating media pipelines by joining media elements of the toolbox Exchange media through Sources Sinks Basic types Endpoints Filters 11

12 Kurento API (II) In one word... connect sourceelement.connect(sinkelement) 12

13 Kurento API (III) A lego game 13

14 WebRTC (I) Importance of WebRTC Before WebRTC Developing the client side After WebRTC Begin Next natural step End Developing the infrastructure side Begin End Unified APIs Standards FOSS Multiplatform Unified APIs Standards FOSS Multiplatform Begin End 14 Unified APIs Standards FOSS Multiplatform

15 WebRTC (II) Real-Time Comunication basics The signaling plane I wan to call you, do you accept? Yes, do it in this way Capture Encode Cipher Transport Network Transport The media plane 15 Decipher Decode Render

16 WebRTC (III) Architecture and Components RTCPeerConnection SDP SRTP DTLS ICE Security SRTP NAT traversal DTLS Network (IP) ICE (Interactive Connectivity Establishment) Media encrypted SDP (Session Description Protocol) Transport and media capabilities negotiation

17 Kurento & WebRTC (I) APIs (basics) RTCPeerConnection WebRtcEndpoint createoffer() generateoffer() createanswer() processoffer() setlocaldescription() processanswer() setremotedescription() addicecandidate() addicecandidate() onicecandidate onicecandidate 17

18 Kurento & WebRTC (II) Signaling example Client Side Signaling plane Application Server Kurento Media Server (KMS) pc.createoffer() sdpoffer pc.setlocaldesc (sdpoffer) sdpoffer WebRtcEp. processoffer(sdpoffer); pc.setremotedesc (sdpanswer); Media plane sdpanswer sdpanswer Media exchange between client and server 18

19 Enjoying Kurento (I) WebRTC loopback (topology) 19

20 Enjoying Kurento (II) WebRTC loopback (code) /* 1. Create pipeline and WebRtcEndpoint */ KurentoClient kurento = KurentoClient.create(); MediaPipeline pipeline = kurento.createmediapipeline(); WebRtcEndpoint webrtcendpoint = new WebRtcEndpoint.Builder(pipeline).build(); /* 2. Media topology: loopback */ webrtcendpoint.connect(webrtcendpoint); /* 3. SDP negotiation */ String sdpoffer = jsonmessage.get("sdpoffer").getasstring(); String sdpanswer = webrtcendpoint.processoffer(sdpoffer); /* 4. Create response */ JsonObject response = new JsonObject(); response.addproperty("sdpanswer", sdpanswer); /* 5. Send response to the client */ session.sendmessage(new TextMessage(response.toString())); 20

21 Enjoying Kurento (III) WebRTC loopback + recording (topology) 21

22 Enjoying Kurento (IV) WebRTC loopback + recording (code) Could you think about how would be the pseudo-code? 22

23 Enjoying Kurento (IV) WebRTC loopback + recording (code) /* 1. Create pipeline, WebRtcEndpoint and RecorderEndpoint */ KurentoClient kurento = KurentoClient.create(); MediaPipeline pipeline = kurento.createmediapipeline(); WebRtcEndpoint webrtcendpoint = new WebRtcEndpoint.Builder(pipeline).build(); RecorderEndpoint recorder = new RecorderEndpoint.Builder(pipeline, RECORDER_FILE_PATH).build(); /* 2. Media topology: loopback + recording */ webrtcendpoint.connect(webrtcendpoint); webrtcendpoint.connect(recorder); /* 3. Start recording */ recorder.record(); /* 4. SDP negotiation */ String sdpoffer = jsonmessage.get("sdpoffer").getasstring(); String sdpanswer = webrtcendpoint.processoffer(sdpoffer); /* 5. Create response */ JsonObject response = new JsonObject(); response.addproperty("sdpanswer", sdpanswer); /* 6. Send response to the client*/ session.sendmessage(new TextMessage(response.toString())); 23

24 Enjoying Kurento (V) WebRTC playing (code) /* 1. Create pipeline, WebRtcEndpoint and PlayerEndpoint */ KurentoClient kurento = KurentoClient.create(); MediaPipeline pipeline = kurento.createmediapipeline(); WebRtcEndpoint webrtcendpoint = new WebRtcEndpoint.Builder(pipeline).build(); PlayerEndpoint player = new PlayerrEndpoint.Builder(pipeline, RECORDER_FILE_PATH).build(); /* 2. Media topology: player webrtcep */ player.connect(webrtcendpoint); /* 3. SDP negotiation */ String sdpoffer = jsonmessage.get("sdpoffer").getasstring(); String sdpanswer = webrtcendpoint.processoffer(sdpoffer); /* 4. Create response */ JsonObject response = new JsonObject(); response.addproperty("sdpanswer", sdpanswer); /* 5. Send response to the client */ session.sendmessage(new TextMessage(response.toString())); /* 6. Start playing */ player.play(); 24

25 Enjoying Kurento (VI) WebRTC playing (topology) Could you think about how would be the topology? 25

26 Enjoying Kurento (VI) WebRTC playing (topology) 26

27 Enjoying Kurento (VII) WebRTC magic mirror (result) 27

28 Enjoying Kurento (VIII) WebRTC magic mirror (topology) Could you think about how would be the topology? 28

29 Enjoying Kurento (VIII) WebRTC magic mirror (topology) 29

30 Enjoying Kurento (IX) WebRTC magic mirror (code) Could you think about how would be the pseudo-code? 30

31 Enjoying Kurento (IX) WebRTC magic mirror (code) /* 1. Create pipeline, WebRtcEndpoint and PlayerEndpoint */ KurentoClient kurento = KurentoClient.create(); MediaPipeline pipeline = kurento.createmediapipeline(); WebRtcEndpoint webrtcendpoint = new WebRtcEndpoint.Builder(pipeline).build(); FaceOverlayFilter faceoverlayfilter = new FaceOverlayFilter.Builder(pipeline).build(); /* 2. Media topology: webrtcep filter webrtcep */ webrtcendpoint.connect(faceoverlayfilter); faceoverlayfilter.connect(webrtcendpoint); /* 3. SDP negotiation */ String sdpoffer = jsonmessage.get("sdpoffer").getasstring(); String sdpanswer = webrtcendpoint.processoffer(sdpoffer); /* 4. Create response */ JsonObject response = new JsonObject(); response.addproperty("sdpanswer", sdpanswer); /* 5. Send response to the client */ session.sendmessage(new TextMessage(response.toString())); 31

32 Enjoying Kurento (X) WebRTC advance videocall (result) 32

33 Enjoying Kurento (XI) WebRTC advanced videocall (topology) Could you think about how will be the topology? 33

34 Enjoying Kurento (XI) WebRTC magic mirror (topology) 34

35 Thank You Miguel París