RTP Payload format for Application and Desktop Sharing

Transcription

1 RTP Payload format for Application and Desktop Sharing Omer Boyaci & Henning Schulzrinne November 18,

2 Application Sharing Sharing an application with multiple users There is only one copy of the application Participants do not need application itself Briefly, participants receive screen updates send keyboard and mouse events 2

3 Terminology Application Host (AH) Participant Remoting protocol Human Interface Protocol (HIP) Different modes of remote access Remote Desktop Connection (Full or partial)screen Sharing Application Sharing 3

4 Screenshot 4

5 Screenshot (2) 5

6 Architecture 6 Remoting protocol

7 Architecture 7 Human Interface Protocol (HIP)

8 Architecture Binary Floor Control Protocol (BFCP) manages the ownership of AH side human interface devices. 8

9 Multimedia Support (Movies) Composite image comparing JPEG and : notice artifacts in JPEG versus solid background. draft-boyaci-avt-png-00 9

10 Comparison of Sharing Systems in terms of web page visiting performance 10

11 Multimedia Support (Movies) 11

12 Comparison of Sharing Systems in terms of multimedia performance Unlimited Bandwidth BASS-Theora BASS-JPEG 12

13 Comparison of Sharing Systems in terms of multimedia performance 3Mb/s Bandwidth BASS-Theora BASS-JPEG 13

14 Remoting Messages Application host (AH) to participants WindowStateInfo RegionUpdate MoveRectangle MousePointerInfo Participants to AH (NACK messages) PLI (Picture Loss Indication) NACK request 14

15 HIP Messages MousePressed MouseReleased MouseMoved MouseWheelMoved KeyPressed KeyReleased KeyTyped 15

16 VNC problems -pull based No multicast support Same encoding for all updates CPU usage increases for each new user 16

17 TeleTeachingTool (VNC based) Problems Modified VNC protocol and clients (TTT server is pushing updates) No compression (only hextile encoding) because packets can get lost 17

18 Open Issues Transport Protocol RTP MSRP Custom Made 18

19 Open Issues How to do retransmission requests for UDP clients Current proposal NACK-based solution NACK suppression to prevent floods RTP-library level retransmissions 19

20 Current Internet Drafts draft-boyaci-avt-app-sharing-00 draft-boyaci-avt-png-00 20

21 Backup Slides 21

22 Protocol Message Structures 22

23 Common remoting/hip Header 23

24 A Window Record 24

25 A Region Update Message 25

26 The MoveRectangle Message 26

27 The MousePressed Message 27

28 The KeyPressed Message 28

29 The KeyTyped Message 29

30 Comparison of Sharing Systems 30

31 Screenshot (Overlapped Windows)

32 Screenshot (Multicast App. Tool)

33 Screenshot (Ultra VNC)

34 Screenshot (BASS)

35 Comparison of Sharing Systems 35

36 VNC problems -pull based No multicast support Same encoding for all updates CPU usage increases for each new user 36

37 TeleTeachingTool (VNC based) 37

38 Information about BASS 38

39 Supported Platforms/OS Windows *nix Mac OS X Server * * is Java based. 39

40 (Viewer) Architecture can Connect to server Wait for incoming connections supports TCP UDP (+Multicast) 40

41 BASS Windows Server Architecture Server Main Thread User space 0 Kernel space X 300,500 Y 200,600 Window coordinates Mirror Driver 41

42 BASS Windows Server Architecture Server Main Thread User space Kernel space X 320,340 Y 100,200 X 320,340 Y 100,200 X 320,340 Y 100,200 X 320,340 Y 100,200 2 X 300,500 Y 200,600 Window coordinates Mirror Driver 42

43 BASS Windows Server Architecture Server Main Thread User space Kernel space X 320,340 Y 100,200 X 320,340 Y 100,200 X 320,340 Y 100,200 2 X 300,500 Y 200,600 Window coordinates Mirror Driver 43

44 BASS Windows Server Architecture T h r e a d- 2 X 3 2 0,3 4 0 X 3 2 0, X 3 2 0,3 4 0 Y 4 0 0,5 0 0 Y 4 0 0, Y 4 0 0,5 0 0 t im e s t a m p t i m e s t a m p t im e s t a m p PN G P NG PN G T h r e a d- 1 Server Main Thread T h r e a d- 3 User space K e rn e l s p a c e X 3 2 0, X 4 2 0, X 3 2 0, Y 4 0 0, Y 2 0 0, Y 4 5 0, W indow coordinates 44

45 Challenges Different client bandwidths/speeds Late Joiner The effects of packet loss Reliable multicast 45

46 Multimedia Support (Movies) 46

47 Multimedia Support (Movies) Our system uses to compress and transmit the region updates is lossless and effective for computer generated images but ineffective for real world captures like pictures or movies JPG is more suitable for photographic images However, JPG is lossy and not effective for computer generated images (text, line, shapes,...) Our system should use both 47

48 Multimedia Support (Movies) Composite image comparing JPEG and : notice artifacts in JPEG versus solid background. 48

49 Multimedia Support ( vs JPG) Image Size Size x 360x K JPG 80 Size 4x 720x K :4 1:20 1:5 1:30 0 Size x Size 4x 49

50 Multimedia Support ( vs JPG) KB/s FPS JPG 600 JPG Ethernet (60Mb/s) 0 Schulzrinne, draft-boyaci-avt-app-sharing-00 Size x Size 4x Size xboyaci & Size 4x 50

51 Multimedia Support ( vs JPG) KB/s FPS JPG 12.5 JPG Wireless (4Mb/s) 0 Size x Size 4x Size x Size 4x 51

52 /JPG Detection Algorithm Region> 40,000px? YES -1,0,1 coordinates Size counter Time Stamp New YES Create a record Region record & Region Start Checking? NO Continue NO Detected YES Use Detected CheckingBoyaci & Schulzrinne, Format? draft-boyaci-avt-app-sharing-00 52

53 Sharing a Movie (Media Player) 53

54 Sharing a Movie File Capturing from the Frame Buffer is expensive. Instead Transcode the movie to Theora beforehand Then stream the theora directly to participants Java client supports theora playback Negligible CPU usage during playback on the server 54

55 Sharing a Movie (Our Method) 55

56 Sharing a Movie File Movie File (wmv,mpg,divx) FFMpeg2Theora Theora Movie Java Streaming Server Java UDP/Multicast Java 56

57 Challenges Different client bandwidths/speeds Late Joiner The effects of packet loss Reliable multicast 57

58 Different Bandwidths/Speeds X 3 2 0,3 4 0 Y 4 0 0,5 0 0 t im e s t a m p PN G T h r e a -d 3 X 3 20,3 4 0 Y 4 00,5 0 0 tim e s t a m p PN G 3M b p s X 3 2 0,3 4 0 Y 4 0 0,5 0 0 ti m e s t a m p T h r e a d S e r v e r M a in T h r e a d X 3 2 0,3 4 0 Y 4 0 0,5 0 0 t im e s t a m p PN G 2M b p s X 3 2 0,3 4 0 X 3 2 0, Y 4 0 0,5 0 0 Y 4 0 0, t i m e s t a m p t im e s t a m p P NG PN G C lie n t T h r e a d- 1 1M b p s

59 Different Bandwidths/Speeds Possible Solutions Slowest one Average speed Fastest one 59

60 Different Bandwidths/Speeds Possible Solutions Slowest one Problem: Penalize everybody except the slowest Average speed Fastest one 60

61 Different Bandwidths/Speeds Possible Solutions Slowest one Problem: Penalize everybody except the slowest Average speed Possible solution Fastest one 61

62 Different Bandwidths/Speeds Possible Solutions Slowest one Problem: Penalize everybody except the slowest Average speed Possible solution (Can we do better?) Fastest one 62

63 Different Bandwidths/Speeds Possible Solutions Slowest one Problem: Penalize everybody except the slowest Average speed Possible solution (Can we do better?) Fastest one The best solution bandwidths are fully utilized 63

64 Different Bandwidths/Speeds X 3 2 0,3 4 0 Y 4 0 0,5 0 0 t im e s t a m p PN G T h r e a -d 3 X 3 20,3 4 0 Y 4 00,5 0 0 tim e s t a m p PN G 3M b p s X 3 2 0,3 4 0 Y 4 0 0,5 0 0 ti m e s t a m p T h r e a d Server Main Thread X 3 2 0,3 4 0 Y 4 0 0,5 0 0 t im e s t a m p PN G 2M b p s X 3 2 0,3 4 0 X 3 2 0, Y 4 0 0,5 0 0 Y 4 0 0, t i m e s t a m p t im e s t a m p P NG PN G T h r e a d- 1 1M b p s

65 Different Bandwidths/Speeds 2Mbps Thread -2 3Mbps Thread Server Main Thread X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 timestamp timestamp timestamp timestamp timestamp timestamp Thread -1 1Mbps

66 Challenges Different client bandwidths/speeds Late Joiner The effects of packet loss Reliable multicast 66

67 Late Joiner Force server to generate full screen update 67

68 Late Joiner Force server to generate full screen update Problems Misbehaving clients can degrade performance If Join/Leave rate is high, too much burden on server Solution Generate full screen updates if really necessary Otherwise start the new client from last full screen update 68

69 Different Bandwidths/Speeds Thread -1 X 320,340 X 320,340 Y 400,500 Y 400,500 timestamp timestamp 1Mbps Server Main Thread 2Mbps Thread -2 3Mbps Thread -3 8Mbps Thread -4 69

70 Different Bandwidths/Speeds Thread -1 X 320,340 X 320,340 Y 400,500 Y 400,500 timestamp timestamp 1Mbps Server Main Thread 2Mbps Thread -2 FR I 3Mbps Thread -3 8Mbps Thread -4 70

71 Different Bandwidths/Speeds 2Mbps Thread -2 3Mbps Thread -3 8Mbps Thread Server Main Thread X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 timestamp timestamp timestamp timestamp timestamp timestamp timestamp Thread -1 1Mbps

72 Different Bandwidths/Speeds 2Mbps Thread -2 3Mbps Thread -3 8Mbps Thread Server Main Thread X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 timestamp timestamp timestamp timestamp timestamp timestamp timestamp Thread -1 1Mbps

73 Different Bandwidths/Speeds 2Mbps Thread -2 3Mbps Thread -3 8Mbps Thread Server Main Thread X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 timestamp timestamp timestamp timestamp timestamp timestamp timestamp timestamp Thread -1 1Mbps

74 Different Bandwidths/Speeds 2Mbps Thread -2 3Mbps Thread -3 8Mbps Thread Server Main Thread X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 timestamp timestamp timestamp timestamp timestamp timestamp timestamp timestamp Thread -1 1Mbps

75 Different Bandwidths/Speeds 3Mbps Thread -3 8Mbps Thread -4 2Mbps Thread Server Main Thread X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 timestamp timestamp timestamp timestamp timestamp timestamp timestamp timestamp Thread -2 2Mbps Thread -1 1Mbps

76 Different Bandwidths/Speeds 3Mbps Thread -3 8Mbps Thread -4 2Mbps Thread Server Main Thread X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 timestamp timestamp timestamp timestamp timestamp timestamp timestamp timestamp Thread -2 2Mbps Thread -1 1Mbps

77 Challenges Different client bandwidths/speeds Late Joiner The effects of packet loss Reliable multicast 77

78 The effects of Packet Loss This problem applies to Multicast UDP The images can be large Regular desktop can be ~900KB ~600 Ethernet packets One packet loss wastes all image 78

79 The effects of Packet Loss Solution Small images Around ~1500 bytes Consist of a few scanlines Disadvantages Increased CPU usage (client&server) Lower compression ratio (%20 lower) Advantages One packet loss = no update for a few scanlines 79

80 The effects of Packet Loss 80

81 Challenges Different client bandwidths/speeds Late Joiner The effects of packet loss Reliable multicast 81

82 Reliable Multicast RTP Library stores last N rtp packets s send NACK for lost packets RTP Library resend the requested packets 82

83 The effects of Packet Loss 83

84 Overview Introduction Demo Architecture Challenges Features Conclusion 84

85 Recording s can record the whole/part session Anybody can play these files locally These files can be streamed to receivers via streaming server Streaming server supports multiple receivers Also late joiners 85

86 Listening waits for incoming connections It can display windows from multiple user Can be used for RGB cable replacement 86

87 Conclusion Application sharing allows users to share a single application with multiple participants. Participants don't need the application. It is not specific to a single application. Extra features like recording is added. 87