Application and Desktop Sharing. Omer Boyaci November 1, 2007

Transcription

1 Application and Desktop Sharing Omer Boyaci November 1, 2007

2 Overview Introduction Demo Architecture Challenges Features Conclusion

3 Application Sharing Models Application specific + Efficient - Participants need application - Application has to be modified Generic - Inefficient (sometimes) + Participants don't need application + All applications are supported

4 Application Sharing Models Application specific + Efficient - Participants need application - Application has to be modified Generic - Inefficient (sometimes) + Participants don't need application + All applications are supported

5 Overview Introduction Demo Architecture Challenges Features Conclusion

6 Application Sharing Sharing an application with multiple users There is only one copy of the application Participants do not need application itself Briefly, participants receive screen updates send keyboard and mouse events Desktop sharing is also supported.

7 Screenshot

8 Screenshot (2)

9 Screenshot (Overlapped Windows)

10 Screenshot (Multicast App. Tool)

11 Screenshot (Ultra VNC)

12 Screenshot (App & Desk Sharing) 1 2 4

13 Supported Platforms/OS Windows *nix Mac OS X * is Java based. Server * + + +

14 Overview Introduction Demo Architecture Challenges Features Conclusion

15 Demo Demo

16 Overview Introduction Demo Architecture Challenges Features Conclusion

17 System Architecture /Server Software Architecture

18 System Architecture /Server Software Architecture Screen Updates

19 System Architecture /Server Software Architecture Keyboard Mouse Events

20 (Viewer) Architecture can Connect to server Wait for incoming connections supports TCP UDP (+Multicast)

21 (Viewer) Architecture receives these commands Open new window Window size changed Pixel update Close window sends BFCP (Binary Floor Control Protocol) commands Keyboard and mouse events

22 Windows Server Architecture Server Main Thread User space 0 Kernel space X 300,500 Y 200,600 Window coordinates Mirror Driver

23 Windows Server Architecture Server Main Thread User space Kernel space X 320,340 Y 100,200 X 320,340 Y 100,200 X 320,340 Y 100,200 X 320,340 Y 100,200 2 X 300,500 Y 200,600 Window coordinates Mirror Driver

24 Windows Server Architecture Server Main Thread User space Kernel space X 320,340 Y 100,200 X 320,340 Y 100,200 X 320,340 Y 100,200 2 X 300,500 Y 200,600 Window coordinates Mirror Driver

25 Windows Server Architecture X 3 2 0,3 4 0 X 3 2 0,3 4 0 X 3 2 0,3 4 0 Y 4 0 0,5 0 0 Y 4 0 0,5 0 0 Y 4 0 0,5 0 0 t im e s t a m p t i m e s t a m p t im e s t a m p P NG PN G T h r e a d -1 T h r e a d -2 Server Main Thread T h r e a d -3 User space K e rn e l s p a c e X 320,340 Y 400,500 X 4 2 0,4 4 0 Y 2 0 0,6 0 0 X 320,340 Y 450,550 2 W i n d o w coordinates

26 Overview Introduction Demo Architecture Challenges Features Conclusion

27 Challenges Different client bandwidths/speeds Late Joiner The effects of packet loss Reliable multicast

28 Multimedia Support (Movies)

29 Multimedia Support (Movies) Our system uses to compress and transmit the region updates is lossless and effective for computer generated images but ineffective for real world captures like pictures or movies JPG is more suitable for photographic images However, JPG is lossy and not effective for computer generated images (text, line, shapes,...) Our system should use both

30 Multimedia Support (Movies) Composite image comparing JPEG and : notice artifacts in JPEG versus solid background.

31 Multimedia Support ( vs JPG) Image Size Size x 360x K Size 4x 720x K JPG :4 1:20 Size x 1:5 1:30 Size 4x

32 Multimedia Support ( vs JPG) FPS KB/s JPG 12.5 JPG Size x Size 4x Size x Ethernet (60Mb/s) Size 4x

33 Multimedia Support ( vs JPG) KB/s FPS JPG 12.5 JPG Size x Size 4x 0 Size x Wireless (4Mb/s) Size 4x

34 /JPG Detection Algorithm -1,0,1 Region> 40,000px? coordinates Size counter YES Time Stamp New Region? YES Create a record & Start Checking NO Continue Checking NO Detected? YES Use Detected Format Region record

35 Sharing a Movie (Media Player)

36 Sharing a Movie File Capturing from the Frame Buffer is expensive. Instead Transcode the movie on the fly to JPGs FFmpeg (subproject of mplayer) FFmpeg is a complete solution to record, convert and stream audio and video. It includes libavcodec, the leading audio/video codec library. FFmpeg is developed under Linux, but it can compiled under most operating systems, including Windows. FFmpeg-Java (subproject of FMJ) Java wrapper (JNA: Java Native Access)

37 Sharing a Movie (Our Method)

38 Sharing a Movie File Movie File (wmv,mpg,divx) decodenextframe() encodeframetojpg() FFmpeg-Java JPG images Java Streaming Server Java UDP/Multicast Java

39 Demo Demo Multimedia Support

40 Challenges Different client bandwidths/speeds Late Joiner The effects of packet loss Reliable multicast

41 Different Bandwidths/Speeds S e r v e r M a in T h r e a d 3Mbps Thread -3 X 3 2 0,3 4 0 X 3 2 0,3 4 0 Y 4 0 0,5 0 0 Y 4 0 0,5 0 0 ti m e s t a m p t i m e s t a m p PN G Thread -2 X 3 2 0,3 4 0 Y 4 0 0,5 0 0 t im e s t a m p 2Mbps X 3 2 0,3 4 0 X 3 2 0,3 4 0 X 3 2 0,3 4 0 Y 4 0 0,5 0 0 Y 4 0 0,5 0 0 Y 4 0 0,5 0 0 t im e s t a m p t i m e s t a m p t im e s t a m p PN G P NG PN G C lie n t Thread -1 1Mbps

42 Different Bandwidths/Speeds Possible Solutions Slowest one Average speed Fastest one

43 Different Bandwidths/Speeds Possible Solutions Slowest one Problem: Penalize everybody except the slowest Average speed Fastest one

44 Different Bandwidths/Speeds Possible Solutions Slowest one Problem: Penalize everybody except the slowest Average speed Possible solution Fastest one

45 Different Bandwidths/Speeds Possible Solutions Slowest one Problem: Penalize everybody except the slowest Average speed Possible solution (Can we do better?) Fastest one

46 Different Bandwidths/Speeds Possible Solutions Slowest one Problem: Penalize everybody except the slowest Average speed Possible solution (Can we do better?) Fastest one The best solution bandwidths are fully utilized

47 Different Bandwidths/Speeds Thread Main Server 3Mbps Thread -3 X 3 2 0,3 4 0 X 3 2 0,3 4 0 Y 4 0 0,5 0 0 Y 4 0 0,5 0 0 ti m e s t a m p t i m e s t a m p PN G Thread -2 X 3 2 0,3 4 0 Y 4 0 0,5 0 0 t im e s t a m p PN G 2Mbps X 3 2 0,3 4 0 X 3 2 0,3 4 0 X 3 2 0,3 4 0 Y 4 0 0,5 0 0 Y 4 0 0,5 0 0 Y 4 0 0,5 0 0 t im e s t a m p t i m e s t a m p t im e s t a m p PN G P NG PN G Thread -1 1Mbps

48 Different Bandwidths/Speeds Server Main Thread 2Mbps Thread -2 3Mbps Thread -3 X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 timestamp timestamp timestamp timestamp timestamp timestamp Thread -1 1Mbps

49 Challenges Different client bandwidths/speeds Late Joiner The effects of packet loss Reliable multicast

50 Late Joiner Force server to generate full screen update

51 Late Joiner Force server to generate full screen update Problems Misbehaving clients can degrade performance If Join/Leave rate is high, too much burden on server Solution Generate full screen updates if really necessary Otherwise start the new client from last full screen update

52 Different Bandwidths/Speeds Thread -1 X 320,340 X 320,340 Y 400,500 Y 400,500 timestamp timestamp 1Mbps Server Main Thread 2Mbps Thread -2 3Mbps Thread -3 8Mbps Thread -4

53 Different Bandwidths/Speeds Thread -1 X 320,340 X 320,340 Y 400,500 Y 400,500 timestamp timestamp 1Mbps Server Main Thread 2Mbps Thread -2 FR I 3Mbps Thread -3 8Mbps Thread -4

54 Different Bandwidths/Speeds Server Main Thread 2Mbps Thread -2 3Mbps Thread -3 8Mbps Thread -4 X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 timestamp timestamp timestamp timestamp timestamp timestamp timestamp Thread -1 1Mbps

55 Different Bandwidths/Speeds Server Main Thread 2Mbps Thread -2 3Mbps Thread -3 8Mbps Thread -4 X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 timestamp timestamp timestamp timestamp timestamp timestamp timestamp Thread -1 1Mbps

56 Different Bandwidths/Speeds Server Main Thread 2Mbps Thread -2 3Mbps Thread -3 8Mbps Thread -4 X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 timestamp timestamp timestamp timestamp timestamp timestamp timestamp timestamp Thread -1 1Mbps

57 Different Bandwidths/Speeds Server Main Thread 2Mbps Thread -2 3Mbps Thread -3 8Mbps Thread -4 X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 timestamp timestamp timestamp timestamp timestamp timestamp timestamp timestamp Thread -1 1Mbps

58 Different Bandwidths/Speeds Server Main Thread 3Mbps Thread -3 8Mbps Thread -4 2Mbps Thread -5 X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 timestamp timestamp timestamp timestamp timestamp timestamp timestamp timestamp Thread -2 2Mbps Thread -1 1Mbps

59 Different Bandwidths/Speeds Server Main Thread 3Mbps Thread -3 8Mbps Thread -4 2Mbps Thread -5 X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 X 320,340 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 Y 400,500 timestamp timestamp timestamp timestamp timestamp timestamp timestamp timestamp Thread -2 2Mbps Thread -1 1Mbps

60 Challenges Different client bandwidths/speeds Late Joiner The effects of packet loss Reliable multicast

61 The effects of Packet Loss This problem applies to Multicast UDP The images can be large Regular desktop can be ~900KB ~600 Ethernet packets One packet loss wastes all image

62 The effects of Packet Loss Solution Small images Around ~1500 bytes Consist of a few scanlines Disadvantages Increased CPU usage (client&server) Lower compression ratio (%20 lower) Advantages One packet loss = no update for a few scanlines

63 The effects of Packet Loss

64 Challenges Different client bandwidths/speeds Late Joiner The effects of packet loss Reliable multicast

65 Reliable Multicast RTP Library stores last N rtp packets s send NACK for lost packets RTP Library resend the requested packets

66 The effects of Packet Loss

67 Overview Introduction Demo Architecture Challenges Features Conclusion

68 Recording s can record the whole/part session Anybody can play these files locally These files can be streamed to receivers via streaming server Streaming server supports multiple receivers Also late joiners

69 Listening waits for incoming connections It can display windows from multiple user Can be used for RGB cable replacement

70 Overview Introduction Demo Architecture Challenges Features Conclusion

71 Conclusion Application sharing allows users to share a single application with multiple participants. Participants don't need the application. It is not specific to a single application. Extra features like recording is added.

72 Thanks Questions?