Media Streaming for Immersive, Collaborative Environments

Size: px

Start display at page:

Download "Media Streaming for Immersive, Collaborative Environments"

Rosaline Wood
5 years ago
Views:

Media Streaming for Immersive, Collaborative Environments Integrated Media Systems Center University of Southern California Los Angeles, CA 90089 NTU / EDB Singapore Visit, February 14, 2006

1 Media Streaming for Immersive, Collaborative Environments Integrated Media Systems Center University of Southern California Los Angeles, CA NTU / EDB Singapore Visit, February 14, Outline High-Definition Media Streaming HYDRA Architecture PWICE application Distributed Immersive Performance (DIP) Application Wireless HD Streaming P2P Media Streaming AudioPeer Stroke Rehabilitation PartyPeer 1

IMSC ERC Research Structure Media Streaming

a rich set of streaming media (haptic, audio,

archiving Need a scalable, general, unified

2 IMSC ERC Research Structure Media Streaming and Archiving Immersive environments produce a rich set of streaming media (haptic, audio, video) Data streams are valuable: session archiving Need a scalable, general, unified streaming media architecture for: multi-stream recording, multi-stream playback, and live streaming (low latency), synchronization. Unique media management platform HYDRA (High-performance Data Recording architecture) [ICEIS 2003] 2

HYDRA Building Blocks Acquisition Rendering Acquisition Rendering Server + Client

Storage Large scale systems can be built Business Application Domain: Remote Problem

3 HYDRA Building Blocks Acquisition Rendering Acquisition Rendering Server + Client Support for different hardware devices that generate or output digital media streams Storage Large scale systems can be built Business Application Domain: Remote Problem Solving Collaborative Site 1 E.g., Technicians of Korean Air, Incheon HD video rendering Collaborative Site 2 E.g., P&W Maintenance Help Desk, Hartford HD video rendering Camera Mics Low latency protocols Internet Camera Mics Immersive audio rendering PC Wireless PC Immersive audio rendering Mobile station E.g., borescope, video and audio transmissions Collaborative Site 3 E.g., Field Service Technician Storage and Retrieval 3

4 HD Live Streaming Approach HD video rendering HD video rendering Camera Mics Low latency protocols IMSC net IMSC net Camera Mics Audio rendering PC Powell Hall (PHE 320) HD resolution: 1280x720, 30 fps, 20 Mb/s Audio interleaving; FireWire & IP protocols Software rendering: MC and idct on GPU Challenges: Precise timing with low latency decoding time for I, P, and B frames varies Error resilience for transmission Independent source/destination clocks Multiple stream synchronization Powell Hall (PHE 106) PC Audio rendering End-to-end latency: ~350 ms Unique, flexible, integrated HYDRA stream management architecture Two-way HD Live Streaming JVC HD10U RTP/ UDP/IP HD-SDI Display FireWire VGA Linux RH 9 MPEG TS over FireWire Decoders: Software-based: use of modern graphics GPU (e.g., nvidia) Hardware-based: use of CineCast HD (Vela Research) 4

5 Two-way HD Live Streaming Live streaming between USC and Inha University in Incheon, Korea USC, October

Challenge: Latency Best case end-to-end latency: ~350 ms (310

camera and local MPEG-2 HD rendering (SW).

Entertainment 04] Live, high-resolution streaming over

at the Fall 2004 Internet2 meeting in Austin, TX Unique: 4 HD

6 Challenge: Latency Best case end-to-end latency: ~350 ms (310 ms) JVC-HD10U Camera With MPEG-2 codec Stream acquisition from camera and local MPEG-2 HD rendering (SW). HD Live Streaming: Accomplishments [ACM Computers in Entertainment 04] Live, high-resolution streaming over commodity IP networks Miró Quartet: Live & Virtual Gala event at the Fall 2004 Internet2 meeting in Austin, TX Unique: 4 HD video channels and 16 immersive audio channels! USC-Inha University, Korea; two-way live streaming, Fall

7 Internet2 Member Meeting (UTA) VIDEO: UT Austin DIP v.2 - Two-way baseline user studies (2004) HYDRA data base/ delay processor mic analysis/data MIDI HYDRA architecture by Zimmermann et al. MIDI analysis MIDI audio streaming by Papadopoulos & Sinha mic video cameras TECHNOLOGY MIDI keyboard MIDI MIDI keyboard audio mixer speakers 7

2 - Two-way baseline user studies (2004) A: first time players perform under delayed conditions B: player 1 and player 2 swap

8 Media Types: HD Video PCM Audio MIDI Example: Stream Archiving for DIP 10:23:56 Common Timestamps RTP/ UDP/IP Backend Server and Storage FireWire Frontend Video, Audio, MIDI archive Performers: The Tosheff Piano Duo DIP v.2 - Two-way baseline user studies (2004) A: first time players perform under delayed conditions B: player 1 and player 2 swap parts (symmetry test) Two players in same room facing each other (visual delay 0ms) Variable audio delay: 0, 10, 20, 30, 40, 50, 75, 100, 150ms VIDEO: Delays 8

Low Latency and Wireless Transmission for High-Definition Video Goal: High-definition streaming over wireless channels for asymmetric applications Challenges: Requires about 23 Mb/s,

Unique accomplishments: Very recent, successful rendering experiments (un-optimized for wireless) on Inspiron XPS Joint work with Prof.

9 Low Latency and Wireless Transmission for High-Definition Video Goal: High-definition streaming over wireless channels for asymmetric applications Challenges: Requires about 23 Mb/s, which pushes the limit of a/g Packet loss characteristics are very different Wireless channel usually shared with other traffic No full duplex! Unique accomplishments: Very recent, successful rendering experiments (un-optimized for wireless) on Inspiron XPS Joint work with Prof. Ahmed Helmy Recent Interactions Thales Avionics s In-Flight Entertainment (IFE) group; Irvine Multiple opportunities for next-generation IFE Improved system High performance Low weight Low power New services Personalization More videos Many opportunities Image Ed Carriazo 9

Mobile P2P Streaming ACTIVE: AudioPeer P2P streaming protocol [MMCN 2005] Observation: in large groups only a subset of

performance The end-to-end delay among active users is reduced by automatically clustering them near the root Education

Education Network Goals and Challenges: Enable discussions and Q&A for courses, labs, office hours Support participation of

10 Mobile P2P Streaming ACTIVE: AudioPeer P2P streaming protocol [MMCN 2005] Observation: in large groups only a subset of participants talk at the same time (are active) ACTIVE dynamically adapts the multicast tree to optimize the system performance The end-to-end delay among active users is reduced by automatically clustering them near the root Education Application Domain: Distance Education Network AudioPeer - Peer-to-peer streaming; in collaboration with USC Distance Education Network Goals and Challenges: Enable discussions and Q&A for courses, labs, office hours Support participation of students, TAs, LAs, and instructors Be scalable: 100+ groups (classes), students Be extensible, e.g., speech transcripts of sessions to allow text search 10

AudioPeer Group Communications Peer-to-peer architecture Adaptive multicast tree protocol called ACTIVE: distinguishes active from passive users Unique: Clustering active users for lower latency

distributed time measurements H ACTIVE Protocol Pipeline User Status Distinguish Active Users 25 R 20 10 A B C 30 D E 20 F 10 G Floor Control Allow Users to Speak [DMS 2004; MMCN 2005] Optimiza

Second Generation (2004) Web based, simple interface Integr

11 AudioPeer Group Communications Peer-to-peer architecture Adaptive multicast tree protocol called ACTIVE: distinguishes active from passive users Unique: Clustering active users for lower latency Quick and automatic failure recovery UPnP support for peers behind NAT firewalls Speex speech codec support (preliminary: software echo cancellation) Integrated NTP library for accurate, distributed time measurements H ACTIVE Protocol Pipeline User Status Distinguish Active Users 25 R A B C 30 D E 20 F 10 G Floor Control Allow Users to Speak [DMS 2004; MMCN 2005] Optimiza tion Cluster Active Users Credit Point System AudioPeer Accomplishments Participant Client Administration Interface Visualization of large groups Monitoring of network conditions, etc. Second Generation (2004) Web based, simple interface Integration with Distance Education Network For large, moderated groups: Allows floor control by moderator (e.g., TA) For small groups (e.g., projects discussions): Automatic floor control to allow complete interactivity Pilot classroom program use with CSci 585 and ISE 515 courses (2003/04/05) User studies Extension to mobile devices [Mobisys 05] 11

12 MStream Position-aware Mobile Music Streaming Integration of a PDA, wireless networking (802.11b) and GPS User location is acquired through GPS receiver Music (or other audio source) is streamed based on the location of the device Applications: outdoor museum guide, games [Mobisys 2005] Medical Application Domain: Tele-Rehabilitation System Tele-therapy can be economical and reduce the burden on patients and their families The tele-therapy system components Therapy program of games and exercises AudioPeer voice and text communication platform Local human-machine dialog system for intake and survey applications Online transcription and archiving of audio in mediated interactions with human therapist Real-time therapist viewing and monitoring of multiple patients as they perform their exercises Hand-over-hand guidance of the patient by the therapist Joint work with Profs. Margaret McLaughlin and Shri Narayanan 12

Social Community Tool: PartyPeer Level 1 Parent of Broadcaster Distributed virtual environment Users are represented as avatars Differently themed rooms ACTIVE

45 kbps 45 kbps...... 45 kbps... Reduced volume Broadcaster.

13 Social Community Tool: PartyPeer Level 1 Parent of Broadcaster Distributed virtual environment Users are represented as avatars Differently themed rooms ACTIVE protocol provides audio conferencing capabilities Proximity audio creates aural landscape Broadcast and interactive modes Level 2 Level 3 Level 4 Full volume Speaker 45 kbps 45 kbps kbps... Reduced volume Broadcaster links per node: 225 kbps Children of Broadcaster Full volume (broadcast mode) Full volume (interactive mode) Reduced volume (interactive mode) Note: - Not all arrows are shown. - Two arrows in the same direction may represent mixed audio. Social Community Tool: PartyPeer 13

14 Thank You! Questions? DMRL Lab Website IMSC Website 14

e-learning, Webinar, Collaboration & More Solution Presentation

e-learning, Webinar, Collaboration & More Solution Presentation Why emeetplus In a better and faster way. Easy to deploy and use Brings the spontaneity, naturalness and interactivity of in-person Learning