POI360 Panoramic Mobile Video Telephony over LTE Cellular Networks Xiufeng Xie University of Michigan-Ann Arbor Xinyu Zhang University of California San Diego CoNEXT 2017
Background: 360 Video for VR 360 camera Sphere view Panoramic frame time 360 video for VR 30FPS
360 Video + Video Telephony = Interactive VR! Mobility Coverage
Challenges & Solution Spaces
Huge VR Traffic Load Calls for Compression 360 frame High VR stream bitrate: 10~20Mbps for 4K MP4 format Exceed LTE UL (5Mbps)/DL (12Mbps) bandwidth Compression based on region of interest (ROI) Human eye can only see part of 360 Compress unseen parts Quality Region-of-Interest (ROI) Spatial position
Challenge 1: Compression Fails over LTE Update ROI knowledge t Compressed frame Low quality High Low quality High Low quality ROI VR stream compressed with new ROI User-perceived VR quality always fluctuates over LTE ROI change ROI quality recover Lower ROI quality for one RTT Does not matter if RTT < VR frame interval (e.g., 33ms for 30fps) Typical wireline network LTE has unstable RTT (5~500ms) depending on traffic & channel
ROI Prediction? Predict the ROI by reviewer s motion? Oculus measurements [1]: Avg. head angular speed: 60 Τs Avg. head angular acceleration: 500 Τs 2 Head can stop rotation within 120ms Typical end-to-end LTE video latency can be more than 500ms Prediction: 120ms Need: 500ms ROI prediction does not work on LTE networks! [1] S.M.LaValle, A.Yershova, M.Katsev, and M.Antonov, Head Tracking for The Oculus Rift, in Robotics and Automation (ICRA), 2014 IEEE International Conference on, 2014.
Video quality Solution: Adaptive Compression ROI center Many ways to redistribute the quality Aggressive Sharp quality drop Spatial position Adaptive compression Conservative Responsive ROI update Aggressive Maximize the user-perceived quality Irresponsive ROI update Conservative Guarantee the stability of VR quality Smooth quality drop
Challenge 2: Irresponsive Rate Control Insufficient VR rate control responsiveness VR users: sensitive to video freezes in immersive environment LTE network: highly dynamic bandwidth Conventional video rate control Measure network-layer statistics Network Request suitable rate Sluggish loop: large RTT over LTE
Solution: Cellular Link-Informed Adaptation Cellular link info as congestion indicator LTE uplink: typical bottleneck for mobile VR telephony Diagnostic interface: status of UL firmware buffer VR Uplink stream congestion LTE control uplink based on UL buffer status Network End-to-end congestion control Shortcut: shorter adaptation path better responsiveness
Challenge 3: UL Bandwidth Underutilization LTE UL firmware buffer Video data UL throughput LTE uplink resource scheduling: UL throughput depends on its own buffer level Before UL congestion, higher buffer level higher uplink rate Existing rate control: unaware of this unique feature Buffer left empty (0 throughput) for 40% of time! UL throughput << available bandwidth
Solution: Adapt to UL Buffer Level Learn relation between UL throughput & buffer level Push firmware buffer level to the sweet region Sweet region: maximize throughput without congestion Buffer level too high: slow down traffic to avoid congestion Buffer level too low: speed up traffic to exploit bandwidth
POI360 System Design
Design Overview Viewer 360 Cam Adaptive Spatial Compression ROI Compressed VR stream Buffer Aware Rate Control Buffer level Sender RTP traffic Firmware Buffer Cellular uplink
Adaptive Spatial Compression Adapt compression mode Balance ROI quality and stability of ROI quality Video quality Aggressive Conservative Design: Switch mode following ROI update responsiveness Responsiveness metric: T3-T1 (duration of lower ROI quality) Spatial position T1: ROI change T3: ROI quality recovered T2: sender updates ROI knowledge
Buffer Aware Rate Control Compressed frame Rate Control PHY buffer level Cross-layer design Learn buffer s sweet region PHY buffer level too high reduce RTP & video bitrate PHY buffer level too low increase RTP & video bitrate PHY bitrate H.264 Encoding Video bitrate RTP bitrate Packet Pacer UL Firmware Buffer Application layer Transport layer Physical layer
Implementation Live stream 360 video VR player QXDM Diag. interface Client s ROI LTE phone
Evaluation
Micro-benchmark Evaluation Validate VR compression design Benchmark algorithm: CMU--Conduit (crop & send ROI) Facebook--Pyramid encoding Video-frame-level delay ROI quality (PSNR) Reduce delay by 15% ROI quality stability 11~13dB of improvement Reduce variation by 5X
Micro-benchmark Evaluation Validate our UL buffer-based rate control design Compare with Google Congestion Control (GCC, default rate control of Google Hangouts & Facebook Messenger) Our rate control FBCC keeps UL buffer level in the sweet region (green) for most of the time Low usage High usage Overuse (saturation)
System-Level Test Test POI360 system under various network conditions Different LTE background traffic load Different physical channel quality Different mobility level Performance metrics Smoothness Video freezing ratio Quality Frame-level PSNR Mean Opinion Score(MOS)
Different Background Traffic Load Light LTE background traffic load (early morning) 1% video freeze Heavy LTE background traffic load (noon) 4% video freeze & 2dB PSNR drop Majority of the frames have either excellent or good quality PSNR & Video freezing ratio MOS
Different Physical Channel Quality Test at places with different channel quality Weak (-115dB RSS); Moderate (-82dB RSS); Strong (-73dB RSS) Better channel: higher PSNR & MOS, less video freezes Even the weak channel achieves <3% video freezes PSNR & Video freezing ratio MOS
Different Mobility Level Test under 3 different mobility levels Slow (15mph); urban driving (30mph); highway (50mph) PSNR & MOS drop with higher mobility. But still have good or excellent quality even under 50mph mobility More freezes with high mobility: 1% for slow driving, 7% for urban driving. Comparable to legacy non-360 LTE video chat PSNR & Video freezing ratio MOS
POI360 Summary Unique challenges when 360 VR video meets LTE Long RTT of LTE breaks spatial VR compression Heavy VR traffic load Low cellular bandwidth utilization POI360: the first adaptive 360 VR compression Adapt compression strategy based on network condition Achieve balance between traffic load & smoothness Leverage cellular statistics to enable responsive rate control Other works in cellular network-informed mobile applications * pistream: Physical Layer Informed Adaptive Video Streaming Over LTE, Xiufeng Xie, Xinyu Zhang, Swarun Kumar, Li Erran Li, ACM MobiCom 15 * Accelerating Mobile Web Loading Using Cellular Link Information, Xiufeng Xie, Xinyu Zhang, Shilin Zhu, ACM MobiSys 17
Thank you! Q & A