Adaptive 360 VR Video Streaming: Divide and Conquer!

Transcription

1 Adaptive 360 VR Video Streaming: Divide and Conquer! Mohammad Hosseini *, Viswanathan Swaminathan * University of Iinois at Urbana-Champaign (UIUC) Adobe Research, San Jose, USA Emai: shossen2@iinois.edu, vishy@adobe.com arxiv: v5 [cs.mm] 7 Nov 207 Abstract Whie traditiona mutimedia appications such as games and videos are sti popuar, there has been a significant interest in the recent years towards new 3D media such as 3D immersion and Virtua Reaity (VR) appications, especiay 360 VR videos. 360 VR video is an immersive spherica video where the user can ook around during payback. Unfortunatey, 360 VR videos are extremey bandwidth intensive, and therefore are difficut to stream at acceptabe quaity eves. In this paper, we propose an adaptive bandwidth-efficient 360 VR video streaming system using a divide and conquer approach. We propose a dynamic view-aware adaptation technique to tacke the huge bandwidth demands of 360 VR video streaming. We spatiay divide the videos into mutipe ties whie encoding and packaging, use MPEG-DASH SRD to describe the spatia reationship of ties in the 360-degree space, and prioritize the ties in the Fied of View (FoV). In order to describe such tied representations, we extend MPEG-DASH SRD to the 3D space of 360 VR videos. We spatiay partition the underying 3D mesh, and construct an efficient 3D geometry mesh caed hexaface sphere to optimay represent a tied 360 VR video in the 3D space. Our initia evauation resuts report up to 72% bandwidth savings on 360 VR video streaming with minor negative quaity impacts compared to the baseine scenario when no adaptations is appied. I. INTRODUCTION Advances in computing hardware and networking technoogies with support of high bandwidth have enabed the use of new 3D media such as 3D immersion and 360-degree VR video appications in the recent years. 360 VR videos are immersive spherica videos, mapped into a 3D geometry as shown in Figure, where the user can ook around during payback using a VR head-mounted dispay (HMD). This gives viewer a sense of depth in every direction. Despite the promising nature of 360 VR videos, existing 360 VR video appications are sti restricted to ower resoutions compared to their 2D counterparts. Unfortunatey, a major chaenge is how to efficienty transmit the buky 360 VR video streams to bandwidth-constrained devices such as wireess VR HMDs given their high bitrate requirements. Especiay with the 4K video resoution being widey viewed as a functiona minimum resoution for current HMDs, and 8K or higher being desired, these new media are extremey bandwidth intensive and difficut to stream at acceptabe quaity eves. Thus there must be a baance between the requirements of streaming and the avaiabe resources on the dispay devices. One of the chaenges to achieving this baance is that we need to meet this requirement without much negative impact on the user s viewing experience. Whie our work is motivated by the 360 VR video appications with 8K and 2K resoutions and the data rate issues that such rich mutimedia system have, a semantic ink between FoV, spatia partitioning, and stream prioritization has not been fuy deveoped yet for the purpose of bandwidth management and high performance 360 VR video streaming. Hence, we propose to utiize this semantic ink in our dynamic adaptations. In this paper, we propose FoV-aware streaming adaptations for efficient deivery of high-resoution 360 VR videos to bandwidth-imited HMDs. We spatiay divide the videos into mutipe ties whie encoding and packaging, use MPEG-DASH SRD to describe the spatia reationship of ties in the 360- degree space, and prioritize the ties in the viewport. We then extend our tiing process to the 360-degree VR environment to conquer the intense bandwidth Fig. : Visua structure of a 360 VR video requirements using viewport adaptation techniques. To achieve that, we spatiay partition the underying 3D mesh, and construct an efficient 3D geometry mesh caed hexaface sphere to optimay represent a tied 360 VR video in the 3D space. On the other hand, due to the absence of a fine-grained prioritized mechanism, most 360 VR streaming systems today hande a the portions of the spherica view as equay important, resuting in sub-optima resource usage. Our approach is to deiver higher bitrate content to regions where the user is currenty ooking and is most ikey to ook, and deivering ower quaity eve to the area outside of user s immediate viewport. Our initia evauation resuts using a rea-patform wireess HMD and mutipe 360 VR video benchmarks show that our adaptations significanty reduces the amount of bandwidth required to deiver a high quaity immersive experience, and increases the overa 360 VR video quaity at a given bandwidth. II. BACKGROUND AND RELATED WORK In this section we briefy present different concepts and categories of state-of-the-art reated to our proposed approach. A. Dynamic Adaptive Streaming One of the main approaches for bandwidth saving on bandwidth-intensive mutimedia appications is adaptive streaming. Dynamic Adaptive Streaming over HTTP (DASH) specificay, aso known as MPEG-DASH [], [2] is an ISO standard that enabes adaptive bitrate streaming whereby a cient chooses the video segment with the appropriate quaity (bit rate, resoution, etc.) based on the constrained resources such as bandwidth avaiabe at the cient. As a part of ISO/IEC :205, a new ammendment caed Spatia Reationship Description (SRD) has been added to MPEG-DASH standard, which aows to define spatia

2 Fig. 2: An exampe visua view of how a 90-degree spherica viewport is mapped on a raw 360 video (Star Wars) reationships between spatiay partitioned mutimedia content [3]. The SRD feature expores how to combine severa spatiay-reated videos in a tied combination, whie at the same time provides backward compatibiity with reguar definition of adaptation sets. It provides a unique approach to tied streaming in the perspective of the faciitated utra high-resoution video dispay, specificay in the context of immersive environments such as those seen in 360 VR videos. There have been some work exporing the features of MPEG- DASH SRD. Le Feuvre et a. in their work [4] expored spatia access of 2D video contents within the context of MPEG- DASH, and discussed how the tiing feature of SRD extension can enabe that. In another work [5], D Acunto et a. expored the use of MPEG-DASH SRD to partition a video into subparts to provide a zooming feature inside 2D video contents. In this work, we extend the semantics of MPEG-DASH SRD towards the context of 360 VR videos, and use that to partition the buky 360 VR videos into spatiay reated ties in the 3D space for the purpose of view-aware adaptation. Our aim is to expore prioritization of the ties by assigning highest resoution ony to those ties that are reevant to the user s FoV and ower resoution to ties outside of the user s FoV. B. Prioritized Mutimedia Streaming Generay, different parts of mutimedia can have different importance given various settings such as view, region, or the context. Hosseini et a. [6], [7] adopted prioritization techniques towards efficienty transmitting, rendering, and dispaying buky 3D contents to resource-imited devices given the importance of various 3D objects in the context. Simiary, in the context of 3D tee-immersive systems, the authors in [8] studied stream prioritization in regards to bandwidth savings. Their approach assigns higher quaity to parts within users viewport given the features of the human visua system. In this paper, we buid upon the concepts from these works to impement an adaptive prioritized view-aware streaming approach to reduce the bandwidth requirements of 360 VR video streaming. III. METHODOLOGY Simiar to the context of 3D graphics, the visua experience of 360 VR videos is aso based upon texturing. Textures are used to provide surface wrapping for a 3D object mesh, with 3D textures being a ogica extension of the traditiona 2D textures. 360 VR videos are created by mapping a raw 360 video as a 3D texture onto a 3D geometry mesh, often a sphere, with the user at the center of that geometry as shown in Figure. In this 360-degree environment however, a user is viewing ony a sma portion of the whoe raw 360-degree video at any given time. Therefore, a user s FoV is aso ony a sma portion equivaent to a specific confined region on the 3D object mesh which is spatiay reated to the corresponding portion of the raw content. For exampe, the Samsung Gear VR HMD offers a 96-degree FoV, meaning it can ony amost cover a quarter of a whoe 360-degree-wide content horizontay. Figure 2 iustrates this concept. The eft subfigure shows an exampe 90-degree viewport as projected on a spherica 3D geometry, whie the right subfigure shows how the mapping of the viewport corresponds to that of a given frame on a raw 360-degree video. One of the major chaenges in streaming 360-degree VR videos is the high bandwidth demands. To decrease the bandwidth requirements, our approach is to assign higher quaity to parts within a user s viewport, and use ower quaity to parts which are not within the immediate viewport of the user. This approach aso makes it possibe to stream ties inside the viewport at highest resoution, at or near the native resoution of the HMD, virtuay enabing a tota resoution of 8K and higher in the 360-degree VR environment. To achieve that, our approach consists of two parts. First, the raw 360-degree video is spatiay partitioned into mutipe ties. Using the features of MPEG-DASH SRD, a reference space is defined for each tie, corresponding to the rectanguar region encompassing the entire raw 360-degree video. Second, we partition the underying 3D geometry into mutipe segments, each representing a subset of the origina 3D mesh with a unique identifier. Finay, a mapping mechanism is defined for spatia positioning of the ties on the 3D space, so that each tie be textured on its corresponding 3D mesh segment. A. 3D Geometry Construction: Hexaface Sphere Most of the cameras avaiabe nowadays output 360 panoramic videos using a equirectanguar format. Whie these videos are mapped into a fat projection for storage, they are inherenty spherica. To achieve a spherica view as a common practice, a 3D sphere mesh is created surrounding the virtua camera inside the main virtua scene. Every individua photo-sphere, an image of a equirectanguar 360 video frame, is wrapped on the interna Fig. 3: A 3D sphere mesh. The highighted area represents a sice. surface of the sphere. The stereoscopic depth of 360 VR video requires two photo-spheres be stored and shown side-by-side representing the sma disparity of eft eye and right eye. The sphere is setup in such a way that it contains vertex ocations and texture coordinates to achieve equirectanguar mapping. The front faces of each of the rendered sphere poygons is cued to enabe their interna surface showing as opposed to externa showing. As a part of our geometry construction, we programmaticay create a 3D sphere mesh in code to be abe to contro and further modify the geometry, and set the norma vectors of a of the mesh segments to point inside, towards the center of the sphere, achieving interna showing. In order to create a 3D sphere mesh, we create an array of vertices by proceduray generating trianges for rendering. We approximate the sphere and provide a quaity parameter

3 Agorithm Our process to generate a 3D sphere mesh M: the number of stacks N: the number of sices m : 0 m M and n : 0 n N, cacuate and store a spatia point P (x, y, z) such that: P x Sin(π m/m) Cos(2π n/n) P y Sin(π m/m) Sin(2π n/n) P z Cos(π m/m) Draw the ine segments between the each vertex. Fig. 4: Visua overview of a generated hexaface sphere. to account for the trade-off between more smoothness and more trianges to render (representing highest quaity) and rendering speed (ower spherica smoothness and chunkier shape). Figure 3 iustrates a visua view of the structure of a 3D sphere mesh. Our quaity parameter is a combination of two major parameters which together contro the smoothness of the sphere: a) number of sices, which represent the number of vertica cuts. Let s assume each sice coides with a sphere s perimeter at a vertica degree θ which π/2 θ π/2; and b) number of stacks, which is the number of horizonta cuts determining the number of rows of rectanges. Let s assume each stack coides with a sphere s perimeter at a horizonta degree φ which π/2 φ π/2. Agorithm presents pseudo-code of our process to create 3D spherica mesh. Next, in order to generate 3D mesh segments, we partition the 3D sphere into mutipe different 3D meshes, in a two-step process. In the first step, we spit the sphere into 3 major parts: The top cap, which incudes meshes from the top point of the sphere (i.e. top poe where θ = +π/2 o ) to the top stack of the midde body (θ = +β o ), totaing (π/2 β) o, The midde body which is a ring of trianges stretching from the bottom stack of the top cap (θ = +β o ) to the top stack of the bottom cap (θ = β o ), totaing 2β o, The bottom cap, which incudes trianges ranging from the bottom stack of the midde body (θ = β o ) to the bottom point of the sphere (i.e. bottom poe where θ = π/2 o ), for π/2 β degrees. 360-degree videos formatted in equirectanguar projection can contain redundant information at the top and bottom ends, but not the midde body. So in the second step, we further spit the midde body into mutipe 3D meshes, each covering α o of the entire 360-degree wide screen. The number of second-step partitions, and therefore α o, can be a function of the horizonta FoV of the VR HMD patform. We spit the midde body into four segments with α o = π/2, so that each segment has a 90- degree cover, and simiary set 2β = π/2 foowing the defaut settings for the vertica FoV. Overa, with this procedure, our projection wi resut into a combination of six 3D meshes of a 3D sphere. We ca our customized 3D geometry a hexaface sphere 3D mesh. Figure 4 iustrates a visua overview of our process to generate a hexaface sphere. It shoud be noted that our assignment of α = π/2 and β = π/4 is not a hard requirement, and is derived from the defaut settings of our VR HMD device as we as the resut of our performance and quaity trade-offs. Our approach is genera, and depending on the performance of underying hardware can be extended towards higher number of ties and 3D meshes. Simiary, our tiing process is genera, and therefore can aso be empoyed in other geometrica projections such as cube maps. B. Viewport Tracking VR device orientation is usuay defined using three types of rotation over the X, Y, and Z axes. The vaues are inherenty represented in the Tait-Bryan anguar system, which uses a specia form of Euer anges that require 3 rotations around each of the 3 axes. To avoid possibe orientation tracking probems such as gimba ock, the anguar Euer system is transformed into a different rotationa system known as a quaternion, which is converted into a unit rotation matrix. To enabe view awareness, we use the foowing three steps to create vaid confines of unit quaternions specificay set for each of the hexaface sphere 3D mesh segments: convert Euer anges to a unit quaternion representation for VR device orientation tracking, cacuate an array corresponding to a normaized direction vector from our quaternion, combine the vaues together to compute the confines of segment-specific quaternion representations inside the hexaface sphere. With the confines of each 3D mesh segment being defined, we then identify which segments and the corresponding video ties intersect with a user s viewport and impement our viewport tracking at every frame. With viewport tracking, we then impement view-aware adaptation, and dynamicay deiver higher bitrate content to the ties within the user s FoV, and assign ower quaity eve to the area outside the user s immediate viewport. IV. EVALUATION To evauate our work, we used Samsung Gear VR HMD mounted with the Samsung Gaaxy S7 smartphone with 4GB RAM and Android Marshmaow 6.0. as our target VR patform. We used Ocuus Mobie SDK.0.3 aong with Android SDK API 24 for deveopment of a 360 VR video streaming appication prototype based on MPEG-DASH SRD. Our VR patform provides a tota resoution of 2560x440 (280x440 per eye), with maximum frame rate of 60 FPS and a horizonta FoV of 96 degrees. As stated before, we set the vertica FoV of our 360 VR video prototype to 90 degrees. We prepared 5 different 360 equirectanguar-formatted sampe videos pubicy avaiabe on Youtube as test sequences for the purpose of appying our adaptations. Tabe?? provides detaied information about our test video sequences. To benefit from the features of MPEG-DASH SRD-based streaming and for the purpose of texturing the hexaface sphere mesh, we deveoped an executabe script based on FFmpeg Zeranoe 64-bit API that given a video input, spatiay crops the video and generates 6 different ties as per requirements of

4 RELATIVE RATIO TABLE I 360 Video Resoution Orig. Bitrate (Mbps) FPS (Hz) Wado 3840x Pane 3840x Karate 920x Star Wars 3840x Racing Car 4096x BANDWIDTH USAGE Proposed (Rep4) Proposed (Rep2) Tied (Rep) Fig. 7: Visua comparison of a specific frame within Wado with the periphera ties having REP 4 (resoution of 240x480) W A L D O P L A N E S T A R W A R S R A C I N G C A R K A R A T E Fig. 6: A comparison of bandwidth savings of streaming different 360 VR videos, using our adaptations, and tied-streaming with no adaptation. our hexaface sphere geometry. We encoded a video segments with H.264 AVC encoder into 4 different representations using a hierarchica resoution degrading, with origina resoutions decreasing poynomiay at every step, as depicted in Figure 5. Fig. 5: Our hierarchica resoution degrading for various representations. We appied various sets of resoutions to different ties to experiment how our prioritized adaptations affects the average bitrate as we as the perceived quaity. Each tria of our experiment was run for a tota of 30 seconds, and during each run we setup the experiment in such a way that each tie is within user s viewport for 5 seconds. We measured the bandwidth usage in terms of average bitrate, when maximum resoution is assigned for ties within immediate FoV, and ower resoutions assigned to the other ties. Figure 6 demonstrates resuts for ony a sma subset of our experiments on a of our benchmarks, with ratios normaized to fit within a unit. We measured the reative bandwidth usage when using our adaptations compared to the baseine case where no adaptation is appied (the 360 VR video is tied; no viewport awareness is present, and a ties are streamed with highest representation (REP ). As can be seen, the resuts show that our adaptations can significanty save bandwidth usage for up to 72% compared to baseine case where our adaptation approach is not empoyed. Figure 7 shows a sampe screenshot of the experiments on Wado. Whie the highest representation possibe (REP - resoution of 960x920) is deivered to the main front tie, owest representation is deivered to the periphera tie on the right identified by the red confines (REP 4 - resoution of 240x480) Our adaptations resuts in minor noticeabe quaity impacts, sometimes not even perceptibe, whie maintaining the origina quaity for the main viewport to ensure a satisfactory user experience. V. CONCLUSION AND FUTURE WORK In this paper, we proposed bandwidth-efficient FoV-aware streaming adaptations to tacke the high bandwidth demands of 360 VR videos. Our nove adaptations expoits the semantic ink of MPEG-DASH SRD with a user s viewport to provide dynamic view awareness in the context of VR videos. We divide the buky 360 VR videos as we as the underying 3D geometry into spatiay partitioned segments in the 3D space, and then conquer the huge streaming bandwidth requirements using a dynamic viewport adaptation. Our initia experimenta resuts shows our adaptations can save up to 72% of bandwidth on 360 VR video streaming without much noticeabe quaity impacts. Appendix I incudes our adaptive rate aocation agorithm for tied streaming given the avaiabe bandwidth. In the future, we aso pan to extend the hexaface sphere towards higher number of 3D segments and measure the bandwidth saving-performance trade-offs. REFERENCES [] MPEG DASH, Information technoogy Dynamic adaptive streaming over HTTP (DASH) Part : Media presentation description and segment formats, ISO-IEC , [2] T. Stockhammer, Dynamic adaptive streaming over HTTP: standards and design principes, in Proc. of 20 ACM Mutimedia Systems, ser. MMSys, 20. [3] O. A. Niamut, E. Thomas, L. D Acunto, C. Concoato, F. Denoua, and S. Y. Lim, MPEG DASH SRD: Spatia Reationship Description, in Proceedings of the 7th Internationa Conference on Mutimedia Systems, ser. MMSys 6. New York, NY, USA: ACM, 206, pp. 5: 5:8. [4] J. Le Feuvre and C. Concoato, Tied-based adaptive streaming using mpeg-dash, in Proceedings of the 7th Internationa Conference on Mutimedia Systems, ser. MMSys 6. New York, NY, USA: ACM, 206, pp. 4: 4:3. [5] L. D Acunto, J. van den Berg, E. Thomas, and O. Niamut, Using mpeg dash srd for zoomabe and navigabe video, in Proceedings of the 7th Internationa Conference on Mutimedia Systems, ser. MMSys 6. New York, NY, USA: ACM, 206, pp. 34: 34:4. [6] M. Hosseini, J. Peters, and S. Shirmohammadi, Energy-budget-compiant adaptive 3D texture streaming in mobie games, in Proceedings of the 4th ACM Mutimedia Systems Conference. ACM, 203, pp.. [7] M. Hosseini, A. Fedorova, J. Peters, and S. Shirmohammadi, Energyaware adaptations in mobie 3D graphics, in Proceedings of the 20th ACM internationa conference on Mutimedia. ACM, 202, pp [8] M. Hosseini, G. Kurio, S. R. Etesami, and J. Yu, Towards coordinated bandwidth adaptations for hundred-scae 3d tee-immersive systems, Mutimedia Systems, pp. 4, 206.

5 VI. APPENDIX I: RATE ALLOCATION PROBLEM Our proposed view-aware tie-based 360 VR video streaming system is adaptive in the sense that it priorities the ties. It deivers higher bitrate content to the ties where the user is currenty ooking at and is most ikey to ook at, and ower quaity eve to the ties outside of a user s immediate viewport. For that purpose, we hereby expain the rate aocation probem within the context of tied 360 VR video streaming, and propose a rate aocation heuristic agorithm to enabe the adaptations within our tie-based 360 VR video streaming system. This appendix aims to mathematicay mode, and propose a rate aocation heuristic agorithm within the context of tie-based 360 VR video streaming to VR devices. The rate seection and aocation probem is the weknown binary Knapsack optimization probem, for which one approach to tacke is to transmit a subset of the whoe ties within the 360-degree VR environment. The binary Knapsack probem is NP-hard, but efficient approximation agorithms can be utiized (fuy poynomia approximation schemes), so this approach is computationay feasibe. However, using this method ony a subset of the whoe ties are seected, which is not desired since the VR user intends to receive a the necessary ties to avoid back views when the user turns head and changes orientation in the 360-degree space. Our proposed agorithms seects a necessary ties, but with different bitrates according to their priorities given the user s viewport. This is a mutipe-choice knapsack probem (MCKP) in which the items (ties in our context) are organized into groups corresponding to the objects. Each group contains the highest bitrate stream corresponding to an object and ower-bitrate versions of the same stream given the adaptation manifest. There are n ties T = {τ, τ 2,..., τ n } in the 360-degree VR environment. The highest possibe representation of each τ h T has a bitrate requirement of s τh, and a priority or importance coefficient p τh given the various priority casses per our definition. With view awareness feature of our tiing process, our agorithm assigns highest priority to the ties within the user s immediate viewport (C ), and owest priority (C 3 ) to the ties in the area outside the user s viewport in the 360-degree space. Figure 8 iustrates how our prioritization approach is appied against ties in the context of tied 360 degree VR videos. Our approach is genera, and can work with any number of priority casses. We use three casses in this piot study. We assume the quaity contribution of a tie τ h is a simpe function q τh = p τh s τh. The avaiabe bandwidth in every interva imits the tota bitrate of a ties that can be received at the headset device to W, which serves as an avaiabe budget. Let X = {x, x 2,..., x n }, be the set of ties that are received at the headset device, serving as the output of running rate aocation agorithm. Each x i X corresponds to an origina tie T. Simiary, each x i has a priority coefficient p xi = p τi. We assume there are L number of representations aviaabe given the manifest, with a representation of eve k noted as R k (0 k L) and the bitrate of a tie with representation R k noted as s R k. We assume the owest bitrate corresponds to the representation with highest ID which is R L which is determined as the minimum bitrate that can be toerated by Fig. 8: An exampe tie prioritization. Ties within viewport are assigned highest priority (C ), whie ties outside of viewport are assigned owest priority (C 3) users. In a simiar way, the quaity contribution of a tie x i is q xi = p xi s xi. A. Heuristic Agorithm Let S be the tota bitrate of a streams, and W be the avaiabe bandwidth budget. The minimum quaity that the user can toerate is given as the representation of eve L noted as R L. Let C, C 2, and C 3 be the cass of ties with the highest priority, medium priority, and owest priority, respectivey. For each tie in T, we cacuate q i as described previousy. This is the contribution that woud make to the average quaity of the system if it were received at highest bitrate possibe. We then cacuate W min = s R L which is the minimum bitrate that is needed to receive a ties at their owest bitrates. In the foowing, assume that W min W so the unused bitrate budget woud be W 0 = W W min. To determine the best bitrate for each tie, our agorithm sorts the prioritized ist of ties by the goba priority from the argest to the smaest. For ease of notation in the foowing, suppose that the ties are re-indexed so that the sorted ist of ties is τ, τ 2,..., τ n. If s τ s R L τ W 0 then there is enough unused budget to receive τ at highest bitrate (R 0 ), so the tie x woud have s x = s τ and woud contribute q to the average quaity. This eaves an unused bandwidth budget of W = W 0 s R0 τ s R L τ for the remaining ties after x. The agorithm repeats for τ 2, τ 3,... unti some ties τ cannot be received at highest bitrate within the remaining budget W. It then determines the highest possibe bitrate at which it can be received by cacuating the owest representation eve L : L L such that s R L W + s R L τ. The tie x wi have bitrate s x = s R L and wi contribute q to the average quaity of the whoe. The remaining bandwidth budget after streaming x wi be W = W s R L. The agorithm repeats this process to determine the proper bitrates, amount of bandwidth budget, and quaity contribution for each of the remaining ties x +, x +2,..., x n. Agorithm?? describes our heuristic agorithm. As a specia feature, our agorithm considers cases where the viewport intersects with mutipe ties in the 360-degree VR environment. Figure 9 iustrates the concept. As a differentiating tweak in that scenario, our muti-tie rate aocation heuristic ensures a visibe ties (ties within a user s viewport) get the same bitrate to avoid edging probems due to quaity variation of ties. For such cases, our agorithm packs a ties within the viewport together and deas with them as a singe tie just for the purpose of rate aocation, so that a visibe ties are assigned the same bitrate. This tweak minimizes any possibe edged views on the tie boundaries, therefore enabing a smooth viewing experience in the tied 360 VR

6 Agorithm 2: Rate aocation heuristic agorithm T : prioritized ist of ties sorted from highest to owest priority : tie with highest bitrate s τi x i : adapted tie with bitrate s xi L: Number of representation eves R L : Leve L representation Cacuate W min = s R L %comment: minimum bitrate requirement for a ties T : s xi s R L %comment: assign R L (minimum bitrate) to a s. W 0 W W min %comment: initiaization whie s τi s R L W i do %comment: i= initiay. s xi s τi W i W i (s τi s R L ) i i + %comment: adapt next tie end whie %comment: above oop repeats unti a tie τ cannot be deivered at highest bitrate within the remaining bandwidth budget W. i %comment: resuting from above oop. Find owest L L such that s R L W + s R L τ %comment: determines the highest bitrate possibe at which τ can be received within remaining budget, by cacuating the owest representation eve L. s x s R L %comment: adapt τ and cacuate s x Fig. 9: Muti-tie adaptation within an exampe 360 video frame. Mutipe ties within viewport are assigned same priority. video scenario. Treating a visibe ties as a singe tie depends on the deveoper preference and is not a hard requirement. It can be a pack of any of visibe ties given how much of the visibe ties intersect with the viewport. The agorithm needs a one-time impementation in the beginning of the session for the main process. Therefore it is impemented in rea-time and does not provide any additiona overhead during the runtime. It is impemented efficienty in O(nogn) time and O(n) space and produces soutions cose optima. The approximation error depends on the difference between the bitrate chosen for the first tie that cannot be received at highest bitrate (i.e. τ ) and the remaining budget avaiabe to receive it.