Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment

Impact of 3D bookmarks on Navigation and Streaming in a Networked Virtual Environment Thomas Forgione 1 thomas.forgione@irit.fr Wei Tsang Ooi 2 weitsang@nus.edu.sg Axel Carlier 1 axel.carlier@enseeiht.fr Vincent Charvillat 1 charvi@enseeiht.fr Géraldine Morin 1 morin@enseeiht.fr 1 Université de Toulouse IRIT 2 National University of Singapore May 11, 2016 T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat MMSys 2016 1 / 36

NVE Networked Virtual Environment 3D environment Streaming Free flying camera 5 degrees of freedom (x, y, z, θ and φ) Figure: An NVE from Sketchfab T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat MMSys 2016 2 / 36

NVE Networked Virtual Environment 3D environment Streaming Free flying camera 5 degrees of freedom (x, y, z, θ and φ) z T θ x P φ y Figure: Spheric coordinate system T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat MMSys 2016 2 / 36

Outline 1 Navigation in NVE Controls 3D Bookmarks 2 Quality of Experience Experimental setup Results 3 Streaming policies Comparison T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat MMSys 2016 3 / 36

Navigation in NVE Outline 1 Navigation in NVE Controls 3D Bookmarks 2 Quality of Experience Experimental setup Results 3 Streaming policies Comparison T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat MMSys 2016 4 / 36

Navigation in NVE Controls State of the art : difficulty of navigation Ruddle et al. [2000] Some interactions can get the user lost in the 3D environment Hyperlinks as such are unusable Jankowski and Hachet [2015] Difficult with a 2D screen Strongly task-dependent T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat MMSys 2016 5 / 36

Navigation in NVE Controls Keyboard controls : difficulty of navigation Esc F1 F2 F3 F4 F5 F6 F7 F8 F9 F10 F11 F12 1 2 3 4 5 6 7 8 9 0 - = \ Del Lock / * - Tab Q W E R T Y U I O P [ ] 7 8 9 Caps A S D F G H J K L ; Enter 4 5 6 Shift Z X C V B N M,. / Shift 1 2 3 Ctrl Alt Alt Ctrl 0. Figure: Keyboard controls z T θ + Enter P φ y x Figure: Spheric coordinate system T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat MMSys 2016 6 / 36

Navigation in NVE Controls Difficulty of navigation T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat MMSys 2016 7 / 36

Navigation in NVE 3D Bookmarks Arrow bookmarks Arrow curved according to the current position of the camera Shows path to targeted bookmark Figure: Bookmarks as arrows 3D data from McGuire [2011] T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat MMSys 2016 8 / 36

Navigation in NVE 3D Bookmarks Fly-to transition Figure: Smooth move from the current position to the bookmark T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat MMSys 2016 9 / 36

Navigation in NVE 3D Bookmarks Viewport bookmarks Computer vision style bookmark Optical center Image plane Figure: Bookmarks as viewports T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat MMSys 2016 10 / 36

Quality of Experience Outline 1 Navigation in NVE Controls 3D Bookmarks 2 Quality of Experience Experimental setup Results 3 Streaming policies Comparison T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat MMSys 2016 11 / 36

Quality of Experience Experimental setup Evaluating the impact of bookmarks with MicroWorkers Crowdsourcing People were recruited on microworkers.com More than 100 started the experiment 51 of them completed it, and were paid Traces of all experiments were recorded in order to be replayed to simulate streaming T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat MMSys 2016 12 / 36

Quality of Experience Experimental setup Problem : task design Without task, people won t browse our 3D scenes They have nothing to do They are probably not interested in our data 3D browsing is difficult = They will browse for a few seconds and leave = We will not see the impact of bookmarks Our solution Ask people to perform a task Find hidden objects in the scene T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat MMSys 2016 13 / 36

Quality of Experience Experimental setup Figure: A coin hidden behind a curtain T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat MMSys 2016 14 / 36

Quality of Experience Experimental setup Task design One micro-job Initial questionnaire (age, gender, etc...), tutorial 3 random tasks Final questionnaire to evaluate QoE One random task Select 1 3D scene out of 3 (loaded beforehand) Select 1 UI out of 3 (no bookmarks, 10 viewports, 10 arrows) Select 8 coins randomly out of 50 (positioned beforehand) No correlation between bookmarks and coin positions Success At least 6 coins found among the 8 hidden coins in the scene T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat MMSys 2016 15 / 36

Quality of Experience Results Stats on 51 users BM type #Exp Mean # coins # completed Mean time No bookmarks 51 7.08 18 4:16 min Arrows 51 7.39 27 2:33 min Viewports 51 7.51 30 2:16 min Table: Analysis of the sessions length and users success by type of bookmarks BM type Total distance Fly-to distance Ratio No bookmarks 610.80 N/A N/A Arrows 586.30 369.77 63% Viewports 546.96 332.72 61 % Table: Analysis of the length of the paths by type of bookmarks T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat MMSys 2016 16 / 36

Quality of Experience Results Questionnaire Questions Answers What was the difficulty level without bookmark? 3.04 / 5 ±0.31 (99%) What was the difficulty level with bookmark? 2.15 / 5 ±0.30 (99%) Did the bookmarks help you to find the coins? 42 Yes, 5 No Did the bookmarks help you to browse the scene? 49 Yes, 2 No Do you think bookmarks can be helpful? 49 Yes, 2 No Which bookmark style do you prefer and why? 32 Arrows, 7 Viewports Did you enjoy the experiment? 36 Yes, 3 No Table: List of questions in the questionnaire and summary of answers. = Quality of experience is better with bookmarks T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat MMSys 2016 17 / 36

Quality of Experience Results From users paths to system queries Percentage of triangles 0.6 0.4 0.2 Without bookmarks With bookmarks 0 10 20 30 40 50 60 Time (in s) Figure: Percentage of polygons queried after a certain period of time Higher demand to the server with bookmarks Potential impact on QoS T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat MMSys 2016 18 / 36

Quality of Experience Results Impact of bookmarks on navigation Faster navigation Better performance w.r.t. the task Good QoE, task perceived as simpler Higher demand to the server T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat MMSys 2016 19 / 36

Streaming simulation Client Server Figure: Streaming model T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat MMSys 2016 21 / 36

Streaming simulation Client Server Client info : Camera position Frustum of the camera Bookmark clicked if any Client info Figure: Streaming model T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat MMSys 2016 21 / 36

Streaming simulation Client Server Client info : Camera position Frustum of the camera Bookmark clicked if any Client info Compute faces to send according to a streaming policy Figure: Streaming model T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat MMSys 2016 21 / 36

Streaming simulation Client Server Client info : Camera position Frustum of the camera Bookmark clicked if any Client info Compute faces to send according to a streaming policy Polygons Figure: Streaming model T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat MMSys 2016 21 / 36

Streaming simulation Client Server Client info : Camera position Frustum of the camera Bookmark clicked if any Client info Compute faces to send according to a streaming policy Polygons Update camera (Evaluate rendering) Figure: Streaming model T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat MMSys 2016 21 / 36

Streaming simulation Client Server Client info : Camera position Frustum of the camera Bookmark clicked if any Client info Compute faces to send according to a streaming policy Polygons Update camera (Evaluate rendering) Client info Figure: Streaming model T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat MMSys 2016 21 / 36

Rendering evaluation One 3D model is fully loaded at the beginning One 3D model is loading progressively We compute the rendering on the fully loaded model R = Pixels already loaded face Pixels all faces Figure: A loading 3D scene with R = 0.9 T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat MMSys 2016 22 / 36

Streaming policies A naive streaming policy C : Frustum culling / Backface culling Send only polygons that may intersect the frustum of the camera Send only polygons that are oriented towards the camera Sort polygons by distance to the camera T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat MMSys 2016 23 / 36

Streaming policies C : Frustum culling / Backface culling Figure: Frustum culling T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat MMSys 2016 24 / 36

Streaming policies Two fold issue 1 0.9 R 0.8 0.7 0.6 0 1 2 3 4 5 6 7 8 Time (in s) Figure: Evaluation of the naive streaming policy T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat MMSys 2016 25 / 36

Streaming policies Two fold issue 1 0.9 R 0.8 0.7 C only 0.6 0 1 2 3 4 5 6 7 8 Time (in s) Figure: Evaluation of the naive streaming policy T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat MMSys 2016 25 / 36

Streaming policies Server-side rendering Find visible triangles Sort them by projected area Figure: Server-side rendering T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat MMSys 2016 26 / 36

Streaming policies Impact of server-side rendering R 1 0.8 0.6 0.4 Culling / distance sort 0.2 Precomputed faces / area sort 0 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 Number of Triangles Received 10 4 Great impact on QoS Not scalable : the server cannot do rendering for everyone But we can do it on bookmarks = Precomputation T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat MMSys 2016 27 / 36

Streaming policies A short review on prefetching Park et al. [2001] Predict the user s interest in an object Sort objects by interest and prefetch Hung and Liu [2006] Cluster the navigation path of users Predict future navigation path Prefetch objects near predicted path Zhou et al. [2015] Learn object access patterns Predict objects that are likely to be accessed and prefetch T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat MMSys 2016 28 / 36

Streaming policies C : Frustum culling / Backface culling Send only polygons that may intersect the frustum of the camera Send only polygons that are oriented towards the camera FD : Fetch destination Send the triangles that are seen by the bookmark clicked PP : Prefetch predicted Send the triangles that are seen by the bookmarks that might be clicked T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat MMSys 2016 29 / 36

Streaming policies PP : Prefetch Predicted Try to guess the next bookmark clicked knowing the previous one Next recommendation clicked 11 10 9 8 7 6 5 4 3 2 1 0 0 1 2 3 4 5 6 7 8 9 10 11 Previous recommendation clicked Figure: Patterns of access for a specific scene T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat MMSys 2016 31 / 36

Streaming policies PP : Prefetch Predicted Try to guess the next bookmark clicked knowing the previous one Next recommendation clicked 11 10 9 8 7 6 5 4 3 2 1 0 0 1 2 3 4 5 6 7 8 9 10 11 Previous recommendation clicked Figure: Patterns of access for a specific scene Frustum culling Bookmark i Bookmark j Figure: Prefetching packet T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat MMSys 2016 31 / 36

Comparison Quality of rendering after a click on a bookmark 1 0.9 R 0.8 0.7 0.6 0 1 2 3 4 5 6 7 8 Time (in s) Figure: Comparison of streaming policies T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat MMSys 2016 32 / 36

Comparison Quality of rendering after a click on a bookmark 1 0.9 R 0.8 0.7 C only 0.6 0 1 2 3 4 5 6 7 8 Time (in s) Figure: Comparison of streaming policies T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat MMSys 2016 32 / 36

Comparison Quality of rendering after a click on a bookmark 1 0.9 R 0.8 0.7 C only V-PP 0.6 0 1 2 3 4 5 6 7 8 Time (in s) Figure: Comparison of streaming policies T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat MMSys 2016 32 / 36

Comparison Quality of rendering after a click on a bookmark 1 0.9 R 0.8 C only 0.7 V-PP V-FD 0.6 0 1 2 3 4 5 6 7 8 Time (in s) Figure: Comparison of streaming policies T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat MMSys 2016 32 / 36

Comparison Quality of rendering after a click on a bookmark 1 0.9 R 0.8 C only V-PP 0.7 V-FD V-PP+FD 0.6 0 1 2 3 4 5 6 7 8 Time (in s) Figure: Comparison of streaming policies T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat MMSys 2016 32 / 36

Improvement with prediction is not significant Conclusion on policies Frustum culling is not satisfying Prefetching is better than frustum culling but not significantly (due to low predictability in 3D) Fetch destination is really efficient Fetch destination + Prefetching is barely better T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat MMSys 2016 33 / 36

Conclusion Bookmarks impact on NVE Good for QoE Drawback on QoS Prerendering the bookmarks can limit this drawback Future work Optimize chunk size allocated for prefetching Adapt fly-to transition to bandwidth Study the influence of the bookmarks positioning on user browsing pattern T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat MMSys 2016 34 / 36

Questions? T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat MMSys 2016 35 / 36

Shao-Shin Hung and Damon Shing-Min Liu. Using prefetching to improve walkthrough latency: Research articles. Comput. Animat. Virtual Worlds, 17(3-4):469 478, July 2006. ISSN 1546-4261. doi: 10.1002/cav.v17:3/4. URL http://dx.doi.org/10.1002/cav.v17:3/4. Jacek Jankowski and Martin Hachet. Advances in interaction with 3d environments. Computer Graphics Forum, 34(1):152 190, 2015. Morgan McGuire. Computer graphics archive, August 2011. URL http://graphics.cs.williams.edu/data. Sungju Park, Dongman Lee, Mingyu Lim, and Chansu Yu. Scalable data management using user-based caching and prefetching in distributed virtual environments. In Proceedings of the ACM Symposium on Virtual Reality Software and Technology, pages 121 126, Banff, Canada, 11 2001. Roy A Ruddle, Andrew Howes, Stephen J Payne, and Dylan M Jones. The effects of hyperlinks on navigation in virtual T. Forgione, A. Carlier, G. Morin, W.T. Ooi, V. Charvillat MMSys 2016 36 / 36