Protected-3DMPS: Remote-Rendering Based 3D Model Publishing System in Digital Museum

Transcription

1 Protected-3DMPS: Remote-Rendering Based 3D Model Publishing System in Digital Museum Cai Su Jin Ping Qi Yue Shen Xukun Beihang University, China {caisu, jinping, qy, Abstract To prevent piracy or misuse of the 3D digital heritage models, we put forward a new 3D model publishing system, called Protected- 3DMPS, by combining the geometry loss compression and remote rendering technology. It consists of a 3D viewer client that includes a point-rendering geometry-compressed model, and a server that renders and returns images of the original high-resolution models according to client requests. This system can effectively defend against different attacks from malicious users and guarantee the security of 3D models. Meanwhile, it also significantly lowers the requirements for the client machine s graphics hardware and network bandwidth. Keywords: digital museum, data security, geometry compression, remote rendering 1. Introduction To protect the rare physical cultural heritages through digitized processing and exhibit it over the Internet constitute as an integral part to the digital museum, which satisfies the demands of education and research [1]. The development of laser scanners and 3D modeling methods is conducive to the acquisition of precise 3D surface data of cultural relics. An example of such a collection is the Stanford Digital Michelangelo Project [2], in which a highresolution digital archive of 10 large statues of Michelangelo was created. For instance, highquality models exist of the David at 1.0 mm resolution (56 million triangles) and St. Matthew at 0.25 mm (372 million triangles). However, the exhibition and publishing of scanned cultural relics involve huge 3D data, which face the following challenges. First, data security. The piracy of digitized cultural relics is strictly prohibited, while current methods cannot guarantee their security. Second, the rendering efficiency of the client s graphics hardware. While the performance of graphics hardware also sees a drastic rise in recent years, however, the ability to handle enormous data sets overloads the capabilities of the state-ofthe-art graphics chip. Third, the bottleneck of network bandwidth. Current network bandwidth failed to serve the needs of huge 3D data transmission, which cannot suffice to offset the explosion of the popularity of 3D models. 2. Related research The majority of current researches focus on the mesh watermarking technique in order to protect the digital media. Watermarking provides a mechanism for copyright protection of digital works by embedding some information (such as the text, serial number, brand image, etc.) in the original data, allowing people to permanently mark their documents and thereby prove claims of authenticity or ownership. Watermark is invisible and it can be detected and extracted from the published data. Among the literature describing 3D mesh watermarking, Praun constructed a set of scalar basis function over the mesh vertices using the multi-resolution analysis and the optimization technique to enhance its robustness [3], and Ohbuchi presented an informed-detection, robust mesh-watermarking algorithm [4] [5] that embeds message bits by deforming the low-frequency components of the shape by using the mesh spectral analysis. However, there exists a fatal drawback in the 3D mesh watermarking technique: It cannot prevent the illicit copying and misuse, but only to detect the piracy after the fact. In regard to the 3D model rendering technique over the Internet, according to the rendering position, it can be divided into 3 methods:

2 (1) Local rendering. The traditional singleresolution model is downloaded from the server and then rendered at the local client. In LOD and progressive methods, before downloading the model, a series of optimization processes will be pretreated at the server: mesh simplification, progressive coding, etc. It can be adapted to high performance of graphic rendering and high real-time interaction application. (2) Server rendering. The server takes charge of the graphics rendering and implements collaboration with the interaction request from the client. After completing the rendering task of the virtual scene, the server sends the result back to the client by images, videos or other media data. Most systems based on the server rendering aim to solve the low performance of graphics rendering at the client. Yoon provided the algorithm of efficient distribution of rendering burden and data transmission between the server and the client [6]. Koller listed and analysed several possible types of attacks on the remote rendering server, and proposed a remote rendering system for protected interactive 3D graphics [7]. (3) Combined rendering. Mann used a powerful graphics-workstation server to incrementally send large collections of textures to a client, i.e. the client interpolates rendered images from available data while the server renders a difference image between the client s and the original view every couple frames and sends missing texture data to the client [8]. Unfortunately, warping causes artifacts, for example gaps in the areas where data is missing on the reference image. To fill those gaps, only the pixels that cover the gaps are sent to the client every frame. Using such an approach, Cohen-Or reported a reduction of the bit-stream size up to a few percent of a corresponding MPEG-2 stream [9]. However, the result did not take into account the transfer of geometry, which had to be available both for the server and the client. Engel presented a concept and system for distributed 3D visualization of volume data for medical applications, combining remote and local 3D visualization [10]. Teler achieved an algorithm by selectively transmitting only parts of 3D model data in the scene for the rendering at the client, while the others are rendered at the server and then sent to the client with images [11]. In a game, for example, game objects (monsters, guns, keys, etc.) are more important than the scenery (plants, mountains, etc.), thus the former was sent by 3D model data and the latter by images. This method integrates the advantages of both the client rendering and the server rendering, however, it highly relies on the complex selectivity algorithm on foreground and background models. We implement a new 3D model publishing system based on the remote rendering technique: Protected-3DMPS. It consists of a viewer client that includes a point-model, and a rendering server, where it returns highresolution 2D images according to the client s request. The user manipulates the point-model when sending some interactive commands (move, rotate, zoom, etc.) via the mouse, keyboard or other input equipments at the client. The high-resolution images will not be sent to the client until the user stops the interactive behavior. In contrast to previous approaches, Protected-3DMPS manages the trade-off between numerous data and limited network bandwidth, and prevents the piracy of original 3D model, which has been applied to the Chinese Digital Museum Project. Section 3 illustrates the client and the server respectively in Protected-3DMPS. The antiattack measures to ensure the data security are described in Section 4. We then give and discuss the experiment result in Section 5. Section 6 concludes the paper and figures out the tasks remained to complete the project in the near future. 3. Protected-3DMPS The architecture of our 3D model publishing system based on remote rendering is shown in Figure 1. Figure 1: The architecture of Protected-3DMPS

3 3.1 Client At the client, the user can manipulate (move, rotate, zoom, etc.) a point-rendering 3D model in a view window. When a behavior stops, a rendering request was sent to the server through network, which includes the current viewpoint position, viewpoint orientation, and transformation matrix to change the model s status, etc. According to the information from the client, the server rendering module renders the high-resolution model and sends the image back to the client, which replaces the current image at the view window of the client. Figure 2 describes the process. Figure 2: An example of manipulating a model at the client In a favorable network environment, the user has the impression of manipulating a highresolution 3D model rather than a lowresolution one. As shown in Figure 2, the Fighter P47, scanned in Beijing Aviation Museum, is displayed in the Chinese Digital Museum Project, where the original model includes 1,089,796 triangles and 546,185 vertices, while the point-model has about 50,000 vertices at the client. 3.2 Server Rendering module The rendering module of the server is to render the high-resolution model according to the request from the client, obtain an image and send it back to the client. Some methods are employed in both the software and hardware to accelerate the process. (1) For some frequent commands of rendering and changing of status, we use a list instead to reduce the number of visiting memory. (2) The vertices coordinates, texture coordinates, normal and color data are stored in the graphics card memory by OpenGL VBO [12]. The speed of GPU, which is faster than that of CPU, has greatly enhanced the performance of graphics rendering. The above measures raise the average speed from 24 fps to 45 fps when rendering a 896,391-triangle model on a P4 3.0GHz CPU, and an ATI Radeon 9800 Pro graphic card Image readback and compression The server must read back the rendered image which was saved in the framebuffer by pixels. Then we should solve two problems: (1) The framebuffer is an on-screen buffer bundled with a window, which limits the information reading from the framebuffer, e.g. the window size and the pixels format. If the bundled window is obstructed by other windows, some object image data in the framebuffer will be lost. In our system, we use P-Buffer instead, which is one of the OpenGL ARB extension [12]. P-Buffer is an off-screen buffer, which is invisible and not bundled with any window. Then the server application can set the P-Buffer property freely. (2) The image extracted from the P-Buffer is a BMP format file with 4 byte per pixel. Usually a BMP image file is 2MB, beyond the capability of network transmission. We compress the image by JPEG compression, which reduces the file size to 15~40 KB without decreasing the image quality seriously.

4 3.2.3 Communication thread To enhance the concurrent capability of Protected-3DMPS, the server application uses the multi-thread technique. The server sets up a communication thread for every client, receives the rendering request, reads the viewpoint, window size, model status, etc, and sends those data to the rendering module. The rendering module and the image cache are both critical resources, therefore we adopt FCFS (First Come First Service) scheduling. 4. System security protection measures Compared with the popular download and play pattern, Protected-3DMPS enjoys high security advantages since the original 3D model is saved at the server, while the client gets only images and a point-model. In [5] a low-resolution model is sent to the client, given that the low-resolution model complexities are of little value to potential thieves. However, the coarse model can still be useful for those malicious users who only need to publish it over the Internet for some commercial motivation. To guard against the above misuse, a point-rendering model with loss geometry compression is adopted, which increases the cost to reconstruct the mesh from the geometry data and achieves a high rate of compression. The geometry-guided compression algorithm is beyond this paper, and we will discuss it in another paper. Therefore it only needs to be noted that the octree decomposition method was employed here. Through the geometry compression technology, we can set the roughness of the object 3D model. For example, a highresolution model with 1,000,000 vertices is simplified to 50,000 vertices, which keeps the profile of the original model, and then is coded effectively to get a smaller file, which lowers the data transmission burden. However, the method noted above does not take account of the attacks through the computer vision technology, which makes it possible to reconstruct 3D model from 2D images [13] [14]. To prevent the attackers from reconstructing the original model by images, we take some measures to enhance the robustness of our system. 4.1 Log analysis Many images captured from a series of viewpoint around the object model can reconstruct the original 3D model, as shown in Figure 3. Figure 3: Capturing images to reconstruct a 3D model To defend against this type of attacks, in our system, the server application labels every client application and tracks and monitors the rendering request lists. By analyzing the log, it can identify those suspicious request lists, such as a huge amount of requests in a short moment, or requests including incrementally selected viewpoints, etc. When detecting a suspicious request list, the server application will close the connection to that client. 4.2 Request limitation Besides the log analysis, if necessary, we can limit the viewpoint condition at the client. For example, forbidden viewpoints are set to prevent the attackers from manipulating the model from some orientations, by which the attackers cannot reconstruct the 3D model due to the missing images. 4.3 Noise distortion Among the current reconstruction algorithm by the computer vision technology, noise in the source images will decrease dramatically the quality of the reconstructed 3D model. Then we can add some noise to the rendered images sent to the client, so as to increase the reconstruction difficulty effectively. In Protected-3DMPS, the ways of noise distortion vary from distorting viewpoint parameters, distorting lighting parameters to adding highfrequency noise to the rendered images. However, noise distortion should be undertaken without disturbing the user browsing the model.

5 5. Result and analysis connecting the server. The results are shown in Table 1. We test several models of cultural relics on Protected-3DMPS with only one client Table 1: Performance Tests of Protected-3DMPS Models Number of triangles in the original model Number of vertices in the original model Number of vertices in the point-model at the client rendering time (ms) P-Buffer readback time (ms) JPEG compression time (ms) compressed image size (KB) FPS at the client Riyue Kwan-yin 896, ,244 15, P47 1,089, ,185 49, Black Widow 2,084,711 1,050,975 56, Terra-cotta Warrior 143,971 72,156 25, Baoding Buddha 411, ,774 25, Test machine Server Client P4 3.0GHz CPU, 2096MB Memory, NVIDIA Quadro FX3400 Graphic Card P4 2.4GHz CPU, 512MB Memory, ATI Radon9550 Graphic Card At the server the bottlenecks lie in the rendering, readback and image compression. For the 896,391-triangle model, the rendering module will consume 42 milliseconds in handling a request at the server mentioned above in the Table 1, in which the rendering process takes about 17 milliseconds and readback and JPEG compression cost about 25 milliseconds. In another word, in average, the server can handle about 24 requests from the client per second. Traditional remote rendering methods usually transmit the image stream without 3D data to the client. To satisfy the real-time interactivity at the client, the image stream needs 30 fps. According to our result, the data stream needs 750 KB/s when an image is about 25 KB, and even if the speed of the image stream is only 24 fps as mentioned above, the data stream reaches 600 KB/s, which remains a heavy burden to the current network. In fact, the high rate of data transmission is not required in our system. Protected-3DMPS pretransmits a point-model to the client, which can be manipulated by the user when he/she performs the interactive behavior. We think there is no need for strict requirement of the detail while the user is manipulating the model. And when the user stops the interactive behavior, he/she can see the rendered highresolution image immediately. Assuming the user observes the model from 60 different positions every minute, the server needs to send 60 images, i.e. the rate of the data transmission is 25KB/s (25KB 60 / 1min ). This method not only remedies the defects of the traditional real-time remote rendering methods, but also avoids substantial losses of the model authenticity. Another experiment is conducted to test the concurrency of Protected-3DMPS. Figure 4: The performance of the capability to handle concurrency. The model is Riyue Kwanyin. Figure 4 shows the average rendering time and the average total time of handling a request (including the rendering time, the readback time, the image compression time, etc.) at the server when the number of clients varies from 1 to 28. Note that every client continuously sends requests to the server. From Figure 4, we see that the average rendering time per request is a constant, while there is a linear relation between the average total time of handling a request and the number of clients.

6 6. Conclusion and future work This paper implements a 3D model publishing system based on remote rendering technique. It avoids sending the high-resolution 3D model of the rare cultural relics to the client directly and guarantees the 3D data security. The main rendering task is completed at the server, which lowers the graphics hardware requirement of the client. The user manipulates the pointrendering 3D model and the client sends rendering requests to the server synchronously. Then the server sends the rendered image of the high-resolution 3D model at a time-cycle, which decreases the server rendering burden and enhances the client real-time interactive capability. Since the server sends a point-model and a series of compressed images, the network bandwidth is no longer a bottleneck. Though our system enjoys an advantage over traditional 3D model publishing systems, there remains some drawbacks to be addressed. Further and broader studies on our work may be carried out in the following aspects: (1) Out-of-core rendering. Out-of-core processing is not new, and, in fact, has long been used to cope with huge amounts of data [15] [16]. Using an out-of-core method is the only solution in the absence of large memory space due to the gigantic graphics for the scanned cultural relics. How to organize the information and implement the data reading from the disk to the memory is the main field of out-of-core algorithms. We plan to use the out-of-core technique to enhance the capability of rendering massive-data-model at the server. (2) Client-server to browser-server. To people who visit the digital museum by the web browser over the Internet, the browser-server architecture is more convenient than the clientserver, since there is no need to publish a special client application. We plan to implement a browser/server application based on the present work. Acknowledgements This work has been supported in part by the Chinese Digital Museum Project and the Research on the Key Technology of Virtual Olympic Museum, a key project of the National Natural Science Foundation of China, No The authors would like to thank all people in the DM team, VRLab, Beihang University for their helpful discussion and suggestion. For the use of the 3D models we would like to thank Hu Yong and Zhao Xuewei for acquiring 3D data and modeling. References [1] Addison, A. Emerging trends in virtual heritage. IEEE Multimedia, Special Issue on Virtual Heritage 7, 2 (April 2000), pp [2] Levoy, M. The digital Michelangelo project archive of 3D models [3] Praun, E., Hoppe, H., and Finkelstein, A Robust mesh watermarking. In Proc. of ACM SIGGRAPH 99, [4] Ohbuchi, R., Mukaiyama, A., and Takahashi, S A frequency-domain approach to watermarking 3d shapes. Computer Graphics Forum 21, 3. [5] Ohbuchi, R., Takahashi, S, Miyazawa, T. and Mukaiyama, A. Watermarking 3D Polygonal Meshes in the Mesh Spectral Domain, in Proc. Graphics Interface 2001, pp. 9-17, [6] Yoon, I. and Neumann, U. Web-based remote rendering with IBRAC (imagebased acceleration and compression), in Proc. of EUROGRAPHICS 2000 (M. Gross and F. HopGood, eds.), vol. 19, Blackell Publishers, Oxford, UK, [7] Koller, D., Turitzin, M., Levoy, M., etc. Protected Interactive 3D Graphics Via Remote Rendering. ACM Trans. Graph. 23(3), pp [8] Mann Y., Cohen-Or D. Selective pixel transmission for navigating in remote virtual environments. Computer Graphics Form (Proceedings of Europraphics 97), 1997, 16(3): 201~206. [9] Cohen-Or D., Mann Y. and Fleishman, S. Deep Compression for Streaming Texture Intensive Animations. Proceedings of the 26th annual conference on Computer graphics and interactive techniques, SIGGRAPH 1999, pp [10] Engel, K., Hastreiter, P., Tomandl, B., Eberhardt, K. and Ertl, T. Combining Local and Remote Visualization Techniques for

7 Interactive Volume Rendering in Medical Applications, IEEE Visualization '00, pp [11] Teler, E., Lischinski, D. Stream of complex 3D scene for remote walkthroughs. Euro Graphics2001, Manchester, UK, [12] OpenGL Extension Registry. [13] Slabaugh, G., Culbertson, W.B., Malzbender, T. and Schafer, R A survey of volumetric scene reconstruction methods from photographs. In Proc. International Workshop on Volume Graphics, pp [14] Slabaugh, G., Culbertson, W.B., Malzbender, T., Stevens, M.R. and Schafer, R.W. Methods for Volumetric Reconstruction of Visual Scenes. International Journal of Computer Vision, Volume 57, Issue 3, 2004, pp [15] Varadhan, G., Manocha, D. Out-ofcore rendering of massive geometric environments. In IEEE Visualization 2002, IEEE, Boston, Massachussetts, R. Moorhead, M. Gross, and K. I. Joy, Eds., pp [16] Yoon, S., Salomon, B., Gayle, R. and Manocha, D. Quick-VDR: Interactive view-dependent rendering of massive models. In VIS 04: Proceedings of the IEEE Visualization 2004 (VIS 04), pp