Development of 3D Mobile Receiver for Stereoscopic Video and Data Service in T- DMB

Transcription

1 Development of 3D Mobile Receiver for Stereoscopic Video and Data Service in T- DMB *Gwangsoon Lee, *Hyun Lee, * Kugjin Yun, *Namho Hur, and *Soo In Lee *Electronics and Telecommunications Research Institute, Republic of Korea ABSTRACT In this paper, we present a development of 3D-T DMB (three-dimensional digital multimedia broadcasting) receiver for providing 3D data service which delivers and plays the 3D data objects together with existing AV stream over T-DMB network. The implemented 3D-DMB receiver has capabilities of supporting a delivery mechanism for 3D data service while securing backward compatibility with existing T-DMB, generating stereoscopic viewing on the glasses-free 3D mobile display. Specially, this paper introduces hardware and software architecture and its implementation of 3D T- DMB receiver. Finally, the effectiveness of 3D data service is verified through the experimental system including the implemented receiver and a variety of service examples. Keywords: 3DTV, mobile broadcasting, data service, stereoscopic 1. INTRODUCTION Recently, efforts to provide realistic stereoscopic contents to home have been in progress with help of the abundant supply of 3D contents from movie industry and release of various types of 3DTV in the markets. 3D contents delivery over mobile broadcasting is expected to be a very attractive service because single-user environment at the mobile terminal is suitable for glasses-free 3D viewing [1]-[2]. The 3DTV service over mobile broadcasting has some of advantages. First, it definitely aims for a single user on the portable multimedia player with a small display. Therefore, it is not only easy to provide the glasses-free 3D viewing but also relatively free from the viewing zone limitation of the auto-stereoscopic display since the user can easily adjust the player for comfortable viewing. Second, a visual fatigue that might be caused from 3D viewing can be reduced due to a small range of binocular parallax. Among the commercialized mobile broadcasting technologies T-DMB is one of applications that have emerged from the Eureka-147 DAB system [3]-[5]. With the help of a development of high-quality auto-stereoscopic technologies, a 3DTV service over T-DMB, what we call a 3D T-DMB service, has recently been introduced to provide further realistic mobile broadcasting service [6]. However, launching a new broadcasting service generally requires the service provider and manufacturer to consider a variety of viewpoint that are closely related to economical efficiency. In other words, it is very important to minimize a bit-budget necessary for the 3D T-DMB service, while maintaining the existing T-DMB service. As a service method to meet such requirement, 3D T-DMB data service, which uses the MPEG-4 binary format for scene (BIFS) technology supported at T-DMB [5], has been introduced to give a 3D effect in a different manner with conventional 3DTV service. In this paper, we introduce the mechanism of 3D data broadcasting service in T-DMB, followed by implementation of 3D T-DMB receiver supporting it. Then, we verify the effectiveness of 3D T-DMB service through several experimental results. 2.1 Overview of 3D Data Service in T-DMB 2. 3D DATA SERVICE IN T-DMB For an additional data service in accompany with basic AV service, T-DMB adopts the BIFS to encode an interactive data related to video contents in addition to MPEG-4 advanced video coding (AVC) for video and MPEG-4 (sliced arithmetic coding) BSAC for audio [5]. Based on MPEG-4 BIFS technology, T-DMB can provide static or dynamic

2 image, graphics and text data associated with a specific video program, using a flexible scene description tool that allows synchronous representation of audio-visual objects in a scene. The BIFS also includes information on visual properties of objects for rendering, spatial position of objects and relative time for rendering. 3D T-DMB data service has been proposed by using MPEG-4 BIFS in T-DMB, as shown in Fig. 1 [5]. In this 3D T- DMB data service, 3D object clip with file formats of JPG, PNG, and MNG4 (Multiple Network Graphics) are delivered through the auxiliary (AUX) data path and played on top of 2D ordinary video. Partial 3D object clip Partial 3D object clip 2D background video Fig. 1. Basic concept of 3D T-DMB Data service. As 3D T-DMB data service is a kind of program associated data service, the 3D objects are related to the 2D video program, highlighting the eye-catching portions on plain 2D background video. The 3D data object is relatively smaller than additional data for the 3D video service. Thus for most of the case, less than 64 kbps is enough to deliver 3D data via T-DMB data channel even though it is depends on the size of 3D objects. Comparing with 3D video service, 3D data service has other advantages like reducing eye fatigue and enhancement of 3D effect because of partial presentation of 3D images on top of 2D background image. We believe this technology will be applied to many 3D broadcasting applications such as advertisement, education, sports, movie, drama, and so on. 2.2 System Configuration As shown in Fig. 2, 3D DMB data broadcasting system is an extension of the T-DMB system, which can provide users with stereoscopic multimedia contents. This system further includes some of devices including the contents authoring tool, data broadcasting server for creation of 3D data and its delivery. In order to fully make sure the compatibility with the legacy T-DMB system, additionally encoded data stream for providing 3D viewing should be multiplexed into the MPEG-TS (transport stream) level. Therefore, the MPEG-2 TS re-multiplexer (ReMUX) generates single stream after combining encoded stream for T-DMB AV service and 3D data service. The multiplexed MPEG-TS is inserted into a preallocated sub-channel at the ensemble multiplexer that forms service channels and combines a variety of service in T- DMB. In the receiver side, 3D DMB receiver has a few of requirements such as adopting auto-stereoscopic 3D display, 2D/3D modes switchable and so on. Fig. 2. System configuration for 3D data service over T-DMB.

3 2.3 3D Data Service Mechanism There will be various configurations of 3D service in mobile broadcasting. A program can be operated in 3D mode. In other words, 3D contents can be delivered all the time over the program. In that case there might be a little problem such as the supply of enough 3D contents, eye fatigue due to watching 3D contents for so long time. Therefore, we believe that 2D and 3D programs should alternate with a pre-determined time schedule as form of 2D, 3D, 2D Based on such service requirement, 3D data service linked with video program was designed to provide at designated time interval as well as at all over program. Note that such scheduling delivery of 3D data contents would be very useful to overcome a constraint of 3D data service such as the supply of 3D contents, eye fatigue etc. As an example, Fig. 3 shows that ordinary DMB video service, monoscopic data service linked with the video program and stereoscopic data service linked with the video program are sequentially provided according to the scheduled time. Further for each service, corresponding data contents with format of JPG, PNG, MNG are also delivered via T-DMB channel. Fig. 3 also shows that the service type is changed according to the scheduled time, in accompany with signaling mechanism to distinguish 2D or 3D service and to let the receiver know which type of service is available on current channel, enabling the corresponding behavior in the display such as turning off a parallax barrier. Fig. 3. Example of service scenario for 3D data service. 2.4 Delivery mechanism of 3D data In order to provide mobile broadcasting service in a manner that monoscopic and stereoscopic data contents are delivered as a form of scheduling event, shown in Fig. 3, a specific signaling mechanism is necessary like any other data broadcasting services. In case of 3D data service in T-DMB, stereoscopicdataservice_descriptor(sds_descriptor) as shown in Table I is defined and inserted into the first descriptor loop that follows program_info_length in program map table (PMT). Table I. Syntax of SDS_descriptor Syntax # of bit SDS_descriptor () { descriptor_tag 8 descriptor_length 8 StereoMono_serviceFlag 1 If (StereoMono_serviceFlag) { reserverd 3 CompositionType 2 LR_first 1 } else{ reserved 2 } } Core information of this descriptor includes a flag distinguishing whether the existing DMB program and new stereoscopic program, a composition type of stereoscopic data and information on reference images (whether left or right). In Table I, StereoMono_serviceFlag denotes that the corresponding program is whether stereoscopic or mono

4 (existing DMB) service. The 3D DMB receiver turns off or on the 3D functionality of display according to the value of StereoMono_serviceFlag. CompositionType denotes the composition type of stereoscopic data, which includes dual file (left and right respectively), side-by-side etc. L_R_first=1 implies that the reference image corresponds to the left images and L_R_first=0 implies that the reference image corresponds to the right image. Herein, the reference image means images designated by independent elementary stream (ES) within the object descriptor (OD). 2.5 Contents Access Procedure Access procedure of stereoscopic contents in T-DMB is shown in Fig. 4. As shown in the figure, the contents access procedure is composed of 9 steps. Each step is described in the following. Fig. 4. Procedure for accessing 3D contents in 3D T-DMB signal. Like any other MPEG-2 system decoder, the receiver first parses the PAT and then access to PMT. In the process of parsing the descriptor loops of PMT, when SDS_descriptor is found the receiver analyzes the information defined Table I. After parsing the IOD_descriptor, ES descriptors related to BIFS and OD are respectively analyzed. Herein after each ES_ID are identified, SL_descriptors that have same ES_ID are obtained in the second descriptor loop. Then, using PID and stream_type that are placed on the SL_descriptor, corresponding video, audio and data packets are extracted from MPEG-2 TS. In the process of parsing the OD, ES_ID and ObjectTypeIndication are identified. For example, in case of JPEG images that can be used for stereoscopic data service, ES corresponded to ObjectTypeIndication=SA JPEG is received and delivered to additional JPEG decoder for composing the stereoscopic image pair. 3. IMPLEMENTATION OF 3D DMB RECEIVER 3.1 Hardware Part Fig. 5 shows the hardware architecture of 3D T-DMB receiver for 3D data service. The basic receiving and decoding functions are performed on the RF tuner, baseband processor and multimedia processor. The host processor equipped with an ARM and DSP core performs most of the functions regarding 3D data services such as parsing of BIFS command, decoding of JPG/MNG/PNG and 3D rendering. The host processor is inter-connected with the multimedia processor via HPI interface, requesting BIFS-related and image data and receiving extracted data as its responses. The proposed 3D T-DMB receiver adopts the parallax barrier method as the glasses-free 3D display technology. The parallax barrier controller in the host processor controls to generate the square wave pulse signal, which operates the parallax barrier placed on the LCD display.

5 Fig. 5. Structure of 3D T-DMB receiver. As illusted before, the implemented 3D T-DMB receiver adpots the parallax barrier method. As the parallax barrier is a kind of TN LCD, it is operated by voltage difference between upper and lower layer of LCD. Light blocking effect occurred by the parallax barrier is known to be almost proportional to the voltage amplitude, therefore the voltage should be decided to make a best quality of 3D viewing. The other factor that impacts on the 3D viewing quality is the frequecy of voltage signal that forms a shape of Pulse Width Modulation (PWM). We experimetally comfirmed that both voltage amplitude and frequcey is closely related to the 3D viewing quality by properly seperarting the left and right light, refrainnig image flicker from generating. The shape of PWM signal that drives the parallax barrier is shown in Figure 6. In case the upper layer (A0) and the lower layer (A1) are respectively driven by the PWM signal and GND or in case the upper layer (A0) and the lower layer (A1) are respectively driven by GND and the PWM signal, the Barrier LCD becomes ON status. On the other hand, In case the upper layer (A0) and the lower layer (A1) are simultaneously driven by GND, the Barrier LCD becomes OFF status, which corresponds to the 2D viewing mode. g e o lta V Upper layer (A0) 0 Parallax Barrier (ON) V op time Parallax Barrier (OFF) f Lower layer (A1) 0 GND. Fig. 6. PWM signal driven to the parallax barrier. 3.2 Software Part The 3D Data Service Module, which performs a main function to visualize the 3D data service, is shown in Fig. 6. Herein, the DMB interface(i/f) Module is made up of Damon and Damon library, which receive the event information from the baseband processor and delivers it into the Event hander. The Event Handler parses the event information and delivers the parsed information into each module including the 3D BIFS Renderer. The EPG browser modifies EPG data based on the parsed information delivered by the Event handler, presenting them through the View Handler (GUI). In case that the events are generated from the Baseband processor or user s request, GUI information is modified through View handler. The BIFS Renderer receives data stream and its additional information such as CTS, IOD, BIFS etc. from the Event handler. The BIFS Library parses CTS, IOD, BIFS and then extracts image data, which is delivered into the Image decoder at the time designated by the CTS. Note that CTS is necessary to synchronize video, audio and related images. The Image decoder performs to decode image data with format of JPEG, PNG, MNG. Specifically, it is composed to a couple of image decoder for synthesizing the stereoscopic images.

6 Fig. 7. Software Structure of 3D Data Service module. The 3D data service module is implemented on Windows CE operating system so that a variety of image processing such as resizing, image filtering, overlay and etc. can be easily supported by the DirectDraw Engine. Decoded image data are overlaid on the video data with help of the DirectDraw engine. The Display finally outputs overlaid stereoscopic images on the 3D display. The auto-stereoscopic 3D display employed in our receiver uses the direction-multiplexed technique being adopted the conventional parallax barrier, in which each eye of the viewer can see only the corresponding view generated by the light of each pixel. Therefore, it is necessary to generate left/right interleaved images that can form stereoscopic view though the glasses-free 3D display as shown in Fig. 8. In this figure, since each resolution of stereoscopic images is QVGA size (320x240), up-sampling process is only done in vertical direction to generate a left/right interleaved image of VGA size (640x480) Left image Right image 640 Horizontal : L/R interleaving Vertical : up-sampling (x 2) 480 Vertical Line Interleaved image Fig. 8. Procedure for generating left/right interleaved image for the Display of 3D T-DMB receiver. 4. Experimental results In order to verify the designed 3D T-DMB service, we have configured the experimental 3D T-DMB data system shown in Fig. 9. First, we implemented a 3D Data service studio that creates a number of 3D data objects encoded by the MPEG-4 BIFS format, a 3D data service scheduler that inserts 3D data objects according to pre-determined time and a

7 3D data service multiplexer that multiplexes encoded 3D data objects with T-DMB stream. Then, the stream are modulated and broadcasted to the air via the portable T-DMB transmitter. Storage 3D Data Service Studio 3D Data Service Multiplexer 3D Data Service Scheduler Portable T-DMB Transmitter 3D T-DMB Receiver 3D Data Service Broadcasting Server Fig. 9. Pictures of experimental 3D T-DMB data broadcasting system. TABLE II The experimental condition Component Video Audio 3D object Total (MPEG-TS rate) Bit rate 320Kbps 54 Kbps 64 Kbps - Each size per object : up to 50 Kbyte - the number of objects : 10 - maximum interval between objects: 10 sec 576 Kbps We tested and finally attained the proper voltage amplitude and frequency of PWM signal that should be driven to the parallax barrier while verifying a best quality of 3D viewing on the 3D display. The proper values may depends on the the types of parallax barrier, in our case the voltage amplitue (V op ) of PWM signal is around 10Vp-p DC and frequeny (f) of PWM singal is 60 Hz. Fig. 10 introduces two applications of 3D data service, 3D log image and 3D advertisement. Fig. 10. Pictures of implemented 3D data service. Fig. 11 (a) shows implemented hardware part that includes the baseband process, multimedia processor, host processor and 3D display. Fig. 11 (b) shows a captured image presenting 3D data service on the implemented 3D T-DMB receiver. As shown in this figure, we confirmed that 3D viewing was successfully generated in specific region on top of video, which was provided by the service mechanism based on MPEG-4 BIFS.

8 RF Tuner 3D Display Baseband Processor Host Processor 3D Object (a) Fig. 11. Implemented 3D T-DMB receiver: (a) hardware part, (b) picture of receiver. (b) 4. CONCLUSIONS In this paper we introduced the 3D data service based on MPEG-4 BIFS technology over T-DMB. Specifically, we presented the implementation of 3D T-DMB receiver that are capable of presenting 3D objects on top of video plain. We also verified the developed 3D T-DMB receiver through the experimental system under various conditions. In this paper we just introduce the 3D data service in T-DMB, which is believed to be very useful as an additional functionality of mobile broadcasting. Moreover, we are currently developing 3D video service over T-DMB, which provides mobile users with more realistic multimedia contents. Therefore, it is expected that various types of 3D services will be available on the mobile broadcasting terminal together with smart touch interface. 5. ACKNOWLEDGMENT We would like to thank the anonymous reviewers whose comments improved this paper. This work is supported by the Ministry of Knowledge Economy and Korea Communications Commission under the title of Development of Next- Generation DTV Core Technology. [1] [2] [3] [4] [5] [6] [7] 6. REFERENCES J. Flack, J. Harrold, G.J. Woodgate, A prototype 3D mobile phone equipped with a next generation autostereosopic display, in: Proceedings of the SPIE Stereoscopic Displays and Virtual Reality Systems XIV, San Jose, CA, USA, J. Flack, J. Harrold, G.J. Woodgate, A prototype 3D mobile phone equipped with a next generation autostereosopic display, Proceedings of the SPIE Stereoscopic Displays and Virtual Reality Systems XIV, San Jose, CA, USA, H. Lee, K. Yun, N. Hur, J. Kim, B.C. Min, J.K. Kim, A structure for 2D/3D mixed service based on terrestrial DMB system, Proceedings of 3DTV Conference, G.S. Lee, K.T. Yang, K.Y. Kim, Y.K. Hahm, C.H. Ahn, and S.I. Lee, Design of Middleware for Interactive Data Services in the Terrestrial DMB, ETRI J., vol. 28, no. 6, Oct. 2006, pp (=>>ETRI. G.S. Lee, S.M. Cho, K.T. Yang, Y.K. Hahm, S.I. Lee, Development of terrestrial DMB transmission system based on Eureka-147 DAB system, IEEE Trans. On Consumer Electronics, vol. 51, Issue 1, 2005, pp ETSI TS v1.1.1, Digital Audio Broadcasting (DAB); DMB video service; User Application Specification, June ITU-T Rec. H.264 ISO/IEC AVC, Advanced Video Coding for Generic Audiovisual Services, Mar