Data Storage Exploration and Bandwidth Analysis for Distributed MPEG-4 Decoding

Transcription

1 Data Storage Exploration and Bandwidth Analysis for Distributed MPEG-4 oding Milan Pastrnak, Peter H. N. de With, Senior Member, IEEE Abstract The low bit-rate profiles of the MPEG-4 standard enable video-streaming for mobile Consumer Electronics (CE) and computing devices. Prediction of the system behavior is a critical issue for the implementation of such applications on a resource-limited (e.g. mobile) platform. The high complexity of the MPEG-4 standard (e.g. the shape-texture decoding) and the resulting dynamic allocation of resources are conflicting with resource limitations. For implementation, we explore a multiprocessor architecture, because it enables an efficient mapping combined with low power consumption. In this paper, we focus on a detailed analysis of the dynamic behavior and bandwidth, using an earlier proposed linear parametrical computation model [1]. The analysis reveals that the actual average bandwidth requirements are about 2.5 times lower than the worst-case estimation. As a consequence of the resulting bandwidth, an additional video stream may be decoded on the average with the same target platform, though control may be needed for worst-case situations 1. Index Terms MPEG-4 coding, shaped video objects, multiprocessor architecture, bandwidth analysis. I. INTRODUCTION The power and cost-efficient implementation of complex multimedia algorithms such as MPEG-4 video coding, requires a careful design process that starts already at an early stage of the system development. The extensive coding tools within the MPEG-4 standard allow a broad area of implementations, in particular low bit-rate video streaming on the Internet and with portable devices. In this paper, we contribute to the analysis of the arbitrary-shaped coding tool of the MPEG-4 standard [2]. First, we focus on the application details that can be explored by an application programmer at an early stage of the product design. Detailed know-how of specific application aspects, such as data-driven complexity and the corresponding memory requirements, may speed-up subsequent SW/HW iterations of the product design process. For example, the predictable memory usage and required bandwidth for the communication between tasks is tightly connected with the application (in our case streaming video). However, a platform-dependent memory organization may 1 Supported by the European Union via the Marie Curie Fellowship program under the project number HPMI-CT Milan Pastrnak is with LogicaCMG Nederland, P.O.Box 7089, NL-5605 JB Eindhoven, The Netherlands. Peter H.N. de With is with Eindhoven University of Technology / LogicaCMG Nederland, NL-5600 MB Eindhoven, The Netherlands. influence the final partitioning of the application. The primary motivation for taking MPEG-4 is that it combines currently used DCT-based coding techniques for video compression with recent algorithms from the computer environment, like 2D and 3D graphics. We have explored the arbitrary-shaped video object (AS-VO) decoding tool, since it contains both features. This tool potentially serves many applications, such as surveillance systems, videophones, multimedia games, etc. Based on earlier work we concluded that a single processor system cannot cope efficiently with the computational complexity of such an advanced application. Therefore, we assume a multiprocessor System-on-Chip (SoC) as the target platform for our system. The purpose of this paper is to study the efficient implementation of AS-VO decoding in MPEG-4. In particular, we focus on the required bandwidth and memory usage. In distributed computing, the bandwidth analysis and the required memory usage are key issues that have to be known at the compile-time design phase. The difference between worst-case and average-case requirements may be exploited for increasing cost efficiency. The outline of this paper is as follows. In Section II, the characteristics of the target application and platform are described. Section III introduces the computation graph that models the application. The problem statement is addressed in Section IV. In Section V, the estimation technique coping with the dynamic change of the resource usage is presented. The parametrical model for the bandwidth estimation is presented in Section VI. Finally, in Section VII we present the conclusions. II. AS-VO DECODING ON A MULTIPROCESSOR SYSTEM-ON- CHIP For realizing AS-VO within MPEG-4 coding, we focused on the coding tools in the standard that offer the arbitraryshaped video object decoding. In MPEG-4, every video object (VO) is represented in several information layers, with the Video Object Plane (VOP) at the base layer. This video object plane is a rectangular frame containing hierarchically lower units, called macroblocks (MB). A macroblock is the smallest data unit of a video object and covers 16 x 16 pixels. Fig. 1 depicts a typical composition of MBs within a VOP, representing an arbitrary-shaped video object. It can be noticed from Fig. 1 that an arbitrary-shaped VO (sometimes referred to as non-rectangular VO) is also covered with a grid /04/$ IEEE. 67

2 of macroblocks, just like the usual rectangular video frames. The standard distinguishes three types of macroblocks: boundary, opaque, and transparent. The boundary macroblock is defined as a macroblock that has transparent pixels and opaque pixels. Additionally to texture information, a compressed boundary MB contains coded shape information. The shape information is represented by a so-called Binary Alpha Block (BAB) of 16 x 16 elements. The texture information is represented by six 8 x 8 blocks in YCrCb color space (4:2:0). Transparent MB BAB macroblocks within one Video Object Plane. Each macroblock starts with the shape information, followed by the texture data. Within the shape-decoding process, we partitioned the decoding into the following subtasks: MacroBlock-type oding (MBtype ), Shape Motion Compensation (Shape), Context Arithmetic oding (CAD), and the Context Block Positioning (CBP) decoding. The texture decoding comprises five steps: Motion vector oding (MvD), IDCT coefficients oding (Coeff ), Texture Motion Compensation (Texture), Inverse Quantization (IQ) and Inverse Discrete Cosine Transformation (IDCT). Video Object Opaque MB Boundary MB 4 IDCT 8x8 blocks (Y) Fig. 1. Example of the macroblock classification for an arbitrary-shaped video object. MBtype CAD CBP Shape shape Shape oding Processing tile I-mem Proc. (ARM) D-mem NI CA Processing tile Processing tile Processing tile MvD Coeff Texture IQ IDCT texture Texture oding FIFOs Fig. 3. Computation graph of a motion-compensated AS-VO decoder. router Network-on-chip Fig. 2. Multiprocessor architecture template. router A possible multiprocessor System-on-Chip (SoC) platform is depicted in Figure 2. The indicated processing tiles represent small self-contained embedded computers which are connected with other tiles via a Network-on-Chip (NoC). The processor in a tile is connected to the NoC via a Network Interface (NI). The transfer of data between the local memory and the network interface is performed by a Communication Assist (CA). The NI units are connected to the routers of the communication network. Network links connect the routers in the desired network topology. III. COMPUTATIONAL MODEL To express the multiprocessor-level parallelism in our model, we use Synchronous Data-Flow (SDF) graphs. Figure 3 shows a computation graph for arbitrary-shaped video object decoding. The depicted graph represents the macroblock decoder as a loop through the number of all For each subtask, the parametrical computation time is estimated. We assume that the computation time can be estimated accurately using a linear model on the parameters p k,i for i = 1, 2,, by t i (j) = c 0,1 + c 1,i.p 2,i (j) + c 1,i.p 2,i (j). (1) The detailed description for each subtask is given in [1]. For example, for one complete Video-Object, the parametrical model of the CAD subtask is t CAD = k k.µ +190 N bits N bits N bits3. It takes kilocycles (indicated by a k) to initialize the task. If the MB is a boundary block, we assign µ = 1, otherwise we assign µ = 0. For µ = 1, the amount of kilocycles have to be spent on decoding the shape, plus cycles for decoding of each bit contained in the arithmetic code. 68

3 IV. PROBLEM STATEMENT The rapid increase of the number of multimedia applications for mobile CE devices is posing new requirements on the system design phase. The applications are more complex and have an increased dynamic behavior. However, the currently used design approaches are tuned towards worst-case processing. The disparity between average and worst-case processing gradually becomes unacceptable for the design of resource-constrained devices. An optimal resource utilization following from a careful distribution of the computations usually conflicts with the usage of communication resources. Simulation models can be used to demonstrate that the system is optimally designed for a particular set of input signals. However, a reliable reasoning about arbitrary input signals requires analytical methods, covering on one hand processor utilization and on the other hand communication and storage optimizations. The balancing of several simultaneously running applications needs to be supported by each application, for example with an estimation of future resource requirements. When focusing on streaming applications, behavior prediction can be supported by the most important coding parameters of the audio / video applications in their packet headers. Let us now introduce an architecture that supports the estimation for AS-VO decoding. The resource estimation is based on the statistical characteristics of input data and the estimation error between the parametrical model and the actual resource usage of previous VOPs. Our previous work on the subtasks timing behavior has revealed that the most important parameters of an MPEG-4 encoded AS-VO are its size and the proportions between different macroblock types. The size of the object is encoded in the VOP header, but the proportion of macroblock types is known only after the decoding of the whole VOP. Fig. 5 illustrates the dynamism in the size change for different test sequences. It is shown that objects can have a constant size, or they are varying in size over time. Video input Predictive VOP header parser Header parameters Resource estimator VOP data Parameters & Estimation difference VOP decoder Resource budget Run-time Resource Manager oded VOPs Parameters Fig. 4. oder architecture with a resource estimator. V. ESTIMATION TECHNIQUE Performance robustness of a system is generally based on the detailed knowledge of the platform resources and the application behavior. To achieve a predictable behavior of applications, analytical methods are developed. For example, the linear timing functions of Section III are based on input data characteristics. However, the exact estimation of resource usage is not possible due to many reasons. The primary reason, which we consider also as the most limiting one, is that only a (very) limited subset of coding parameters composing the timing functions is encoded in the VOP headers. To decrease the uncertainty of the application behavior, we introduce an architecture using predictive parsing of the VOP header information (see Figure 4). For this purpose, we concentrate on a few important architectural aspects of the proposed AS-VO decoder system. The predictive parser of VOP headers is performed N VOPs in advance, so that the platform can cope with the increase demand on resources. The subset of available parameters can vary between very small, e.g. the size of a VOP, or much larger, such as e.g. the full set of timing parameters as used by a complete encodingdecoding chain. An estimation of the required resources combines the available parameters from VOP headers with the complete set of timing parameters available after the VOP decoding. Macroblocks VOP size change VOP index Fig. 5. Variable size of video objects expressed in macroblocks as a function of succeeding VOP indexes. The resource estimator combines the size and proportion parameters using the knowledge of the actual VOP size and the average values of the proportion of macroblock types in the previous VOP. The proportion of macroblock types is initially set to the average values for the transparent, opaque and boundary blocks, respectively. These values are replaced with the actual proportion values after decoding the first VOP. The deviation of the proportion due to object resizing is negligible for the used test sequences. Without loss of generalization, we implicitly assume that the resizing of the object only smoothly changes over time, while preserving the 69

4 proportion parameters. Hence, we formulate our estimation function E to be dependent on the actual size (S) and the macroblock types proportions (P) of the VOP at time t-1 by E = f (P t-1, S t ). (2) The function f is further detailed for the communication resources in Section VII. Proportion of macroblock types The resource estimator computes the VOP based on estimations of the resource usage from the linear equations. In the following section we provide such a linear model for the communication resources for each connection within our computation graph presented in Fig. 3. TABLE 1. DATA INDEPENDENT CONNECTIONS OF AS-VO COMPUTATION GRAPH. Start connection End connection Bandwidth (kbyte / s) Macroblocks 100% 80% 60% 40% Boundary Opaque Transparent MBtype CAD MBtype Shape 9.6 CAD CBP 3.75 CBP MvD 5.62 MvD Coeff % 0% VOP index Fig. 6. Example of the varying proportions of different macroblock types within a VOP. For our target system, we propose a scenario-based resource allocation [3]. For this method, we have to provide several scenarios (sometimes also called modes) in which the application can be executed. A mode depends on some input parameters, like in our case the video-object size. The switching of the modes usually evokes the allocation of extra resources or the release of some resources. In our particular case, we have high interest in predicting such a switch in advance and reasoning on a possible dynamical reconfiguration of the platform over time. Using Equation (2), we can express the minimum number γ of VOP headers that should be parsed in advance as a ratio between the time for switching modes and the frame period (usually 1/30 Hz), hence γ = nint( T switch(m,n) / T frame rate ) (3) The above-described predictive parser, in combination with the estimation technique, increases the predictability of the resource usage of the system. A serious limitation of this estimation model is the unpredictability of the worst-case switching time between two modes. However, we have found that even information on the average-case switching time, based on the simulation results, decreases the uncertainty of the reconfiguration of the platform. VI. BANDWIDTH ANALYSIS In this section, we focus on the detailed analysis of the bandwidth allocation for in-between tasks communication. Based on the bandwidth analysis, our task model can be repartitioned. For example, if there is a high communication bandwidth is required and the computation load is not a limiting factor, we can decide to map two tasks on the same processor. This decision may save a significant amount of communication between processing tiles. However, the optimal balancing of all main resources (processing, memory and communication) remains still a critical task for the system designer. Starting with the proposed computation graph, we calculate the worst-case estimation for our model. We assume that in the worst-case, all macroblocks within a VOP are boundary macroblocks. The second assumption refers to coded texture data: all texture blocks are coded with maximum-valued coefficients (64), to handle even nonrealistic situations. Our analysis reveals that we can distinguish the types of connection in the graph. The first class represents the inputdata dependent communications. The second class covers the communication of data that are not directly depending on the input data. These types of connections transfer the implementation-specific data and usually cover a small percentage of the overall communication. Table 1 covers the second class of connections. For example, the task MBtype communicates to the task CAD for every macroblock its type and the position in the input bit stream. This holds also for the Shape task, but now some extra information (e.g. reference position of the macroblock) is required. These connections have very static characteristics, and because they represent just 1.8% of the overall communication, we focus on the second class of connections. 70

5 MBtype CAD CBP MvD Coeff 9,6 Shape 491,5 7,5 Texture IQ 491,5 shape IDCT texture MBtype CAD CBP MvD Coeff 9,6 Shape 164,2 4,5 38 Texture a) worst-case estimation b) real execution IQ 165,9 shape 393,6 430 IDCT texture Fig. 7. Example of the required bandwidth (kbyte/s) for AS-VO decoding 30Hz). In the next step, we provide the model for the bandwidth computation based on Equation (2), for input data dependent communications. For better understanding, let us describe in detail the connection between tasks Shape and CAD. If the BAB macroblock is motion compensated (see for more details in [2]), the Shape task has to provide the reference BAB for the CAD task. Intrinsically, if the current macroblock is of the boundary type, the decoded shape has to be stored in the shape buffer for later rendering with the other video objects. Fig. 7 illustrates an exemplary overview of analytical worstcase bandwidth and measured values from our software implementation of the decoder. The parametrical model for each connection can be described by the linear parametrical equation TABLE 2. PARAMETRICAL FUNCTIONS OF DATA GRAPH. Start connection End connection Parametrical function (kbyte / s) Shape CAD ξ CAD Shape 256.Ψ MvD Texture 4.ω Texture IDCT ω Coeff IQ 14.θ.χ IQ IDCT 384.χ IDCT Texture 384.χ Parameter TABLE 3. PARAMETER DESCRIPTION. Description ξ 1, if the macroblock shape is motion compensated; 0, otherwise. Ψ 1, if the macroblock is boundary type; 0, otherwise. ω 1, if the texture is motion compensated; 0, otherwise. θ Number of non-transparent sub-blocks in the macroblock, hence θ 6. χ 1, if the macroblock is not transparent; 0, otherwise. 430 b i (j) = n 0,1 + n 1,i.l 2,i (j) + n 1,i.l 2,i (j). (4) We can model both types of connections as they are classified above. The data independent connections are modeled without the parametrical part. For example, if the j-th macroblock contains encoded shape information and belongs to the Inter type of VOP (ξ = 1), the required communicated data between the tasks Shape and CAD is equal to b Shape,CAD (j) = ξ [Bytes]. When knowing the proportion between macroblocks within one VOP, we obtain for the complete VOP the expression b Shape,CAD = ( ξ ). P boundary, (5) where P boundary denotes the percentage of the boundary macroblocks. In order to find the prediction of the required communication bandwidth for the next VOP, we use an estimation function similar to Equation (5) for every connection and multiply it by the number of macroblocks inside the VOP. Table 2 provides details of these equations for the task connections. The detailed description of the parameters is given in Table 3. To evaluate the correctness of the estimation, using the statistical properties of the decoded input data and the new size of the object, we introduce an estimation-error function. This function is evaluated after each VOP, based on the estimated and real requirements of the AS-VO decoder. The final results are portrayed by Fig. 8. The simulations have shown that our estimation technique for the bandwidth requirements was consistently 4.7% too low. If we add a threshold of 5% to the estimator, the requested bandwidth fulfills the demand on the communication resources for all VOPs Estimation Estimation with t reshold Worst-case estimation Fig. 8. Example of several estimation functions compared to the worstcase estimation. 71

6 VII. CONCLUSIONS We have studied the dynamic bandwidth usage of arbitraryshaped video-object decoding of the MPEG-4 standard. The presented research concentrates on the MPEG-4 Core Profile, Level 2. As a target platform for mapping, we have assumed a Multiprocessor Network-on-Chip. We have extended an earlier developed task-level model, for obtaining more details on the required memory bandwidth, both for actual and worstcase performance. A linear timing model was employed based on earlier work, which was extended for a detailed communication analysis, in which the communication paths in the computation graph were classified individually. Estimation functions were presented exploiting the coding parameters, so that the bandwidth requirements can be better predicted. We have found a factor of 2.5 between the worst-case and the actual measured bandwidth in our experiments. Besides these numerical findings, the most important result of this study is that it enables an accurate prediction of the platform operation. The proposed model for the estimation, combined with the concept of predictive parsing, decreases the demand for the allocation of the communication bandwidth in a ratio that is sufficient for the decoding of a secondary average video sequence on the same platform. This template is applicable to any streaming application with dynamic usage of resources. REFERENCES [1] M. Pastrnak, P. Poplavko, P.H.N. de With, J. van Meerbergen, On Resource Estimation of MPEG-4 Video oding for A Multiprocessor Architecture, Proceedings of PROGRESS 2003, Workshop on Embedded Systems, Veldhoven, The Netherlands, October [2] ISO/IEC :199/amd.1:2000, Coding of Audio-Visual Objects Part 2:Visual, Amendment 1: Visual Extensions, Maui, ember [3] P. Yang et al., Managing Dynamic Concurrent Tasks in Embedded Real-Time Multimedia Systems, ACM Proceedings ISSS 02, pp ,