Scalable Codec Architectures for Internet Video-on-Demand

Transcription

1 Scalable Codec Architectures for nternet Video-on-Demand Bernd Girod, Niko Farber, and Uwe Horn Telecommunications Laboratory University of Erlangen-Nuremberg Cauerst. 7,91058 Erlangen, Germany { girod,faerber,uhorn}@nt.e-technik.uni-erlangen.de Abstract The heterogeneous structure of the nternet is a great obstacle for establishing real-time video services. Scalable video codecs, generating bit-streams decodable at different rates, have been proposed to address the heterogeneity problem. n this paper, we will review standard-compliant and non-compliant codec architectures for nternet videoon-demand. For compression based on the H.263 standard, we have developed a compatible architecture that allows to switch between preencoded bit-streams of diferent bitrates. This architecture provides excellent streaming performance for point-to-point communication scenarios that ofer a low delay feedback channel. Then we will present a non-compliantfully scalable video codec based on a spatiotemporal resolution pyramid. This approach can encode embedded lower bit-rate layers at the same overall bit-rate as needed by H.263 single-layer coding and can also support multicasting. The complexity of both schemes is sufjiciently low to allow software-only implementations of nternet video services. This is demonstrated by means of an implemented World Wide Web video server application. 1. ntroduction nformation systems and distributed applications for the nternet show a growing demand for real-time audiovisual services. Real-time audio services can already be found on several World Wide Web (WWW) servers as well as within the nternet MBone tools. For video services the heterogeneous structure of the nternet is still a great obstacle. Reservation protocols alone, e.g. RSVP (resource reservation protocol [ 14]), cannot solve the heterogeneity problem. Consider an nternet video server with sequences encoded at a predetermined bit-rate. f the bit-rate is sufficiently low, nearly everybody connected to the ntemet can view the videos in real-time at the cost of very poor quality. But those who can connect to the server at higher speeds do not bene- fit from higher bandwidth because the sequences have been encoded at a much lower bit-rate. On the other hand, if the sequences were encoded at higher bit-rates, those connecting to the server at lower speeds cannot view the videos in real-time any longer. Storing bit-streams for different bitrates is possible but requires certain precautions to allow switching between different bit-rates during a transmission. Scalable video coding has been proposed to address the above problem. A scalable video coder produces a bitstream, decodable at different bit-rates. t allows computation time and memory limited decoding on less powerful hardware platforms [S, and it can substantially improve the quality and acceptance of video services on the nternet. For video servers, sequences need only to be encoded once. Depending on the available bit-rate, the server selects more or fewer bits from the bit-stream to be sent to the receiver. For broadcast applications a layered multicast scheme can be used. The different bit-rates can be distributed into several layered multicast groups. Picture quality increases with the number of received multicast groups. For videoconferences with several participants, the RSVP protocol can be used for bandwidth allocation over each intermediate link. f necessary, the scalable bit-stream can be pruned easily at routers such that the allocated bandwidth can be utilized in the most efficient way. n this paper we propose two different approaches for ntemet Video-on-Demand and compare their performance with H.263 simulcast. The comparison is based on a simulation scenario with two quality levels which is described in the following section. After comparing the resulting bitrates for a given target PSNR we finally provide simulation results for the transmission over a constant bit-rate channel. 2. Simulation scenario The simulation results in this paper were obtained under the assumption of an nternet Video-on-Demand situation. We require that the encoder can generate two different picture qualities such that in case of network congestion the /98 $ EEE 357

2 higher quality bit-stream can be discarded. We further assume that the bandwidth needed for the transmission of the low bit-rate layer can be guaranteed during the whole transmission. The low quality layer is encoded in QCF resolution at 15 fps, while the high quality layer is encoded in CF resolution at 30 fps. For both layers the target PSNR is 34 db which is achieved with only little variations for all investigated approaches. The sequences are encoded with fixed quantizers and no rate control. f the decoder wants to switch from lower to higher bandwidth during the transmission, the delay until pictures of better quality are displayed is an important design parameter. We require a maximum delay of 320 msec (i.e. every 8 frames) for the simulations in this paper. All results are based on the MPEG-4 test sequence SEAN, frames H.263 simulcast We compare the proposed architectures with simulcasting of two H.263 encoded bit-streams. Since we desire a maximum delay of 320 msec for switching from lower to higher quality, an 1-frame has to be inserted in the CF layer every 8 frames. -frames in the QCF layer are not needed as long as the QCF H.263 decoder keeps decoding the low quality layer even if the higher quality layer is used for display. Note that this requires two running decoders at the receiver. H.263 coding simulations were carried out with mode decision according to TMN-5. No special coding options were used. Fig. 1 illustrates the used simulation model for simulcast and the dependencies of frames caused by prediction. The frame types are '' (intra) and '' (motion compensated, temporal prediction). proach results in a fully scalable codec as proposed in the next section. time [1/30 s] Figure 2. Simulcast: Transmitted bits during a switch from low to high quality. 4. Scalable coding Our approach to scalable video coding is based on a spatio-temporal resolution pyramid. This kind of a multiresolution decomposition was first proposed by Uz and Vetterli in 1991 [12, 131. The simulation model used is shown in Fig. 3 and is very similar to simulcast except for the two additional frame types 'Pst' and 'Ps' which are described below. Pst i Figure 3. Simulation model for the scalable codec. Figure 1. Simulation model for simulcast. Fig. 2 shows the transmitted bits during a switch from low to high quality. t can be seen that -frames in the CF layer require about 8 times as many bits as -frames and therefore significantly contribute to the overall bit-rate. Because both bit-streams are encoded independently, no advantage is taken from the fact that the QCF layer could be used to improve the coding efficiency of the CF layer. This could be achieved by using interpolated frames from the QCF layer to spatially predict frames from the CF layer or by encoding side information more effectively. This ap- A scalable codec superior to simulcasting has to exploit spatio-temporal redundancies of the pyramid decomposition by an efficient compression technique. n our approach we have combined low complexity downsanipling and interpolation filters with highly efficient lattice vector quantization. Decoder complexity is sufficiently low to allow software-only implementations on today's PCs and workstations. Encoder complexity is mainly determined by motion estimation. For -frames, the original frame is successively filtered and downsampled by a simple averaging filter with coefficients (1 1), separately applied in horizontal and vertical direction. The lowest resolution layer is encoded by a DPCM 358

3 technique [l ]. For all other layers a spatial prediction is formed by interpolating the lower resolution layer by a filter with coefficients (1 3 3 l) again applied horizontally and vertically. Spatially predicted frames are denoted Ps' and can also be used for any other type of lower resolution frames, e.g. -frames. n the above simulation model Psframes are used to allow switching from the QCF to the CF layer and are inserted every 8 frames in the CF layer. The residual prediction error quantizer uses an 8- dimensional lattice vector quantizer (LTVQ) as shown in Fig. 4 [6]. We selected the &-lattice since it is the best known lattice quantizer in eight dimensions [3]. Furthermore, in [, 21 fast algorithms for nearest lattice point rounding are given. For encoding, a 2 x 4 block of neighboring samples is mapped into an 8-dimensional vector. This vector is scaled by a factor l/s, s >> 1 where s corresponds to the quantizer step size in one-dimensional quantization. By varying s, the bit-rate of the quantizer can be controlled. The scaled vector, a point in Rs, is rounded to its nearest &-lattice point by using the algorithm given in [l]. From the obtained lattice point an index is computed which then is transmitted to the decoder. The decoder can reconstruct the lattice point from the received index either by computation or by a simple table-lookup. By rescaling the reconstructed lattice point with s, the finally reconstructed 2 x 4 input block is obtained. 1 /s Scaling spatially from the interpolated lower resolution frame of the same temporal reference or temporally from the previous frame of the same resolution. Fig. 5 shows the transmitted bits during a switch from low to high quality. The performance of P-frames in the QCF and CF layer are very similar to H.263. nterestingly the sizes of Pst-frames and -frames in the CF laycr arc almost identical, indicating that no advantage can be taken from the presence of the QCF layer. However, a significant gain can be obtained by replacing the -frames in the CF layer by spatially predicted Ps-frames. The size of those frames is reduced from approximately 80 kbit to 60 kbit respectively. time [ /30 s] Figure 5. Scalable codec: Transmitted bits during a switch from low to high quality. Prediction P 5 Rescaling Recomtructed prediction \ error kmge Table-lookup for most probable vectors Figure 4. Encoding and decoding with lattice vector quantization (LTVQ). Besides -frames we are also using predicted frames in our coding scheme. Motion-compensated prediction is based on 16 x 16 blocks and works within each layer similar to motion-compensated hybrid coders [7, 81. Motion vectors are estimated and coded in a hierarchical way. Motion vectors found for a lower resolution layer are used to predict motion vectors in the next higher layer. Within predicted frames, we distinguish between temporal prediction (frame type ' ' ) and spatial-temporal prediction (frame type ' Pst' ). Spatial-temporal prediction is only possible when a lower resolution frame with the same temporal reference is available. Then each block can be predicted either Based on the scalable coder described above, we have implemented a software-only video server application called NetClip [9] and integrated it into the World Wide Web (WWW). The decoder is available as a Netscape plugin for various Unix hardware platforms [ 101. The HTTP server machine has on its local disk several video clips which have been compressed by our scalable codcc. 5. S-frame architecture n both of the above approaches the periodic transmission of -frames (Ps-frames) significantly effect the ovcrall bit-rate for the high quality bit-stream. Also notc that -frames (Ps-frames) do not provide any additional functionality during normal transmission of the CF layer. n [4] we have therefore proposed a new architecture for the standard compatible storage and transmission of video that avoids the overhead of periodic -frames (Ps-frames). According to this architecture both layers are encoded indcpendently using only the most effective temporally prcdictcd -frames. A switch to the CF layer is possible by dynamically assembling the bit-stream at the vidco scrvcr 359

4 whcncvcr a switch is rcquested from the client. n between the last frame of the QCF layer and the first frame of the CF layer an especially encoded "Switch-frame" (S-frame) is inserted that is also preencoded and stored for each position a switch shall be supported, i.e., every 8 frames. Because this approach is feedback driven, it is only feasible in a point-to-point communication scenario such as nternet video-on-demand. n contrast to the above two approaches it would therefore not be suitable for multicasting or broadcasting. Furthermore it requires a reasonable low delay on the feedback channel to allow a quick response time and interactivity. Under those conditions, however, it provides excellent streaming performance as will be shown below. Fig. 6 shows the assumed simulation model where ' S' indicates the switch-frame. are only transmitted during a switch from the QCF to the CF layer. Therefore their influence on the average transmitted bit-rate is very low. For storage, however, S-frames have to be considered every 8 frames. Therefore we need to distinguish between the effective bit-rate for storage and transmission. Fig. 7 illustrates that during a switch from low to high quality a high peak in bit-rate is encountered. After the switch, however, the S-frame architecture provides excellent streaming performance. + CF 1 time [130 s] Figure 7. S-Frame architcture: Transmitted bits during a switch from low to high quality. Figure 6. Simulation model for S-frame architect u re. The purposc of S-frames is to resynchronize the frame buffers at the encoder and decoder before any following frames of thc CF layer are transmitted. Therefore, instead of encoding the original frames, the reconstructed frames from the CF layer are used for encoding the S-frames. As a reference frame the last reconstructed frame of the QCF layer is used. With this special encoding strategy, the content of thc frame buffer in the CF layer can be approximated with increasing accuracy by decreasing the quantization paramcter Qp. For all simulations in this paper we use Qp=3, which results in a remaining loss of PSNR of less than 0.2 db compared to the case that the CF layer is decoded without any switch. After the switch, the loss of picture quality decreases over time. Note that an S-frame does not require a new frame type but can bc cncodcd as an H.263 compatible -frame. However the change in spatial resolution requircs the use of the optional coding mode ' Rcference Picture Resampling' that is part of a series of new options for H.263 that have recently be adopted by the TU-T under the working term ' H.263+'. Because an S-frame has to approximate the reconstructed frame from thc CF layer very closely, it requires a higher bit-rate than an -frame. On the other hand S-frames 6. Comparison of coding performance Table 1 summarizes the coding performance of the three approaches. Note that the scalable codec can encode both layers at a lower rate than required by H.263 for the CF layer alone. For the S-frame architecture the given rate does not include any S-frames since those are only transmitted during a switch. The rate that is needed to store the bit-streams including all S-frames is provided in brackets. The results indicate that the S-frame architecture is advantageous, if the network bit-rate rather than the storage capacity is the limiting factor. The peak in bit-rate may be a problem for packet networks and may require buffering as described below. Simulcast Scalable S-frame (635.9) Table 1. Summary of coding performance. 360

5 ' ' Sa D - :. v) W 5 - m is mainly achieved by using spatially predicted frames instead of -frames to allow switching from low to high quality. The third approach is limited to point-to-point communication scenarios that offer a low delay feedback channel. A switch from low to high quality is accomplished by inserting especially encoded switch-frames that cause a high peak in bit-rate. After the switch, however, the approach offers excellent streaming performance. The S-frame architecture is fully standard compatible but requires increased storage capacity. "0 lo M) QCF QCF +CF time [1/30 s] Figure 8. Comparison of buffer levels for the transmission over a constant bit-rate channel. n addition to the above results we have simulated the transmission over a constant bit-rate channel. The bit-rate has been chosen in between the low and high video bit-rates, such that switching between the two layers is necessary to avoid buffer over- and underflow. The simulations are based on a channel bit-rate of 200 kbps and a buffer size of 300 kbit. To simulate the transmission of a longer sequence, the encoded frames have been transmitted repeatedly. Therefore -frames are transmitted every 96 frames for all approaches. Fig. 8 shows the simulation results. Because the performance of all approaches in the QCF layer is very similar, the time to empty the buffer after buffer overflow is about the same in all three simulations (white areas). After buffer underflow the high rate is transmitted and CF frames can be displayed as long as the increased rate does not cause buffer overflow. This time period (gray areas) is longest for the S-frame architecture and shortest for simulcasting. 7. Conclusions The heterogeneity of the nternet remains a great obstacle for providing reliable video services. Scalable video codecs can substantially improve the quality of video services for many applications. The three approaches to scalable video coding that have been compared in this paper show very similar performance in the QCF layer where only temporal prediction is used. n the CF layer the proposed scalable codec performs significantly better than H.263 simulcast. Both layers can be encoded at a lower rate than required by H.263 for the CF layer alone. This gain References [] J. H. Conway and N. J. A. Sloane. Fast quantizing and decoding algorithms for lattice quantizers and codes. EEE Trans. on lnformafion Theory, T-28(2): , Mar [2] J. H. Conway and N. J. A. Sloane. A fast encoding method for lattice codes and quantizers. EEE Trans. on lnforniation Theory, T-29(6): , Nov [3] J. H. Conway and N. J. A. Sloane. Sphere Packings, Latiices and Groups. Springer, [4] N. Farber and B. Girod. Robust H.263 compatible video transmission for mobile access to video servers. n Proc. lcp'97, Santa Barbara, USA, Oct [5] B. Girod. Scalable video for multimedia systems. Computers & Graphics, 17(3): , [6] B. Girod, F. Hartung, and U. Hom. Subband image coding. n A. N. Akansu and M. J. Smith, editors, Subband and Wavelei Transforms, chapter 7, pages Kluwer Academic Publishers, Norwell, MA, [7] B. Girod, U. Hom, and B. Belzer. Scalable video coding with multiscale motion compensation and unequal error protection. n Y. Wang, S. Panwar, s.-p. Kim, and H. L. Bertoni, editors, Multimedia Coniniunicutions and Video Coding, pages Plenum Press, New York, Oct [8] U. Hom and B. Girod. Performance analysis of multiscale motion compensation techniques in pyramid coders. n Proc. ClP'96, volume 111, pages , [9] U. Hom, B. Girod, and B. Belzer. Scalable video coding for multimedia applications and robust transmission over wireless channels. n 7th nternational Workshop on Packet Video, pages 43-48, Brisbane, Mar [O] U. Hom and J. Weigert. Netscape pyramid decoder plugin. www4.informatik.uni-erlangen.deiprojects/netclipi, Mar ] A. N. Netravali and B. G. Haskell. Digital Pictures. Plenum Press, New York, M. Uz, M. Vetterli, and D. LeGall. nterpolative multiresolution coding of advanced television with compatible subchannels. EEE Trans. on Circuits and Systenis for Video Technology, ( ):86-99, Mar M. Vetterli and K. Uz. Multiresolution coding techniques for digital television: A review. Multidimensional Systenis and Signal Processing, 3: , [ 141 L. Zhanp, S. Deering, D. Estrin, S. Shenker, and D. Zappala. RSVP: A new resource reservation protocol. EEE Network, 7(5):8-18, Sep