Compression and File Formats 1 Compressing Moving Images Methods: Motion JPEG, Cinepak, Indeo, MPEG Known as CODECs compression / decompression algorithms hardware and software implementations symmetrical / asymmetrical Moving versus still images: Temporal as well as spatial compression Data transfer rates are critical How much data is needed to maintain 24 frames/sec? Chapman & Chapman, Chapter 10 2 1
Spatial Compression Compress each frame (still image) independently - also known as intra-frame compression Animated GIFs single file containing multiple images compressed using LZW compression only suitable for simple, cartoon-like animations Motion JPEG (MJPEG) compresses each frame using JPEG suitable for videos of real-world action implemented in hardware for real-time capture and compression of video no standard file format! (MJPEG-A emerging) 3 Temporal Compression Also, known as inter-frame compression - relies on only small changes between frames: 4 2
Temporal Compression Also Basic idea is this: Certain frames designated as key frames e.g. every sixth frame (may be user-settable) Key frames either are left uncompressed or only spatially compressed Each frame between key frames is replaced by a difference frame records only differences between frame and preceding frame or preceding key frame Difference frames readily compressed with RLE or dictionary-based schemes 5 Software CODECs for Multimedia Multimedia is likely delivered on CD-ROM or over the Internet Low to very low data transfer rates High compression required Software implementations of a variety of codecs Cinepak Intel Indeo MPEG-1 6 3
Cinepak and Indeo Similar codec methods that use vector quantization Each frame divided into small rectangular blocks of pixels (the vectors) Use a code book consisting of typical vectors (representing typical patterns that might occur in a image, e.g.) areas of flat colour sharp or soft edges different textures (the dictionary of still image compression) Vector Quantization is the process of matching each image vector with its closest vector in the code book Each image vector is replaced by its index in the code book Image can be similarly re-constructed from these indexes and the codebook - de-compression efficient 7 Cinepak and Indeo (2) Highly asymmetric: compression takes much longer than decompression 150:1 for Cinepak Also use temporal compression based on key and difference frames full-motion full screen playback cannot be achieved with these codecs Good enough for 320x240 pixels at 12 frames/sec not great viewing! New Sorenson codec does a bit better (can compress video to give a frame-rate of 50 kbytes/s) 8 4
MPEG-1 Several international standards for compressed digital video are grouped together under the name, MPEG Motion Picture Experts Group Defines a data stream syntax and a decompressor Compression is left to the software developer competitive advantage in the marketplace MPEG-1 Standard does define compression implicitly So we can describe MPEG-1 compression which combines: Spatial compression similar to JPEG, and Temporal compression via difference frames and motion compensation. 9 Temporal Compression in MPEG-1 Naïve approach subtracts each pixel in current frame from corresponding pixel in previous frame result is 0 for pixels that do not change between frames difference frames will contain large numbers of zeros and so will compress well, using e.g. RLE BUT can possibly do better when there is large movement between frames pictures usually composed of whole objects (e.g. a walking person) that move between frames Difference frame would be better if we could match pixels from the moving object between frames Motion compensation (if we can identify coherent area of change corresponding to object, then only need to record object s movement together with changed pixels in the area) 10 5
Motion Compensation Motion compensation in MPEG does not attempt to identify objects in a scene hard computer vision / artificial intelligence problem!! Divides each frame into 16 by 16 pixel blocks, known as macroblocks different from the smaller blocks used in spatial compression Attempts to predict where the corresponding macroblock is in the next frame simple searching in a limited area around the macroblock (all possible displacements within a limited range are tried and best best match chosen) Difference frame is then constructed by subtracting matching macroblocks (i.e. each macroblock from its predicted counterpart in next frame) BUT also need to store displacement vectors (describing predicted displacement of macroblocks between frames)! 11 Frame Types in MPEG Temporal compression has to start somewhere (e.g. Keyframe) Key frames called I-pictures (I for intra) Difference frames based on previous frames called P- pictures (predictive pictures) MPEG also allows difference frames (motion compensation) based on later frames: B-pictures (bi-directional prediction) A video clip can be encoded in compressed form as a sequence of I-, P- and B- pictures MPEG Display order: IBBPBBIBBPBBI Encoders normally use a repeating sequence, called Group of Pictures (GOP): IBBPBB (always starts with Key frame) Display order is converted into a Bitstream order (for transmission): e.g. IPBBIBBPBBIBB Decoder needs to reconstruct display order from bitstream 12 6
MPEG Variations MPEG-1: Delivers 1.2Mbps of video and 250Kbps of 2- channel audio from CD-ROM best for multimedia applications However, MPEG-1 cannot handle interlacing or HDTV formats, hence need for MPEG-2 for broadcasting & studio work MPEG-2: Needs higher data rates of 3-15Mbps, but delivers better picture quality, interlaced video formats and multichannel audio designed for broadcast digital video (e.g. DVD) MPEG-4: emerging standard offering data rates from 5Kbps to 4Mbytesps, with interactive multimedia as well as video 13 New and Future CODECs The challenge of sending video over the Internet has led to much research into new compression methods very low bitrates required (e.g. 28kbps over a phoneline) More sophisticated motion estimation Wavelets, fractals etc H.261 and H.263 designed for video conferencing use multiple compression techniques better than nothing performance over a phoneline 14 7
File Formats for Video Like file formats for still images, file formats for video provide a packaging that includes details of the contents of the file, such as the codec used Various formats, such as... Microsoft s AVI QuickTime MPEG not really a file format, just defines a bit (data) stream (like JPEG) 15 Video for Windows AVI: Audio and Video Interleaved Video and synchronized audio in a small window at around 15 frames-per-second without specialized hardware Video is interleaved with the audio Variety of codecs (identified by 4-byte code, FOURCC): Cinepak (CVID) Intel Indeo Microsoft Video Microsoft RLE (MRLE) others 16 8
QuickTime De facto standard (as of version 3.0 in 1998) QuickTime (QT) movie (.mov ) is a framework for organizing and manipulating the data that make up the actual video frames many video codecs supported can contain media other than video media not necessarily stored in the.mov file A movie plays according to a time base (which records rate at which it should be played back & its current position) frames may be dropped to keep in time enables constant playback speed on any system maintains synchronization between video and audio 17 QuickTime Success of QT is due to its Component-based architecture Plug in new components (small programs conforming to certain interface conventions) to deal with new formats or provide new operations Components include: Compressors (for implementing e.g. Cinepak, Indeo) Sequence grabbers: digitize video Movie player component is used to provide a user-interface for playing movies: will handle AVI and MPEG Transcoders: translate between data formats that use the same compression algorithm 18 9
Streamed Video Playback as it arrives over the Internet Current method of delivery on WWW: Progressive download or HTTP streaming download movie to hard disk start playing back when playback time of downloaded content equals download time of remainder of movie still a large delay before playback begins and not suitable for live video True streaming never stored on user s disk must arrive at playback rate (not possible on dial-up internet connections using V90 modems or basic rate ISDN) different movie formats depending on connection speed Streaming QuickTime and RealVideo 19 File Formats for Animation Macromedia s Shockwave Flash Native file format for Flash movies (and Director) Vector images and animations Metafile: can incorporate bitmap images and sound De facto standard for simple animation on the Web Browser plugins freely available VRML Virtual Reality Modelling Language 3D interactive worlds Browser plugins freely available 20 10
Shockwave Flash (SWF) File contains two broad classes of items Definitions: dictionary of symbols that appear in the animation Control items: instructions to place, remove or move a symbol movement specified using transformation matrices Rather like a program (cf. FIG, POVRay, VRML) stored in compressed binary format: SMALL! Flash creates simple animations Director creates more complicated movies 21 VRML Code directly, or output from 3D modelling package Interactive 3D worlds: 22 11
VRML Example #VRML V1.0 ascii Separator { DirectionalLight { direction 0 0-1 PerspectiveCamera { position -8.6 2.1 5.6 orientation -0.1352-0.9831-0.1233 1.1417 focaldistance 10.84 Separator { # The red sphere Material { diffusecolor 1 0 0 # Red Translation { translation 3 0 1 Sphere { radius 2.3 Separator { # The blue cube Material { diffusecolor 0 0 1 # Blue Transform { translation -2.4.2 1 rotation 0 1 1.9 Cube { 23 End of Lecture Exam revision/hints for my part of the course during tutorials next week! Good luck with all your exams! 24 12