Multimedia What is multimedia? Media types + Text +Graphics +Audio +Image +Video Interchange formats Petri Vuorimaa 1
What is multimedia? Multimedia = many media User interaction = interactivity Script = time Petri Vuorimaa 2
Most common media types Text Graphics Audio Image Video Petri Vuorimaa 3
Continuous media Animations (virtual reality) Audio Video What is the difference between normal and continuous media? Petri Vuorimaa 4
Continuous media processing Capture Preprocessing A/Dconversions Compression Transfer Play D/Aconversion Postprocessing Decompression Petri Vuorimaa 5
Interactivity User can control the progress of the presentation Level of interaction + user interface, application, server, content producer Amount of interaction + www page, www application, video, game, virtual reality Petri Vuorimaa 6
Time Media elements are placed in time dimension Different components are synchronized Presentation system takes cares of synchronization (orchestration) Petri Vuorimaa 7
Applications Multimedia presentations +PowerPoint Multimedia communications + video conference Multimedia services + video on demand Integrated hybrid applications + record store Petri Vuorimaa 8
Text ASCII Text documents + Microsoft Word, Adobe Acrobat Structured documents + SGML, HTML, XML Hypertext +HyperCard Petri Vuorimaa 9
Graphics Bitmap graphics + paintings + Microsoft Paint Vector graphics + drawings +OpenGL + Postscript Petri Vuorimaa 10
Physical properties of sound Amplitude + db = 20 log 10 (A/B) + hearing threshold 0 db, pain limit 100-120 db Period / frequency +Hz = 1/s + Hearing band 20 Hz - 20 khz Petri Vuorimaa 11
Pulse code modulation Samples are taken at sample frequency Sample frequency has to be at least twice of maximum frequency (so called Nyquist frequency) Common sample frequencies are 8, 44.1, and 48 khz Petri Vuorimaa 12
Pulse code modulation (cont.) Signal amplitude at sample moment is converted into numerical value + Pulse Code Modulation (PCM) Sampling causes a quantization error Petri Vuorimaa 13
Frequency transformation Natural sounds are composed of base frequency and its harmonic frequencies Thus, sound can be presented also in frequency dimension Petri Vuorimaa 14
Clarinet sound Petri Vuorimaa 15
Frequency transformation Petri Vuorimaa 16
Frequency transformation (cont.) The Fourier transformation coefficients present the sound in frequency dimension Stationary signals can be presented accurately with Fourier transformation With dynamic signals discrete Fourier transformation has to be used Usually, fast Fast Fourier Transformation (FFT) algorithm is used Petri Vuorimaa 17
Psycho acoustics The properties of human hearing should be considered in coding of sound Sound should be processed in frequency dimension For example, hearing threshold depends on frequency Ear is sensitive to the valleys and hills of spectrum + for example, detection of vowels Petri Vuorimaa 18
Threshold of hearing Petri Vuorimaa 19
Psycho acoustics (cont.) Amplitude So called masking effect A loud sound at certain frequency raises the threshold of hearing at wider frequency area Frequency Petri Vuorimaa 20
Compression methods For example, masking effect can be used in sound coding Signal is divided into frequency bands, which are coded separately (subband coding) For example, Mini Disc records (Sony), DCC Cassettes (Philips), and MPEG audio Petri Vuorimaa 21
Begin Subband analysis Scale factor calculation Coding of scale factors FFT analysis Calculation of masking and required bit allocation Determination of nontransmitted subbands Adjustment to fixed bit-rate Quantization of samples Coding of samples Coding of bit allocation Formatting and transmission End
Image and video coding Methods are either lossy or lossless Most common lossy method is Discrete Cosine Transformation (DCT) For example, Huffmann method is lossless Petri Vuorimaa 23
Coding methods Image coding + JPEG (Joint Photographic Expert Group) Video coding + H.261, H.263 + MPEG (Motion Picture Expert Group) Coding methods utilize usually several coding techniques Petri Vuorimaa 24
JPEG Goals Compression ratio vs. image quality can be selected Works with all kinds of images Both software and hardware implementation Four modes: + sequential coding (original order) + progressive coding (multiphase coding) + lossless coding (perfect copy) + hierarchical coding (multiple resolutions) Petri Vuorimaa 25
JPEG Architectures Lossy modes use DCT for 8 x 8 pixel blocks Sequential mode outputs the DCT-coefficients block by block Progressive mode outputs the DCT-coefficient in groups Hierarchical mode encodes several resolutions at the same time Petri Vuorimaa 26
Sequential JPEG Petri Vuorimaa 27
Progressive JPEG Petri Vuorimaa 28
Hierarchical JPEG Petri Vuorimaa 29
Lossless JPEG Petri Vuorimaa 30
DCT and quantization The DCT-coefficients can be represented as a matrix The quantization is done according to a quantization table The coefficients are put in Zig-Zag order This places the zero coefficients in the end of the run Finally Run-Length coding eliminates the zeros Petri Vuorimaa 31
DCT coding Petri Vuorimaa 32
Statistical coding Uses either Huffman or arithmetic coding Huffman coding requires a separate table Arithmetic coding does not require a table, but need more computation In addition, the compression ratio of arithmetic coding is 5-10 % better Petri Vuorimaa 33
Lossless encoding Lossless encoding utilizes prediction Seven different alternatives + how many and which pixels are used Predictive encoding can reach compression ratio of 2:1 Petri Vuorimaa 34
Efficiency 0,25-0,5 bpp: reasonable - good quality 0,5-0,75 bpp: good - very good quality 0,75-1,5 bpp: very good quality 1,5-2,00 bpp: same as original Petri Vuorimaa 35
MPEG Resembles JPEG method In addition, properties of moving images have been used + following images are very similar + images have few moving objects + image series change seldom Implementation is more complicated Requires usually special hardware Petri Vuorimaa 36
Motion estimation Moving objects are searched from following images Motion estimation are calculated for the changing blocks Estimations are called motion vectors and they are transmitted as part of the coded information Petri Vuorimaa 37
Difference images Difference images are calculated between the original and predicted images Only changed (non-zero) areas are transmitted MPEG utilizes both forward an backward prediction: + I = Intra - original image + P = forward Prediction + B = Bidirection prediction Petri Vuorimaa 39
MPEG image sequence Petri Vuorimaa 40
Interchange formats The co-operation and transfer of multimedia applications requires common interchange formats Interchange format defines: + time, place, structure, and operation (procedures) Without an interchange format content produced with one application cannot be read and used with other application Conversion tools are not a good solution Petri Vuorimaa 41
Application areas A common interchange format can be used for + storage format (e.g., Macromedia Director) + transfer format (e.g., CD-ROM) + real-time transfer format (e.g., digital TV) + content transfer between applications (e.g., group work) Petri Vuorimaa 42
Requirements Data model: + time, synchronization, different formats, pointers, links, interactivity Scripts: + programming language or graphical programming Capacity: + definitions should not take much space Retrieval time: + decompression has to be fast, progressive resolution, etc. Interdependency: + hardware and platform neutral Scalability: + new formats, attributes, etc. Petri Vuorimaa 43
MHEG ISO working group Object based multimedia and hypermedia interchange format Supports interactivity and real-time transfer Defines final presentation format Petri Vuorimaa 44
Properties Defines a set of platform independent components for interactive applications Two ways to implement interactivity: + components are linked to events + scripting language is used Events are caused by timers or user s actions Petri Vuorimaa 45
Properties (cont.) Defines both temporal and spatial location Macros allow the definition of complex objects based on models Contains also support for real-time transfer Petri Vuorimaa 46
MHEG class hierarchy MH-object Behavior Action Link Script Component Content Interaction Selection Modification Composite Descriptor Macro Macro Definition Macro Use Petri Vuorimaa 47
HTML HTML is also an interchange format Limited features HTML = structural document, links + DOM model is coming ECMAScript (JavaScript) = interactivity Style sheets (Cascading Style Sheets) = placement Synchronization is still missing + Synchronized Multimedia Integration Language (SMIL) Petri Vuorimaa 48