Multimedia Application An (usually) interactive piece of software which communicates to the user using several media e.g Text, graphics (illustrations, photos), audio (music, sounds), animation and video. Multimedia Applications are developed with ADITDEM Implementation is using presentation software or multimedia authoring software. Web pages in HTML can be implemented using a text editor e.g. (Notepad) or WYSIWYG (What you see is what you get) software (e.g. FrontPage which allows both). Text Editor WYSIWYG Advantages Text editing software is simple, placing few demands on the computer hardware Provides greater control over the application Allows for easier editing of the application Disadvantages Output can be difficult to predict or calculate Knowledge of programming or textbased commands is required More complex software places greater demands on the computer hardware Authoring Software provides the ability for the developer to customise the application using a variety of methods, e.g. using program code or scripting, or using an icon-based method to set up complex properties without having to learn programming code. Presentation Software usually doesn t have programming features and is still very limited when compared to proper multimedia authoring software. Multimedia data such as graphics or sounds can either be in separate files from the application e.g. in web pages or can be embedded in the application file e.g. in PowerPoint. Embedding data makes it more secure and easy to keep control of files but makes the application file very large. All the embedded data and application would usually have to be downloaded before the application starts to run. It is often easier to update the individual files if not embedded Streamed data data broadcast over a network. The entire file does not need to be downloaded before viewing or listening to it. Codecs The structure of multimedia data files varies. A codec is programming code to encode the data or decode it it may be part of a program or a plug-in. Usually codecs compress or decompress the multimedia data Container file multimedia files can often contain different types of data or it may be compressed in different ways. Container files allow this by having a header which holds information about the type of data and the codec used. An example is RIFF files (see later). Bitmapped Graphics Input Hardware for capturing graphic data CCD Charge Coupled Device a grid of sensors which detect light levels. They can be used to detect the levels of red, green and blue light. In a scanner the sensors are arranged linearly; in a camera there is a 2D array. Resolution is determined by the number of sensors and hence the number of pixels saved e.g. a camera could have 3 Mega pixels and a scanner scan at 4800 dots per inch. 1
Storing Graphic Data RGB data pixel s data can be stored as a 24 bit code (8bits each Red, Green Blue). Colour Depth gives the number of colours available. E.g. a colour depth of 8 bits per pixel allows 2 8 = 256 different colours CLUT (Colour Look Up Table) or indexed colour instead of saving 24 bits for each pixel, a colour lookup table can be used so that for example with 4 bit colour, 4 bits are saved for each pixel. The 4 bit code would be the index to a lookup table with 16 (2 4 ) different codes linked to 16 different RGB colours. RLE Compression (Run Length Encoding) a lossless compression method. The compressed data consists of the colour index and the number of pixels of that colour. This works best for solid blocks of colour. Gif (Graphics Interchange Format) Efficient format for storing images to be transferred using the Internet. Compresses using LZW (Lempel, Ziv, Welch) lossless compression which uses a string table of data in a similar way to a CLUT. However the algorithm is patented. 8 bit colour Supports transparency Supports animation Supports interlacing (interleaving) ( e.g. showing every 8 th row, every 4 th row..) Good for blocks of colour, poor for photos. Jpeg (Joint Photographic Experts Group) designed for compressing full colour or greyscale real world scenes such as photographs. Stores data as 24bit as does not use a CLUT. Uses lossy compression the degree loss can be varied by the user, trading file size against image quality. Loses data e.g. small colour changes, which are perceived less well by humans. Can cause artefacts marks in the image. After data has been removed, file is compressed further by a lossless compression. Effective for large files Full colour or greyscale Does not support transparency Does not support animation but there is the mpeg format for this. Does not support interlacing PNG (Portable Network Graphic) designed as an improvement to GIF and with a patent free compression method. Compression similar to GIF lossless Allows a range of colour depths. Depths below 24bit use CLUTs. Supports transparency and partial transparency Does not support animation but there is MNG for this Supports interlacing Not yet supported by all browsers. Dithering simulates extra colours by mixing pixels of some of the allowed colours close together. This can reduce banding. Anti-aliasing Simulates extra resolution by adding extra in-between colours along sharp edges to reduce the stepping effect (jaggies). 2
Increasing resolution If an image is captured with a low colour depth or low resolution, increasing the colour depth or resolution afterwards will have no effect. The software does not know what colours or data is missing. Resampling is an attempt to increase resolution to reduce the effects of pixellation by estimating the missing data e.g. by calculating the colour of the new pixel by giving it the average colour of its 4 neighbours (bicubic method). Calculations Width = w pixels number of pixels given Height = h pixels No of bits = w x h x colour depth resolution (r) given in dots per inch Width = w inches Height = h inches No of pixels = w x r x h x r No of bits = w x r x h x r x colour depth Note that it is possible that the horizontal and vertical resolutions are different Output Hardware for Graphic Data Graphics card responsible for generating images. It contains Video Random Access Memory (VRAM), a Digital to Analogue (DAC) display and Graphics Processing Unit (GPU). There should be enough VRAM to support the pixel data needed for the display DAC converts the digital signal from the computer to the analogue one for the monitor. (Some LCD s can now take a digital signal). The GPU is a digital signal processor (DSP) The GPU generates the images very quickly The GPU applies special effects to the images e.g. anti-aliasing, texturing Digitised Sound Data Input Hardware for Capturing Sound Data Microphone captures analogue sound and converts it to an electrical signal Sound card - Analogue to Digital Converter samples the electrical signal at a certain sampling frequency eg.44100 Hz. The value of the sample is converted to a binary value stored at a sampling resolution or sampling depth (a number of bits) e.g. 16 bit sound. Sound data formats RAW captured sound stored as it is without further processing. When the binary input data is stored sequentially it is called Pulse Code Modulation (PCM). PCM allows data to be transmitted over a network. It is the format of data on CD-Audio, DVD Audio etc. 3
Multimedia Technology RIFF (Resource Interchange File Format) a multimedia container file format. The header of the RIFF file contains data about the type of file. WAV files are RIFF files containing digitised sound data having the header WAVE WAV files can have data encoded in alternative ways e.g. with different compression algorithms. ADPCM (Adaptive Pulse Code Modulation) is a compression codec for sound data which can be used in WAV files. It encodes the sound data as not the value of the volume of the sound, but the difference between the sample and the last one. This can be done using a smaller number of bits. The data is still recorded sequentially and on a network can be decoded as it is received. MP3 Part of the video compression codec (MPEG) - MPEG Audio Layer 3 A lossy compression scheme which is based on what humans hear best Too high or low pitched sounds are removed If two sounds play at the same time and one is significantly louder, the quieter one is removed. Emphasis is placed on frequencies humans hear best. Can adjust quality and compression can get to about 10% of original size Audio terms Bit-rate The rate at which digital signals need to be processed to analogue for us to hear them. The network bandwidth has to be sufficient to transfer data. Bit-rate (bits per second) = sampling depth (bits) x sampling frequency (Hz) Normalising Stretching the amplitude of a sound signal so that the full dynamic range is used. Calculation of File Size File size (bits) = sampling depth(bits) x sampling frequency(hz) x sound time(s) Clipping If the volume of a sound signal is beyond the dynamic range, then the top of signal is lost. This can be caused by recording the sound using too high a volume. Stereo More than one sound track (usually two right and left) recorded by more than one microphone. Will take up twice the disc space of a single (monaural) track. Surround sound Most are 5.1 systems 5 main speakers and an extra subwoofer (0.1) for low frequency effects. Fading Needs 6 channels. Compression is needed; otherwise the bit-rate is too high. Decoding done by hardware. This is a change in volume. It can be used to prevent abrupt starts and finishes. Fade in gradually increases the volume. Fade out gradually decreases the volume. 4
Output Hardware Sound card Digital to Analogue Converter (DAC) changes the computer s digital signal to an analogue one. Amplifier increases the volume of the sound fed to the speakers Digital Signal Processor can apply effects to the sound and can generate surround sound. If the sound card doesn t have a DSP then the computer s processor would have to do the task Video Data Input Hardware for capturing digital video data Digital Video Camera (Camcorder) Similarly to still cameras, they use a CCD array. Expensive cameras have 3 arrays for the 3 primary colours; cheaper ones use a filter with a single CCD. CCD s usually lower resolution, but can capture data very quickly. Webcam Designed for transmitting over Internet so they use low resolution video which has a low bit rate. Have low resolution CCD s and low quality lenses. Video Capture Card Used to convert analogue video (e.g. from video recorder, TV etc.) to digital video. Some graphics cards can do this, but for best results a special video capture card is needed Has a fast ADC to capture video and sound. DSP uses hardware to capture and encode the data to a compressed format e.g. to MPEG see below. Storage of Video Data Uncompressed AVI AVI (Audio Video Interleave) is a type of RIFF file, information on type of file being stored in the header, so an AVI file can be compressed or uncompressed. Uncompressed AVI files are so large that they are almost never used. Audio and Video data is mixed in the file so that it can be streamed. MPEG The Moving Pictures Experts Group has defined a series of standards for compressing video and audio data. Just some frames are stored the key frames. Then data on what has changed since the last key frame The video compression for a frame uses the same lossy technique as for JPEG. The audio compression is MP3. 5
Vector data Calculations File size Calculate size of one frame (a still picture) and multiply by the number of frames. File size(bits) = frame size(bits) x frame rate(frames per second) x time of video(s) Technical Terms Timeline In a video editor, the timeline displays each frame of the video, allowing effects to be applied to a selection of frames Transition Transitions are effects that can be used to join two video clips together e.g. Fade and Wipe Sequencing Putting video clips into different orders. Usually done on the Story board of the video editor. Transitions can be added as clips are connected. Output Hardware Most modern graphics cards act as video output cards. The DSP (in this case a GPU) uses hardware on the graphics card to decode the digital video signal (e.g. from MPEG). While most people still own analogue displays, a DAC on the card is needed to convert the digital signals to analogue. Storing data about objects is more efficient than storing bits. Since the attributes of the objects are stored, the output quality is determined by the resolution of the output device. Relatively easy to convert vector data into bitmapped data, but difficult to convert bitmapped data to vectors. Pattern recognition is required. Vector graphics Common attributes of 2D vector graphic objects are: type of shape, position, size, rotation, line thickness, fill colour etc Common attributes of 3D vector graphic objects are: type of shape, position, size, rotation, texture, etc. Scalable Vector Graphics (SVG) This is a format that encodes vector graphics in XML (extensible Markup Language) which can display in web pages on most browsers with a plug-in. <svg> <circle cx = "200" cy ="200" r ="150" stroke-width = "2" fill = "red" /> </svg> displays a circle at attributes, position (200,200), radius 150, stroke width 2, fill colour red. The transform attribute can be used to rotate objects. i.e. transform = "rotate(angle cx cy) " where the angle is in degrees and (cx, cy) is the centre of rotation. 6
3D Images - VRML Virtual Reality Mark-up (or Modelling) Language A standard for describing interactive 3D objects to display as web pages ( a plug-in is needed) VRML text files are called WORLD files (.wrl) The text file needs a header to tell the vrml application which version of vrml is being used. Shape { appearance Appearance { material Material{ diffusecolor 0 1 0 shininess 1}} geometry Cylinder { radius 0.5 height 4}} draws a shiny cylinder of colour green (R = 0, G = 1, B = 0) The position, rotation, and texture of the objects can be changed and they can be animated. Synthesised Sound (MIDI Musical Instrument Digital Interface) Allows musical instruments to communicate with computers and has become a standard file type for files that contain data that enables the music to be recreated. Midi file format contains data in binary format. It contains a header, which has information about tempo (speed to play the notes). It can have up to 16 channels each playing an instrument. Each channel consists of a sequence of data Start of a note Channel to use Pitch of note Volume to play it at End of note The sound card synthesises the music usually using samples of music data stored on the card Wavetable sound. There may also be control messages e.g. to use an effect like echo or to change instrument. Advantages of MIDI Smaller file size All aspects of music can be edited Effects can be applied to individual instruments No background noise Disadvantages of MIDI Quality depends on sound card Cannot have voice Limited effects can be applied 7
Implications of Use of Multimedia Technology (see Computer Systems Unit) Communications Faster communication allows greater amounts of multimedia data to be transferred and wireless communication makes it accessible anywhere. Wireless WiFi up to 54 Mbps and Bluetooth 2 Mbps Buses USB2-480Mbps and Firewire 1394b 800Mbps Storage Solid State increasingly used in many devices e.g. silicon flash memory cards up to 4GB Magnetic discs now cheaper faster smaller e.g. in Apple Ipod. Optical disc storage CD s and DVD s most common distribution for Multimedia along with Internet. Blue laser (instead of red laser) DVD now available offering increased storage capacity Holographic storage uses lasers. One form of medium is a 3D cube, which would offer increased storage and speed. Not yet available commercially. Processor Many more devices have cheap processors being put into many more devices leading to convergence of technologies. Video processors now often more powerful than the main processor on computers. Research into using light rather than electrons for processors to increase speed. Display technologies LCD TFT displays are becoming much cheaper allowing display of multimedia in more and more places. OLED (Organic Light Emitting Diode) displays can be made in flexible material. Virtual 3D displays now available without virtual reality helmets or goggles (e.g. Sharp display made from two LCD screens one behind the other. Compatible with some graphic cards. Problem with 3D effect if head is not a standard distance from screen.) Real 3D volumetric displays now available, (E.g. www.actuality-systems.com) 8