Contents Media Files and Formats CSCU9B2 Prehistory: analogue Digital formats Compression Proprietary vs Free formats Image formats Audio formats Video formats CSCU9B2 1 CSCU9B2 2 The prehistory The prequel: analogue CSCU9B2 3 Pre-Digital Formats In the 1980s and before, most media was analogue Music was recorded on vinyl or magnetic tape Photos were taken onto light sensitive film and printed onto paper Nothing was turned into numbers storage was physically related to the sound or look of the media In that sense what was recorded was analogous to what was being stored CSCU9B2 4
Audio recording This was first invented in 1877 by Thomas Edison (on a cylinder) (analogue) Moved to disc from about 1889. Commercial recordings started in the 1900 s, and the disc overtook the cylinder about 1910. CSCU9B2 5 Audio analog recording techniques Early recordings were analogue. Precise shape of groove is a representation of the original signal. It is recorded by replicating the pressure wave into a pattern made in a groove on the substrate (wax/shellac/vinyl) And played back by placing a fine stylus in the groove, turning the signal into an analogue electrical signal, amplifying it, and then turning the analogue signal back into a pressure wave. CSCU9B2 6 Analogue Audio: tape Physical recordings are write-once/read often Tape recordings Made by encoding the pressure wave in the pattern of magnetisation on the magnetic particles on a tape allow write often/read often usage Useful for Portable recordings Erasable recordings Formats: Reel to reel Cassette European format and 8 track But still analogue What happened? Analog recording was the dominant technology for most of the 20 th century Subject to continuous improvement 45 rpm singles, 33 rpm LP s Better styli, better transducers, better amplifiers, better tape, better loudspeakers Stereo Became a mature and dominant technology And, indeed, a large industry Music recording, distribution, retailing, Note that copying an analogue recording results in a reduction in quality CSCU9B2 7 CSCU9B2 8
Analogue Images & Video Mechanical assistance Albrecht Dürer (1471-1528) used a grid to aid his reproductions. CSCU9B2 9 CSCU9B2 10 Technological image creation History of Photography? See Wikipedia for details Daguerrotype from 1839 Lincoln, 1846 Many innovations on its way to a pervasive mass technology Moving images http://en.wikipedia.org/wiki/file:muybridge _horse_gallop_animated_2.gif (1878) CSCU9B2 11 CSCU9B2 12
Moving images Victorian toys: zoetrope Film Multiple photography Initially travelling showmen, Later full commercial industry Needed industrialization of process, accurate machining etc. Places to show films Editing film Whole industry: Modern ideas of entertainment Documentaries and newsreels Imagination and illusion» Envelopment and escapism CSCU9B2 13 CSCU9B2 14 Snapshot: mid-1960 s analogue image industry Large industries: film, cameras, developers, consumer movies (super-8, and the developers), cinema, studios, production, distribution, showing: even photocopying (literally!). Picture-based printed media: newspapers, magazines, posters etc. Altogether a major industry pre-existed before anyone had really thought about using digital technology in this area Hollywood, studios in many other countries as well. CSCU9B2 15 Digital Formats As computers became faster and storage cheaper, it became sensible to store media in digital format This involves turning the physical (analogue) media into a long list of numbers which represented its form The form the media takes can be thought of as its signal. We will look at two digital signals image and sound CSCU9B2 16
Advantages of Digital Error free transmission and duplication Indefinitely copying without loss of quality Storage is generally safe and backups are easy to make Will the stored data last forever? See last lecture Storage of large quantities in small spaces Becomes better and better Fast transmission and sharing, including error correction Ease of searching, sorting, organising and editing Now hugely popular Image storage CSCU9B2 17 CSCU9B2 18 Electromagnetic Spectrum (1: visible is very small part 2: not all colours are present in the rainbow!) Images The analogue signal we record from an image is light specifically spectral content CSCU9B2 19 Digital formats do not store wavelength data, they use colour theory The most common format is called RGB CSCU9B2 20
RGB Colour Format Additive colour mixing produces colours by mixing light Mix colours of all wavelengths evenly and you get white (simplification) The additive primary colours are red, green and blue any colour can be made with the right combination of amounts of these three colours (simplification) So, to encode a colour, you need to store three values the amount of red, green, and blue it has CSCU9B2 21 24 Bit Colour You probably know that in computing, a byte is made up of 8 bits Each bit (binary digit) can take one of two values 0 or 1 With 8 bits, you can represent 2 8 = 256 different values, in this case the integers from 0 to 255 Standard 24 bit RGB colour uses 3 bytes to represent a colour, one for each primary colour using additive mixing CSCU9B2 22 RGB Colours 24 bit colour gives 256 x 256 x 256 16m different colours The eye is thought to be able to see about 10m different colours, so this should be fine for human consumption Smaller colour models are also used, for example 8 bit, either using a palette or 3 bits for red, 3 for green and 2 for blue: RGB Mixing Each of the three RGB components takes a value from 0 (no light of this colour) to 255 (full power!) 0,0,0 is black 255,255,255 is white 255,0,0 is bright red See (right) that red + green = yellow so 255,255,0 = yellow CSCU9B2 23 CSCU9B2 24
Using a value of less than 255 for any component reduces its quantity in the mix RGB Mixing Hexadecimal A small technical complication: Computer scientists don t always use base 10 You ve met binary base 2 Now here is base 16: 0,1,2,3,4,5,6,7,8,9,A,B,C,D,E,F,10,11 Base 16 is known as hexadecimal, or hex for short Each hex digit encodes 4 binary digits CSCU9B2 25 CSCU9B2 26 Hex and Colours 8 bit hex numbers run from 0 to FF So FF=255, FE=254, etc. RGB values are often represented in hex as a number from 0 to FF So white is FFFFFF, pale turquoise is AFEEEE, etc. Grey scales have all three values the same, e.g. ABABAB Web browsers expect colours to be specified in hex, as do some graphics programs Web Colour Names When web browsers and computer monitors could not display all 16m colours, there were a set of web safe colours defined. These don t tend to get used any more There is now a list of browser defined colours, given names such as Red (FF0000) and SpringGreen (00FF7F) CSCU9B2 27 CSCU9B2 28
RGBA - Transparency Representing an Image The RGBA colour model has a fourth component, known as the alpha (a) channel The alpha channel dictates the level of transparency in the colour This gives a 32 bit model 8 bits for alpha, plus the 24 for RGB Only some graphics formats can cope with alpha data (more later on this) CSCU9B2 29 Image is considered as an M by N array of picture elements Pixels M N <R, G, B> CSCU9B2 30 Representing an image cont d Mostly, images are assumed to be rectangular in shape, so they have a width and a height Each pixel has a colour represented with RGB The image is stored in memory or a file Image Files If you want to keep an image for long, you need to store it in a file You will need to store the numbers in such a way that image viewing software will know how to turn them back into an image This means that you will choose a format for your image CSCU9B2 31 CSCU9B2 32
Image File Formats This is not a new issue! There are lots of image file format standards A file format defines a number of things about how to convert from image to numbers and back to image again What makes the biggest difference is the compression method used More on this later Raster Images Images stored as a set of pixel values are known as raster images. Old-fashioned term from the days of cathode ray tubes This is the most common way to represent an image, particularly for photographs and many other graphics. Later we will look at an alternative vector graphics There are also mixtures of raster and vector There are also 3D image formats, which we won t be looking at CSCU9B2 33 CSCU9B2 34 Raster Image File Format When storing an image in a file, there are generally two types of data to store: Information about the image Dimensions and resolution Size in bytes Colour palette Etc. Pixel data in some format File Size and Quality Three things that make a difference to image quality and file size: Colour depth (number of bits used to encode colour 24 is very good) Resolution number of pixels in image. Obviously, more pixels = higher resolution Compression more on this later CSCU9B2 35 CSCU9B2 36
.BMP files A very simple file format,.bmp stores the RGB (and sometimes alpha) channels explicitly Also known as a Device Independent Bitmap The header starts with BM to show it is a bitmap file Pixel data starts in the lower left corner of the image, row by row.bmp Example Here is an enlarged image of 4 pixels Following the header, the file would contain: FFFF00 00FF00 FF0000 0000FF Note, the spaces are for you to read it more easily, they don t appear in the file CSCU9B2 37 CSCU9B2 38 The Trouble with BMP Images can get quite big Let s say we want to print at 300dpi (dots per inch) A 6 x8 image would be 300x6 x 300x8 = over 4 million pixels, in 24 bit RGB that is nearly 13MB Not huge by today s standards, but slow if you are downloading a lot and not ideal if you have small memory cards CSCU9B2 39 Compression Image compression stores fewer numbers to represent an image Image compression may be: Lossless meaning that the re-created image is a perfect reproduction of the uncompressed original Lossy meaning that some information is thrown away (which may or may not be visible to the human eye) CSCU9B2 40
Lossless Compression Need to be able to reproduce the source image exactly Sensible for images on buttons, icons, etc. which are small anyway and need to look perfect Dictionary Encoding Based on a dictionary of pixel patterns found in the image Each pattern gets a code The codes are stored in the file To reproduce the image, look up each code and display the pattern it refers to An enhancement is to use variable length codes where common patterns have short codes CSCU9B2 41 CSCU9B2 42 Lossy Compression Reducing the colour depth can make an image file size smaller, as can scaling an image to smaller dimensions, but these acts do not really count as compression, though both are lossy Lossy compression generally relies on what we know about the human eye, and how it is better at seeing some things than others Subsampling The human visual system is more sensitive to changes in brightness than colour By keeping brightness information and discarding colour information, we can store less data and leave an image looking nearly as good as the original CSCU9B2 43 CSCU9B2 44
Scalability Many lossy compression algorithms allow the user to choose the amount of compression applied More compression = more loss = poorer image quality = smaller file Here is a jpeg image, compressed progressively from right to left Common Image Formats It pays to use the right image format at the right time Image quality Photo or low-colour graphics Animation Transparency Storage and transmission limitations Patents and freedom CSCU9B2 45 CC Michael Gäbler / Wikimedia Commons CSCU9B2 46 GIF 8 bit colour only Lossless compression (given the above) Can handle transparency Can be animated Was once patented, but this was challenged and it is no longer protected Good for web graphics PNG Created as open source version of GIF Supports 8 bit and 24 bit colours Transparency and animation Lossless (better than GIF, given colour depth) Generally preferred to GIF CSCU9B2 47 CSCU9B2 48
JPEG JPEG Lossy uses sub sampling compression Good for photos, bad for icons and illustrations No transparency nor animation Bad for text To probe further: see http://en.wikipedia.org/wik i/image_file_formats CSCU9B2 49 Next. Audio Formats CSCU9B2 50 Audio Formats Audio signals are produced by sound Sound can be thought of as a wave that travels from what ever makes the sound (the source) to whatever hears it (the receptor). Analogue sound through air causes a pressure wave, which means that the pressure at the receptor is continuously changing If the receptor is an ear, this continuous change in pressure is perceived as sound Plotting the Wave If we plot the change in pressure at the receptor over time, we see the familiar sound wave: CSCU9B2 51 CSCU9B2 52 CC Fillbit / Wikimedia Commons
Continuous Vs Digital Something we ignored when we looked at image formats: Analogue signals are continuous (usually) there is no number inside their operating range that they cannot take In theory, this allows for infinite accuracy Digital computers are not able to represent infinite ranges Binary Limitations Remember we said that 8 bit bytes can represent numbers from 0 to 255? More bits allow more numbers: 16 bits allows a range from 0 to 65,535, for example (the rate CDs are coded at) The number of bits used in representing the sound as a number is known as the word length, or resolution. CSCU9B2 53 CSCU9B2 54 Quantization The process of converting a continuous signal into a series of discrete values is known as quantization The accuracy lost by this process is called the quantization error Sampling Rate Just as the range of values are continuous, so is the signal over time To digitise a signal, we must sample at some fixed interval the samples are a series of snap shots Anything that happens between the samples is not recorded The higher the sampling rate, the better the sound quality CSCU9B2 55 CC Wikimedia Commons CSCU9B2 56
Sampling Rate Sound is made up of waves The pitch (bass to treble) of a note is largely determined by the frequency of the wave The frequency of a wave is the number of times it cycles in one second (measured in Hertz, Hz) High frequency waves require high frequency sampling To capture information at a frequency f Hz, you need to sample at a rate of more than 2f Hz This is known as the Nyquist sampling theorem. CSCU9B2 57 Audio Frequencies The (young!) human ear can hear sounds from around 20Hz to 20,000Hz (Hz = Hertz = cycles per second) CD quality sampling is at 44,100Hz, so is over the Nyquist rate for the human ear Of course, sound contains higher frequencies, but it doesn t matter that they are lost They do show up as a problem known as aliasing and need to be removed CSCU9B2 58 Sound Quality So, two things affect the quality of digital sound: Word length (accuracy of the representation) Sampling rate Usually 44,100Hz, but can be more or less These two factors combine to produce the usual measure of digital audio quality bit rate Bit Rate Bit rate describes the number of bits per second used to encode digital audio Stated in kilobits per second (1000 bits) It is simply the number of bits of the word length multiplied by the sampling frequency (times 2 for stereo) So 44,100Hz x 16bits x 2 = 1,411,200 = 1,411 kbit/s CSCU9B2 59 CSCU9B2 60
Compression You could use fewer bits per second if you dropped the sampling frequency or word length, but that would make a small reduction in size and a large reduction in sound quality Compression techniques can reduce the bit rate further (sometimes 10:1) without much loss of quality Compression Compression algorithms reduce the bit rate of an audio file while trying not to reduce sound quality Many algorithms for compressing sound, and they have a special name: codecs Coder/decoders Generally, the codec used will be specified by the file extension, e.g file.mp3 CSCU9B2 61 CSCU9B2 62 Compression Codecs Just as with image compression, there are two types of codec with respect to loss: Lossy codecs reduce the sound quality Lossless ones keep the sound quality Other things to consider about a codec Will hardware devices play it? Is it patent free? Does it include digital rights management (DRM)? Lossless Compression First, a quick mention of.wav files Microsoft s non-compressed audio format Can store 16bit 44.1kHz Sound quality is perfect, file size is large FLAC Free Lossless Audio Codec Patent and royalty free Typical 50% file size reduction Not all devices can play FLAC files CSCU9B2 63 CSCU9B2 64
Lossless Compression ALAC - Apple Lossless Audio Codec Proprietary owned by apple Works on all idevices Shares the.m4a file extension with Apple s lossy format, AAC Around 50% compression rate Lossy Compression Perhaps the best known MP3 Bit rates from 8 to 320 kbit/s Sampling rate up to 48 khz Patented some issues with Firefox, for example DRM rarely used Widely supported by hardware and software CSCU9B2 65 CSCU9B2 66 Advanced Audio Coding (AAC) Designed to replace MP3 Better audio quality for same bit rates Chosen by Apple for itunes store format Includes DRM restrictions FairPlay Widely supported by software, smart phones, games consoles, etc. Codec of choice on YouTube Windows Media Audio Owned by Microsoft Lossy and Lossless versions + one for voice 48 khz, up to 768 kbit/s DRM support Not as widely supported as some other formats (need to convert it to AAC to play on idevices, for example) CSCU9B2 67 CSCU9B2 68
Vorbis Ogg Ogg is a freeware audio codec with no patents and no licence fees Can sample up to 200 khz Format used in HTML5 No DRM Well supported by hardware, phones, etc. Finally. Video Formats CSCU9B2 69 CSCU9B2 70 Video Codecs Digital video overview: Video is a series of images shown in order at a speed the creates the illusion of movement It also often has an audio track We have already looked at digital image and audio compression We will now have a quick look at video Movie is made up of a sequence of still images Each still image could be coded as for images But this would need a lot of space Some stills are coded this way Rest coded by differencing Deltas Key Frames CSCU9B2 71 CSCU9B2 72
WMV Windows Media Video Owned by Microsoft Can support DRM Software support in Windows Media Player, RealPlayer and others Some hardware support e.g. Windows mobile devices File extension is.wmv H.264 / MPEG-4 AVC AVC = Advanced Video Coding One of the codec standards for Blu-ray Used by YouTube, Adobe Flash, itunes, and various HDTV Used on idevices too Patent protected arguments about use in HTML5 abound File extension is usually.mp4 or.avc CSCU9B2 73 CSCU9B2 74 Flash Video One of the most common online video format Plays in the Adobe Flash Player Supports a variety of codecs including H.264 Not supported by idevices File extensions are.flv,.f4v Also can be contained in.swf files CSCU9B2 75 Theora Free lossy compression format Same organisation that brought you.ogg audio Supported by HTML5 Patent free, open source No idevice support Many software packages will play it File extension is.ogg or.ogv CSCU9B2 76
Summary Images, audio and video all have in common: A wide choice of file format and compression methods, quality and file size Issues of quantization, resolution and sampling rate A choice of free or patented methods Limitations in which software and devices will recognise them CSCU9B2 77