ICT 514 Multimedia Systems Topic 4: Audio

Transcription

1 ICT 514 Multimedia Systems Topic 4: Audio Ref: Chapman N. & Chapman J., Digital Multimedia 2 nd Edition Chapter 9 Lance Fung master of science in information technology

2 Overview Objectives of this lecture Sound Hearing What is sound? Digital sound Sampling Resolution Storing MIDI Streaming Editing Playback ICT 514_06_Lect 4 2

3 Objectives At the end of this lecture, you will be able to: describe what sound is and how we hear it outline what is meant by sampling sound compare WAV and MP3 in terms of file size and the use of compression understand what MIDI is and how MIDI files are created discuss the advantages/disadvantages of streaming sound technology ICT 514_06_Lect 4 3

4 Sound The theory of sound is complex - digital sound is even more complex. Sound is one of the most engaging and influential elements of multimedia, be it music, special effects, or spoken words. Sound, particularly music, can stimulate emotions and associations. How often do you hear a particular song and immediately remember some event, person, experience associated with it? Like colours that create different emotions (see lecture on design principles), sound can make you feel dreamy, romantic, annoyed, frightened, shocked, etc. ICT 514_06_Lect 4 4

5 Sound Physical disturbance: Sound is an organised movement of molecules caused by a vibrating body in some medium. Sensation (psychological): Sound is the auditory sensation produced through the ear by the alteration in pressure. ICT 514_06_Lect 4 5

6 Sound How is sound heard? Sound originates when a body moves back and forth rapidly enough to send a wave through the medium in which it is vibrating. Sound is produced when two or more objects collide, releasing a wave of energy which in turn forces changes in the surrounding air pressure. For example, clapping your hands. Sound waves move in all directions from the disturbance, like ripples produced when a stone is dropped into a pond. ICT 514_06_Lect 4 6

7 Sound The vibrations travel as waves through some medium like air and reach our ears. This analogue wave causes our eardrum to vibrate and the vibrations are transmitted by nerves to the brain. We talk of digital sound but what we actually hear is analogue (sound travelling through the air). The ear is made of 3 parts: Outer Ear Middle Ear Inner Ear ICT 514_06_Lect 4 7

8 Outer ear Middle ear Inner ear

9 ICT 514_06_Lect 4 9

10 Hearing Outer Ear The Outer Ear The outer ear is comprised of: auricle (or pinna) ear canal

11 Hearing Outer Ear Ear Canal Each ear leads into an ear canal - an irregular cylinder with an average diameter of less than 0.8 mm and about 2.5 cm long. The ear canal is open at the outer end which is surrounded by the pinna (or auricle). The pinna plays an important spatial focusing role in hearing. The canal narrows slightly and widens towards its inner end, which is sealed off by the eardrum (tympanic membrane). The canal is therefore like an organ pipe. It has a shaped tube enclosing a resonating column of air - with the combination of open and closed ends. ICT 514_06_Lect 4 11

12 Hearing Outer Ear The ear canal supports (resonates or enhances) sound vibrations best at the frequencies which the human ears hear most sharply. This resonance amplifies the variations of air pressure that make up sound waves, placing a peak pressure directly at the eardrum. For frequencies between approximately 2 KHz and 5.5 KHz, the sound pressure level at the eardrum is approximately 10 times the pressure of the sound at the auricle.

13 Hearing Outer Ear Eardrum The eardrum is the interface between outer and middle ear. Airborne sound waves reach only as far as the eardrum. Here they are converted into mechanical vibrations in the solid materials of the middle ear. Sounds (air pressure waves) first set up sympathetic vibrations in the taunt membrane of the eardrum, just as they do in the diaphragm of some types of microphone. The eardrum passes these vibrations on to the middle ear structure.

14 Hearing Middle Ear The Middle Ear

15 Hearing Middle Ear The Middle Ear The middle ear contains three small bones known as: Malleus (hammer) Incus (anvil) Stapes (stirrup) These bones form a system of levers which are linked together and driven by the eardrum. The malleus pushes the incus, the incus pushes the stapes. ICT 514_06_Lect 4 15

16 Hearing Middle Ear Working together as a lever system, these bones amplify the force of sound vibrations. In combination, these bones (which are the smallest bones in the human body) double or triple the force of the vibrations at the eardrum. The muscles of the middle ear (the tiniest muscles in the human body) modify the performance of these bones as an amplifying unit. These muscles also act as safety devices to protect the ear against excessively large vibrations from very loud sounds - a sort of automatic volume control. This is also the part that sometimes gets stuffed up when we have a cold. ICT 514_06_Lect 4 16

17 Hearing Middle Ear From Outer to Inner Ear Within the 4 cm or so occupied by the outer and middle ears, three distinct physical principles operate to magnify weak vibrations in air so that they can establish pressure waves in a liquid: The organ pipe resonance of the ear canal may increase the air pressure force 10 times. The mechanical advantages of the bone system may nearly triple it. The pinpointing arrangement of the eardrum and the oval window may provide another thirty fold increase. The result of these three mechanisms may be an amplification of a sound wave by more than 800 times before it sets the liquid of the inner ear in motion. ICT 514_06_Lect 4 17

18 Hearing Inner Ear The Inner Ear Semicircular canals (for balance) Cochlea (for sensing sound)

19 Hearing The semicircular canals are the body's balance mechanism and it is thought that it plays no part in hearing. The cochlea includes: vestibular canal tympanic canal cochlea duct

20 Hearing Inner Ear The fibres (hair cells) in the cochlea detect sound. The fibres are long at one end of the cochlea and gradually become shorter at the other. The length of the fibres seems to determine the frequency of sound to which the fibre is sensitive. High frequency tones occur at the base of the cochlea and the lower frequencies towards the apex. ICT 514_06_Lect 4 20

21 Hearing Inner Ear Sound in the cochlea branch of the auditory nerve is interpreted in the brain in terms of: Pitch (determined by frequency of sound) Loudness (amplitude of sound) Timbre (quality, determined by the waveform of the sound). It is in the brain that we actually hear. Human beings are sensitive to sound frequencies from about 16Hz to 20KHz. For comparison, dogs are sensitive to sound frequencies over 25KHz, and bats at about 100KHz. ICT 514_06_Lect 4 21

22 Sound properties Pitch Analogous to colour in light (wavelength or frequency). Sound pitch depends on the frequency. Intensity Intensity of sound is the energy per second (power). Relative intensities are measured in decibels. Loudness Loudness is a sensation and it is therefore difficult to measure. Usually the greater the intensity, the greater the loudness. Quality or Timbre If the same note is played on a violin and a piano, you can tell the difference. Each instrument introduces different overtones (background frequencies). ICT 514_06_Lect 4 22

23 What is sound? Sound - in its analogue form - has a waveform, which has 2 characteristics: 1. Frequency Frequency determines the pitch of a sound. It is a measurement of how many waveforms there are in a given period of time. The more waveforms there are moving one after another, the higher the pitch of the sound. Frequencies are measured in Hertz, which is the number of oscillations or waves per second, i.e 1 Hz = 1 cycle per second 2. Amplitude The amplitude of a sound wave is the height of the sound wave at a particular time. This is the loudness or volume of the sound. Simplistically, the higher the wave, the louder the sound. ICT 514_06_Lect 4 23

24 + Frequency 0 - Amplitude Time Analogue waveform ICT 514_06_Lect 4 24

25 High Pitch: this wave corresponds to a pure tone of 3,000 Hz (3 khz) Low Pitch: this wave corresponds to a pure tone of 300 Hz. ICT 514_06_Lect 4 25

26 Digital sound We ll look at different processes in creating a digital sound file: Sampling Storing File formats (WAVE, MP3, VQF) Streaming MIDI Editing Playback ICT 514_06_Lect 4 26

27 Sampling Natural sound sources are analogue but sound on the Web and the Internet is digital because a computer is used to record or playback the sound. To record sound, an Analogue to Digital Converter (ADC) is needed. To playback sound, a Digital to Analogue Converter (DAC) is needed. ADCs convert analogue to digital by sampling the analogue signal. Sampling a sound wave consists of determining the amplitude of the sound at a number of discrete times within a given time frame. ICT 514_06_Lect 4 27

28 Sampling To record an analogue sound on your computer, you need to take a snapshot of the sound wave every so often and then join the dots to reconstruct an image of the original wave. ICT 514_06_Lect 4 28

29 This process is called sampling. Digitised sound is sampled sound. Every fraction of a second, a sample of sound is taken and stored as digital information in bits. The number of times a sample is taken is called the sampling rate or sampling frequency. The sampling rate is measured in Hertz or KiloHertz (thousands of samples per second). ICT 514_06_Lect 4 29

30 Time Sample Amplitude Sample Amplitude Sampling the waveform Low sampling frequency Time

31 Sampling Obviously, the more often a sample is taken, the better the representation of the original, continuous sound. If a low sampling rate is used to sample high frequency sound, then much of the sound information is lost. What sampling rate should be used? ICT 514_06_Lect 4 31

32 In the 1920s, H. Nyquist showed that a signal of bandwidth (H) could be completely reconstructed by making only 2H samples per second. Simply, this means that the sampling rate must be at least twice the highest frequency that is to be recorded. A bandwidth is the range of frequencies needed to transmit a signal. The larger the bandwidth, the larger the information carrying capacity. (Rather like the size of a pipe that is needed for a certain amount of liquid to flow.) ICT 514_06_Lect 4 32

33 How often to sample? If we sample at 1 time per cycle we ll think it s a constant. If we sample at 1.5 per cycle we ll think it s a a low frequency.

34 Sampling If the sampling rate is low, then aliasing will occur. If the sampling frequency is too high, it is a waste of storage space. For sound that we can hear, the highest frequency is approximately 20KHz. Therefore CD Audio is sampled at 44.1KHz. If a human voice is to be digitised then a sampling rate of about 10KHz would be enough because speech rarely contains frequencies above 5KHz ICT 514_06_Lect 4 34

35 Sampling Example sampling rates (samples/sec) Telephone 8,000 AM radio 11,025 FM radio 22,050 Compact disk 44,100 ICT 514_06_Lect 4 35

36 Sampling There are three sampling frequencies most often used in multimedia khz is sufficient for speech (it s about as low as you can go and still get reasonable results) khz is OK depending on other factors khz is CD quality frequency for music (Note that the first two are sub-multiples of CD quality 44.1) Obviously, the more samples/second, the larger the sound file. ICT 514_06_Lect 4 36

37 Resolution Each sample of data is stored with a certain amount of information about the amplitude of the sound at that moment in time. The number of bits used to store the amplitude determines the resolution. If we use only 4 bits to store this amplitude value (i.e. 4 bits per sample), then only 16 possible values are available for the amplitude. 8 bits give 256 possible values. 16 bits give 65,536 different values. Again, the more values the better the representation and the more values the larger the file. ICT 514_06_Lect 4 37

38 Resolution In the process of saving the amplitude to a certain number of bits, some data may be lost. If the number of bits available to record the amplitude is low, then there may be some clipping of the top and bottom of the wave, which severely distorts the sound. ICT 514_06_Lect 4 38

39 The resolution at which a recording is made is determined by your sound card: If your sound card is only capable of recording in 8 bit then that is the best resolution you can get. If your sound card is capable of recording at 16 bit, then you can use the recording software to record in either 8 or 16 bit. ICT 514_06_Lect 4 39

40 Resolution It may not necessarily be good to store the highest and lowest amplitude in a sound signal (i.e. the full dynamic range). These high and low amplitudes do not occur frequently so if values are allocated to them then the more common amplitudes have a sparser allocation of values. This means that the accuracy at which the more common amplitudes are stored is low. ICT 514_06_Lect 4 40

41 So sound can be recorded at a reduced amplitude range. By reducing the range, the highest and lowest amplitudes will be chopped off at the limit set. This is called clipping. 8-bit resolution is good enough for voice but music requires 16-bit resolution for high fidelity. ICT 514_06_Lect 4 41

42 Resolution Finally, there is the option of recording in mono or stereo. We hear with two ears and catch different sounds in each ear. Stereophonic sound simulates this, so are more lifelike. Mono recordings pick up everything, but sound dull and flat. ICT 514_06_Lect 4 42

43 Stereo is achieved by recording on two channels instead of one. This means, of course, that twice the information is stored for every sample that is taken. Obviously, stereo files will be twice as large for each sampling frequency as mono files. ICT 514_06_Lect 4 43

44 Resolution CoolEdit ICT 514_06_Lect 4 44

45 Resolution For 1 second 16-bit resolution stereo on a CD: 44,100 x 16 x 2 = 1,411,200 bits For 1 hour: 44,100 x 16 x 2 x 3,600 = ~635 Mb ICT 514_06_Lect 4 45

46 Storing When the sampling has been done and the bits recorded for every sample, this data needs to be stored. Now you need to decide in which format to store the file: WAV MP3 VQF Liquid-Audio a2b RealAudio Windows Audio 4.0 We will look at some of these formats ICT 514_06_Lect 4 46

47 Storing : WAV Wave (.wav) is the most basic and oldest sound file format, and is the Windows sound system format. Windows sound effects are recorded as wav files. WAV is to sound what BMP is to graphics. ICT 514_06_Lect 4 47

48 WAV doesn t use any compression techniques. They are constructed simply by taking samples of the sound and then recording those as digital data. The file is simply a recording of all data from all the samples taken. Just as BMP is the largest of the graphic file formats, WAV files are the largest of the sound file formats. An 8 sec recording of Mozart in wav format is 1.3 Mb... ICT 514_06_Lect 4 48

49 Storing : WAV Wave (.wav) is the most basic and oldest sound file format, and is the Windows sound system format. Windows sound effects are recorded as wav files. MacOS sound format is AIFF UNIX sound format is AU All have broadly similar capabilities now. Each platform supports compressed and uncompressed data, with a range of compressors. ICT 514_06_Lect 4 49

50 Storing : MP3 MP3 stands for MPEG-1 Audio Layer 3. Layer 3 is one of three coding schemes (layer 1, layer 2 and layer 3) for the compression of audio signals. Each layer increases in complexity in the encoding process. Layer 3 uses compression to discard sounds which cannot be heard by humans, that is, the very low and very high frequencies. ICT 514_06_Lect 4 50

51 Storing : MP3 MP3 uses a lossy compression technique This means some data is lost but the loss in quality is not noticed. The compression is impressive - up to 12 times smaller than a corresponding WAV file. MP3 has become popular as a means of compressing audio, particularly music, for downloading over the Internet, both legally and illegally. ICT 514_06_Lect 4 51

52 Storing : VQF VQF is another audio compression format. It is similar to MP3 in one regard: it takes large sound files, and compresses them down to very small files. However, VQF is regarded as doing it better than MP3s. ICT 514_06_Lect 4 52

53 There are three important aspects to sound encoding: 1. File size VQF files are approximately 30-35% smaller than MP3 files. 2. Sound quality The sound quality of VQFs is much better than MP3s. 3. CPU usage Here VQFs are more cumbersome than MP3s. However, they were meant to be so. When MP3s were developed, Pentiums were superior. With Pentium IIIs and later CPUs, and other multimedia enhanced computers, the load can be handled by most machines. This is what allows it to pack as much (or more) sound data into a 30% smaller file!! ICT 514_06_Lect 4 53

54 Streaming Concerts, speeches, lectures, etc. are now recorded in a way that allows you to listen to them in real time. The technique used is called streaming. Streaming audio is not a file format, but a technique for playing back audio (and other) data transmitted over the Internet and other communications channels. Normally when you click on a sound file link, the sound file is downloaded to the computer s memory, your player is opened and then the sound begins to play. ICT 514_06_Lect 4 54

55 With the streaming technique, the sound is compressed and then sent through the communication channels in a continuous stream. This means that the user doesn t have to wait for a large file to download. ICT 514_06_Lect 4 55

56 Streaming Streaming uses a lossy compression. It reduces a file by a factor of 100. The quality, of course, is not as good as MP3. Streaming audio must be encoded and played back with the right software, such as RealAudio. ICT 514_06_Lect 4 56

57 The problem with streaming data is that often the speed at which the data comes across the Internet and the speed at which the player can play the data are not the same. The data may arrive too fast and bits have to be cached (stored temporarily) until the player can catch up. The data may arrive too slowly, because of congestion on the transmission channels, and the sound becomes very jerky. ICT 514_06_Lect 4 57

58 MIDI MIDI stands for Musical Instrument Digital Interface. MIDI is a standard adopted by the electronic music industry for controlling computerised devices such as synthesisers. It is a standard language for recording the sound from electronic music hardware. Because of this standard language, electronic instruments and computers can communicate with each other. ICT 514_06_Lect 4 58

59 Rather than sampling existing sound, MIDI is a technique and format for recording original music which is played into the computer. Normally this is done using a synthesiser (a computerised musical instrument which produces digital rather than analogue sound). But you can simulate the synthesiser with the computer keyboard. ICT 514_06_Lect 4 59

60 MIDI MIDI files, then, do not record music. MIDI files store a description of the music that is played by the synthesiser. The MIDI representation of a sound always includes values for the note s pitch, length and volume. The file may also include additional characteristics of the music. ICT 514_06_Lect 4 60

61 MIDI files are much smaller than WAV. MIDI files are to sound what vector graphics are to images. A 2-minute.mid recording is only 12K. Listen to the range of instrumental sounds that can be replicated with MIDI, but also notice that the music does sound synthetic, a computergenerated replica of music ICT 514_06_Lect 4 61

62 MIDI and Computers MIDI interface allows computer to send MIDI data to instruments Store MIDI sequences in files, exchange them between computers, incorporate into multimedia Computer can synthesize sounds on a sound card, or play back samples from disk in response to MIDI instructions Computer becomes primitive musical instrument (quality of sound inferior to dedicated instruments) ICT 514_06_Lect 4 62

63 MIDI Messages Instructions that control some aspect of the performance of an instrument Status byte indicates type of message 2data bytes values of parameters e.g. Note On + note number (0..127) + key velocity Running status omit status byte if it is the same as preceding one ICT 514_06_Lect 4 63

64 General MIDI Synths and samplers provide a variety of voices MIDI Program Change message selects a new voice, but mapping from values to voices is not defined in the MIDI standard General MIDI (addendum to standard) specifies 128 standard voices for Program Change values Actually GM specifies voice names, no guarantee that identical sounds will be produced on different instruments ICT 514_06_Lect 4 64

65 Editing As with a graphics file, you can manipulate a sound file. One of the obvious things you might want to do is to remove parts of the file that you don t like (like cropping a photo), and add other parts. Or reverse the sound (like Paul Simon, Gracelands) Remember, though, to edit sound files well, you need to have a good knowledge of music and digital sound files. But you can experiment - maybe this is what will interest you most and you will learn more about it. CoolEdit will not edit MP3 or MIDI files. MP3 is a final (compressed) file format. You will need an original digital WAV file to edit. ICT 514_06_Lect 4 65

66 Sound Editing Timeline divided into tracks Sound on each track displayed as a waveform 'Scrub' over part of a track e.g. to find pauses Cut and paste, drag and drop May combine many tracks from different recordings (mix-down) ICT 514_06_Lect 4 66

67 Effects and Filters Noise gate Low pass and high pass filters Notch filter De-esser Click repairer Reverb Graphic equalizer Envelope Shaping Pitch alteration and time stretching etc ICT 514_06_Lect 4 67

68 Compression In general, lossy methods required because of complex and unpredictable nature of audio data CD quality, stereo, 3-minute song requires over 25 Mbytes Data rate exceeds bandwidth of dial-up Internet connection Difference in the way we perceive sound and image means different approach from image compression is needed ICT 514_06_Lect 4 68

69 Companding Non-linear quantization Higher quantization levels spaced further apart than lower ones Quiet sounds represented in greater detail than loud ones ICT 514_06_Lect 4 69

70 ADPCM Differential Pulse Code Modulation Similar to video inter-frame compression Compute a predicted value for next sample, store the difference between prediction and actual value Adaptive Differential Pulse Code Modulation Dynamically vary step size used to store quantized differences ICT 514_06_Lect 4 70

71 Perceptually-Based Compression Identify and discard data that doesn't affect the perception of the signal Needs a psycho-acoustical model, since ear and brain do not respond to sound waves in a simple way Threshold of hearing sounds too quiet to hear Masking sound obscured by some other sound ICT 514_06_Lect 4 71

72 The Threshold of Hearing ICT 514_06_Lect 4 72

73 Masking ICT 514_06_Lect 4 73

74 300 Compression Algorithm Split signal into bands of frequencies using filters Commonly use 32 bands Compute masking level for each band, based on its average value and a psycho-acoustical model i.e. approximate masking curve by a single value for each band Discard signal if it is below masking level Otherwise quantize using the minimum number of bits that will mask quantization noise ICT 514_06_Lect 4 74

75 Incorporating into your web page You can choose if you want your sound file to be foreground music or background music. Foreground music: For WAV or MID files, use the regular href tags, e.g. <a href= filename.mid >Mozart music</a> <a href= filename.wav >Yesterday music</a> ICT 514_06_Lect 4 75

76 Playback Background music: More problematic, because Netscape and IE use different tags. Netscape uses <embed> tags. IE uses <bgsound> tags (add this as an attribute in the <body> tag). ICT 514_06_Lect 4 76

77 To have both browsers playback, use the following HTML code: <embed src="filename.mid" width= 145 height= 55 > <noembed> <bgsound src= filename.mid > </noembed> There are other attributes you can add to the embed tag, such as: autostart= true -- for playing once autostart= true loop= true -- for playing continuously For more about HTML for sound, see the Annabella s URI in the reference section. ICT 514_06_Lect 4 77

78 Summary How do we hear? Where is the sound interpreted? What is sound? What are the two characteristics of a waveform? How do you digitise sound? What is the sampling rate? What sampling frequency would you use for speech? CD? What is an uncompressed format of a sound file? What sort of compression is an MP3 file? What is streaming? What sort of compression does streaming use? ICT 514_06_Lect 4 78

79 References Beekman, Brent and Rathswohl, pp , 57-59, Play it again Sam, APC, August Nyquist, erial/notes/chap3/chap3.1/chap3.1.html URLs: Physiology of Hearing, 1.html ng.html ound.htm ICT 514_06_Lect 4 79

80 HTML Help for Music: Beginner s MP3 Player Setup: Digitalising Sound: MIDI: Downloads for audio/video: Sound cards: ICT 514_06_Lect 4 80