CG087 Time-based Multimedia Assets 5. Chapter 5 Sampling & Sequencing: Combining MIDI and audio. Introduction. Recording sound

Transcription

1 CG087 Time-based Multimedia Assets 5 Chapter 5 Sampling & Sequencing: Combining MIDI and audio In Zen they say: if something is boring after two minutes, try it for four. If still boring, try it for eight, sixteen, thirty-two, and so on. Eventually one discovers that it s not boring at all, but very interesting. John Cage: Silence Introduction To create audio assets we often want to combine MIDI data with audio (for example to put some sound effects on top of a musical soundtrack). We know from last week that MIDI is not an audio format. So, how can we combine multiple audio files and MIDI data into one artefact? We do this with a special program called a sequencer. Like a multi-track tape recorder a sequencer allows us to build up songs/tracks from component parts. However, unlike a multi-track tape deck, a sequencer is far more flexible, offers more tracks, and allows the combination and mixing of many different formats (e.g. mono audio, stereo audio, MIDI, etc.) Recording sound Before we address the mixing of sound and MIDI in a sequencer we first need to revisit some audio principles. Recall that all sounds comprise one or more sinusoidal components that are continually varying analogue signals. We can record these signals to analogue tape using a microphone, but how do we get sounds into a digital computer? The conversion of an analogue audio signal into a digital one is called sampling. The process is analogous to filming a scene with a movie camera. The camera samples the scene by taking a photograph every 1/30 th of a second. When the film is projected back at 30 fps then we perceive continuous motion. To sample an audio signal instead of a camera we use an analogue-to-digital Converter (ADC). The ADC is presented with continually varying analogue signal which is inspected/sampled at regular intervals and the signal s voltage converted to a binary value. Each binary value is called a sample 1. The sound is reproduced by sending the samples in turn to a digitalto-analogue converter (DAC). Once a sound has been sampled (stored in a digital format) it can be easily edited using a computer. 1 Thus a monaural signal sampled at 44.1 KHz would have 44,100 sample values for each second of audio. At a resolution of 16 bits this requires 44, bits (2 bytes), or Kb of storage per second, or approximately 5Mb per minute. 79

2 80 CG087 Time-based Multimedia Assets Sampling theory A sound is fed via a microphone into an analogue-to-digital converter (ADC) which turns the sound into a series of digital values which are then stored in computer memory (RAM). The stored sound is played back by reversing the sampling process, that is, feeding the digital values held in RAM into a digital-to-analogue converter (DAC) which produces an analogue signal which can in turn be fed into an amplifier for listening. Figure 1 shows how an ADC works. Analogue input Amplifier } Incoming } To voltage convert input voltage to correct voltage range (e.g. 0-5V) Sample and Hold Vs Vc Comparator } 'Remembers' } Produces input at specified time Comparison voltage generator } Generates 0-5V whilst counter increases logic 1 as long as Vs > Vc Counter } Increases until Vc >= Vs then releases digital value Digital representation Figure 1 An analogue to digital converter The ADC is presented with a continually varying analogue signal (provided by a microphone, for example). It then examines the incoming signal at regular time intervals and converts the input signal (a voltage on the microphone cable) into a numeric value (like the camera converting the moving scene into a snapshot every thirtieth of a second). The result is a series of numbers that represent the incoming voltage. So when this series of numbers is sent out through a DAC a sound is heard which should be the same as that which was fed into the ADC. Figure 2 shows the sample points on a 528 Hz wave that has been sampled at 6 KHz. The wave is fed into the ADC which looks at the incoming signal every ms (1/6000) and converts the input voltage to a numeric value. The result is a set of numbers representing the co-ordinates of the sampled waveform. The wave in Figure 2 has a frequency of 528 Hz, therefore, the duration of 1 wavelength is 1/528 = 1.89

3 ms. Thus, at 6,000 samples per second (or 6 samples per millsecond) the 528 Hz wave would be sampled times per wavelength (6 1.89). The individual sample points are shown as dots along the waveform in Figure 2. f=528 Hz Sampling & Sequencing 81 Resolution = 16 bits Although the concept of sampling is fairly simple, there are factors that complicate matters. The first of these is the sampling rate, that is, how often the sound signal is to be sampled. This problem can be easily understood by returning to the movie camera analogy. The idea of a movie camera is to capture on film a series of images which when played back T=1.89 ms (1/f) Figure 2 Sample points on a 528 Hz wave sampled at 6 KHz in quick succession through a projector create a moving picture. The movement must be fluid enough that it does not appear jerky, yet there is no need to have it smoother than the human eye can detect. What speed is the optimum then? Well, in the case of television, a screen refresh rate of 25Hz, or 25 pictures per-second proves to be quite satisfactory. In the same way, a sampler must sample at such a rate that when played back through a DAC (the sampling equivalent of the movie projector) the sample retains the integrity of the original sound without being so detailed that it uses up more memory space than is necessary. So again, what sampling rate will suffice? Well, with sound there is no single value which will provide samples of acceptable integrity with the minimum amount of memory storage across the board, so every solution is a compromise. So far as human speech is concerned it a sampling rate of 8Khz (8,000 samples per-second) provides an acceptable level of reproduction. Figure 3 A complex wave that has been undersampled Sample rate = 6 KHz = 6 samples per ms = samples per wavelength However, less-complex sounds can be represented with a lower sample rate than is needed to accurately capture a more complex wave. Also, the lower the frequency of the sound, the lower the sampling rate needed. Figure 3 shows a higher-frequency complex wave that has also been sampled at 6 KHz. The dotted line joining the sampling points shows an approximation of the resultant sampled wave. We can see that the essential shapes of the original

4 82 CG087 Time-based Multimedia Assets and the digitised waves are quite different. This unrepresentative (and undesirable) result is called aliasing. It is clear that the sampling rate needs to be faster than the incoming frequency, but how much faster? The minimum necessary sampling rate is given by Nyquist s theorem (also called sampling theorem) which says that a waveform can be accurately sampled at a rate equal to (or higher than) twice the highest frequency component in the signal. Thus, a 528 Hz tone can be accurately sampled at KHz, whilst a 2 KHz tone needs to be sampled at a rate of 4 KHz. This is fine if you re recording simple tones of a single frequency, but what about real-world sounds (e.g. music) where the actual tones to be sampled vary across a wide frequency range and where the tones themselves are complex tones that contain many harmonics at very high frequencies? Well, it is generally accepted that the human auditory system does not hear sounds greater than about 20 KHz 2, and so we need a sampling rate of at least 40 KHz to cover all the frequencies that we can hear in a signal. The compact disc standard uses a rate of 44.1 KHz allowing accurate sampling of signals up to 22,050 Hz 3. Digital audio tape (DAT) recorders use a slightly higher rate of 48 KHz. In addition, to prevent aliasing caused by the under-sampling of signals above the Nyquist limit our analogue-to-digital converter first needs to use a low-pass filter to remove all the frequency components that cannot be accurately sampled. This filtering is important because frequencies above the Nyquist limit fold over 4 to produce aliasing effects. The aliasing can be easily calculated. Any frequency f in the range SR/2 SR (where SR = sampling rate) becomes f i where f i = f SR. For example, if the sampling rate (SR) is Hz, then a frequency of 25,000 Hz would become = 19,100 Hz. A frequency of 35 KHz would become 35 KHz 44.1 KHz = 9.1 KHz. A good example of aliasing can be found in television western films. When a stage coach goes past at high speed, the wheels are often observed to be turning slowly backwards. This is because the frame speed (sampling rate) of the film is just slower than the turning speed (frequency) of the wheels. Suppose the film is running at 25 frames per-second and the stage coach wheels are turning thirty times per-second. The camera would capture an image of the wheels on film every 25th of a second thereby placing the spokes of the wheels behind rather than ahead of their last observed position. Whilst a sampling rate of 44.1 KHz allows reasonably good general purpose sampling of music and other everyday sounds, it is wasteful for signals that contain many low frequency components. This is part of the principle behind audio compression techniques (such as MP3) that analyse the signal and sample the low-frequency components at a lower sampling rate. Sample resolution and dynamic range Having decided upon an appropriate sampling rate the next factor to consider is the resolution of the ADC. Suppose an analogue device has an output voltage to be sampled. Let us assume that the output voltage can vary anywhere from 0v to +63v DC. The voltage values can then be represented in steps of 1 volt in six bits ( ). Thus if the input voltage is +20v, the digital value will be However, if the signal is +20.3v the six-bit value will not completely describe the input, but will instead be an approximation (this process of approximation is called quantising). Obviously 2 This is a disputed assertion and there is some evidence to show that even though we cannot hear frequencies above KHz, their presence affects our perception of the signal. Most modern sampling systems now support sampling rates up to 96 KHz which is twice that of the DAT standard. 3 Actually, it samples up to 20 KHz, but the sampling rate is set at 44.1 KHz in to avoid signal contamination from the filter roll-off. 4 A bit like when overflow occurs in integer variables in programs. E.g. 32, = -32,768.

5 Sampling & Sequencing 83 if the number of bits used to represent the values were greater then the precision or resolution of the converter would be greater thereby reducing quantising errors and increasing the accuracy of the sample. For example, suppose seven bits were used instead of six to represent the input. It would now be possible to express 2 7 (128) values. Thus steps of 0.5v could be handled. The CD audio format specifies a sampling resolution of 16 bits which allows the amplitude of the signal to be divided into 65,536 steps. Modern samplers support higher resolutions of 20, 24, and even 30 bits. If the amplitude or intensity of the signal is represented by the voltage presented to the ADC and the resolution of the ADC determines the number of discrete steps into which the voltage range can be broken down, then it follows that the higher the resolution, the greater the range of amplitudes that can be accurately reproduced. The difference between the lowest and highest possible intensities of a system is called the dynamic range. Different applications require different dynamic ranges. For example, most pop music (especially that which is edited to be played over FM radio) has a very narrow dynamic range, that is, there is not a great variation between the quiet and louder passages. Pop music is usually put through a compressor in the studio which boosts the low intensity signals and attenuates the high intensity signals to reduce the dynamic range. Radio stations often compress the signals further to avoid large changes in volume inter- and intra-tracks. On the other hand, orchestral music has a much greater dynamic range with some passages played very quietly on a single instrument and others played very loudly using the whole orchestra. If you spend a few minutes listening to BBC Radio 1 and then a little while longer listening to BBC Radio 3 you should find this difference in dynamic ranges to be quite noticeable. Usually you can set Radio 1 to a medium volume level and leave it there. However, when listening to Radio 3 you may find yourself turning the radio up to hear the very quiet passages and then having to turn the volume down again when the volume picks up. Most people are happy with the dynamic range offered by the 16-bit resolution of compact discs. However, adding just a single bit would effectively double the sampler s resolution which is roughly equivalent to adding about 6 db to the dynamic range. Table 1 Storage requirements in Kb per second for various sampling formats Mono Stereo Sampling rate 8 bit 16 bit 24 bit 8 bit 16 bit 24 bit 8 KHz KHz KHz KHz KHz Sampling, therefore, is a compromise between storage efficiency and the faithfulness of reproduction. Clearly, using a sampling rate of 96 KHz at 24 bit resolution would give excellent results in all cases but requires a lot of storage. Also, it is not necessary to sample at this level for all applications. The spoken voice will have a fairly small dynamic range as well as a narrow frequency band and so can be satisfactorily sampled at lower rates and resolutions. Table 1 gives the storage requirements in kilobytes-per-second for audio sampled at a range of sampling rates and resolutions (ignoring the extra information (e.g. error checking) also required by some formats such as compact disc audio). You can see that 1 minute of stereo audio sampled at 24 bit/96 KHz needs

6 84 CG087 Time-based Multimedia Assets 33,750 Kb (32.95 Mb) of disc space. This compares with Mb for a minute of CD-quality stereo audio. File formats Deciding on a sampling rate and resolution is important, but the next choice is how to store the audio file. There are many different file formats used for storing audio (just as different word processor programs have their own file formats). Table 2 Summary of some common audio file types Extension.SAM Description (taken from Cooledit help system) 8-bit signed :This format is popular for building MOD files, since audio in MOD files is 8-bit signed. Many MOD editors allow samples to be inserted from files, or exported to files in this format. 8-bit signed raw format data with the.sam extension is assumed to be 8-bit signed raw data with no header. The sample rate is assumed to be 22050Hz, but the actual sample rate can be changed once loaded using /Edit/Adjust Sample Rate. A/mu-Law Wave: A-Law and mu-law formats (CCITT standard G.711) are common in telephony applications. These encoding formats compress original 16-bit audio down to 8 bits (for a 2:1 compression ratio) with a dynamic range of about 13-bit. Thus, a-law and mu-law encoded waveforms have a higher s/n ratio than 8-bit PCM, but at the price of a bit more distortion than the original 16-bit audio. The quality is higher than you would get with 4-bit ADPCM formats. Mu-law 8-bit is the international standard telecommunications encoding format, while A-Law is a slight variation found in European systems..wav DVI/IMA ADPCM: The International Multimedia Association (IMA) flavor of ADPCM compresses 16-bit data to 4- bits/sample (4:1) using a different (faster) method than Microsoft ADPCM, and has different distortion characteristics, which can give better, or worse results depending on the sample being compressed. This format also allows for 3-bit compression (5.3:1) as well at a slightly lower quality, though few sound drivers support the 3-bit ADPCM Microsoft ADPCM: The Microsoft ADPCM format consists of 4-bit per channel compressed data (providing 4:1 compression). Windows PCM: Microsoft Windows format. Windows WAV files support both mono and stereo files at a variety of resolutions and sample rates. This file type follows the RIFF (Resource Information File Format) specification, and allows for extra user information to be embedded and saved with the wave file. The standard Windows PCM waveform contains PCM coded data, which is pure, uncompressed pulse code modulation formatted data..aiff,.aif,.snd.vox.mp3.au.ra Apple AIFF: This is Apple's standard wave file format. Like Windows WAV, AIFF files support mono or stereo, 16- bit or 8-bit, and a wide range of sample rates. Cool Edit Pro only supports the PCM encoded portion of the data, even though this format (like Windows WAV) can contain any one of a number of data formats. The AIFF format is a good choice for PC/Mac cross-platform compatibility. Dialogic ADPCM: The Dialogic ADPCM format is commonly found in telephony applications, and has been optimized for low sample rate voice. It will only save mono 16-bit audio, and like other ADPCM formats, it compresses to 4-bits/sample (for a 4:1 ratio). This format has no header, so any file format with the extension.vox will be assumed to be in this format. When opening VOX files in Cooledit, you will be prompted for a sample rate unless "Don't Ask" is checked. Take note of the sample rate of your audio before saving as Dialogic VOX, you will need to enter it upon reopening the file. MPEG. Various compression formats; see chapter 6 dealing with compression. This format is the standard found on NeXT and Sun computers, and has many data types. Cool Edit Pro supports the CCITT mu-law, A-Law, G.721 ADPCM, and linear PCM data variants. Like Windows WAV and AIFF files, this support can support mono or stereo, 16-bit or 8-bit, and a wide range of sample rates when saved as linear PCM. The most common use for the AU file format is for compressing 16-bit data to 8-bit mu-law data. AU is used quite extensively for distribution on the Internet, and for inclusion in JAVA applications and applets. RealAudio: This is Progressive Networks's compressed format used for real time audio streaming over the Internet. RealAudio files can be encoded using several different algorithms. Each encoding algorithm is optimized for a particular type of audio and connection speed bandwidth. In Cooledit you can save to this format from 8- or 16-bit, mono or stereo, with valid sampling rates of 8 khz, khz, 16 khz, khz, and 44.1 khz. You can use a stereo source to produce a mono or a stereo RealAudio file, however you cannot use a mono input file to produce a stereo output file.

7 Sampling & Sequencing 85 Some of the file types in Table 2 use compression (e.g ADPCM), whilst others do not (e.g. standard Windows PCM). We will look at compression in a little more detail in the next chapter. While programs like Windows Media Player will happily cope with a range of audio file formats (useful for auditioning assets you find on the web), some audio sequencing programs are more particular about the formats they will accept. Voyetra s Digital Orchestrator Pro, for example, will only allow you to import Windows PCM audio (.WAV files). Unfortunately, as Table 2 shows, the WAV file specification supports a range of different audio formats. Therefore, if your audio sequencing software is particular about the kinds of audio it will accept, you should use a program like Cool Edit Pro to save a copy of the audio in a compatible format. In the case of Digital Orchestator Pro you can easily save any audio clip as a Windows PCM file. Before we leave the topic of file formats, it is worth mentioning the current effort to specify a universal file format. This is an ongoing debate with much arguing to and fro 5. Don t hold your breath. Sequencing audio and MIDI It is important to know about sampling rates resolutions when we come to combining individual audio files into a single asset. For the reasons discussed above you will find that audio files produced by third parties (such as you find on the Internet) come in a wide range of sampling rates and bit resolutions. Many short sound-effects-type clips are stored in KHz or even KHz formats. Some spoken voice clips come in non-standard rates (e.g. 6.2 KHz) that have been calculated to provide an optimum balance between quality of reproduction and storage. This needs to be taken into account when we try to combine audio files of different formats and from different sources into a single asset. Some sequencer programs automatically resample every piece of audio that is imported, but not all do so, and some do but with strange results. In some programs where the default sampling rate has been set at 44.1 KHz, importing an audio file that has been sampled at KHz leads to the audio being heard at double speed. This is because some programs do not actually resample the incoming audio but simply play it back at the default speed. This is analogous to playing a 33 rpm long-playing-record at 45 rpm on a dual-speed turntable. A proper resampling algorithm will take the source (at whatever sampling rate it has been recorded) and will then sample it at the preferred rate as if it were recording a live sound. To illustrate these points, open Cool Edit Pro and load in the girl.wav audio file found on the Blackboard page for this week. The status bar at the bottom of the Cool Edit window shows that this file has a sampling format of 16 bit 16 KHz mono audio. Play the file to familiarise yourself with it. Now select Edit Adjust Sample Rate and choose a new sampling rate of 48 KHz. What does the file sound like when played back now? This function simply lets you play an audio file at any sample rate; it does not convert (or resample) the audio file data to the new rate. Close girl.wav and reopen it to restore it to its original settings. Now select Edit Convert Sample Type and choose a rate of 48 KHz. When you play back the file now it should sound like it did before you adjusted it. Convert Sample Type directly processes the samples within the file, or re-samples the data, so that the audio will retain the same pitch and duration as the original file. If you save the file (with a different name, say girl2.wav) and compare it to the original girl.wav you will see that its file size is much bigger. girl.wav should use 199 Kb and girl2.wav should take 594 Kb. If you now try downsampling (choose a 5 See for insight into the problems of defining a universal audio file format.

8 86 CG087 Time-based Multimedia Assets sample rate of KHz and save this version as girl3.wav you will see that this file is smaller at 137 Kb. Digital Orchestrator Pro (DOP) can only import audio files that have sampling rates of KHz, KHz, and 44.1 KHz. Therefore, it is important to become familiar with how to up- and down-sample audio files in Cool Edit Pro as many audio files you will find on the web have sampling rates outside DOP s acceptable range. If you try and import a file such as girl.wav into DOP, it will ask you if you want to convert the file to the nearest acceptable sampling rate. If you do this and play back the audio you will notice that its pitch has been changed as DOP has not properly resampled the audio. Once you have your audio in a format that a program like DOP can handle you can start experimenting with sequencing it. Sequencing is the arranging of audio/midi data into a desired order. Sequencers work around the notion of tracks of data. DOP lets you have as many tracks as you like (though the processor and memory of your PC will determine the practical limits of what is possible). Figure 4 shows a typical project in DOP which is one I created in a couple of hours with the aid of some sampled drum loops from a sample CD, a Roland D-50 on which I played the lead melodies, and a Boss DS-330 on which the piano parts were played. The D-50 and DS-330 parts were recorded as MIDI data. I then created audio tracks (21-24) onto which I recorded the audio output from the synthesisers as the MIDI tracks (1, 2, 4, & 5) were played back. The whole project, now in audio, was mixed down to a single stereo.wav file. I then used the Lame encoder to produce an.mp3 format file which you can find on the Blackboard page for this week (the original.orc file is over 19 Mb, so I haven t brought that in from home). Figure 4 Audio and MIDI tracks in Digital Orchestrator Pro

9 Sampling & Sequencing 87 The first two tracks hold MIDI data (there is a picture of a MIDI DIN plug in the type column). The next track holds audio data (denoted by the waveform icon), and so on. Tracks run horizontally from left to right with the left-hand end being the start of the piece. The track window is organised into bars, or measures, as sequencers are normally used for composing music. The tempo, or speed, of the piece is specified in the tempo box (the one with the picture of the metronome next to it in this case 130 beats-per-minute). Looking at Figure 4 we can see that the first audio track has audio data playing from bar 4 whereas track 15 plays from the beginning of the piece. By positioning (using drag and drop) your audio data along the timeline you can make your chosen audio events occur just when you want them 6. You can download an MP3 recording of this piece MIDI tracks can be created by capturing MIDI data from a MIDI instrument, entering MIDI data directly via the editor, or by importing MIDI files. When importing MIDI files you need to be aware that the imported data will be played back at the tempo of the current project. This is important as a piece that sounds fine at its original tempo may not sound so good if played much faster or slower. For example, a MIDI file that was recorded at a temp of 160 beats-per-minute (bpm) is going to sound very strange if played back at, say, 120 bpm. When combining MIDI files you need to be careful to select pieces with similar tempi so that they won t be affected by small adjustments in tempo. You need to be especially careful when mixing MIDI with audio. Speeding up and slowing down MIDI playback is a simple affair. However, adjusting the tempo of an audio file is problematic. Simply playing an audio file faster will lead to an increase in pitch, whilst slowing it down will decrease its pitch. This is the same problem that occurs when you try to play a 33 rpm record at 45 rpm and vice versa. Conversely, adjusting the pitch of a piece of audio to match that of MIDI data or other audio files can be done by speeding up or slowing down the audio. However, this will lead to a change in the duration of the audio. This is problematic if the audio is time-based and intended to fit into a syncrhonised framework. The solution to both audio problems is to use software that can change the pitch and tempo of a piece of audio without affecting its other attributes. Changing the pitch of audio without changing its duration is called pitch shifting. Changing the duration while leaving the pitch unchanged is called time-stretching. Audio editors like Cool Edit Pro and SoundForge have a range of pitch shifting and time stretching algorithms available 7. These algorithms are very sophisticated, but even so, their results are only really successful when small adjustments are made to a sound. For example, say you have a sampled note (E) from a bass guitar that you want to use to create an entire bass line. You could pitch shift the sample up to, say, a G# and down to an B quite happily (a shift of 5 semitones in either direction which is equivalent to a frequency change of ± 33.48% 8 ). This is about the useful 6 If you find you want a sound to start in the middle of a bar, or just after the start of a bar, then you can edit the audio more precisely using the Digital Audio editor (either by pushing the button at the bottom of the DOP screen with the picture of a waveform on it, or choose Window New Digital Audio. In either case, highlight the audio you want to edit. In the Digital Audio view you can select, drag, and drop the audio at much higher time resolutions. You can increase or decrease the resolution by quantising to different note values. See the DOP manual (available on Blackboard) for detailed instructions. 7 SoundForge has a particularly good set with algorithms optimised for specific applications such as speech, instrumental music, vocal music, drum sounds, and so on. 8 The frequency of a pitch doubles with each octave. For example, the pitch of A above middle C is 440 Hz. The frequency of A' (one octave higher) is 880 Hz. Since the time of Bach, musicians have adopted the equal-

10 88 CG087 Time-based Multimedia Assets limit of a pitch shifting algorithm as shifting outside this range gives results that do not sound convincing. Therefore, you really need samples of a few notes from the bass in order to reproduce the full twelve tones of an octave. To create convincing digital pianos that use samples of real acoustic (and prohibitively expensive) pianos requires sampling the acoustic piano every three semitones. The notes in between can be emulated by pitch shifting. Pitch shifting examples On Blackboard you will find a number of MP3 files accompanying these notes. First, play the file C Mutes.mp3. This is a recording of a guitar strumming some C chords. Now listen to Amutes.mp3. This is a version of C Mutes.mp3 that has been pitch-shifted down three semitones. Now play D Mutes.mp3 (two semitones up) and E Mutes.mp3 (a four semitone rise). All of these sound reasonably convincing, that is, they sound much as we would expect the guitar to sound if the chords A, D, and E had been played. Finally, listen to C Mutes Octave.mp3. This file has had its pitch raised by a whole octave (twelve semitones). It sounds very unconvincing. Time stretching example To here how time stretching can work, listen to Bongo fills 125 bpm (mono).mp3. This is a drum loop originally recorded at 125 bpm. I wanted to convert this to a slower tempo, and so used a time stretching algorithm to change its tempo to 104 bpm. The result can be heard by playing Bongo fills 104 bpm (mono).mp3. The result is quite good, though you can hear some slight chorusing on some of the drum hits. Summary So, when creating an audio asset that combines MIDI and audio you need to be very careful about your selection of source material. Shifting the tempo and the pitch of audio is possible but is only effective across a narrow range. Pitch shifting is very useful for making small adjustments (of a few Hz) to audio files that are only slightly out of tune with each other. On the other hand, the pitch of MIDI data can easily be shifted without detriment. Select MIDI source material of similar tempi. If including music stored in audio files, then the pitch and tempo of the other material is very important as it s hard to make big changes to the music audio. Commercially-produced sample CDs always indicate the tempo and pitch of each file making it much easier to match source material correctly. If you want to combine some audio files without any MIDI data, then Cool Edit Pro will allow you to do this. It has a multi-track view which lets you position audio files on separate tracks along a timeline. Why mix audio and MIDI? Many multimedia products have musical aspects, usually in the form of a soundtrack. MIDI is a particularly convenient means of transporting musical data, although, as we noted in the last chapter, it offers no control over how the music will sound on the client machine. If you want to tempered scale in which the octave is divided into twelve equally-spaced steps corresponding to the twelve semitones A, A#, B, C, C#, D, D#, E, F, F#, G, G#, A'. Thus, the frequency difference between two semitones is 2 1/12, or So, if A = 440 Hz, then A# = = Hz, and so on. Therefore, an increase of 5 semitones, such as A to D is given by = = Hz.

11 Sampling & Sequencing 89 guarantee that a MIDI music track will sound the same on every machine, then you need to record the audio output of your MIDI hardware and distribute the asset as sampled audio. The disadvantage with this solution is that audio files take up far more space than MIDI files. The main reason we would mix MIDI with audio is for ease of use at the composition and construction stage. MIDI data is easily captured from synthesisers and as long as the synthesiser is attached to the computer on which you are creating your soundtrack you can get access to their sounds. The MIDI data takes up very little space on the hard disc and requires little RAM for working storage. It is very easy and quick to process. For example, transposing a MIDI file up a semitone is simple, whereas transposing an audio recording of the same piece is computationally expensive, time consuming, and not guaranteed to give an acceptable result. Thus, MIDI is a useful medium for work in progress. Consider the piece jingle.mp3 mentioned above, and shown in Figure 4. When I created this piece I used MIDI to record the music I played on the synthesisers. If I played a wrong note it could be easily corrected or removed using the MIDI editor in DOP. Any timing errors could also be easily corrected. Only when I was happy with the performance did I then record the MIDI track to audio ready for the final mixdown. By keeping the data in the MIDI domain editing was very easy. Imagine trying to correct the pitch of a single wrong note in an audio file? Of course, once everything is mixed down to audio, the file storage requirements increase. Jingle is about a minute long, and so needs about 10 Mb to store it as 16 bit 44.1 KHz stereo audio. The file size can be reduced through compression (jingle.mp3 is only 1.2 Mb), though the quality starts to suffer. I have created a RealAudio version of the piece optimised for a 56K modem. The file jingle.rm is only 254 Kb, but it does not sound as good. An alternative is to maintain the quality but to stream the file to obviate the need for a single large download. We shall deal with compression and streaming in more detail in the next chapter. When does MIDI become audio? We know that MIDI is not an audio format. But jingle.mp3 is an audio file and has been produced by combining MIDI performance data with sampled loops. So when exactly does MIDI become audio? The transformation occurs when a MIDI tone generator (such as a synthesiser, or a MIDI-equipped sound card) interprets MIDI data and plays it. When you download a MIDI file and play it with Microsoft Media Player it becomes audio when Media Player sends the MIDI data in the file to your sound card. The only difference between this and the audio data in jingle.mp3 is that I did the conversion to audio on my computer and recorded the output of the synthesiser/sound card. The advantage of the former is that the MIDI file is small. The disadvantage is that the quality of the playback is entirely dependent on the quality of the sound card/synthesiser attached to the computer. The advantage of the latter approach is that we know that the music will sound the same on every machine (though the quality of the loudspeakers will have an effect on this). The disadvantage of the latter approach is that the file size is much larger. However, we are not restricted to the choice between deploying small MIDI files whose reproduction we cannot control and large audio files that will sound fine, but that are large. Technologies like Beatnik s extensible music format (XMF) allow a hybrid approach 9. Beatnik s system allows developers to create files that contain both MIDI data. and sampled audio. The MIDI playback is via the Beatnik Audio Engine (BAE) that is installed on the client machine. The BAE provides a set of sampled MIDI instrument definitions through which the MIDI data are played. This means that file size economies can be achieved, yet consistency of experience is maintained. 9 See

12 90 CG087 Time-based Multimedia Assets MIDI and audio can also be deployed via a programming approach. The Java Media Framework (JMF) provides a set of APIs that enable programmers to created and playback a variety of media formats from within a Java program or applet 10. JMF supports audio sample rates from 8KHz to 48KHz, though the cross-platform version only supports the rates 8, , , 16, 22.05, , 32, 44.1, and 48 KHz. 10 See for full details of supported formats