AUTOMATING THE DESIGN OF SOUND SYNTHESIS TECHNIQUES USING EVOLUTIONARY METHODS. Ricardo A. Garcia *

Transcription

1 AUTOMATING THE DESIGN OF SOUND SYNTHESIS TECHNIQUES USING EVOLUTIONARY METHODS Ricardo A. Garcia * MIT Media Lab Machine Listening Group 20 Ames St., E5-49, Cambridge, MA 0239 rago@media.mit.edu ABSTRACT Digital sound synthesizers, ubiquitous today in sound cards, sotware and dedicated hardware, use algorithms (Sound Synthesis Techniques, SSTs) capable o generating sounds similar to those o acoustic instruments and even totally novel sounds. The design o SSTs is a very hard problem. It is usually assumed that it requires human ingenuity to design an algorithm suitable or synthesizing a sound with certain characteristics. Many o the SSTs commonly used are the ruit o experimentation and a long reinement processes. A SST is determined by its unctional orm and internal parameters. Design o SSTs is usually done by selecting a ixed unctional orm rom a handul o commonly used SSTs, and perorming a parameter estimation technique to ind a set o internal parameters that will best emulate the target sound. A new approach or automating the design o SSTs is proposed. It uses a set o examples o the desired behavior o the SST in the orm o inputs + target sound. The approach is capable o suggesting novel unctional orms and their internal parameters, suited to ollow closely the given examples. Design o a SST is stated as a search problem in the SST space (the space spanned by all the possible valid unctional orms and internal parameters, within certain limits to make it practical). This search is done using evolutionary methods; speciically, Genetic Programming (GP).. INTRODUCTION Sound synthesizers are usually implemented as computer programs and algorithms that run in digital computers and produce digital sound samples (waveorms). These algorithms or sound generation are termed Sound Synthesis Techniques (SSTs). An SST can be dissected into a unctional orm and internal parameters. The unctional orm (also known as topology) describes the relationship between the unctions and elements in the algorithm, while the internal parameters are variables that take a particular value at the moment o implementation o the algorithm (depending on the desired behavior). Design o a SST is customarily limited to the selection o a unctional orm rom a set o algorithms (i.e. classic SSTs) ollowed by application o a mathematical technique or estimation o the internal parameters to match a target sound. The design o SSTs, more speciically their unctional orm, is a very hard problem. It is usually assumed that it requires human ingenuity to design an algorithm suitable or synthesizing sound with certain characteristics. Many o the SSTs commonly used are the ruit o experimentation and a long reinement processes. The eorts or automating the design o SSTs have been mainly ocused into automating the parameter estimation stage o the internal parameters or a given unctional orm. Horner et al. [] proposed an approach or automating the internal parameter estimation o FM synthesizers using evolutionary methods, in particular Genetic Algorithms (GA). Johnson [2] proposed an interesting approach to use evolutionary methods and human listeners in an interactive system to explore the parameter space o (Fonction d Onde Formantique) FOF synthesis. Wehn [3] used GA to explore the parameter space o FM-like synthesizers, and allowed some degree o variation in the unctional orms. Our goal is to propose a general approach capable o suggesting valid unctional orms and internal parameters or a SST to synthesize a target sound, using a known set o inputs (time varying signals). This problem is related to the system identiication, or symbolic regression problem stated in control theory. The inputs and outputs o the system are known, but the system is unknown... Approach The SST space is deined as the space spanned by all the possible combinations o a given set o unctional elements and their connections. Every point in the SST space deines completely a unctional orm and its set o internal parameters. Design o a SST is then regarded as a search in the SST space. The goal is to ind a point in this space that is capable o producing a sound close to the target sound using the given inputs. This measure o closeness is done using an error metric. The search in this space is perormed using a class o evolutionary computation method called Genetic Programming (GP). GP has shown outstanding empirical perormance in searching complex multidimensional spaces [4-6]. Custom SST representations in the orm o topology graphs and expression trees are used along * Now with Chaoticom. 83 Laayette Road Hampton Falls, N.H , USA DAFX-

2 with their required mapping rules. Topology graphs are the most widely used representation or SST, but they are diicult to manipulate. Expression trees acilitate the level o manipulation required to use GP or exploring the SST space. A complete discussion o this research can be ound in [7]. 2. REPRESENTATION OF SSTS SSTs are computer algorithms designed to produce sound samples. Algorithms are usually represented using: mathematical ormulas, instruction lists (pseudocode) or topology graphs (low diagrams). All o them are equivalent representation (posses the same inormation), but oer dierent advantages rom the viewpoint o a human designer. evaluate sin(x) assign result to P evaluate 2*P b) assign result to Q evaluate Q+0.5 y = 2 sin( x) assign result to Y output Y 0.5 Y a) + c) X 2 sin( ) Figure. Representations o algorithm: a) ormula, b) pseudocode and, c) topology graph 2.. Representing SSTs as expression trees Topology graphs o complex SSTs are usually in the orm o cyclic graphs (with closed loops). Manipulating this kind o graph is very diicult. Addition, deletion or reconnection o unctional elements can render a topology invalid very easy. In x addition, i the designer in charge o manipulating these topologies is a computer program (as it is the goal), the easier the manipulation, the less probability o creating useless SSTs. Expression trees are graphs that are very easy to manipulate by ollowing a small set o construction rules. It makes sense to try to ind a mapping between an easy-to-manipulate expression tree graph into a diicult-to-manipulate topology graph. An ingenious idea borrowed rom developmental biology suggests a way o doing this. The idea is to encode in the expression tree the instructions or the development o an embryonic topology. The process begins with a very simple embryo, and ollowing the instructions it grows the ully developed topology. It can even include the development or the internal parameters associated with the unctional elements. Problems similar to this one that involved development o topology graphs rom expression trees were suggested by Koza et al. [6, 8] that used a mapping into analog circuit topologies; and Gruau [9] that mapped trees into neural network amilies Suggested mapping: expression tree into topology graph The nodes in the expression trees are instructions that when executed will result in a ully developed topology graph. Every expression tree will render a unique topology graph, but it is possible to have more than one tree to render the same topology graph. The initial topology graph is called embryo, and in our case it is a simple topology with our blocks, as seen in Figure 2 The embryo has a single modiiable object TYPE_A with no unctional element assigned yet, and connected to two sources and one renderer. The sources and the renderer will remain the same during the whole development process, but the modiiable object will change and new blocks and connections will be created. This coniguration o the embryo could be dierent (to suit the design speciications, i.e. the number o time varying inputs), but has been chosen or explicatory purposes here. Figure 2 shows a simple expression tree, embryo and irst steps o development o a topology. The irst node o the expression tree Step 0 EMBRYO SOURCE Step SOURCE Step SOURCE Step 3 Step SOURCE ADD 3 2 SOURCE KOSCIL ADD Figure 2. Example o mapping rom expression tree into topology graph. Note the development o the embryo into a more complex topology DAFX-2

3 is the START node, and this is ignored during the development process. The second node is pointing to the modiiable object number. When executed, this node will change the topology graph, more speciically the object that is pointed to in some way. In this case, the node has the instruction, so the type is assigned to the particular block in the topology graph. The next node has the instruction SERIES. The eect o this instruction is to add some new blocks and connections to our topology graph, and to create more pointers to dierent nodes in dierent new branches o the expression tree. Ater executing this Topology Modiying Function, several new blocks and connections are introduced into the topology graph. Each one o the new objects is modiiable, and has an associated node pointing to it. The rest o the nodes are executed, and this adds, modiies, or changes blocks and their connections into the topology graph. A complete repertoire o topology development unctions can be ound in [7] Functional elements From an analysis o several classic SSTs [0-2], a set o commonly used basic unctional elements was extracted. They are called classic SSTs because they have been used and studied by researchers and musicians over many years, and oer a good approximation or a set o unctional elements. This is a list o the main types o unctional elements ound in our taxonomy: Sinusoidal oscillators (variable amplitude, requency, phase over time) Wavetable oscillator (variable amplitude, read index) Delay (memory) or one or more samples Controlled gain ilter Noise generator Time varying ilters (coeicients can change over time) Addition Multiplication 3. DESIGN OF SST Design o an algorithm is deined as the process that conceives the structural orm and internal parameters o an algorithm, capable o producing a desired set o outputs, using a known set o inputs. The speciications or the design are usually given as sets o examples. Each example comprises a set o inputs and a target output (desired behavior). It is necessary to speciy as well an Error Metric to measure the perormance o a given SST. Design o an SST is usually a two-stage process: irst selection o a unctional orm and, second parameter estimation. 3.. Classic design In classic SSTs design a human realizes the irst stage o the process. Functional orms are usually never conceived rom scratch or a particular target sound. Instead, the designer selects one template rom a set o known unctional orms (i.e. the classic SST) based on the characteristics o the target sound, and the known capabilities o the tentative unctional orms. In the second stage, the designer selects an approach or parameter estimation, and uses it to ind the internal parameters that better it the selected unctional orm to produce a sound close to the target sound. This part o the process has been automated with high success [] or FM synthesis parameter estimation. The designer usually tries a handul o unctional orms to select the one that results in a better match to the target sound. LPC ADDITIVE FM FM selection by human Parameter Estimation INPUTS target OUTPUT use SST human judgement Figure 3 Classic design o SST. Human selects a ixed Functional Form rom a pre-deined set (i.e. classic SST) and uses a parameter estimation method. Human judges i the results are satisactory or repeats process with new unctional orm 3.2. Proposed approach or design Our proposed approach tries to remove as much human intervention rom the design process as possible. The irst change (and maybe the most important) is to replace the irst stage o selection o a pre-made unctional orm, with a unctional-orm suggesting mechanism. This mechanism will suggest valid unctional orms that can be tested to see i they are good or not or the desired goal. The second stage remains the same, and it consists in the parameter estimation or the selected unctional orm to try to match the target sound. Another point where the human intervention can be reduced is in the error comparison between the output sound and the target sound. This comparison (error metric) will return a value that will be used or suggesting a new unctional orm, and it will try to minimize the error. The procedure is repeated until the error alls within acceptable limits. SST suggestion unctional orm + initial parameters unctional elements Parameter Estimation INPUTS target OUTPUT error unction use SST Figure 4 Suggested approach or design. Automated suggestion o Functional Forms, parameter estimation, and automated comparison o target/output sounds ed back in suggestion block. DAFX-3

4 3.3. Parameter estimation Once a unctional-orm has been selected (or suggested), the number and type o internal parameters remains ixed. A technique or parameter estimation can be used to ind a set o values or those parameters that will reduce the error between the produced output and the target sound. This is usually done using either mathematical analysis or optimization methods. The approach selected depends mainly in the type o unctional orm to be optimized and in the precision needed or its internal parameters. Some o the mathematical analysis tools commonly used include Fourier and Cepstral analysis. When a direct mathematical analysis is not convenient, it is possible to use a numerical method to approximate a good set o internal parameters. These methods explore the parameter space in a guided manner, and return usually a locally optimal set o parameters that accomplish the desired behavior or the SST. Note that the error metric plays a undamental role in the exploration o the parameter space and, ultimately, in the selection o the parameters. is repeated until a candidate solution that shows a itness value that ulills the speciications is ound, or a maximum allowed number o generations have been tested. 4. DESIGN AS A SEARCH IN SST SPACE All the possible valid combinations o unctional elements, connections and internal parameters compose the SST space. Hypothesis: Given a set o inputs, a target sound and an error metric, it is possible to ind the unctional-orm and internal parameters o a SST capable o synthesizing an output sound close to the target sound. Our original goal o designing a SST (unctional orm and internal parameters) can then be stated as a search problem in the SST space. The next step is to deine a search strategy to eiciently and adaptively explore this space and ind an acceptable solution to our problem. The SST space has many dimensions and is highly non linear. Each point in this space represents a dierent SST (with dierent unctional orm and internal parameters). Evolutionary methods have been shown to perorm very well in complex search spaces. [4, 3]. 4.. Genetic Programming (GP) Genetic Programming (GP) is an optimization/search method that has been gaining popularity in the last decade. It is an extension o Genetic Algorithms, and both belong to the ield o Evolutionary Computation. The idea with GP is to have a population o candidate solutions (in our case, suggested SSTs) that will be evaluated and a itness value assigned to each. The itness unction gives an analytical measure o the perormance o the individual and its output. Once all the individuals in the population have computed their itness value, a new population o candidate solutions is created by probabilistically selecting individuals and perorming genetic operations on them. The probability o being selected to be part o a genetic operation is directly related to the itness o the individual: the better the itness, the higher the probability. The genetic operations will create new individuals by: copy (identical copy o an individual), mutation (random alteration o an individual unctional orm and/or internal parameters), or crossover (characteristics o two individuals are used together to create a new one). The process Figure 5 Genetic Programming loop 4.2. Fitness unctions In any kind o optimization or search method, it is undamental to have a way to measure the perormance o the candidate solution. This perormance metric is usually called a itness unction or error metric. Fitness unctions (FF) give some numerical grade to the dierence between the outputs o the system compared to a desired target. The eatures that are measured in a itness unction vary rom application to application. In our case, or sound synthesis techniques and sound sample sequences (waveorms) as targets, it is usual to deine itness unctions that measure the distance between two sounds, or how similar they are. An analytical itness unction that uses the Least Squares Error (LSE) o the magnitude spectrograms (o Target and produced sounds) has been successully applied by Horner et al. [], and showed good results with this project. An enhancement or this itness unction was to include phase inormation. The LSE o the phase spectrograms, weighted with the magnitude o the target sound was successully used in most o the test perormed [7, 4]. Perceptual itness unctions are more diicult to compute because o their subjective nature. A itness unction that incorporates a psychoacoustic model o simultaneous requency masking [5-7] was developed and used successully in some tests perormed [7]. DAFX-4

5 5. AGeSS SYSTEM A system implementing the proposed approach called AGeSS (Automatic Generation o Sound Synthesizers) has been developed and tested. The system is implemented as a set o binaries (compiled or ANSI C++) and Matlab scripts. The user is required to supply the parameters or the GP run, as well as the examples or the system. USER Inputs - Run parameters - Examples AGeSS SYSTEM Output Suggested expression tree Figure 6 AGeSS system: user input (parameters, examples o control signals and Target). Output (suggested expression tree) target sound and uses this to calculate the threshold o masking o the target. This inormation, along with the spectrogram o the output sound is used to calculate a distance metric. Figure 8 Spectrogram and waveorm o TARGET signal ormed by two notes (A880, B988). 6. EXAMPLE The developed AGeSS system was used to perorm a series o experiments to explore the potential o the suggested approach. One o them is outlined here. A simple FM synthesis ormula was chosen or this experiment [0, 2], as shown in equation. This SST has been explored in depth by many researchers and musicians. The value o the internal parameters was taken rom the original values suggested by Chowning or simulating a woodwind sound [8]. C M s( t) = A( t)sin 2π + 2πIM sin 2π () FS FS C = Carrier requency = 880, 988 = (t) M = Modulator requency = 880/3, 988/3 = (t)/3 I = index o modulation = 2 FS = sampling requency = 8000 A(t) = time varying envelope This SST uses two time varying inputs: A(t) and (t) and 3 internal parameters I, D, M. For the generation o the Target sound, two time varying signals were generated (using Matlab) to simulate the brass sound o two distinct notes (A880, B988) o 0.3 seconds each. These can be seen in Figure 7. Figure 9 Spectrogram, waveorm and topology or best individual o Generation 220 Note that the inal spectrum agrees with the target in all the requencies o the harmonics. But the inal spectrum has higher energy at the high end o the spectrum. The topology evolved in generation 220 is shown in Figure 9. It is possible to analyze the unctional elements and their connections to ind the close orm ormula representation o the topology. In this case, it is represented in equation 2. ( k ( t) + ( t) oscil ( ( t) )) ( oscil ( k ( t ) ) s ( t) = k A( t) oscil ) + k (2) Comparison between equations and 2 shows a close similarity in their unctional orm CONCLUSIONS Figure 7 Input signals (top) Envelope or two notes. (bottom) Normalized pitch or two notes (A880, B988). The selected itness unction uses the simultaneous requency masking itness unction. It calculates the spectrogram o the Design is stated as a search in the multidimensional SST space. Each point in this space will represent a dierent unctional orm and set o internal parameters. The goal is then to ind a point in the SST space that will ulill the speciications o design. It is not clear how neighboring points are related in this representation. In addition, the number o possible points in this space is huge, making it impossible to do a thorough search o the space. These characteristics make the search o the SST space a DAFX-5

6 very complex problem. Evolutionary methods, such as Genetic Programming, have proved satisactory when dealing with these types o problems. The experiments show that the selected set o unctional elements and the representation scheme are eective or the automated design o some common synthesis algorithms, especially the requency modulation techniques. For more inormation about this research project, please visit [7] P. R. Cook, "Music, cognition, and computerized sound : an introduction to psychoacoustics,". Cambridge, Mass.: MIT Press, 999. [8] J. Chowning, "The synthesis o complex audio spectra by means o requency modulation," Journal o the Audio Engineering Society, vol. 2, pp , REFERENCES [] A. Horner, J. Beauchamp, and L. Haken, "Machine Tongues.6. Genetic Algorithms and Their Application to Fm Matching Synthesis," Computer Music Journal, vol. 7, pp. 7-29, 993. [2] C. G. Johnson, "Exploring the sound-space o synthesis algorithms using interactive genetic algorithms," presented at Proceedings o the AISB Workshop on Articial Intelligence and Musical Creativity, Edinburgh, 999. [3] K. Wehn, "Using ideas rom natural selection to evolve synthesized sounds," presented at Proceedings o the Digital Audio Eects DAFX98 workshop, Barcelona, 998. [4] J. R. Koza, Genetic programming : on the programming o computers by means o natural selection. Cambridge, Mass.: MIT Press, 992. [5] J. R. Koza, Genetic programming II : automatic discovery o reusable programs. Cambridge, Mass.: MIT Press, 994. [6] J. R. Koza, Genetic programming III : darwinian invention and problem solving. San Francisco: Morgan Kaumann, 999. [7] R. A. Garcia, "Automatic Generation o Sound Synthesis Techniques," in Program in Media Arts & Sciences: Massachusetts Institute o Technology, 200, pp. 98 p. [8] J. R. Koza, F. H. Bennett, III, D. Andre, M. A. Keane, and F. Dunlap, "Automated synthesis o analog electrical circuits by means o genetic programming," IEEE Transactions on Evolutionary Computation, vol., pp , 997. [9] F. Gruau, "Cellular encodign o genetic neural networks," Ecole Normale Superiéure de Lyon, Lyon 92-2, May [0] C. Roads, The computer music tutorial. Cambridge, Mass.: MIT Press, 994. [] G. Depoli, "A Tutorial on Digital Sound Synthesis Techniques," Computer Music Journal, vol. 7, pp. 8-26, 983. [2] R. C. Boulanger, "The Csound book : perspectives in sotware synthesis, sound design, signal processing, and programming,". Cambridge, Mass.: MIT Press, [3] N. A. Gersheneld, The nature o mathematical modeling. New York: Cambridge University Press, 999. [4] R. A. Garcia, "Growing Sound Synthesizers using Evolutionary Methods," presented at Sixth European Conerence on Artiicial Lie. Workshop on Artiicial Lie Models or Musical Applications, Prague, Czech Republic, 200. [5] J. G. Roederer, The physics and psychophysics o music : an introduction, 3rd ed. New York: Springer-Verlag, 995. [6] K. C. Pohlmann, Principles o digital audio, 3rd ed. New York: McGraw-Hill, 995. DAFX-6