Institutionen för datavetenskap

Transcription

1 Institutionen för datavetenskap Department of Computer and Information Science Master thesis Advanced Real Time sound techniques for Virtual Reality headsets by Johan Yngman and Emil Westergren Linköping Civilingenjör Datateknik ISRN: LIU IDA/LITH EX A 14/014 SE Advisor: Anders Fröberg 1

2 Abstract Virtual reality headsets, like the Oculus Rift, uses visual tricks to enhance the feeling of immersion in virtual reality applications. This thesis explores the possibilities of creating an immersive sound experience to go along with the visuals. A sound propagation model designed for real time simulations is presented along with techniques to implement binaural synthesis. The thesis includes an implementation of a 3D cube world written in C# using OpenGL for graphics and FMOD for sound. The implementation is evaluated in the context of realism and possible improvements and optimizations are suggested. 2

3 Table of contents Abstract Table of contents Background Purpose and problem statement Theory What is a Virtual Reality (VR) headset? Binaural sound What is Binaural sound? Head related impulse response (HRIR) Head related transfer functions (HRTF) Virtual binaural sound effect Model for sound propagation General effects Sound speed Geometric spreading Air absorption Direct path Early reflections Reverberation Distinct echo Transmission Other sound phenomena Diffraction Refraction Interference Implementation Synthesizing binaural sounds Lomont s Fast Fourier Transform (FFT) HRTF database HRTF interpolation Delay filter Sound propagation Ray casting algorithm Direct path transmission Reflections Ray casting with GPU First order reflections First order reflections algorithm Second order reflections Directional reflections Reverberation Blending listener and source environment Directional reverb Result Direct path transmission Reflections Reverberation Binaural filter Delay filter Optimizations Conclusion Discussion References 3

4 Background Virtual reality (VR) is as of today expected to seriously establish itself in the market of gaming. While the realism and quality of visual computer graphics have quickly evolved over the years to be sufficient for VR, the sound techniques have remained primitive without any major evolution towards increased realism. Purpose and problem statement The purpose of the thesis is to explore the possibilities of implementing real time dynamic sound propagation techniques in combination with binaural sound synthesis for VR headsets. Which techniques are suitable for real time dynamic sound propagation? How can binaural sound synthesis be used in combination with dynamic sound propagation? How can the above techniques be further optimized? 4

5 Theory What is a Virtual Reality (VR) headset? When writing this, the most anticipated VR headset is the Oculus Rift. The company, Oculus, are yet to release a consumer version but have created a development kit that developers can use to create VR content before the consumer release. This thesis work is primarily targeted towards this development kit. An Oculus Rift, Development kit The development kit is a pair of goggles that presents a stereoscopic 3D image with 110 degrees field of view to the user. It also tracks the users head rotation with low latency and high precision. This creates a strong feeling of immersion in the virtual world and a sense of actually being in another place. 1 1 The Oculus Rift and Immersion through Fear. ADAM HALLEY PRINABLE. (accessed ) 5

6 Binaural sound This section describes an algorithm for synthesizing the binaural effect in real time and introduces the terms used later in the implementation section. What is Binaural sound? Binaural sound can be created by placing microphones in the ears of a real person or on a dummy head. Whenever a sound is recorded with this setup, the left ear will record the left part of a stereo channel and the right ear will record the right part. If one would listen to this recording using stereo headphones, it will sound close to what it would sound like in reality. 2 A binaural recording setup The effect works best if the recording is done with a perfect body model of the listener. This is because every person have a unique body and ear shape which affects how the sound is picked up by the microphones. This means that no one will perceive a sound in the exact same way as another person. The way that a sound is manipulated for a specific person is called the Head related transfer function (HRTF). Head-related impulse response (HRIR) The Head related impulse response (HRIR) is the recorded ear impulse response from a specific angle. Head-related transfer functions (HRTF) The way that a sound is manipulated for a specific person, and angle, is called the Head related transfer function (HRTF). The HRTF is the fourier transform of the Head related impulse response. 2 The Inventor of Stereo: The Life and Works of Alan Dower Blumlein Robert Charles Alexander (Focal Press, 1999) 6

7 Fast Fourier Transform (FFT) The Fast Fourier Transform (FFT) is an algorithm to calculate the Discrete Fourier Transform (DFT). It transforms a signal from the time domain to the frequency domain. 3 Virtual binaural sound effect To apply the binaural effect to a sound in a virtual world, an HRTF/HRIR is needed for each angle that the sound can be heard from. Or rather, as many as it takes to make the difference between two neighbouring HRTFs/HRIRs unnoticable. The sound also have to be in mono. To simulate that a sound is heard from a certain angle one can convolve the HRIR with the sound. This is slow since the time complexity of convolution is O (n 2 ) where n is the number of samples. It is possible to reduce the time complexity to O (n log n) by applying a Fast Fourier Transform (FFT) to the sound and do pointwise multiplication in the frequency domain with the corresponding HRTF. Pseudo code for creating the binaural effect in a simulation 4 These steps are repeated every time the simulation updates: Calculate the angle between the listener and the sound source Select the HRIR for the closest matching listening angle Convolve the sound with the HRIR Play the convolved sound Interpolating between HRTFs Interpolation is needed for at least two reasons. Whenever the angle change too drastically, there might be a stutter in the sound. A simulation which runs at 60 frames per second has a ~16.6 millisecond delay between each frame. That timespan is enough to turn your head over such a long distance so that this sound artifact might appear. The other reason is because of the denseness of the available HRIRs. There might be a noticeable difference between two neighbouring angles if there aren t enough HRIRs. This could be solved by approximating new HRIRs for non existing angles. 5 3 Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest. Introduction to Algorithms. Third edition. Clifford Stein. The MIT Press HRTF Measurements of a KEMAR Dummy Head Microphone, Bill Gardner and Keith Martin (MIT Media Lab 1994) (accessed ) 5 Two approaches for HRTF interpolation. Gustavo H. M. de Sousa, Marcelo Queiroz (accessed ) 7

8 Model for sound propagation When a sound source emits sound waves the waves are affected in various ways by the environment before reaching the listener. These effects play an important role in how the sound is perceived by the human brain. General effects Sound speed The speed of sound is highly dependent on the medium it travels through. Even in air the speed is not constant since the condition of the air will vary. This thesis will however neglect this and assume a constant sound speed of 343 m/s (approximate speed in air) regardless of the medium. Geometric spreading The intensity of a sound will decrease over distance due to geometric spreading. The loss is independent of frequency and, assuming a point source, the reduction will be relative to the inverse square distance. In ideal conditions (a point sound source with nothing but air surrounding it) the following formula can be used: r 2 L 2 = 10 lg( 1 r 2 2 ) + L 1 where L 2 is the intensity (in db) at distance r 2 and L 1 is a known intensity at distance r 1. This results in a 6 db reduction for each doubling of the distance from the source. 6 Air absorption All sound waves are also affected by energy absorption when travelling through air. This will cause the intensity of the sound (amplitude of the wave) to decrease. The energy loss depends on the frequency of the wave. A higher frequency will suffer a greater energy reduction over distance. The reduction is also dependent on the temperature and humidity of the air. Air absorption will however only be noticeable when dealing with long distances or very high frequencies and can in many cases be neglected. 3 Direct path The direct path is the straight path between the listener and the sound source. This is often where the first sound waves to reach the listener comes from since it is the shortest possible path. 6 Sound propagation, Simon Fraser University studio/handbook/sound_propagation.html (accessed ) 8

9 Early reflections Sound waves that reflect once or twice off a surface and reach the listener within approximately ms after the first perceived wave are called early reflections. Even though these waves do not reach the listener simultaneously they will not be perceived as separate but instead tend to reinforce the sound. They will however give the listener information about the environment, so called spatial impression. The reflections also usually originates from other directions than the 7 8 sound source which will affect how the listener perceives the sound. Reverberation Reflected sound waves reaching the listener after the early reflections are called reverberation (or late reflections). Consisting of a large number of echoes reaching the listener in very quick succession these reflections can not be distinguished from each other but are instead perceived as a persistence of the original sound rather than a new sound. Over time these waves will arrive with gradually lower intensity due to the energy loss from a higher number of reflections. This will produce a fade in the intensity of the reverb. The characteristics of the reverb are highly dependent on the environment, specifically the size of the room and the materials used in the room. In a small room the waves will quickly bounce off the walls a number of times and therefore quickly lose enough energy to be inaudible. In a large room the energy reducing reflections will be less time frequent which produces a longer lasting reverb. The material of a reflection point determines how much energy is lost. Soft materials will likely cause a high energy reduction while the opposite is the case for hard materials. This property is defined by an absorption coefficient which is a value between 0 and 1 where 0 will absorb 0% of the energy and 1 will absorb 100% of the energy. The time for a sound to become inaudible is called reverberation time. This is defined as the time for a sound to drop 60 db from its original intensity. The most common approximation used to calculate this time is Sabine s equation: T = V Sα where T is the reverberation time (in seconds), V the volume of the room, S the surface area in the room and α the average absorption coefficient. 9 Distinct echo Reflections reaching the listener with a distinct delay (at least 100 ms) after the first perceived wave and with sufficient intensity to distinguish itself from reverberation will be perceived as a 7 Physical audio signal processing, Julius O. Smith III (accessed ) 8 Room acoustics, McGill University (accessed ) 9 Calculation of reverberation time, Davidson physics (accessed ) 9

10 separate sound by the human brain. This produces a distinct echo which will not mix with the original sound as in the case with reverberation. 10 Transmission Sound transmission is the basic principle of sound waves travelling through a medium. Part of the sound the listener hears will usually (unless total isolation) travel directly from the sound source to the listener. Even though a direct path is blocked by an obstacle between a source and a listener, sound waves may still be able to travel through the obstacle. This ability is highly dependent on the material and thickness of the obstacle. Lower frequencies have a better ability to pass through obstacles which is very noticeable when opening and closing a door to a room with a sound source (e.g. music player). 11 Other sound phenomena Diffraction Diffraction is a known wave phenomena and therefore also applies to sound waves. It gives sound waves the ability to bend around obstacles and pass through small openings. The result is that even though a direct path does not exist between the sound source and the listener, 12 sound waves can still reach the listener without the need to reflect off a surface. This phenomena will be neglected in this thesis. Refraction When a sound wave travels from one medium to another, refraction causes it to change its direction. This phenomena will be neglected in this thesis. 10 Echo vs. Reverberation, The Physics Classroom (accessed ) 11 Sound transmission, Gerald S. Wilkinson (accessed ) 12 Reflection, Refraction, and Diffraction, The Physics Classroom (accessed ) 10

11 Interference Interference is the phenomena caused by multiple waves affecting each other when travelling in the same medium. Depending on the difference in phase of the waves, they can either cause constructive interference (the amplitudes will be added) or destructive interference (the amplitudes will be subtracted). This phenomena will be neglected in this thesis. Implementation The environment for the implementation consists of a cube based 3D world written in C# and OpenGL. The FMOD audio library has been used for playing and modifying audio. FMOD includes DSP effects for low pass filtering and reverb and also support for custom DSP effects. Screenshot from the cube world. Synthesizing binaural sounds This chapter describes how the binaural effect is implemented. Lomont s Fast Fourier Transform (FFT) An existing C# implementation of the Fast Fourier Transform (Chris Lomont, 2010) is used. This particular FFT implementation was designed for sound synthesis and it uses a faster, one dimensional, more memory efficient version of the transform The Fast Fourier Transform, Chris Lomont (accessed ) 11

12 HRTF database The HRIR s used were recorded by Bill Gardner and Keith Martin (MIT Media Lab 1994). They recorded 368 impulse responses on a half sphere around the left ear, using a KEMAR dummy head at a fixed distance of 1.4 meters. This data can be used for both the left and right ear since the impulse responses are symmetrical. 14 Each impulse response is stored in a wav file and there are 128 samples per ear in each file. The angle relative to the listener is embedded in the file name using polar coordinates. The vertical angle is called elevation and the horizontal is called azimuth. 15 When the simulation starts, every HRIR is loaded into memory and transformed using FFT to get a HRTF for every angle. A direction vector is calculated which matches the cube world coordinate system for every HRTF. This vector is later used to select the closest matching HRTF for the different listening angles. 1. // Calculatedirectionvectortobeusedwhenselectinghrtf 2. Vector3d=newVector3(0, 0, 1); 3. floatradazim=(azimuth/ 180.0f)*(float)Math.PI; 4. floatradelev=2.0f*(float)math.pi-(elevation/ 180.0f)*(float)Math.PI; 5. d=d* Matrix4.CreateRotationX(radElev); 6. d=d* Matrix4.CreateRotationY(radAzim); directionvector=d; 9. directionvector.normalize(); Code snippet showing how a direction vector is calculated from the azimuth and elevation of a HRIR. The values for elevation and azimuth are extracted from the file name. To then select the correct HRTF, all the loaded transfer functions are iterated and the one with the closest matching direction vector is selected. 14 HRTF Measurements of a KEMAR Dummy Head Microphone, Bill Gardner and Keith Martin (MIT Media Lab 1994) (accessed ) 15 HRTF Measurements of a KEMAR Dummy Head Microphone, Bill Gardner and Keith Martin (MIT Media Lab 1994) (accessed ) 12

13 HRTF interpolation This implementation focus on interpolation over time. It is needed because sound artifacts can be heard when the angle between the listener and the sound source differ too much between frames. Instead of instantly switching to a new HRTF every frame, all the samples in the previous HRTF are iterated and pulled towards the new ones. This makes the transition to the new HRTF last for several frames. 1. publicvoidinterpolatetowards(hrtfother) 2. { 3. for(inti=0; i<samples.length*2; i++) 4. { 5. doublediffleft=other.leftfourier[i]- LeftFourier[i]; 6. doublediffright=other.rightfourier[i]- RightFourier[i]; LeftFourier[i] +=diffleft/ transition; 9. RightFourier[i] +=diffright/ transition; 10. } 11.} This code snippet shows how the varying HRTF is pulled towards the new one (called other). The code is called once per frame. The speed of the transition can be altered by changing the transition variable. Interpolation for non existing HRTFs is ignored in this thesis. 13

14 Binaural filter implementation The sounds are played with a sample rate of 44.1 khz. This means that when the simulation runs at 60 frames per second, at least = 735 samples needs to be processed every frame 60 for each sound playing. The filter function works in chunks of 128 samples which is comfortable since it s the size of the HRTFs. It is therefore needed to call the filter function at least 6 times per frame for each sound. More than 6 is recommended to avoid sound artifacts when the frame rate drops. It is a tradeoff between responsiveness and robustness. A simplified figure of the filter function. The filter function works like this: 128 samples from a playing sound is sent as argument to the filter function. FFT is then applied to the samples, one time for each ear. The sound is multiplied with the interpolated HRTF and the result is inverse transformed. The result is the same as convolution with the sound and the impulse response. The result of this convolution is 255 samples long. The first 128 of those samples are sent to the outbuffer (and out through the headphones) and the rest is saved in a temporary buffer. The content of this temporary buffer is added to the output the next time the filter function is called. The reason for this is that the output buffer size of a filter function in FMOD has to be the same as the size of the input buffer. 14

15 Delay filter A custom delay filter is used to delay the sound based on the distance travelled before it reaches the listener. It uses a rotating buffer with index pointers which are updated when the delay is changed. s oundspeed = s amplerate = 343 m/s hz d elaysamples = samplerate distance soundspeed 1. uintsaveindex=(currentindex+delaysamples)% MAX_DELAY; // Savetodelaybufferandsetoutbuffers 4. for(inti=0; i<length; i++) 5. { 6. delaybufferleft[(i+saveindex)% MAX_DELAY] =inbuffer[i* 2]; 7. delaybufferright[(i+saveindex)% MAX_DELAY] =inbuffer[i* 2+1]; outbuffer[i* 2] =delaybufferleft[(i+currentindex)% MAX_DELAY]; 10. outbuffer[i* 2+1] =delaybufferright[(i+currentindex)% MAX_DELAY]; 11.} currentIndex=(currentIndex+length)% MAX_DELAY; Code snippet from the delay filter. saveindex keeps track on where to save the outgoing samples in the delay buffers. delaysamples is the number of samples to delay the sound. MAX_DELAY is the longest possible delay and is set to samples in this implementation (~0.37 milliseconds) 15

16 Sound propagation To deal with the multiple number of directions and effects of a single sound, the sound source is divided into multiple internal sound sources. These internal sources require an individual sound channel and DSP effects. Overview of the sound source structure Ray casting algorithm Only one ray per sound source needs to be casted for the direct path calculations, hence the ray casting is not a performance issue. The implemented algorithm takes advantage of the simple geometrical nature of the cube world and uses Bresenham s line algorithm for three dimensions to trace the intersected cubes in a line The Bresenham Line Drawing Algorithm, Colin Flanagan (accessed ) 16

17 Direct path transmission In most cases when a sound source is heard, part of the sound is transmitted through the direct path. Without any obstacle blocking the direct path this would result in the sound reaching the listener relatively unaffected. What is more important are the cases where the direct path is blocked, particularly when the direct path is the only part of the sound reaching the listener, i.e. no reflections or diffractions arrive. The most important effect when the sound transmits through obstacles is the possible loss of high frequencies. One could therefore treat the direct path as a low pass filtered source, where the cutoff frequency is dependent on the properties of the obstacles in the direct path. For example, a perfectly sound isolated wall would block all frequencies while a thin layer of fabric may have almost no effect at all. To determine if the direct path should be low pass filtered, and with what cutoff frequency, the direct path needs to be traced. This is achieved by using the implemented ray casting algorithm which detects obstacles and takes into account a transmission coefficient for the specific obstacle. The coefficient is a value between 0 and 1 where 0 means no frequencies can be transmitted and 1 means all frequencies can be transmitted. For multiple obstacles the coefficients are multiplied. The maximum human hearing frequency is approximately 20 khz and by using the calculated coefficient the cutoff frequency can be calculated by the following formula: f c = * K total where f c is the cutoff frequency and K total is the total obstacle coefficient. The direct path is treated as an individual sound source in combination with a delay, a low pass filter and a binaural DSP. Reflections Dynamic reflection calculation is by nature a ray casting problem and by using the implemented ray casting algorithm all first and second order reflection points can be found with ease. This however quickly becomes computationally expensive with n rays casted for the first order reflections resulting in a total of n 2 rays for each point source. For decent results at least n=100 is needed resulting in a total of rays for each sound source. Even though this can easily be parallelized using multiple CPU cores it would still require a very efficient ray casting algorithm and would be highly dependent on the geometrical complexity of the environment. 17

18 Ray casting with GPU This thesis explores the possibility to calculate the reflections on the GPU instead, taking advantage of the efficient parallelization nature of the GPU. Complex geometry will still affect the performance but with a much lower factor than with the CPU based ray casting algorithm. First order reflections The problem of calculating first order reflections is a similar problem as calculating point light shadows in a 3D application. This is because both problems need to find the points in the environment which can be seen from both the position of the source and the position of the viewer (or listener in this case). Note that the normal vector of the point is neglected here meaning the reflection points will not necessarily be perfect reflections but rather points where a reflection is highly probable. 17 The solution for shadow calculations is called shadow mapping and a similar technique is used in this implementation. One advantage in this case is that very low resolutions are sufficient for the sound reflections (compared to shadow calculations) and these can easily be scaled for increased precision or better performance. This implementation uses textures for the listener cube map and for the sound sources. This is equivalent of 32*32*6 = 6144 casted rays for the listener and 16*16*6 = 1536 rays for each sound source. First order reflections algorithm Render distance cube map for the listener For each sound source: Render distance cube map with listener distance cube map in VRAM: In pixel shader: Output the distance to the current pixel in alpha channel (needed later for reverb estimation). Retrieve the distance value from the listener cube map in the direction from the listener to the current pixel position in the shader Compare this distance with the length of the vector from the listener position and the position of the current pixel If the distance is shorter from the listener s point of view then the point is not a reflection point: Output zero vector. If not then the point is a first order reflection point: Output the world position of the reflection as pixel data. The resulting textures are then downloaded from the GPU to the CPU and the reflection points are extracted. 1. #version Shadow Mapping, OpenGL Tutorial tutorial.org/intermediate tutorials/tutorial 16 shadow mapping/ (accessed ) 18

19 3. outvec4fragcolor; 4. uniformvec3soundposition; 5. uniformvec3listenerposition; 6. uniformfloatradius; 7. uniformsamplercubelistenerdistances; 8. uniformfloatlistenerradius; 9. uniformfloatbias; smoothinvec3worldPos; 12.smoothinvec3worldNormal; voidmain(void) 15.{ 16. //Vectorfromthesoundsourcetothepixelinworldcoordinates 17. vec3distancevec=worldpos- soundposition; //Outputthescalardistanceforreverbestimation 20. floatdisttosound=length(distancevec); 21. fragcolor.a=disttosound/radius; //Vectorfrompixeltolistener 24. vec3listenervec=worldpos- listenerposition; //Vectorfromthelistenertothesoundsource 27. vec3listenertosound=soundposition- listenerposition; //Invertzandytomakecubemapsamplingcorrect 30. listenervec.z=-listenervec.z; 31. listenervec.y=-listenervec.y; //Samplethenormalizeddistancetothepixelfromthelistener'sviewand scalewithradius 34. floatlistenerdist=texture(listenerdistances, listenervec)*listenerradius; 19

20 //Performthedistancecomparisontodetectreflectionpoint 37. if(listenerdist<length(listenervec)- bias) 38. { 39. //Noreflection 40. fragcolor.r=0; 41. fragcolor.g=0; 42. fragcolor.b=0; 43. } 44. else 45. { 46. //Reflection, outputworldposition 47. fragcolor.r=worldpos.x; 48. fragcolor.g=worldpos.y; 49. fragcolor.b=worldpos.z; 50. } 51.} Code showing the pixel shader for the first order reflection pass Second order reflections The computational complexity of second order reflections is very high even with a GPU based algorithm and it would not be feasible to repeat the above algorithm for each ray from the source. The second order reflections are therefore neglected in this implementation. Directional reflections Treating each reflection point as an individual sound source would produce a good result but this is not an option when each sound source needs heavy computations for its binaural synthesis. The number of sound sources each sound is divided into should be kept at a minimum to maintain decent performance. However, since reflections can reach the listener from any direction it is important to have some form of directional presence from the reflections. To achieve this the directions from the listener s point of view are divided into four areas according to the world axes: positive x, negative x, positive z and negative z. The y axis (vertical axis in the world) is neglected for optimization reasons. It also does not affect the listener as 20

21 much as the other axes since the human brain is much less sensitive to changes in vertical position of a sound source. 18 An average position for all reflection points contained in each of the areas is then calculated. This position is used as a sound source representing the average reflections from one of the mentioned world axes. The number of reflection points dictates the intensity of the sound source. The image illustrates how the reflection points are divided into areas around the listener. The sound from the reflection sources are independently delayed according to the total distance travelled. This can easily be calculated by simply adding the distance from the reflection source to the sound source to the distance from the reflection source to the listener and dividing by the speed of sound. This individual delay results in a very evident effect of directional reflections which is most noticeable when only one of the average reflection points is far away from the listener. Reverberation The extreme complexity of calculating reverberation ray by ray makes it much more feasible to use an approximation. The most important factors deciding the reverb in a room are the size of the environment and the materials in the room. For the reflection calculations the distances to the surrounding environment were calculated for both the listener and each sound source. Both these can be used for the reverb estimation. With the average distance the environment can be approximated as a sphere with the radius equal to 18 Spatial Sound Localization in an Augmented Reality Environment, Sodnik, Tomazic, Grasset, Duenser, Billinghurst OZCHI SpatialSoundInAR.pdf (accessed ) 21

22 4μ the average distance μ. The volume of the sphere therefore is 3 3 and the surface area 4πμ2. Inserting this in Sabine s equation results in: T = 0.017μ α where T is the reverberation time (in seconds), μ the average distance in the room and α is the average absorption coefficient. This is the formula used in the implementation. Blending listener and source environment The reverb is not only dependent on the environment of the sound source but also the environment of the listener. How much each affects the resulting sound depends on the ability of the sound source environment to absorb reflections. To blend these the following formula can be used: μ = βμ l + (1 β )μ s where μ is the blended distance average, β is the blending weight, μ l the average distance for the listener and μ s the average distance for the sound source. 19 β is dependent on the sound absorption in the environment around the sound source and can be calculated by β = 1 6 log(1 α) + 1 where β is the blending weight and α is the average surface absorption. This results in the listener environment being more important if the environment surrounding the sound source has a high surface absorption and vice versa. 1. publicvoidupdatereverb(floatsounddistance, floatlistenerdistance, ref FMOD.DSPreverb) 2. { 3. //Constantairabsorptionintheimplementation 19 Aural proxies and directionally varying reverberation for interactive sound propagation in virtual environments, Antani L, Manocha D (accessed ) 22

23 4. floatb=0.7f; //Blendtheaveragedistanceofthesoundenvironmentandthelistener environment 7. floatu=b* sounddistance+(1- b)* listenerdistance; //CalculatereverberationtimeaccordingtoSabine'sequation 10. floatt=(0.017f* u)/ 0.1f; //Setthedecaytime(reverberationtime) 13. reverb.setparameter((int)fmod.dsp_sfxreverb.decaytime, T); //Calculateandsetreverbdelay 16. floatreverbdelay=u/ f; 17. reverb.setparameter((int)fmod.dsp_sfxreverb.reverbdelay, reverbdelay); 18. } This code snippet shows how the reverb is updated. Directional reverb Using the already implemented sound sources for directional reflections, directional reverb can be achieved by simply adding individual reverb DSP effects to each reflection source. The individual reverb time for the reflection source is calculated as above but by only using distance values from the specific world axis for the calculation of the listener environment. This results in a directionally dependent reverb. 23

24 Result Direct path transmission The direct path sound source gives the listener a steady directional sense of the sound position and the low pass filter creates a realistic muffled effect for occluded sounds. There are some issues however; the transition from unfiltered to filtered and vice versa can feel abrupt and unnatural and additional work on interpolation is needed for a smooth experience. The technique is dependent on a ray casting algorithm for the environment but since only one ray per sound source needs to be casted, performance should not be an issue. Reflections The approximation of using four reflection channels with individual reverb and delay is sufficient to give the listener a directional sense of reflections, echoes and reverb. For example, the reflections in combination with a muffled direct path can change the primary sense of the sound direction for the listener. This is expected when a sound source is around a corner. The direct path will be blocked so the full range of frequencies will come from reflections and diffractions which alters the listener s perception of the sound direction. The sound direction will be perceived as to the corner instead of to the sound source. The individual delay and reverb for the reflection sources make sure the directions of significant differences in delay and reverb can be perceived by the listener. The absence of second order reflections naturally causes too little sound to reach the listener, which is most apparent when no first order reflections exist. The real time calculations for real second order reflections is however an expensive task and is preferably avoided. The problem can be slightly disguised by using a low volume non directional ambient channel with only a delay effect applied. The volume can be relative to the distance to the source but should have a low maximum to not intervene too much with the physically correct sound. The reflections also suffer from too abrupt changes in some cases and need to be smoothened by interpolation. Reverberation The dynamic reverb estimation produces a noticeable difference in reverb when moving a sound source to a different environment. The transitions are smooth and the blending of the listener and sound environment makes sure the technique can handle more complex scenarios. The environment analysis can be easily implemented on the GPU with low resolution cube maps, eliminating the need of ray casting on the CPU. 24

25 Binaural filter The quality of the interpolation, as it is currently implemented, depends on the transition rate. With a fast transition the sound might crackle but it is more responsive. With a slow transition there is no crackle but the transition can be noticed. In this implementation, the transition rate was set to the fastest possible where the sound never crackled. The transition is slightly noticeable on rapid head movements. It is hard to make objective comments about how much the binaural filter enhances the spatial location feeling of the sound. The experience varies from person to person since everyone have their unique HRTF. Even though the HRTFs from the KEMAR dummy head seems to be sufficient for most people, the experience could be enhanced by using tailor made HRTFs for each listener. Since all the impulse responses used in this thesis were recorded at a fixed distance of 1.4 meters, an important feature of binaural recordings were lost. That is all the HRTFs for different distances. When the distance is greater, the HRTFs stay almost the same, but they change dramatically closer to the head. This means that without the distance variable, certain effects like someone whispering in your ear, will not seem realistic. It is hard to record HRTFs for both every angle and distance with good results. It might be possible to run a simulation using a model of a human head, approximate the HRTF for each angle and distance and then load them all at runtime. The listener experience could be greatly enhanced by using personalized HRIRs. However, to record impulse responses for different angles and distances, one would need a sound chamber and expensive equipment. The HRIRs would probably also differ a lot with different recording setups. A more efficient way to personalize the HRIRs would be to estimate them using ear and body measurements. This would ensure consistency but might not achieve the same level of realism as a good recording. Another thing worth exploring is how the length of the HRTFs affect the spatial quality of the sound. A long HRTF is computationally heavy to convolve but might result in a richer experience. Delay filter When moving the listener towards or away from a sound source, the sound will sometimes crackle. This is because the filter instantly switches to another point in time of the sound. In reality there would be a doppler effect and the sound would stretch or shrink. For future improvement, one could recreate this doppler effect in the filter function. Another solution to the sound crackles could be to have two alternating delay buffers and interpolate between them. Or mix between them when changing the delay to smooth out the transition. This would however create a small flanger effect. 25

26 Optimizations Pixel transfer optimization Transferring pixels from the graphics card to the RAM is usually an expensive process and may bottleneck an application by making the CPU wait until the transfer is completed. This can be improved by using Pixel Buffer Objects which allow the memory transfer to be performed asynchronously and not stall the CPU. The pixel transfers for each sound source in this implementation is not necessarily needed every single frame. All sound sources can be split into two groups and each frame the groups will alternate between initiating a pixel transfer and actually reading the pixels. In optimal conditions this may almost completely remove the slow process of transferring the pixels to the RAM. The drawback is that the data from the pixel read will be one frame old, this should however not be noticeable in this specific application. Reducing draw calls with geometry shader In the above implementation, each sound source (and the listener) updates a cube map with information about distance and/or reflection points. This update requires 6 draw calls for each cube map (one call per cube face). This is possible to reduce to one single draw call by using a geometry shader. This optimization can be useful when the number of draw calls in the application is critical. Convolution on GPU The performance of the sound convolution was greatly improved when FFT was used instead of HRIR convolution. A way to improve it even further could be to apply the FFT on the GPU. In this thesis it could be done by uploading the samples in a texture and make a draw call from the binaural filter function. The result could be written and later read from a texture render target. 26

27 Conclusion Major improvements over conventional sound techniques in gaming have been implemented and shown to run in real time with at least 60 FPS. Several sound propagation phenomenon have been implemented; direct path transmission, first order reflections and reverberation estimation. These have been successfully combined with binaural synthesis to achieve three dimensional directional sound perception. The computation costs are not negligible however and may need to be compromised with other aspects of realism in applications requiring heavy computations. Optimizations are therefore key and the thesis encourages further analysis of possible optimizations. Discussion This thesis has covered the theory behind sound propagation and binaural synthesis and explored the possibilities to use this theory in development towards VR headsets like the Oculus Rift. An implementation has been programmed with dynamic real time sound propagation in combination with binaural synthesis where the results have been evaluated. With the new generation of VR expected to transform the possibilities of visual immersion, an evolution in sound immersion should be highly anticipated. To achieve perfect VR all human senses need to be realistically triggered and with visual graphics approaching photorealism and 3D perception, sound should definitely be next in line. Not much have happened in sound immersion in gaming in the last decade except for a few exceptions. Binaural sound in games have been a hard sell because of the headphone requirement. However, this requirement makes more sense when already wearing a headset for the visuals. It seems like a logical step to take advantage of the head tracking for a richer, more immersive, sound experience. The result of the thesis have encouraged the writers to investigate the matter further. We believe realistic sound immersion will become more important in the future when VR becomes mainstream and our results show that a big improvement is definitely feasible in sound immersion for gaming. A slight problem is the competition over computational power with graphics, physics and other game logic. Optimizations and the growing capability of graphics cards should help remove this problem and in VR a slight drop in graphics might be worth a significant boost in sound immersion. 27

28 References The Fast Fourier Transform, Chris Lomont (accessed ) HRTF Measurements of a KEMAR Dummy Head Microphone, Bill Gardner and Keith Martin (MIT Media Lab 1994) (accessed ) Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest. Introduction to Algorithms. Third edition. Clifford Stein. The MIT Press The Inventor of Stereo: The Life and Works of Alan Dower Blumlein Robert Charles Alexander (Focal Press, 1999) Two approaches for HRTF interpolation. Gustavo H. M. de Sousa, Marcelo Queiroz (accessed ) A Structural Model for Binaural Sound Synthesis. C. Phillip Brown and Richard O. Duda (accessed ) The Oculus Rift and Immersion through Fear. ADAM HALLEY PRINABLE. (accessed ) Sound propagation, Simon Fraser University studio/handbook/sound_propagation.html (accessed ) Physical audio signal processing, Julius O. Smith III (accessed ) Room acoustics, McGill University (accessed ) Calculation of reverberation time, Davidson physics (accessed ) Echo vs. Reverberation, The Physics Classroom (accessed ) Sound transmission, Gerald S. Wilkinson (accessed ) Reflection, Refraction, and Diffraction, The Physics Classroom (accessed ) The Bresenham Line Drawing Algorithm, Colin Flanagan (accessed ) Shadow Mapping, OpenGL Tutorial tutorial.org/intermediate tutorials/tutorial 16 shadow mapping/ (accessed ) Spatial Sound Localization in an Augmented Reality Environment, Sodnik, Tomazic, Grasset, Duenser, Billinghurst OZCHI SpatialSoundInAR.pdf (accessed ) Aural proxies and directionally varying reverberation for interactive sound propagation in virtual environments, Antani L, Manocha D (accessed ) 28