OVERVIEW AND REFERENCE

Transcription

1 V I R T U A L A U D I O S P A C E with integrated Dense-Array EEG OVERVIEW AND REFERENCE Andrew Butcher, M.Sc. Matthew Tata, Ph.D. Canadian Centre for Behaioural Neuroscience The Uniersity of Lethbridge

2 Table of Contents Audio Hardware! Studio Monitors! Audio Interface! EEG Hardware and Software! Audio Software! Core Audio! The AudioGraph and SimpleAudioGraph Libraries! Timing and the arious models of running an experiment! Panning! PanningDemo 1! Integration with EGI s Netstation! The Netstation.h library! Timing and the benefits of scheduling! Resources! iii iii iii i i i i ii i

3 Oeriew The Virtual Audio Space at the Uniersity of Lethbridge was designed by Andrew Butcher and Matthew Tata in The goal was to create a system capable of reproducing complex and dynamic auditory displays under programatic control and with millisecond timing accuracy suitable for EEG experiments. The system consists of up to 14 speakers controlled by a digital audio interface on a Mac OSX computer. A software library called AudioGraph was created to proide easy access to OS X s CoreAudio frameworks. A second library called SimpleAudioGraph was created to proided easy access to AudioGraph, although the experienced programmer might choose to bypass this library entirely. The system is designed to integrate with the Electrical Geodesics (EGI) Netstation EEG acquisition software and hardware. To this end, a third library called Netstation proides methods to interact with Netstation programatically and send time-stamped triggers for EEG experiments. Considerable effort was dedicated to achieing the lowest possible latency jitter between stimuli and their associated timestamps. This is why some aspects of the approach might seem odd or inefficient. Howeer when done right, the system achiees a lag of 5 ms +/-1 ms between stimuli and trigger timestamps. ii

4 Audio Hardware Studio Monitors The speakers you use depend largely on the degree of precision and control you need oer the fidelity of the sounds you will be trying to reproduce. For some applications the simplest desktop speakers might be sufficient. For auditory psychophysics it is desirable to hae high-quality speakers with a flat frequency response oer a broad range of frequencies. The Virtual Audio Space at the Uniersity of Lethbridge uses up to 14 Mackie HR-264 Actie Near-Field monitors. Actie monitors are probably preferable because they are independent of the load at other speakers and you don t need to deal with separate amplifier hardware. The main drawbacks are: they are expensie, you end up with lots of power cords lying around and you can run into 60Hz noise problems in EEG if you re not careful. Audio Interface A digital-audio interface is a deice that extends the audio capabilities of your computer. Think of it like an external audio card. Most good ones were historically firewire deices, although USB deices are becoming popular and good. The AudioGraph and SimpleAudioGraph libraries are entirely agnostic about the audio interface. The code you write to control audio presentation will likewise be almost, but not entirely, agnostic about the interface. For the most part the Mac OS takes care of the interface ia the AudioMIDI setup utility aailable in many OS X ersions. Once your hardware is configured the channels of the deice are exposed to CoreAudio and we can directly interact with them ia our audio graph. A good audio interface will hae seeral analog outputs, ultra-low (studio quality) latency and accurate A-to-D conersion oer a broad frequency range. Many manufacturers make such deices. We use the M-Audio Firewire 410. This has 8 analog outputs to drie speakers so we daisychain two of these together and link them as aggregate deices in AudioMIDI Setup. This is where your code might hae to take some hardware-specific steps. For example, on the M-Audio Firewire 410, channels 9 and 10 are SPIDF (which we don t use). Thus on the aggregate deice that CoreAudio knows about, the 16 analog outputs aailable are [0,1,2,3,4,5,6,7,10,11,12,13,14,15,16,17]; Other hardware will hae other quirks. iii

5 EEG Hardware and Software The EEG hardware consists of an EGI 128-ch amplifier and A-to-D conerter to which geodesic dense-array nets are attached. Data are acquired on an imac with the Netstation software. It is this acquisition system that is the target of time-stamped eent triggers generated optionally by code that you link to the Netstation library. Many EEG systems try to send triggers in real time and emphasize achieing low time-of-flight latencies. EGI s approach is entirely different. It is reasoned that as long as the stimulus presentation and EEG acquisition computers hae synchronized clocks, then real-time triggers are unnecessary and problematic. Instead, eents generated by the presentation system are time stamped according to the presentation computer s clock and forwarded to Netstation ia TCP/IP. Netstation schedules them flexibly into the EEG data. It doesn t care when the packets arrie - it just writes them into the data. In principle one could run an entire session, record the triggers on the stimulus presentation machine, and then upload them at the end of the session before closing the Netstation session file. In practice we work the other way around. We synchronize the clocks on the two systems and then immediately pre-schedule all the known triggers by sending them to Netstation before the stimulus eent has actually happened. This irtually eliminates the problem of clock drift because the triggers are time stamped within a few milliseconds of clock sync. i

6 Audio Software Core Audio! Core Audio is Apple s OS leel audio framework. The API exposes methods to create notional audio systems called graphs to which functional modules called audio units can be attached. Our use of CoreAudio is ery simple: We create a single instance of a graph object, attach one or more file player units, attach.wa files to these file player units, and put a matrix mixer unit in between the file players and the output channels so that we can control the gain with which each sound source is sent to each speaker. By contrast, CoreAudio also forms the basic framework for sophisticated software such as GarageBand. The AudioGraph and SimpleAudioGraph Libraries The AudioGraph library is a set of C++ methods that call into the CoreAudio framework. These set up a graph with a gien number of input and output channels, attach any audio units that are needed, interact with these units, etc. This library probably exposes all or most of what the typical deeloper might need to create irtual-reality immersie audio with any number of panning techniques (we e used arious ersions of Ambisonics and Vector-Based Panning; see Andrew Butcher s M.Sc. thesis for details). Because the detail exposed by AudioGraph is probably more than the simplest audio presentations require, a second library was created to streamline the basic functionality of AudioGraph. This library is called SimpleAudioGraph. It simplifies by wrapping seeral steps into calls such as prime() and reset() and play(). SimpleAudioGraph probably proides all the access one needs to create simple auditory experiments, een ones that feature multiple sound streams in motion. Timing and the arious models of running an experiment In EEG, timing is eerything. Fortunately the same applies to studio audio and CoreAudio was meant to gie deelopers tools to create Pro Audio software to run on the Mac OS. Consequently, CoreAudio proides ways to get ery good control oer what the audio hardware is doing and when it does it. There are two basic approaches we can take in designing our stimulus control program: Programmatically cue and play brief sound eents as needed. One way to arrange a series of audio eents is to create a different.wa file for each stimulus and attach each to a different fileplayer unit. For example one might want to play a series of 100-ms tones. The tones could be generated by any means you choose so long as it can write a.wa file to disk. PanningDemo 1 shows how to do this with some generic mono.wa files (although these are probably rather longer than would be typical). In this model the experiment control program keeps track of what needs to happen and when. If, for example, a 1000Hz pure tone is required, then it tells the fileplayer unit that has the 1000Hz.wa file attached to it to play. This is probably the most intuitie approach, but it isn t really the best way to use CoreAudio. Pre-build and pre-schedule a series of audio eents contained within one (or a few) long.wa files. Notice that the play() method of SimpleAudioGraph takes two arguments. These are the fileplayer unit to play, and the system time in nanoseconds at which to play it. If you send it zero, CoreAudio will do its best to start playback immediately, but in practice this can neer be perfectly instantaneous. Lags on the order of tens of milliseconds are entirely possible. A better approach is to tell CoreAudio to start playback at a known time in the future. CoreAudio will then schedule playback on a separate thread (called the render thread) and will do a ery good job of starting playback when you

7 told it to. For example, imagine you want a series of ten pure tones to play, one eery 1000 ms. You might pre-build a 10,000 ms long mono.wa file in whicheer is your faorite audio package (MATLAB and Praat are good options). You can attach this.wa file to a fileplayer unit and tell it to start playing at a known system time in the future (say 5 seconds from the present time). CoreAudio will start playback of the entire.wa file at almost exactly the requested time (usually within a fraction of a millisecond). Since you know when the stimuli occur in the.wa file with sample-leel accuracy (because you built it), simply add those times to the requested start time of playback. Now you know the system time at which each eent will occur with sub-millisecond precision. Notice that the Netstation library for communicating EEG triggers to the EEG acquisition computer works similarly in that you can pre-schedule trigger eents proided you know when they will occur. By also prebuilding the entire sequence of audio eents, and thus knowing the exact system time at which each eent will occur, it is possible to send ery precise timestamps to Netstation. These are passed along before the stimuli een start playing! It is important to recognize the limitations to this approach and spot where a hybrid approach might be necessary. For example the pre-built sequence of stimuli can t interact with the listener (although one could turn up the olume on different fileplayer units in near-real-time - an approach which has not been explored). One might pre-build a sequence of tones to which a listener must respond and attach this to a fileplayer unit. Two different feedback tones - one for correct and one for incorrect - might be associated with two different fileplayer units. The code might then programmatically control which feedback stimulus was played based on responses from the listener. Notice also that coneying those responses to Netstation is usually desirable. The Cocoa framework makes a timestamp aailable for a gien keypress eent which can be used to get ery accurate response triggers as well. Panning! THIS SECTION IS A WORK IN PROGRESS PanningDemo 1! THIS SECTION IS A WORK IN PROGRESS i

8 Integration with EGI s Netstation The Netstation.h library! THIS SECTION IS A WORK IN PROGRESS. Timing and the benefits of scheduling! THIS SECTION IS A WORK IN PROGRESS ii

9 Resources Naturally this document and the examples proided aren t going to be a complete tutorial. There is a lot of other information for you to seek out. The CoreAudio System Guide proided by Apple along with the CoreAudio SDK is a critical reference. For more background reading, especially about the design process and details of the Virtual Audio Space at The Uniersity of Lethbridge, look into Andrew Butcher s M.Sc. thesis proided at For papers that hae used the system see Ponjaic-Conte et al. (2012) NeuroReport and Butcher et al. (2010) Hearing Research. iii