Solving Audio Subsystem Challenges Using a Crossfade/Level Controller Algorithm in Portable Media Applications

Size: px

Start display at page:

Download "Solving Audio Subsystem Challenges Using a Crossfade/Level Controller Algorithm in Portable Media Applications"

Sydney Blair
6 years ago
Views:

1 Soling Audio Subsystem Challenges Using a Crossfade/Leel Controller Algorithm in Portable Media Applications Danny Jochelson DSP Solutions R&D Digital Audio Branch Texas Instruments, dsjochel@ti.com Jason Kridner Imaging and Audio Group Texas Instruments, jdk@ti.com

2 Agenda Motiation Common Digital System Architectures and Challenges Crossfade/Leel Controller Features Stream Relationships Construction of Volume Fades Real-time eent functionality Implementation for Common Scenario Results Demonstrations

3 Leel Control Motiation More multimedia capabilities on portable deices yields more simultaneous audio streams Standard media Music (including multi-stream mixing, e.g. music with MIDI background track Audio with ideo User alerts User Interface sounds Cellular ringtones (MIDI, MP, etc. Communication streams Voice streams

Crossfade Motiation Crossfades often employed in music clubs for smooth transition between songs Can also perform beat-matched crossfade for een smoother

4 Crossfade Motiation Crossfades often employed in music clubs for smooth transition between songs Can also perform beat-matched crossfade for een smoother transition Feature not commonly found on consumer media deices (e.g. cell phones, portable media players Can greatly improe user experience (i.e. perception of quality

5 Digital Audio Common Use Case Crossfade with UI/ringtone (alert sound Establish stream relationships Stream relationships change oer time Outgoing music Equally important Affect each other equally Incoming music UI sound or ringtone Most important Influences other streams

6 Compressed Data PCM Samples from radio/adc or preious DEC Digital Audio Architectures DEC MIDI EQ TSM AGC SRC EQ AGC SRC SRC XFC/ MIX PCM data to DAC Streamer Decryptor Decoder Effects Renderer Compressed Data streamer for transferring data to internal memory for playback Decryption module like WMDRM Audio decoder like MP, AAC, WMA Algorithms for audio effects like Equalizer, TSM DAC Drier for PCM audio samples playback

7 Digital Audio Architectures PCM data to DAC EQ TSM AGC SRC DEC XFC/ MIX SRC MIDI EQ AGC SRC Compressed Data for Storage ENC Compressed Data PCM Samples from radio/adc or preious DEC

8 Digital Audio Architectures Data stream STREAM PARSER Video Decoder Video Postprocessing Display Drier A/V Sync YUV/RGB data DEC MIDI EQ TSM AGC SRC SRC XFC/ MIX PCM data to DAC EQ AGC SRC PCM Samples from radio/adc or preious DEC ENC Compressed Data for Storage

9 Digital Audio Design Challenges Compressed Data PCM Samples from radio/adc or preious DEC DEC MIDI EQ TSM AGC SRC EQ AGC SRC SRC XFC/ MIX Saturation and dynamic range trade-offs Limited bits per sample in fixed-point architecture Gains from mixing could cause saturation Gain reduction aoids saturation, but hurts dynamic range perceied by user Synchronous changes to DAC gain Needed to compensate for gain reductions in aoiding saturation PCM data to DAC VOL

10 Digital Audio Design Challenges Compressed Data DEC MIDI EQ TSM AGC SRC SRC XFC/ MIX PCM data to DAC PCM Samples from radio/adc or preious DEC EQ AGC SRC OEMs might choose different effects Change number of channels/streams Remoe AGC, too many cycles Perform EQ after XFC/MIX

11 Digital Audio Design Challenges Unexpected end of stream Corrupted bit-stream or real-time error User actiity, such as stop or pause Frequent messages add MIPS and complexity Aoid needing to send a message for eery gain change No single idea of a good crossfade Club-DJ might want slow/transparent transition Radio-DJ might want long decay and quick ramp

12 Crossfade Controller Features Volume leel control for N streams User control of relationships between streams Automatic changes in olume targets based upon stream relationships Multi-segment olume profiles to emulate irtually any possible fade shape Depop logic remoes pops from sudden stops in the audio Real-time eents for fades, addition, and deletion of streams

13 XFC Algorithm Architecture Crossfade module inputs from application: User Volume Profiles Set desired multi-segment olume fades for each stream Model Coefficients Defines stream relationships Input PCM data for each stream Crossfade module contains: XFC Leel Controller Uses olume profiles and model coefficients to determine how to alter Volume Adjust Objects Volume Adjust Objects Control/change olume for each of N streams (*N objects for stereo processing Depop Smooths out noise from sudden stops in audio Mixer Final summing of streams User Volume Profiles Figure : Structure of the XFC algorithm Model Coefficients XFC Leel Controller Input PCM Stream Volume Adjust Object Input PCM Stream Mixer Volume Adjust Object Depop Audio PCM output Input PCM Stream Volume Adjust Object

14 Types of Stream Relationships Hierarchical A Volume of only one stream alters olume of other stream, without being affected by other stream (i.e. high s. low priority Example: Priority(V >Priority(V V ' V V ' ( V Generic form: V ' ( α α V V V A Equal Volumes of two streams affect each other equally Example: Priority(V Priority(V V ' V V ' Generic form: V V V ' V β β V A V A

15 Types of Stream Relationships (cont d Multiple influence Volume of two or more streams equally alter olume of other stream, without being affected by other stream. Example: Priority(V Priority(V >Priority(V V ' V V ' V V ' ( V V V ' V Generic Form: V ' V V [ ] γ ' ( γ γ V ' γ V ' V V ' ( γ γ V ' A V ' A A

16 Stream Relationships for XFC Hierarchical α,α used to generate terms b α bitmap matrix designates to which streams to apply these terms Equal β, β used to generate terms b β bitmap matrix designates to which streams to apply these terms Multiple influence γ, γ used to generate terms b γ bitmap matrix designates to which streams to apply these terms A A A A A A A

17 Setting XFC Stream Relationships ( ( ( ( ( g Stream ID Number Priority Leel Highest β γ α g g g g ( ( ( ( (

18 Gain matrix Terms Bit Matrix Setting XFC Stream Relationships { 4 β β β t { } { } {} { ' ' ' ' ' ' 4 γ γ γ t ( g α { } { } {} { 4 α α α t { } b α b β { } b γ ( ( g β ( ( g γ

19 Generating Stream Relationships from Priority Leel Stream relationships can be explicitly set in model coefficients Or if strict priority-based structure is desired, model coefficients can be generated through XFC s XFC_coeff_gen( function #define NUM_STREAMS 4 Int6 coeff_gen_p {,,, }; XFC_ol_adj_params model_coeff; XFC_coeff_gen(NUM_STREAMS, coeff_gen_p, model_coeff;

$profiles consist of seen parts stream_id unique ID for each stream num_eents number of segments in user olume profile del_t array that holds lengths of all segments (i.e. {t, t, t, t } s array that holds slopes of all segments (i.$ $e. {s, s, s, s } _init the initial alue for start of the olume profile status the current state for the stream end_of_profile_status designates the status when the end of the user olume$

20 User Volume Profiles Multi-segment user olume profiles proide great flexibility in shapes of fades Any shape can be emulated Not just limited to linear or exponential decays User olume profiles consist of seen parts stream_id unique ID for each stream num_eents number of segments in user olume profile del_t array that holds lengths of all segments (i.e. {t, t, t, t } s array that holds slopes of all segments (i.e. {s, s, s, s } _init the initial alue for start of the olume profile status the current state for the stream end_of_profile_status designates the status when the end of the user olume profile is reached. _init Example of User Volume Profile Shape s s s s t t t t Single-Segment, Exponential Fade-In Multi-Segment, Linear Fade-In

21 XFC State Machine Exclusie state machine for each stream States STREAM_DELAY_FADE stream inactie and not mixed in, state presered STREAM_NEW_FADE userinitiated state to designate start of a new user olume profile STREAM_ON stream actie and mixed in STREAM_STOP user-initiated hard stop STREAM_DEPOP applying depop logic STREAM_OFF stream inactie and not mixed in, automatic transition to deletion or STREAM_DELAY_FADE STREAM_ STOP State Transition Diagram for XFC module STREAM_ NEW_ FADE STREAM_ ON STREAM_ DEPOP STREAM_ OFF Deletion eops or eops eops eops STREAM_ DELAY_ FADE Red designates user-triggered state changes, while all other state changes occur automatically within the XFC algorithm.

22 End Of Profile Status Flag End of Profile Status Flag (eops means of automatically designating the status upon completion of user olume profile (i.e. no user interention needed at end of fade Three possible flag alues delete the stream upon completion of fade stream is resident and silent upon completion of fade stream is actie and playing upon completion of fade STREAM_ STOP State Transition Diagram for XFC module STREAM_ NEW_ FADE STREAM_ ON STREAM_ DEPOP STREAM_ OFF Deletion eops or eops eops eops STREAM_ DELAY_ FADE Red designates user-triggered state changes, while all other state changes occur automatically within the XFC algorithm.

Depop Processing Problem: Pop sound from discontinuity at stop of audio Bitstream corruption or realtime error causes stop in middle of song without warning User stops in middle of song without

23 Depop Processing Problem: Pop sound from discontinuity at stop of audio Bitstream corruption or realtime error causes stop in middle of song without warning User stops in middle of song without warning End of song reached with no fade-out End of song reached with partial fade-out Solution: Depop processing lessens discontinuity through exponential decay STREAM_ON Example of Depop Processing Stream is turned off when amplitude of both channels are below a threshold STREAM_OFF -> STREAM_DELAY_FADE or Deletion STREAM_DEPOP

24 Motiation for Eent-Drien XFC: Multiple Streams.5 Case: / Title: Nested Streams with Equal Priorities / Full Volume Input Vol Output Vol Sum Tim e.5 SRC SRC SRC Input Vol Tim e..8 SRC SRC SRC Output Vol Tim e Volume Leels/Targets for each stream need to change to accommodate unanticipated streams

25 Motiation for Eent-Drien XFC: Asynchronous Fades Case: 6 / Title: quick ram p-up with slow ram p-down, asynchronous.5 Input Vol Output Vol Sum Tim e.5 SRC SRC Input Vol Tim e. SRC SRC.8 Output Vol Tim e

26 Volume Targets and Eents XFC is eent-drien in order to handle interruptions by new streams At each real-time eent, target olumes and delta times are recalculated based on stream relationships Eents Start of Fade (SOF beginning of user olume profile End of Fade (EOF conclusion of user olume profile End of Segment (EOS end of segment within user olume profile XFC controller keeps track of time to next eent Low Priority Stream High Priority Stream SOF SOF EOF SOF EOF EOF Target Calculator Operating Example Original Volume Profile Adjusted Volume Profile Original Targets Adjusted Targets

27 Common Scenario: Crossfade with UI sound.5 Case: / Title: M IDI sound during first and second half of XF In p u t V o l O utput V ol Sum Tim e.5 SRC SRC SRC Input Vol Tim e..8 SRC SRC SRC Output Vol Tim e

28 Implementation Steps by Client. Setup model coefficients for three streams. Create user olume profiles (fade shapes and olume targets for each stream. Input model coefficients and user olume profiles into XFC 4. When desired, change stream status to STREAM_NEW_FADE to begin olume ramps

29 Common Scenario: XFC Stream Relationships Stream ID Number Priority Leel Highest ( ( ( ( g { 4 4 β γ α g g g g ( ( Or generate model coefficients automatically: #define NUM_STREAMS Int6 coeff_gen_p {,, }; XFC_ol_adj_params model_coeff; XFC_coeff_gen(NUM_STREAMS, coeff_gen_p, model_coeff;

30 Gain matrix Terms Bit Matrix Common Scenario: XFC Stream Relationships { β β β t 44 {} { } {} { ' ' ' γ γ γ t 44 ( ( g α { } { } {} { α α α t 4 4 { } b α b β { } b γ ( ( g β g γ

31 Common Scenario: Setting Up User Volume Profiles XFC_user_olume_profile slow_linear_fadeout { };, // stream id MULTISEGMENT_NUM, // num_eents slow_linear_fadeout_del_t // del_t slow_linear_fadeout_s, // s - 768, // _init STREAM_NEW_FADE, // stream state // end of profile status XFC_fade(XFCHandle, xfcstatus, fast_linear_fadein; XFC_fade(XFCHandle, xfcstatus, slow_linear_fadeout; XFC_fade(XFCHandle, xfcstatus, slow_linear_fadein;.. XFC_user_olume_profile fast_linear_fadein {, // stream id MULTISEGMENT_NUM, // num_eents fast_linear_fadein_del_t // del_t fast_linear_fadein_s, // s - 4, // _init (Q8.8 (-4dB STREAM_DELAY_FADE, // stream state // end of profile status }; if (timelater { fast_linear_fadein.status STREAM_NEW_FADE; } XFC_fade(XFCHandle,xfcStatus, fast_linear_fadein;

32 Results Flexible leel/crossfade controller User control of stream relationships Multi-segment olume fades Eent-drien module Implemented on C55x DSP Performance Memory (including Mixer and Volume objects Program 9.kB Data.4kW Instance ~ words MIPS Test Scenario 5-second crossfade of two streams Aerage 4. MIPS Worst Case 8. MIPS

33 Demo Equal Priority Sinusoids.5 Case: / Title: Nested Streams with Equal Priorities / Full Volume Input Vol Output Vol Sum Time.5 SRC SRC SRC Sum (No XFC Input Vol Time..8 SRC SRC SRC Sum (With XFC Output Vol Time

34 Demo- XF with Alert Sound (Music With XFC No XFC

35 Conclusions Discussed today Need to handle multiple streams and crossfades Audio System Architectures and Challenges Crossfade/Leel Controller Highlights Stream Relationships Construction of Volume Fades Real-time eent functionality Implementation for Common Scenario Results Demonstrations Possible Future Work Currently based on maximum olume of stream, could use current olume leel of stream

36 Soling Audio Subsystem Challenges Using a Crossfade/Leel Controller Algorithm in Portable Media Applications Danny Jochelson dsjochel@ti.com Jason Kridner jdk@ti.com

Completing the Multimedia Architecture

Copyright Khronos Group, 2011 - Page 1 Completing the Multimedia Architecture Erik Noreke Chair of OpenSL ES Working Group Chair of OpenMAX AL Working Group Copyright Khronos Group, 2011 - Page 2 Today