A 3D Speech Visualization Tool for Turkish based on MRI

Transcription

1 A 3D Speech Visualization Tool for Turkish based on MRI Maviş Emel Kulak Kayıkcı, PhD, Hacettepe University, Turkey Can Ölçek, MSc, Sobee Studios, Turkey Erdal Yılmaz, PhD, Sobee Studios, Turkey Burce Özgen, MD, Hacettepe University, Turkey Emrah Tomur, PhD, INNOVA IT Solutions, (Current Affiliation İzmir Technology Development Zone)

2 DISCLOSURE STATEMENT Mavis Emel Kulak Kayıkcı have relevant financial relationship to the disclose: scientific consultant to INNOVA IT Solutions. The other authors don t have any financial or nonfinancial relationship to the disclose.

3 VOCAL TRACT MEASUREMENT

4 Articulators differ Location Shape Structural Composition Speed and Complexity of movement

5 Tissue consistency Soft tissue (tongue, lips, velum) Hard tissue (jaw, hard palate) Visible-invisible structures Lip vs velum Articulator Motion rate Slow motion (jaw) Fast motion (tongue) Interaction between articulator Jaw-tongue interaction

6 Device Insertion Speech distortion

7 LABORATORY TECHNIQUES Imaging Techniques X-Ray Tomography CT MRI Ultrasound Point Tracking Measurements ElectroMagnetic Articulometer (EMA) X-Ray Microbeam Optotrak Tongue-Palate Interaction Measurement Tongue-Palate Contact Tongue-Palate Pressure

8 METHOD PROS CONS COMMENT CT EMA X-RAY (MICROBEAM) ULTRASOUND MRI OPTOTRAK High Temporal and spatial resolution Capture pharyngeal structures 3-D possible High Spatial and Temporal Resolution 3-D High Temporal and spatial resolution Flesh point tracking not possible for pharyngeal structures High Temporal and spatial resolution Non-invasive, safe Good audio can be obtained simultaneously Non-invasive, safe Captures pharyngeal structures 3-D possible Tagged MRI allows fleshpoint tracking Tracks points on face Non-invasive 3D Exposure to Radiation Provides Spatially sparse point tracking data Cannot capture pharyngeal structures Exposure to radiation Images show only a projection through volume which makes contour extraction difficult Not widely available Spatially sparse data Noisy images Detects only first Tissue-air boundary Not suitable for anterior tongue tip and lip imaging Detector is in contact with jaw and may effect speech production process Relatively poor spatial and temporal resolution Expensive Limited to subjects without major dental work/implants Supine position generally required Simultaneous audio recording difficult due to scanner noise Limited to external use Rarely used in speech research Often used in speech production Rarely used now in speech research Existing databases are still being used Used Primarily for tongue body imaging An emerging technique for speech research Used primarily to track lip, jaw and face motion Bresch et al, 2008 Stone, 1999

9 MAGNETIC RESONANCE IMAGING

10 The study of the upper airway has not been the focus of much previous MRI development, and most equipment and imaging methods are not optimized for this region of interest.(bzech et al, 2008)

11 A midsaggital image of the vocal tract from the glottis to the lips can be acquired MRI

12 Real Time-MRI refers to the continuous acquisition of MR images with frame rates sufficient to capture the underlying motion or physiology of interest, typically 5 50 frames per second

13 Automatic segmentation techniques have been developed for the analysis of real-time MRI (Bresch&Narayanan 2009, Proctor et al. 2010) Direct image analysis of rtmri data (Lammert et al. 2010, Lammert et al.2011, Proctor 2011)

14 GOAL Converting MR images to Animation models

15 MRI DATA ACQUISITION

16 MRI data were acquired at the Hacettepe University Hospital. Sagittal FLASH images were obtained from the head, using an 8 channel head and neck coil with a 1.5 T MR scanner TR/TE: 4.2/1.83, Flip angle:20, NEX:1, Slice thickness:6mm, gap:3mm, FOV:300, number of dynamic images:20 (Symphony, Siemens, Erlangen, Germany). A single dynamic image was acquired in 1.7 seconds.

17 VISUALISATION STUDIES

18 The virtual tongue was modelled with 6 control points (animation bones). The mesh model was created by 3D artists using typical modelling tools.

19 Assumptions Jaw animation is fixed on one axis. There is only one motion (open/close) Uvula animation is fixed on one axis. There is only one motion.(velar elevation-depression) Tonque animation supports two axes (x&z, y axis is fixed)

20 MRI High frequency MRI setup to get as much sample as possible. The subject repeated the same phoneme to ensure to have maximum amount of samples. We determine 4-5 images (with homogenous intervals) per phoneme to capture animation data. We determine the resting position (initial frame) to find the correct order of MRI data.

21 Mapping An operator defines the positions of 6 check points. These points are the basis of animation bones.

22 Animation Editor An animation editor was developed similar to well-known graph editors. Using this interactive tool, an operator marks the time stamps (key frames) considering MRI data time tags. The figure shows an editing operation for a single check-point (bone).

23 Animation Editor The blue Mexican-hat-like curve shows the motion of a single bone (check point) in a single axis (z-axis in this example).

24 Animation Editor The graphs of a single check-point are prepared for all axes (x,y,z) Same time-steps are used.

25 Animation Editor The graph preperation operation is repeated for all bones and all-axes.

26 Tuner This Interface is used for; Jaw animation (duration and synchronization) Uvula animation (animation type, delay and synchronization) Exports an XML file that containes all animation data.

27 MRI SEQUENCE (/t /)

28 GENERATED 3D ANIMATION

29 MRI & ANIMATION

30 MOBILE & WEB APPLICATION

31 FUTURE WORK Direct creation of 3D models of speech samples from MRI recordings Synchronized audio recordings Enrichment of 3D characters (female, kids etc..)

32 Acknowledgement N.Alpay Karagöz, PhD, INNOVA IT Solutions

33