Contents. Resumen. List of Acronyms. List of Mathematical Symbols. List of Figures. List of Tables. I Introduction 1

Transcription

1 Contents Agraïments Resum Resumen Abstract List of Acronyms List of Mathematical Symbols List of Figures List of Tables VII IX XI XIII XVIII XIX XXII XXIV I Introduction 1 1 Introduction Motivation Scientific and Technological Objectives Document Structure... 5 Bibliography Background Statistical Natural Language Recognition Preprocessing and Feature Extraction Automatic Speech Recognition Features O -line Handwriting Text Recognition Features On-line Handwriting Text Recognition Features Tandem Features Statistical Modelling Morphological Modelling Language Modelling I

2 Advances on the Transcription of Historical Manuscripts Lexicon Modelling Decoding The Viterbi Algorithm Recognition Output Formats Assistive Transcription of Historical Manuscripts Crowdsourcing for Natural Language Processing Tasks Evaluation Measures Natural Language Recognition Evaluation Language Model Evaluation Computer Assisted Transcription Evaluation Multimodal Crowdsourcing Statistical Significance Datasets Historical Manuscript Corpora (O -line Handwriting) Touch Screen Handwriting Corpus (On-line Handwriting): UNIPEN Training Speech Corpus: Albayzin Multimodal (Text - Speech) Corpora Bibliography II Multimodality 41 3 Combining Handwriting and Speech Introduction Hypothesis Combination on Natural Language Recognition Recogniser Output Voting Error Reduction (ROVER) N-best ROVER Lattices Rescoring Our proposal: Bimodal Confusion Network Combination Subnetworks Based Alignment Composing a New Confusion Network Conclusions Bibliography Multimodal Experimental Results Experimental Framework Datasets Features Models II

3 4.1.4 Evaluation Metrics Experimental Setup Experiment 1: Iterative and Non-Iterative Combination Experiments with Cristo Salvador Experiments with Rodrigo Experiment 2: Unimodal and Multimodal Combination Baseline Experiments Unimodal Combination Experiments Multimodal Combination Experiment Di culty of Reaching the Oracle Values Experiment 3: Multimodal Combination Comparative Conclusions and Future Work Bibliography III Interactivity 65 5 Assistive Transcription Computer Assisted Transcription Overview Multimodal Computer Assisted Transcription Multimodal Hypotheses Combination in CATTI Multimodal Hypotheses Correction in CATTI Conclusions Bibliography Interactivity Experimental Results Experimental Framework Datasets Features Models Evaluation Metrics Experimental Setup Experiment 1: Multimodal Hypotheses Combination Experiments with Cristo Salvador Experiments with Rodrigo Experiment 2: Multimodal Hypotheses Correction O -line and On-line HTR Results CATTI and Multimodal CATTI Results Experiment 3: Multimodal Hypotheses Combination and Correction III

4 Advances on the Transcription of Historical Manuscripts Post-Edition Baseline Results CATTI Results Multimodal CATTI Results Conclusions and Future Work Bibliography IV Crowdsourcing 85 7 Collective Collaboration Multimodal Crowdsourcing Framework Language Model Interpolation Multimodal Combination Reliability Verification Lines Selection Client Application for Speech Acquisition Conclusions Bibliography Crowdsourcing Experiments Experimental Conditions Datasets Features Models Evaluation Metrics Experimental Setup Experiment 1: Supervised Multimodal Crowdsourcing Baseline and Framework Adjustment Speaker Ordering ASR Reliability Verification Absence of Speech Utterances Collaborator E ort Optimisation Experiment 2: Unsupervised Multimodal Crowdsourcing Baseline and Framework Adjustment Preliminary Experiments ASR Reliability Verification and Collaboration E ort Collaboration E ort per Line Conclusions and Future Work Bibliography IV

5 V Conclusions and Future Work Conclusions and Future Work Conclusions Scientific Work and Contributions Future Work Bibliography V