Basic Concepts in Information Theory and Coding. The Adventures of Secret Agent 00111

Transcription

1 Basic Concepts in Information Theory and Coding The Adventures of Secret Agent 00111

2 Applications of Communications Theory Series Editor: R. W Lucky, Bellcore Recent volumes In the series: BASIC CONCEPTS IN INFORMATION THEORY AND CODING: The Adventures of Secret Agent Solomon W. Golomb, Robert E. Peile, and Robert A. Scholtz COMPUTER COMMUNICATIONS AND NETWORKS John R. Freer COMPUTER NETWORK ARCHITECTURES AND PROTOCOLS Second Edition. Edited by Carl A. Sunshine DATA COMMUNICATIONS PRINCIPLES Richard D. Gitlin, Jeremiah F. Hayes, and Stephen B. Weinstein DATA TRANSPORTATION AND PROTECTION John E. Hershey and R. K. Rao Yarlagadda DIGITAL PHASE MODULATION John B. Anderson, Tor AuIin, and Carl-Erik Sundberg DIGITAL PICTURES: Representation and Compression Arun N. NetravaIi and Barry G. Haskell FUNDAMENTALS OF DIGITAL SWITCHING Second Edition. Edited by John C. McDonald AN INTRODUCTION TO BROADBAND NETWORKS: LANs, MANs, ATM, B-ISDN, and Optical Networks for Integrated Multimedia Telecommunications Anthony S. Acampora AN INTRODUCTION TO PHOTONIC SWITCHING FABRICS H. Scott Hinton OPTICAL CHANNELS: Fibers, Clouds, Water, and the Atmosphere Sherman Karp, Robert M. Gagliardi, Steven E. Moran, and Larry B. Stotts PRACTICAL COMPUTER DATA COMMUNICATIONS William J. Barksdale SIMULATION OF COMMUNICATIONS SYSTEMS Michel C. Jeruchim, Philip Balaban, and K. Sam Shanmugan A Continuation Order Plan is available for this series. A continuation order will bring delivery of each new volume immediately upon publication. Volumes are billed only upon actual shipment. For further information please contact the publisher.

3 Basic Concepts in Information Theory and Coding The Adventures of Secret Agent Solomon W. Golomb Departments of Electrical Engineering and Mathematics University of Southern California Los Angeles, California Robert E. Peile Racal Research, Limited Reading, Berkshire, United Kingdom Robert A. Scholtz Department of Electrical Engineering University of Southern California Los Angeles, California Springer Science+Business Media, LLC

4 LIbrary of Congress CatalogIng-In-PublIcatIon Data Golomb, Solomon W. (Solomon Wolf) Basic concepts in information theory and coding, the adventures of secret agent / SoloMon W. GOlOMb, Robert E. Peile, Robert A. Scholtz. p. em. -- (Applications of communications theory) Includes bibliographical references and index. 1. Coding theory. 2. InforMation theory. I. Peile, Robert E. II. Scholtz, Robert A. III. Title. IV. Series. CA268.G '.54--dc CIP ISBN ISBN (ebook) DOl / Springer Science+Business Media New York Originally published by Plenum Press, New York in Softcover reprint of the hardcover I st edition 1994 All rights reserved No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, microfilming, recording, or otherwise, without written permission from the Publisher

5 Preface Basic Concepts in Information Theory and Coding is an outgrowth of a onesemester introductory course that has been taught at the University of Southern California since the mid-1960s. Lecture notes from that course have evolved in response to student reaction, new technological and theoretical developments, and the insights of faculty members who have taught the course (including the three of us). In presenting this material, we have made it accessible to a broad audience by limiting prerequisites to basic calculus and the elementary concepts of discrete probability theory. To keep the material suitable for a one-semester course, we have limited its scope to discrete information theory and a general discussion of coding theory without detailed treatment of algorithms for encoding and decoding for various specific code classes. Readers will find that this book offers an unusually thorough treatment of noiseless self-synchronizing codes, as well as the advantage of problem sections that have been honed by reactions and interactions of several generations of bright students, while Agent provides a context for the discussion of abstract concepts. Information theory and coding has progressed from its earliest beginnings in Shannon's epochal paper, "A Mathematical Theory of Communication," and the initial work on error-correcting codes by Hamming and Golay (all in the late 1940s) to revolutionize all aspects of information handling, storage, and communication and transform how information is viewed in fields as v

6 vi Preface diverse as biology, linguistics, and philosophy. It is our hope that this book will introduce the fascinating subject of information theory to many new readers. Solomon W. Golomb Los Angeles Robert E. Peile Reading, England Robert A. Scholtz Los Angeles

7 Contents CHAPTER 1. Introduction 1.1. Agent 00 III... 1 l.2. Uncertainty and Measurement l.2.l. Agent Ponders Pricing... 2 l.2.2. Axioms for Uncertainty and the Entropy Function... 4 l.3. Information Gain Agent 00 Ill's Sliding Scale of Success Charges Mutual Information and Equivocation Handling Large Amounts of Information l.4.l. The Problem for Agent A General Model for an Information Source Tutorial on Homogeneous Markov Sources l.6. The Number of Typical Sequences l.6.l. Agent Uncovers a Puzzle List Length and Entropy l.7. The Utility ofinformation Source Models l. 7.l. Agent 00 III and Language Generation l.7.2. Language Models and Generation Notes References CHAPTER 2. Coding for Discrete Noiseless Channels 2.1. The Problem Agent 's Problem Problem Statement vii

8 viii Contents 2.2. An Algorithm for Determining Unique Decodability in the UF and Us Cases A Simple Coding Theorem for Fixed-Rate Sources The Significance of Information Theory Tree Codes A Coding Theorem for Controllable Rate Sources Huffman's Coding Procedure Efficiently Encoding Markov Sources Variable Symbol Duration Channels Lempel-Ziv Coding Procedure Agent 's Problem The Algorithm The Lempel-Ziv Algorithm and Entropy The Lempel-Ziv Approach and Sequence Complexity Results Notes References CHAPTER 3. Synchronizable Codes 3.1. An Untimely Adventure Identifying U I Dictionaries The Hierarchy ofsynchronizable Codes A Bound on U I Dictionary Size Fixed-Word-Length UI Dictionaries Maximal Comma-Free Codes Prefixed Comma-Free Codes Path-Invariant Comma-Free Codes Lexicographic U I Codes Comparing Fixed-Word-Length Synchronizable Codes Variable-Word-Length Synchronizable Codes Necessary Conditions for the Existence of UI Dictionaries Cyclic Equivalence Class Occupancy and the Sufficiency of Iteratively Constructed Codes Constructing Maximal Comma-Free Codes of Odd Word Length Automating Binary Bounded Synchronization Delay Codes Cyclic Equivalence Class Representations Encoding and Decoding Notes Appendix: The Mobius Inversion Formula References

9 Contents ix CHAPTER 4. Infinite Discrete Sources 4.1. Agent 00 III Meets the Countably Infinite The Leningrad Paradox Mean vs. Entropy in Infinite Discrete Distributions Run-Length Encodings Decoding Run-Length Codes Capacity-Attaining Codes The Distribution Waiting Times and Performance of Elias- Shannon Coding Optimal, Asymptotically Optimal, and Universal Codes The Information-Generating Function of a Probability Distribution Uniqueness ofthe Inverse Composition of Generating Functions Notes References CHAPTER 5. Error Correction I: Distance Concepts and Bounds 5.1. The Heavy-Handed Cancellation Problem Discrete Noisy Channels Decoding Algorithms A Hamming Distance Design Theorem Hamming Bound Plotkin's Bound Elias Bound Gilbert Bound Perfect Codes Equidistant Codes Hamming Distance Enumeration Pless Power Moment Identities and the Welch, McEliece, Rodemich, and Rumsey (WMR) Bound Finite State Channels Pure Error Detection Notes References CHAPTER 6. Error Correction II: The Information-Theoretic Viewpoint 6.1. Disruption in the Channel Data-Processing Theorem and Estimation Problems

10 x Contents 6.3. An Upper Bound on Information Rate for Block-Coding Schemes The Chernoff Bound Linked Sequences Coding Theorem for Noisy Channels The Situation for Reliable Communication Convex Functions and Mutual Information Maximization Memoryless Channel Capacity Computations Notes References CHAPTER 7. Practical Aspects of Coding 7.1. Agent Is Not Concerned Types of Practical Codes Convolutional Codes Block Codes: A General Overview Reed-Solomon Codes Interleaving Concatenated Codes Coding and Modulation Hybrid Forward Error Correction (FEC) and Retransmission (ARQ) Schemes General Comments Some Basic ARQ Strategies Type-1 Hybrid FECI ARQ Type-2 Hybrid ARQ/FEC Chase Code Combining Coding and Future Networks A. Appendix: Lattices and Rings References Author Index Subject Index

11 Basic Concepts in Information Theory and Coding The Adventures of Secret Agent 00111