Basic Concepts in Information Theory and Coding The Adventures of Secret Agent 00111
Applications of Communications Theory Series Editor: R. W Lucky, Bellcore Recent volumes In the series: BASIC CONCEPTS IN INFORMATION THEORY AND CODING: The Adventures of Secret Agent 00111 Solomon W. Golomb, Robert E. Peile, and Robert A. Scholtz COMPUTER COMMUNICATIONS AND NETWORKS John R. Freer COMPUTER NETWORK ARCHITECTURES AND PROTOCOLS Second Edition. Edited by Carl A. Sunshine DATA COMMUNICATIONS PRINCIPLES Richard D. Gitlin, Jeremiah F. Hayes, and Stephen B. Weinstein DATA TRANSPORTATION AND PROTECTION John E. Hershey and R. K. Rao Yarlagadda DIGITAL PHASE MODULATION John B. Anderson, Tor AuIin, and Carl-Erik Sundberg DIGITAL PICTURES: Representation and Compression Arun N. NetravaIi and Barry G. Haskell FUNDAMENTALS OF DIGITAL SWITCHING Second Edition. Edited by John C. McDonald AN INTRODUCTION TO BROADBAND NETWORKS: LANs, MANs, ATM, B-ISDN, and Optical Networks for Integrated Multimedia Telecommunications Anthony S. Acampora AN INTRODUCTION TO PHOTONIC SWITCHING FABRICS H. Scott Hinton OPTICAL CHANNELS: Fibers, Clouds, Water, and the Atmosphere Sherman Karp, Robert M. Gagliardi, Steven E. Moran, and Larry B. Stotts PRACTICAL COMPUTER DATA COMMUNICATIONS William J. Barksdale SIMULATION OF COMMUNICATIONS SYSTEMS Michel C. Jeruchim, Philip Balaban, and K. Sam Shanmugan A Continuation Order Plan is available for this series. A continuation order will bring delivery of each new volume immediately upon publication. Volumes are billed only upon actual shipment. For further information please contact the publisher.
Basic Concepts in Information Theory and Coding The Adventures of Secret Agent 00111 Solomon W. Golomb Departments of Electrical Engineering and Mathematics University of Southern California Los Angeles, California Robert E. Peile Racal Research, Limited Reading, Berkshire, United Kingdom Robert A. Scholtz Department of Electrical Engineering University of Southern California Los Angeles, California Springer Science+Business Media, LLC
LIbrary of Congress CatalogIng-In-PublIcatIon Data Golomb, Solomon W. (Solomon Wolf) Basic concepts in information theory and coding, the adventures of secret agent 00111 / SoloMon W. GOlOMb, Robert E. Peile, Robert A. Scholtz. p. em. -- (Applications of communications theory) Includes bibliographical references and index. 1. Coding theory. 2. InforMation theory. I. Peile, Robert E. II. Scholtz, Robert A. III. Title. IV. Series. CA268.G575 1994 003'.54--dc20 93-48869 CIP ISBN 978-1-4419-3236-5 ISBN 978-1-4757-2319-9 (ebook) DOl 10.1007/978-1-4757-2319-9 1994 Springer Science+Business Media New York Originally published by Plenum Press, New York in 1994. Softcover reprint of the hardcover I st edition 1994 All rights reserved No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, microfilming, recording, or otherwise, without written permission from the Publisher
Preface Basic Concepts in Information Theory and Coding is an outgrowth of a onesemester introductory course that has been taught at the University of Southern California since the mid-1960s. Lecture notes from that course have evolved in response to student reaction, new technological and theoretical developments, and the insights of faculty members who have taught the course (including the three of us). In presenting this material, we have made it accessible to a broad audience by limiting prerequisites to basic calculus and the elementary concepts of discrete probability theory. To keep the material suitable for a one-semester course, we have limited its scope to discrete information theory and a general discussion of coding theory without detailed treatment of algorithms for encoding and decoding for various specific code classes. Readers will find that this book offers an unusually thorough treatment of noiseless self-synchronizing codes, as well as the advantage of problem sections that have been honed by reactions and interactions of several generations of bright students, while Agent 00111 provides a context for the discussion of abstract concepts. Information theory and coding has progressed from its earliest beginnings in Shannon's epochal paper, "A Mathematical Theory of Communication," and the initial work on error-correcting codes by Hamming and Golay (all in the late 1940s) to revolutionize all aspects of information handling, storage, and communication and transform how information is viewed in fields as v
vi Preface diverse as biology, linguistics, and philosophy. It is our hope that this book will introduce the fascinating subject of information theory to many new readers. Solomon W. Golomb Los Angeles Robert E. Peile Reading, England Robert A. Scholtz Los Angeles
Contents CHAPTER 1. Introduction 1.1. Agent 00 III... 1 l.2. Uncertainty and Measurement.......................... 2 l.2.l. Agent 00111 Ponders Pricing... 2 l.2.2. Axioms for Uncertainty and the Entropy Function... 4 l.3. Information Gain... 13 1.3.1. Agent 00 Ill's Sliding Scale of Success Charges... 13 1.3.2. Mutual Information and Equivocation............. 16 1.4. Handling Large Amounts of Information... 21 l.4.l. The Problem for Agent 00111... 21 1.4.2. A General Model for an Information Source... 22 1.5. Tutorial on Homogeneous Markov Sources... 27 l.6. The Number of Typical Sequences... 45 l.6.l. Agent 00111 Uncovers a Puzzle... 45 1.6.2. List Length and Entropy... 46 l.7. The Utility ofinformation Source Models... 53 l. 7.l. Agent 00 III and Language Generation... 53 l.7.2. Language Models and Generation... 53 1.8. Notes... 59 References... 61 CHAPTER 2. Coding for Discrete Noiseless Channels 2.1. The Problem... 63 2.1.1. Agent 00111 's Problem... 63 2.1.2. Problem Statement... 64 vii
viii Contents 2.2. An Algorithm for Determining Unique Decodability in the UF and Us Cases... 67 2.3. A Simple Coding Theorem for Fixed-Rate Sources... 74 2.4. The Significance of Information Theory... 80 2.5. Tree Codes... 83 2.6. A Coding Theorem for Controllable Rate Sources... 91 2.7. Huffman's Coding Procedure........................... 95 2.8. Efficiently Encoding Markov Sources... 101 2.9. Variable Symbol Duration Channels... 105 2.10. Lempel-Ziv Coding Procedure... 116 2.10.1. Agent 00111 's Problem... 116 2.10.2. The Algorithm... 116 2.10.3. The Lempel-Ziv Algorithm and Entropy... 123 2.10.4. The Lempel-Ziv Approach and Sequence Complexity Results... 127 2.11. Notes... 128 References... 129 CHAPTER 3. Synchronizable Codes 3.1. An Untimely Adventure............................... 131 3.2. Identifying U I Dictionaries... 132 3.3. The Hierarchy ofsynchronizable Codes... 140 3.4. A Bound on U I Dictionary Size... 143 3.5. Fixed-Word-Length UI Dictionaries...................... 147 3.5.1. Maximal Comma-Free Codes... 148 3.5.2. Prefixed Comma-Free Codes... 151 3.5.3. Path-Invariant Comma-Free Codes................ 153 3.5.4. Lexicographic U I Codes......................... 155 3.6. Comparing Fixed-Word-Length Synchronizable Codes... 160 3.7. Variable-Word-Length Synchronizable Codes.............. 164 3.8. Necessary Conditions for the Existence of UI Dictionaries... 170 3.9. Cyclic Equivalence Class Occupancy and the Sufficiency of Iteratively Constructed Codes........................... 174 3.10. Constructing Maximal Comma-Free Codes of Odd Word Length 182 3.11. Automating Binary Bounded Synchronization Delay Codes.. 189 3.11.1. Cyclic Equivalence Class Representations... 189 3.11.2. Encoding and Decoding... 196 3.12. Notes... 199 Appendix: The Mobius Inversion Formula... 200 References... 204
Contents ix CHAPTER 4. Infinite Discrete Sources 4.1. Agent 00 III Meets the Countably Infinite... 207 4.2. The Leningrad Paradox... 208 4.3. Mean vs. Entropy in Infinite Discrete Distributions... 211 4.4. Run-Length Encodings... 219 4.5. Decoding Run-Length Codes... 222 4.6. Capacity-Attaining Codes... 223 4.7. The Distribution Waiting Times and Performance of Elias- Shannon Coding... 226 4.8. Optimal, Asymptotically Optimal, and Universal Codes... 230 4.9. The Information-Generating Function of a Probability Distribution........................ 237 4.9.1. Uniqueness ofthe Inverse... 238 4.9.2. Composition of Generating Functions... 240 4.10. Notes... 241 References... 241 CHAPTER 5. Error Correction I: Distance Concepts and Bounds 5.1. The Heavy-Handed Cancellation Problem... 243 5.2. Discrete Noisy Channels............................... 244 5.3. Decoding Algorithms... 248 5.4. A Hamming Distance Design Theorem................... 256 5.5. Hamming Bound... 260 5.6. Plotkin's Bound... 265 5.7. Elias Bound......................................... 269 5.8. Gilbert Bound... 273 5.9. Perfect Codes........................................ 275 5.10. Equidistant Codes... 280 5.11. Hamming Distance Enumeration... 284 5.12. Pless Power Moment Identities and the Welch, McEliece, Rodemich, and Rumsey (WMR) Bound... 293 5.13. Finite State Channels... 295 5.14. Pure Error Detection... 302 5.15. Notes... 306 References... 307 CHAPTER 6. Error Correction II: The Information-Theoretic Viewpoint 6.1. Disruption in the Channel... 309 6.2. Data-Processing Theorem and Estimation Problems... 310
x Contents 6.3. An Upper Bound on Information Rate for Block-Coding Schemes 315 6.4. The Chernoff Bound... 323 6.5. Linked Sequences... 331 6.6. Coding Theorem for Noisy Channels..................... 336 6.7. The Situation for Reliable Communication... 346 6.8. Convex Functions and Mutual Information Maximization... 348 6.9. Memoryless Channel Capacity Computations.............. 358 6.10. Notes... 367 References... 368 CHAPTER 7. Practical Aspects of Coding 7.1. Agent 00111 Is Not Concerned... 369 7.2. Types of Practical Codes............................... 369 7.2.1. Convolutional Codes... 370 7.2.2. Block Codes: A General Overview... 374 7.2.3. Reed-Solomon Codes... 377 7.2.4. Interleaving... 382 7.2.5. Concatenated Codes............................ 383 7.3. Coding and Modulation... 388 7.4. Hybrid Forward Error Correction (FEC) and Retransmission (ARQ) Schemes...................................... 389 7.4.1. General Comments... 390 7.4.2. Some Basic ARQ Strategies...................... 390 7.4.3. Type-1 Hybrid FECI ARQ... 391 7.4.4. Type-2 Hybrid ARQ/FEC... 393 7.4.5. Chase Code Combining... 395 7.4.6. Coding and Future Networks... 395 7.A. Appendix: Lattices and Rings........................... 397 References... 416 Author Index... 419 Subject Index... 423
Basic Concepts in Information Theory and Coding The Adventures of Secret Agent 00111