Lecture 17. Lower bound for variable-length source codes with error. Coding a sequence of symbols: Rates and scheme (Arithmetic code)
|
|
- Paulina Shana Gibson
- 5 years ago
- Views:
Transcription
1 Lecture 17 Agenda for the lecture Lower bound for variable-length source codes with error Coding a sequence of symbols: Rates and scheme (Arithmetic code) Introduction to universal codes 17.1 variable-length source code with error In the previous lecture we saw that variable-length source codes with error can do significantly better than their error-free counterparts. However, it is very easy to convert any error-free source code to one which allows error and gain a factor of roughly (1 ɛ) in the average length. Therefore, as far as code design is concerned, it suffices to design error-free variable-length codes. But can we gain even more than a factor of (1 ɛ) by a more sophisticated design? In other words, do we have a matching lower bound for our upper bound on L ɛ (X) derived in the previous lecture. Well, nearly so. Theorem For a discrete source X and 0 < ɛ < 1, H(X) log(eh(x)) ɛ log X 1 L ɛ (X) (1 ɛ)h(x) c Himanshu Tyagi. Feel free to use with acknowledgement. 1
2 Proof. Proof is very similar to that of the lower bound in Theorem The only problem is that the equality H(X) = H(Y 1,..., Y n ) does not hold anymore since we don t have a oneto-one map between the symbol X and the corresponding codeword Y 1,..., Y n. Nevertheless, using the codeword we can recover X with probability of error less than ɛ. Therefore, by Fano s inequality H(X) H(Y 1,..., Y n ) + H(X Y 1,..., Y N ) H(Y 1,..., Y n ) + ɛ log X + 1. The rest of the proof proceeds exactly as that of Theorem Coding a sequence of symbols Up to this point, we considered coding a single symbol generated by a known source distribution. In practice, however, we don t have just one symbol but a sequence of symbols. Consider a DMS (X 1,..., X n ) with a common distribution P. Before we proceed to the schemes for encoding a sequence of symbols, let us derive some benchmark bounds Optimal rates We have already seen that the optimal rates for fixed-length codes satisfy R(X) = R ɛ (X) = H(X) for every 0 < ɛ < 1. For variable length codes, defining R ɛ (X) analogously to R ɛ (X) with L ɛ (X n ) replacing L ɛ (X n ), the bounds obtained in the previous section give (show this yourself) H(X) ɛ log X R ɛ (X) (1 ɛ)h(x). In fact, an improved lower bound can be obtained and we can show that R ɛ (X) = (1 ɛ)h(x). 2
3 Note that R(X) := lim ɛ 0 R ɛ (X) = H(X). Thus, while the optimal asymptotic rate for variable-length codes is much smaller than that for fixed-length codes when error is allowed, both are same for vanishing error. Furthermore, the optimal rate with error can be obtained simply by modifying an error-free scheme using a randomized encoder. In fact, as noted earlier, we can obtain a prefix-free code of average length less than (1 ɛ)h(x n ) + 1 when error is allowed. The rate of such codes approach the optimal rate (1 ɛ)h(x) as n Arithmetic code for compressing a sequence of symbols We now explore practical schemes for compressing a sequence of symbols. As we have seen, it suffices to design schemes for the error-free case. The first, perhaps naive, scheme will simply apply one of the codes designed above for 1 symbol to each symbol. Recall that all our schemes simply guarantee that we can come within 1 bit of H(X). Therefore, when we apply our schemes to each symbol X i and denote by l(x) the length of codeword assigned to x X, the average length E [ n i=1 l(x i)] can be anywhere between nh(p) and nh(p) + n. Instead, we can treat the entire sequence (X 1,..., X n ) as a single symbol generated from P n and apply, say, Huffman code to this sequence. This will yield a code of average length H(P n ) + 1 = nh(p) + 1; indeed, it will yield a code of optimal length. In fact, this is one of the main lessons of Information theory: DMSs can be compressed better if we process a large number of symbols together. We also encountered this principle in fixed-length code where if we applied the code given by our single-shot results symbol-wise we will not be able to achieve the optimal rate of H(P). Attaining H(P) requires identifying small cardinality subsets of X n of large P n -probability directly. So, are we done here? Well, in theory yes. But we are not even close from an implementation point of view. The main bottleneck in implementing either Shannon-Fano or Huffman code is sorting large sequences. Even if somehow design the code, encoding each sequence will require us to navigate a huge look-up table since we cannot immediately 3
4 identify any additional structure in the code. A similar obstacle is faced in implementing a Shannon-Fano-Elias code as well. This problem is solved using a variant of the Shannon-Fano-Elias code called an arithmetic code. The key feature of an arithmetic code is that it can compress a symbol x i using just the knowledge of the probability P(x i ) and yet ensure that a sequence x 1,..., x n is compressed to a codeword of length log P(x 1 )...P(x n ). In fact, the coding scheme can be applied to any arbitrary distribution P(x 1,..., x n ) by using distribution P(x i x 1,..., x i 1 ) for the ithe symbol. Furthermore, it satisfies the First In First Out (FIFO) property, namely the symbols are decoded in the order in which they are encoded. Arithmetic coding, too, represents codewords as interval subsets of [0, 1]. Specifically, corresponding to a sequence (x 1,..., x n ), the scheme produces an interval of length P(x 1,..., x n ). Different intervals corresponding to different codewords are disjoint. In each interval we can find a number with binary representation of length l(x) = log P(x 1,..., x n ), which we use to represent the interval. Specifically, each interval of length l(x) must contain one number from the set {0, 2 l(x), 2.2 l(x), 3.2 l(x),..., 1} = { , , , , ,..., }. We use one of these sequences to uniquely represent the interval. It remains to describe the encoding process of obtaining the interval HEAD and the decoding process of recovering the sequence from the interval. ======= and the decoding process of removering the sequence from the interval. The encoder maintains an interval at each step represented by its starting point C and the width A and successively updates the interval for each new symbol. At every step, a new interval is constructed by partitioning the previous interval into parts of length proportional to the probabilities of the symbol and move to the part corresponding to the encoded symbol. We begin with the interval [0, 1]. As such, every new interval is a subset of the previous interval. Furthermore, intervals corresponding to different sequences of the same length constitute a partition of 4
5 [0, 1] as the length of the sequence increases the partition becomes successive finer. The formal description of the encoder is as follows: Input: A sequence of symbols (x 1,..., x n ) X n and associated pmfs (P 1,..., P n ) on X. Output: An interval [C, C + A) [0, 1]; C is called the current code and A the augend. 1. Initialize C = 0 and A = for i =1,..., n (i) update C C + A a<x i 1 P i(a). For convenience, we have assumed an ordering between the symbols of X ; (ii) update A A P i (x i ). As an illustration, consider the sequence acbb where each symbol is generated from the same pmf P such that P(a) = 1/8, P(b) = 1/2, P(c) = 1/4, P(d) = 1/8; we assume the ordering a < b < c < d. For this sequence, the encoder above updates the interval as follows: [0, 0.001) [ , ) [ , ) [ , ). Note that the width of the final interval is 2 9. A number in the final interval which has a binary representation of length 9 is given by Thus, the codeword corresponding to acbb is The decoding process simply inverts the encoding process and is accoplished by magnitude comparisons. At each iteration, we consider the partion of [0, 1] with parts proportional to probabilities used in the encoding that symbol (with the same ordering as the encoding process). The value of the symbol is obtained by simply checking which part the binary value of the codeword lies in. Once the symbol is identified, the codeword is updated by renormalizing the previous interval of the decoded symbol to [0, 1]. 5
6 Formally, each iteration of the decoder proceeds as follows: Input: A number v in [0, 1] and a pmfs P on X. Output: A symbol x X and the updated (codeword) number 1. Return x such that x 1 i=1 P i v < x i=1 P i; ( 2. update v v ) x 1 i=1 P i /P x. As an illustration, consider our foregoing example once more. Upon observing the codeword , the symbols and the updated codewords at each step produced by the decoder above are as follows: (a, ) (c, ) (b, 0.001) (b, 0). The final value of zero represents the completion of the decoding process. Remark (i) The elegant algorithm above can be easily implemented if one can multiply and divide real numbers with arbitrary precision. Of course, that is not the case. In practice, we are forced to work with finite precision. Nevertheless, variants of arithmetic coding have been implemented to work with finite precision. (ii) Note that in our description above, we have assumed the availability of the probability model both at the encoder and the decoder. In a practical implementation, this model, too, has to be constructed recursively from the data. Even if we make the model available at the encoder, at the decoder the model pmf for decoding the next symbol must be obtained using the previously decoded symbols. We shall encounter one such method for obtaining probability models in universal source coding Universal source coding By now we are in a position to implement basic data compression algorithms, provided someone gives us the probability distribution used to generate the data. In practice, of 6
7 course, this distribution must be ascertained from the data itself. In general, our compression algorithms should include the process of modeling the data. Such source codes which are constructed to operate without the knowledge of the probability distribution generating the data are called universal sorce codes. Clearly, any code can be thought of as a universal source code. The key point is that we would like good performance guarantees for every reasonable probability model for the data. In particular, for this course we shall require guarantees for all i.i.d. pmfs. As before, we shall consider both fixed-length and variable-length codes. The first step is to define our measures of performance for each setup. Specifically, we define what is the benchmark performance that we seek from universal source codes Rate optimality for fixed-length universal source codes A sequence of fixed-length source codes C n for a source alphabet X n is of rate R if their lengths l n satisfy l n lim n n = R. We consier the limit definition for simplicity. Other variants based on lim sup and lim inf can also be considered and lead to the same results for i.i.d. sources. Denote by ɛ(p n, C n ) the probability of error for the code C n under the i.i.d. pmf P n with common distribution P. A sequence of fixed-length source codes of rate R is universally rate optimal if for every pmf P on the common alphabet X such that H(P) < R, the probability of error ɛ(p n, C n ) satisfies lim n ɛ(pn, C n ) = Minmax regret for variable-length universal source codes For variable-length universal source codes, we pursue a slightly different benchmark of performance. For each distribution P on the common alphabet X, we define the regret 7
8 for our universal code for n symbols from X as the difference between its average length under the i.i.d. pmf P n and the average length for the optimal code with the knowledge of P. To make the problem more tractible, we simply use nh(p) as a proxy for the latter quantity; our previous analysis tells us that this will only take us off by at most a bit. Specifically, for a code C which assigns codewords of lengths l(x) to x X n, define the regret for P as r n (C, P) := P n (x)l(x) nh(p). x X n The worst-case regret of code C is given by r n (C) := max r n(c, P). P P(X ) We seek to characterize the minimum of the worst-case regret over all possible uniquely decodable codes C and develop codes which achieve this minmax regret. The desired notion of minmax regret is given by r n = min C:C C u r n(c), where 853f3f922a09036b6dbfb00df81d61f9e76042d4 8
Lecture 15. Error-free variable length schemes: Shannon-Fano code
Lecture 15 Agenda for the lecture Bounds for L(X) Error-free variable length schemes: Shannon-Fano code 15.1 Optimal length nonsingular code While we do not know L(X), it is easy to specify a nonsingular
More informationInformation Theory and Communication
Information Theory and Communication Shannon-Fano-Elias Code and Arithmetic Codes Ritwik Banerjee rbanerjee@cs.stonybrook.edu c Ritwik Banerjee Information Theory and Communication 1/12 Roadmap Examples
More informationCOMPSCI 650 Applied Information Theory Feb 2, Lecture 5. Recall the example of Huffman Coding on a binary string from last class:
COMPSCI 650 Applied Information Theory Feb, 016 Lecture 5 Instructor: Arya Mazumdar Scribe: Larkin Flodin, John Lalor 1 Huffman Coding 1.1 Last Class s Example Recall the example of Huffman Coding on a
More informationLecture 13. Types of error-free codes: Nonsingular, Uniquely-decodable and Prefix-free
Lecture 13 Agenda for the lecture Introduction to data compression Fixed- and variable-length codes Types of error-free codes: Nonsingular, Uniquely-decodable and Prefix-free 13.1 The data compression
More informationDavid Rappaport School of Computing Queen s University CANADA. Copyright, 1996 Dale Carnegie & Associates, Inc.
David Rappaport School of Computing Queen s University CANADA Copyright, 1996 Dale Carnegie & Associates, Inc. Data Compression There are two broad categories of data compression: Lossless Compression
More informationDigital Communication Prof. Bikash Kumar Dey Department of Electrical Engineering Indian Institute of Technology, Bombay
Digital Communication Prof. Bikash Kumar Dey Department of Electrical Engineering Indian Institute of Technology, Bombay Lecture - 29 Source Coding (Part-4) We have already had 3 classes on source coding
More informationChapter 5 VARIABLE-LENGTH CODING Information Theory Results (II)
Chapter 5 VARIABLE-LENGTH CODING ---- Information Theory Results (II) 1 Some Fundamental Results Coding an Information Source Consider an information source, represented by a source alphabet S. S = { s,
More informationThe Encoding Complexity of Network Coding
The Encoding Complexity of Network Coding Michael Langberg Alexander Sprintson Jehoshua Bruck California Institute of Technology Email: mikel,spalex,bruck @caltech.edu Abstract In the multicast network
More informationCourse notes for Data Compression - 2 Kolmogorov complexity Fall 2005
Course notes for Data Compression - 2 Kolmogorov complexity Fall 2005 Peter Bro Miltersen September 29, 2005 Version 2.0 1 Kolmogorov Complexity In this section, we present the concept of Kolmogorov Complexity
More informationMove-to-front algorithm
Up to now, we have looked at codes for a set of symbols in an alphabet. We have also looked at the specific case that the alphabet is a set of integers. We will now study a few compression techniques in
More informationTopic: Local Search: Max-Cut, Facility Location Date: 2/13/2007
CS880: Approximations Algorithms Scribe: Chi Man Liu Lecturer: Shuchi Chawla Topic: Local Search: Max-Cut, Facility Location Date: 2/3/2007 In previous lectures we saw how dynamic programming could be
More informationInformation Theory and Coding Prof. S. N. Merchant Department of Electrical Engineering Indian Institute of Technology, Bombay
Information Theory and Coding Prof. S. N. Merchant Department of Electrical Engineering Indian Institute of Technology, Bombay Lecture - 11 Coding Strategies and Introduction to Huffman Coding The Fundamental
More informationITCT Lecture 8.2: Dictionary Codes and Lempel-Ziv Coding
ITCT Lecture 8.2: Dictionary Codes and Lempel-Ziv Coding Huffman codes require us to have a fairly reasonable idea of how source symbol probabilities are distributed. There are a number of applications
More informationDistributed source coding
Distributed source coding Suppose that we want to encode two sources (X, Y ) with joint probability mass function p(x, y). If the encoder has access to both X and Y, it is sufficient to use a rate R >
More informationAchievable Rate Regions for Network Coding
Achievable Rate Regions for Network oding Randall Dougherty enter for ommunications Research 4320 Westerra ourt San Diego, A 92121-1969 Email: rdough@ccrwest.org hris Freiling Department of Mathematics
More informationLecture 19. Lecturer: Aleksander Mądry Scribes: Chidambaram Annamalai and Carsten Moldenhauer
CS-621 Theory Gems November 21, 2012 Lecture 19 Lecturer: Aleksander Mądry Scribes: Chidambaram Annamalai and Carsten Moldenhauer 1 Introduction We continue our exploration of streaming algorithms. First,
More informationSubset sum problem and dynamic programming
Lecture Notes: Dynamic programming We will discuss the subset sum problem (introduced last time), and introduce the main idea of dynamic programming. We illustrate it further using a variant of the so-called
More informationCompressing Data. Konstantin Tretyakov
Compressing Data Konstantin Tretyakov (kt@ut.ee) MTAT.03.238 Advanced April 26, 2012 Claude Elwood Shannon (1916-2001) C. E. Shannon. A mathematical theory of communication. 1948 C. E. Shannon. The mathematical
More information16 Greedy Algorithms
16 Greedy Algorithms Optimization algorithms typically go through a sequence of steps, with a set of choices at each For many optimization problems, using dynamic programming to determine the best choices
More informationTreewidth and graph minors
Treewidth and graph minors Lectures 9 and 10, December 29, 2011, January 5, 2012 We shall touch upon the theory of Graph Minors by Robertson and Seymour. This theory gives a very general condition under
More informationScribe: Virginia Williams, Sam Kim (2016), Mary Wootters (2017) Date: May 22, 2017
CS6 Lecture 4 Greedy Algorithms Scribe: Virginia Williams, Sam Kim (26), Mary Wootters (27) Date: May 22, 27 Greedy Algorithms Suppose we want to solve a problem, and we re able to come up with some recursive
More informationOpen and Closed Sets
Open and Closed Sets Definition: A subset S of a metric space (X, d) is open if it contains an open ball about each of its points i.e., if x S : ɛ > 0 : B(x, ɛ) S. (1) Theorem: (O1) and X are open sets.
More informationOUTLINE. Paper Review First Paper The Zero-Error Side Information Problem and Chromatic Numbers, H. S. Witsenhausen Definitions:
OUTLINE Definitions: - Source Code - Expected Length of a source code - Length of a codeword - Variable/fixed length coding - Example: Huffman coding - Lossless coding - Distortion - Worst case length
More informationComputing intersections in a set of line segments: the Bentley-Ottmann algorithm
Computing intersections in a set of line segments: the Bentley-Ottmann algorithm Michiel Smid October 14, 2003 1 Introduction In these notes, we introduce a powerful technique for solving geometric problems.
More informationGreedy Algorithms CHAPTER 16
CHAPTER 16 Greedy Algorithms In dynamic programming, the optimal solution is described in a recursive manner, and then is computed ``bottom up''. Dynamic programming is a powerful technique, but it often
More informationFundamentals of Multimedia. Lecture 5 Lossless Data Compression Variable Length Coding
Fundamentals of Multimedia Lecture 5 Lossless Data Compression Variable Length Coding Mahmoud El-Gayyar elgayyar@ci.suez.edu.eg Mahmoud El-Gayyar / Fundamentals of Multimedia 1 Data Compression Compression
More informationSIGNAL COMPRESSION Lecture Lempel-Ziv Coding
SIGNAL COMPRESSION Lecture 5 11.9.2007 Lempel-Ziv Coding Dictionary methods Ziv-Lempel 77 The gzip variant of Ziv-Lempel 77 Ziv-Lempel 78 The LZW variant of Ziv-Lempel 78 Asymptotic optimality of Ziv-Lempel
More informationIEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 53, NO. 10, OCTOBER
IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 53, NO. 10, OCTOBER 2007 3413 Relay Networks With Delays Abbas El Gamal, Fellow, IEEE, Navid Hassanpour, and James Mammen, Student Member, IEEE Abstract The
More informationTopology and Topological Spaces
Topology and Topological Spaces Mathematical spaces such as vector spaces, normed vector spaces (Banach spaces), and metric spaces are generalizations of ideas that are familiar in R or in R n. For example,
More information2386 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 52, NO. 6, JUNE 2006
2386 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 52, NO. 6, JUNE 2006 The Encoding Complexity of Network Coding Michael Langberg, Member, IEEE, Alexander Sprintson, Member, IEEE, and Jehoshua Bruck,
More informationUC San Diego UC San Diego Previously Published Works
UC San Diego UC San Diego Previously Published Works Title Universal lossless compression via multilevel pattern matching Permalink https://escholarshiporg/uc/item/39k54514 Journal IEEE Transactions on
More informationCPSC 536N: Randomized Algorithms Term 2. Lecture 10
CPSC 536N: Randomized Algorithms 011-1 Term Prof. Nick Harvey Lecture 10 University of British Columbia In the first lecture we discussed the Max Cut problem, which is NP-complete, and we presented a very
More informationText Compression through Huffman Coding. Terminology
Text Compression through Huffman Coding Huffman codes represent a very effective technique for compressing data; they usually produce savings between 20% 90% Preliminary example We are given a 100,000-character
More informationMA651 Topology. Lecture 4. Topological spaces 2
MA651 Topology. Lecture 4. Topological spaces 2 This text is based on the following books: Linear Algebra and Analysis by Marc Zamansky Topology by James Dugundgji Fundamental concepts of topology by Peter
More information3 No-Wait Job Shops with Variable Processing Times
3 No-Wait Job Shops with Variable Processing Times In this chapter we assume that, on top of the classical no-wait job shop setting, we are given a set of processing times for each operation. We may select
More informationWe show that the composite function h, h(x) = g(f(x)) is a reduction h: A m C.
219 Lemma J For all languages A, B, C the following hold i. A m A, (reflexive) ii. if A m B and B m C, then A m C, (transitive) iii. if A m B and B is Turing-recognizable, then so is A, and iv. if A m
More information16.Greedy algorithms
16.Greedy algorithms 16.1 An activity-selection problem Suppose we have a set S = {a 1, a 2,..., a n } of n proposed activities that with to use a resource. Each activity a i has a start time s i and a
More informationData Compression Techniques
Data Compression Techniques Part 1: Entropy Coding Lecture 1: Introduction and Huffman Coding Juha Kärkkäinen 31.10.2017 1 / 21 Introduction Data compression deals with encoding information in as few bits
More information6. Advanced Topics in Computability
227 6. Advanced Topics in Computability The Church-Turing thesis gives a universally acceptable definition of algorithm Another fundamental concept in computer science is information No equally comprehensive
More informationSources for this lecture. 3. Matching in bipartite and general graphs. Symmetric difference
S-72.2420 / T-79.5203 Matching in bipartite and general graphs 1 3. Matching in bipartite and general graphs Let G be a graph. A matching M in G is a set of nonloop edges with no shared endpoints. Let
More information14.1 Encoding for different models of computation
Lecture 14 Decidable languages In the previous lecture we discussed some examples of encoding schemes, through which various objects can be represented by strings over a given alphabet. We will begin this
More informationOn Generalization of Fuzzy Concept Lattices Based on Change of Underlying Fuzzy Order
On Generalization of Fuzzy Concept Lattices Based on Change of Underlying Fuzzy Order Pavel Martinek Department of Computer Science, Palacky University, Olomouc Tomkova 40, CZ-779 00 Olomouc, Czech Republic
More informationEngineering Mathematics II Lecture 16 Compression
010.141 Engineering Mathematics II Lecture 16 Compression Bob McKay School of Computer Science and Engineering College of Engineering Seoul National University 1 Lossless Compression Outline Huffman &
More informationConvexity: an introduction
Convexity: an introduction Geir Dahl CMA, Dept. of Mathematics and Dept. of Informatics University of Oslo 1 / 74 1. Introduction 1. Introduction what is convexity where does it arise main concepts and
More informationLecture 17: Continuous Functions
Lecture 17: Continuous Functions 1 Continuous Functions Let (X, T X ) and (Y, T Y ) be topological spaces. Definition 1.1 (Continuous Function). A function f : X Y is said to be continuous if the inverse
More informationLecture notes on the simplex method September We will present an algorithm to solve linear programs of the form. maximize.
Cornell University, Fall 2017 CS 6820: Algorithms Lecture notes on the simplex method September 2017 1 The Simplex Method We will present an algorithm to solve linear programs of the form maximize subject
More information* (4.1) A more exact setting will be specified later. The side lengthsj are determined such that
D D Chapter 4 xtensions of the CUB MTOD e present several generalizations of the CUB MTOD In section 41 we analyze the query algorithm GOINGCUB The difference to the CUB MTOD occurs in the case when the
More informationTheorem 2.9: nearest addition algorithm
There are severe limits on our ability to compute near-optimal tours It is NP-complete to decide whether a given undirected =(,)has a Hamiltonian cycle An approximation algorithm for the TSP can be used
More informationSource Coding with Distortion through Graph Coloring
Source Coding with Distortion through Graph Coloring Vishal Doshi, Devavrat Shah, and Muriel Médard Laboratory for Information and Decision Systems Massachusetts Institute of Technology Cambridge, MA 02139
More information6. Finding Efficient Compressions; Huffman and Hu-Tucker
6. Finding Efficient Compressions; Huffman and Hu-Tucker We now address the question: how do we find a code that uses the frequency information about k length patterns efficiently to shorten our message?
More informationChapter 5 Lempel-Ziv Codes To set the stage for Lempel-Ziv codes, suppose we wish to nd the best block code for compressing a datavector X. Then we ha
Chapter 5 Lempel-Ziv Codes To set the stage for Lempel-Ziv codes, suppose we wish to nd the best block code for compressing a datavector X. Then we have to take into account the complexity of the code.
More informationHorn Formulae. CS124 Course Notes 8 Spring 2018
CS124 Course Notes 8 Spring 2018 In today s lecture we will be looking a bit more closely at the Greedy approach to designing algorithms. As we will see, sometimes it works, and sometimes even when it
More informationRecitation 4: Elimination algorithm, reconstituted graph, triangulation
Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science 6.438 Algorithms For Inference Fall 2014 Recitation 4: Elimination algorithm, reconstituted graph, triangulation
More informationTrees. 3. (Minimally Connected) G is connected and deleting any of its edges gives rise to a disconnected graph.
Trees 1 Introduction Trees are very special kind of (undirected) graphs. Formally speaking, a tree is a connected graph that is acyclic. 1 This definition has some drawbacks: given a graph it is not trivial
More informationGreedy Algorithms 1 {K(S) K(S) C} For large values of d, brute force search is not feasible because there are 2 d {1,..., d}.
Greedy Algorithms 1 Simple Knapsack Problem Greedy Algorithms form an important class of algorithmic techniques. We illustrate the idea by applying it to a simplified version of the Knapsack Problem. Informally,
More informationLecture 25 Spanning Trees
Lecture 25 Spanning Trees 15-122: Principles of Imperative Computation (Fall 2018) Frank Pfenning, Iliano Cervesato The following is a simple example of a connected, undirected graph with 5 vertices (A,
More informationEdge Colorings of Complete Multipartite Graphs Forbidding Rainbow Cycles
Theory and Applications of Graphs Volume 4 Issue 2 Article 2 November 2017 Edge Colorings of Complete Multipartite Graphs Forbidding Rainbow Cycles Peter Johnson johnspd@auburn.edu Andrew Owens Auburn
More informationEfficiently decodable insertion/deletion codes for high-noise and high-rate regimes
Efficiently decodable insertion/deletion codes for high-noise and high-rate regimes Venkatesan Guruswami Carnegie Mellon University Pittsburgh, PA 53 Email: guruswami@cmu.edu Ray Li Carnegie Mellon University
More informationInterval Algorithm for Homophonic Coding
IEEE TRANSACTIONS ON INFORMATION THEORY, VOL 47, NO 3, MARCH 2001 1021 Interval Algorithm for Homophonic Coding Mamoru Hoshi, Member, IEEE, and Te Sun Han, Fellow, IEEE Abstract It is shown that the idea
More informationError-Correcting Codes
Error-Correcting Codes Michael Mo 10770518 6 February 2016 Abstract An introduction to error-correcting codes will be given by discussing a class of error-correcting codes, called linear block codes. The
More informationROUGH MEMBERSHIP FUNCTIONS: A TOOL FOR REASONING WITH UNCERTAINTY
ALGEBRAIC METHODS IN LOGIC AND IN COMPUTER SCIENCE BANACH CENTER PUBLICATIONS, VOLUME 28 INSTITUTE OF MATHEMATICS POLISH ACADEMY OF SCIENCES WARSZAWA 1993 ROUGH MEMBERSHIP FUNCTIONS: A TOOL FOR REASONING
More informationDiagonalization. The cardinality of a finite set is easy to grasp: {1,3,4} = 3. But what about infinite sets?
Diagonalization Cardinalities The cardinality of a finite set is easy to grasp: {1,3,4} = 3. But what about infinite sets? We say that a set S has at least as great cardinality as set T, written S T, if
More information1 Linear programming relaxation
Cornell University, Fall 2010 CS 6820: Algorithms Lecture notes: Primal-dual min-cost bipartite matching August 27 30 1 Linear programming relaxation Recall that in the bipartite minimum-cost perfect matching
More informationLecture 3: Art Gallery Problems and Polygon Triangulation
EECS 396/496: Computational Geometry Fall 2017 Lecture 3: Art Gallery Problems and Polygon Triangulation Lecturer: Huck Bennett In this lecture, we study the problem of guarding an art gallery (specified
More informationProbabilistic Graphical Models
School of Computer Science Probabilistic Graphical Models Theory of Variational Inference: Inner and Outer Approximation Eric Xing Lecture 14, February 29, 2016 Reading: W & J Book Chapters Eric Xing @
More informationWelcome Back to Fundamentals of Multimedia (MR412) Fall, 2012 Lecture 10 (Chapter 7) ZHU Yongxin, Winson
Welcome Back to Fundamentals of Multimedia (MR412) Fall, 2012 Lecture 10 (Chapter 7) ZHU Yongxin, Winson zhuyongxin@sjtu.edu.cn 2 Lossless Compression Algorithms 7.1 Introduction 7.2 Basics of Information
More informationBELOW, we consider decoding algorithms for Reed Muller
4880 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 52, NO. 11, NOVEMBER 2006 Error Exponents for Recursive Decoding of Reed Muller Codes on a Binary-Symmetric Channel Marat Burnashev and Ilya Dumer, Senior
More information4 Fractional Dimension of Posets from Trees
57 4 Fractional Dimension of Posets from Trees In this last chapter, we switch gears a little bit, and fractionalize the dimension of posets We start with a few simple definitions to develop the language
More informationCSC 310, Fall 2011 Solutions to Theory Assignment #1
CSC 310, Fall 2011 Solutions to Theory Assignment #1 Question 1 (15 marks): Consider a source with an alphabet of three symbols, a 1,a 2,a 3, with probabilities p 1,p 2,p 3. Suppose we use a code in which
More informationGraph Contraction. Graph Contraction CSE341T/CSE549T 10/20/2014. Lecture 14
CSE341T/CSE549T 10/20/2014 Lecture 14 Graph Contraction Graph Contraction So far we have mostly talking about standard techniques for solving problems on graphs that were developed in the context of sequential
More informationData Compression - Seminar 4
Data Compression - Seminar 4 October 29, 2013 Problem 1 (Uniquely decodable and instantaneous codes) Let L = p i l 100 i be the expected value of the 100th power of the word lengths associated with an
More informationEE67I Multimedia Communication Systems Lecture 4
EE67I Multimedia Communication Systems Lecture 4 Lossless Compression Basics of Information Theory Compression is either lossless, in which no information is lost, or lossy in which information is lost.
More informationEntropy Coding. - to shorten the average code length by assigning shorter codes to more probable symbols => Morse-, Huffman-, Arithmetic Code
Entropy Coding } different probabilities for the appearing of single symbols are used - to shorten the average code length by assigning shorter codes to more probable symbols => Morse-, Huffman-, Arithmetic
More informationComputational Geometry: Lecture 5
Computational Geometry: Lecture 5 Don Sheehy January 29, 2010 1 Degeneracy In many of the algorithms that we have discussed so far, we have run into problems when that input is somehow troublesome. For
More informationLossless Compression Algorithms
Multimedia Data Compression Part I Chapter 7 Lossless Compression Algorithms 1 Chapter 7 Lossless Compression Algorithms 1. Introduction 2. Basics of Information Theory 3. Lossless Compression Algorithms
More informationMultidimensional Divide and Conquer 2 Spatial Joins
Multidimensional Divide and Conque Spatial Joins Yufei Tao ITEE University of Queensland Today we will continue our discussion of the divide and conquer method in computational geometry. This lecture will
More informationMatching Algorithms. Proof. If a bipartite graph has a perfect matching, then it is easy to see that the right hand side is a necessary condition.
18.433 Combinatorial Optimization Matching Algorithms September 9,14,16 Lecturer: Santosh Vempala Given a graph G = (V, E), a matching M is a set of edges with the property that no two of the edges have
More informationData Structures and Algorithms Dr. Naveen Garg Department of Computer Science and Engineering Indian Institute of Technology, Delhi.
Data Structures and Algorithms Dr. Naveen Garg Department of Computer Science and Engineering Indian Institute of Technology, Delhi Lecture 18 Tries Today we are going to be talking about another data
More informationAlgorithms Dr. Haim Levkowitz
91.503 Algorithms Dr. Haim Levkowitz Fall 2007 Lecture 4 Tuesday, 25 Sep 2007 Design Patterns for Optimization Problems Greedy Algorithms 1 Greedy Algorithms 2 What is Greedy Algorithm? Similar to dynamic
More informationCSC 373: Algorithm Design and Analysis Lecture 4
CSC 373: Algorithm Design and Analysis Lecture 4 Allan Borodin January 14, 2013 1 / 16 Lecture 4: Outline (for this lecture and next lecture) Some concluding comments on optimality of EST Greedy Interval
More informationMOST attention in the literature of network codes has
3862 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 56, NO. 8, AUGUST 2010 Efficient Network Code Design for Cyclic Networks Elona Erez, Member, IEEE, and Meir Feder, Fellow, IEEE Abstract This paper introduces
More informationGenerell Topologi. Richard Williamson. May 27, 2013
Generell Topologi Richard Williamson May 27, 2013 1 1 Tuesday 15th January 1.1 Topological spaces definition, terminology, finite examples Definition 1.1. A topological space is a pair (X, O) of a set
More information1 (15 points) LexicoSort
CS161 Homework 2 Due: 22 April 2016, 12 noon Submit on Gradescope Handed out: 15 April 2016 Instructions: Please answer the following questions to the best of your ability. If you are asked to show your
More information1 Overview. 2 Applications of submodular maximization. AM 221: Advanced Optimization Spring 2016
AM : Advanced Optimization Spring 06 Prof. Yaron Singer Lecture 0 April th Overview Last time we saw the problem of Combinatorial Auctions and framed it as a submodular maximization problem under a partition
More informationCS 337 Project 1: Minimum-Weight Binary Search Trees
CS 337 Project 1: Minimum-Weight Binary Search Trees September 6, 2006 1 Preliminaries Let X denote a set of keys drawn from some totally ordered universe U, such as the set of all integers, or the set
More informationprinceton univ. F 15 cos 521: Advanced Algorithm Design Lecture 2: Karger s Min Cut Algorithm
princeton univ. F 5 cos 5: Advanced Algorithm Design Lecture : Karger s Min Cut Algorithm Lecturer: Pravesh Kothari Scribe:Pravesh (These notes are a slightly modified version of notes from previous offerings
More informationMassachusetts Institute of Technology Department of Electrical Engineering and Computer Science Algorithms For Inference Fall 2014
Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science 6.438 Algorithms For Inference Fall 2014 Recitation-6: Hardness of Inference Contents 1 NP-Hardness Part-II
More informationLecture 15 : Review DRAFT
CS/Math 240: Introduction to Discrete Mathematics 3/10/2011 Lecture 15 : Review Instructor: Dieter van Melkebeek Scribe: Dalibor Zelený DRAFT Today slectureservesasareviewofthematerialthatwillappearonyoursecondmidtermexam.
More informationDepartment of Computer Applications. MCA 312: Design and Analysis of Algorithms. [Part I : Medium Answer Type Questions] UNIT I
MCA 312: Design and Analysis of Algorithms [Part I : Medium Answer Type Questions] UNIT I 1) What is an Algorithm? What is the need to study Algorithms? 2) Define: a) Time Efficiency b) Space Efficiency
More informationThe Encoding Complexity of Network Coding
The Encoding Complexity of Network Coding Michael Langberg Alexander Sprintson Jehoshua Bruck California Institute of Technology Email mikel,spalex,bruck @caltech.edu Abstract In the multicast network
More informationTowards the Proof of the PCP Theorem
CS640 Computational Complexity Towards the Proof of the PCP Theorem Instructor: Manindra Agrawal Scribe: Ramprasad Saptharishi Last class we saw completed our discussion on expander graphs. We shall now
More informationCS2 Algorithms and Data Structures Note 10. Depth-First Search and Topological Sorting
CS2 Algorithms and Data Structures Note 10 Depth-First Search and Topological Sorting In this lecture, we will analyse the running time of DFS and discuss a few applications. 10.1 A recursive implementation
More informationChapter 5: Data compression. Chapter 5 outline
Chapter 5: Data compression Chapter 5 outline 2 balls weighing problem Examples of codes Kraft inequality Optimal codes + bounds Kraft inequality for uniquely decodable codes Huffman codes Shannon-Fano-Elias
More informationEDAA40 At home exercises 1
EDAA40 At home exercises 1 1. Given, with as always the natural numbers starting at 1, let us define the following sets (with iff ): Give the number of elements in these sets as follows: 1. 23 2. 6 3.
More informationMathematical and Algorithmic Foundations Linear Programming and Matchings
Adavnced Algorithms Lectures Mathematical and Algorithmic Foundations Linear Programming and Matchings Paul G. Spirakis Department of Computer Science University of Patras and Liverpool Paul G. Spirakis
More information6. Lecture notes on matroid intersection
Massachusetts Institute of Technology 18.453: Combinatorial Optimization Michel X. Goemans May 2, 2017 6. Lecture notes on matroid intersection One nice feature about matroids is that a simple greedy algorithm
More informationMassachusetts Institute of Technology Department of Electrical Engineering and Computer Science Algorithms For Inference Fall 2014
Suggested Reading: Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science 6.438 Algorithms For Inference Fall 2014 Probabilistic Modelling and Reasoning: The Junction
More informationAlgorithms and Data Structures
Algorithms and Data Structures Spring 2019 Alexis Maciel Department of Computer Science Clarkson University Copyright c 2019 Alexis Maciel ii Contents 1 Analysis of Algorithms 1 1.1 Introduction.................................
More information8 Integer encoding. scritto da: Tiziano De Matteis
8 Integer encoding scritto da: Tiziano De Matteis 8.1 Unary code... 8-2 8.2 Elias codes: γ andδ... 8-2 8.3 Rice code... 8-3 8.4 Interpolative coding... 8-4 8.5 Variable-byte codes and (s,c)-dense codes...
More informationLecture 4: examples of topological spaces, coarser and finer topologies, bases and closed sets
Lecture 4: examples of topological spaces, coarser and finer topologies, bases and closed sets Saul Glasman 14 September 2016 Let s give the definition of an open subset of R. Definition 1. Let U R. We
More information