Coding Theorems for Reversible Embedding

Transcription

1 Coding Theorems for Reversible Embedding Frans Willems and Ton Kalker T.U. Eindhoven, Philips Research DIMACS, March 16-19, 2003

2 Outline 1. Gelfand-Pinsker coding theorem 2. Noise-free embedding 3. Reversible embedding 4. Robust and reversible embedding 5. Partially reversible embedding 6. Remarks 1

3 I. The Gelfand-Pinsker Coding Theorem W Y N Z N Y N = e(w, X N ) Ŵ = d(z N ) Pc(z y, x) X N Ps(x) Messages: Pr{W = w} = 1/M for w {1, 2,, M}. Side information: Pr{X N = x N } = Π N n=1 P s(xn) for x N X N. Channel: discrete memoryless {Y X, Pc(z y, x), Z}. Error probability: P E = Pr{Ŵ W }. Rate: R = 1 N log 2(M). Ŵ 2

4 Capacity The side-information capacity C si is the largest ρ such that for all ɛ > 0 there exist for all large enough N encoders and decoders with R ρ ɛ and P E ɛ. THEOREM (Gelfand-Pinsker [1980]): C si = max Pt(u,y x) I(U; Z) I(U; X). (1) Achievability proof: Fix a test-channel Pt(u, y x). Consider sets Aɛ( ) of strongly typical sequences, etc. (a) For each message index w {1,, 2 NR }, generate 2 NR u sequences u N at random according to P (u) = x,y Ps(x)Pt(u, y x). Give these sequences the label w. (b) When message index w has to be transmitted choose a sequence u N having label w such that (u N, x N ) Aɛ(U, X). Such a sequence exists almost always if Ru > I(U; X) (roughly). 3

5 (c) The input sequence y N results from applying the channel P (y u, x) = Pt(y, u x)/ y Pt(u, y x) to u N and x N. Then y N is transmitted. (d) The decoder upon receiving z N, looks for the unique sequence u N such that (u N, z N ) Aɛ(U, Z). If R + Ru < I(U; Z) (roughly) such a unique sequence exists. The message index is the label of u N. Conclusion is that R < I(U; Z) I(U; X) is achievable. Observations A: As an intermediate result the decoder recovers the sequence u N. B: The transmitted u N is jointly typical with the side-info sequence x N, i.e. (u N, x N ) Aɛ(U, X) thus their joint composition is OK. Note that P (u, x) = y Ps(x)Pt(u, y x). 4

6 II. Noise-free Embedding X N Y N Ps(x) Y N = e(w, X N ) W Ŵ = d(y N ) Ŵ Messages: Pr{W = w} = 1 M for w {1, 2,, M}. Source (host): Pr{X N = x N } = Π n=1,n Ps(xn) for x N X N. Error probability: P E = Pr{Ŵ W }. Rate: R = 1 N log 2(M). Embedding distortion: Dxy = E[ N 1 n=1,n Dxy(Xn, en(w, X N ))] for some distortion matrix {Dxy(x, y), x X, y Y}. 5

7 Achievable region noise-free embedding A rate-distortion pair (ρ, xy) is said to be achievable if for all ɛ > 0 there exists for all large enough N encoders and decoders such that R ρ ɛ, Dxy xy + ɛ, P E ɛ. THEOREM (Chen [2000], Barron [2000]): The set of achievable rate-distortion pairs is equal to G nfe which is defined as G nfe = {(ρ, xy) : 0 ρ H(Y X), xy x,y P (x, y)dxy(x, y), for P (x, y) = Ps(x)Pt(y x)}. (2) Again {X, Pt(y x), Y} is called test-channel. 6

8 Proof: Achievability: In the Gelfand-Pinsker achievability proof, note that Z = Y (noiseless channel) and take the auxiliary random variable U = Y. Then (x N, y N ) Aɛ(X, Y ) hence Dxy is OK. For the embedding rate we obtain R = I(U; Z) I(U; X) = I(Y ; Y ) I(Y ; X) = H(Y X). Converse: Rate part: log2(m) H(W ) H(W Ŵ ) + Fano term H(W X N ) H(W X N, Y N ) + Fano term = I(W ; Y N X N ) + Fano term H(Y N X N ) + Fano term n=1,n H(Yn Xn) + Fano term NH(Y X) + Fano term, 7

9 where X and Y are random variables with Pr{(X, Y ) = (x, y)} = 1 N n=1,n for x X and y Y. Note that for x X Pr{(Xn, Yn) = (x, y)}, Pr{X = x} = Ps(x). Distortion part: Dxy = x N,y N Pr{(X N, Y N ) = (x N, y N )} 1 N n Dxy(xn, yn) = x,y Pr{(X, Y ) = (x, y)}dxy(x, y). Let P E 0, etc. 8

10 III. Reversible Embedding X N Y N Ps(x) Y N = e(w, X N ) W (Ŵ, ˆX N 1 ) = d(y N ) Ŵ ˆX 1 N Messages: Pr{W = w} = 1 M for w {1, 2,, M}. Source (host): Pr{X N = x N } = Π n=1,n Ps(xn) for x N X N. Error probability: P E = Pr{Ŵ W ˆX N 1 XN }. Rate: R = 1 N log 2(M). Embedding distortion: Dxy = E[ N 1 n=1,n Dxy(Xn, en(w, X N ))] for some distortion matrix {Dxy(x, y), x X, y Y}. Inspired by Fridrich, Goljan, and Du, Lossless data embedding for all image formats, Proc. SPIE, Security and Watermarking of Multimedia Contents, San Jose, CA,

11 Achievable region for reversible embedding A rate-distortion pair (ρ, xy) is said to be achievable if for all ɛ > 0 there exists for all large enough N encoders and decoders such that R ρ ɛ, Dxy xy + ɛ, P E ɛ. RESULT (Kalker-Willems [2002]): The set of achievable rate-distortion pairs is equal to Gre which is defined as Gre = {(ρ, xy) : 0 ρ H(Y ) H(X), xy x,y P (x, y)dxy(x, y), for P (x, y) = Ps(x)Pt(y x)}. (3) Note that {X, Pt(y x), Y} is the test channel. 10

12 Proof: Achievability: In the Gelfand-Pinsker achievability proof, note that Z = Y (noiseless channel) and take the auxiliary random variable U = [X, Y ]. Then x N can be reconstructed by the decoder and (x N, y N ) Aɛ(X, Y ) hence Dxy is OK. For the embedding rate we obtain R = I(U; Z) I(U; X) = I([X, Y ]; Y ) I([X, Y ]; X) = H(Y ) H(X). Converse: Rate part: log2(m) H(W ) H(W, X N Ŵ, ˆX 1 N ) + Fano term = H(W, X N ) H(W, X N Ŵ, ˆX 1 N ) H(XN ) + Fano term H(W, X N ) H(W, X N Y N, Ŵ, ˆX N 1 ) H(XN ) + Fano term = I(W, X N ; Y N ) H(X N ) + Fano term = H(Y N ) H(X N ) + Fano term n=1,n [H(Yn) H(Xn)] + Fano term N[H(Y ) H(X)] + Fano term, 11

13 where X and Y are random variables with Pr{(X, Y ) = (x, y)} = 1 N n=1,n for x X and y Y. Note that for x X Pr{(Xn, Yn) = (x, y)}, Pr{X = x} = Ps(x). Distortion part: Dxy = x N,y N Pr{(X N, Y N ) = (x N, y N )} 1 N n Dxy(xn, yn) = x,y Pr{(X, Y ) = (x, y)}dxy(x, y). Let P E 0, etc. 12

14 Example: Binary source, Hamming distortion px 1 py d1 1 X Y 0 d0 0 Since we can write xy pxd1 + (1 px)d0 py = px(1 d1) + (1 px)d0 py xy + px(1 2d1). Assume w.l.o.g. that px 1/2. First let xy be such that xy +px 1/2 or xy 1/2 px. Then we have py xy + px 1/2, 13

15 and hence ρ h(py) h(px) h(px + xy) h(px). However ρ = h(px + xy) h(px) is achievable with xy by taking d1 = 0 and d0 = xy 1 px. Note that the test channel is not symmetric and that d0 = xy 1 px 1/2 p x 1 px 1/2. For xy + px 1/2 the rate is bounded as ρ 1 h(px) but also achievable. 14

16 15 Horizontal axis xy, vertical axis ρ, for px = 0.2. Maximum embedding rate 1 h(0.2) DISTORTION RATE in BITS Plot of rate-distortion region Gre

17 Another perspective y N (k) y N (k + 1) w(k) w(k + 1) x N (k) x N (k + 1) Consider a blocked system with blocks of length N. In block k message bits can be (noise-free) embedded with rate H(Y X) and corresponding distortion. Then in block k + 1 message bits are embedded that allow for reconstruction of x N (k) given y N (k). This requires NH(X Y ) bits. Therefore the resulting embedding rate is R = H(Y X) H(X Y ) = H(Y, X) H(X) H(X Y ) = H(Y ) H(X). 16

18 IV. Robust and Reversible Embedding X N Y N Ps(x) Y N = e(w, X N ) W Pc(z y) Z N (Ŵ, ˆX N 1 ) = d(zn ) Ŵ ˆX 1 N Messages: Pr{W = w} = 1 M for w {1, 2,, M}. Source (host): Pr{X N = x N } = Π n=1,n Ps(xn) for x N X N. Channel: discrete memoryless {Y, Pc(z y), Z}. Error probability: P E = Pr{Ŵ W ˆX N 1 XN }. Rate: R = 1 N log 2(M). Embedding distortion: Dxy = E[ N 1 n=1,n Dxy(Xn, en(w, X N ))] for some distortion matrix {Dxy(x, y), x X, y Y}. 17

19 Achievable region for robust and reversible embedding A rate-distortion pair (ρ, xy) is said to be achievable if for all ɛ > 0 there exists for all large enough N encoders and decoders such that R ρ ɛ, Dxy xy + ɛ, P E ɛ. RESULT (Willems-Kalker [2003]): The set of achievable rate-distortion pairs is equal to Grre which is defined as Grre = {(ρ, xy) : 0 ρ I(Y ; Z) H(X), xy x,y P (x, y)dxy(x, y), for P (x, y, z) = Ps(x)Pt(y x)pc(z y)}. (4) 18

20 Proof: Achievability: In the Gelfand-Pinsker achievability proof again take the auxiliary random variable U = [X, Y ]. Then x N can be reconstructed by the decoder and since (x N, y N ) Aɛ(X, Y ) the embedding distortion Dxy is OK. For the embedding rate we obtain R = I(U; Z) I(U; X) = I([X, Y ]; Z) I([X, Y ]; X) = I(Y ; Z) H(X). Converse: Rate part: log2(m) H(W ) H(W, X N Ŵ, ˆX 1 N ) + Fano term = H(W, X N ) H(W, X N Ŵ, ˆX 1 N ) H(XN ) + Fano term H(W, X N ) H(W, X N Z N, Ŵ, ˆX N 1 ) H(XN ) + Fano term = I(W, X N ; Z N ) H(X N ) + Fano term = I(Y N ; Z N ) H(X N ) + Fano term n=1,n [I(Yn; Zn) H(Xn)] + Fano term N[I(Y ; Z) H(X)] + Fano term, 19

21 where X, Y and Z are random variables with Pr{(X, Y, Z) = (x, y, z)} = 1 N n=1,n for x X, y Y, and z Z. Note that for x X Pr{(Xn, Yn, Zn) = (x, y, z)}, Pr{X = x} = Ps(x). and for y Y and z Z Pr{Z = z Y = y} = Pc(z y). Distortion part: Dxy = x N,y N Pr{(X N, Y N ) = (x N, y N )} 1 N n Dxy(xn, yn) = x,y Pr{(X, Y ) = (x, y)}dxy(x, y). Let P E 0, etc. 20

22 Example: Binary source, Hamming distortion, binary symmetric channel px py 1 pz d1 1 α 1 X Y Z d0 α Similar analysis as before. 21

23 22 Horizontal axis xy, vertical axis ρ for px = α = 0.1. Minimal distortion 0.218, maximum embedding rate 1 h(0.1) h(0.1) DISTORTION RATE in BITS Plot of achievable region Grre

24 The zero-rate case: Robustification X N Y N Ps(x) Y N = e(x N ) Z N ˆX 1 N Pc(z y) ˆX N 1 = d(zn ) Source (host): Pr{X N = x N } = Π n=1,n Ps(xn) for x N X N. Channel: discrete memoryless {Y, Pc(z y), Z}. Error probability: P E = Pr{ ˆX N 1 XN }. Robustification distortion: Dxy = E[ N 1 n=1,n Dxy(Xn, en(w, X N ))] for some distortion matrix {Dxy(x, y), x X, y Y}. 23

25 Achievable distortions for robustification A distortion xy is said to be achievable if for all ɛ > 0 there exists for all large enough N encoders and decoders such that Dxy xy + ɛ, P E ɛ. RESULT: The set of achievable distortions is equal to G rob which is defined as G rob = { xy : xy x,y P (x, y)dxy(x, y), for P (x, y, z) = Ps(x)Pt(y x)pc(z y) such that H(X) I(Y ; Z)}. (5) Related to Shannon s separation principle! Robustification is not possible if H(X) > max P I(Y ; Z). t(y) 24

26 V. Partially Reversible Embedding X N Y N Ps(x) Y N = e(w, X N ) W Ŵ = d(y N ) V N = f(y N ) Ŵ V N Messages: Pr{W = w} = 1 M for w {1, 2,, M}. Source (host): Pr{X N = x N } = Π n=1,n Ps(xn) for x N X N. Error probability: P E = Pr{Ŵ W }. Rate: R = 1 N log 2(M). Embedding distortion: Dxy = E[ N 1 n=1,n Dxy(Xn, en(w, X N ))] for some distortion matrix {Dxy(x, y), x X, y Y}. Restoration distortion: Dxv = E[ N 1 n=1,n Dxv(Xn, fn(y N ))] for a distortion matrix {Dxv(x, v), x X, z V}. 25

27 Achievable region for partially reversible embedding A rate-distortion triple (ρ, xy, xv) is said to be achievable if for all ɛ > 0 there exists for all large enough N encoders and decoders such that R ρ ɛ, Dxy xy + ɛ, Dxv xv + ɛ, P E ɛ. RESULT (Willems-Kalker [2002]): The set of achievable rate-distortion triples is given by Gpre which is defined as Gpre = {(ρ, xy, xv) : 0 ρ H(Y ) I(X; Y, V ), xy P (x, y, v)dxy(x, y), x,y,v xv x,y,v P (x, y, v)dxv(x, v), for P (x, y, v) = Ps(x)Pt(y, v x)}. (6) 26

28 Proof: Achievability: In the Gelfand-Pinsker achievability proof, note again that Z = Y (noiseless channel) and take the auxiliary random variable U = [Y, V ]. Then v N can be reconstructed by the decoder and since (x N, y N, v N ) Aɛ(X, Y, V ) both Dxy and Dxv are OK. For the embedding rate we obtain R = I(U; Z) I(U; X) = I([Y, V ]; Y ) I([Y, V ]; X) = H(Y ) I(X; Y, V ). Converse: Rate part: log2(m) = H(Y N, V N, W ) H(Y N, V N W ) H(Y N ) + H(W Ŵ ) I(Y N, V N ; X N W ) H(Y N ) H(X N W ) + H(X N W, Y N, V N ) + Fano term H(Y N ) H(X N ) + H(X N Y N, V N ) + Fano term H(Yn) H(Xn) + H(Xn Yn, Vn) + Fano term n=1,n n=1,n n=1,n N[H(Y ) H(X) + H(X Y, V )] + Fano term = N[H(Y ) I(X; Y, V )] + Fano term, 27

29 where X, Y and V are random variables with Pr{(X, Y, V ) = (x, y, v)} = 1 N n=1,n for x X, y Y, and v V. Note that for x X Pr{(Xn, Yn, Vn) = (x, y, v)}, Pr{X = x} = Ps(x). Distortion parts: Dxy = x N,y N Pr{(X N, Y N ) = (x N, y N )} 1 N n Dxy(xn, yn) = x,y Pr{(X, Y ) = (x, y)}dxy(x, y). Dxv = x N,v N Pr{(X N, V N ) = (x N, v N )} 1 N n Dxv(xn, vn) = x,v Pr{(X, V ) = (x, v)}dxv(x, v). Let P E 0, etc. 28

30 The other perspective again Consider a blocked system with blocks of length N. In block k a message can be (noise-free) embedded with rate H(Y X) and the corresponding distortion. Then in block k + 1 data is embedded that specifies a restoration sequence v N (k) given y N (k). This requires NI(X; V Y ) bits. Therefore the remaining embedding rate is R = H(Y X) I(X; V Y ) = H(Y X) H(X Y ) + H(X Y, V ) = H(Y ) H(X) + H(X Y, V ) = H(Y ) I(X; Y, V ). 29

31 The zero-rate case: Self-Embedding Ps(x) X N Y N Y N = e(x N ) V N = f(y N ) V N Source (host): Pr{X N = x N } = Π n=1,n Ps(xn) for x N X N. Embedding distortion: Dxy = E[ N 1 n=1,n Dxy(Xn, en(w, X N ))] for some distortion matrix {Dxy(x, y), x X, y Y}. Restoration distortion: Dxv = E[ N 1 n=1,n Dxv(Xn, fn(y N ))] for a distortion matrix {Dxv(x, v), x X, z V}. 30

32 Achievable distortions for self-embedding A distortion pair ( xy, xv) is said to be achievable if for all ɛ > 0 there exists for all large enough N encoders and decoders such that Dxy xy + ɛ, Dxv xv + ɛ. RESULT (Willems-Kalker [2002]): The set of achievable distortion pairs is equal to Gse which is defined as Gse = {( xy, xv) : xy x,y,v xv x,y,v P (x, y, v)dxy(x, y), P (x, y, v)dxv(x, v), for P (x, y, v) = Ps(x)Pt(y, v x) such that H(Y ) I(X; Y, V )}. (7) Self-embedding is putting a vector quantizer into a scalar quantizer. Or making an abstract index to a restoration vector v N meaningful. 31

33 VI. Remarks 1. Our results are related to results of Sutivong, Cover, et al. Slightly different setups however. Embedding distortion. 2. We cannot do the partially reversible AND robust case. An achievable region would be similar to the Sutivong, Cover, Chiang, Kim [2002] region. No converse. 3. Coding techniques for the reversible case have been studied (with Deran Maas [2002]). 4. Open problems: (A) Arimoto-Blahut methods to compute the ratedistortion functions, (B) Coding techniques, especially for the zerorate cases. 32