Will Monroe July 21, with materials by Mehran Sahami and Chris Piech. Joint Distributions

Transcription

1 Will Monroe July 1, 017 with materials by Mehran Sahami and Chris Piech Joint Distributions

2 Review: Normal random variable An normal (= Gaussian) random variable is a good approximation to many other distributions. It often results from sums or averages of independent random variables. X N (μ, σ ) 1 x μ ( 1 σ ) f X ( x)= e σ π

3 Review: Normal fact sheet mean X N (μ, σ ) variance (σ = standard deviation) PDF: CDF: 1 x μ σ ( ) 1 f X ( x)= e σ π x x μ F X ( x)=φ σ = dx f X ( x) ( ) (no closed form) expectation: variance: E[ X ]=μ Var( X )=σ

4 Review: Normal approximation to the binomial large n, medium p P( X =k ) Bin (n, p) N (μ, σ ) k

5 Linear transform of a normal Adding a constant to a normal? Add the constant to the mean. Multiplying a normal by a constant? Multiply the mean by the constant and the variance by the square of the constant. X N (μ, σ ) a X +b N (a μ+b, a σ )

6 Most real-world problems involve multiple random variables

7 Joint distributions A joint distribution combines multiple random variables. Its PDF or PMF gives the probability or relative likelihood of both random variables taking on specific values. p X,Y (a, b)=p ( X =a,y =b)

8 A table of probabilities Two random variables: X, Y Each can take on values {0, 1, } Y X P(X = 0, Y = 0) and all add up to 1 P(X = 1, Y = )

9 A just-for-fun demo

10 Joint probability mass function A joint probability mass function gives the probability of more than one discrete random variable each taking on a specific value (an AND of the + values). p X,Y (a, b)=p ( X =a,y =b) Y X

11 Joint probability density function A joint probability density function gives the relative likelihood of more than one continuous random variable each taking on a specific value. P (a1 < X a, b1 <Y b )= a b dx dy f X,Y ( x, y ) a1 b1

12 Joint probability density function f X, Y (x, y ) y x P (a1 < X a, b1 <Y b )= a b dx dy f X,Y ( x, y ) a1 b1 plot by Academo

13 Multiple integrals (without tears) X and Y are two continuous random variables: 0 X 1, 0 Y f X, Y (x, y)= 1 ( 1 { x y if 0 x 1, 0 y 0 otherwise evaluate the inner integral (treat outer variable as constant) ) dy dx ( x y)= dy dx ( x y ) y=0 x=0 y=0 x=0 1 1 = dy y x x=0 y=0 1 1 = dy y = y 4 y=0 [ ] [ ] now evaluate the outer integral = =1 4 y=0

14 Marginalization Marginal probabilities give the distribution of a subset of the variables (often, just one) of a joint distribution. Sum/integrate over the variables you don t care about. p X (a)= p X,Y (a, y) y f X (a)= dy f X,Y (a, y)

15 A just-for-fun demo

16 Defects on a hard drive A single point defect is uniformly distributed over a disk or radius R. 1 if x + y R f X, Y (x, y)= π R 0 otherwise { f X ( x)= dy f X,Y (x, y ) 1 = dy π R y : x + y R + R x 1 R x = dy = π R y= R x πr

17 Break time!

18 Joint cumulative distribution function F X, Y (x, y)=p( X x, Y y ) to 1 as x +, y +, y to 0 as x -, y -, x plot by Academo

19 Probabilities from joint CDFs P (a1 < X a, b1 <Y b )=F X,Y (a, b ) F X,Y (a1, b ) F X,Y (a, b1 ) + F X,Y (a1, b1 ) b b1 a1 a

20 Probabilities from joint CDFs P (a1 < X a, b1 <Y b )=F X,Y (a, b ) F X,Y (a1, b ) F X,Y (a, b1 ) + F X,Y (a1, b1 ) b b1 a1 a

21 Probabilities from joint CDFs P (a1 < X a, b1 <Y b )=F X,Y (a, b ) F X,Y (a1, b ) F X,Y (a, b1 ) + F X,Y (a1, b1 ) b b1 a1 a

22 Probabilities from joint CDFs P (a1 < X a, b1 <Y b )=F X,Y (a, b ) F X,Y (a1, b ) F X,Y (a, b1 ) + F X,Y (a1, b1 ) b b1 a1 a

23 Probabilities from joint CDFs P (a1 < X a, b1 <Y b )=F X,Y (a, b ) F X,Y (a1, b ) F X,Y (a, b1 ) + F X,Y (a1, b1 ) b b1 a1 a

24 Probabilities from joint CDFs P (a1 < X a, b1 <Y b )=F X,Y (a, b ) F X,Y (a1, b ) F X,Y (a, b1 ) + F X,Y (a1, b1 ) b b1 a1 a

25 Probabilities from joint CDFs P (a1 < X a, b1 <Y b )=F X,Y (a, b ) F X,Y (a1, b ) F X,Y (a, b1 ) + F X,Y (a1, b1 ) b b1 a1 a

26 Probabilities from joint CDFs P (a1 < X a, b1 <Y b )=F X,Y (a, b ) F X,Y (a1, b ) F X,Y (a, b1 ) + F X,Y (a1, b1 ) b b1 a1 a

27 Review: Linearity of expectation Adding random variables or constants? Add the expectations. Multiplying by a constant? Multiply the expectation by the constant. E [ax +by +c ]=ae [ X ]+be [Y ]+c X X X + 4

28 Expectation of a function of two variables Exactly like a function of one variable, but with the joint PMF! E [ g( X, Y )]= g(x, y) p X,Y (x, y ) x y E [ g( X, Y )]= dx dy g(x, y ) f X,Y ( x, y )

29 Proof of expectation of sum E [ X +Y ]=E [ g( X, Y )] = g(x, y) p X,Y ( x, y ) x y = [ x+ y ] p X,Y ( x, y ) x y = x p X,Y (x, y)+ y p X,Y ( x, y ) x y x y = x p X,Y (x, y)+ y p X,Y ( x, y ) x y y = x p X ( x)+ y py ( y) x =E[ X ]+ E [Y ] y x

30 Another way to compute expectation FY ( y ) You can integrate y times the PMF, or you can integrate 1 minus the CDF! E [Y ]= dy P(Y > y) 0 = dy (1 F Y ( y)) 0 y

31 Non-negative RV expectation lemma: Rearranging terms E [ X ]=0 P ( X=0)+1 P( X =1)+ P( X =)+3 P( X =3)+ =0 P( X =0)+1 P( X =1)+ P( X =)+3 P ( X =3)+ +1 P( X1 =1)+ P( X =)+3 P ( X =3)+ +1 P( X =1)+ P( X =)+3 P ( X =3) =0 P( X =0)+1 P( X =1)+ P( X =)+3 P ( X =3)+ +1 P( X =1)+ P( X =)+3 P ( X =3)+ +1 P( X =1)+ P( X =)+3 P ( X =3)+ =0 P( X =0)+1 P( X 1) +1 P( X =1)+ P( X ) +1 P( X =1)+ P( X =)+3 P ( X 3)+ = P( X i) [Addendum] i=1

32 Non-negative RV expectation lemma: Graphically p_x(x) E [ X ]= P(X 4) P(X 3) P(X ) P(X 1) 0 0 P(X=0) 1 P(X=1) P(X=) 3 P(X=3) x [Addendum] 4 P(X=4)

33 Defects on a hard drive A single point defect is uniformly distributed over a disk or radius R. 1 if x + y R f X, Y (x, y)= π R 0 otherwise { D= X +Y R R a E [ D]= da P (D>a)= da 1 R 0 0 ( 3 a = a 3R [ ] R ) = R 3 a=0

34 Multinomial random variable An multinomial random variable records the number of times each outcome occurs, when an experiment with multiple outcomes (e.g. die roll) is run multiple times. X 1,, X m MN (n, p1, p,, p m) vector! P ( X 1 =c 1, X =c,, X m=c m) c c c n = p1 p p m c 1, c,, c m ( ) 1 m

35 Roll all of the dice! A 6-sided die is rolled 7 times. What is the probability we get: 1 one 1 two 0 threes fours 0 fives 3 sixes? X 1,, X 6 MN (7,,,,,, ) P ( X 1 =1, X =1, X 3 =0, X 4 =, X 5 =0, X 6 =3) = =40 6 1,1,0,,0, ( )( ) ( ) ( ) ( ) ( ) ( ) 7 ()