[PDF] The Bivariate Normal Distribution

34609_6Bivariate_Normal.pdf

The Bivariate Normal Distribution

This is Section 4.7 of the 1st edition (2002) of the book Introduc- tion to Probability, by D. P. Bertsekas and J. N. Tsitsiklis. The material in this section was not included in the 2nd edition (2008). LetUandVbe two independent normal random variables, and consider two new random variablesXandYof the form

X=aU+bV,

Y=cU+dV,

wherea,b,c,d, are some scalars. Each one of the random variablesXandYis normal, since it is a linear function of independent normal random variables.† Furthermore, becauseXandYare linear functions of the same two independent normal random variables, their joint PDF takes a special form, known as thebi- variate normalPDF. The bivariate normal PDF has several useful and elegant properties and, for this reason, it is a commonly employed model. In this section, we derive many such properties, both qualitative and analytical, culminating in a closed-form expression for the joint PDF. To keep the discussion simple, we restrict ourselves to the case whereXandYhave zero mean.

Jointly Normal Random Variables

Two random variablesXandYare said to bejointly normalif they can beexpressedintheform

X=aU+bV,

Y=cU+dV,

whereUandVare independent normal random variables. Note that ifXandYare jointly normal, then any linear combination Z=s 1 X+s 2 Y †For the purposes of this section, we adopt the following convention. A random variable which is always equal to a constant will also be called normal, with zero variance, even though it does not have a PDF. With this convention, the family of normal random variables is closed under linear operations. That is, ifXis normal, thenaX+bis also normal, even ifa=0. 1

2The Bivariate Normal Distribution

has a normal distribution. The reason is that if we haveX=aU+bVand Y=cU+dVfor some independent normal random variablesUandV,then Z=s 1 (aU+bV)+s 2 (cU+dV)=(as 1 +cs 2 )U+(bs 1 +ds 2 )V. Thus,Zis the sum of the independent normal random variables (as 1 +cs 2 )U and (bs 1 +ds 2 )V, and is therefore normal. A very important property of jointly normal random variables, and which will be the starting point for our development, is that zero correlation implies independence.

Zero Correlation Implies Independence

If two random variablesXandYare jointly normal and are uncorrelated, then they are independent. This property can be verified using multivariate transforms, as follows. Suppose thatUandVare independent zero-mean normal random variables, and thatX=aU+bVandY=cU+dV,sothatXandYare jointly normal. We assume thatXandYare uncorrelated, and we wish to show that they are independent. Our first step is to derive a formula for the multivariate transform M X,Y (s 1 ,s 2 ) associated withXandY. Recall that ifZis a zero-mean normal random variable with varianceσ 2Z , the associated transform is E[e sZ ]=M Z (s)=e σ 2Z s 2 /2 , which implies that E[e Z ]=M Z (1) =e σ 2Z /2 .

Let us fix some scalarss

1 ,s 2 ,andletZ=s 1 X+s 2

Y. The random variableZ

is normal, by our earlier discussion, with variance σ 2Z =s 21
σ 2X +s 22
σ 2Y . This leads to the following formula for the multivariate transform associated with the uncorrelated pairXandY: M X,Y (s 1 ,s 2 )=E[e s 1 X+s 2 Y ] =E[e Z ] =e (s 21
σ 2X +s 22
σ 2Y )/2. Let nowXandYbeindependentzero-mean normal random variables with the same variancesσ 2X andσ 2Y asXandY, respectively. SinceXandYare independent, they are also uncorrelated, and the preceding argument yields M X,Y (s 1 ,s 2 )=e (s 21
σ 2X +s 22
σ 2Y )/2. .

The Bivariate Normal Distribution3

Thus, the two pairs of random variables (X,Y)and(X,Y) are associated with the same multivariate transform. Since the multivariate transform completely determines the joint PDF, it follows that the pair (X,Y) has the same joint

PDF as the pair (

X,Y). SinceXandYare independent,XandYmust also

be independent, which establishes our claim.

The Conditional Distribution ofXGivenY

We now turn to the problem of estimatingXgiven the value ofY.Toavoid uninteresting degenerate cases, we assume that bothXandYhave positive variance. Let us define† ˆ

X=ρσ

X σ Y

Y,˜X=X-ˆX,

where

ρ=E[XY]

σ X σ Y is the correlation coefficient ofXandY.SinceXandYare linear combinations of independent normal random variablesUandV, it follows thatYand˜Xare also linear combinations ofUandV.Inparticular,Yand˜Xare jointly normal.

Furthermore,

E[Y˜X]=E[YX]-E[YˆX]=ρσ

X σ Y -ρσ X σ Y σ 2Y =0. Thus,Yand˜Xare uncorrelated and, therefore, independent. SinceˆXis a scalar multiple ofY, it follows thatˆXand˜Xare independent. We have so far decomposedXinto a sum of two independent normal ran- dom variables, namely,

X=ˆX+˜X=ρσ

X σ Y

Y+˜X.

We take conditional expectations of both sides, givenY,toobtain

E[X|Y]=ρσ

X σ Y

E[Y|Y]+E[˜X|Y]=ρσ

X σ Y

Y=ˆX,

where we have made use of the independence ofYand˜Xto setE[˜X|Y]=0.We have therefore reached the important conclusion that the conditional expectation E[X|Y] is a linear function of the random variableY. Using the above decomposition, it is now easy to determine the conditional PDF ofX. Given a value ofY, the random variableˆX=ρσ X

Y/σ

Y becomes †Comparing with the formulas in the preceding section, it is seen thatˆXis defined to be the linear least squares estimator ofX,and˜Xis the corresponding estimation error, although these facts are not needed for the argument that follows.

4The Bivariate Normal Distribution

a known constant, but the normal distribution of the random variable˜Xis unaffected, since˜Xis independent ofY. Therefore, the conditional distribution ofXgivenYis the same as the unconditional distribution of˜X,shiftedbyˆX. Since˜Xis normal with mean zero and some varianceσ

2˜X

, we conclude that the conditional distribution ofXis also normal with meanˆXand the same variance σ

2˜X

. The variance of˜Xcan be found with the following calculation: σ

2˜X

=E? ?

X-ρσ

X σ Y Y? 2 ? =σ 2X -2ρσ X σ Y

ρσ

X σ Y +ρ 2 σ 2X σ 2Y σ 2Y =(1-ρ 2 )σ 2X , wherewehavemadeuseofthepropertyE[XY]=ρσ X σ Y . We summarize our conclusions below. Although our discussion used the zero-mean assumption, these conclusions also hold for the non-zero mean case and we state them with this added generality; see the end-of-chapter problems.

Properties of Jointly Normal Random Variables

LetXandYbe jointly normal random variables.

•XandYare independent if and only if they are uncorrelated. •The conditional expectation ofXgivenYsatisfies

E[X|Y]=E[X]+ρσ

X σ Y ?Y-E[Y]?.

It is a linear function ofYand has a normal PDF.

•The estimation error˜X=X-E[X|Y] is zero-mean, normal, and independent ofY, with variance σ

2˜X

=(1-ρ 2 )σ 2X . •The conditional distribution ofXgivenYis normalwith meanE[X|Y] and varianceσ

2˜X

.

TheFormoftheBivariateNormalPDF

Having determined the parameters of the PDF of

˜Xand of the conditional PDF

ofX, we can give explicit formulas for these PDFs. We keep assuming that

The Bivariate Normal Distribution5

XandYhave zero means and positive variances. Furthermore, to avoid the degenerate where˜Xis identically zero, we assume that|ρ|<1. We have f ˜X (˜x)=f

˜X|Y

(˜x|y)=1⎷

2π⎷1-ρ

2 σ X e-˜x 2 /2σ

2˜X

, and f X|Y (x|y)=1⎷

2π⎷1-ρ

2 σ X e-? x-ρσ X σ Y y? 2 /2σ

2˜X

, where σ

2˜X

=(1-ρ 2 )σ 2X .

UsingalsotheformulaforthePDFofY,

f Y (y)=1⎷

2πσ

Y e -y 2 /2σ 2Y , and the multiplication rulef X,Y (x,y)=f Y (y)f X|Y (x|y), we can obtain the joint PDF ofXandY. This PDF is of the form f X,Y (x,y)=ce -q(x,y) , where the normalizing constant is c=1

2π⎷1-ρ

2 σ X σ Y . The exponent termq(x,y) is a quadratic function ofxandy, q(x,y)=y 2 2σ 2Y +? x-ρσ X σ Y y? 2

2(1-ρ

2 )σ 2X , which after some straightforward algebra simplifies to q(x,y)=x 2 σ 2X -2ρxy
X σ Y +y 2 σ 2Y

2(1-ρ

2 ). An important observation here is thatthe joint PDF is completely deter- mined byσ X ,σ Y ,andρ. In the special case whereXandYare uncorrelated (ρ= 0), the joint PDF takes the simple form f X,Y (x,y)=1

2πσ

X σ Y e-x 2 2σ 2X -y 2 2σ 2Y ,

6The Bivariate Normal Distribution

which is just the product of two independent normal PDFs. We can get some insight into the form of this PDF by considering its contours, i.e., sets of points at which the PDF takes a constant value. These contours are described by an equation of the formx 2 σ 2X +y 2 σ 2Y = constant, and are ellipses whose two axes are horizontal and vertical. In the more general case whereXandYare dependent, a typical contour is described byx 2 σ 2X -2ρxy
X σ Y +y 2 σ 2Y = constant, and is again an ellipse, but its axes are no longer horizontal and vertical. Figure

4.11 illustrates the contours for two cases, one in whichρis positive and one in

whichρis negative. xyxy Figure 4.11:Contours of the bivariate normal PDF. The diagram on the left (respectively, right) corresponds to a case of positive (respectively, negative) cor- relation coefficientρ. Example 4.28.Suppose thatXandZare zero-mean jointly normal random variables, such thatσ 2X =4,σ 2Z =17/9, andE[XZ] = 2. We define a new random variableY=2X-3Z. We wish to determine the PDF ofY, the conditional PDF ofXgivenY,andthejointPDFofXandY. As noted earlier, a linear function of two jointly normal random variables is also normal. Thus,Yis normal with variance σ 2Y =E ? (2X-3Z) 2 ? =4E[X 2 ]+9E[Z 2 ]-12E[XZ]=4·4+9·17

9-12·2=9.

Hence,Yhas the normal PDF

f

Y(y)=1⎷

2π·3e

-y2/18 .

The Bivariate Normal Distribution7

We next note thatXandYare jointly normal. The reason is thatXandZare linear functions of two independent normal random variables (by the definition of joint normality), so thatXandYare also linear functions of the same independent normal random variables. The covariance ofXandYis equal to

E[XY]=E

?

X(2X-3Z)

? =2E[X 2 ]-3E[XZ] =2·4-3·2 =2. Hence, the correlation coefficient ofXandY, denoted byρ,isequalto

ρ=E[XY]

σXσY

=2

2·3=13.

The conditional expectation ofXgivenYis

E[X|Y]=ρσ

X σY Y=1

3·23Y=29Y.

The conditional variance ofXgivenY(which is the same as the variance of˜X=

X-E[X|Y]) is

σ

2˜X

=(1-ρ 2 )σ 2X = ? 1-1 9 ? 4=32 9, so thatσ ˜X=⎷32/3. Hence, the conditional PDF ofXgivenYis f X|Y (x|y)=3⎷

2π⎷32e-

? x-(2y/9) ? 2

2·32/9.

Finally, the joint PDF ofXandYis obtained using either the multiplication rulef

X,Y(x,y)=fX(x)f

X|Y (x|y), or by using the earlier developed formula for the exponentq(x,y), and is equal to f

X,Y(x,y)=1

2π⎷32e-y

2 9+x 2

4-23·xy2·3

2(1-(1/9)).

We end with a cautionary note. IfXandYare jointly normal, then each random variableXandYis normal. However, the converse is not true. Namely, if each of the random variablesXandYis normal, it does not follow that they are jointly normal, even if they are uncorrelated. This is illustrated in the following example. Example 4.29.LetXhave a normal distribution with zero mean and unit variance. LetZbe independent ofX,withP(Z=1)=P(Z=-1) = 1/2. Let

8The Bivariate Normal Distribution

Y=ZX, which is also normal with zero mean. The reason is that conditioned on either value ofZ,Yhas the same normal distribution, hence its unconditional distribution is also normal. Furthermore,

E[XY]=E[ZX

2 ]=E[Z]E[X 2 ]=0·1=0, soXandYare uncorrelated. On the other handXandYare clearly dependent. (For example, ifX=1,thenYmust be either-1or1.)IfXandYwere jointly normal, we would have a contradiction to our earlier conclusion that zero correlation implies independence. It follows thatXandYarenotjointly normal, even though both marginal distributions are normal.

The Multivariate Normal PDF

The development in this section generalizes to the case of more than two random variables. For example, we can say that the random variablesX 1 ,...,X n are jointly normal if all of them are linear functions of a setU 1 ,...,U n of independent normal random variables. We can then establish the natural extensions of the results derived in this section. For example, it is still true that zero correlation implies independence, that the conditional expectation of one random variable given some of the others is a linear function of the conditioning random variables, and that the conditional PDF ofX 1 ,...,X k givenX k+1 ,...,X n is multivariate normal. Finally, there is a closed-form expression for the joint PDF. Assuming that none of the random variables is a deterministic function of the others, we have f X 1 ,...,Xn =ce -q(x 1 ,...,xn) , wherecis a normalizing constant and whereq(x 1 ,...,x n ) is a quadratic function ofx 1 ,...,x n that increases to infinity as the magnitude of the vector (x 1 ,...,x n ) tends to infinity. Multivariate normal models are very common in statistics, econometrics, signal processing, feedback control, and many other fields. However, a full de- velopment falls outside the scope of this text. Solved Problems on The Bivariate Normal Distribution Problem 1.LetX1andX2be independent standard normal random variables.

Define the random variablesY

1andY2by

Y

1=2X1+X2,Y2=X1-X2.

FindE[Y

1],E[Y2], cov(Y1,Y2), and the joint PDFfY1,Y2.

Solution.The means are given by

E[Y

1]=E[2X1+X2]=E[2X1]+E[X2]=0,

E[Y

2]=E[X1-X2]=E[X1]-E[X2]=0.

The Bivariate Normal Distribution9

The covariance is obtained as follows:

cov(Y

1,Y2)=E[Y1Y1]-E[Y1]E[Y2]

=E ? (2X1+X2)·(X1-X2) ? =E ? 2X 2 1 -X1X2-X 2 2 ? =1. The bivariate normal is determined by the means, the variances, and the correlation coefficient, so we need to calculate the variances. We have σ 2Y1 =E[Y 21
]-μ 2Y1 =E[4X 21
+4X1X2+X 22
]=5.

Similarly,

σ 2Y2 =E[Y 22
]-μ 2Y2 =5. Thus

ρ(Y

1,Y2)=cov(Y

1,Y2)

σY1σY2

=1 5.

To write the joint PDF ofY

1andY2, we substitute the above values into the formula

for the bivariate normal density function. Problem 2.The random variablesXandYare described by a joint PDF of the form f

X,Y(x,y)=ce

-8x2-6xy-18y2 . Find the means, variances, and the correlation coefficient ofXandY. Also, find the value of the constantc. Solution.We recognize this as a bivariate normal PDF, with zero means. By comparing 8x 2 +6xy+18y 2 with the exponent q(x,y)=x 2 σ 2X -2ρxy
XσY +y 2 σ 2Y

2(1-ρ

2 ) of the bivariate normal, we obtain σ 2X (1-ρ 2 )=1/4,σ 2Y (1-ρ 2 )=1/9,(1-ρ 2 )σXσY=-ρ/3.

From the first two equations, we have

(1-ρ 2 )σXσY=1/6, which implies thatρ=-1/2. Thus,σ 2X =1/3, andσ 2Y =4/27. Finally, c=1 2π ?

1-ρ

2

σXσY

=⎷ 27
π.

10The Bivariate Normal Distribution

Problem 3.Suppose thatXandYare independent normal random variables with the same variance. Show thatX-YandX+Yare independent. Solution.It suffices to show that the zero-mean jointly normal random variables X-Y-E[X-Y]andX+Y-E[X+Y] are independent. We can therefore, without loss of generality, assume thatXandYhave zero mean. To prove independence, under the zero-mean assumption, it suffices to show that the covariance ofX-YandX+Y is zero. Indeed, cov(X-Y,X+Y)=E ? (X-Y)(X+Y) ? =E[X 2 ]-E[Y 2 ]=0, sinceXandYwere assumed to have the same variance. Problem 4.The coordinatesXandYof a point are independent zero-mean normal random variables with common varianceσ 2 . Given that the point is at a distance of at leastcfrom the origin, find the conditional joint PDF ofXandY.

Solution.LetCdenote the event thatX

2 +Y 2 >c 2 . The probabilityP(C)canbe calculated using polar coordinates, as follows:

P(C)=1

2πσ

2 ? 2π 0 ? ∞ c re -r2/2σ2 drdθ = 1 σ 2 ? ∞ c re -r2/2σ2 dr =e -c2/2σ2 .

Thus, for (x,y)?C,

f X,Y|C (x,y)=f

X,Y(x,y)

P(C)=12πσ

2 e - 1 2σ 2 (x 2 +y 2 -c 2 ).

Problem 5.

*Suppose thatXandYare jointly normal random variables. Show that

E[X|Y]=E[X]+ρσ

X σY ?

Y-E[Y]

? . Hint:Consider the random variablesX-E[X]andY-E[Y] and use the result established in the text for the zero-mean case. Solution.Let˜X=X-E[X]and˜Y=Y-E[Y]. The random variables˜Xand˜Y are jointly normal. This is because ifXandYare linear functions of two independent normal random variablesUandV,then˜Xand˜Yare also linear functions ofUand

V. Therefore, as established in the text,

E[˜X|˜Y]=ρ(˜X,˜Y)σ

˜X

σ˜Y

˜Y.

Note that conditioning on

˜Yis the same as conditioning onY. Therefore,

E[˜X|˜Y]=E[˜X|Y]=E[X|Y]-E[X].

The Bivariate Normal Distribution11

SinceXand˜Xonly differ by a constant, we haveσ

˜X=σXand, similarly,σ˜Y=σY.

Finally,

cov(

˜X,˜Y)=E[˜X˜Y]=E

??

X-E[X]

??

Y-E[Y]

?? =cov(X,Y), from which it follows thatρ(˜X,˜Y)=ρ(X,Y). The desired formula follows by substi- tuting the above relations in the formula forE[˜X|˜Y].

Problem 6.

* (a) LetX1,X2,...,Xnbe independent identically distributed random variables and letY=X

1+X2+···+Xn. Show that

E[X

1|Y]=Y

n. (b) LetXandWbe independent zero-mean normal random variables, with posi- tive integer varianceskandm, respectively. Use the result of part (a) to find E[X|X+W], and verify that this agrees with the conditional expectation for- mula for jointly normal random variables given in the text.Hint:Think ofX andWas sums of independent random variables.

Solution.(a) By symmetry, we see thatE[X

i|Y]isthesameforalli.Furthermore, E[X

1+···+Xn|Y]=E[Y|Y]=Y.

Therefore,E[X

1|Y]=Y/n.

(b) We can think ofXandWas sums of independent standard normal random vari- ables: X=X

1+···+X

k ,W=W1+···+Wm. We identifyYwithX+Wand use the result from part (a), to obtain E[X i|X+W]=X+W k+m. Thus,

E[X|X+W]=E[X

1+···+X

k |X+W]=k k+m(X+W). This formula agrees with the formula derived in the text because

ρ(X,X+W)σ

X

σX+W

=cov(X,X+W) σ 2X+W =k k+m.

We have used here the property

cov(X,X+W)=E ?

X(X+W)

? =E[X 2 ]=k.