[PDF] 18 The Exponential Family and Statistical Applications

View - Download 18 The Exponential Family and Statistical Applications

Previous PDF

Next PDF

18 The Exponential Family and Statistical Applications

The Exponential family is a practically convenient and widely used uni¯ed family of distributions on ¯nite dimensional Euclidean spaces parametrized by a ¯nite dimensional parameter vector.

Specialized to the case of the real line, the Exponential family contains as special cases most of the

standard discrete and continuous distributions that we use for practical modelling, such as the nor- mal, Poisson, Binomial, exponential, Gamma, multivariate normal, etc. The reason for the special status of the Exponential family is that a number of important and useful calculations in statistics can be done all at one stroke within the framework of the Exponential family. This generality contributes to both convenience and larger scale understanding. The Exponential family is the usual testing ground for the large spectrum of results in parametric statistical theory that require nential family have an element of mathematical neatness. Distributions in the Exponential family have been used in classical statistics for decades. However, it has recently obtained additional im- portance due to its use and appeal to the machine learning community. A fundamental treatment of the general Exponential family is provided in this chapter. Classic expositions are available in Barndor®-Nielsen (1978), Brown (1986), and Lehmann and Casella (1998). An excellent recent treatment is available in Bickel and Doksum (2006).

18.1 One Parameter Exponential Family

Exponential families can have any ¯nite number of parameters. For instance, as we will see, a normal distribution with a known mean is in the one parameter Exponential family, while a normal distribution with both parameters unknown is in the two parameter Exponential family. A bivariate normal distribution with all parameters unknown is in the ¯ve parameter Exponential family. As another example, if we take a normal distribution in which the mean and the variance are functionally related, e.g., theN(¹;¹ 2 ) distribution, then the distribution will be neither in the one parameter nor in the two parameter Exponential family, but in a family called acurved Exponential family. We start with the one parameter regular Exponential family.

18.1.1 De¯nition and First Examples

We start with an illustrative example that brings out some of the most important properties of distributions in an Exponential family. Example 18.1. (Normal Distribution with a Known Mean).SupposeX»N(0;¾ 2 ). Then the density ofXis f(xj¾)=1

¾p2¼e

x2

2¾2

I x2R This density is parametrized by a single parameter¾. Writing

´(¾)=¡1

2¾ 2 ;T(x)=x 2 ;Ã(¾) = log¾;h(x)=1p

2¼I

x2R we can represent the density in the form f(xj¾)=e

´(¾)T(x)¡Ã(¾)

h(x); 498
for any¾2R

Next, suppose that we have an iid sampleX

1 ;X 2 ;¢¢¢;X n

»N(0;¾

2 ). Then the joint density of X 1 ;X 2 ;¢¢¢;X n is f(x 1 ;x 2 ;¢¢¢;x n j¾)=1 n (2¼) n=2 e

Pni=1x2i

2¾2

I x1;x2;¢¢¢;xn2R

Now writing

´(¾)=¡1

2¾ 2 ;T(x 1 ;x 2 ;¢¢¢;x n n X i=1 x 2i ;Ã(¾)=nlog¾; and h(x 1 ;x 2 ;¢¢¢;x n )=1 (2¼) n=2 I x1;x2;¢¢¢;xn2R once again we can represent the joint density in the same general form f(x 1 ;x 2 ;¢¢¢;x n j¾)=e h(x 1 ;x 2 ;¢¢¢;x n We notice that in this representation of the joint densityf(x 1 ;x 2 ;¢¢¢;x n j¾), the statisticT(X 1 ;X 2 ;¢¢¢;X n is still a one dimensional statistic, namely,T(X 1 ;X 2 ;¢¢¢;X n )=P n i=1 X 2i . Using the fact that the sum of squares ofnindependent standard normal variables is a chi square variable withndegrees of freedom, we have that the density ofT(X 1 ;X 2 ;¢¢¢;X n )is f T (tj¾)=e t

2¾2

t n 2 ¡1 n 2 n=2 n 2 )I t>0

This time, writing

´(¾)=¡1

2¾ 2 ;S(t)=t;Ã(¾)=nlog¾;h(t)=1 2 n=2 n 2 )I t>0 once again we are able to write even the density ofT(X 1 ;X 2 ;¢¢¢;X n )=P n i=1 X 2i in that same general form f T (tj¾)=e

´(¾)S(t)¡Ã(¾)

h(t):

Clearly, something very interesting is going on. We started with a basic density in a speci¯c form,

namely,f(xj¾)=e

´(¾)T(x)¡Ã(¾)

h(x), and then we found that the joint density and the density of the relevant one dimensional statisticP n i=1 X 2i in that joint density, are once again densities of exactly that same general form. It turns out that all of these phenomena are true of the entire family of densities which can be written in that general form, which is the one parameter

Exponential family. Let us formally de¯ne it and we will then extend the de¯nition to distributions

with more than one parameter.

De¯nition 18.1.LetX=(X

1 ;¢¢¢;X d )bead-dimensional random vector with a distribution P ;µ2£µR.

SupposeX

1 ;¢¢¢;X d are jointly continuous. The family of distributionsfP ;µ2£gis said to belong to theone parameter Exponential familyif the density ofX=(X 1 ;¢¢¢;X d ) may be represented in the form f(xjµ)=e

´(µ)T(x)¡Ã(µ)

h(x); 499
for some real valued functionsT(x);Ã(µ)andh(x)¸0. IfX 1 ;¢¢¢;X d are jointly discrete, thenfP ;µ2£gis said to belong to the one parameter Ex- ponential family if the joint pmfp(xjµ)=P (X 1 =x 1 ;¢¢¢;X d =x d ) may be written in the form p(xjµ)=e

´(µ)T(x)¡Ã(µ)

h(x); for some real valued functionsT(x);Ã(µ)andh(x)¸0. Note that the functions´;Tandhare not unique. For example, in the product´T, we can multiply Tby some constantcand divide´by it. Similarly, we can play with constants in the functionh.

De¯nition 18.2.SupposeX=(X

1 ;¢¢¢;X d ) has a distributionP ;µ2£, belonging to the one parameter Exponential family. Then the statisticT(X) is calledthe natural su±cient statisticfor the familyfP g.

The notion of a su±cient statistic is a fundamental one in statistical theory and its applications.

Su±ciency was introduced into the statistical literature by Sir Ronald A. Fisher (Fisher (1922)). Su±ciency attempts to formalize the notion ofno loss of information. A su±cient statistic is supposed to contain by itself all of the information about the unknown parameters of the underlying distribution that the entire sample could have provided. In that sense, there is nothing to lose

by restricting attention to just a su±cient statistic in one's inference process. However, the form

of a su±cient statistic is very much dependent on the choice of a particular distributionP for modelling the observableX. Still, reduction to su±ciency in widely used models usually makes just simple common sense. We will come back to the issue of su±ciency once again later in this chapter. We will now see examples of a few more common distributions that belong to the one parameter

Exponential family.

Example 18.2. (Binomial Distribution).LetX»Bin(n;p);withn¸1 considered as known, and 01¡p +nlog(1¡p) I fx2f0;1;¢¢¢;ngg

Writing´(p) = log

p

1¡p

;T(x)=x;Ã(p)=¡nlog(1¡p), andh(x)=¡ n x ¢I fx2f0;1;¢¢¢;ngg ,wehave represented the pmff(xjp) in the one parameter Exponential family form, as long asp2(0;1). Forp= 0 or 1, the distribution becomes a one point distribution. Consequently, the family of distributionsff(xjp);02¼e x2 2 +¹x¡ ¹2 2 I x2R 500
which can be written in the one parameter Exponential family form by witing´(¹)=¹;T(x)= x;Ã(¹)= 2 2 ,andh(x)=e x2 2 I x2R . So, the family of distributionsff(xj¹);¹2Rgforms a one parameter Exponential family. Example 18.4. (Errors in Variables).SupposeU;V;Ware independent normal variables, with

UandVbeingN(¹;1) andWbeingN(0;1). LetX

1 =U+WandX 2 =V+W. In other words, a common error of measurementWcontaminates bothUandV.

LetX=(X

1 ;Xquotesdbs_dbs29.pdfusesText_35

[PDF] Dérivées des fonctions usuelles Opérations sur les dérivées - XyMaths

[PDF] Petit supplément sur les fonctions ? valeurs complexes - Tourbillon

[PDF] fonction d 'une variable complexe - CMAP, Polytechnique

[PDF] Tableau des dérivées élémentaires et règles de - Lycée d 'Adultes

[PDF] DÉRIVÉES USUELLES ET DIFFÉRENTIELLES

[PDF] Fiche : Dérivées et primitives des fonctions usuelles - Institut de

[PDF] derivation des fonctions composees - Maths54

[PDF] Fonctions dérivées - Académie en ligne

[PDF] La fonction exponentielle complexe

[PDF] ROC : dérivée d 'une fonction composée

[PDF] Exercices corrigés sur l 'étude des fonctions composées

[PDF] Tableaux des dérivées Dérivées des fonctions usuelles Notes

[PDF] Intégrales dépendant d 'un paramètre - Math France

[PDF] Dérivées et différentielles des fonctions de plusieurs variables

[PDF] Dérivées partielles - Exo7 - Emathfr