[PDF] On model selection criteria in multimodel analysis

View - Download On model selection criteria in multimodel analysis

Previous PDF

Next PDF

On model selection criteria in multimodel analysis

Ming Ye,

1

Philip D. Meyer,

2 and Shlomo P. Neuman 3 Received 2 January 2008; accepted 1 February 2008; published 27 March 2008. [1]Hydrologic systems are open and complex, rendering them prone to multiple conceptualizations and mathematical descriptions. There has been a growing tendency to postulate several alternative hydrologic models for a site and use model selection criteria to (1) rank these models, (2) eliminate some of them, and/or (3) weigh and average predictions and statistics generated by multiple models. This has led to some debate among hydrogeologists about the merits and demerits of common model selection (also known as model discrimination or information) criteria such asAIC,AICc,BIC, andKIC and some lack of clarity about the proper interpretation and mathematical representation of each criterion. We examine the model selection literature to find that (1) all published rigorous derivations ofAICandAICcrequire that the (true) model having generated the observational data be in the set of candidate models; (2) thoughBICandKICwere originally derived by assuming that such a model is in the set,BIChas been rederived by Cavanaugh and Neath (1999) without the need for such an assumption; and (3)KIC reduces toBICas the number of observations becomes large relative to the number of adjustable model parameters, implying that it likewise does not require the existence of a true model in the set of alternatives. We explain whyKICis the only criterion accounting validly for the likelihood of prior parameter estimates, elucidate the unique role that the Fisher information matrix plays inKIC, and demonstrate through an example that it imbuesKICwith desirable model selection properties not shared byAIC,AICc,orBIC. Our example appears to provide the first comprehensive test of howAIC,AICc,BIC, and KICweigh and rank alternative models in light of the models' predictive performance under cross validation with real hydrologic data.

Citation:Ye, M., P. D. Meyer, and S. P. Neuman (2008), On model selection criteria in multimodel analysis,Water Resour. Res.,44,

W03428, doi:10.1029/2008WR006803.

1. Introduction

[2] Hydrologic environments are open and complex, rendering them prone to multiple interpretations and math- ematical descriptions regardless of the quantity and quality of available data. This recognition has led to a growing tendency among hydrologists to postulate several alterna- tive hydrologic models for a site and use various criteria to (1) rank these models, (2) eliminate some of them, and/or (3) weigh and average predictions and statistics generated by multiple models (Neuman[2003],Neuman and Wierenga [2003],Ye et al.[2004],Poeter and Anderson[2005],Beven [2006] and references therein pertaining to GLUE, and Refsgaard et al.[2006]). This in turn has brought about a debate among hydrogeologists about the merits and demerits of various model selection (also known as model discrimi- nation or information) criteria such as the information- theoretic criteriaAIC[Akaike, 1974] andAICc[Hurvich and Tsai, 1989] and the Bayesian criteriaBIC[Schwarz,

1978] andKIC[Kashyap, 1982]. These criteria discriminate

between models based on how closely they reproduce hydrologic observations using maximum likelihood esti- mates of model parameters (favoring models that reproduce observed behavior most closely) and how many such param- eters they contain (penalizing models that contain many). KICadditionally considers the likelihood of the parameter estimates in light of their prior values (when such are available) and contains a Fisher information matrix term that as we shall see, imbues it with desirable model selection properties not shared byAIC,AICc,orBIC. Models associ- the criterion being irrelevant. [3] Consider a setMofKalternative models,M k ,k=1,

2,...,K. ThenAIC,AICc,BIC, andKICare defined for

modelM k as AIC k

¼?2lnL

b k z*j

þ2N

k

ð1Þ

AICc k

¼?2lnL

b k z*j

þ2N

k 2N k N k

þ1ðÞ

N z ?N k ?1

ð2Þ

BIC k

¼?2lnL

b k z*j þN k lnN z

ð3Þ

1 School of Computational Science and Department of Geological Sciences, Florida State University, Tallahassee, Florida, USA. 2 Pacific Northwest National Laboratory, Richland, Washington, USA. 3 Department of Hydrology and Water Resources, University of Arizona,

Tucson, Arizona, USA.

Copyright 2008 by the American Geophysical Union.

0043-1397/08/2008WR006803$09.00

W03428

WATER RESOURCES RESEARCH, VOL. 44, W03428, doi:10.1029/2008WR006803, 2008 Click Here for Full

Article

1of12 KIC k

¼?2lnL

b k z*j ?2lnp b k þN k lnN z =2pðÞþln F k jj;

ð4Þ

where b k is the maximum likelihood (ML) estimate of a vectorb k ofN k adjustable parameters (which may include statistical parameters of the calibration data) associated with modelM k ;z* is an observed vector ofN z random (hydrologic) system state variableszin space-time, the randomness of which may be inherent (and thus modeled stochastically) or resulting from an additive random error (typically taken to be associated with measurements), common to allKmodels in the set;?ln[L( b k jz*)] is the minimum of the negative log-likelihood (NLL) function ?ln[L(b k jz*)] occurring, by definition, at b k ;p( b k )is the prior probability ofb k evaluated at b k and F k =F k /N z is the normalized (byN z ) observed (implicitly conditioned on the observationsz* and evaluated at the maximum likelihood parameter estimates b k ) Fisher information matrixF k having elements [Kashyap, 1982] F kij 1 N z F kij 1 N z 2 lnLb k z*j @b ki @b kj b k b k :ð5Þ [4] The first term of each criterion,?2ln[L( b k jz*)] measures goodness of fit between predicted and observed system states,^zandz*, respectively; the smaller this term, the better the fit. The terms containingN k represent measures of model complexity. The criteria thus embody (to various degrees) the principle of parsimony, penalizing models for having a relatively large number of parameters if this does not bring about a corresponding improvement in model fit. [5] Expressions equivalent to (1)-(4) for the case of Gaussian likelihood functions, which correspond to param- eter estimation schemes based on weighted least squares of the kind employed in some hydrologic inverse codes (e.g.,

PEST [Doherty, 2006], UCODE

_

2005 [Poeter et al., 2005],

and MODFLOW2000 [Hill et al., 2000]), are given in

Appendix B.

[6] In the ensuing discussion we drop the subscriptk from all terms other thanM k andN k unless required for clarity. We start by noting thatAIC[Akaike, 1974;Linhart and Zucchini, 1986;Bozdogan, 1987] is based on the Kullback-Leibler (K-L) information, a measure of the discrepancy between a true but unknown representation (model) of reality from which the observationsz* arise, denoted here asf, and an approximate representation of the same reality, denoted here as the modelghaving parameters b. The Kullback-Leibler information can be expressed as [Akaike, 1974]

If;g?jbðÞ=

Z ln fz*ðÞ gz*jbðÞ fz*ðÞdz*:ð6Þ [7]AICis an asymptotically unbiased estimator of

E[I(f;g(?j

b))] = R

I(f;g(?j

b))f(z*)dz* withgevaluated at the maximum likelihood estimate bofb, the expectation being taken with respect tof(z*). AsN z /N k decreasesAIC becomes progressively more biased, a property improved approximation ofE[I(f;g(?j b))] with better small sample performance.Burnham and Anderson[2002] advocate the use ofAICcwhenN z /N k is less than 40. [8]BICwas derived in a Bayesian context bySchwarz [1978] asanasymptoticapproximationtoatransformationof the posterior probability of a candidate model (Cavanaugh and Neath[1999]; other derivations are due toAkaike[1977] andRissanen[1978]).Cavanaugh and Neath[1999] noted that in the case of large samples,BICfavors the model which is a posteriori most probable, i.e., is most plausible in light of the available data. Assuming (as did Schwarz) that the data were generated by a model which belongs to the set of candidate models rendersBICconsistent in the sense that as the sample sizeN z increases relative toN k , the criterion tends to identify this generating (operating), or true model with probability one. [9]KICwas derived in a Bayesian context byKashyap [1982] as an asymptotic approximation to the model likelihood, i.e., the marginal probability densityp(z*jM k R p(z*jb,M k )p(bjM k )dbof the observations conditional onagivenmodelM k inasetofKsuchmodels.LikeBIC,KIC having generated the data is in the set of candidate models. KICis asymptotic in the sense that the approximation improves asp(z*jb,M k ) becomes more peaked about b, which will generally occur as the numberN z of observations increases.KICis closely related to the asymptotic Laplace approximation ofp(z*jM k )[Kass and Vaidyanathan, 1992; Kass and Raftery, 1995]. The model likelihood arises when evaluating alternative models using Bayes factors [Kass and Raftery, 1995] and model probabilities using Bayesian model averaging (BMA).Neuman[2003] proposed using, andYe et al.[2004, 2005] as well asMeyer et al.[2007] have implemented,KICin the context of maximum likelihood BMA (MLBMA). It is well established (and we show in Appendix A) thatKICreduces asymptotically to

BICasN

z becomes large in comparison toN k (i.e., asN z /N k !1). WhenN z is not large,BICsometimes prefers models with too few parameters [Bozdogan and Haughton, 1998], in which caseKICis a more appropriate criterion to use. We note also that whenN z > 8 the penalty termN k lnN z inBIC is larger than 2N k , in which caseBICplaces more emphasis on parsimony than doesAIC(compare equations (1) and (3)). [10] Selecting the prior probability in the second term, ?2lnp( b), ofKICis considered byKass and Wasserman [1996], and estimating it in the model selection context is discussed byKass and Raftery[1995]. In the special case where prior parameter measurementsb* are available,p(b) may be taken to represent the probability density fuction (pdf) of corresponding measurement errors (b*?b), and p( b) the pdf of associated residuals (b*? b), as done absorbing?2lnp( b) into the leading negative log likelihood term,?2ln[L( bjz*)], and included it also in the leading term ofAICandBIC.As?2lnp( b) drops out ofKICin the asymptotic limit of largeN z /N k (see Appendix A), this term should be excluded fromBIC. We are not aware of any theoretical justification for including?2lnp( b)inAIC,AIC c and/orBICas allowed byHill[1998],Hill and Tiedeman 2of12 W03428YE ET AL.: ON MODEL SELECTION CRITERIA IN MULTIMODEL ANALYSISW03428 [2007], andPoeter and Hill[2007]. In the absence of such justification the presence of?2lnp( b)inKICappears to be a unique feature of this criterion. [11] There has been much debate in the model selectionquotesdbs_dbs27.pdfusesText_33

[PDF] BIC Montpellier Méditerranée Métropole - Gestion De Projet

[PDF] bic rasoir mousquetaire lame

[PDF] BIC SPORT : LA PASSION DES SPORTS DE GLISSE

[PDF] bic techno 283 - Palavas-les - Anciens Et Réunions

[PDF] BIC-Planning 2015- 2016 V3 novembre 2015 - Conception

[PDF] BICA 3 mai - Collège Ahuntsic - Anciens Et Réunions

[PDF] Bicarbonate de potassium

[PDF] Bicarbonate de potassium Carbonate de calcium - Anciens Et Réunions

[PDF] BICARBONATE DE SODIUM Substance de base

[PDF] Bicarbonate de sodium, citrate de sodium - Colorants Et Pigments

[PDF] Bicci di Lorenzo, Bapteme Christ - Musée des Beaux

[PDF] BICCS 2K ACCELERATOR ADDITIVE

[PDF] bicentenaire 1813-2013 - France

[PDF] BICENTENAIRES VERDI ET WAGNER - Italie

[PDF] BICHE, OH MA BICHE - Anciens Et Réunions