“Bayesian information criterion”的意思、由来-开放百科全书

In statistics, the Bayesian information criterion (BIC) or Schwarz information criterion (also SIC, SBC, SBIC) is a criterion for model selection among a finite set of models; the model with the lowest BIC is preferred. It is based, in part, on the likelihood function and it is closely related to the Akaike information criterion (AIC).

When fitting models, it is possible to increase the likelihood by adding parameters, but doing so may result in overfitting. Both BIC and AIC attempt to resolve this problem by introducing a penalty term for the number of parameters in the model; the penalty term is larger in BIC than in AIC.

The BIC was developed by Gideon E. Schwarz and published in a 1978 paper,^[1] where he gave a Bayesian argument for adopting it.

Definition

Konishi and Kitigawa (2008, p. 217) derive the BIC to approximate the distribution of the data, integrating out the parameters using Laplace's method, starting with the following:

The log(likelihood),

, is then expanded to a second order Taylor series about the MLE,

, assuming it is twice differentiable as follows:

where

is the average observed information per observation, and prime (

) denotes transpose of the vector

. To the extent that

is negligible and

is relatively linear near

, we can integrate out

to get the following:

where BIC is defined as above, and

either (a) is the Bayesian posterior mode or (b) uses the MLE and the prior

has nonzero slope at the MLE. Then the posterior

Properties

Limitations

Gaussian special case

Under the assumption that the model errors or disturbances are independent and identically distributed according to a normal distribution and that the boundary condition that the derivative of the log likelihood with respect to the true variance is zero, this becomes (up to an additive constant, which depends only on n and not on the model):^[5]

When testing multiple linear models against a saturated model, the BIC can be rewritten in terms of the

When picking from several models, the one with the lowest BIC is preferred. The BIC is an increasing function of the error variance

The BIC generally penalizes free parameters more strongly than the Akaike information criterion, though it depends on the size of n and relative magnitude of n and k.

It is important to keep in mind that the BIC can be used to compare estimated models only when the numerical values of the dependent variable are identical for all estimates being compared. The models being compared need not be nested, unlike the case when models are being compared using an F-test or a likelihood ratio test.{{Citation needed|date=February 2019}}

BIC for high-dimensional model

For high dimensional model with the number of potential variables

, and the true model size is bounded by a constant, modified BICs has been proposed in Chen and Chen (2008) and Gao and Song (2010). For high dimensional model with the number of variables

, and the true model size is unbounded, a high dimensional BIC has been proposed in Gao and Carroll (2017). The high dimensional BIC is of the form:

Gao and Carroll (2017) proposed a pseudo-likelihood BIC for which the pseudo log-likelihood is used instead of the true log-likelihood. The high dimensional pseudo-likelihood BIC is of the form:

where

is an estimated degrees of freedom, and the constant

is an unknown constant.

To achieve the theoretical model selection consistency for divergent

, the two high dimensional BICs above require the multiplicative factor

. However, in practical use, the high dimensional BIC can take a simpler form:

where various choices of the multiplicative factor

can be used. In empirical studies,

can be used and it is shown to have good empirical performance.

See also

Notes

1. ^{{citation | last=Schwarz |first=Gideon E. |title=Estimating the dimension of a model |journal= Annals of Statistics |year=1978 |volume=6 |issue=2 |pages=461–464 |doi=10.1214/aos/1176344136 |mr=468014 }}.
2. ^{{Cite journal| doi = 10.1111/j.1467-9574.2012.00530.x| volume = 66 | issue = 3 | pages = 217–236| last = Wit | first = Ernst |author2=Edwin van den Heuvel |author3=Jan-Willem Romeyn| title = 'All models are wrong...': an introduction to model uncertainty| journal = Statistica Neerlandica| year = 2012}}
3. ^NOTE: The AIC, AICc and BIC defined by Claeskens and Hjort (2008) is the negative of that defined in this article and in most other standard references.
4. ^¹{{cite book|last=Giraud|first=C.|year=2015|title=Introduction to high-dimensional statistics|publisher=Chapman & Hall/CRC|isbn=9781482237948}}
5. ^{{cite book|last=Priestley|first=M.B.|year=1981|title=Spectral Analysis and Time Series|publisher=Academic Press|isbn=978-0-12-564922-3}} (p. 375).
6. ^¹{{Citation| doi = 10.2307/2291091| issn = 0162-1459| volume = 90| issue = 430| pages = 773–795| last1 = Kass| first1 = Robert E.| last2= Raftery| first2= Adrian E.| title = Bayes Factors| journal = Journal of the American Statistical Association| year = 1995| jstor = 2291091}}.