- Entropy rates for Markov chains
- See also
- References
In the mathematical theory of probability, the entropy rate or source information rate of a stochastic process is, informally, the time density of the average information in a stochastic process. For stochastic processes with a countable index, the entropy rate 
is the limit of the joint entropy of 
members of the process 
divided by , as tends to infinity:
when the limit exists. An alternative, related quantity is:
For strongly stationary stochastic processes, 
. The entropy rate can be thought of as a general property of stochastic sources; this is the asymptotic equipartition property. The entropy rate may be used to estimate the complexity of stochastic processes. It is used in diverse applications ranging from characterizing the complexity of languages, blind source separation, through to optimizing quantizers and data compression algorithms. For example, a maximum entropy rate criterion may be used for feature selection in machine learning .[1]
Entropy rates for Markov chains
Since a stochastic process defined by a Markov chain that is irreducible, aperiodic
and positive recurrent has a stationary distribution, the entropy rate is independent of the initial distribution.
For example, for such a Markov chain 
defined on a countable number of states, given the transition matrix 
, 
is given by:
where 
is the asymptotic distribution of the chain.
A simple consequence of this definition is that an i.i.d. stochastic process has an entropy rate that is the same as the entropy of any individual member of the process.
See also
- Information source (mathematics)
- Markov information source
- Asymptotic equipartition property
- Maximal Entropy Random Walk - chosen to maximize entropy rate
References
- Cover, T. and Thomas, J. (1991) Elements of Information Theory, John Wiley and Sons, Inc., {{ISBN|0-471-06259-6}} [https://archive.today/20121216133431/http://www3.interscience.wiley.com/cgi-bin/bookhome/110438582?CRETRY=1&SRETRY=0]
- Lincoln Lawyer (disambiguation)
- Lincoln Leadership Academy Charter School
- Lincoln-Liberty Building
- Lincoln Library of Essential Information
- Lincoln Lincoln
- Lincoln local elections
- Lincoln L-series
- Lincoln L series
- Lincoln Lunatic Asylum
- Lincoln Maazel
- Lincoln MacMillan
- Lincoln, Macoupin, Illinois
- Lincoln, Maine
- Lincoln M. Alexander
- Lincoln Manor (disambiguation)
- Lincoln Manor (San Francisco)
- Lincoln mark 8
- Lincoln McCarthy
- Lincoln Medical Center
- Lincoln Memorial Circle
- Lincoln Memorial Railsplitters
- Lincoln Memorial Railsplitters baseball
- Lincoln Memorial Stadium
- Lincoln-Mercury
- Lincoln metropolitan area