
related topics 
{math, number, function} 
{rate, high, increase} 
{math, energy, light} 
{language, word, form} 
{style, bgcolor, rowspan} 
{system, computer, user} 
{work, book, publish} 
{theory, work, human} 

In information theory, entropy is a measure of the uncertainty associated with a random variable. In this context, the term usually refers to the Shannon entropy, which quantifies the expected value of the information contained in a message, usually in units such as bits. Equivalently, the Shannon entropy is a measure of the average information content one is missing when one does not know the value of the random variable. The concept was introduced by Claude E. Shannon in his 1948 paper "A Mathematical Theory of Communication".
Shannon's entropy represents an absolute limit on the best possible lossless compression of any communication, under certain constraints: treating messages to be encoded as a sequence of independent and identicallydistributed random variables, Shannon's source coding theorem shows that, in the limit, the average length of the shortest possible representation to encode the messages in a given alphabet is their entropy divided by the logarithm of the number of symbols in the target alphabet.
A fair coin has an entropy of one bit. However, if the coin is not fair, then the uncertainty is lower (if asked to bet on the next outcome, we would bet preferentially on the most frequent result), and thus the Shannon entropy is lower. Mathematically, a coin flip is an example of a Bernoulli trial, and its entropy is given by the binary entropy function. A long string of repeating characters has an entropy rate of 0, since every character is predictable. The entropy rate of English text is between 1.0 and 1.5 bits per letter,^{[1]} or as low as 0.6 to 1.3 bits per letter, according to estimates by Shannon based on human experiments.^{[2]}
Contents
Full article ▸


related documents 
Numerical analysis 
Kernel (algebra) 
Huffman coding 
Cardinal number 
Gaussian elimination 
Sequence alignment 
Denotational semantics 
Complete lattice 
Infinity 
Lua (programming language) 
Hash function 
Stone–Čech compactification 
Interval (mathematics) 
Absolute value 
Simplex 
Logic programming 
RSA 
Series (mathematics) 
Ruby (programming language) 
Newton's method 
Compass and straightedge constructions 
Integration by parts 
Power law 
Functor 
Direct sum of modules 
Proofs of Fermat's little theorem 
Riemann zeta function 
Boolean satisfiability problem 
List of trigonometric identities 
Central limit theorem 
