Shannon entropy

Entropy is a quantity that measures “disorder”. In thermodynamics it is introduced as the “exchange rate” between energy and disorder: $\Delta E = T\Delta S$ (Borghi, Entropioni, 2019). Claude Shannon in the 1940s discovered that the same formula describes the average information transmitted by a probabilistic source (Shannon, 1948).

Definition — Shannon entropy

Let $X$ be a discrete random variable with values $x_1,\ldots,x_n$ and probabilities $p_1,\ldots,p_n$ (with $\sum p_i=1$). The entropy of $X$ is $H(X) = -\sum_{i=1}^{n} p_i \log_2 p_i,$ measured in bits. Convention: $0\cdot\log 0 = 0$ (limit of $p\log p$ as $p\to 0^+$). Changing the base of the logarithm changes the unit: base $e$ gives nats, base $10$ gives hartleys.

Links

Topics: Distribuzioni probabilita
Concepts: Entropia · Variabile aleatoria
Skills: Modellizzare
People: Shannon