Entropy of a coin

The simplest case — a possibly biased coin — shows how entropy is a function of the probability $p$, with a well-defined maximum.

Example — Tossing a coin

$X\in\{T,C\}$ with $P(T)=p$, $P(C)=1-p$. Then $H(X) = -p\log_2 p - (1-p)\log_2(1-p).$ Function study: maximum at $p=1/2$ with $H=1$ bit (total uncertainty); minimum at $p=0$ or $p=1$ with $H=0$ (no surprise).

Entropy of the coin $H(p)=-p\log_2 p-(1-p)\log_2(1-p)$: maximum $H_{\max}=1$ bit at $p=1/2$, zero at the extremes $p=0$ and $p=1$.

Links

Topics: Distribuzioni probabilita
Concepts: Entropia
Skills: Interpretare grafico · Studiare funzione