Operational meaning of entropy

The definition of entropy becomes concrete if you read it as a question: how many “yes / no” questions are needed on average to discover the value of $X$?

Remark — Operational meaning

$H(X)$ is the minimum average number of yes / no questions (or of bits, in an optimal binary encoding) needed to identify the value of $X$. Uniform distribution over $2^k$ values: $H = k$ bits (exactly $k$ questions are needed). Deterministic distribution: $H = 0$ (there is nothing to discover).

This reading explains why entropy is measured in bits: a bit is exactly the answer to a well-posed binary question. The more “predictable” a distribution, the fewer questions are needed, and the lower its entropy.

Links

Topics: Distribuzioni probabilita
Concepts: Entropia
Skills: Interpretare grafico