Entropy of a binomial

Question

Compute $H(X)$ for $X$ binomial $B(3;\,1/2)$ (four values $0,1,2,3$).

Solution

The probabilities of the four values are $\binom{3}{k}/2^3$: $p_0 = \tfrac18,\quad p_1 = \tfrac38,\quad p_2 = \tfrac38,\quad p_3 = \tfrac18.$ Then $H(X) = -2\cdot\tfrac18\log_2\tfrac18 - 2\cdot\tfrac38\log_2\tfrac38.$ Since $\log_2\tfrac18 = -3$ and $\log_2\tfrac38 = \log_2 3 - 3 \approx -1{,}415$: $H(X) = 2\cdot\tfrac18\cdot 3 + 2\cdot\tfrac38\cdot 1{,}415 = 0{,}75 + 1{,}061 \approx \boxed{1{,}811 \text{ bit}}.$ Less than the $2$ bits of a uniform distribution over $4$ values: the binomial concentrates probability on the central values, reducing uncertainty.

Links

Topics: Probability distributions
Concepts: Binomial distribution · Entropy
Skills: Computing
Exercise type: Expression evaluation