Information Theory and Digital Communications · Discrete channels, mutual information, and capacity
Let X → Y be a binary symmetric channel of crossover p=1/4, and let X be uniform on {0,1}
Problem
Information Theory and Digital Communications practice problemLet be a binary symmetric channel of crossover , and let be uniform on . The information density of a pair isWrite for the random variable obtained by evaluating at the random pair . Mutual information is the mean of this random variable.
1. Compute the two possible values of and the probabilities of
those two values. Identify which value occurs on the event and which occurs on .
2. Compute the mean and simplify it to
. Confirm that the same number equals by expanding .
3. Compute the variance of . Use the two-point formula
and simplify the gap .
4. Audit both sentences: (i) “information density is just mutual
information, so is the constant on every pair”; (ii) “the variance is the Bernoulli variance with no logarithmic gap, or else the mean is .”
Do not replace the BSC by a BEC, and do not compute a cascade crossover in place of the density.
Hint
HintUniform input forces . The channel likelihoods are and , so the two densities are and .
Check your work
Work the problem yourself first. Then open it in Training to check your answer and read the full worked solution.
The answer check and full solution for this problem come with ProofAnvil Practice membership ($19 USD monthly). See membership. Or start with the free Information Theory and Digital Communications sample problem: Try the free sample problem.
More Information Theory and Digital Communications practice problems
- Let X → Z → Y be the cascade of two independent binary symmetric channels: X → Z has…Discrete channels, mutual information, and capacity