MATH3411 Chap.4 Entropy, Information and Channel Capacity
Entropy, Information and Channel Capacity
Information theory measures uncertainty independently of message meaning. Entropy averages surprise, source-coding results bound compression, and joint, conditional and mutual information track how variables share uncertainty. Noisy-channel models add transition probabilities, while channel capacity maximises mutual information over input distributions.
The binary symmetric channel provides the central calculation and interpretation for the current exam scope. Every quantity belongs to a stated probability model and carries a unit determined by the logarithm base. Joint tables are the reliable starting point because their margins and normalised slices reveal exactly which distribution enters each entropy.
Mutual information can be calculated as shared uncertainty or as the reduction produced by an observation, and agreement between those forms is a useful check. For channels, transition probabilities alone do not determine mutual information; they combine with an input distribution to create the joint law. Capacity then optimises over that choice.
Endpoint and symmetry tests distinguish a plausible information rate from a decimal produced by reversed conditioning or a mistaken logarithm.
What this chapter covers
- 01
Self-information and entropy
- 02
Maximum entropy
- 03
Source-coding bounds
- 04
Joint and conditional entropy
- 05
Mutual information
- 06
Binary symmetric channels
- 07
Channel capacity
Worked example · free
Measure Capacity under Symmetric Noise
- +1Calculate binary entropy H2(0.1), approximately 0.469 bits.
- +1Subtract from one to obtain capacity 0.531 bits per use.
- +1Explain why a balanced input is optimal by symmetry.
Key terms
- Self-information
- The negative logarithm of an event probability.
- Conditional entropy
- The uncertainty remaining in one variable after another is known.
- Channel capacity
- The maximum mutual information achievable per channel use.
Entropy, Information and Channel Capacity FAQ
Why is a fair binary source most uncertain?
Its two outcomes are equally plausible, so observing either removes the largest possible average uncertainty for a binary choice. Moving probability toward either endpoint makes one outcome easier to predict and lowers entropy.
What is the difference between entropy and expected code length?
Entropy is a distribution property and lower bound, while expected length also depends on the chosen integer-length code. The gap records coding overhead and can shrink when longer source blocks are encoded together.
Why can a channel that flips 90% still carry information?
A predictable flip can be reversed. Maximum uncertainty occurs at a 50% crossover, where output becomes independent of input. Capacity measures usable information rate, not the raw proportion of unchanged symbols.
Exam move
Build information-theory fluency by making every decimal answer pass a qualitative test. Start with binary entropy at probabilities zero, one quarter, one half, three quarters and one. Sketch the symmetry around one half and label the maximum of one bit. Before calculating a new distribution, predict whether moving probability toward one outcome should raise or lower uncertainty.
Write each logarithm base and include zero-probability terms only through the usual limiting convention. For sources with more than two outcomes, check that the result lies between zero and the logarithm of the alphabet size. Joint questions should begin with the full probability table. Confirm that cells total one, sum rows and columns for marginals, and normalise the correct row or column for a conditional distribution.
Compute the chain rule in both directions on selected examples so that the equality of the totals becomes familiar without suggesting that the two conditional entropies are equal. Derive mutual information once from marginal and joint entropies and again as the reduction in uncertainty. Agreement between the two routes is a strong arithmetic check.
For channel questions, label input, output and transition probabilities separately. Combine the input distribution with the transition matrix to obtain a joint table, derive the output marginal, and then calculate mutual information. In the binary symmetric case, explain why balance maximises output entropy and why the remaining uncertainty is binary entropy of the crossover probability.
Test capacity at zero noise, complete ambiguity and a near-certain flip. These endpoints stop you from confusing information rate with the fraction of unchanged bits. Maintain an error ledger for wrong logarithm base, unnormalised conditionals, reversed conditioning, missing probability weights, negative mutual information and units omitted from a rate.
Redo each faulty computation using exact fractions where possible before entering decimals. A final-week circuit should include one source entropy, one joint table, one conditional identity, one mutual-information comparison and one capacity interpretation. State what is being maximised when using the word capacity, and respect the course boundary by not substituting an excluded coding theorem for the required calculation.
Close every solution with a range, symmetry, independence or endpoint check. Those short interpretations supply evidence that the number belongs to the model rather than merely to a calculator display.
Working through Entropy, Information and Channel Capacity in MATH3411? Sia is AskSia’s AI Mathematics tutor — ask any MATH3411 Entropy, Information and Channel Capacity question and get a clear, step-by-step explanation grounded in how MATH3411 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
