The Hong Kong University of Science and Technology · FACULTY OF ELECTRICAL ENGINEERING

ELEC1010 Chap.8 Information, Entropy and Capacity

- one subject, every graph, every model, every mark
6 Chapters4-page Bible
Our own words - no uploaded lecturer files
Updated for this semester
Chapter 8 of 10 · ELEC1010

Information, Entropy and Capacity

Measure uncertainty and transmission limits

Self-information measures surprise: an unlikely outcome supplies more information when it occurs than a likely outcome. Entropy balances surprise with occurrence probability by weighting each outcome's logarithmic information and summing. With base-two logarithms the unit is bits.

Confirm that probabilities lie between zero and one and sum to one before interpreting the result as a source average.

A certain outcome has zero entropy because observing it resolves no uncertainty. Two equally likely binary outcomes produce the maximum binary entropy of one bit per symbol. Bias makes prediction easier and reduces the average.

The two labels still exist, but their probabilities allow a long code to spend fewer bits on common patterns and more on rare ones.

Entropy is a lower bound on long-run average representation, not a claim that one isolated symbol occupies a fractional physical bit. Grouping many outcomes and using variable-length codewords allows an average to approach the bound.

Keep semantic importance separate: a predictable alarm can be crucial yet have low source entropy, while random noise can carry high uncertainty without useful meaning.

Channel capacity addresses the other side of communication. The course uses C equals B log2 of one plus SNR, with bandwidth B in hertz and signal-to-noise ratio in linear form. The result is bits per second.

Bandwidth scales the ceiling directly, whereas SNR lies inside a logarithm and therefore yields diminishing proportional gains as it increases.

Capacity is a theoretical upper bound under the model, not a guaranteed equipment data rate. Practical coding overhead, implementation loss and margin keep an operating rate lower. Do not substitute a decibel number directly where a linear ratio is required.

Convert units first, evaluate the logarithm, multiply by bandwidth, and label the result as a ceiling.

Entropy in bits per symbol cannot be compared directly with capacity in bits per second. Multiply the coded source requirement by symbols per second to form a source bit rate. A feasible design keeps that rate below channel capacity.

Revising the complete chain—probabilities to entropy, code to average length, symbol rate to source rate, and channel resources to capacity—prevents the two Shannon bounds from being reversed.

In this chapter

What this chapter covers

  • 01

    Outcome probability

  • 02

    Self-information

  • 03

    Source entropy

  • 04

    Binary entropy bounds

  • 05

    Bandwidth and noise

  • 06

    Shannon capacity

Worked example · free

Compute binary entropy

Q [9 marks]. A source reports Safe with probability 0.75 and Alert with probability 0.25. Find entropy. The marks shown are an AskSia study allocation, not the University's published marking scheme.
Binary entropy follows the uncertainty, not the number of labels.Binary entropy follows the uncertainty, not the number of labels. The curve peaks at one bit when p equals one half and falls toward zero as either outcome becomes certain.Binary entropy is highest when the outcomes are equally likelyp = 0.5H = 1 bit
Binary entropy follows the uncertainty, not the number of labels. The curve peaks at one bit when p equals one half and falls toward zero as either outcome becomes certain.
  • 4Write one weighted logarithm per outcome.
  • 3Add the two contributions.
  • 2Check the zero-to-one bound.
Entropy is 0.75 log2(1/0.75) + 0.25 log2(1/0.25), approximately 0.311 + 0.500 = 0.811 bit per report.
Sia tip — For a binary source, reject any entropy below zero or above one before trusting the calculator.
Glossary

Key terms

Self-information
The information associated with observing one outcome of stated probability.
Entropy
The probability-weighted average information of a source.
Source Rate
The coded information produced by a source per second.
Channel Capacity
The theoretical maximum reliable information rate under the channel model.
Signal-to-noise Ratio
The ratio comparing signal power with noise power.
FAQ

Information, Entropy and Capacity FAQ

Why does certainty have zero entropy?

A certain outcome supplies no new answer when observed because it was already known. Its probability is one, so the logarithmic information contribution is zero.

Can one symbol use less than one physical bit?

An isolated binary word cannot have fractional length, but a long sequence can approach a fractional average by grouping outcomes and assigning short words to common patterns.

How does capacity differ from entropy?

Entropy is a source-side lower bound measured in bits per symbol. Capacity is a channel-side upper bound measured in bits per second. A symbol rate is needed before comparing them.

Why is a decibel SNR not inserted directly?

The Shannon expression uses a linear power ratio inside the logarithm. A decibel value is a logarithmic representation and must be converted before evaluating one plus SNR.

Study strategy

Exam move

Compute several binary entropies and check symmetry around probability one half. For capacity, convert bandwidth to hertz and use linear SNR before the logarithm. Label every result as a lower or upper bound. Finally convert entropy to a source bit rate so it can be compared with channel capacity in matching units. Audit the direction and unit of every bound.

Entropy is a source-side lower bound in bits per symbol, while capacity is a channel-side upper bound in bits per second. Check binary entropy against the zero-to-one range and verify that increasing bandwidth or linear SNR does not reduce capacity. Only after multiplying by symbol rate should source demand be compared with the channel ceiling.

Efficient coding may approach but cannot beat entropy, and reliable operation cannot be justified above capacity by merely relabelling symbols. These directional checks catch correct arithmetic attached to the wrong engineering conclusion.

Working through Information, Entropy and Capacity in ELEC1010? Sia is AskSia’s AI Electrical Engineering tutor — ask any ELEC1010 Information, Entropy and Capacity question and get a clear, step-by-step explanation grounded in how ELEC1010 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.

A+Everything unlocked
Unlocks this Bible + all 13 of your subjects at The Hong Kong University of Science and Technology, plus 1,000+ more Bibles across the AskSia library.
Sia - your ELEC1010 tutor, unlimited, worked the way the exam marks it
The full 4-page Bible + practice bank with worked solutions
Chrome extension - sync your LMS so Sia knows your deadlines
Bilingual EN / Chinese on every Bible and every Sia answer
$0.99 Trial
30-day money-back · cancel in one tap · how it works