ECE5179 Chap.8 Generative Adversarial Networks
Generative Adversarial Networks
Week 8 moves from prediction to generation. A generative adversarial network trains a generator to turn random noise into realistic samples and a discriminator to tell real from generated, in alternating steps.
The chapter writes both losses, explains why the non-saturating generator loss is preferred, derives the optimal discriminator and its link to the Jensen-Shannon divergence, and covers mode collapse, convolutional and conditional GANs, and how to judge training progress.
What this chapter covers
- 01
Generator, discriminator and the two-player game
- 02
Alternating training and the equilibrium D = ½
- 03
The discriminator loss as binary cross-entropy
- 04
Minimax versus non-saturating generator losses
- 05
The optimal discriminator for a fixed generator
- 06
Mode collapse, DCGAN and conditional GANs
Worked example · free
Read the optimal discriminator off two densities
- 1For a fixed generator, D*(x) = P_real(x) / (P_real(x) + P_G(x)).
- 1At x = 1 the two Gaussians have equal density, so D*(1) = 0.5.
- 1At x = 0 the real density is about 0.399 and the generator's about 0.054, so D*(0) ≈ 0.881.
- 1If P_G equalled P_real, D* would be ½ everywhere and the discriminator loss would reach 2 log 2 ≈ 1.386.
Key terms
- Generator
- The generator is the GAN network that maps random noise vectors to synthetic samples.
- Discriminator
- The discriminator is the GAN network that outputs the probability that its input is a real sample.
- Non-saturating loss
- The non-saturating loss trains the generator to maximise log D(G(z)), which keeps gradients large early in training.
- Jensen-Shannon divergence
- The Jensen-Shannon divergence is a symmetric measure of difference between two distributions, bounded between 0 and log 2.
Generative Adversarial Networks FAQ
Why is the non-saturating generator loss preferred?
Early in training the discriminator rejects fakes confidently, so D(G(z)) is near zero and the minimax loss log(1 − D(G(z))) is almost flat. Measured against the discriminator's logit, its gradient is −D, which is tiny, while the non-saturating loss −log D(G(z)) gives −(1 − D), close to −1 in the same direction.
What is mode collapse?
Mode collapse happens when the generator produces convincing samples that cover only part of the real distribution, for example only one kind of image from a varied data set. The discriminator can still separate the missing kinds, so the pair is not at equilibrium, but training may stay stuck there.
How can I tell whether GAN training is going well?
Not from the loss curve, which oscillates and tells you remarkably little according to the Studio 8 worksheet. Generate images from a fixed batch of noise vectors at regular intervals, with the generator in evaluation mode, and judge their quality and variety over time.
What does the Jensen-Shannon divergence have to do with GANs?
If the discriminator is optimal for the current generator, its loss equals 2 log 2 minus twice the Jensen-Shannon divergence between the real and generated distributions. Training the generator therefore reduces that divergence. When the two distributions do not overlap, the divergence is stuck at log 2, which starves the generator of gradient.
What are DCGANs and conditional GANs?
A deep convolutional GAN builds the generator from transposed convolutions that grow a small feature map into an image and makes the discriminator a convolutional classifier. A conditional GAN feeds a class label to both networks, so a single trained generator can be asked to produce a chosen class on demand.
Why train the generator and discriminator in alternating steps?
Each network's loss depends on the other, so they cannot be minimised jointly as one objective. Holding one fixed while updating the other keeps each step well defined: the discriminator learns to separate the current fakes, then the generator learns to fool that discriminator, and the cycle repeats.
Exam move
Write both losses from memory, labelling which term uses real data and which uses generated data, then evaluate them for a few pairs of discriminator outputs. Derive the optimal discriminator once from the concave function a log u + b log(1 − u), and practise using it on mixtures, such as a generator that produces only one kind of image.
Explain in two sentences why the non-saturating loss helps and why the loss curve is a poor progress measure. Add one equilibrium question of your own to the practice set: choose a real mixture of two kinds of image and a generator that favours one kind, compute D* for each kind, and state whether the pair is at equilibrium and which way the generator will be pushed.
Working through Generative Adversarial Networks in ECE5179? Sia is AskSia’s AI Computer Science tutor — ask any ECE5179 Generative Adversarial Networks question and get a clear, step-by-step explanation grounded in how ECE5179 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
