ECE5179 Chap.9 Recurrent Neural Networks and LSTMs
Recurrent Neural Networks and LSTMs
Week 9 turns to sequences such as text and signals, where inputs vary in length and order matters. A recurrent network carries a hidden state, updated by one shared cell at every step, so it can be unrolled into a deep network with tied weights. The chapter works a small recurrence by hand, explains exploding and vanishing gradients through time, and develops the gated LSTM cell, the GRU and the parameter counts of each.
What this chapter covers
- 01
Why fixed-size feedforward networks struggle with sequences
- 02
The vanilla RNN cell and weight sharing across time
- 03
Unrolling a recurrence by hand
- 04
Exploding and vanishing gradients through time
- 05
Gradient clipping and gated recurrent cells
- 06
LSTM forget, write and output gates
- 07
GRUs, stacked RNNs and parameter counts
Worked example · free
Unroll a tanh cell over three steps, passing the hidden state forward
- 1h1 = tanh(1) ≈ 0.7616.
- 1h2 = tanh(0.5 × 0.7616 + 0.5) = tanh(0.8808) ≈ 0.7068.
- 1h3 = tanh(0.5 × 0.7068 − 1) = tanh(−0.6466) ≈ −0.5694.
- 1Sensitivity: (1 − h1²) × 0.5(1 − h2²) × 0.5(1 − h3²) ≈ 0.42 × 0.250 × 0.338 ≈ 0.036.
Key terms
- Hidden state
- A hidden state is the vector a recurrent network carries forward to summarise the sequence so far.
- Gradient clipping
- Gradient clipping rescales a gradient whose norm exceeds a threshold before the parameter update.
- Forget gate
- The forget gate is the LSTM sigmoid that decides how much of the previous cell state to keep.
- Gated recurrent unit
- A gated recurrent unit is a recurrent cell with update and reset gates and no separate cell state.
Recurrent Neural Networks and LSTMs FAQ
Why do vanilla RNNs forget long-range information?
Backpropagating across k steps multiplies k factors, each the recurrent weight times a tanh derivative of at most one. When those factors are below one the product shrinks geometrically, so early inputs barely affect the gradient. Factors above one make it explode instead, which clipping can control but vanishing cannot be fixed that way.
How does an LSTM avoid vanishing gradients?
Its cell state is updated by multiplying by the forget gate and adding new content, so the derivative from one cell state to the previous one is simply the forget gate. When that gate stays near one, the gradient passes back almost unchanged, much like the identity shortcut in a residual network.
How many parameters does an LSTM layer have?
Each of its four affine maps, for the forget, write and output gates and the candidate, takes the concatenated hidden state and input, so it has H(H + D) weights and H biases. Multiply by four. With hidden size 128 and input size 64, that is 98,816 parameters.
What is the difference between the cell state and the hidden state in an LSTM?
The cell state is the internal memory, updated only by forgetting and adding, so information and gradients can travel along it for many steps. The hidden state is the gated output, the output gate times tanh of the cell state, which is passed to the next step and to any layer above.
Why does batch normalisation suit recurrent networks poorly?
Sequences in a batch have different lengths, so per-step batch statistics are unreliable, and they would change at every time step. Layer normalisation computes statistics within each sample and step instead, which is why it became the standard normaliser for recurrent networks and transformers.
Exam move
Unroll a short scalar recurrence by hand with your own weights, then compute how strongly the last state depends on the first input, to feel the vanishing effect numerically. Write the LSTM update equations from memory and evaluate one step from given gate values. Count parameters for vanilla, GRU and LSTM layers at two sizes using one affine map as the building block.
Finish by explaining, in a few sentences, why the cell-state path and a residual shortcut solve the same gradient problem. Finally, rehearse the language-modelling view: at each step the hidden state feeds a softmax over the vocabulary, and training minimises cross-entropy on the true next token, the same loss as in the classification chapter.
Working through Recurrent Neural Networks and LSTMs in ECE5179? Sia is AskSia’s AI Computer Science tutor — ask any ECE5179 Recurrent Neural Networks and LSTMs question and get a clear, step-by-step explanation grounded in how ECE5179 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
