IS6400 Chap.8 Neural Networks and Recurrent Models
Neural Networks and Recurrent Models
A network starts with weighted sums
Each neural unit multiplies inputs by weights, adds a bias and applies an activation. Layers repeat that calculation to represent nonlinear combinations.
Hidden activations are representations rather than probabilities, and dimensions should be tracked at every layer to prevent silent shape errors.
Recurrent state carries sequence history
A recurrent model passes information from one position to the next, which is useful when order changes meaning.
Split by entity and time according to the deployment claim, mask padding and compare with engineered lag features so recurrence is used for evidence rather than appearance.
Reduce a forward pass to arithmetic
For each unit, multiply inputs by weights, sum the products, add the bias and apply the activation.
ReLU returns zero for a negative pre-activation and the positive value otherwise; a sigmoid maps a value into a bounded score. Track the vector or matrix shape at every layer because broadcasting can hide an incorrect calculation.
Hidden activations are learned representations, not automatically calibrated probabilities or human-readable concepts.
Make the split match the sequence claim
If deployment predicts later events for known customers, preserve time within each customer. If it predicts for new customers, hold out entire customers. Fragments of one sequence on both sides leak identity and history even when timestamps differ.
Specify padding, masking, truncation and maximum length so the network cannot treat administrative formatting as signal.
Compare with lag features or another simple sequential baseline to establish whether recurrence adds useful order information.
Use training curves as diagnostic evidence
Optimisation updates weights to reduce a loss on training batches, while validation data supports architecture and stopping choices.
Falling training loss with worsening validation loss suggests overfitting; two poor flat curves suggest representation, optimisation or data problems. Reserve the final test until choices are complete. After deployment, monitor input ranges, sequence lengths, score distributions and segment errors, and retain the preprocessing shape receipt needed to reproduce a live prediction.
What this chapter covers
- 01
Weighted sums and bias
- 02
ReLU and sigmoid activations
- 03
Layer dimensions
- 04
Training loss and validation
- 05
Recurrent hidden state
- 06
Entity and time leakage
Worked example · free
Compute two neural activations
- 3Multiply inputs and weights with signs. The weighting here is not a published University assessment scheme.
- 2Add bias and apply ReLU.
- 2Explain why the first contribution is positive.
Key terms
- Weighted Sum
- The sum of input values multiplied by learned weights, plus a bias.
- Activation Function
- A nonlinear transformation applied to a neural unit's pre-activation value.
- ReLU
- An activation returning zero for negative input and the input itself otherwise.
- Hidden State
- A recurrent representation carried from earlier sequence positions into later calculations.
- Sequence Leakage
- Information crossing evaluation boundaries because related positions or entities occur on both sides of a split.
Neural Networks and Recurrent Models FAQ
Why use an activation after the weighted sum?
Without nonlinear activations, stacked linear layers collapse into another linear transformation. Activations allow the network to represent relationships that cannot be captured by one linear combination.
When does an RNN add value?
It adds value when ordered history changes the prediction and a recurrent state outperforms simpler lag features under an entity- and time-safe split. Sequence-shaped input alone is not enough.
How can padding damage a sequence model?
If padding is not masked, the model can learn sequence length or formatting artifacts as signal. Apply masks consistently and test whether performance survives changes in padding and batch composition.
Exam move
Work several single-unit calculations until signed products, bias and activation are automatic. Draw a feed-forward network and a recurrent cell, then explain the one extra arrow in plain language. For each sequence case, state whether deployment predicts later events for known entities or events for new entities, because that claim determines the split.
Compare the recurrent model with a simple lag-feature baseline and inspect performance across sequence lengths. Annotate every layer with batch, sequence and feature dimensions, then trace one row through weighted sum, bias and activation. Create one safe split for known entities over time and another for unseen entities, explaining why fragments cannot cross the boundary in the second case.
Sketch training and validation loss for overfitting and underfitting, propose the smallest response to each, and reserve the final test until architecture and stopping choices are complete. Rehearse a deployment receipt containing tensor shape, padding rule, split unit, baseline, threshold and monitored sequence-length range so preprocessing remains reproducible.