City University of Hong Kong · FACULTY OF INFORMATION TECHNOLOGY

IS6400 Chap.8 Neural Networks and Recurrent Models

- one subject, every graph, every model, every mark
6 Chapters2-page Bible
Our own words - no uploaded lecturer files
Updated for this semester
Chapter 8 of 10 · IS6400

Neural Networks and Recurrent Models

A network starts with weighted sums

Each neural unit multiplies inputs by weights, adds a bias and applies an activation. Layers repeat that calculation to represent nonlinear combinations.

Hidden activations are representations rather than probabilities, and dimensions should be tracked at every layer to prevent silent shape errors.

Recurrent state carries sequence history

A recurrent model passes information from one position to the next, which is useful when order changes meaning.

Split by entity and time according to the deployment claim, mask padding and compare with engineered lag features so recurrence is used for evidence rather than appearance.

Reduce a forward pass to arithmetic

For each unit, multiply inputs by weights, sum the products, add the bias and apply the activation.

ReLU returns zero for a negative pre-activation and the positive value otherwise; a sigmoid maps a value into a bounded score. Track the vector or matrix shape at every layer because broadcasting can hide an incorrect calculation.

Hidden activations are learned representations, not automatically calibrated probabilities or human-readable concepts.

Make the split match the sequence claim

If deployment predicts later events for known customers, preserve time within each customer. If it predicts for new customers, hold out entire customers. Fragments of one sequence on both sides leak identity and history even when timestamps differ.

Specify padding, masking, truncation and maximum length so the network cannot treat administrative formatting as signal.

Compare with lag features or another simple sequential baseline to establish whether recurrence adds useful order information.

Use training curves as diagnostic evidence

Optimisation updates weights to reduce a loss on training batches, while validation data supports architecture and stopping choices.

Falling training loss with worsening validation loss suggests overfitting; two poor flat curves suggest representation, optimisation or data problems. Reserve the final test until choices are complete. After deployment, monitor input ranges, sequence lengths, score distributions and segment errors, and retain the preprocessing shape receipt needed to reproduce a live prediction.

In this chapter

What this chapter covers

  • 01

    Weighted sums and bias

  • 02

    ReLU and sigmoid activations

  • 03

    Layer dimensions

  • 04

    Training loss and validation

  • 05

    Recurrent hidden state

  • 06

    Entity and time leakage

Worked example · free

Compute two neural activations

Q [7 marks]. AskSia-authored practice. The listed marks are AskSia study allocations, not a University rubric. Inputs are x1 = -2 and x2 = 1, weights are -0.5 and 0.3, and bias is -0.1. Find z and the ReLU output. Then explain the sign check.
The left network sends information forward once.The left network sends information forward once. The right network also sends a hidden state onward, allowing later predictions to depend on earlier sequence positions.Feed-forward paths end; recurrent paths carry a state to the next stepstate tstate t+1memory connection
The left network sends information forward once. The right network also sends a hidden state onward, allowing later predictions to depend on earlier sequence positions.
  • 3Multiply inputs and weights with signs. The weighting here is not a published University assessment scheme.
  • 2Add bias and apply ReLU.
  • 2Explain why the first contribution is positive.
The weighted sum is z = (-0.5)(-2) + 0.3(1) - 0.1 = 1 + 0.3 - 0.1 = 1.2. ReLU returns 1.2 because z is positive. The first contribution is positive because multiplying two negative values gives a positive value. If z had been negative, ReLU would return zero; that boundary belongs to the activation and should not be confused with a missing input or zero probability.
Sia tip — Compute each signed contribution on its own line before summing; neural arithmetic is ordinary arithmetic at scale.
Glossary

Key terms

Weighted Sum
The sum of input values multiplied by learned weights, plus a bias.
Activation Function
A nonlinear transformation applied to a neural unit's pre-activation value.
ReLU
An activation returning zero for negative input and the input itself otherwise.
Hidden State
A recurrent representation carried from earlier sequence positions into later calculations.
Sequence Leakage
Information crossing evaluation boundaries because related positions or entities occur on both sides of a split.
FAQ

Neural Networks and Recurrent Models FAQ

Why use an activation after the weighted sum?

Without nonlinear activations, stacked linear layers collapse into another linear transformation. Activations allow the network to represent relationships that cannot be captured by one linear combination.

When does an RNN add value?

It adds value when ordered history changes the prediction and a recurrent state outperforms simpler lag features under an entity- and time-safe split. Sequence-shaped input alone is not enough.

How can padding damage a sequence model?

If padding is not masked, the model can learn sequence length or formatting artifacts as signal. Apply masks consistently and test whether performance survives changes in padding and batch composition.

Study strategy

Exam move

Work several single-unit calculations until signed products, bias and activation are automatic. Draw a feed-forward network and a recurrent cell, then explain the one extra arrow in plain language. For each sequence case, state whether deployment predicts later events for known entities or events for new entities, because that claim determines the split.

Compare the recurrent model with a simple lag-feature baseline and inspect performance across sequence lengths. Annotate every layer with batch, sequence and feature dimensions, then trace one row through weighted sum, bias and activation. Create one safe split for known entities over time and another for unseen entities, explaining why fragments cannot cross the boundary in the second case.

Sketch training and validation loss for overfitting and underfitting, propose the smallest response to each, and reserve the final test until architecture and stopping choices are complete. Rehearse a deployment receipt containing tensor shape, padding rule, split unit, baseline, threshold and monitored sequence-length range so preprocessing remains reproducible.

A+Everything unlocked
Unlocks this Bible + all 31 of your subjects at City University of Hong Kong, plus 1,000+ more Bibles across the AskSia library.
Sia - your IS6400 tutor, unlimited, worked the way the exam marks it
The full 2-page Bible + practice bank with worked solutions
Chrome extension - sync your LMS so Sia knows your deadlines
Bilingual EN / Chinese on every Bible and every Sia answer
$0.99 Trial
30-day money-back · cancel in one tap · how it works