MATH7005 Chap.5 Feed-Forward Network Architecture
Feed-Forward Network Architecture
Representations built layer by layer
A feed-forward network alternates affine transformations with nonlinear activations. The chapter uses a dimension ledger to show how input width, hidden width and output design determine every weight and bias shape.
It compares sigmoid, tanh and ReLU by range and derivative behaviour rather than by slogans.
What to verify
Preserve both pre-activations and activations, assert broadcast shapes and match the final activation to the target structure.
A more complex architecture should also be compared with a simpler baseline under the same data and validation conditions.
Dimensions express the design
Input width comes from the feature representation, hidden widths set the learned intermediate spaces and output width comes from the target. Write these numbers before creating weight matrices.
A bias has one entry per output unit and broadcasts across records; accidentally giving it a batch dimension changes the model rather than merely changing syntax.
Activation choice changes gradient flow
Sigmoid and tanh compress extreme values and may produce small derivatives, while ReLU passes positive values with unit local slope and blocks negative ones.
Inspect activation distributions across a batch instead of judging from one record. A unit returning zero once is ordinary; a unit remaining inactive throughout the data indicates a more serious problem.
Complexity must earn its place
Count parameters, runtime and validation performance beside the result from logistic regression or another simple baseline.
Additional width and depth can fit richer structure and more noise. If the improvement is small, explain whether latency, interpretability or maintenance makes the simpler model preferable for the data product.
What this chapter covers
- 01
Affine layer
- 02
Weight and bias shapes
- 03
Nonlinear activation
- 04
Hidden representation
- 05
Output activation
- 06
Parameter counting
Worked example · free
Trace a small hidden layer
- 3Multiply the input by the weight matrix.
- 2Add the bias.
- 2Apply ReLU componentwise.
Key terms
- Affine Layer
- A weight-matrix multiplication followed by addition of a bias vector.
- Hidden Layer
- An intermediate learned representation between model input and output.
- Pre-Activation
- The affine-layer output before a nonlinear activation is applied.
- ReLU
- An activation returning zero for a negative input and the input itself when positive.
- Output Layer
- The final transformation whose shape and activation must match the prediction target.
Feed-Forward Network Architecture FAQ
Why are nonlinear activations necessary?
Without them, consecutive affine layers collapse into one affine transformation. Nonlinearity lets the network build curved boundaries and hierarchical representations that a single linear map cannot express.
How do I count dense-layer parameters?
Multiply input width by output width for the weight entries, then add one bias per output unit. Batch size changes the number of records processed, not the number of trainable parameters.
Is a deeper network always better?
No. Depth can represent more complex compositions but raises optimisation, computation and overfitting risks. The architecture should earn its complexity through validation evidence against a simpler baseline on the same task.
What does a zero ReLU output mean?
For one record it means the pre-activation was not positive. A dead unit is a stronger pattern in which the pre-activation remains negative across relevant data, so inspect the distribution rather than diagnosing from one example.
Why preserve the pre-activation as well as the activation?
The backward pass evaluates the activation derivative at the pre-activation, and debugging needs to distinguish an affine output from the nonlinear gate. Saving both also reveals whether unexpected zeros came from weights, bias or activation.
How should I justify a hidden-layer width?
Treat width as a capacity choice tested against a simpler baseline. Compare parameter count, validation behaviour, runtime and failure cases under the same preprocessing and split, then retain added units only when the evidence supports their cost.
Where does the batch dimension appear in a dense layer?
A batch adds a leading row dimension to the activation matrix, while the shared weight matrix and bias retain their model-defined shapes. Broadcasting the bias across rows does not create a separate trainable bias for each record.
Assessment move
Draw a small network with the dimension written under every layer. Derive each weight and bias shape, count parameters and then trace one record through the affine calculation, stored pre-activation and nonlinear output. Change an activation input from positive to negative and predict how ReLU alters the forward value and local gradient.
In code, assert tensor shapes and test bias broadcasting on a batch with more than one record. Match the final layer to binary, mutually exclusive multiclass, multi-label and regression targets in turn. For project evidence, compare the network with a simple baseline on the same split, record runtime and inspect activation distributions rather than presenting only the final accuracy.
Working through Feed-Forward Network Architecture in MATH7005? Sia is AskSia’s AI Computer Science tutor — ask any MATH7005 Feed-Forward Network Architecture question and get a clear, step-by-step explanation grounded in how MATH7005 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
