MATH2089 Chap.1 Numerical Computing and Error
Numerical Computing and Error
Floating point arithmetic replaces the real number line with a finite representable set, so every numerical result must be read through an error model. This chapter separates signed, absolute and relative error, distinguishes roundoff from truncation, explains cancellation and connects accuracy with operation count, memory and big O growth.
The practical goal is to state what was approximated, how error was scaled and which check makes the answer credible. A machine number stores a sign, a finite significand and an exponent. Values between representable numbers must be rounded, and the absolute spacing grows as magnitude grows even when relative precision remains broadly controlled.
That observation explains why a printed decimal is not the mathematical value itself and why reporting many digits cannot recover information that was never stored. Signed error records whether an approximation is above or below a reference. Absolute error records distance in the original unit, while relative error compares that distance with a meaningful scale.
A reliable solution identifies the reference used and avoids a relative-error formula when the reference is zero or too close to zero for the ratio to be informative. Roundoff arises from finite representation and arithmetic. Truncation arises when an infinite or exact mathematical process is replaced by a finite rule, such as a difference formula or a stopped iteration.
Reducing a step size can lower truncation but can also increase the influence of roundoff through subtraction and division. The best step is therefore not automatically the smallest one available. Cancellation is especially important when nearly equal stored quantities are subtracted. Shared leading digits disappear, leaving the result sensitive to earlier rounding.
Reformulating the calculation may be more valuable than changing output precision. Stability asks whether an algorithm amplifies small perturbations, while conditioning asks whether the mathematical problem itself is sensitive. A stable method cannot restore information lost in an ill-conditioned problem. Efficiency belongs in the same judgement.
A flop count estimates arithmetic work, memory records storage cost, and big O notation describes dominant growth as the input size increases. Constants and data movement still matter for a real run, but growth rate explains why a method that works for a small example may fail at scale.
End each computation with an audit: name the approximation, classify its main errors, scale the discrepancy, state the resource cost and provide an independent check such as refinement, a residual or a known limiting case.
What this chapter covers
- 01
Floating point spacing and finite precision
- 02
Signed, absolute and relative error
- 03
Roundoff, truncation and cancellation
- 04
Floating point operation counts
- 05
Big O growth and resource tradeoffs
- 06
Stability and independent checks
Worked example · free
Scale an approximation error
- +1Signed error is 12.43 minus 12.48, which equals negative 0.05.
- +1Absolute error is the magnitude, 0.05 units.
- +1Relative error is 0.05 divided by 12.48, about 0.004006.
- +1As a percentage, the approximation is low by about 0.401% of the reference.
Key terms
- Floating point
- A finite sign, significand and exponent representation used to store approximate real numbers.
- Relative error
- Absolute error divided by a reference magnitude, allowing comparisons across scales.
- Cancellation
- Loss of significant digits when nearly equal stored values are subtracted.
Numerical Computing and Error FAQ
Why is the smallest step not always best?
A smaller step can reduce truncation error, but it can also magnify roundoff through subtraction and division. Numerical accuracy often improves only until an optimal region, after which finite precision begins to dominate.
What does big O omit?
Big O records the dominant growth rate as problem size increases. It omits constants, lower order work and hardware effects, so a method with better asymptotic growth can still lose on a small input.
How should a numerical result be checked?
Use a check targeted at the method: a residual, a known identity, a refinement ratio or comparison with a higher precision reference. The check should be capable of exposing a real failure.
Exam move
Build a four-column error ledger for representation, error source, computational cost and verification. Recalculate the chapter example with new values and practise explaining why extra printed digits cannot recover information lost through cancellation. For every later algorithm, add its dominant error and an independent check to the same ledger.
Run a step-size experiment for one difference formula and mark where truncation gives way to roundoff. Compare signed, absolute and relative errors for values on very different scales, including a case where a relative denominator is unsuitable. For cost, write the dominant operation and storage growth beside each method instead of memorising big O labels without a model.
End a timed response with one sentence that states what information is trustworthy and why. That final judgement forces representation, stability, conditioning, efficiency and verification into a single answer rather than leaving them as isolated definitions.
Working through Numerical Computing and Error in MATH2089? Sia is AskSia’s AI Mathematics tutor — ask any MATH2089 Numerical Computing and Error question and get a clear, step-by-step explanation grounded in how MATH2089 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.