INFO2222 · Computing 2 Usability and Security
Usability Evaluation
Week 4 teaches how to check that a design works: evaluating with users (usability testing, controlled experiments, field studies) versus without users (analytical methods — inspection, heuristic evaluation, walkthroughs, analytics). The exam frames this as method selection under constraints and correct labelling of an experiment's variables, so it is one of the most reliably tested usability topics.
What this chapter covers
- 01Three broad evaluation categories: controlled-with-users (testing/experiments), natural-with-users (field studies), without-users (analytical)
- 02Analytical methods without users: inspection, heuristic evaluation, walkthroughs, models and analytics
- 03Evaluation process: determine goals → explore questions → choose approach/methods → evaluate, interpret, present
- 04Quality criteria: reliability, validity, biases, scope, ethics — and internal vs external validity trade-off
- 05Experiment design: hypotheses (null H0 vs alternative H1), and the four variable roles
- 06Independent variable (manipulated, ≥2 levels), dependent variable (measured), control variable (held constant), random variable (allowed to vary)
- 07Participant assignment: within-subjects vs between-subjects, counterbalancing and the Latin square; A/B testing
- 08Think-aloud method and its users-by-tasks results grid; data analysis is conceptual (no t-test calculation examined)
Design and label a menu-type usability experiment
- +1(a) Alternative hypothesis (H1): menu type affects selection time (e.g. a flat menu is faster than a hierarchical one). Null hypothesis (H0): menu type has no effect on selection time — H0 is simply the inverse, predicting no difference.
- +1(b) Independent variable = menu type, with two levels {flat, hierarchical} (the IV must have at least two consistent levels). Dependent variables = task completion time and error rate (the measured aspects of performance).
- +1Control variable (held constant) = screen resolution (or keyboard type); random variable (allowed to vary) = participant expertise. Holding more variables constant raises internal validity but lowers generalisability.
- +1(c) With few participants and large individual differences, use a within-subjects design so every participant does both conditions, removing person-to-person variance. This introduces order effects (learning and fatigue), so counterbalance the presentation order — e.g. with a Latin square — so each condition appears equally often in each position.
Key terms
- Analytical evaluation
- Evaluation without directly involving users: experts inspect, apply heuristics, walk through tasks, build models or read analytics to predict usability problems. Contrasts with evaluation involving users (usability testing, experiments, field studies).
- Internal vs external validity
- Internal validity is confidence that the manipulation caused the result (strong in controlled experiments). External validity is how well results generalise to real settings (strong in natural/field studies). Controlling more variables raises internal but lowers external validity.
- Independent / dependent variable
- The independent variable (IV) is what the experimenter manipulates, with at least two levels (e.g. menu type = flat/hierarchical). The dependent variable (DV) is the measured aspect of performance related to it (e.g. task time, error rate).
- Control vs random variable
- A control variable is kept constant while the IV is tested (e.g. screen resolution), raising internal validity. A random variable is allowed to vary (e.g. participant expertise), raising external validity but reducing reproducibility.
- Within- vs between-subjects; counterbalancing
- In a within-subjects design every participant does all conditions; in between-subjects each does only one. Within-subjects needs counterbalancing (e.g. a Latin square) to cancel order effects — learning/practice and fatigue/sequence effects.
- Think-aloud
- A usability method where a participant verbalises their thoughts while doing a task, revealing their mental model. Results are reported in a users-by-tasks grid marking each cell as completed easily, completed with help, or not completed.
Usability Evaluation FAQ
How do I choose an evaluation method?
Work from the taxonomy and your constraints. If you can involve users in a controlled setting, use usability testing or a controlled experiment; if you want real-world behaviour, run a field study in natural settings; if you cannot recruit users, use analytical methods (inspection, heuristic evaluation, walkthroughs, models/analytics). Then weigh reliability, validity (internal vs external), bias, scope and ethics. Early lo-fi designs favour cheap inspection or think-aloud; near-final products favour comparative usability tests.
What is the difference between internal and external validity?
Internal validity is how confident you are that the thing you changed (the IV) actually caused the measured effect — highest in tightly controlled experiments. External validity is how well the findings transfer to other, real settings — highest in natural field studies. They trade off: the more you control, the more internally valid but the less generalisable your study becomes.
Do I have to calculate t-tests for the exam?
No. The teaching team states you will not hand-calculate statistics (t-tests, ANOVA) or recite equations. You do need the concepts: what a hypothesis, p-value (p < 0.05 as the usual significance threshold) and standard deviation mean, and when a parametric test (normal data) versus a non-parametric test (e.g. Wilcoxon rank-sum) is appropriate. Understand the selection logic, not the arithmetic. Confirm scope in the Week 12 revision on Canvas.
Can AI help me design a usability study for INFO2222?
Yes, as a study aid. Sia can help you label IV/DV/control/random variables, decide between within- and between-subjects designs, explain why counterbalancing with a Latin square is needed, and reason about which evaluation method fits a scenario. Use it to rehearse for the quizzes and to think through your project's evaluation plan; it does not run your graded study or write your report, and the University of Sydney academic-integrity policy applies.
Exam move
The two exam-heavy skills here are method selection and variable labelling, so drill both. For selection, memorise the taxonomy as a small tree — with users {controlled: testing/experiments; natural: field studies} vs without users {analytical: inspection, heuristics, walkthroughs, models/analytics} — and practise picking one for a given constraint plus one advantage and one disadvantage. For experiments, take any comparison (GUI vs CLI, dark vs light theme) and rehearse writing H0/H1 and labelling the four variable roles, then decide within- vs between-subjects and whether counterbalancing is needed. Keep the internal-vs-external validity trade-off and the reliability/validity/bias/scope/ethics criteria on a card. Treat the statistics conceptually — know when each test applies, not how to compute it. This chapter directly informs your project's evaluation phase. Confirm the examinable depth on Canvas.
Working through Usability Evaluation in INFO2222? Sia is AskSia’s AI Computer Science tutor — ask any INFO2222 Usability Evaluation question and get a clear, step-by-step explanation grounded in how INFO2222 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
