COMP90054 Chap.8 SARSA and Q-Learning Control

SARSA and Q-Learning Control

Define temporal-difference learning

The course material gives this chapter a concrete anchor: Week 8 materials directly cover model-free control through SARSA and Q-learning. That temporal-difference learning anchor controls how SARSA is explained and how Q-learning is tested in changed practice.

SARSA and Q-Learning Control turns temporal-difference learning, SARSA and Q-learning into executable reasoning.

The chapter's practical target is to calculate and compare on-policy and off-policy action-value updates, so every explanation should connect syntax to program state, control flow and observable output.

Treat temporal-difference learning as a precise program object, not a loose label.

Identify the value or responsibility of temporal-difference learning before execution, then trace what can read it, change it or depend on it.

This makes state changes visible before they become debugging guesses.

Trace SARSA

Use SARSA to explain the program's next move.

Work through one representative SARSA input by hand and name the branch, iteration or call that follows. If the SARSA trace cannot be stated, the code may run by accident rather than by understood design.

Bring in Q-learning as the test of structure.

Compare normal, boundary and invalid inputs for Q-learning; state the expected behaviour first; then use the mismatch between expectation and result to localise the defect.

For the application — calculate and compare on-policy and off-policy action-value updates — write the smallest complete example that exposes the rule.

Explain why the Q-learning result works, what would break it and how the program should signal or recover from that failure.

Test with Q-learning

Before running an example involving temporal-difference learning, make a trace table with the important state before and after each operation.

Include the value associated with temporal-difference learning, the control decision governed by SARSA and the output or object affected by Q-learning. The temporal-difference learning table turns an unexplained result into a sequence that can be tested one transition at a time.

Test three inputs: an ordinary case, a boundary case and an invalid case.

State the expected Q-learning result for each before execution, then compare it with what the program actually does. A useful test of SARSA isolates one rule; changing several conditions at once cannot reveal which condition caused the failure.

Practise explaining the solution without reading the code.

For comp90054, name the data representation, the control flow, the responsibility of each function or class and the reason the chosen design supports calculate and compare on-policy and off-policy action-value updates.

This Q-learning rehearsal matters when a written test or interview asks why the program works rather than whether it produces one correct output.

Transfer to SARSA and Q-Learning Control

A complete response should make the task visible before the detail: identify what must be decided, define the relevant terms, connect the evidence to SARSA, and use Q-learning to test the result.

The final sentence about Q-learning should answer the question actually asked rather than merely repeat the topic.

The controlling limit is specific: Function approximation, correlated data, exploration and changing targets can destabilise learning.

Keep that Q-learning limit beside the worked example, because it separates a careful comp90054 answer from one that sounds confident but claims more than the task or evidence supports.

For revision, retrieve temporal-difference learning, SARSA and Q-learning without notes, explain their relationship aloud, then complete a changed version of the application: calculate and compare on-policy and off-policy action-value updates.

Record the first failed SARSA reasoning move and repair it before attempting another case.

What this chapter covers

• 01

  temporal-difference learning

• 02

  SARSA

• 03

  Q-learning

• 04

  Applying temporal-difference learning

• 05

  Limits of SARSA and Q-learning

Apply one Q-learning update

Key terms

temporal-difference learning

Value learning from a reward plus a bootstrapped next-state estimate before an episode ends. This chapter uses the concept when students calculate and compare on-policy and off-policy action-value updates. Use this definition when the task is to calculate and compare on-policy and off-policy action-value updates.

SARSA

On-policy control update using the next action actually selected by the behaviour policy. It helps explain the reasoning required to calculate and compare on-policy and off-policy action-value updates. Use this definition when the task is to calculate and compare on-policy and off-policy action-value updates.

Q-learning

Off-policy control update using the maximum estimated next-state action value. Its limit matters because function approximation, correlated data, exploration and changing targets can destabilise learning. Use this definition when the task is to calculate and compare on-policy and off-policy action-value updates.

SARSA and Q-Learning Control FAQ

Exam move

Reconstruct the relationship among temporal-difference learning, SARSA and Q-learning; complete the chapter application without notes; then test the result against this limit: Function approximation, correlated data, exploration and changing targets can destabilise learning.

Working through SARSA and Q-Learning Control in COMP90054? Sia is AskSia’s AI Artificial Intelligence tutor — ask any COMP90054 SARSA and Q-Learning Control question and get a clear, step-by-step explanation grounded in how COMP90054 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.