COMP90054 Chap.9 Policy Gradient and Actor-Critic

Policy Gradient and Actor-Critic

Define policy gradient

The course material gives this chapter a concrete anchor: Stable Week 10 materials cover policy-gradient and actor-critic methods. That policy gradient anchor controls how actor is explained and how critic is tested in changed practice.

Policy Gradient and Actor-Critic turns policy gradient, actor and critic into executable reasoning.

The chapter's practical target is to derive the direction of a policy-gradient or actor-critic update and interpret its estimator, so every explanation should connect syntax to program state, control flow and observable output.

Treat policy gradient as a precise program object, not a loose label.

Identify the value or responsibility of policy gradient before execution, then trace what can read it, change it or depend on it.

This makes state changes visible before they become debugging guesses.

Trace actor

Use actor to explain the program's next move.

Work through one representative actor input by hand and name the branch, iteration or call that follows. If the actor trace cannot be stated, the code may run by accident rather than by understood design.

Bring in critic as the test of structure.

Compare normal, boundary and invalid inputs for critic; state the expected behaviour first; then use the mismatch between expectation and result to localise the defect.

For the application — derive the direction of a policy-gradient or actor-critic update and interpret its estimator — write the smallest complete example that exposes the rule.

Explain why the critic result works, what would break it and how the program should signal or recover from that failure.

Test with critic

Before running an example involving policy gradient, make a trace table with the important state before and after each operation. Include the value associated with policy gradient, the control decision governed by actor and the output or object affected by critic.

The policy gradient table turns an unexplained result into a sequence that can be tested one transition at a time.

Test three inputs: an ordinary case, a boundary case and an invalid case. State the expected critic result for each before execution, then compare it with what the program actually does.

A useful test of actor isolates one rule; changing several conditions at once cannot reveal which condition caused the failure.

Practise explaining the solution without reading the code.

For comp90054, name the data representation, the control flow, the responsibility of each function or class and the reason the chosen design supports derive the direction of a policy-gradient or actor-critic update and interpret its estimator.

This critic rehearsal matters when a written test or interview asks why the program works rather than whether it produces one correct output.

Transfer to Policy Gradient and Actor-Critic

A complete response should make the task visible before the detail: identify what must be decided, define the relevant terms, connect the evidence to actor, and use critic to test the result.

The final sentence about critic should answer the question actually asked rather than merely repeat the topic.

The controlling limit is specific: Gradient estimates can have high variance, critic bias and unsafe exploration, and local optimisation gives no global guarantee.

Keep that critic limit beside the worked example, because it separates a careful comp90054 answer from one that sounds confident but claims more than the task or evidence supports.

For revision, retrieve policy gradient, actor and critic without notes, explain their relationship aloud, then complete a changed version of the application: derive the direction of a policy-gradient or actor-critic update and interpret its estimator.

Record the first failed actor reasoning move and repair it before attempting another case.

What this chapter covers

• 01

  policy gradient

• 02

  actor

• 03

  critic

• 04

  Applying policy gradient

• 05

  Limits of actor and critic

Interpret an advantage update

Key terms

policy gradient

Direct optimisation of parameterised policy performance using an estimated return gradient. This chapter uses the concept when students derive the direction of a policy-gradient or actor-critic update and interpret its estimator. Use this definition when the task is to derive the direction of a policy-gradient or actor-critic update and interpret its estimator.

actor

Component selecting actions according to a parameterised policy. It helps explain the reasoning required to derive the direction of a policy-gradient or actor-critic update and interpret its estimator. Use this definition when the task is to derive the direction of a policy-gradient or actor-critic update and interpret its estimator.

critic

Component estimating value or advantage to guide actor updates. Its limit matters because gradient estimates can have high variance, critic bias and unsafe exploration, and local optimisation gives no global guarantee. Use this definition when the task is to derive the direction of a policy-gradient or actor-critic update and interpret its estimator.

Policy Gradient and Actor-Critic FAQ

Exam move

Reconstruct the relationship among policy gradient, actor and critic; complete the chapter application without notes; then test the result against this limit: Gradient estimates can have high variance, critic bias and unsafe exploration, and local optimisation gives no global guarantee.

Working through Policy Gradient and Actor-Critic in COMP90054? Sia is AskSia’s AI Artificial Intelligence tutor — ask any COMP90054 Policy Gradient and Actor-Critic question and get a clear, step-by-step explanation grounded in how COMP90054 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.