The University of Melbourne · FACULTY OF ARTIFICIAL INTELLIGENCE

COMP90054 Chap.10 Deep Tree Search and Learned Guidance

- one subject, every graph, every model, every mark

5 Chapters2-page Bible

Our own words - no uploaded lecturer files

Updated for this semester

Chapter 10 of 11 · COMP90054

Deep Tree Search and Learned Guidance

Define policy prior

The course material gives this chapter a concrete anchor: Week 11 stable materials cover deep learning with tree search, AlphaGo, AlphaZero and MuZero. That policy prior anchor controls how value network is explained and how self-play is tested in changed practice.

Deep Tree Search and Learned Guidance turns policy prior, value network and self-play into executable reasoning.

The chapter's practical target is to explain how learned policy and value estimates guide tree search and training, so every explanation should connect syntax to program state, control flow and observable output.

Treat policy prior as a precise program object, not a loose label. Identify the value or responsibility of policy prior before execution, then trace what can read it, change it or depend on it.

This makes state changes visible before they become debugging guesses.

Formula checkpoint: policy prior

PUCT-style score

Score(s,a)=Q(s,a)+cP(s,a)\frac{\sqrt{N(s)}}{1+N(s,a)}

A common learned-tree score combines estimated value with a prior-weighted exploration bonus.

Trace value network

Use value network to explain the program's next move.

Work through one representative value network input by hand and name the branch, iteration or call that follows. If the value network trace cannot be stated, the code may run by accident rather than by understood design.

Bring in self-play as the test of structure.

Compare normal, boundary and invalid inputs for self-play; state the expected behaviour first; then use the mismatch between expectation and result to localise the defect.

For the application — explain how learned policy and value estimates guide tree search and training — write the smallest complete example that exposes the rule.

Explain why the self-play result works, what would break it and how the program should signal or recover from that failure.

Test with self-play

Before running an example involving policy prior, make a trace table with the important state before and after each operation. Include the value associated with policy prior, the control decision governed by value network and the output or object affected by self-play.

The policy prior table turns an unexplained result into a sequence that can be tested one transition at a time.

Test three inputs: an ordinary case, a boundary case and an invalid case. State the expected self-play result for each before execution, then compare it with what the program actually does.

A useful test of value network isolates one rule; changing several conditions at once cannot reveal which condition caused the failure.

Practise explaining the solution without reading the code. For comp90054, name the data representation, the control flow, the responsibility of each function or class and the reason the chosen design supports explain how learned policy and value estimates guide tree search and training.

This self-play rehearsal matters when a written test or interview asks why the program works rather than whether it produces one correct output.

Transfer to Deep Tree Search and Learned Guidance

A complete response should make the task visible before the detail: identify what must be decided, define the relevant terms, connect the evidence to value network, and use self-play to test the result.

The final sentence about self-play should answer the question actually asked rather than merely repeat the topic.

The controlling limit is specific: Distribution shift, compute concentration, calibration and objective design can produce confident failure.

Keep that self-play limit beside the worked example, because it separates a careful comp90054 answer from one that sounds confident but claims more than the task or evidence supports.

For revision, retrieve policy prior, value network and self-play without notes, explain their relationship aloud, then complete a changed version of the application: explain how learned policy and value estimates guide tree search and training.

Record the first failed value network reasoning move and repair it before attempting another case.

In this chapter

What this chapter covers

01
policy prior
02
value network
03
self-play
04
Applying policy prior
05
Limits of value network and self-play

Worked example · free

Interpret a PUCT choice

Q [4 marks]. AskSia-authored practice. An action has modest current Q, a strong policy prior and few visits. Why might search select it?

1Identify value and prior-weighted exploration terms.
1Relate exploration to parent and child visit counts.
1Explain how more visits reduce the bonus.
1Audit prior error and compute budget.

PUCT can favour the underexplored action because its prior-weighted bonus is large; repeated visits should shift selection toward accumulated evidence, subject to budget and model error.

Sia tip — Learned guidance saves search only when its errors remain discoverable.

Glossary

Key terms

policy prior: Learned probability distribution guiding which actions a tree search explores. This chapter uses the concept when students explain how learned policy and value estimates guide tree search and training. Use this definition when the task is to explain how learned policy and value estimates guide tree search and training.
value network: Function approximator estimating expected outcome from a state. It helps explain the reasoning required to explain how learned policy and value estimates guide tree search and training. Use this definition when the task is to explain how learned policy and value estimates guide tree search and training.
self-play: Generation of learning experience by agents playing or interacting against versions of themselves. Its limit matters because distribution shift, compute concentration, calibration and objective design can produce confident failure. Use this definition when the task is to explain how learned policy and value estimates guide tree search and training.

FAQ

Deep Tree Search and Learned Guidance FAQ

What is the main task in Deep Tree Search and Learned Guidance?

Explain how learned policy and value estimates guide tree search and training.

How do policy prior and value network work together?

Use policy prior to establish the object or condition, then use value network to explain how it changes the outcome being analysed.

What must a comp90054 answer qualify here?

Distribution shift, compute concentration, calibration and objective design can produce confident failure.

How should I revise Deep Tree Search and Learned Guidance?

Retrieve policy prior, value network and self-play, apply them to a changed case, and correct the first point where the evidence no longer supports the conclusion.

Study strategy

Exam move

Reconstruct the relationship among policy prior, value network and self-play; complete the chapter application without notes; then test the result against this limit: Distribution shift, compute concentration, calibration and objective design can produce confident failure.

Working through Deep Tree Search and Learned Guidance in COMP90054? Sia is AskSia’s AI Artificial Intelligence tutor — ask any COMP90054 Deep Tree Search and Learned Guidance question and get a clear, step-by-step explanation grounded in how COMP90054 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.

Keep going — explore the course

A+Everything unlocked

Unlocks this Bible + all 24 of your The University of Melbourne subjects - and 1,000+ Bibles across every Australian university.

Sia - your COMP90054 tutor, unlimited, worked the way the exam marks it

The full 2-page Bible + practice bank with worked solutions

Chrome extension - sync your LMS so Sia knows your deadlines

Bilingual EN / Chinese on every Bible and every Sia answer

$0.99 Trial

30-day money-back · cancel in one tap · how it works