PMGM7023 Chap.9 Web Scraping Within the Rules
Web Scraping Within the Rules
Permission is checked before technique
The project rubric asks teams to collect their own data through web scraping, so this tutorial is a compliance step as well as a method. The method is short: a program reads the structured markup behind a page and writes the values into a table.
The compliance step comes first, and it has three gates that must all pass.
The first asks whether the content is reachable without logging in or paying. Material behind a login or a paywall is not freely accessible and is not collectable, whether or not you personally hold an account.
The second asks whether the site's own terms permit automated access; some prohibit it explicitly, and a prohibition holds even where the pages are public.
The third asks whether the site's crawler instructions disallow your target path, in a short file listing which automated visitors the rules apply to and which paths are allowed or disallowed.
Reading the crawler instructions properly
Two details catch people out. The rules are stated per visitor, so a block aimed at whole-site copying tools may not apply to you while a general rule does.
And permissions are stated per path, so a site can allow its listing pages and disallow its account or comparison pages. Checking the domain and stopping is not the check; the target path has to be found in the list.
Silence is treated differently at different gates. Terms that say nothing about automated access pass the second gate.
A crawler file that explicitly disallows your path is a stop even where the terms are silent and the page is public.
Four collection moves, chosen by what you already know
With a known address, read that page and return the fields. Where the content only appears after an interaction, open the page and then read what appears, since reading alone returns the pre-interaction state.
Where you know the site but not the specific pages, list the site's addresses first to build a collection frame without reading any content. Where you have only a topic, search, verify the candidates, then read them.
The choice follows from your knowledge of the target rather than from the size of the job, and most wasted effort comes from using the first move where the third was needed.
What belongs in every collected file
Two fields belong in any scraped dataset regardless of the question. The address each row came from lets any value be traced back and rechecked.
The date and time of collection turn a changing page into a dated observation, and without it rows gathered a week apart are silently treated as contemporaneous. Sample a few pages by hand before running anything at scale, both to confirm the fields are present in the markup and to check that what you are reading is what a reader sees.
Record the outcome of all three permission gates for every source.
The project asks for the source website, the URLs, the collection dates and the procedure, and the permission record is the part that shows the collection was allowed as well as possible.
What this chapter covers
- 01
A page becomes a table by reading structured markup
- 02
Three permission gates, in order, all of which must pass
- 03
Choosing a collection move from what you know about the target
- 04
Address and collection time as mandatory fields in any scraped file
Planning a price collection across a retail chain
- 3Run and record the three permission checks.
- 4Choose the collection move and justify it.
- 3Name the fields that make the file traceable.
Key terms
- Crawler Instructions
- Crawler instructions are a file at a website's root stating which automated visitors the rules apply to and which paths they may or may not visit.
- Terms of Service
- Terms of service are a site's stated conditions of use, which may prohibit automated access even where the pages themselves are public.
- Collection Frame
- A collection frame is the list of addresses a collection will cover, built before any content is read so that coverage can be checked afterwards.
- Static Page
- A static page delivers its content in the markup that arrives with the page, so reading it returns what a person would see without interacting.
Web Scraping Within the Rules FAQ
Can I scrape a website just because it is public?
No. Three checks must all pass first. The content must be reachable without logging in or paying; the site's terms must not prohibit automated access; and the site's crawler instructions must not disallow the specific path you want. A failure at any one gate ends the collection for that path, and collecting more slowly does not change permission. Record all three verdicts with the date checked.
Which move should I use when I do not know the page addresses?
List the site's addresses first to build a collection frame, then read the pages that match the pattern you need. Reading is for a known address, opening and then reading is for content that appears only after an interaction, and searching first is for cases where you have only a topic. The choice depends on what you already know about the target rather than on how much data you want.
Exam move
Run the three permission checks on two unfamiliar sites this week and write the verdicts as you would in the supporting document. Practise identifying which collection move each situation calls for before writing any request. Add the source address and collection timestamp to every file you build, including practice ones.
Working through Web Scraping Within the Rules in PMGM7023? Sia is AskSia’s AI Management tutor — ask any PMGM7023 Web Scraping Within the Rules question and get a clear, step-by-step explanation grounded in how PMGM7023 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
