DATA2001 · Data Science, Big Data and Data Variety
Web APIs and NoSQL
Week 7 opens the data-variety arc through the web: retrieving data from REST APIs and handling semi-structured JSON and XML, then storing flexible/optional fields in PostgreSQL's JSONB and contrasting structured SQL with NoSQL stores. It is grounded in the group assignment's use of a Points-of-Interest API to build a dataset, and the JSON/JSONB and pagination ideas here appear in the 50% final exam.
What this chapter covers
- 01XML — user-defined tags, well-formed rules (one root, case-sensitive, quoted attributes, closed tags), DTD validity
- 02JSON — name:value pairs, objects { }, arrays [ ], values (string/number/object/array/boolean/null); easier to parse than XML
- 03JSON in PostgreSQL: json (raw text) vs jsonb (binary, indexable, queryable); use columns for stable data, JSONB for flexible/optional
- 04JSONB accessors: -> returns a JSON value, ->> returns text, and chained access for nested fields
- 05Web APIs: client HTTP request to a URL, server response often as JSON; requests.get(url, params=...), status 200, response.json()
- 06Paginated APIs: count (total available) vs number fetched per page, and the next URL; json.dumps / json.loads
- 07Structured vs semi-structured data, and NoSQL as flexible-schema storage for variety
Pagination: total available vs number fetched
- +1One call. A single request returns one page — at most 40 records — so the first call yields 40 records, not 250. The count field (250) is the total available across all pages, not the number returned now.
- +1Number of calls. To fetch all 250 at 40 per page, take the ceiling of 250 ÷ 40 = 6.25, which rounds up to 7 calls (6 full pages plus a partial one).
- +1Final page size. Six full pages carry 6 × 40 = 240 records; the remaining 250 - 240 = 10 records are on the seventh (final) call.
- +1Why they differ. APIs paginate to bound response size and server load, so each response returns one page and exposes the total in count and a next URL to follow. You must loop on next (or increment an offset) until you have gathered all count records.
Key terms
- REST API
- A way for software to talk over the web using HTTP: a client sends a request to a URL, the server processes it and returns a response, often JSON. In Python: response = requests.get(url, params={...}); check response.status_code == 200; data = response.json().
- JSON
- JavaScript Object Notation — a human-readable, semi-structured format of name:value pairs. Objects use { }, arrays use [ ], and values are string, number, object, array, boolean or null. Names are always double-quoted.
- XML vs JSON
- Both are self-describing, semi-structured formats carrying data from a server. XML allows user-defined tags and needs an XML parser (harder to parse); JSON maps directly to native objects/arrays and is lighter. Well-formed XML needs one root, closed case-sensitive tags and quoted attributes.
- JSONB in PostgreSQL
- A binary, indexable JSON column type (preferred over the raw json type) for storing flexible or optional fields that vary between records — use fixed columns for stable core data and JSONB for the variable part.
- JSONB accessors -> and ->>
- Inside a JSONB value, -> returns a JSON object/value (e.g. details->'brand') while ->> returns the value as text (e.g. details->>'brand' = 'TechCorp'). Chain them for nested fields, such as details->'features'->>'camera'.
- Pagination
- Splitting a large API result into pages. A response exposes the total via count and a next URL (or offset) for the following page; a single call returns only one page, so count usually exceeds the number fetched at once.
Web APIs and NoSQL FAQ
What is the difference between JSON and XML?
Both are semi-structured, self-describing formats used to transport data. XML wraps data in user-defined, case-sensitive tags and needs a dedicated parser (and optionally a DTD to be 'valid'), which makes it heavier. JSON represents data as objects and arrays of name:value pairs that map straight onto native language structures and parse more easily, which is why most modern web APIs return JSON.
When should I use a JSONB column instead of normal columns?
Use ordinary columns for the stable, always-present core fields, and a JSONB column for data that is optional, varies between records, or changes shape over time — for example a product's spec sheet where different products list different attributes. JSONB is binary, indexable and queryable with the -> and ->> accessors, giving you schema flexibility inside a relational database rather than forcing a rigid column for every possible field.
Why does an API's count not match how many records I got?
Because APIs paginate. count reports the total number of records available, but a single response returns only one page (say 40 records) to keep responses small and the server responsive. The response also gives a next URL; you loop on it, accumulating results, until you have all count records. Confusing count with the page size is a common bug.
Can AI help me work with APIs and JSON in DATA2001?
Yes. Sia can walk you through calling a REST API with requests, checking the status code, parsing response.json(), navigating nested JSON, designing a JSONB column, and looping through pagination, and it will check your code and reasoning. Practise on a neutral public API. It is a study aid and does not do graded assessment; the University of Sydney academic-integrity policy applies.
Exam move
Build one clean end-to-end API pattern you can reproduce: requests.get(url, params=...), check status_code == 200, response.json() into a Python dict, then navigate the nested structure. Practise pagination explicitly — loop on the next URL until you have all count records, and be able to explain why count exceeds a single page. Get comfortable moving between JSON and Python (json.dumps / json.loads) and between JSON and a DataFrame (pd.read_json). For storage, know when to reach for a JSONB column and how the -> and ->> accessors differ, since that is a compact exam question. Keep the structured vs semi-structured vs unstructured distinction sharp, and the well-formed-XML rules handy. Because the group assignment pulls from a Points-of-Interest API, this pattern is directly useful. Ask Sia to drill you on parsing nested JSON and JSONB queries; confirm assessment details on Canvas.
Working through Web APIs and NoSQL in DATA2001? Sia is AskSia’s AI Data Science tutor — ask any DATA2001 Web APIs and NoSQL question and get a clear, step-by-step explanation grounded in how DATA2001 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
