ECON7880 Chap.2 Python Notebooks and Reproducible Data Work
Python Notebooks and Reproducible Data Work
Make state visible
Jupyter supports an execute-explore workflow, but hidden execution order can make a notebook appear complete when it cannot run from the top.
This chapter treats restart-and-run as a validity check and follows data from an unchanged raw file through typed, checked transformations.
Fit preparation inside training data
Imputation, scaling and encoding can leak holdout information if they are learned before splitting.
The worked example repairs that order and uses assertions to protect keys, target values, partition overlap and predictor alignment.
Preserve raw inputs and explicit types
Keep the original file unchanged and make transformations in code rather than editing cells manually. Inspect column names, row count, missingness, duplicate keys and target balance before cleaning.
A postcode may contain digits without being a quantity, while a numeric field may arrive as text because one record contains a symbol.
Explicit conversion and category mappings prevent silent comparisons between incompatible representations.
Turn exploration into a deterministic route
Interactive exploration is useful for discovering what needs attention, but the final notebook should express one reproducible path. Consolidate successful cleaning steps, remove abandoned branches and print compact checkpoints after major changes.
A restart followed by run-all must rebuild every table and figure from the raw input. If a cell depends on a variable created later, the current output reflects hidden memory rather than the documented sequence.
Keep learned transformations inside validation
Means for imputation, scaling parameters, selected features and encoded categories are fitted objects.
Learn them only from the current training partition, then apply them unchanged to validation or test rows. When cross-validation is used, the preparation must be refitted separately inside each fold.
A pipeline object is valuable because it preserves this order automatically and travels with the final estimator.
Use invariants to catch plausible mistakes
Assertions can require unique identifiers, legal target labels, non-overlapping partitions and identical predictor columns after encoding. These checks fail near the source of a defect instead of allowing a plausible model score to conceal it.
Record the input filename, seed and package imports, but exclude credentials, personal paths and student identifiers. The aim is a notebook another analyst can run, inspect and challenge.
What this chapter covers
- 01
Kernel state and execution order
- 02
Raw-file preservation
- 03
Explicit data types
- 04
Train-only preparation
- 05
Random seeds and repeatability
- 06
Assertions and data invariants
Worked example · free
Move preparation behind the split
- 2Identify the two leakages.
- 3Reorder the pipeline.
- 2Add an invariant check.
Key terms
- Hidden State
- Values retained in memory from an execution order that the visible notebook does not reproduce.
- Deterministic Pipeline
- A saved transformation sequence that produces the same result from the same inputs.
- Data Invariant
- A condition such as unique keys or legal target values that must remain true throughout processing.
Python Notebooks and Reproducible Data Work FAQ
Why restart a notebook kernel?
Restarting removes hidden variables and proves that the visible cell order contains every dependency. A notebook that only works in the author’s current memory state is not reproducible.
Why split before scaling?
Scaling parameters learned from all rows contain information about the test distribution. Fit them on training rows only, then apply the stored transformation to the holdout without learning anything new from it.
Which checks belong in code?
Assert unique keys where required, legal target values, no partition overlap and consistent predictor columns. Print compact shape and missingness summaries after major transformations, and fail early when a replacement file violates the documented schema or category mapping. Run these checks before model fitting begins and again after any new data version arrives.
Exam move
Audit one notebook by tracing the raw load, target creation, split, fitted preparation and metric in order. Restart the kernel, run all cells and explain every change in row count or column type. Add assertions for key uniqueness, target legality and partition overlap. Then deliberately move scaling before the split and explain why the resulting workflow is contaminated even if the score improves.
Finish by rebuilding the safe pipeline and documenting the seed, input file and feature cutoff so the analysis can be repeated on another machine. Practise diagnosing three plausible outputs: a missing category in test data, a duplicated identifier and a sudden change in row count after a join. For each, write the assertion or summary that would expose it earliest.
Save one clean notebook version with no manual edits to the raw table and verify that another folder location does not break its paths. Compare final predictor columns with the data dictionary, and explain how unseen categories, missing fields and a replacement input file will be handled safely.
Working through Python Notebooks and Reproducible Data Work in ECON7880? Sia is AskSia’s AI Economics tutor — ask any ECON7880 Python Notebooks and Reproducible Data Work question and get a clear, step-by-step explanation grounded in how ECON7880 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
