CDT502 Chap.6 Reproducible Decision Studies
Reproducible Decision Studies
This chapter turns the group project's marking points into an analytical workflow. A bounded decision question drives data auditing, feature construction, time-aware validation, baseline selection, common-fold model comparison and segmented error analysis.
Reproducibility is treated as the ability to regenerate every table and figure, while predictive importance is kept separate from causal effect and model ranking is tied to operational loss. A focused question identifies the outcome, unit, prediction or decision horizon and intended action. It also states what lies outside scope. This prevents a project from drifting toward whatever pattern looks interesting after exploration.
The dataset is then judged against the question: coverage, fields, time order, missingness and operating states should be documented before modelling begins. Feature construction must respect information availability. A transformation learned from all records leaks holdout information into training, even when the final estimator never sees the outcome column directly.
Fit scaling, imputation and encoding inside each training fold. For a future-facing problem, preserve chronology through expanding or rolling validation and reserve a later block for one final evaluation. Baselines make improvement interpretable. Compare a transparent current rule and candidate models on identical folds, outcomes and features available at prediction time.
Report metrics in the outcome unit and inspect the segments where an error changes action. A small average gain can be unacceptable if performance deteriorates during the costly peak, vulnerable group or critical operating state. Reproducibility completes the argument. Deliver inputs, code, environment instructions, random settings, fold boundaries and generated outputs so another analyst can recreate every table and chart.
Predictive association, coefficient size and feature importance do not establish causal effects. Translate the result into a trial, monitoring rule and bounded implication that the study design actually supports.
What this chapter covers
- 01
Decision questions and scope
- 02
Data audits and preprocessing
- 03
Time-aware validation
- 04
Transparent baselines
- 05
Error metrics and segmentation
- 06
Reproducibility and decision translation
Worked example · free
Prefer the model that wins where error costs most
- 3Calculate overall gains against the baseline.
- 2Compare candidate errors during the costly segment.
- 3Recommend a bounded operational trial.
Key terms
- Expanding Window
- A time-aware validation scheme that trains on an increasing past period and validates on a later block.
- Holdout Set
- A reserved dataset used once for final evaluation after model selection is complete.
- Mean Absolute Error
- The average absolute distance between predictions and observed outcomes in the outcome's unit.
Reproducible Decision Studies FAQ
Why is a random split risky for time-indexed data?
It can place later observations in training and earlier ones in testing, allowing future seasonal or policy information to influence a model evaluated as though the future were unknown. Use expanding or rolling windows and preserve a final later block so evaluation matches deployment direction.
Why does the project need a baseline model?
A baseline embodies the current rule or a transparent pattern and gives candidate performance operational meaning. Compare every model on identical folds, outcomes and available features. A complex model that cannot beat a suitable baseline, or loses in the costly segment, has not justified deployment.
How do I keep predictive results from becoming causal claims?
Use verbs such as predicts, is associated with or adds held-out information. Feature importance and coefficients describe a fitted predictive relationship under the dataset and model. Recommending an intervention requires a design that identifies what would happen if the feature were changed while other relevant conditions were controlled.
Exam move
Build one small project from clean inputs to final report. Freeze the question and holdout before modelling, fit every transformation inside training folds, and compare a simple baseline with candidates on identical periods. Reproduce the complete report in a fresh environment, then audit each conclusion for prediction-to-causation drift.
Keep a decision log that records why each exclusion, feature, fold boundary and metric was chosen before final evaluation. Compare model rankings overall and in the operational segment with the largest loss, and write what would make you retain the baseline. Break the pipeline deliberately by fitting a transformation on all data, then identify exactly where holdout information entered.
Prepare a one-minute explanation of the validation design without model jargon. Finally, hand the project to a classmate with only the delivered instructions and record every undocumented dependency they encounter; those failures identify the reproducibility work still required.
Working through Reproducible Decision Studies in CDT502? Sia is AskSia’s AI Statistics tutor — ask any CDT502 Reproducible Decision Studies question and get a clear, step-by-step explanation grounded in how CDT502 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
