INT6067 Chap.7 Quantitative Research Methods and Analysis
Quantitative Research Methods and Analysis
Design first, analysis second
The course gives quantitative method two taught sessions, a practical session with statistical software and a further session on structural equation modelling. That sequence is the shape of the work: decide a design, collect with an instrument, then choose an analysis the design has already made available.
Choosing the analysis first is how proposals end up promising a model they have no data for.
Three decisions come before any number is collected. The first is the unit that carries the outcome, since allocating whole classes and then analysing individuals treats thirty learners as thirty independent observations when they share one teacher and one room.
The second is when the outcome is measured, because a single measurement after the fact cannot show change. The third is what the comparison is, and in a classroom the honest answer is usually ordinary teaching rather than nothing at all.
Realistic scale in this field
Published experiments in educational technology commonly run with twenty to thirty learners per condition rather than hundreds.
Studies in this course's readings include fifty-nine children across three robot roles, ninety children in a randomised story reading comparison, and thirty-one students in an evaluation of automatically generated quizzes.
Surveys reach larger numbers because the cost per response is small, and one survey of writing tool use reached sixty-eight educators and one hundred and fifty-eight students by asking about six writing tasks rather than by running anything.
Ambition in a proposal is measured by what the design can rule out, not by the size of the sample.
The design ladder
Five designs appear in the readings and each rules out something more than the one below it. A single group measured only afterwards rules out nothing. The same group measured before and after removes the cohort problem and leaves maturation and ordinary teaching.
Two groups chosen by a teacher removes timing and paper differences and leaves selection. Two groups allocated by a stated rule removes selection if the rule is stated. Random allocation removes it on average, including on characteristics nobody measured. Each rung costs a concrete thing: a second measurement, a second class, a teacher willing to accept an allocation rule.
Strength is bought with access, never with a more elaborate statistic.
Four analysis families, and what a result does not mean
Description answers how much and how often. Association answers whether two things move together. Difference answers whether groups or time points differ more than noise.
Modelling answers whether an outcome can be predicted or a structure fitted, and it includes the classification methods used on learning logs and the structural models introduced late in the course. Two errors recur. A statistically reliable difference is not necessarily a large one.
And in a study with twenty learners per group, failing to find a difference is the expected outcome even when a real one exists, so a null result is not evidence of equivalence.
What this chapter covers
- 01
Deciding the unit that carries the outcome
- 02
When the outcome is measured, and why one measurement cannot show change
- 03
What the comparison actually is in a classroom
- 04
Realistic participant numbers in published studies in this field
- 05
Five comparison designs ranked by what each rules out
- 06
Quasi-experiments and the duty they transfer to the argument
- 07
Reliability against validity, and the case for borrowing an instrument
- 08
Description, association, difference and modelling
- 09
Why a null result in a small study is uninformative
One weak comparison, upgraded twice, with each upgrade priced
- 3Name what is wrong with the historical comparison.
- 3Give the cheap upgrade and the claim it supports.
- 3Give the expensive upgrade and what it costs in access.
Key terms
- Allocation Rule
- The stated procedure by which participants reached their condition. A rule a reader can inspect is what separates a defensible comparison from one that depends on who volunteered.
- Historical Comparison
- A comparison against a previous cohort rather than a concurrent group. It leaves differences in paper, term and teacher as candidate explanations for any result.
- Reliability
- Whether an instrument produces consistent answers on repeated use. It says nothing about whether the instrument measures the construct it claims.
- Validity
- Whether an instrument measures the thing it names. A scale that reliably measures how much learners like the researcher is reliable and invalid.
- Base Measurement
- A measurement taken before an intervention, used so that change can be reported rather than level. It is the cheapest single upgrade available to most classroom designs.
- Classification Method
- An analysis that assigns cases to categories from earlier records, used on learning logs to identify students who may need support. It supports prediction, not a claim about cause.
- Null Result
- A comparison that found no reliable difference. In a small study it is the expected outcome even when a real difference exists, so it is not evidence that two conditions are equivalent.
- Latent Construct
- Something not measured directly but inferred from several observed items, such as motivation. Models that relate several of these at once need considerably more respondents than a group comparison.
Quantitative Research Methods and Analysis FAQ
How many participants does my study need?
The honest answer in a proposal is that the number follows from the comparison rather than from a rule of thumb, and that published work in this field commonly runs with twenty to thirty learners per condition.
What a marker looks for is consistency: a design promising to detect a small difference with two classes has promised something arithmetic will not deliver, and a design that names its limits explicitly reads as more competent than one that claims a sample it cannot recruit.
Can I use a questionnaire I designed myself?
You can, and a marker will ask two questions about it: whether it measures what you named rather than something adjacent, and whether it gives stable answers. Neither can be answered for an instrument used once and never examined.
Adopting a published scale used in comparable studies settles both questions with a citation, fixes the number of items and the response format, and lets you compare your results with theirs in the discussion.
My school will not allow random allocation. Is my study still worth doing?
Yes, and most classroom studies in this field are in the same position. Comparing existing classes makes the study a quasi-experiment, which transfers work from the design to the argument: report something about both groups before the intervention to show they were comparable, and name the differences you could not measure.
One study in the course readings runs two field quasi-experiments and adds a focus group afterwards, which is a workable template.
What can learning log data actually establish?
Association and prediction. Records collected during a term and matched to a later grade can show which features move with attainment and how well various methods predict it, which is why studies of this kind report correlation alongside classification performance.
What such a design cannot establish is difference or cause, because nothing was varied and no group was compared, so a feature that predicts failure is a flag for support rather than a lever that raises attainment.
Assessment move
Take any study you have already read and redraw it one rung up and one rung down the design ladder, writing what each version would let the authors claim. Doing that three or four times is what makes the ladder usable under pressure, and it is directly the judgement the group critique asks for.
Then apply it to your own proposal and price the upgrade you cannot afford, because naming it is worth more than pretending the design is stronger than it is.
Working through Quantitative Research Methods and Analysis in INT6067? Sia is AskSia’s AI Education tutor — ask any INT6067 Quantitative Research Methods and Analysis question and get a clear, step-by-step explanation grounded in how INT6067 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
