MEDD8001 Chap.8 Experimental Designs and Technology in Learning
Experimental Designs and Technology in Learning
A field whose findings come in matched pairs
The issue is whether technology helps or harms teaching and learning, and the surveyed literature does not deliver a verdict. Adaptive systems are reported to tailor material and comment to the individual learner, so that a wrong idea can be caught early. Willingness to keep working, and satisfaction with the experience, are reported to rise.
Better understanding and better results are reported against conventional teaching, for learners working alone and together.
Against those sit a drop in how much a learner directs their own work, unequal access to devices, privacy, bias, and worry about the tools that pulls willingness down where confidence and a sense of usefulness pull it up.
Those are not contradictions so much as findings about different students, settings and tools, which is exactly the situation in which a design that isolates one manipulated variable becomes valuable.
What makes a study an experiment
Three characteristics, all required.
Control means shutting down every influence outside the one being tested, so that its work cannot be read as the work of the variable you manipulated; the principle behind it is that where two settings differ in nothing else that matters, whatever changes afterwards may be credited to the thing introduced.
Manipulation means an operation is performed on the independent variable, with treatment conditions being its levels.
Observation and measurement means checking after the treatment whether the hypothesised change occurred. Three kinds of group follow, and the middle one matters in schools: an experimental group receives the treatment, a control group receives none, and a comparison group receives a different treatment.
In a school the realistic contrast is almost never nothing, so writing control when you mean comparison overstates what the design isolates.
Four validities and ten threats
Internal validity concerns whether an observed effect reflects a causal relationship. Statistical conclusion validity concerns inferences about covariation between treatment and outcome.
Construct validity concerns the constructs involved in subjects, settings, treatments and observations. External validity concerns whether the relationship holds elsewhere.
Ten threats are named: history, maturation, the testing effect, instrumentation, regression to the middle, unequal groups at the start, uneven drop-out, what the researcher gives away, what the participant comes to believe, and leakage of the treatment sideways.
Grouping them by where they enter turns a list into a review procedure, since selection and regression are present before the study begins, history and diffusion accumulate while it runs, three are introduced by the act of researching, and mortality is only visible at the end.
Six controls, and two that are opposites
Random assignment allocates participants by chance and is the reference response to selection bias.
Randomised matching first matches on relevant variables and then assigns randomly. Homogeneous selection chooses participants as similar as possible on a characteristic. Building variables into the design uses analysis of variance to see the main and interactive effects of a covariate. Statistical control uses analysis of covariance to remove a covariate's effect from the outcome.
Controlling situation differences makes conditions alike except for the treatment.
The fourth and fifth are opposite intentions rather than two versions of one move, and choosing the wrong one answers a different question from the one asked.
One experiment, and the argument against the whole design
The lecture walks a published experiment on laptop multitasking through in full, and most of what makes it interpretable is procedural: a lecture written for the study and checked for accuracy by a second specialist, a treatment pinned down as a count of errands and a share of the session, an outcome measured at two difficulty levels with items intermixed, physically random seat allocation, fidelity monitoring with a stated discard rule, and a blind manipulation check on note quality.
It then closes with a published argument that the design has failed repeatedly in educational inquiry across three historical waves, on the ground that what it controls away, the teaching method, the size of the group, who the teacher is and how they work, where the children come from and the sheer novelty of a thing, is exactly what education most needs to understand.
What this chapter covers
- 01
Matched pairs of findings on learning with technology
- 02
Control, manipulation, observation and measurement
- 03
The law of the single significant variable
- 04
Experimental, control and comparison groups
- 05
Two conditions a causal claim has to meet
- 06
Internal, statistical conclusion, construct and external validity
- 07
Ten threats, grouped by the point in a study where each one arrives
- 08
Six controls, and the two with opposite intentions
- 09
Quasi-experiments and selection bias
- 10
Procedure, fidelity and the manipulation check
- 11
Why a non-significant interaction is a finding
- 12
Three waves of experiment in education, and four objections
Four threats admitted by one convenient design
- 4Name the threats the design admits and say why each one enters here.
- 3Identify the largest and the smallest change that removes it.
- 3Say what to do if that change is impossible.
Key terms
- Comparison Group
- A group receiving a different treatment from the experimental group, as opposed to a control group which receives none, and the realistic contrast in most school settings.
- Random Assignment
- Allocating members of a sample to conditions by a chance procedure, which addresses selection bias including on variables the researcher has not thought of.
- Diffusion
- The threat that the untreated group hears about the treatment from the treated group, so that their results move as well.
- Mortality
- Uneven drop-out of people from one group compared with another during a study, which can create a difference between conditions that owes nothing to the treatment.
- Statistical Control
- Removing a covariate's effect from outcome scores through analysis of covariance, which is the opposite intention from building the variable into the design.
- Quasi-experiment
- A design using non-experimental variation in the independent variable, where exposure is not randomly assigned, and whose main threat is therefore selection bias.
- Fidelity Measure
- A check that participants actually received the treatment as intended, together with a stated rule for what happens to data from those who did not.
Experimental Designs and Technology in Learning FAQ
Can I run an experiment for a Master of Education project?
Sometimes, and the constraint is usually assignment rather than ambition. Schools rarely permit random allocation of students to conditions, and assignment by volunteering teacher or by existing class bundles willingness and teacher quality into the treatment. The honest route when randomisation is unavailable is a quasi-experiment: say so, measure the pre-existing differences between the groups, and adjust for them.
That is a weaker claim made openly, which reads far better than a strong claim the design cannot support.
What is the difference between a control group and a comparison group?
A control group receives no treatment; a comparison group receives a different treatment. In a school the alternative to a new approach is almost always the usual approach rather than no teaching at all, so most educational studies have comparison groups.
It matters because the finding is then about the difference between two approaches rather than about the effect of the new one against nothing, and a reader will infer the second claim from the word control.
If experiments are criticised this heavily, why learn them?
Because the criticisms are about over-reliance and about inference in context rather than about the logic of control.
The published argument the course sets out is that scaling up from a single successful experiment works in engineering and may not in education, that an idea helping some children can be discarded because it did nothing on average, and that the very things a trial strips out, who teaches, how large the group is, where the children come from, are what teachers most need to understand.
Being able to state those objections and still design a clean comparison is the skill; rejecting the design wholesale is not.
Assessment move
Take any intervention study you read and write its four validity questions in the margin, then mark which of the ten threats its design leaves open. Do this before reading the authors' own limitations section, then compare. The threats you found that they did not name are the ones worth writing about, and the ones they named that you missed are the ones you will miss in your own design.
Working through Experimental Designs and Technology in Learning in MEDD8001? Sia is AskSia’s AI Education tutor — ask any MEDD8001 Experimental Designs and Technology in Learning question and get a clear, step-by-step explanation grounded in how MEDD8001 is taught and assessed. Read this chapter free, then take your hardest questions to Sia.
