Who gets
missed?
A model learns from one year of students and is tested on the next year's, people it has never seen. Among students over 35 who went on to fail or withdraw, it missed
at a threshold chosen on the training data, after two weeks of a course. A better threshold closed the gender gap. It did not close this one.
Early-warning systems are tested on the wrong students
Universities use click data from the first weeks of a course to flag students who might fail. Most published models are tested on a random split: the test students come from the same run of the same course as the training students. A real system has to work next year, on new people. So I asked two things.
Transfer
How well do the first 2 or 4 weeks of activity predict who passes when the model is trained on 2013 and tested on 2014, and how much lower is that than a random split?
Fairness
Is the model equally good for every group of students: gender, age, the deprivation level of where they live, and disability?
Committed before any data. The questions, splits, models, metrics and hypotheses were written down and committed to Git before the dataset was downloaded. Every later change is in the deviation log, with its reason.
Two ways to test the same model
The data is OULAD: seven modules at the Open University, each run up to four times in 2013 and 2014. Six modules ran in both years. The difference between the two tests is who ends up in the test set.
From seven tables to one prediction
Everything the model sees is dated before the cutoff day. Hover a box for details.
Only what is known on the cutoff day
Students who had already left by the cutoff are removed: there is nothing left to predict for them. The outcome is pass or distinction against fail or withdrawal.
Four models, settings fixed in advance
Logistic regression
Standardised features, L2 penalty, C = 1. The simple baseline.
Gradient-boosted trees primary
scikit-learn defaults. Handles missing values and non-linear effects.
GRU sequence model
Reads clicks day by day and by activity type, plus the static features. Five seeds.
Boosted trees + demographics
Also sees gender, age, deprivation, disability, region and education. Reported separately.
The random split overstates accuracy, but only a little
Every model loses about 0.01 to 0.02 AUROC when tested on next year's students. The drop is real (its 95% interval stays above zero for every model), but small. The bigger story is how weak early prediction is: no model reaches 0.75.
Show as a table
What moves accuracy, and by how much
Changes in later-run AUROC at day 14, except the first, which compares day 28 with day 14 on the same students. Waiting two more weeks is worth more than any model change.
The same model is less accurate for some students
AUROC within each group, on the later run. The model ranks women less accurately than men at both cutoffs. Older students are ranked about as well as younger ones, but more of them are missed, as the next chart shows.
Gap intervals for deprivation (six levels) are biased upward: the largest-minus-smallest gap grows with noise. Treat them as rough.
Who gets missed, before and after choosing the threshold properly
The plan fixed the flagging threshold at 0.5 in advance. That missed about half of at-risk students. A threshold chosen on the training data (to flag 80% of at-risk training students) was added afterwards, and is labelled as a post hoc analysis.
Move the threshold yourself
Every test student gets a predicted chance of passing. Anyone below the threshold is flagged as at risk. Drag the line: flag more students and fewer at-risk ones are missed, but more students who would have passed get flagged too. Watch the two age groups: the gap between them never closes.
Values at the slider's 0.01 steps; the threshold chosen on training data was .
What this does not show
One institution, one year apart
The Open University in 2013 and 2014, the same modules. A different university or a longer gap could drop much more.
Biased gap intervals
A gap is the largest minus the smallest group, so noise always inflates it. For six deprivation levels the point estimate even falls below its interval.
The threshold analysis is post hoc
It was added after seeing the 0.5 results. The 0.5 results remain the planned, primary report.
Withdrawal counts as at risk
Some students withdraw for reasons a click log cannot show: work, illness, family.
Tutor-marked scores unused
OULAD does not record when a tutor-marked assessment was graded, so its score may not have been known on the cutoff day.
Data from a mirror
The Open University link now returns 404; the data came from the UCI repository copy, checksummed.
Run it yourself
- Clone the repository and install the pinned requirements (PyTorch from the CPU index).
- Download OULAD, put the seven CSV files in
data/raw/, and check them againstdata/SHA256SUMS.txt. - Run the four steps below from
src/. The full run takes about 15 minutes on a laptop CPU.
python -m oulad.features # feature tables for days 14 and 28
python -m oulad.run # all models, splits and bootstrap intervals
python -m oulad.report # results/RESULTS.md and figures
python -m oulad.threshold # post hoc threshold analysis
Cite the data
Kuzilek, J., Hlosta, M. and Zdrahal, Z. (2017). Open University Learning Analytics dataset. Scientific Data, 4, 170171. CC BY 4.0.