Who gets missed? · OULAD
Early-warning models · Open University data

Who gets
missed?

A model learns from one year of students and is tested on the next year's, people it has never seen. Among students over 35 who went on to fail or withdraw, it missed

31% vs 23% under 35

at a threshold chosen on the training data, after two weeks of a course. A better threshold closed the gender gap. It did not close this one.

Accuracy on next year's students
–
AUROC after 14 days, gradient-boosted trees
What a random split overstates
–
AUROC, 95% interval
Two more weeks of data
–
AUROC, same students
01 · THE QUESTION

Early-warning systems are tested on the wrong students

Universities use click data from the first weeks of a course to flag students who might fail. Most published models are tested on a random split: the test students come from the same run of the same course as the training students. A real system has to work next year, on new people. So I asked two things.

1

Transfer

How well do the first 2 or 4 weeks of activity predict who passes when the model is trained on 2013 and tested on 2014, and how much lower is that than a random split?

2

Fairness

Is the model equally good for every group of students: gender, age, the deprivation level of where they live, and disability?

Committed before any data. The questions, splits, models, metrics and hypotheses were written down and committed to Git before the dataset was downloaded. Every later change is in the deviation log, with its reason.

02 · METHOD

Two ways to test the same model

The data is OULAD: seven modules at the Open University, each run up to four times in 2013 and 2014. Six modules ran in both years. The difference between the two tests is who ends up in the test set.

Who is trained on, and who is tested
Enrolment in a 2013 run Enrolment in a 2014 run Student already seen in 2013, removed from the test

From seven tables to one prediction

Everything the model sees is dated before the cutoff day. Hover a box for details.

Only what is known on the cutoff day

Students who had already left by the cutoff are removed: there is nothing left to predict for them. The outcome is pass or distinction against fail or withdrawal.

A course run, from the model's point of view

Four models, settings fixed in advance

Logistic regression

Standardised features, L2 penalty, C = 1. The simple baseline.

Gradient-boosted trees primary

scikit-learn defaults. Handles missing values and non-linear effects.

GRU sequence model

Reads clicks day by day and by activity type, plus the static features. Five seeds.

Boosted trees + demographics

Also sees gender, age, deprivation, disability, region and education. Reported separately.

03 · RESULTS

The random split overstates accuracy, but only a little

Every model loses about 0.01 to 0.02 AUROC when tested on next year's students. The drop is real (its 95% interval stays above zero for every model), but small. The bigger story is how weak early prediction is: no model reaches 0.75.

AUROC by model: random split vs later run
Dots are point estimates; whiskers show the 95% interval (later run) and the range over 5 seeds (random split).
Random split (mean of 5)Later run (95% interval)
Show as a table

What moves accuracy, and by how much

Changes in later-run AUROC at day 14, except the first, which compares day 28 with day 14 on the same students. Waiting two more weeks is worth more than any model change.

Raises AUROCLowers AUROC
04 · GROUPS

The same model is less accurate for some students

AUROC within each group, on the later run. The model ranks women less accurately than men at both cutoffs. Older students are ranked about as well as younger ones, but more of them are missed, as the next chart shows.

Later-run AUROC by group
The hairline marks the model's AUROC on all test students. Gaps are largest minus smallest, with 95% intervals.

Gap intervals for deprivation (six levels) are biased upward: the largest-minus-smallest gap grows with noise. Treat them as rough.

Who gets missed, before and after choosing the threshold properly

The plan fixed the flagging threshold at 0.5 in advance. That missed about half of at-risk students. A threshold chosen on the training data (to flag 80% of at-risk training students) was added afterwards, and is labelled as a post hoc analysis.

Share of at-risk students the model misses
Gradient-boosted trees, later run. Lines that converge mean the gap closed.
Women / under 35Men / 35 and over

05 · TRY IT

Move the threshold yourself

Every test student gets a predicted chance of passing. Anyone below the threshold is flagged as at risk. Drag the line: flag more students and fewer at-risk ones are missed, but more students who would have passed get flagged too. Watch the two age groups: the gap between them never closes.

Predicted chance of passing, day 14, later run
Students who passed above the line, students who failed or withdrew below. Shaded: flagged.
PassedFailed or withdrew
Students flagged
–
At-risk missed
–
Missed, under 35
–
Missed, 35+
–
Passing students flagged
–

Values at the slider's 0.01 steps; the threshold chosen on training data was .

06 · LIMITS

What this does not show

One institution, one year apart

The Open University in 2013 and 2014, the same modules. A different university or a longer gap could drop much more.

Biased gap intervals

A gap is the largest minus the smallest group, so noise always inflates it. For six deprivation levels the point estimate even falls below its interval.

The threshold analysis is post hoc

It was added after seeing the 0.5 results. The 0.5 results remain the planned, primary report.

Withdrawal counts as at risk

Some students withdraw for reasons a click log cannot show: work, illness, family.

Tutor-marked scores unused

OULAD does not record when a tutor-marked assessment was graded, so its score may not have been known on the cutoff day.

Data from a mirror

The Open University link now returns 404; the data came from the UCI repository copy, checksummed.

07 · REPRODUCE

Run it yourself

  1. Clone the repository and install the pinned requirements (PyTorch from the CPU index).
  2. Download OULAD, put the seven CSV files in data/raw/, and check them against data/SHA256SUMS.txt.
  3. Run the four steps below from src/. The full run takes about 15 minutes on a laptop CPU.
python -m oulad.features    # feature tables for days 14 and 28
python -m oulad.run         # all models, splits and bootstrap intervals
python -m oulad.report      # results/RESULTS.md and figures
python -m oulad.threshold   # post hoc threshold analysis

Cite the data

Kuzilek, J., Hlosta, M. and Zdrahal, Z. (2017). Open University Learning Analytics dataset. Scientific Data, 4, 170171. CC BY 4.0.