An IELTS speaking assessment built as a measuring instrument

Why this is the hard problem

Two trained human examiners marking the same IELTS speaking interview agree at a correlation of about 0.90 [1]. That is the ceiling. An automated marker is measured against a standard that people do not hit perfectly themselves, so the useful question is not whether it is right every time. It is whether you can see how it reached a band, and catch it when it is wrong.

The regulatory floor moved while this was being built. ETS's 2026 update to the TOEFL marks all eleven speaking items on the test as AI scored [2]. Ofqual fined Cambridge English £875,000 over automated-marking errors that ran undetected for more than two years [3]. Under the EU AI Act, a system that evaluates learning outcomes is high-risk by classification rather than by anyone's opinion of it [4].

So the design problem was never how to get a model to output a band. It was how to build something an examiner, a regulator and a candidate's appeal can all read.

The five layers

The system listens to the whole interview rather than sampling it, and each layer produces evidence the next one can check. The AI never does the arithmetic.

1 · Capture

Transcription with word-level timestamps, hesitations preserved rather than tidied away. A fluency judgement that cannot see where somebody paused is a guess.

2 · Acoustic measurement

Speech rate, pause length and position, where hesitation falls, pitch range: measured deterministically in signal-processing code. These are the feature families ETS SpeechRater has used for over a decade [5], and they are arithmetic rather than opinion, so the same audio gives the same numbers every time.

3 · Pronunciation

A dedicated phoneme-level model on the Goodness-of-Pronunciation method, trained against a corpus scored by five expert raters. General-purpose multimodal models were tested for this job and were not good enough at phoneme and stress judgement, at 0.21 against 0.61 to 0.74 for specialist models [6].

4 · The examiner model

One pass over the whole interview with every measured value in front of it, applying the official band descriptors. It runs three times, the median is taken, and disagreement between the runs routes the case to a human without anyone asking. It sits behind a swappable interface, on OpenAI audio models with a Gemini fallback, so the instrument does not depend on one vendor staying still.

5 · Scoring and guardrails

The four criteria are averaged, rounded by a documented rule and capped where an answer is off topic, all of it in plain code. Integrity checks for memorised answers, read-aloud delivery and synthetic voice raise a review flag; none of them silently alters a score.

Confidence routing, and the number the design is built around

Cambridge publishes the figures that make the case for this shape. Its Linguaskill automarker awarded the same CEFR grade as the examiners on 56.8% of tests marking alone, and on 95.6% under the hybrid model, where a response the computer is not confident about goes to a human [7]. Escalation is what moves that number, not a better model.

So the instrument is built around knowing when to stop. Roughly 15% of interviews route to a human reviewer, and a further 5% are audited at random whether the system was confident or not, because a gate tested only on the cases it flagged tells you nothing about the ones it let through. Every reviewed case comes back as labelled data for the next version.

Fairness and accuracy are release gates

These are gates, not results. Stating them the other way round is the failure this whole design exists to avoid.

Accent fairness

The pronunciation layer's output is evidence, never the score. Score differences by candidate first language are tested as a formal gate, within 0.10 standard deviations, and a build that fails it does not ship.

Agreement with examiners

Quadratic-weighted agreement of at least 0.70 with human examiners, the threshold Williamson, Xi and Breyer set out for automated scoring [8].

The validation is scheduled

The corpus is 300 interviews, each marked by two certificated examiners with a third resolving disagreements. It is booked rather than finished, so this page publishes the gates and no number against them. An accuracy figure without its validation behind it is the first thing a regulator would take apart.

Where the build stands

The pipeline is complete and has been independently verified. Around 1,170 automated tests cover the speaking pipeline, among them a regression suite where disabling any one guardrail breaks specific frozen cases, so a guardrail cannot be quietly dropped without a test going red. The core mathematics was hand-verified against worked examples rather than only against itself.

Seven regulator-facing governance documents are drafted, and the rollout is staged behind controls rather than switched on.

How it started

The brief was a frontend. Building it meant living inside the product, and the mock test made the ceiling clear: it could tell a learner whether an answer was right, but not why, and not in a language the learner was comfortable being taught in. We built an AI tutor that teaches English in the learner's own language, and the engagement widened from there into the question the CEO actually had, which was where AI belonged across the company and in what order.

The estate underneath

The engagement is now end-to-end technical ownership of the live production estate: the exam platform, the APIs and the data. On 21 August 2026 the production database held 3,141 live student records, 1,086 of them added in the previous ninety days, and 393 mock assessments completed since November 2025. Those are counts from the system, not a projection.

The database was replatformed into Google Cloud's Doha region to meet Gulf data-residency requirements, and the automated test estate across the platform went from around 300 tests to more than 1,400.