How AI proctoring actually works (and why most implementations fail)
AI proctoring isn't a model — it's a multi-signal pipeline with a trust problem. The mechanics, three failure modes that kill naive builds, and what works.
by Ankor
A national driver’s-license authority asks a small team to “use AI to catch cheats.” The same week, a university wants to stop students using ChatGPT on take-home finals. A hiring platform needs to know if the candidate on the video call is the one typing.
These look like the same problem. They’re not. And the difference is why most AI proctoring builds ship a demo that impresses the room and a system that fails in the field — usually within a semester.
We built XAM’s proctoring stack for 80+ organisations running everything from K-12 quizzes to competitive test prep. The lesson we keep relearning is that the hard part isn’t the detection — it’s what you do after detection, and what you do for the 99% of candidates who weren’t cheating at all.
What “AI proctoring” actually means
Strip the marketing and an AI proctoring system is a multi-signal pipeline, not a single model. In production it usually looks like this:
┌──────────────┐
Webcam ─┤ Gaze / face ├─┐
Mic ────┤ Voice / audio├─┤
Browser ┤ Focus / tab ├─┼──► Event log ──► LLM summary ──► Human reviewer
OS ─────┤ Process / OS ├─┘
└──────────────┘Each input is processed independently, scored, timestamped, and stored as an event. An LLM (or a rules engine) is then asked to produce a human-readable summary of the session — “Candidate looked away from screen 14 times in 38 minutes, longest 9 seconds; off-screen audio at 22:14 sounds like a second voice; window focus lost twice during the last ten minutes.” A human reviewer reads that summary in under thirty seconds and decides whether to escalate.
That’s the whole shape. The interesting questions are all in the details: which signals you trust, how you score them, what threshold makes a flag, what you show the reviewer, and — most importantly — what you decide the AI is not allowed to decide.
The three failure modes that kill naive implementations
Most failed proctoring builds we’ve audited fall into one of three traps. None of them are model problems. They’re architecture decisions.
1. The false-positive tax
A detection model with 99% recall and 1% false-positive rate sounds great on paper. Run it across ten thousand exams and you’ve produced a hundred false flags a day. Reviewers learn to dismiss flags. Real cheating gets ignored alongside noise. The model isn’t wrong — it’s unusable because the human-in-the-loop has been trained out of trusting it.
This is the same problem we described in why AI agents fail silently: the system looks fine on dashboards while the actual reliability is collapsing because the operators have stopped believing it. With proctoring, the failure shows up in appeal rates. Honest candidates whose flags were overturned stop recommending your platform. Universities quietly switch vendors.
The fix: Treat precision and reviewer throughput as first-class metrics, alongside recall. Every flag that gets overturned by a reviewer is a bug — feed those cases back into your eval set, not your training set, until the precision is high enough that the queue is actually actionable. We target a flag-to-overturn ratio below 5% on XAM. Anything above that, we assume the threshold is wrong before we assume the candidate is.
2. The surveillance backlash
In 2020–2021, the first wave of COVID-era remote proctoring produced predictable headlines: students protesting webcam mandates, privacy regulators opening investigations, lawsuits over biometric data. The pattern has repeated every year since — most recently in the 2025 coverage of national-scale exam integrity systems where the AI itself was sound but the deployment lost the room.
The technical mistake is collecting signals you don’t need, retaining them longer than you should, or sharing them with parties the candidate didn’t consent to. The political mistake is treating privacy as a legal checkbox instead of a design constraint.
The fix: Decide upfront what the AI is allowed to see, store, and share — and build the data retention story into the product, not the legal review. For XAM we settled on: gaze direction and presence detection only (no facial recognition), audio stored as feature vectors not raw recordings, all signals deletable on request within 30 days. None of this is technically interesting. All of it is the reason the platform is still in use at universities that have banned two of our competitors.
3. The audit-trail collapse
This is the one nobody warns you about. When you let an LLM summarise a session, you’re trusting it to compress a multi-hour event log into a paragraph without losing what mattered. A naive prompt — “summarise this session” — produces a confident, fluent summary that quietly omits the one moment the reviewer needed to see. The summary looks professional. The audit trail is now unrecoverable because the human never read the underlying events.
We saw a version of this at a hiring platform last year: the LLM produced a clean summary that didn’t mention a forty-second audio anomaly in the third hour. The reviewer approved. The candidate had been reading answers aloud to a confederate. The platform only learned because the candidate bragged about it on social media.
The fix: Two rules. First, the LLM’s job is summarisation, never verdict — it cannot decide to flag or clear a session, only describe what happened. Second, the summary must be grounded: every claim in the summary links back to a timestamped event in the underlying log. If the reviewer wants to verify, the link is one click away. We wrote about this discipline more broadly in harness engineering — the model is the easy part, the harness around it is what makes the system trustworthy.
The pipeline, in detail
If you’re building this, here’s the shape that has worked for us across XAM and similar systems. Five stages, each with a defined failure mode:
| Stage | What it does | Failure mode if you skip it |
|---|---|---|
| Capture | Webcam, mic, browser, OS events into a normalised stream | You build the rest of the pipeline on unparseable logs |
| Detect | Per-signal models: gaze drift, multi-face, off-screen audio, tab switches, forbidden processes | You have one model doing all of it and can’t isolate failures |
| Score | Combine signals into event objects with confidence and timestamp | You produce raw detections that reviewers can’t triage |
| Summarise | LLM turns the event stream into a short, grounded narrative | Reviewers drown in raw events or trust ungrounded summaries |
| Review | Human invigilator reads summary, drills into events, decides | The whole system becomes theatre |
The key architectural choice is making every stage independently testable. We ship a harness that runs regression evals on each stage: a new gaze model goes in, the eval tells us whether the event-level precision changed. A new summarisation prompt goes in, the eval tells us whether the groundedness score changed. Without that rig, you’re tuning five coupled systems at once and you can’t tell which knob fixed what.
What this looks like in production
A few numbers from XAM that we’re comfortable sharing publicly:
- 80+ organisations running assessments on the platform, from K-12 to competitive hiring.
- 99.17% uptime across exam windows — proctoring fails during exams, so availability is a product feature, not an SLO.
- Reviewer throughput that holds steady under load because the summary is short, grounded, and the events are drillable — not because the model got cleverer.
- Appeal rate that didn’t spike when we tightened detection thresholds, because we tightened them on the eval set, not in production.
None of those are model metrics. They’re all system metrics. That’s the lesson.
When AI proctoring is the wrong call
Worth saying out loud: there are cases where AI proctoring is the wrong answer, and the engineering-honest move is to say so.
- Low-stakes formative assessments. If the exam is for learning, not certification, the surveillance cost isn’t worth the integrity gain. Use better question design and adaptive difficulty instead — both cheaper and more pedagogically sound.
- Populations without reliable hardware. A model that requires 1080p webcams and modern browsers excludes the candidates you usually most want to include. If your candidates are on shared devices or low-bandwidth networks, you’ll measure hardware privilege, not cheating.
- Cases where the human can’t review. If your business model means no human looks at the flags before consequences land — auto-fail, auto-dismiss — you don’t have a proctoring system, you have an automated accusation pipeline. Don’t ship that.
If any of those describe your situation, the right move isn’t a better model. It’s a different assessment design.
What to do next
If you’re building or evaluating AI proctoring:
- Decide what the AI is not allowed to do before you decide what it will do. Verdict, grading, biometric identification — out of scope. Detection, summarisation, surfacing — in scope. The list is short and worth writing down.
- Measure precision, not just recall. A model that flags everything isn’t strict, it’s broken. Track flag-to-overturn ratio as a first-class metric and refuse to ship a model that can’t beat your current threshold.
- Ground every summary in events. If the reviewer can’t click a claim in the summary and land on the underlying evidence, the summary is doing harm. Test this with adversarial sessions — candidates who do nothing wrong — and confirm the summary says so explicitly.
- Treat privacy as a design constraint. Retention windows, biometric scope, and consent flows are part of the product spec, not the legal review. If you can’t explain them in plain language, your candidates will explain them in protest.
- Build the harness before you tune the model. You cannot tell whether a model change made things better without an eval set built from your own failure cases. We covered the why in harness engineering — it’s the same lesson at a different scale.
Proctoring is one of those AI problems where the model is the easy part and everything around the model is the actual product. The systems that work treat it that way from day one. The ones that don’t usually figure it out in the appeal queue.
Ankor builds production-grade AI systems for organisations that can’t afford to demo well and fail in the field — including XAM’s proctoring platform and a range of LLM-integrated products for edtech, hiring, and assessment. If you’re evaluating an AI proctoring build and want a second pair of eyes on the architecture, book a 30-minute call or write to us at team@ankor.us.
