Doc 2 · owner: Rebekah · 2026-08-06 · companion to The Plan

The
Measure

Purpose: produce credible proof for investors (and the launch story) that 3 weeks of Kelas Sekejap improves spoken English. Three instruments that must converge: a blind-rated speaking task, objective transcript metrics, and self-report. We measure change within each student, not absolute level — so the same instrument works for Form 1 and Form 5.

1 · The Speaking Check (in-app, pre + post)

Delivered by the app, not by a person (decided 2026-08-06 — human-run assessments across four sites were a logistical nightmare and don't scale). It's a ~5-minute activity inside the app: the pre runs as the student's very first activity — Day 1 stays locked until it's done, which guarantees a true baseline and enforces "joining = doing both" — and the post unlocks after Day 21 as the final activity.

Flow per round (~5 min, identical pre and post)

1. Intro screen (B1 copy): "This is not an exam. There are no wrong answers. Find a quiet place. Speak in English and try to keep talking until the timer ends."
2. Confidence check (one tap): "How confident do you feel speaking English?" — 1–10 scale. Asked identically pre and post; this is the before/after confidence measure, captured in-app so collection is 100%.
3. Warm-up (not recorded): one easy question ("What did you do last weekend?"), ~30 seconds — settles nerves so Task 1 isn't cold.
4. Task 1 — Describe (2 min, recorded): personal-descriptive prompt with 3 guide points on screen, countdown timer visible.
5. Task 2 — Opinion (1 min, recorded): simple agree/disagree question.
6. Confirmation screen: upload confirmed ("We got it — thank you!"). A failed upload retries until confirmed and flags the account for follow-up.

Rules that protect the measure

• Single take. No re-record, no listening back before submit — same as real speech. (Only a failed upload retries; a completed recording is never redone.)
• A/B auto-assigned by the app and stored with the recording — counterbalancing needs no admin work.
• Audio + transcript land through the existing recorder → R2 → transcription pipeline, tagged with round (pre/post) + version (A/B) — no phone voice memos, no file wrangling.
• Timing gates: pre before Day 1 unlocks; post only after Day 21.

Task versions (counterbalanced)

Half the students get A pre / B post; the other half B pre / A post — assigned automatically. It kills the "they just saw the question before" objection and cancels out one version being slightly easier.

2 · Rubric (three transcript-gaugeable criteria)

Three criteria, 1–5 each, total /15. Pronunciation and delivery were deliberately dropped: they can only be judged from audio, and the scoring runs on transcripts so a model can score every recording consistently. Confidence still shows up in the self-report and the objective metrics; pronunciation is out of scope for this study.

Scoring protocol — model-scored, human-validated

1. Pull the transcripts and rename each to a random code (R014); the code→student/round key lives in one spreadsheet only the admin (Rebekah) sees. Strip anything that reveals the round.
2. Claude scores every transcript against the rubric — one batch run, fixed JSON output schema (score + one-line justification per criterion). Blind by construction: the model is never told whether a transcript is pre or post. Perfectly consistent, rerunnable, no rater fatigue. Cost for the whole study: under RM5.
3. Rebekah blind-rates a validation sample — ~20 recordings (audio + transcript), same three criteria, shuffled, round unknown. About 1.5 hours.
4. Report model–human agreement on the sample (% of criterion scores within 1 point) — the line that pre-empts the "AI graded its own product" objection.

3 · Objective transcript metrics

• Speaking Check recordings: transcripts arrive automatically through the app's own pipeline. Compute per student: total words in Task 1, words per minute, longest uninterrupted stretch, distinct-word count (vocabulary diversity).
• In-app trajectory (continuous, free): the backend stores a transcript for every attempt. Compare each student's week-1 vs week-3 attempts on words per attempt and WPM — 21 days of data, not just two points; the growth curve is often more persuasive than the endpoint comparison.

4 · Self-reported confidence (in-app, pre + post)

We gauge the students' own confidence at the start and at the end. The question lives inside the Speaking Check — one screen, one tap — so collection is 100%:

• "How confident do you feel speaking English?" — 1–10 scale, identical wording in the pre check and the post check.
• The behaviour questions (answered a question in English class · spoke English outside class · avoided speaking English) move to the exit interview — they need a conversation, not a tap.

Secondary evidence only, but "average confidence went from 4 to 7" is a line investors remember.

5 · What investors see

Logistics & cost

• Nothing to administer — the app runs both rounds. Kickoff sessions are onboarding only: Seremban — Eugene (~1 hour, solo); Catholic High + SMK Putrajaya — Rebekah in person; Melaka — video call. Students do the pre Speaking Check that evening at home; Day 1 unlocks after it.
• Human time that remains: ~1.5 hours blind-rating the validation sample (Rebekah); chasing stragglers; exit interviews (those stay human).

The build

The one engineering deliverable — ships before kickoff, ~Aug 16

A new in-app activity, deliberately thin — it reuses the existing recorder → R2 upload → transcription → attempts machinery:

• Six screens per the flow above (intro / confidence 1–10 / warm-up / task 1 / task 2 / confirm), B1 copy, countdown timers, no re-record.
• A/B version auto-assignment + the 1–10 confidence answer, stored with the submission.
• Submissions tagged with round (pre/post) + version — recordings and transcripts queryable without file wrangling.
• Gates: pre required before Day 1; post unlocks after Day 21.
• Feature-flagged for focus-group accounts (same pattern as the listening gate), so it doesn't leak before launch.
• End-to-end test on a real phone: record → upload confirmed → transcript lands.

Checklist