Right or Wrong Is a Thin Signal. Here Is What to Track Instead
Next-gen learning apps look at how a kid reached an answer, not just whether it was right. What to track, why it matters, and what it costs.
When my middle schooler gets a math problem wrong, the "wrong" is almost the least interesting part. What I want to know is where it went wrong. Was it the first step or the last? Did they misunderstand the idea, forget a step, or just copy a number down badly? Did they fix it on the second try? Could they explain it afterwards? A red X answers none of those questions. For years, most learning software has been built around that red X, and after time spent building a learning app, I am convinced it is one of the least useful things we could be measuring.
The scoreboard is not the game tape
In sports, the scoreboard tells you who won. The game tape tells you why. No serious coach prepares for next week by staring at the final score. They watch the tape: the missed rotation, the late pass, the moment the defense lost its shape.
Right or wrong is the scoreboard. It is a thin signal, a single bit of information about a process that may have involved a dozen small decisions. Two kids can get the same problem wrong for completely different reasons. Two kids can get it right with very different levels of understanding. One knows it cold. The other guessed, got lucky, or followed a hint step by step without really following the logic.
If all you collect is the final answer, every one of those situations collapses into the same data point. Whatever you practice next is then based on a guess.
What process evidence looks like
Process evidence is the game tape. Here are the pieces I have found most useful.
The line where it went off course
When a kid shows their work, you can find the exact step where things went sideways. Everything up to line three is solid. Line four is where the negative sign vanished. That is far more actionable than "wrong," because it tells you the kid already knows most of the procedure. You do not need to reteach the whole thing, just one step.
The kind of mistake
Not all mistakes are equal. I find it useful to sort them into three rough buckets:
- Concept: the kid has the wrong idea. Adding the denominators when adding fractions is a classic. More practice of the same kind will just reinforce it.
- Procedure: the idea is right, but a step is shaky or out of order. This responds well to a worked example and a few focused repetitions.
- Careless: they know it, they rushed. Copying a 6 as a 0. The fix here is habits, like checking work, not more teaching.
Teachers do this sorting by instinct. Software should too, and it should be humble about it, because the line between careless and conceptual is not always obvious from a single attempt.
Hints used
A correct answer reached after three hints is a different thing from a correct answer reached alone. Neither is bad. Hints are scaffolding, and scaffolding is how learning works. But recording which hints were used, and how far into the problem, tells you how much of the structure the kid is carrying on their own.
The second try
What a kid does right after a mistake might be the most revealing moment of all. Do they spot the error and fix it? Start over from scratch? Change the answer at random until something turns green? A thoughtful second try is a sign that understanding is forming. Random guessing is a sign the kid is lost, or just tired.
Explaining it back
Asking a kid to explain an answer in their own words is a powerful check, and a powerful way to learn, which is a nice bonus. An explanation shows whether the understanding is real or borrowed. "You flip the second fraction because dividing by a half is the same as asking how many halves fit" is understanding. "Because that's the rule" is memory.
Doing it with help switched off
Finally, the real test: can they do a similar problem later, with no hints and no worked example on screen? This is where formative assessment meets retrieval practice. Pulling something out of memory on your own, after some time has passed, is both strong evidence of learning and one of the best ways to make learning stick.
Why this changes what you practice next
Here is the payoff. Once you know how a kid got something wrong, the next step becomes much clearer, and it is different for each kind of mistake.
A concept error calls for a different explanation or a picture, not more of the same problems. A procedure error calls for a worked example and a few targeted reps on that one step. Careless errors call for slowing down and checking. Heavy hint use suggests the kid should see the skill again soon, with a little less support. A strong explanation plus a clean independent attempt suggests it is time to space it out and come back later, which is the core idea behind spaced repetition.
It is like cooking. If a dish comes out too salty, you do not just cook it again the same way and hope. You figure out whether the recipe was wrong, the measuring was off, or you got distracted and salted it twice. Each answer leads to a different fix.
This is also where desirable difficulties come in. The right next problem is often a little harder than comfortable, mixed in with other topics, or given after a gap. Process evidence lets you tune that difficulty to the actual kid instead of guessing.
The privacy cost, and how to keep it small
Now the uncomfortable part. Process data is far more intimate than a list of right and wrong answers. It can include handwriting, recordings of explanations, the timing of every pause, every abandoned attempt. Collected carelessly, it adds up to a detailed portrait of how a child thinks, struggles and gives up. That is exactly the kind of data that should never be used to profile, rank or market to a kid, and it does the most harm if it leaks.
So the rule I try to hold is simple: collect the evidence you need to decide what to practice next, and nothing more. In practice:
- Record events, not surveillance. "Mistake on step four, procedural, fixed on the second try" is enough. A keystroke-by-keystroke log is not needed.
- Summarize, then let go of the raw material. Transcribe an explanation, keep a short excerpt as evidence, and discard the audio.
- Put time limits on everything. Old process data loses value fast. Last month's mistakes matter much less once the skill is solid.
- Keep it in context. Learning data should be used for learning. Not advertising, not predictions about the child, not anything the family did not sign up for.
- Let parents see it. If you would be uncomfortable showing a parent what you collected, you collected too much.
Takeaways
- A right or wrong answer is the scoreboard. Process evidence is the game tape.
- The most useful signals are where the work went off course, the type of mistake, hints used, the second try, explaining it back, and working with help switched off.
- Sort mistakes into concept, procedure and careless, because each one needs a different fix.
- Use process evidence to choose the next practice, never to label the child.
- Collect the minimum, summarize, discard raw recordings and set time limits.
- Parents: ask any learning app what it records about the process, why, and for how long.
