A study of 760,000 students still needed a normal-person sentence

My daily AI news brief handed me a very appealing conclusion: a massive international study had found that unguided AI use was associated with worse learning, while purposeful, taught AI use was associated with better outcomes.

That was convenient. Almost too convenient. It lined up neatly with what I already teach in AI-literacy classes, which is exactly when I become nervous. Evidence that agrees with you deserves at least as much suspicion as evidence that does not.

I asked AI to check what the OECD study actually found, then help me explain it in a normal-person way.

“Normal person” has become my shorthand for the average person who is not Christopher. We have established that I have what we will generously call a translation challenge. I like the full evidentiary foundation, the qualifications, and the mental model behind the qualifications. A person walking into an introductory class generally does not want to begin with my private festival of methodological restraint.

The scale of the study was real: PISA 2025 covered more than 760,000 15-year-olds across 91 countries and economies. This was not twelve undergraduates and a questionnaire. But the clean conclusion from my brief did not quite survive contact with the source.

The study did not prove that teaching students to use AI caused better learning. It was observational, not a controlled experiment. Students who used AI were not one uniform group, either. The pattern changed with frequency, purpose, task, and whether students had learned to question and evaluate what AI gave them.

The data said something messier. Simply giving someone access to AI did not automatically make them better at anything. The way they used it mattered.

We tried a calculator analogy. Handing someone a calculator does not teach mathematics. A calculator can replace thinking or support thinking. The difference is whether the person understands the underlying work, knows what the tool is doing, and can recognize when its answer makes no sense.

Then we reached the line I could actually imagine saying to a room:

“Using AI isn't what seems to matter most. Knowing how to use it does.”

There was another line that cut even closer to what worries me about these tools:

“AI can make your answer better without making you better.”

The goal is to get both.

I have not used those lines in a class yet, so there is no heartwarming ending about the person in the back row whose relationship with technology changed forever. The useful work happened earlier: the first claim sounded perfect, the source made it messier, and the sentence improved once it stopped pretending the evidence said more than it did.

Finding a memorable line was helpful. Checking whether I had earned the right to say it was better.