I was struck by a recent article in the Economist about whether AI is stopping children from learning.

The headline finding is extraordinary.

Recent research, based on 30 months of data from 26,811 Chinese secondary-school students, found that after adopting generative AI, students’ homework scores increased (on average by 18%), while the time they spent doing their homework fell (by 30%).

So far, so good. Except that their exam scores subsequently fell by 20%.

In other words, students were producing better homework, faster, but learning less.

The authors found that the problem was concentrated among students whose behaviour was consistent with outsourcing their homework to AI. Those who used AI but continued to spend roughly the same amount of time on their work experienced only small learning losses.

The Power of Productive Struggle

This made me think about productive struggle.

Learning is not simply about getting the right answer. It involves building and retrieving knowledge, making connections, trying something, getting it wrong, receiving feedback, revising your thinking and trying again.

That process requires effort and involves some friction or struggle. In the case of learning, in contrast to that of work, that effort is not a bug in the process but rather a key part of it.

This is consistent with the idea of “desirable difficulties” developed by Robert and Elizabeth Bjork: conditions that make learning feel harder or less efficient in the short term can, under the right circumstances, produce more durable learning.

The Tutoring Paradox

I keep coming back to Benjamin Bloom’s famous “2 sigma” research. Bloom found remarkably large gains from one-to-one tutoring compared with conventional classroom instruction. Shouldn’t using AI (tutors) result in similar learning gains?  

That would be the Holy Grail, but maybe we’re not there yet.  Tutoring isn’t simply a matter of giving each student the right answer. The tutor continually diagnoses where the learner is, asks questions, provides feedback, adjusts the level of challenge and keeps the learner engaged in the task. There is a human relationship involved too.

This is the question for AI in education.  How can AI provide the benefits of a tutor without removing the struggle that makes the learning happen?

Augmentation Works

Interestingly, a recent randomised experiment by Zara Contractor and Germán Reyes found that students using AI to learn an unfamiliar subject performed better on subsequent unaided knowledge tests, with the gains persisting a week later. But the strongest delayed gains came from students who used AI to augment their learning. For example, by asking it to explain concepts rather than using it to automate the production of their work. The authors found that AI users shifted time away from drafting and towards reading and searching for information.

To Be or Not To Be

The real distinction we should be looking at isn’t between “AI” and “no AI”.

It is between AI that does the thinking for you and AI that makes you think better.

If we optimise education for productivity (higher homework scores in less time) we may accidentally optimise away something much more important: the learning itself.