I’ve been thinking quite a lot recently about what happens to learning when we introduce generative AI.

My concern is that if AI makes it easier to complete a task, are we also making it easier to avoid the thinking that gets us there?

That’s why I was interested in a recent paper looking at human-AI collaboration at scale.

AI, Claudius

The researchers analysed almost 250,000 real-world conversations with Claude. Rather than looking at what AI could do in a controlled experiment, they looked at what people were actually doing with it.

There’s quite a lot of learning going on.

In 67% of learning and upskilling conversations, Claude engaged in some form of teaching. This wasn’t just school or university work. Software development, health and lifestyle, and business were among the most common areas.

The most common approach was combining conceptual explanation with step-by-step guidance.

So far, so good. But teaching and learning are not the same thing.

I can ask AI to explain how something works. I can also ask it to do the work for me. The two interactions might look superficially similar, but the consequences for learning are very different.

Then I read that almost half of the conversations involved some form of friction.

The AI misunderstood the question, made a mistake, couldn’t do what the user wanted, or perhaps the user’s instructions weren’t clear enough.

Normally, we think of this as something technology should get better at eliminating.

However, in 78.7% of these cases, users tried to recover from the problem, and when people challenged the AI’s reasoning, those attempts were successful 81.5% of the time.

This is much more interesting than the headline that AI can teach.

Resistance is not futile

There is a version of AI-supported learning that looks like this:

Ask a question. Get an answer. Move on.

There is a more productive version:

Ask a question. Get an answer. Notice something doesn’t make sense. Challenge it. Work through the disagreement. Correct the answer.

The second interaction looks rather like learning.

There is effort, uncertainty, feedback. There is the possibility of being wrong. The learner still has to think.

This brings me back to a question I have been exploring in some of my recent writing about AI and education:

Are we using AI to make learning more effective, or to make tasks more efficient?

A student who uses AI to produce a better piece of homework may well get a better piece of homework. But that doesn’t tell us whether the student has become better at the underlying task.

On the other hand, an AI system that helps a student formulate a question, tests their understanding, gives them feedback and challenges their reasoning could potentially be a very powerful learning tool.

The difference is what the human is doing.

The answer isn’t to keep AI away from learning. Quite the opposite. The potential is enormous.

But we should be careful about celebrating every reduction in effort.

Effort is the point. Productive effort drives learning.

If AI removes all of it, we may discover that we have superficially made learning “easier” but also thrown the baby out with the bathwater.

The interesting question isn’t whether AI can teach: it can.

The question is whether we can use AI to make learning easier without taking the learning out of it.