ChatGPT has been banned in some classrooms and embraced in others, but what does the evidence actually say? Does ChatGPT enhance student learning?
This is a question posed by a group of Chinese researchers who conducted 69 experiments in an attempt to come up with an answer.
The 35-page paper makes for interesting reading, so students, teachers, and academics listen up.
Hello and welcome to the AI Six podcast, where I teach you AI in just six minutes a day as we all try to keep up with this tech and societal revolution.
So what did the researchers uncover? Is ChatGPT a helpful aid to students, or will it forever limit critical thinking?
First, about the study
The five researchers, Ruiqi Deng, Maoli Jiang, Xinlu Yu, Yuyan Lu, and Shasha Liu are associated with the Hangzhou Normal University in Eastern China, known for its education programmes. The Chinese Education Modernization Research Institute plays a role in shaping forward-looking education policy, including integrating digital tools and innovations like AI in classrooms.
Together, they conducted a review of 69 experimental studies on ChatGPT’s impact on student learning.
Most studies were done in university settings. Around 85 per cent of them, in fact. The most common subjects were language education, computing, health, and physics. Nearly all of the experiments took place in real classroom environments, rather than isolated lab settings. And most of them used ChatGPT as a direct learning tool, where students interacted with it during lessons or as part of assignments.
Limitations of the study
Many of the experiments did not include a power analysis, meaning the sample sizes may have been too small to yield reliable results. Some studies also failed to control for pre-existing differences between groups, which makes it harder to isolate the effect of ChatGPT itself.
There was also evidence of publication bias, positive results are more likely to be published than neutral or negative ones, which can skew the overall picture.
Finally, the long-term impact remains uncertain.
The review points to a possible novelty effect, where initial improvements in motivation and engagement may fade over time as the excitement around the technology wears off.
Top-line takeaways
- Performance improved significantly when students used ChatGPT – but here’s the catch: we need better ways to measure if students are actually learning vs just getting better outputs
- Students felt more motivated and engaged when using ChatGPT. But is this just because it’s new and exciting?
- Students reported better critical thinking when using ChatGPT – though we need more objective ways to measure this beyond self-reporting
- No significant impact on students’ confidence in their own abilities
- Mental effort decreased – but this could be good or bad depending on whether it helps or hinders deeper learning
- The big takeaway: ChatGPT shows real promise in education, but we need smarter ways to use and evaluate it.
Four recommendations
The researchers offer four clear recommendations for future work.
- First, we need better assessments, ones that go beyond standardised tests to include project-based work and originality.
- Second, we need to study the long-term effects of ChatGPT on motivation and learning behaviour.
- Third, researchers should use objective methods to evaluate critical thinking and problem solving, not just student surveys.
- And finally, studies must be better designed with proper sample sizes and clear controls.
Final Thoughts
The AI revolution is disrupting education, just like every other sector.
If you’re using ChatGPT in the classroom or thinking about it, I’d love to hear your experiences. You can find me on LinkedIn or message me through the podcast website.
If you want to learn how to use AI without crossing ethical lines, my AI Diploma for Marketers covers exactly that. Use the code AISIX10 for a discount. Until next time, stay AI curious!
About the Podcast Host
Joanne Sweeney is a specialist in public-sector marketing and digital communications. She’s the author of Public Sector Marketing Pro and the creator of the Professional Diploma in AI for Public Sector Marketing. With years of experience in training, consulting, and advocating for digital transformation, Joanne provides insight and guidance for government agencies navigating the digital landscape. She is also the host of the award-winning Public Sector Marketing Show podcast.
