Machine Learning & AI

A central assumption in AI-assisted education is that AI struggles with highly cognitive-demanding tasks just as humans do. This study tests that assumption directly by evaluating ChatGPT and GPT-4 on 54 NAEP science assessment items coded by cognitive complexity and dimensionality, then comparing AI performance to actual student scores.

Results show both AI tools consistently outperformed most students across Grades 4, 8, and 12. Crucially, while students required higher ability to succeed on more cognitively demanding items, the AI tools showed no such sensitivity — their performance was largely unaffected by increases in cognitive demand.

Findings: Current cognitive intensity tasks are insufficient for differentiating human from AI capability, calling for a shift toward creativity, critical thinking, and novel problem-solving in science education and assessment.

Scientific argumentation requires coordinating three skills: making claims (C), using evidence (E), and providing warrants (W). Traditional total scores obscure which of these skills individual students actually possess. This study builds ML scoring algorithms for 19 constructed-response argumentation items, then uses CDM to reveal fine-grained cognitive patterns.

ML scoring achieved strong human agreement (average Cohen's κ = 0.73). CDM analysis identified 21 distinct skill-mastery profiles among 932 Grades 5–8 students — with the 9 most common profiles covering over 70% of the sample. As the radar charts illustrate, three students with the same total score can have very different argumentation skill patterns.

Findings: CDM provides actionable diagnostic information that total scores cannot — enabling targeted instructional feedback on claims, evidence, or warrants for each student.