Can Artificial Intelligence Identify Reading Fluency and Level? Comparison of Human and Machine Performance

Abstract

This study examined whether an artificial intelligence-based automatic speech recognition system can accurately assess students' reading fluency and reading level. Participants were 120 fourth-grade students attending public schools in T & uuml;rkiye. Students read a grade-level text out loud while their voice was recorded. Two experts and the artificial intelligence-based automatic speech recognition system analyzed the recordings for reading errors. Following the analysis, a word error rate was calculated for both the experts and the artificial intelligence-based automatic speech recognition system. Word error rates were converted into reading accuracy rate scores. Inter-rater agreement and linear regression analyses were used to compare the raters' reading fluency scores, and logistic regression analyses were used to compare the classification of readers according to their reading levels. Results showed that the difference between the scores of the artificial intelligence-based automatic speech recognition system and the expert scores was minimal. This is because there was a very high level of agreement between the artificial intelligence-based automatic speech recognition system and the experts scores. Linear regression analyses showed that the artificial intelligence-based automatic speech recognition system significantly predicted the scores of experts. According to the logistic regression analysis results, the artificial intelligence-based automatic speech recognition system was at least 93% as successful as human raters in classifying readers as poor and good. These results give us hope that reading assessments at classroom, school, regional, national, and even international levels can be conducted more accurately and economically by using artificial intelligence-based systems in the coming years.