Adaptive learning systems like MemoryLab have changed the way we learn, by tailoring digital learning sessions to the needs of individual learners. To date, such systems have been particularly helpful in memorizing vocabulary and factual knowledge. In a recent collaboration with College language, a publisher of language learning materials for applied university education in the Netherlands, we examined the possibility of extending our existing learning algorithm, which was developed for efficient memorization of simple facts, to learning complex concepts like grammar rules in the context of language learning. In this blog post, you can read about the background, development and test of an adaptive learning system that aims to keep track of a learners' mastery of Dutch grammar rules.
From simple facts to complex grammar rules
Traditionally, adaptive learning systems have helped learners memorize simple paired associates such as vocabulary or toponymy items. Questions are presented to learners on a computer screen, and the learner is asked to type the correct answer (eg, translation of the vocabulary item, location or name of a country or city, etc). These systems adjust the frequency and difficulty of the content based on a learner's performance, using algorithms to predict which items a learner is likely to forget and when. This method has proven effective in increasing efficiency and retention compared to non- or less- adaptive learning strategies (eg, see van Rijn et al., 2009; Lindsey et al., 2014, Papousek et al., 2014; Memory Lab). In this project, we worked together with College language. Hogeschooltaal is a popular online learning and practice program through which students learn spelling and grammar for Dutch and/or English at university of applied sciences level. The main aim of the project was to see if we could use the above-described adaptive learning methods—which have proven to be successful in improving the efficiency of simple fact learning— to also improve learning more complex materials. More specifically, we aimed to investigate whether the adaptive learning algorithm could keep track of a learners' mastery of grammar rules using randomly presented, example questions (see Table 1 for an example grammar rule with a few example questions in Dutch). Although some prior studies have attempted to keep track of a learner's memory of common grammar rules in a computer model (Taatgen & Anderson, 2002; Taatgen, 2001), none of these studies have attempted to use this information to help a learner memorize these rules , as we do in this project.
Here, we aimed to examine the extent to which performance on specific instance questions is indicative of performance on other instance questions of the same rule, and if the estimated speed at which a learner forgets a grammar rule is predictive of the performance of a learner on a new question for that rule on a later test.
Pilot study
In total, 42 participants completed the pilot study. The study consisted of a practice block and a test. During the practice block, participants studied a total of 18 grammar rules, based on randomly chosen example practice questions (see Table 1 for an example). During the test block, participants received a new example question for each grammar rule. If the response would be correct, learners would simply advance to the next question. If an incorrect answer was given, learners would receive the correct answer, together with an explanation of the grammar rule. After the learning session, a test, on which for all grammar rules followed a new example question would be presented.
For each example question, we measured the correctness of the answer, and the response time. The rationale behind keeping track of these performance metrics is simple: if a learner is able to give a correct answer to a question quickly, the learner is likely to master the underlying grammar rule. If more time is needed to produce the answer or if the answer is incorrect, the learner does not yet master the rule and more practice is required. In our adaptive learning algorithm, the response time and accuracy of the response are summarized in a single metric: the speed of forgetting. Easy items, for which many quick and correct responses are given, have a low speed of forgetting, whereas more difficult items have a higher speed of forgetting.
Using instance questions to infer rule performance
The primary goal of this research was to determine if it is possible to estimate a learner's mastery of grammar rules based on their responses to specific example questions. We observed a clear pattern showing that some grammar rules were consistently more challenging than others, as indicated by generally lower accuracy scores on these rules' instance questions. Additionally, there was a noticeable improvement trend: as learners repeated grammar rules using different example questions, their accuracy increased and their response times decreased. This suggests that participants were effectively learning the underlying grammar rules through varying examples. Moreover, we found a strong positive link between the average accuracy during the learning phase and the accuracy on new, related questions in subsequent tests. This indicates that how well participants responded to example questions during learning could predict their performance on similar questions in tests. Essentially, the way participants answered questions during learning appeared to reflect their grasp of the grammar rules, both throughout the learning process and in the following assessments.
Predicting performance using speed of forgetting.
The second aim of this project was to quantify how well learners grasped grammar rules using the “speed of forgetting” parameter discussed above. Figure 1a illustrates the average speed of forgetting for each rule over several repetitions, updated continually based on how accurately and quickly learners responded to instance questions about each rule. Figure 1b explores how this speed of forgetting, determined during the learning phase, correlates with how accurately learners answered new questions about the same rules during the test with new example questions that followed the learning session. We found that rules with a higher speed of forgetting (meaning learners forgot them more quickly) during the learning phase tended to be answered less accurately in the test. This suggests that our adaptive learning model effectively monitors how well learners understand and retain grammar rules throughout the learning process to predict their performance in future assessments.
Conclusion
In this project, we asked participants to study Dutch grammar and spelling rules through exposure to specific instances of each rule. We show that it is possible to use the learner's answers to instance questions to estimate their performance on new instances of the same rules. We show that we can estimate how well learners have memorized the rules using accuracy scores and response times on randomly introduced example questions. Although future research should explore how these estimations of a learner's rule performance can be exploited to increase learning efficiency, these results pave the way for the development of adaptive learning applications that allow for rule learning based on instances.
References
Lindsey, R. V., Shroyer, J. D., Pashler, H., & Mozer, M. C. (2014). Improving students' long-term knowledge retention through personalized review. Psychological Science, 25(3), 639-647.
Papousek, J., Pelánek, R., & Stanislav, V. (2014). Adaptive practice of facts in domains with varied prior knowledge. Educational Data Mining..
Taatgen, N. A. (2001). Extending the past-tense debate: A model of the German plural. In Proceedings of the annual meeting of the cognitive science society (Vol. 23).
Taatgen, N. A., & Anderson, J. R. (2002). Why do children learn to say “broke”? a model of learning the past tense without feedback. Cognition, 86(2), 123–155.
Van Rijn, H., Van Maanen, L., & Van Woudenberg, M. (2009). Passing the test: Improving learning gains by balancing spacing and testing effects. In Proceedings of the 9th international conference of cognitive modeling (Vol. 2, pp. 7–6).
Wozniak, P. A., & Gorzelanczyk, E. J. (1994). Optimization of repetition spacing in the practice of learning. Acta Neurobiologiae Experimentalis, 54, 59-59.
