Traditional knowledge assessment
How do we assess a learner’s knowledge level? We test them. A traditional assessment might involve asking the learner short-answer or multiple-choice questions, or requiring them to demonstrate their knowledge by writing an essay, for example. We then infer what a learner knows from the correctness of their responses.
While these traditional forms of assessment are commonplace in education, they have some clear drawbacks, as we discussed in a previous blog post. For one, traditional assessments offer a momentary snapshot of a learner’s capabilities. This means that while learners may score well on a test, that does not necessarily mean that they will remember what they’ve studied for long. Indeed, many learners recognise the feeling of having forgotten much of what they learned for a test only a few weeks or months ago. What’s more, basing knowledge assessment on performance at a single moment in time increases the risk of test anxiety, stress or other external factors distorting the outcome (Yang et al., 2023). Lastly, it simply takes time and effort on the part of teachers to make high-quality assessments, which can detract from other pedagogical activities (Swiecki et al., 2022).
Rethinking knowledge assessment in the digital age
The adoption of digital learning environments offers the opportunity to rethink how we assess knowledge, and what the role of assessment in learning is. Digital environments make it possible to continuously measure learners’ activity at a fine-grained level. Put differently, where a traditional assessment measures the product of learning, a digital learning environment can also measure the learning process itself (Lodge, 2018). This kind of low-stakes, continuous assessment has several notable advantages. It can promote learners’ self-efficacy, by providing immediate feedback on performance during the learning process and/or by making progress tangible to learners through analytics in the learning environment (Shute and Kim, 2013). For example, analytics provided by the learning environment can empower learners to make better decisions about which topics to study when (Sense, van der Velde, & van Rijn, 2021). Assessing performance throughout the process of learning also enables real-time adaptation of the learning task, based on the knowledge state of the learner. For example, a learning environment can focus on items that a learner is yet to master, and drop items once they do reach a mastery criterion. This kind of assessment-based adaptation can significantly speed up learning. Finally, incorporating the learning process itself into the assessment opens the door to making more precise predictions about a learner’s future performance. After all, we are not only interested in what a learner knows right now, but also how that learner’s knowledge state is likely to evolve in the future. With the tools available in digital learning environments, we can look beyond traditional knowledge assessment methods towards real-time, actionable and predictive modelling of learners’ knowledge over time.
Knowledge modelling in practice
While the promise of a continuous, integrated assessment in digital learning environments may be clear, there are many possible ways of implementing this idea in practice. This kind of knowledge assessment is typically referred to using terms like knowledge modelling or knowledge tracing (Pelánek, 2017). I have summarised some of the most important demands for effective knowledge modelling in this (non-exhaustive) list:
Real-time descriptive accuracy — Does the knowledge model provide an accurate reflection of current performance?
Predictive accuracy — Can the knowledge model accurately predict future performance?
Cognitive fidelity — Is the knowledge model aligned with cognitive science theories of learning and forgetting?
Interpretability — Can the knowledge model be easily understood and interpreted by learners to guide decision making?
Current approaches to knowledge modelling can be roughly broken down into three categories, which I will refer to as heuristic, statistical, and cognitive.
| Type | Description | Examples |
| Heuristic (rule-based) | Infer if a knowledge component has been mastered using a simple rule of thumb, like counting the number of correct answers | N-Consecutive Correct in a Row; Tug-of-War (Doroudi, 2020) |
| Statistical (data-driven) | Infer if a knowledge component has been mastered from a learner’s interactions using probabilistic models or machine learning | Bayesian Knowledge Tracing; Logistic Knowledge Tracing; Deep Learning-based Knowledge Tracing (Liu, 2022) |
| Cognitive (theory-driven) | Infer if a knowledge component has been mastered from a learner’s interactions, using a computational cognitive model that simulates learning and forgetting processes | ACT-R; Model-Based Mastery; Predictive Performance Equation (Collins et al., 2022) |
These different categories of knowledge modelling represent a different choice of trade-offs between the demands listed above. Heuristic methods excel in their simplicity, which makes them popular (Baker, 2016): they are easy to implement, and metrics like counting the number of correct answers feel intuitively familiar to learners. On the other hand, they align quite poorly with our scientific understanding of the learning process. For example, they ignore temporal dynamics in the learning process. If a learner answers an item correctly on three separate practice attempts, the timing matters: larger time gaps between attempts both make it harder to respond correctly (e.g., due to forgetting), and make the resulting memory trace more resistant to forgetting. Conversely, it is much easier to give three correct answers if the practice attempts follow one another in close succession. Disregarding such factors, as heuristic methods do, therefore increases the risk of making inaccurate knowledge assessments. These methods can also be frustrating for learners to work with: requiring a sequence of three correct responses in a row means that making a simple slip (e.g., a typo) can be a costly mistake.
Statistical methods tend to be substantially more complex than heuristics. This allows them to capture more nuanced performance patterns and make a more sophisticated interpretation of learners’ interactions. For instance, methods like Bayesian Knowledge Tracing allow for the possibility that learners have some amount of prior knowledge, can make a lucky guess when they really don’t know the answer, or make a typo (Corbett & Anderson, 1995). As with heuristic-based methods, however, it is still uncommon for statistical methods to take the passage of time into account in their inference of the learner’s knowledge state, even though that can be highly relevant for correctly interpreting performance in the context of learning and forgetting, particularly when learning takes place on longer timescales (Sense et al., 2019). To give an indication: only 5 of the 25 knowledge tracing models surveyed by Liu et al. (2022) incorporated time-related dynamics in some way. When it comes to predicting future performance on material that is currently mastered, cognitive science tells us that learners will forget at least some of what they have learned, and different learners will forget at different rates, which makes timing information an important omission in many of these methods. In addition, the complexity of some of these statistical methods can be a problem, particularly those using deep learning-based approaches: in cases where assessments of a learner’s knowledge state are intended to be presented to the learner, e.g., to support their study decisions, a method that the learner perceives as an uninterpretable “black box” can be a drawback (Sense et al., 2022). Learners may feel less confident in making study decisions based on outputs they do not fully understand, or feel frustrated when the assessment does not align with their expectations.
Cognitive methods of knowledge modelling stand out from the others by being explicitly grounded in cognitive science theories, which gives them an advantage in accurately simulating learning and forgetting processes. For example, the ACT-R model of declarative memory (which forms the basis of the adaptive algorithm used in MemoryLab) describes in detail how the availability of memorised knowledge evolves during and after learning, allowing it to correctly account for important properties of memory in its interpretation of responses, like the spacing, frequency, recency, and testing effects (Pavlik & Anderson, 2005). This enables it to model each learner’s knowledge within a practice session and beyond in a cognitively realistic manner (van der Velde et al., 2024). It is therefore not surprising that a distinct advantage of these theory-driven methods is that they tend to require less data to perform well than state-of-the-art statistical methods, which still need to learn the statistical regularities of underlying memory processes from the data (Sense et al., 2022). This makes cognitive methods powerful tools in the common scenario when relatively little data is available, for instance when a learner only practises an item a handful of times. Interestingly, combining the theoretical strength of a cognitive method with the computational power of a statistical approach may lead to further improvements in predictive accuracy (Sense et al., 2022). Like statistical methods, cognitive methods are also substantially more complex than simple heuristics, which means that interpretability of assessments remains an important challenge. Cognitive knowledge models have an advantage in that they can represent their assessment of the learner’s knowledge state in psychologically meaningful and pedagogically relevant terms. For instance, our own Model-Based Mastery system describes mastery in terms of predicted knowledge retention in the future. This allows us to show a learner that they have mastered this knowledge for now, but also conveys the temporal nature of this assessment, incentivising them to pursue effective learning strategies.
Conclusion
In summary, the continuous knowledge modelling made possible by advances in digital learning environments can reshape how we assess what learners know. Knowledge modelling methods offer an opportunity to gain more accurate and nuanced insight into learners’ abilities than traditional tests tend to offer. Different approaches to knowledge modelling make different trade-offs between accuracy, cognitive fidelity, interpretability and ease of implementation. Moving forward, combining the strengths of theory-driven and data-driven approaches may lead to further improvement of modelling what learners know.
References
Baker, R. S. (2016). Stupid Tutoring Systems, Intelligent Humans. International Journal of Artificial Intelligence in Education, 26(2), 600–614. https://doi.org/10.1007/s40593-016-0105-0
Collins, M. G., Sense, F., Krusmark, M., & Jastrzembski, T. (2022). Extending the Predictive Performance Equation to Account for Multivariate Performance. eScholarship, University of California. https://escholarship.org/uc/item/93z8b61d
Corbett, A. T., & Anderson, J. R. (1995). Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modelling and User-Adapted Interaction, 4(4), 253–278. https://doi.org/10.1007/BF01099821
Doroudi, S. (2020). Mastery Learning Heuristics and Their Hidden Models. In I. I. Bittencourt, M. Cukurova, K. Muldner, R. Luckin, & E. Millán (Eds.), Artificial Intelligence in Education (Vol. 12164, pp. 86–91). Springer International Publishing. https://doi.org/10.1007/978-3-030-52240-7_16
Lodge, J. M. (2018). A Futures Perspective on Information Technology and Assessment. In R. Latiner Raby & E. J. Valeau (Eds.), Handbook of Comparative Studies on Community Colleges and Global Counterparts (pp. 1–13). Springer International Publishing. https://doi.org/10.1007/978-3-319-53803-7_43-1
Pavlik, P. I., & Anderson, J. R. (2005). Practice and Forgetting Effects on Vocabulary Memory: An Activation-Based Model of the Spacing Effect. Cognitive Science, 29(4), 559–586. https://doi.org/10.1207/s15516709cog000014
Pelánek, R. (2017). Bayesian knowledge tracing, logistic models, and beyond: An overview of learner modeling techniques. User Modeling and User-Adapted Interaction, 27(3–5), 313–350. https://doi.org/10.1007/s11257-017-9193-2
Sense, F., Jastrzembski, T., Mozer, M.C., Krusmark, M., & van Rijn, H. (2019). Perspectives on Computational Models of Learning and Forgetting. Proceedings of the 17th International Conference on Cognitive Modeling. Montreal, Canada.
Sense, F., van der Velde, M., & van Rijn, H. (2021). Predicting University Students’ Exam Performance Using a Model-Based Adaptive Fact-Learning System. Journal of Learning Analytics, 8(3), 155–169. https://doi.org/10.18608/jla.2021.6590
Sense, F., Wood, R., Collins, M. G., Fiechter, J., Wood, A., Krusmark, M., Jastrzembski, T., & Myers, C. W. (2022). Cognition‐Enhanced Machine Learning for Better Predictions with Limited Data. Topics in Cognitive Science, 14(4), 739–755. https://doi.org/10.1111/tops.12574
Shute, V. J., & Kim, Y. J. (2014). Formative and Stealth Assessment. In J. M. Spector, M. D. Merrill, J. Elen, & M. J. Bishop (Eds.), Handbook of Research on Educational Communications and Technology (pp. 311–321). Springer New York. https://doi.org/10.1007/978-1-4614-3185-5_25
Swiecki, Z., Khosravi, H., Chen, G., Martinez-Maldonado, R., Lodge, J. M., Milligan, S., Selwyn, N., & Gašević, D. (2022). Assessment in the age of artificial intelligence. Computers and Education: Artificial Intelligence, 3, 100075. https://doi.org/10.1016/j.caeai.2022.100075
van der Velde, M., Sense, F., Borst, J. P., & Rijn, H. V. (2024). Large-scale evaluation of cold-start mitigation in adaptive fact learning: Knowing “what” matters more than knowing “who”. User Modeling and User-Adapted Interaction. https://doi.org/10.1007/s11257-024-09401-5
Yang, C., Li, J., Zhao, W., Luo, L., & Shanks, D. R. (2023). Do Practice Tests (Quizzes) Reduce or Provoke Test Anxiety? A Meta-Analytic Review. Educational Psychology Review, 35(3), 87. https://doi.org/10.1007/s10648-023-09801-w
