Can context improve AI-generated MCQ distractors?

Multiple-choice questions (MCQs) are popular among learners, they’re convenient, accessible, and require relatively little effort. Unfortunately, MCQs are often criticised for promoting recognition rather than active recall, making them less effective for long-term memory retention (Karpicke & Roediger, 2008). Recognition of the correct response is especially easy if the distractors, the incorrect answer options, are not carefully chosen. For example, if one needs to choose between New York, Tokyo, and Rotterdam for the question “Where was  Erasmus born?”, most people will answer correctly even though they do not know the answer.  However, when distractors are carefully designed to challenge the learner without misleading them, they can foster deeper processing and improve retention (Little & Bjork, 2015). Experience shows that creating high-quality distractors is not an easy task. It demands time, expertise, and a deep understanding of the content domain.

Recent advancements in large language models (LLMs), such as GPT-4, have raised interest in whether these tools can support or automate distractor generation. In this blog, we explore the use of GPT-4 for distractor generation, with a focus on vocabulary learning tasks. Specifically, we investigated whether adding lesson context, such as a short article or educational paragraph, improves the quality of generated distractors compared to a prompt that relies on a wordlist alone.

Can LLMs help with distractor generation?

While large language models have demonstrated strong performance in text generation tasks, prior research has pointed out several limitations when applying them to MCQ distractor generation (Alhazmi et al., 2024). In particular, high-quality distractors are typically expected to meet three core criteria: they must be plausible, clearly incorrect, and semantically diverse. However, LLMs often struggle to consistently meet these standards.

A common issue is that the model generates distractors that are too close in meaning to the correct answer (e.g., near-synonyms), which can confuse learners. Conversely, the model may produce distractors that are too vague or generic, reducing the cognitive challenge of the item (Alhazmi et al., 2024). These shortcomings can undermine the effectiveness of MCQs in promoting deep learning or meaningful retrieval practice.

Why context might matter

Recent work suggests that the quality of educational outputs from LLMs can be significantly improved by providing context. For instance, Zhang et al. (2024) found that when LLMs generate questions based on contextual input, the results tend to be more descriptive and better aligned with the source material. Similarly, other studies in the area of automated question and answer generation have shown that adding relevant context helps models generate output that are of improved quality (Alhazmi et al., 2024; Wei et al., 2022).

Our method

To further investigate this, we performed a controlled comparison. Using our carefully designed distractor-generation prompt, we asked GPT-4 to generate distractors for Dutch vocabulary items. Here we tested our prompt in two different conditions (see figure 1):

  1. Wordlist-Only Prompting: The prompt is provided with cue–answer pairs;
  2. Contextual Prompting: A contextual passage in which the vocabulary appears in use was added alongside the prompt and cue–answer pairs. This helped contextualise the words both semantically and pragmatically.
Figure 1. Overview of the two prompting conditions used for distractor generation. 1: Wordlist-Only Prompting. 2: Contextual Prompting.
Figure 1. Overview of the two prompting conditions used for distractor generation. 1: Wordlist-Only Prompting. 2: Contextual Prompting.

We created a structured distractor-generation prompt based on NLP research (Alhazmi et al., 2024), asking for distractors that are:

  • Similar in part of speech
  • Similar in category (e.g. food, clothes, working tools)
  • Semantically distinct but contextually plausible 
  • Not synonymous nor antonymous with the answer

We approached this comparison with two main expectations. First, we expected that including contextual material would help the model disambiguate the intended meaning of the correct answer, thereby reducing the risk of generating vague or synonymous distractors. Second, we wondered whether this added context would lead the model to produce more thematically consistent but potentially less varied distractors, for example, staying too close to the topic or reusing phrasing from the text, at the expense of diversity.

Our findings

Our results show that both techniques produced generally good-quality distractors. However, contextual prompting led to improvements in several key areas:

1. Semantic clarity

With context, the model more consistently generated distractors that were clearly distinct in meaning from the correct answer. For example, for “invloed hebben op” (“to have influence on”), the no-context prompt gave “controle hebben over” (“to have control over”): a near-synonym. With context, distractors like “informatie geven over” (“to give information about”) were suggested: still relevant, but more distinct in meaning and function.

2. Thematic consistency

Contextual prompting also slightly improved the thematic alignment of distractors. By placing vocabulary in a lesson context, the model produced distractors that better matched the intended use of the correct answer without overlapping too closely. For example, for “herkenbaar” (“recognisable”), the no-context prompt included distractors like “vindbaar” (“findable”) and “hoorbaar” (“audible”):  similar in form, but thematically inconsistent. With context, distractors like “zichtbaar” (“visible”) and “merkbaar” (“noticeable”) were more closely related to the concept of recognition. However, these differences are relatively subtle and open to interpretation.

3. Grammatical alignment and diversity

Both methods performed similarly in terms of grammatical accuracy. In nearly all cases, distractors matched the part of speech of the correct answer, ensuring grammatical plausibility.

Discussion and conclusion

This exploration supports the idea that including lesson context in LLM-based distractor generation can improve the semantic and thematic quality of MCQ items. While both prompt types yielded grammatically accurate and diverse distractors, contextual prompting led to slightly clearer semantic distinctions and better alignment with the functional use of the target vocabulary. A potential caveat is that the reduction of near-synonyms when context is provided will work well in most cases of learning, but might be less suitable when the expected levels of proficiency are at a near-native level. Importantly, diversity across distractors remained stable across both conditions, indicating that context enhances relevance without limiting variation.

This suggests that educators can improve LLM output by pairing vocabulary with authentic context. In the future, it may be worth exploring how different types of context (e.g., story-based vs. informational) influence output, and how future models can be better guided through prompting for educational use.

References

Alhazmi, E., Sheng, Q. Z., Zhang, W. E., Zaib, M., & Alhazmi, F. (2024). Distractor generation in multiple-choice tasks: A survey of methods, datasets, and evaluation. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), 14437–14458. https://doi.org/10.18653/v1/2024.emnlp-main.799 

Karpicke, J. D., & Roediger, H. L. (2008). The critical importance of retrieval for learning. Science, 319(5865), 966–968. https://doi.org/10.1126/science.1152408 

Little, J. L., & Bjork, E. L. (2015). Optimizing multiple-choice tests as tools for learning. Memory & Cognition, 43(1), 14–26. https://doi.org/10.3758/s13421-014-0452-8

Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., & Zhou, D., et al. (2022). Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35, 24824–24837

Zhang, Y., Liu, X., Sun, Y., Alharbi, A., Alzahrani, H., Shi, T., Alomair, B., & Song, D. (2025). Can LLMs ask good questions? arXiv. https://doi.org/10.48550/arXiv.2501.03491

Offerte aanvragen?

Vul het onderstaande formulier in:

Of mail naar:

Aan de slag!

Wilt u meer weten?

Vul het onderstaande formulier in:

Of mail naar: