Choosing the next question in Memoza

Two students open the same course. One can execute the main procedures but confuses when to use them. The other chooses appropriate methods but repeatedly loses track of an algebra step.

Giving them different questions would be easy. Giving them different questions for defensible reasons is the real design problem.

This article sets out the recommendation architecture we want Memoza to support. The signals and decisions described below are design requirements; the article is not a verified inventory of every feature currently running in production.

First decide what can reasonably be selected

A recommendation starts with an eligible set. An activity must belong to the relevant course and version, support the chosen mode, have a usable grading path, and respect the learner's explicit topic selection where appropriate.

Prerequisites complicate this step. If a task depends on knowledge the student has not demonstrated, the system could offer a prerequisite check or a supported version. A missing observation should not automatically mean the student lacks the skill. They may have learned it elsewhere.

Eligibility also includes content availability. If a needed objective has no suitable question, that is a coverage gap to surface. A ranking algorithm cannot repair it by repeatedly selecting the closest available item.

Rank the needs behind the activities

Within the eligible set, several reasons might make an activity useful. A recent error could deserve repair. An old success could deserve a retention check. An untested topic could deserve diagnosis. A well-established method could be tested in a less familiar context.

These reasons involve different uncertainties. We might be uncertain about the student's knowledge, the item's difficulty, or the diagnosis of an error. Combining all three into one “mastery gap” can conceal what the system actually knows.

Research on spaced repetition provides one example of a defined prediction target. Settles and Meeder's half-life regression model used learner interaction data to estimate recall over time in language learning (Settles and Meeder, 2016). That target is more specific than a general claim to measure understanding. Adapting such an idea to university problem solving requires a new validation case.

Our proposed policy should similarly specify what each signal predicts and what decision it informs. A suspected retention risk might justify a delayed independent attempt. It does not, by itself, justify a harder problem.

Difficulty should serve the purpose

A diagnostic task, a fluency task, and a transfer task can reasonably use different difficulty levels. If the goal is to identify a prerequisite failure, adding several other demands may obscure the answer. If the goal is to assess method selection, explicitly naming the method defeats part of the purpose.

The choice between breadth and depth enters here. Repeated errors on the same step may justify staying nearby. Successful execution combined with poor method selection may justify comparison across topics. A long gap since an earlier success may justify review before introducing another variant.

These are hypotheses about useful responses to learning evidence. A first policy could implement them as transparent rules. Calling them adaptive would describe their dependence on learner state; it would not establish that they improve outcomes.

Occasionally check what the system is unsure about

A policy focused entirely on known weaknesses can neglect areas it has never measured. A policy focused entirely on exploration can spend too much time discovering what the student already knows.

For our two students, a brief comparison task might help determine whether method selection is actually the obstacle. A short algebra check might distinguish a procedural gap from a one-off error. Each diagnostic should earn its place by changing a later decision.

This is also why prediction accuracy alone is an inadequate evaluation. If the system selects only easy, familiar questions, it may predict correct answers very well while providing little useful practice.

Make each recommendation reconstructable

An auditable decision should preserve the eligible activities, the information available at selection time, the policy version, and the reason for the choice. If two otherwise equivalent candidates were selected randomly, that fact should also be recoverable.

The learner-facing explanation can be simpler: “You chose the right method but made the same transformation error twice,” or “You have not tried this objective without hints.” These are examples of the explanations the proposed architecture should support, rather than claims about current interface text.

Reconstructability makes debugging possible. It does not prove that a recommendation was beneficial. That needs comparison with a credible alternative and a later learning outcome.

For Memoza, the standard should progress in stages: verify that the policy follows its specification, check whether its estimates predict relevant performance, and test whether its decisions improve learning. Each stage answers a different question. The final stage is the one that justifies claims of educational advantage.

Reconstructability makes debugging possible. It does not prove that a recommendation was beneficial.

How Memoza fits

Two parts of this are built today: the eligibility step and the record. The candidates for a step are the concepts of the session's own course version that some template of the chosen activity type can actually serve through an answer channel this build accepts, narrowed by the session's mode and, in a topic session, by the topic that session names. What ranks them is a practice need built from seven weighted terms, and each candidate keeps the signed contribution of every one of them rather than only a total. The stored decision then holds that whole ranking rather than the winner alone, the reason codes derived from those same numbers, which policy configuration produced them, which sequencing policy governed the step, and how many concepts were eligible before the candidate cap. Ties break on concept id, so two concepts with identical evidence produce a stable order rather than a coin toss. What none of that settles is the question this article ends on: whether the choice helped.

References

Settles, B., & Meeder, B. (2016). A trainable spaced repetition model for language learning. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, Volume 1, 1848–1858. https://doi.org/10.18653/v1/P16-1174 Computational research paper.