Building a compiler for university courses

A probability course arrives as a collection of files: lecture slides, tutorial sheets, a syllabus, perhaps a formula sheet and a past paper. The files contain knowledge, but they do not yet specify a learning environment.

They do not reliably say which ideas are prerequisites, which exercises are introductory, or what evidence would show that a student can handle the final assessment. Some of that information is explicit. Some requires interpretation. Some is missing.

At Memoza, “course compiler” is our name for an architectural idea: turn these materials into an explicit, reviewable course representation before using them to plan practice. The term is an engineering analogy. A successful compilation would mean the required structure and checks are present; it would not certify educational quality by itself.

Begin with what the student must be able to do

A topic label such as “conditional expectation” is too broad to define a useful activity. A course might require students to calculate it from a joint distribution, explain a property, or apply it within a proof. Those performances overlap, but one question will not necessarily assess all of them.

Biggs' account of constructive alignment connects intended learning outcomes, teaching activities, and assessment (Biggs, 1996). For a course model, the implication is to preserve the requested performance rather than extract only a list of nouns. “Define,” “derive,” “interpret,” and “implement” can imply substantially different tasks. This is a design principle, not experimental proof of a particular software pipeline.

Suppose an objective requires students to justify why an estimator is unbiased. A flashcard asking for the definition of bias covers relevant vocabulary. It does not provide the same evidence as a derivation in which the learner must manipulate an expectation and state the assumptions.

The course representation should preserve that difference before any activity is generated.

Separate the course from its assessment

A syllabus describes the territory of a course. An assessment profile describes the conditions under which some of that territory will be examined: response formats, permitted tools, time limits, and marking expectations.

Keeping these representations separate prevents a common error. The frequency of a word in the slides should not silently become its predicted share of exam marks. A long chapter can provide background for a short assessed skill. A brief theorem can support several demanding questions.

The frequency of a word in the slides should not silently become its predicted share of exam marks.

Evidence about assessment also varies in strength. A current official specification is different from an inference based on one old paper. Where a topic weight is unknown, the model should preserve that uncertainty instead of inventing a precise percentage.

For a course spanning several years of materials, provenance becomes practical. Which syllabus version supports this objective? Does the permitted-calculator rule still apply? Was the example introduced as an examinable technique or as enrichment?

Build the argument behind each task

Evidence-centered assessment design asks how observed responses support claims about a learner's capabilities. Mislevy, Steinberg, and Almond connect those claims to the tasks, evidence rules, and operational processes used to assess them (Mislevy et al., 2003). That framework offers a useful foundation for specifying what a generated question is supposed to reveal.

For the unbiasedness example, the intended claim might be that the student can construct an expectation-based argument. The required evidence could include the correct expectation operation, a justified transformation, and a conclusion tied to the parameter. The question and rubric should make that evidence observable.

This gives generation a concrete specification. It also allows a reviewer to reject a fluent question for the right reason: it may be accurate yet test only substitution into a formula.

What the compiler should preserve

The proposed output is a set of connected records: objectives, concepts, prerequisite relationships, assessment requirements, activity specifications, grading expectations, and source references. Human review is needed wherever those relationships depend on interpretation.

Validation can then ask targeted questions. Does every activity map to an objective? Does its answer exist under the stated assumptions? Is its response format supported by the grader? Does an assessment requirement contradict a newer source? Are major objectives missing practice opportunities?

Some checks can be automated. Others require disciplinary judgement and evidence from actual learners. In particular, assigning a difficulty label during authoring does not establish how difficult students will find the item.

The pipeline will also need revision in both directions. An unanswerable question may expose a missing assumption in the specification. Repeated student errors may reveal that a supposed single skill actually combines several demands. A compiler metaphor should not force a one-way process onto work that benefits from iteration.

The ambition is to make these decisions explicit enough to inspect and improve. Before Memoza can choose a good next activity, it needs a defensible account of what that activity means within the course.

How Memoza fits

The private course builder is this idea as far as it has been built. Uploaded material becomes an extraction that a person has to approve before anything is derived from it, and generation never approves its own output; the approval is what a deterministic materialisation step reads to write the course structure, which is topics, concepts, learning objectives, the prerequisite edges the extraction stated, and a grading definition minted for that course version. What is known about the assessment is kept in a separate record, where a stated fact carries its own provenance, a per-topic weight carries its own, and “unknown” is a value somebody records rather than a blank the system quietly fills: a course does not count as ready while an assessment fact has neither a value nor an explicit claim. It is measured against nine named readiness gates before it can be studied, and a person reviews each drafted question on a course-alignment dimension before it becomes practice. None of that settles the harder half of the argument above, which is whether the representation is a good one.

References

Biggs, J. (1996). Enhancing teaching through constructive alignment. Higher Education, 32, 347–364. https://doi.org/10.1007/BF00138871 Instructional design framework.

Mislevy, R. J., Steinberg, L. S., & Almond, R. G. (2003). On the structure of educational assessments. CSE Technical Report 597. National Center for Research on Evaluation, Standards, and Student Testing, University of California, Los Angeles. Assessment design research report.