How Margins is evaluated

Margins has two evaluations. A retrieval eval measures how well the app maps a topic to the chunks of the course material that explain it. A generation eval runs the app's real generators (notes, worked examples, practice problems, quiz) on two real courses, then checks the output two ways: automatic checks in code, and scores from a model judge.

The numbers on this page are copied from the result files in the repository, which are linked at the end. The runs are from October 20 and 21, 2026.

Retrieval

When the extract-topics job finds a topic, it looks up that topic's source chunks, keeps the top 24 and stores them. Every notes, quiz and exam generator later reads only those chunks. If the right chunks are not in the stored 24, the generated material cannot be grounded in them.

The harness answers: for a topic whose answer we know the location of, how many of the right chunks are in that top 24, and how high are they ranked?

Method

• The corpus is chapters from three OpenStax textbooks (CS, science, social science): 306 chunks, ingested through the app's real chunker and real embedder.
• There are 38 topics, each with a name, a description and keywords, shaped like what the topic extractor produces. Each lists the section whose chunks count as relevant.
• For every topic the harness calls the app's retrieval the way the app does, with the same arguments and the same number of results. Relevance is binary and averaged over topics.

Results

The old search found nearly all the relevant chunks within 24 but ranked them badly. Fusion is the large change. Against vector search only the gap is smaller: MRR 0.961 against 0.934, and nDCG 0.932 against 0.917.

recall@24 is high for almost any method on this corpus, so the useful signal is in the ordering (MRR, nDCG and hit@5).

Limits

• Small corpus. About 300 chunks across three documents, and K = 24 is 8% of it. The corpus is also tidy: well-structured textbook prose, not scanned slides or lecture notes.
• 38 topics, one writer. The topics were written by one person, not generated by the topic extractor, so they are cleaner than real extractor output. Per-topic numbers are noisy.
• Relevance by section is an approximation. A chunk in another section that also explains the topic is counted as wrong.
• Tuned on this corpus. The similarity floor and the full-text query were chosen by looking at these 38 topics. A different corpus may want a different floor.
• One embedder, English only. Results are for the default local embedding model, and the full-text search is English only.

Generation quality

The latest run is from October 21, 2026. It generated 8 topics (4 from MIT 18.05, probability and statistics, and 4 from MIT 6.006, algorithms) with Claude Opus at medium effort. Each topic got notes, worked examples, practice problems and a quiz: 3 examples of each kind, 10 quiz questions, medium difficulty. The topics came from the app's course planner.

Automatic checks

These involve no model. They are lexical heuristics: they find things worth reading, they do not show content is right. The ones below include every check the output failed; the report has the rest.

The recall-copy check was tightened after the run. On the stored outputs it now finds 4 of 32 numeric quiz questions (12%) that restate a worked example. The report's own table still shows the earlier 50%.

Judge scores

The judge is Claude Opus at medium effort, the same model that wrote the content. It sees the source excerpts and is not told which model wrote what, but a model grading its own output is not an independent reviewer, so read the judge scores as limited evidence.

The judge listed 8 errors, 2 of them mathematical: a heap-building explanation that gave about 1.44n swaps where the exact total is at most n - 1, and an index range that was off by one. The report lists all 8. The judge can be wrong, so check an error before relying on it.

Cost per topic, all four content types: $0.585. The run cost $6.99 in total, with the judge.

Limits

• One sample per cell. Each topic, model and content type is generated once. Model output varies run to run.
• Small corpora. Two MIT OpenCourseWare courses, four topics each.
• Excerpts, not the whole course. The judge and the grounding check see only the 24 excerpts each topic is mapped to.
• Checks are lexical. Zero hits means none were found, not that none exist.

Model choice

Before the run, a rule was set: per purpose, the cheapest model whose judge correctness mean is at least 4.5 with no mathematical error, and whose teaching and question-quality mean is within 0.3 of Opus. The runs used the same two courses (4 topics each) and the same Opus judge, which is also one of the three models.

Sonnet and Haiku fail the rule for every content type: maths errors in notes, worked examples and quizzes, and teaching about a point below Opus. Opus at low effort costs $0.52 per topic (16% less), but its sample was partial and its teaching mean was lower, so it is not used.

Notes, worked examples, practice, quizzes and exams stay on Claude Opus at medium effort. Cost is controlled by generating on demand, when a student opens a topic, rather than by a weaker model.

Raw results

Every table above comes from one of these files in the public repository.

• Retrieval, hybrid search with fusion (2026-10-20)
• Retrieval, the old search (2026-10-20)
• Retrieval, the two compared (2026-10-20)
• Generation quality, with every error the judge listed (2026-10-21)
• Model choice run (2026-10-20)
• Opus low against medium effort (2026-10-21)
• The model decision (2026-10-21)
• The evals README: method, metrics and limitations

The courses are MIT OpenCourseWare materials and the textbooks are from OpenStax, both under CC BY-NC-SA 4.0, so the source text is not committed to the repository.

Margins was built by Harish Manoharan.