Certainty-Based Marking (CBM) and Moodle
Certainty-Based Marking (CBM: see LAPT) is a strategy to reward
students for distinguishing between uncertain and reliable answers in a quiz.
CBM options for Moodle users are as follows:
Downloads. For full CBM functionality in Moodle, code patches are available (as listed above) from here. These do not affect database structures at all, nor functionality when CBM behaviour modes are not selected. Installation is easily reversed if you prefer, since the modifications don't affect the database. There is a forum for any discussion, suggestions or report of problems.
- Moodle 1.9-2.0: CBM was not included in standard Moodle code. Good CBM behaviour is available if you replace a few files with versions available in download patches, below.
- Moodle 2.1: Basic CBM behaviour was implemented in standard Moodle code, but CBM users are advised to upgrade from this version.
- Moodle 2.2-2.5: CBM behaviour can be much improved with patches available as downloads on this site.
- Moodle 2.6: CBM behaviour was improved and works well for students, without patches. CBM Grade Reports for teachers require simple installation of a PLUGIN, from the Moodle site, or from here.
Demonstration Sites: These are demo sites to show what effects the patches for older Moodle versions will have
(1) with Moodle 2.4 Dec 2012. Login as a 'test student' with username = 'moodler', password = 'moodler' . Email me for access as a teacher.
(2) with Moodle 2.2 (Moodle 2.3 available code is similar). Login as a 'test student' with username = 'moodler', password = 'moodler'. Email me for access as a teacher.
(3) with Moodle 1.9.
Login as a 'test student' with username = 'moodler', password = 'moodler', or register in your own name and ask to have the role of 'teacher' (further instructions on each site).
CBM with Moodle 2.6 (or with 2.2-2.5 plus patches)
These include the following main features:
1. Scores are presented not only in terms of the total number of marks, but as "Accuracy" (percentage of answers correct) with a "CBM bonus" indicating how well a student has discriminated reliable from uncertain responses. In combination, these give an enhanced measure of performance (= Accuracy + Bonus, maximum = 100%), that has greater statistical reliability predicting performance than simple "Accuracy". The bonus provides feedback to students as to how successfully they are acknowledging sound and weak areas of knowledge.
2. If total guesses (with no knowledge) might be expected to elicit a significant proportion of correct answers,
the simple "Accuracy" and enhanced "CB Accuracy" scores are also expressed as "Percent Knowledge", scaled so guesses would on average give zero,
and confident correct answers would give 100%.
3. When a student chooses to respond just to some of the questions in a quiz, all scores are presented both as percentages
of the number of questions in the quiz and also relative to the questions chosen for response. For example, if you respond to 10 questions in a
quiz that has 20 questions, and you get them all right, you will get two accuracy scores: 50% (relative to the quiz) and 100% (for your responses).
This is important to help students when they are selecting the topics on which they wish to assess themselves.
4. Different weights or grades assigned to different questions in a quiz are ignored when CBM behaviour is selected, to keep things simple. Links have been added to the interface to provide explanations of CBM and of the various CBM scores.
Older Code Versions)
Earlier code downloads (for Moodle 1.7 - 2.0, before Feb 2012) used a different strategy for presenting final scores.
In this, a Certainty-Based Score (CBS) is calculated with a non-linear transformation from the CBM marks, so that
an average student gets about the same score as their "Percent Knowledge" (see 2 above) calculated directly from accuracy.
If they discriminate well, their CBS score is better than their crude "Percent Knowledge".
Though this works well from an examiners' point of view, it was seen by Moodlers as hard for students to understand.
The CBS score and the new "Certainty-Based Accuracy" have similar benefits from the point of view of reliability and validity.
In other respects the older code versions are broadly similar to the new, and are also available (for Moodle 1.9.9 to 2.07)
from the download site.
LAPT (London Agreed Protocol for Teaching)
LAPT (www.ucl.ac.uk/lapt is software developed
at University College London (UCL) for student self-tests with CBM. In many respects it is more efficient and easier to use for this purpose than Moodle, and is freely available for use by other institutions. It presents scores as "Accuracy", "Knowledge" (corrected for guessing), with and without the "CBM Bonus" for good discrimination. The Moodle downloads described above include import/export plugin files for transfer of content between Moodle and LAPT formats.
Links to Moodle / CBM Discussions
Marks, accuracy and CBM 2013
Certainty Based Marking in Moodle (Isabelle Langeveld's blog) 2011
Quiz Report Averages 2011
Testing CBM 2011
Tony Gardner-Medwin (UCL)
Revised March 2013