The assessment of students' subjective estimates of the probability that they are giving correct answers is a potentially valuable educational tool (Shuford et al., 1966; Gardner-Medwin, 1995). It helps students identify when they have only a tentative grasp of a subject, and it encourages the habit of checking conclusions and re-reading the question. We have employed this technique, loosely called "confidence assessment", in computer-aided exercises in physiology and allied subjects over two years.
Students are asked true/false questions (e.g. "ADH increases urine osmolarity, T/F?") or open questions (e.g. "What is a typical g.f.r. in man?"). After each answer is entered, the student is asked for his/her confidence level on a scale of 1, 2 or 3. If the answer is correct, then this is the mark awarded. If not, marks of 0, -2, or -6 are awarded. Students are advised to use the upper confidence levels only when their subjective probability exceeds the thresholds (67%, 80%) that are appropriate to maximise expected scores.
Data were analysed for 130,000 presentations of questions to UCL and CXWMS students. 91% of the questions were answered. 77% of the answers were correct. Confidence levels C=1,2,3 were assigned to 20%, 25% and 54% of the answers, respectively. The percentages of correct answers at each level were 60%, 73% and 84% (within the optimal ranges). Statistics for individual questions reveal information about particular topics that can be used to improve teaching programmes. Some answers were both correct and confident (e.g. "Conduction velocity is decreased in demyelinating diseases, T/F?": 95% correct; 88% C=3). Others were largely correct but unconfident (e.g. "Tissue fluid formation decreases when tissue pressure increases, T/F?": 92% correct; 50% C=3). Serious problems were revealed where answers were both wrong and confident, e.g. "Electric current in tissue is due to flow of electrons, T/F?": 63% at C=3, of which 63% were incorrect; "1ml = 1 cubic millimetre, T/F?": 39% at C=3, of which 71% were incorrect. Such topics are ones where students do not appreciate their problems and, unless given assistance, must be disinclined to practice or study. Analysis of student questionnaires suggests that students benefit from the additional feedback and regard the time taken to think about and enter their confidence as worthwhile.
Several exercises used in this study were written by colleagues at UCL and CXWMS. The programs and exercises (LAPT: London Agreed Protocol for Teaching) are available by File Transfer Protocol from ftp.ucl.ac.uk in the directory pub/users/cusplap.
Gardner-Medwin, A.R.(1995) Association for Learning Technology Journal 3:80-85
Shuford E.H., Albert A. & Massengill H.E. (1966) Psychometrika 31:125-145
Go to LAPT Home Page .
Queries / Problems: e-mail to cusplap@ucl.ac.uk