Confidence marking

Added by Marc Eberhard over 6 years ago

Dear all,

while looking into hMCQs I stumbled across a suggestion to allow students to state their confidence level in answering a question, which then affects the mark they can obtain. The example given was that students can tick "Low", "Medium" or "High" as their confidence level for each individual question. The marks given are then:

        | Low | Medium | High
--------+-----+--------+-----
Correct |   +1|     +2 |  +3
Wrong   |    0|     -2 |  -6

One can argue about the specific numbers used, but I do like the idea that the marks depend on how certain a student believes in having given the correct answer.

The big question is how this can best be implemented in AMC? I'm struggling to think of a way how this could be done, especially with a separate answer sheet. Any suggestions or a short LaTeX template would be great!

Thanks,
Marc


Replies (7)

RE: Confidence marking - Added by Alexis Bienvenüe over 6 years ago

See attached files for a starting point, using development version of AMC.
For each question, I created two AMC questions (hiding the "Question" header for the second one): the first for the confidence level, and the second for the question itself.
You have to adapt the code for questionmult, depending on your scoring strategy.

confidence_standard.tex - without separate answer sheet (1.4 kB)

confidence.tex - with separate answer sheet (1.8 kB)

RE: Confidence marking - Added by Marc Eberhard over 6 years ago

Dear Alexis,

many thanks for the quick reply, that's a great starting point!

I'm using version 1.3.0 from AUR on Arch. Does the development version include any additional features needed here or should this work with the version I use?

Trying the example I get:

Marks Dialog

It does pick up the confidence level, but doesn't seem to calculate a mark for the actual question. No matter which box I tick it returns a 0 mark. So, I do wonder if setglobal.SCOREOK and setglobal.SCOREWRONG don't work properly. If I replace them with +1 and -1 in \scoring for the actual question I do get those values back. It seems that both default to 0 in my case and are not passed from the confidence part to the actual question. Unfortunately, I'm not enough of an expert on the inner workings of AMC to debug this. Is there anything I can do to try to trace why the values don't get passed through?

All the best,
Marc

Marks.png - Marks Dialog (12 kB)

RE: Confidence marking - Added by Alexis Bienvenüe over 6 years ago

I'm using version 1.3.0 from AUR on Arch. Does the development version include any additional features needed here?

Yes. The key point is the use of the setglobal.XXX directive, that was recently introduced with revision hg:1815c3151934. See #273.

RE: Confidence marking - Added by Marc Eberhard over 6 years ago

Ah! I have now applied the diff to 1.3.0 and it sort of works...

The mark values for a correct answer at certainty low (0) and high (2) seem to have been swapped. Am I doing something wrong here?

Also, the layout of the answer sheet isn't great:

Is there a way to influence the layout and have the answers first and the certainty levels after them in one line like


Question 1: |A| |B| |C| |D| |E|    |L| |M| |H|

Marks.png (10.4 kB)

confidence.tex (3.9 kB)

Layout.png (32.5 kB)

RE: Confidence marking - Added by Alexis Bienvenüe over 6 years ago

The mark values for a correct answer at certainty low (0) and high (2) seem to have been swapped. Am I doing something wrong here?

I'm afraid that setglobal.XXX was not correctly tested: this is a bug, that should be fixed with hg:2f5ae49dfa71. Can you test it?

RE: Confidence marking - Added by Alexis Bienvenüe over 6 years ago

I'm afraid it is not possible to get confidence level after answers, but you can get a single-line layout with (right after the \def\AMCformQuestion#1{...} line)

\def\AMCformBeforeQuestion{\ifnum\value{AMCquestionaff}<0\hspace{1em}\else\vspace{\AMCformVSpace}\par\fi}

single_line.png (10.4 kB)

RE: Confidence marking - Added by Marc Eberhard over 6 years ago

Brilliant, this all works now as far as I can tell!

I'm attaching my latest test template for future reference for others that might want to try this marking approach too and the original research paper "Confidence-Based Marking - towards deeper learning and better exams" by A.R. Gardner-Medwin, Dept. Physiology, University College London.

Many thanks Alexis for the quick responses and bug fixes!

All the best,
Marc

confidence.tex - Template for certainty based marking scheme (4 kB)

Certainty_Based_Marking.pdf - Research Paper on Certainty Based Marking (186.8 kB)

(1-7/7)