Report on first experience

Added by Michaël Cadilhac about 1 year ago

Hi there all,

I ran my first exam with AMC; it was 50% quiz, 50% problems, and I used a separate score sheet. There were 5 variants of the exam, distributed at random to 50 students.

I've hit a few roadblocks, shouted a bit, and suffered at times, but it went well. I'm reporting here on my experience — some of these items may be turned into feature requests, some bug reports, and I'd appreciate your input on it. This list will look like an airing of grievances, but really, I enjoyed my experience overall and I'm looking forward to using AMC again, in a more proficient way. Thanks for a great piece of software.

1. I graded the problems myself on a tablet using xournalpp, and reported the grade on the separate score sheet (an example is attached). Sometimes, the annotation did not get read by AMC, and I had to use the dreadful "convert in.pdf out.pdf" to rasterize the whole file myself. I'm glad I caught this but that wasn't obvious.

2. I was frustrated to have to rescan a handful of papers just because a black speck was here or there and AMC got confused. I would really have liked to be able to tell AMC: Here are the dots.

3. Speaking of, while going through the "reject" pile, it would have been quite helpful to have the filenames given, so I know what to rescan. The input filenames contain the name of the user; 1/24:3 says nothing to me. Similarly, some students forgot to put their name in the name box (the answer sheet); it would have been really useful to get the filename displayed in the script-to-name dialog. Also, when manual-marking the problems, I had the scoresheet opened in AMC and I was opening on the side the script of the student to read their answer; making the connection between the script in AMC and the PDF wasn't easy.

4. About reviewing the automarking; my main concern was that a student darkening of the box wasn't enough (or slightly off) or that they had filled two boxes (usually indicating that they changed their mind). I would have loved to have access to all questions of all copies that were either double-filled or left empty. Going through these would have made point 1. above much more obvious.

5. In a similar fashion, one copy got their scoresheet scanned as the wrong page number; I couldn't see a flag raised somewhere that that student didn't get their scoresheet scanned. In the script-to-name dialog, I had filled out all the matches but it was still displaying "One copy is not associated to a student name". Tracking which one was a pain.

6. When matching names to scripts, I was typing the name of the student, then when it was the only name matched, I had to click on it. Hitting "Enter" rather than clicking would have been much more comfortable.

7. In terms of confidence in the autoscan, I would have loved a dialog in which the autoscanned page number/exam number were displayed next to the script scan, so that I could check it was correctly matched. This goes hand in hand with 2.: Sometimes, AMC makes a mistake, or fails to scan a script for a reason or another, and being able to scan just one script entirely manually or to edit AMC's result would be very useful.

8. When manual marking, I wanted to go through a specific quiz question where I messed up the correct answer, and change the students answers (e.g., if they had put A I'd change it to B). But I couldn't use the "focus on specific question" in a single score sheet situation, because it wasn't zooming on that question, it was bottoming out on the screen (so that the whole bottom half page was shown, and I had no clue which quiz question I was supposed to look at).

9. When exporting the grades to CSV, I'd have like to see the whole grade, not out of 20. Further, a comma is used as the decimal point, which is incorrect in English (the locale should be used to decide the decimal separator).

Sorry for the lengthy list, I know it's bad taste in a forum…

Hope we can solve some of these before my next exam in the Fall!


example.pdf (49.9 kB)

Replies (1)

RE: Report on first experience - Added by Gregory F about 1 year ago

Hi Michaël,

Welcome to AMC users community :-)

I hope you enjoyed the first experience though there is a learning curve and some bumps, hence your long list :-)

I have been using AMC for the past 8 years now, I am still struggling at time but I always find an answer and features I didn't know existed and that would have made my life much easier if only I knew.

I'll try to reply to some of your points to the best of my abilities. Bear in mind I am only a simple user here, just like you. And sorry for the huge screenshots, I don't know how to make them smaller.

1. Yes, we have to flatten the PDFs if marks were applied on a device rather than directly on paper. I this this is more related to the PDF format that AMC. Perhaps a box to flatten the PDF during the capture process could be an option. Not sure if this one could help

2. Not sure I am getting this. From my personal experience, I only had a few cases of missed recognitions due to wrapped pages or students drawing around the circles.

3. Why were the copies rejected? Normally when your inspect the "unrecognized scans", you should have the information of filename and page number:

If you are talking about missing pages, I spent a lot of time in the past searching PDF files for copy number (needle in hay stack), until I discovered that you can display the filename in the list of captures pages. That made my life much easier :-)

4. On this one, I will explain my process for marking:
- After applying the marking, I open the manual capture screen
- I display the scan and I sort by "sensitivity", from highest to lowest:

- Then I process all the highly sensitive pages and check from badly recognised boxes (I think I have a pretty efficient threshold, but this can be adjusted in settings). HINT: Select the first page and click the right arrow once. Then you can press SPACE to move between pages.
- Once I reach a relatively low sensitivity (around .5 or lower depending) I switch to "I&E" to only focus on Invalid or empty questions.

5 - 8. not sure I can be really helpful with this :-)

9. You can change the decimal delimiter in the settings

As for displaying the score in addition to the mark, that a fair point for a feature request. I never use CSV export myself but I understand the need. The score is displayed in the OpenOffice export.

I hope I was able to answer a few of your questions.