I want te be able to put my answers into the same files as the question for easy reference and having the obligation to have a correction template (on paper)
for all exams. I think it is best to keep things together.

Alexis pointed me to the availability of the \ifAMC@correc flag, which is set when in catalog or answer mode.

Because of the @ character it is a bit tricky to use in a normal .tex file.
Thus the solution I choose is to query the flag early in the tex processing and set a special toggle which is easily used in normal LaTeX.
I do that in a \usepackage{} file I use anyway and created it to avoid this @ handling.
You need to use \usepackage{etoolbox} and then in a style file or a \makeatletter ... \makeatother block pick up the setting like so

\newtoggle{AMCAnswers}
\ifAMC@correc\toggletrue{AMCAnswers}\else\togglefalse{AMCAnswers}\fi

Then use it like

    \iftoggle{AMCAnswers}{
    Answers, for the correction template

    }{
    question box, for instance a figure or a lined box or some such.

    }

It is probably best to keep the stuff you have LaTeX render to be the same vertical size, to avoid any problems or different pagination in the generated pdf files.
something like using a box with a fixed height in both cases.