Now that the project to upgrade my old multiple choice applet to a more modern and collaborative format is underway (see this server-side demo and this javascript/wiki demo, as well as the discussion here), I thought it would be a good time to collect my own personal opinions and thoughts regarding how multiple choice quizzes are currently used in teaching mathematics, and on the potential ways they could be used in the future.  The short version of my opinions is that multiple choice quizzes have significant limitations when used in the traditional classroom setting, but have a lot of interesting and underexplored potential when used as a self-assessment tool.

— Multiple choice quizzes in the classroom —

In principle, it would seem that the unambiguous and precise nature of mathematical statements would lend itself well to the multiple choice format; in contrast to some other disciplines of knowledge, many questions in mathematics do have a single and objective correct answer, with all other answers being agreed upon as being incorrect.  With a multiple choice quiz, a student can be tested on such questions in an objective manner; indeed, the grading for such quizzes can even be automated to be done by a computer or scanning machine.  As long as the question was phrased unambiguously (and that the solution key is correct), the grading of such quizzes is less subject to dispute than other means of examination.  As a final positive, the multiple choice format is extremely familiar to virtually all college students (who have probably had to have taken standardised multiple choice tests as part of the university admission process) and so the rules of the quiz require very little explanation.

On the other hand, the multiple choice format, as it is currently used in maths exams, has a number of serious weaknesses which, in my opinion, render it inferior to other examination options for most upper-division maths courses, although there are ways to remove the most glaring defects of the format.  Perhaps the most obvious problem is the zero-tolerance approach to mistakes, which can distort the relationship between aptitude and credit: a student who had the right approach to a question, but made a single sign error or misunderstood the question slightly, could lose all points for a question, whereas a student who had no clue whatsoever what to do, and is simply guessing randomly, could manage to earn credit for a multiple choice question by pure luck, which is much harder to achieve in other examination formats.  (Admittedly, one can mitigate this problem by keeping the questions simple and unambiguous, and ensuring that the incorrect answers obtainable from sign errors and the like are not given as one of the alternatives.)  Another issue is that multiple choice quizzes are more susceptible to certain types of cheating and corruption than other examination formats, since the answer key is easy to copy and use, even by students who do not actually understand the material.  (This particular problem can be guarded against to some extent by shuffling the questions separately for each student, though this of course makes it more difficult to grade the quiz, or to provide a solution key afterwards.)  A third problem is that if the student arrives at an answer that is not among the options listed, this often encourages a rather tortured and not particularly logical fudging of the computation in order to arrive at one of the listed answers, which is not a good habit to instill in a mathematician.

A more insidious problem, however, is that these quizzes give a misleading impression of what mathematical problem solving is, and how one should go about it.  In actual mathematical research, problems do not usually come with a list of five alternatives, one of which is correct; often, figuring out what the potential, plausible, or likely answers could be, or even what type of answers one should expect or whether one should ask the question at all, is as important as actually identifying the correct answer.  Multiple choice quizzes also tend to reward quick-and-dirty or sloppy approaches to problem solving, as opposed to careful, deliberate, and nuanced approaches; in particular, such quizzes tend to encourage the mindless application of formal rules in order to arrive at an answer, without devoting much thought as to whether these rules are actually applicable for the problem at hand.  (Indeed, overthinking a multiple quiz problem, searching for some subtle trick, degeneracy, or exception in the wording of the problem, or trying to play some sort of “metagame” in which one is trying to divine the intent of the examiner [see this scene from “The Princess Bride” for an extreme example of this], can mean that the brighter students who actually understand the material can end up doing worse on the quiz than those who are simply applying the rules they are taught without that understanding.  Conversely, an overly tricky quiz problem that is designed to trap those who carelessly apply a standard rule without checking to see that it applies will usually be (quite rightly) perceived as being rather unfair on the student.)  While some drill of basic formal rules (e.g. the laws and algorithms of calculus) is certainly necessary, especially at the high-school and lower-division undergraduate levels of mathematics, by the time one transitions to upper-division mathematics, one needs to start understanding the underlying theory and justification behind such rules, as part of developing a more conceptual grounding in the subject.  (Also, when one gets to more advanced topics, there tend to be so many exceptions and weaknesses to any given rule that it becomes quite dangerous to apply such rules mindlessly.  For instance, computing an integral by unilaterally shifting contours is quite likely to give an incorrect answer if one does not have a good sense of which contour integrals will safely go to zero in some limit, and which ones do not.  It is self-defeating to look for some easily memorised rule that will say which integrals are safe and which ones are not, as there are so many different variations, especially in real-world applications; the only reliable way to proceed is to actually understand the business of estimating integrals and computing limits.)

But perhaps most of all, multiple choice questions promote the idea that the answer to a mathematical question is more important than the process used to arrive at that answer (and the insights acquired during that process, and the art of communicating that process effectively to others).  In truth, the process is far more important than the answer, particularly if the answer is to an artificial question, such as one designed specifically for examination purposes.  Knowing the thought processes that were used by a student to arrive at an answer – even an incorrect one – would give me a detailed picture of how well equipped the student would be to handle similar (or more complicated) questions in the future, whereas the mere knowledge that the student selected one correct answer out of five alternatives gives me much less information in this regard.  This also allows for much more valuable feedback in the grading process than simply reporting whether a given question was answered correctly or incorrectly, by identifying specific strengths and weaknesses in a student’s reasoning.

— Multiple choice quizzes as self-examination —

I have discussed my reservations about the use of multiple choice quizzes in classroom examinations, particularly in upper-division mathematics courses.   On the other hand, I do feel that such quizzes can play a very useful supporting role in self-examination for such courses, particularly with regards to foundational material (e.g. definitions or basic rules of calculation).   I will illustrate this with a hypothetical course in high school algebra, though the point is certainly applicable to more advanced mathematical courses.

Suppose this algebra course is intended to teach students how to solve various algebraic equations.  There are of course several standard pitfalls that the student encounters when actually trying to solve such equations; a common one is starting with an equation such as x^2=y and concluding incorrectly that x = \sqrt{y}, when instead the best one can say is that x = \sqrt{y} or x = -\sqrt{y}.  Now, one can caution against this error in classes, and the student may even write down this warning when taking notes, but it still happens all too frequently that this error is committed while solving a more complicated algebra problem in an examination (or worse, in a real life application of one’s high school algebra).   At that point, the student may well realise the cause of the error – but this feedback may come days or weeks after first learning the material; without reinforcement, the same error may then recur later in the course, or in subsequent courses.  Repeated exposure to algebra will eventually eliminate the error, but it can be an inefficient process.

This is where a self-administered multiple choice quiz (in particular, an online quiz) can help, with questions such as

Question 1. If x and y are real numbers such that x^2=y, then the best we can say about x is that

  1. x = \sqrt{y}.
  2. x = y^2.
  3. x = y^{-2}.
  4. x = \sqrt{y} or x = -\sqrt{y}.
  5. x = y^{-2} or x = -y^{-2}.

mixed together with variants such as

Question 2. Let x and y be real numbers.  Which of the following statements is not sufficient to imply that x^2=y?

  1. x = \sqrt{y}.
  2. x = - \sqrt{y}.
  3. x is either equal to +\sqrt{y} or -\sqrt{y}.
  4. x = y^2.
  5. y = x^2.

and

Question 3. If x and y are real numbers such that x^3=y, then the best one can say about x is that

  1. x = \sqrt{y}.
  2. x = y^{1/3}.
  3. x = +y^{1/3} or x = -y^{1/3}.
  4. x = y^{-3}.
  5. x = y^{-3} or x = - y^{-3}.

Such questions can address quite directly (and with immediate feedback) whether one has any misunderstanding on this specific point, without needing the live intervention of a lecturer or teaching assistant.  (Ideally, an automated quiz should not only respond immediately as to whether the selected answer was true or false, but also to explain what the error was in the latter case.)

Note some differences between these sorts of multiple choice questions and the ones that appear in a classroom examination.  In an exam setting, one usually wants to have more complex questions that test several aspects of the material at once (e.g. factorisation, gathering terms, substitution, etc.) rather than focusing narrowly and simply on a single aspect.  (In particular, a student who actually knows the material should be able to answer each question here readily, without the need for significant computation.) Also, while classroom quizzes take pains to make the correct answer quite distinct from the incorrect alternatives (to separate those who basically understand the material from those who are truly lost), it is more effective for self-examinations to have only quite subtle differences between the correct answer and the other answers, in order to encourage the student to think carefully and to address any misconceptions head-on; these kinds of “trick questions” would be rather unfair in the stressful environment of an assessed classroom exam, but can be safely administered in a self-examination.

Multiple choice questions seem most effective here for reinforcing the precise definition of a key concept (was it “For every \varepsilon there exists a \delta“, or “for every \delta there exists an \varepsilon?”), the precise formulation of some rule (is the derivative of f/g equal to (fg'-gf')/g^2, or (f'g-gf')/g^2, or (f'g-gf')/f^2, etc.?), or the direct testing of a specific and commonly made error (if x<y, does this imply that -x<-y, or -x>-y?  See also this list of common errors in college maths).  But with a bit of imagination, one could come up with some useful multiple choice questions for self-examination for other purposes, even for rather advanced maths topics.  For instance, consider the following question to test one’s grasp of the properties of the Fourier transform:

Question 4. Let f: {\Bbb R} \to {\Bbb C} be a function.  Among all the hypotheses listed below, which one is the weakest that still implies that the Fourier transform \hat f: {\Bbb R} \to {\Bbb C} exists and is continuous?

  1. f is smooth and rapidly decreasing.
  2. f is absolutely integrable.
  3. f is square-integrable.
  4. f is continuous.
  5. f is continuous and compactly supported.
  6. f is a tempered distribution.

The type of knowledge in Fourier analysis that this question is probing seems difficult to examine by other types of questioning (other than an oral examination).

Another interesting possibility is to use multiple choice quizzes to explore specific mathematical problem solving tactics, which is an issue which is only indirectly addressed by most examination methods.  For instance, in a single-variable calculus course, one could focus on integration tactics, using questions such as these:

Question 5. Which of the following techniques would you feel to be a good first step towards finding an antiderivative \int x^2 \log(1+x)\ dx of the function x^2 \log(1+x)?

  1. Integration by parts, differentiating x^2 and integrating \log(1+x).
  2. Integration by parts, differentiating \log(1+x) and integrating x^2.
  3. Substitution, setting y = x^2.
  4. Substitution, setting y = 1+x.
  5. Substitution, setting y = \log(1+x).
  6. Trial differentiation, using functions such as x^3 \log(1+x).
  7. Sketching a graph of x^2 \log(1+x).
  8. Taylor series expansion of \log(1+x).
  9. Start up Maple, Mathematica, or SAGE. :-)

Note that this question is of a more subjective nature than the preceding questions, with different answers having different strengths and weaknesses; there is no single “correct” or even “best” answer here.   As such, this would be a terrible question to place in an assessed exam, but I think it would be a good thought-provoking question to give in a self-administered quiz.  (This would be one example of a question where the process of arriving at one’s chosen answer is definitely more valuable than the answer itself.  Also, having a place to discuss the various answers to a question such as this – as would be the case, for instance, if the question was hosted on a wiki – would also add an extra dimension to this exercise.)  Note the difference between the above question and the more traditional “Compute the antiderivative of x^2 \log(1+x).”; the emphasis is now on tactics rather than computation.

In summary, I believe that there are a number of interesting ways – many of which appear to be underexplored at present – in which some well-designed and self-administered online multiple choice questions can efficiently assess one’s strengths and weaknesses in a given mathematical subject.  Of course, having one-on-one interaction with a lecturer or teaching assistant would be a greatly preferable way to achieve this sort of instant feedback, but this is impractical for larger classes.  It is also true that a certain level of maturity and discipline is needed on the student’s part in order to actually benefit from these sort of self-assessments, especially since they are not directly contributing to the student’s grade in the class, but my philosophy here is to give the students the benefit of the doubt in this regard; I feel that being able to explore beyond the bare minimum of what is needed to obtain a passing grade is part of what an upper-division course should be about.