Creating multiple choice questions

The use of multiple choice tests are attractive to instructors for many reasons. From a time management perspective, multiple choice tests are very practical – particularly in large class settings. Other advantages include:

Versatility: Multiple choice test items can be written to assess various levels of learning outcomes, from basic recall to application, analysis, and evaluation. Because students are choosing from a set of potential answers, however, there are obvious limits on what can be tested with multiple choice items. For example, they are not an effective way to test students’ ability to organize thoughts or articulate explanations or creative ideas.

Reliability: Reliability is defined as the degree to which a test consistently measures a learning outcome. Multiple choice test items are less susceptible to guessing than true/false questions, making them a more reliable means of assessment. The reliability is enhanced when the number of MC items focused on a single learning objective is increased. In addition, the objective scoring associated with multiple choice test items frees them from problems with scorer inconsistency that can plague scoring of essay questions.

Validity: Validity is the degree to which a test measures the learning outcomes it purports to measure. Because students can typically answer a multiple choice item much more quickly than an essay question, tests based on multiple choice items can typically focus on a relatively broad representation of course material, thus increasing the validity of the assessment. (Brame, 2013)

A multiple-choice question consists of a stem (question or problem) and a list of possible answers (alternatives) containing the best answer to the question and a number of conceivable but incorrect answers. Students respond to multiple choice question by indicating the alternative that they believe best answers or completes the stem. Here are some tips for designing effective stems and alternatives:

• Express the full problem in the stem. When creating the item, ask yourself if the students would be able to answer the question without looking at the options. This makes the purpose of the question clear.
• Put all relevant material in the stem. Do not repeat in each of the alternatives information that can be included in the stem. This makes options easier to read and understand, and makes it easier for students to answer the question quickly.
• The stem of the question should clearly indicate what the student is to do (e.g., identify the best answer, find the most recent accomplishment, identify the answer with the best order of events, etc.). Often, poorly worded questions do not clearly state what the student is to do.
• The stem should be in the form of a question and be worded positively if possible. Irrelevant material should be avoided.
• Eliminate excessive wording and irrelevant information from the stem. Irrelevant information in the stem confuses students and leads them to waste time.
• Limit the number of alternatives. Use between three and five alternatives per question. Research shows that three-choice items are about as effective as four or five-choice items, mainly because it is difficult to come up with plausible distractors.
• Make sure there is only one best answer. Avoid having two or more options that are correct, but where one is “more” correct than the others. The distractors should be incorrect answers to the question posed in the stem.
• Make the distractors appealing and plausible. If the distractors are farfetched, students will too easily locate the correct answer, even if they have little knowledge. When testing for recognition of key terms and ideas keep the distractors similar in length and type of language as the correct solution. When testing conceptual understanding, distractors should represent common mistakes made by students.
• Make the choices grammatically consistent with the stem. Read the stem and each of the choices aloud to make sure that they are grammatically correct.
• Place the choices in some meaningful order. When possible, place the choices in numerical, chronological or conceptual order.
• Randomly distribute the correct response. The exam should have roughly the same number of correct answers that are a’s, b’s, c’s, and d’s (assuming there are four choices per question).
• Avoid using “all of the above”. If “all of the above” is an option and students know two of the options are correct, the answer must be “all of the above”. If they know one is incorrect, the answer must not be “all of the above”. A student may also read the first option, determine that it is correct, and be misled into choosing it without reading all of the options.
• Avoid using “none of the above”. The option “none of the above” does not test whether the student knows the correct answer, but only that he/she knows the distractors aren’t correct.
• Refrain from using words such as always, never, all, or none. Most students know that few things are universally true or false, so distractors with these words in them can often be easily dismissed.
• Use the words ‘best answer’ rather than ‘most correct answer’ as there may be exceptions and this phrasing will avoid any arguments.
• Avoid overlapping choices. Make the alternatives mutually exclusive. It should never be the case that if one of the distractors is true, another distractor must be true as well.
• Avoid questions of the form “Which of the following statements is correct?” There is no clear question being asked, and the choices are often heterogeneous. Such questions are better presented in the form of True/False questions.
• Use capital letters (A. B. C. D.) rather than lower case letters (e.g., “a” gets confused with “d” and “c” with “a” for those with vision problems, poor photocopying, dyslexia, etc.)
• Make all responses fairly equal in length. Avoid making the correct response either the longest or the shortest in length.
• If “no” or “not” is used, underline it. Try to avoid using negative constructions in the stem.

Sources

Centre for Innovation and Excellence in Teaching, Vancouver Island University. (n.d.). Writing Effective Quiz Questions. Retrieved from: https://ciel.viu.ca/learning-technologies-innovation/viulearn/assessment-tools/writing-effective-quiz-questions

Designing multiple-choice questions. Centre for Teaching Excellence, University of Waterloo. Retrieved from: https://uwaterloo.ca/centre-for-teaching-excellence/teaching-resources/teaching-tips/developing-assignments/assignment-design/designing-multiple-choice-questions

The following is an example of a bad multiple choice question, and the reasons why, from Roy Bishop’s article in The Teaching of Astronomy.

Q: Space debris is of most interest to astronomers for which one of the following reasons?

1. It contains much gold and silver.
2. It all came from the Moon.
3. It all came from comets.
4. It tells us about the early Solar System.
5. It tells us about the last ice age on Earth.

Comment: The question itself is not good because the term “space debris” is not defined (fragments of satellites in near-Earth orbit? Interplanetary dust and meteoroids?) Also, answers A, B, and E should be obviously incorrect to even a relatively poor student, so a guess at the remaining answers would mean a 50% chance at choosing the correct answer. 

The following is an example of a good multiple choice question, and the reasons why, again from Roy Bishop’s article in The Teaching of Astronomy.

Q: Suppose you are in Williamstown. If, towards the northwest, you see a first quarter Moon near the horizon, what month is it?

1. August
2. March
3. December
4. June
5. September

Comment: This question is not trivial, and it does not involve regurgitation of standard material. The student must know about: i) the phases of the moon and the associated Sun-Earth-Moon geometry; ii) the inclination of the ecliptic; iii) that the Moon’s orbit lies (approximately) near the ecliptic; iv) the possible range of compass directions of the intersection of the ecliptic with the horizon at mid-northern latitudes; and v) how the Sun’s position to the ecliptic is related to the time of year. Furthermore, the student must be able to correlate these various things and fir them to the question. In brief: A low Moon in the northwest means that the summer solstice point (most northerly point) of the ecliptic must be in that vicinity. The first quarter phase means that the sun must be about 90 degrees further westward along the ecliptic…near the spring equinox. Therefore, the answer is B.

Source: Bishop, R. L. (1990). Multiple Choice Questions. The Teaching Of Astronomy. 83-87

In the following examples of effective and ineffective multiple choice questions, students explore potential energy, or the energy that is stored by an object.

#1. Good Stem, Poor Distractors

Potential energy is:

1. the energy of motion of an object.
2. not the energy stored by an object.
3. the energy stored by an object.
4. not the energy of motion of an object.

In this question the good stem is clear, brief, and presents the central idea of the question through positive construction. However, the distractors are confusing: b) and d) are written in negative constructions that force students to reinterpret the stem, while c) and d) have overlapping, inconsistent content that confuses and tests reading comprehension over content recall. Finally, choices do not move logically by grouping content, failing to visualize and test larger concepts for students.

#2. Poor Stem, Good Distractors

Potential energy is not the energy:

1. of motion of a particular object.
2. stored by a particular object.
3. relative to the position of another object.
4. capable of being converted to kinetic energy.

In this question the poor stem contains the word “not,” which fails to identify what potential energy is, and tests grammar over student understanding. However, the good distractors are written clearly, cover unique content, and follow a logical and consistent grammatical pattern.

#3. Good Stem, Good Distractors

Potential energy is:

1. the energy of motion of an object.
2. the energy stored by an object.
3. the energy emitted by an object.

In this example both the stem and the distractors are written well, remain consistent, and test a clear idea.

Source:

Yale Center for Teaching and Learning. (n.d.). Designing quality multiple choice questions. Retrieved from: https://ctl.yale.edu/MultipleChoiceQuestions

Workshops on a variety of tools, including the quiz tool to develop multiple choice questions, are available for sessional instructors, instructors, librarians, pre- and tenured faculty members.

Tap here for UM Learn workshop times and locations. When attending a UM Learn workshop please bring your own laptop.

You can also find online support for developing quizzes through UM Learn.

• Login at UM Learn
• Tap Support to reveal the drop down menu (top right hand of the screen)
• Tap Help
• In the Search Topics text box (top left hand corner), type “quiz”
• Scroll down to Quizzes – Campus
• Choose from a variety of videos to help you develop your quiz
• For a more in depth understanding, scroll to the Assessments – Instructor Guide (pp. 71 – 102)

14 Rules for Writing Multiple-Choice Questions. Timothy W. Bothell, Brigham Young University. 2001.

Brame, C., (2013) Writing good multiple choice test questions. Retrieved [todaysdate] from https://cft.vanderbilt.edu/guides-sub-pages/writing-good-multiple-choice-test-questions/.

Centre for Innovation and Excellence in Teaching, Vancouver Island University. (n.d.). Writing Effective Quiz Questions. Retrieved from: https://ciel.viu.ca/learning-technologies-innovation/viulearn/assessment-tools/writing-effective-quiz-questions

Centre for Teaching Excellence, University of Waterloo. (n.d.). Designing multiple-choice questions.Retrieved from: https://uwaterloo.ca/centre-for-teaching-excellence/teaching-resources/teaching-tips/developing-assignments/assignment-design/designing-multiple-choice-questions

Writing Multiple-Choice Questions that Demand Critical Thinking. Georgeanne Cooper, University of Oregon. 2007 

Yale Center for Teaching and Learning. (n.d.). Designing quality multiple choice questions. Retrieved from: https://ctl.yale.edu/MultipleChoiceQuestions