Mastering the Art of Creating Effective Questions

Blog by John Dillon

Have you ever taken a quiz and didn’t know the subject matter, but, because the questions were so poorly written, you got a good score despite not knowing the topic? In this blog, I will highlight a few basic things to help you create better question formats. I will break it down into some basic rules I try to follow when I create a quiz.

The problem with Multiple-Choice

Ironically, the most common question format we encounter, the multiple-choice format, is also the most challenging to create. The quality of multiple-choice items hinges on the plausibility of the distractors (Dutke & Barenberg). Creating distractors requires practice, as coming up with the wrong answers is the real challenge.

The Basics of the Multiple-Choice Format

Let's start with the stem, the heart of a multiple-choice question. It's the part that poses the problem or asks the question. To ensure clarity, it's best to state the stem as a question or an incomplete statement. If you choose the latter, make sure the stem is grammatically correct and the options are similar in grammar, length, complexity, and style.

For Example:

Incomplete Statement
Idaho typically ranks first nationally in ________ production.
a.) potato*
b.) apple
c.) green bean
d.) corn

However, you could make this a question by simply rewording the sentence.

Question Example
Idaho is widely known for what agricultural product?
a.) potatoes*
b.) apples
c.) green beans
d.) corn

Avoid Negative Wording

Students often fail to observe negative wording, and it can confuse them. As a result, students familiar with the material often make mistakes when answering negatively worded questions. Negative questions, also known as odd-one-out questions, are a type of multiple-choice question in which the correct answer is false, surrounded by true distractors. In general, avoid having any negatives in the stem or the options. In the rare cases where you use negatives, emphasize the keywords by putting them in uppercase and bolding or underlining them.

For example:

Which city is NOT a state capital?
a.) St. Paul, Minnesota
b.) Sacramento, California
c.) Springfield, Illinois*
d.) Hartford, Connecticut

Keep Option Lengths Similar

Avoid making your correct answer long or short. In an interesting read from William Poundstone, author of Rock Breaks Scissors: A Practical Guide to Outguessing and Outwitting Almost Everything, noticed that the longest answer on multiple-choice tests was usually correct. "Test makers have to make sure that right answers are indisputably right," he says. "Often, this demands some qualifying language. They may not try so hard with wrong answers." If one choice is noticeably longer than its counterparts, he says it's likely the correct answer. A good rule of thumb is to have all answers in similar lengths.

Place the choices in some meaningful order

Place the options in numerical, chronological, or conceptual order when possible. A better-structured question is easier to read and respond to. Note that the answers below appear in order from low to high. Hint: The answer is 50% of the athletes.

How many of the 10,500 athletes taking part in Paris 2024 will be women?
a.) 4,750
b.) 5,000
c.) 5,250*
d.) 6,350

All/None of the Above Formats

Instructors often use “All of the above” as an answer choice when writing multiple-choice questions, but there are several reasons to avoid this. With four answer choices, where “All of the above” is the correct answer, a student only needs to know two are correct to know that “All of the above” is the correct answer. Additionally, if "None of the above" is an answer choice, the test-takers are only presented with incorrect information. The basic idea is to reinforce the right answers, not remind them of the wrong answers. These questions test partial knowledge and test-taking skills, making guessing easier for the savvy student (Gronlund, 1998). Also, authors tend to use “All of the above” to ensure that the student knows a comprehensive list of items; therefore, it is often the correct answer. As for none of the above, you will never know if students know the correct answer.

True-False Questions

Compared to the multiple-choice format, the easiest question format to create is the true-false question. However, despite their simplicity in creating, most instructors avoid true-false because of the ease of guessing the correct answer. However, we must discuss the benefits of creating true-false formats over multiple choice. First, true-false can quickly test knowledge and inspire more critical thinking than other question styles. Creating a true-false question is simply writing a statement that is either correct or incorrect. I recently created a blog on Using True/False Questions in Assessment and how confidence weighting can give you a better indicator of knowledge, making them a much better option for assessment. What are confidence-weighted questions? R.L. Ebel (1965). The term confident weighting refers to a special mode of responding to objective test items and a special mode of scoring those responses. In general terms, the examinee is asked to indicate not only what [they] believe to be the correct answer to a question but also how certain [they are] of the correctness of [their] answer. R.L. Ebel, 51.

References

Gronlund, N. E. (1998). Assessment of student achievement, (6th ed). Boston, MA: Allyn and Bacon.

Waugh, C. K. & Gronlund, N. E. (2013). Assessment of student achievement (10th ed.). Upper Saddle River, NJ: Pearson44

Ebel, R. (1965) Confidence Weighting and Test Reliability 1 Paper read at the Annual Meeting of the National Council on Measurement in Education, Chicago, Illinois, February 1965. https://doi.org/10.1111/j.1745-3984.1965.tb00390

Haldyna, T. M. and Rodriguez, M.C. (2013). Developing and Validating Test Items. Routledge, New York. ISBN-10: 0415876052.

Haldyna, T. M. (1996). Writing Test Items to Evaluate Higher Order Thinking. Routledge, New York. ISBN-13: 978-0205178759

Poundstone, W. Rock (2015) Breaks Scissors, Little Brown Books, New York, ISBN-10. 0316228052

TRY IT in your next training session

Get a FREE 14-day unlimited license to use in your next training session. Try it in the classroom, on a webinar, or in an eLearning session and see for yourself how powerful gamified training can be.