Using True/False Questions in Assessment
Blog by John Dillon
True-False Weighted Questions is a Game Changer
What is confidence weighting? Ebel (1965) The term confidence weighting refers to a special mode of responding to objective test items and a special mode of scoring those responses. In General terms, the examinee is asked to indicate not only what [they] believe to be the correct answer to a question but also how certain [they are] of the correctness of [their] answer. Ebel ( pg.251).
We see a true-false statement in the screenshot above from the CONFIDENCE game, offered by C3 Softworks. In this game, the student selects an answer, and then must select a confidence rating from the scale. In this case, they could choose five if they are highly confident or a lower rating if they are unsure of the answer. Unlike a traditional quiz, points are deducted for wrong answers based on their confidence level. In the example above, the respondent chose 4 as their rating, because they are more confident that they are answered correctly. Based on this selection, they would earn or lose 80 points depending on their correctness. In the screenshot below, the scale was set to 0-3. Two is the max points at 100, and 0, with no points gained or lost will be selected if they are unsure or just guessing, and don’t want to risk losing points.
Scoring is done in three ways with CONFIDENCE:
1) The standard overall score of correct answers
2) The confidence level of answers selected
3) The points earned
Confidence weighting provides insight into those questions that the respondents answered correctly. With weighting applied, you may see that 70% of the respondents got the question right. However, the confidence level is 40%, which tells you that most of the students were not confident in their answers. This, in turn, gives you a better idea of what they know.
Encouraging Students to Give Honest Responses
Ebel (1965) states that confidence weighting is applied more frequently to the type of test item most seriously affected by guessing, the true-false item. Ebel, (pg49) We want the students to answer the question like they normally would, but we are also asking them to rate their confidence. Scoring is done in three ways: the overall percentage of correct responses, the confidence level of answers selected, and the points earned. In the paper titled “Confidence Weighting and Test Reliability” Ebel (1965) Ebel suggests getting more honest answers from students, you score the results differently. Essentially confidence weighting is intended to get a more accurate assessment of a student's knowledge. Sometimes, the instructor would rather have a student say, “I don’t know,” than take a wild guess at an answer. Ebel (pg 1) BRAVOZONE’s CONFIDENCE handles the overconfident responses by applying negative points if the question is wrong. Sure, a respondent could choose the highest rating for all their responses, but, the incorrect percentages and points will tell a different story.
In this Screenshot from CONFIDENCE from C3 SoftWorks BRAVOZONE, the presenter displayed that 75% of the respondents got the answer correct, but the overall confidence level of the answer selected is 61%.
My first experience with true-false weighted questions happened at a conference over 10 years ago using audience response pads. That experience has stuck with me. First, let's discuss the elephant in the room: Not every instructor is a fan of true or false questions. There are many reasons for this, but probably the biggest is that students have a higher probability of guessing the correct answer in this question format. Because of this, it is hard to determine if students know the content. This makes it more difficult for a presenter or instructor to know what the audience truly knows. While I used to share these beliefs, I have formed a different opinion on this since I learned about confidence-weighted questions. As the title suggests, confidence-weighted questions are a game changer for true-false.
There is no question in my mind that there are definite advantages to using true-false formats. The biggest is the simplicity of question creation.
Compared to multiple-choice items, constructing true-false statements is less laborious than authoring multiple-choice items. The quality of multiple-choice items depends on the plausibility of the distractors. (Dutke & Barenberg, 2015,p. 251)
True or false questions can quickly test knowledge and inspire more critical thinking than other question styles. Creating good multiple-choice questions is a skill, and one needs to learn many good rules of thumb which I won’t address in this blog. I have seen many assessment questions, on topics I have no knowledge of, but found I could answer many questions correctly, because of the poor structure of the questions. On the other hand, a true-false statement is one that is either yes or no.
Final Thoughts
It is important to remember that true-false questions have their place. They are best used, but not limited to, learning definitions and correct statements, and they are much easier for a student to process. But like any tool, you need to know your objective up front. By combining the advantages of creating true-false questions and the ease of the respondents processing the questions with confidence weighting, many instructors may change their opinion of the value of using true-false statements in their training. After all, the goal is to find ways to find out what your students or audience knows.
Tips for Developing True-False Questions
Avoid using negative constructions in questions. What is an example of a negative contraction?
A negative contraction is a negative verb construction that ends in -'nt.
Every question must present itself as an unequivocally true or false statement that does not allow exceptions.
Keep the proportion of false statements slightly higher than factual statements. Students tend to guess “true” on uncertain questions.
Avoid trivia. Make sure that your actual/false questions directly assess learning goals.
Keep your questions short and to the point. Never cover multiple ideas or notions with a single true/false question.
Do not introduce ambiguity to your questions by using words such as “sometimes,” “the majority,” “most likely,” “usually not,” “best,” “worst,” etc.
References
Dutke, S., Barenberg, J (2015). Easy and Informative: Using Confidence-Weighted True–False Items
Knowledge Tests in Psychology Courses. Psychology Learning & Teaching 2015, Vol.
14(3) 250–259. DOI: 10.1177/1475725715605627
https://journals.sagepub.com/doi/pdf/10.1177/1475725715605627
Dutke, S. (2015). Easy and Informative: Using Confidence-Weighted True–False Items for
Knowledge Tests in Psychology Courses. Psychology Learning & Teaching 2015, Vol.
14(3) 250–259. DOI: 10.1177/1475725715605627
https://journals.sagepub.com/doi/pdf/10.1177/1475725715605627
Ebel, R. (1965) Confidence Weighting and Test Reliability
1 Paper read at the Annual Meeting of the National Council on Measurement in Education,
Chicago, Illinois, February 1965. https://doi.org/10.1111/j.1745-3984.1965.tb00390.x
Sparck, E.M., Bjork, E.L. & Bjork, R.A. On the learning benefits of confidence-weighted testing.
Research 1, 3 (2016). https://doi.org/10.1186/s41235-016-0003-x
TRY IT in your next training session
Get a FREE 14-day unlimited license to use in your next training session. Try it in the classroom, on a webinar, or in an eLearning session and see for yourself how powerful gamified training can be.