Using Confidence Weighted Questions as a Tool for Assessment

Blog by John Dillon


Using a Likert Scale or Rating System to Represent the User's Confidence in their Response

Let's discuss how using a Likert scale will help create weighted questions. If you are unsure what a Likert scale is, I have no doubt you have already seen this type of scale but didn’t know the name.  A Likert scale is commonly used in surveys. Since, in this example, we will be rating confidence in response to an answer selected, we are using a scale from 1-5 in the screenshot above. Five is the highest confidence, and one is the lowest confidence. Points are assigned to each level, and unlike a traditional assessment, points will be deducted if the question is wrong. Each rating has a specific number of points; in the screenshot above, one hundred is assigned to 5 and drops to 20 for one.  The scale can be adjusted based on the instructor's preferences. 

What Prevents a Student from Just Picking the Highest Confidence Level?

It is important to note that confidence weighted questions apply to those who got it correct. Unlike traditional question formats, students are penalized points based on two factors: getting it correct and the confidence rating they chose. Unlike traditional scoring, CONFIDENCE will apply negative points or penalize the student for wrong answers. The goal of the Likert scale is to allow the student to rate their confidence, and if they are not confident they are correct, they would be best to choose a lower rating; otherwise, they could lose 100 points.


See a Sample Assessment using Confidence Weighting

I first saw an example of a confidence-weighted question about 10 years ago at a conference, and ever since, I have been intrigued by the concept. I have since found that this is not a new concept in my research. It goes back as far as 1932. And as I understand it, there is so much value in this form of assessment. Robert L. Ebel of Michigan State University, in a 1965 paper titled Confidence Weighting and Test Reliability, explains the basic concept of confidence-weighting questions. 

R.L Ebel (1965) The term confidence weighting refers to a special mode of responding to objective test items and a special mode of scoring those responses. In General terms, the examinee is asked to indicate not only what [they] believe to be the correct answer to a question, but also how certain [they are] of the correctness of [their] answer. R.L. Ebel, 51

Applying a Likert scale to scoring in the training tool, CONFIDENCE

In the C3 SoftWorks' CONFIDENCE game, found in the BRAVOZONE, scores are calculated using three methods: the traditional overall correct answer percentage, the confidence level percentage of questions answered, and the actual points earned. Since CONFIDENCE can be a stand-alone self-paced session or an instructor-led in front of a classroom, this gives the instructor options. In self-paced, the students will answer the questions and rate their confidence, but the most important is the overall percentage of correct responses. After completion, the instructor will access all three scores in the reports. In an instructor-led session, CONFIDENCE can look and feel like a game, and after students answer a question and rate their confidence, the class and instructor will see the percentage of selected answers. Then, the instructor can show the average confidence percentage for that question at any time. In the screenshot below, 75% of the students chose the right answer, but the confidence level of those who got it correct was 61%. The overall percentage point of the correct answers helps, but you now have an additional tool to see how confident they were in their answer selection. The bigger the gap from the overall percent and the confidence level tells you that the students were not that confident in their response even though they got it correct.

Final Thoughts

The confidence-weighted questions format is an excellent way to determine where the knowledge gaps are in training. Traditionally, instructors had to rely on the overall percentage correct; for example, 75% of the students got it right, but of those 75%, how confident were they in their selection? Confidence weighting will give you more information on the respondent's confidence level. As an instructor, this helps me identify the knowledge gaps in the training or areas I need to focus on going forward.  Looking back on my training history, I would have used it to review assignments, pre-tests, overviews after a conference presentation, etc.  And if you want to gamify it, like in a classroom, you can also do that.


References

Dutke, S. (2015). Easy and Informative: Using Confidence-Weighted True–False Items for

Knowledge Tests in Psychology Courses. Psychology Learning & Teaching 2015, Vol. 

14(3) 250–259. DOI: 10.1177/1475725715605627 

https://journals.sagepub.com/doi/pdf/10.1177/1475725715605627

Ebel, R. (1965) Confidence Weighting and Test Reliability

1 Paper read at the Annual Meeting of the National Council on Measurement in Education, 

Chicago, Illinois, February 1965. https://doi.org/10.1111/j.1745-3984.1965.tb00390.x

Sparck, E.M., Bjork, E.L. & Bjork, R.A. On the learning benefits of confidence-weighted testing.    Cogn. Research 1, 3 (2016). https://doi.org/10.1186/s41235-016-0003-x

TRY IT in your next training session

Get a FREE 14-day unlimited license to use in your next training session. Try it in the classroom, on a webinar, or in an eLearning session and see for yourself how powerful gamified training can be.