Thursday, March 05, 2009

First Instinct Fallacy

As a high school teacher in a state with end of course testing, I am a regular administrator of multiple choice tests. So it's worth revisiting an old debate; if uncertain with your answer, is it better to change it or stick with you first instinct? Here's an abstract that may challenge the traditional beliefs:

Most people believe that they should avoid changing their answer when taking multiple-choice tests. Virtually all research on this topic, however, suggests that this strategy is ill-founded: most answer changes are from incorrect to correct, and people who change their answers usually improve their test scores. Why do people believe in this strategy if the data so strongly refute it? We argue that the belief is in part a product of counter factual thinking. Changing an answer when one should have stuck with one’s original answer leads to more “if only…” self-recriminations than does sticking with one’s first instinct when one should have switched. As a consequence, instances of the former are more memorable than instances of the latter. This differential availability provides individuals with compelling (albeit illusory) personal evidence for the wisdom of always following their first instinct, with sub-optimal test scores the result.
One of their studies showed: 25% changed their answer from right to wrong, 23% went from wrong to wrong and 51% changed from wrong to right.


  1. The data lies! The data lies! Aoooooooooooga! Aoooooooooooooga!
    Why? Because this data depends on the fact that there are multiple wrong answers but only one correct answer. In other words, you can only change to a right answer IF YOU BEGAN WITH THE WRONG ANSWER. This is not simple statistics my friend, it's an analysis that is based on two different but linked statistical choices. It's easy to say, "Look, 50% of the switchers ended up right!" But of those 50%, 100% started out wrong. And if you start wrong, it is always statistically favorable to switch. There is no "Right to Right" option because those people didn't switch.
    Let's flip the scenario around and say there are 3 correct choices and 1 incorrect. Your chances of ending up correct if you switch are much smaller compared your overall chance to be correct. You would see very strange results regarding switchers because YOU HAVE A MUCH GREATER CHANCE OF BEING WRONG IF YOU SWITCH. Why? Simple because you have a much greater chance of first choosing a correct answer. I cannot unlink my second choice from the first.
    The solution? Figure out if your original answer is wrong and if it is, switch. Oh wait, if you knew that you wouldn't have picked it in the first place.

  2. ^^^^^^^^^^Show-off!^^^^^^^^^

    Still a very well-explained point. I have nothing to add.

  3. Wow, great point Paul, let me make sure I understood you. So you think the study is flawed because there are always 3 wrong answers and only 1 right one. So there is statistically a greater chance to bubbling the wrong answer than the right one. Is that right?

    Although a great point, does that really change the thesis that people underestimate the benefit of switching your answer?

  4. Wait a minute. If the answer is A. But you put D. If you change it, based solely on a guess you only have a 1/3 chance of choosing the right one. But the study showed 51% chose A. I don't think I get your point.

  5. I will not try to pretend that completely understand his point but I hope I can add a little insight.

    When answering a typical MC question statistically you have a 25% chance of being right and 75% chance of being wrong. Based on those numbers, it appears reasonable that 25% of those who switch answers go from right to wrong. Leaving approx. 75% for all remaining possibilities (wrong to wrong and wrong to right). Now, prior to seeing this study I would have estimated that those who answer wrong initially would have have a 1/3 (25% of the remaining 75%) changing to the correct answer, while this study makes it look closer to 2/3 (50% of remaining 75%). I am willing to concede that, but all this tells me is that if you start with a wrong answer, you have a 2/3 chance of switching to the correct one. However, if you initially choose correctly. then change your answer, you have a 100% chance of making an incorrect choice (assuming only 1 correct answer). The truth is... its always going to be in your best interest to switch answers if you start with the wrong one. The kicker is knowing your first choice is incorrect (which I believe is Paul's point). If you ask me... if you don't know the answer from the start, flip a coin and you will get similar results.

  6. Wow Paul, great point. Can't believe I didn't see that before.


You are the reason why I do not write privately. I would love to hear your thoughts, whether you agree or not.