Friday, July 27, 2012

Objectively Subjective

Something that has always bugged me is human’s ability to translate subjective, mental information about probabilities, happiness, guilt etc. into objective, quantifiable numbers. For me the problem really became pronounced when reading a paper asking students to assess from 0-100% how likely they felt their answer to a simulated SAT question was correct. Say that you are given 5 choices: A, B, C, D, and E. Ignore for a moment the inherent difficulty in forecasting using data, and instead focus on the process one would go through to make the assessment of how certain they are. First, you would have to think how well do I know this subject area? If your answer is ‘not too well’, then you have to translate that ‘not too well’ into some range. Let’s say I have a 30% chance I know the correct answer with certainty. This number is compared to the 20% chance that if you guess, you got it right. However, now you start going through the answers themselves and determine how ‘reasonable’ they seem. The mental machinations may exclude one obviously incorrect choice, but now we’re stuck with what do we mean by ‘reasonable’ in the context of some finite number. Let’s say I’m 100% certain D is wrong is wrong and 85% sure E is wrong. Knowing this means we might as well only select from A, B and C. With this restricted choice set, my probability of being right, incorporating my prior belief state, is about 35.29%, now isn’t that a nice number? However, we are ignoring one key problem, my initial assumption that I was 30% certain I knew the right answer. How am I to know that because I got cut off earlier in the day by some bozo, I am now just a little more pessimistic in my outlook? Because of me being in this “hot state”, I shave off 10% from my initial assumption. 
            
           This question gets even more interesting when looking at how people make absolute comparisons. George Miller, who recently passed away, was a pioneer in the field of short-term memory, writing a now rather famous paper entitled, ‘The Magical Number SevenPlus or Minus Two: Some Limits on our Capacity for Processing Information’.  One experiment of particular interest to this discussion deals with peoples’ ability to discern differences in tones, Prof. Miller sums it up nicely:

“When only two or three tones were used the listeners never confused them. With four different tones confusions were quite rare, but with five or more tones confusions were frequent. With fourteen different tones the listeners made many mistakes. These data are plotted in Fig. 1. Along the bottom is the amount of input information in bits per stimulus. As the number of alternative tones was increased from 2 to 14, the input information increased from 1 to 3.8 bits. On the ordinate is plotted the amount of transmitted information. The amount of transmitted information behaves in much the way we would expect a communication channel to behave; the transmitted information increases linearly up to about 2 bits and then bends off toward an asymptote at about 2.5 bits. This value, 2.5 bits, therefore, is what we are calling the channel capacity of the listener for absolute judgments of pitch. 
 So now we have the number 2.5 bits. What does it mean? First, note that 2.5 bits corresponds to about six equally likely alternatives. The result means that we cannot pick more than six different pitches that the listener will never confuse. Or, stated slightly differently, no matter how many alternative tones we ask him to judge, the best we can expect him to do is to assign them to about six different classes without error. Or, again, if we know that there were N alternative stimuli, then his judgment enables us to narrow down the particular stimulus to one out of N /6.”
The takeaway from all his results is that humans have an innate capacity to make an absolute judgment among 7 different items on a uni-dimensional scale. For example, if someone were given 14 shades of green and asked which ones are different, humans would usually say that 7 of them are the same and 7 are different. Now one must remember that these are questions about single dimensions of OBJECTIVELY knowable items, like color, sound, taste etc. The world out there is filled with the unknown. When pollsters and academics ask questions to subjects relating to “enthusiasm to vote”, “dislike with the president’s economic policy” or “probabilities that your answers are right”, participants are doing their best to bring all these factors together and spit out a number. What I’m saying is that the results that these processes glean may be telling us little about people’s true tastes and instead depend heavily on how many choices participants are given, as well as, other factors related to framing of the questions asked.