November 17, 2012
-
The Quantitative Fallacy
JDN 2456250 EDT 19:06
I’ve noticed a common mistake of reasoning, technically an informal fallacy, which once I started looking for it turned up all over the place. I call it the quantitative fallacy.
Quantitative fallacy: The use of easily-quantifiable measures that do not accurately reflect the phenomenon in question, especially when this produces false or misleading conclusions.
Quantitative measures are very useful, don’t get me wrong. But we must be careful to remind ourselves that not everything is easily quantifiable, and any simple quantitative measure is going to leave out important information. It’s perfectly fine to make a simplified model, as long as you recognize it as a simplified model. The mistake lies in taking the conclusions of the model too seriously and applying them to the real world without first making sure that the conclusions still apply.
Here are a few examples of the fallacy in action:
1. We use national GDP to assess economic prosperity. Even adjusted for inflation and purchasing power, it still ignores some very important things, like inequality in distribution and economic activity that isn’t paid for, like volunteer work and housework. And then there’s the fact that more stuff doesn’t necessarily mean a better society. But of course GDP is so much easier to measure!
2. We use grades and test scores to assess education. How would you really measure whether a student is a creative, critical thinker? How would you determine whether a student has the potential to make great discoveries as a researcher? That’s so hard! How much easier it is to simply count up the correct answers to multiple-choice questions. Nevermind that multiple-choice by definition eliminates the most important classes of problems that people need to learn to solve. (Multiple choice reduces all NP problems to P problems, and eliminates any need to creatively generate possible solutions.)
3. We use IQ scores to assess intelligence. Human intelligence is one of the most complex phenomena in the universe; indeed, it is so complex that it is actually capable of understanding many features of the universe and even, most remarkably, itself. So the idea that it could be measured in a single number is pretty much risible on its face. Yes, some people are smarter than others, in various ways. And maybe we could average over them all and say that people have an overall “smartness”, which we would call IQ. But this is necessarily going to be an oversimplification of their abilities. Someone with a higher IQ will not be better at everything than someone with a lower IQ, and not even necessarily better at traditionally intellectual tasks like chess or calculus. As for social skills, manual labor, athletic achievements–these have no connection to IQ at all, yet are vitally important, and require complex brain functioning that currently no robot is capable of matching.
In fact, the quantitative fallacy can prop up in more subtle ways. Consider the use of a company’s market capitalization, profit, and debt-to-revenue ratio as an assessment of the company’s value. At first this seems totally reasonable: Doesn’t a company exist, after all, to make profits? But there are many other factors that can cause these values to fluctuate, factors which may not have anything to do with the real functioning of the company. There are also externalities, impacts the company has on the world which are not measured in their profits and stock price. A company is a very complex system, and ultimately it cannot be reduced to a single number or small set of numbers.
How do we deal with this fallacy? First, we need to recognize it. Then, we need to continually remind ourselves that while numbers are useful, the fact that something is a number doesn’t automatically make it useful. A quantitative measure that doesn’t measure what we want is no better than a qualitative measure that does–in fact, it can be considerably worse.
