Yes, I generalize. Yes, I overwhelmingly rely on statistics instead of individual accounts. And now I’m going to explain why you should, too.

I am extremely sensitive to sample bias, because I tend to surround myself with extremely atypical individuals. This is not uncommon; if you study physics, all your friends can probably do calculus in their heads, and very soon you start believing that it’s something everybody can do, or that it is much more common than it actually is. The problem is magnified by the echo-chambers of the Internet, because similar people tend to form interest groups that exclude the outside world and are in fact rather hostile to opinions that are uncommon in their group. After a while, their perception of reality is so severely skewed by their personal sample bias, that what they think is going on in the world has very little in common with the actual world.

You can call it the Marie Antoinette syndrome, if you like; I’m referring to the anecdote where she was told about the riots on the streets and she asked about the cause, and when she was told that the people have no bread, she allegedly responded along the lines of “Stupid mob, they should eat cake instead”. That’s what I’m talking about when I refer to sample bias. When you’re surrounded only by rich people you can’t really understand that the poor people don’t have the option to choose between this or that food, because they can afford only the cheapest kind, and if it isn’t available, they will starve. You have only what one would call the first world problems (“the line at Starbucks was too long so I didn’t have coffee this morning”).

I’m not saying that applying personal preference to our personal choice of company is a bad thing. The whole point is to find people who are more like us, so that we can function at our peak potential, which would be impossible if we were surrounded by people who have interests and abilities so different from our own that there’s no significant intersection. It’s a good thing. What I’m saying is, it isn’t healthy to use our personal experience, formed by an extremely skewed sample of the general population, in order to form opinions about the state of society in general.

This is why I rely on statistics. If someone did a meticulous scientific analysis of some social group, using large unbiased samples, I am going to rely on his findings much more than I’m going to rely on my limited experience with members of that group, because if you’re in heaven, you will tend to think that everybody is a saint, and if you’re in hell, you’ll tend to think that everybody is a demon. What you need is a wider picture, which tells you how many people in total there are, of which how many are in hell and how many are in heaven, and of those in each group, you will want a breakdown by certain characteristics in order to see a pattern. Essentially, what you need to do is remove yourself from the picture and acquire sufficient distance, in order to gain perspective.

But this wider perspective doesn’t influence the way I treat individual people. I can have generalized opinions about a certain group of people based on statistics, but that individual you are dealing with can be normal for that group, or extremely atypical. It’s like trying to form opinions about me based on general statistical facts about Croats. Not the best idea. On an individual level, you need to treat people like individuals, and do your best to perceive the actual person you’re dealing with. However, this individual approach is actually dangerous when you’re dealing with large populations, and if you don’t resort to statistics you will be unable to form useful opinions. What you need are generalizations – you need to know what an expected median sample of a population is, and for an individual, you need to know where he is placed on the histogram of his population group. If you’re talking to a +3 sigma individual, you know how many of those you can realistically expect to find where he came from. If you’re in Jet Propulsion Laboratory you need to be aware that each individual there is probably extremely atypical for any population, to the point where he’s more alien than human; the average person there has a PhD and Mensa level IQ. If you’re in the army, everyone you encounter is most likely representative of the general population and it would be unlikely for you to encounter an atypical individual.

If you’re an atypical individual, it is exceedingly difficult for you to find others like you in a general population, and the best thing about the Internet is that it enables you to find others like you much more easily, allowing you to skip the arduous task of checking out uninteresting individuals with very low odds of finding what you’re looking for. However, this opens you to extreme sample bias, because the ease with which you can meet other atypical individuals can blind you to the placement of your interest group on the population histogram. Essentially, you tend to think that everybody is like you and that your group represents what people normally are, and that actually has an unknown probability of being true.