Seth Roberts is one of my favorite bloggers. In terms of bloggers who introduce me to new ideas that I end up trying to internalize, he is up there with Robin Hanson, Tyler Cowen, Eric Falkenstein, Arnold Kling and Bryan Caplan. His posts on self experimentation, health care (including the health benefits of omega 3) and insiders vs outsiders are particularly interesting.
However, recently Seth Robert's goes too far in a post where he thinks that certain claims sound too much like real thinking but don't touch on real data, and therefore prevent analysis more often than they help the analysis.
1. Absence of evidence is not evidence of absence. Øyhus explains why this is wrong. That such an Orwellian saying is popular in discussions of data suggests there are many ways we push away inconvenient data. [Me: This is wrong, the phrase is not meant to deal with probabilities. Evidence of absence means that a claim is unlikely or false, and the lack of evidence for a claim when evidence shouldn't necessarily be present does not automatically make the claim false even if a claim with some supporting evidence is stronger than a claim without any evidence all things held constant.]
2. Correlation does not equal causation. In practice, this is used to mean that correlation is not evidence for causation. At UC Berkeley, a job candidate for a faculty position in psychology said this to me. I said, “Isn’t zero correlation evidence against causation?” She looked puzzled.
3. The plural of anecdote is not data. How dare you try to learn from stories you are told or what you yourself observe!
Orwell was right. People use these sayings — especially #1 and #3 — to push away data that contradicts this or that approved view of the world. Without any data at all, the world would be simpler: We would simply believe what authorities tell us. Data complicates things. These sayings help those who say them ignore data, thus restoring comforting certainty.
Maybe there should be a term (antiscientific method?) to describe the many ways people push away data. Or maybe preventive stupidity will do.
I responded in the comments.
While these sayings are used incorrectly many times, there are good reasons these types of arguments are mentioned in public discourse. Rhetorical strategies in the public space often make use of really bad arguments that are countered by those sayings. Your annoyance is how much of a crutch these arguments are among people who should be arguing in good faith and at a higher level than average.
1. “Absence of evidence is not evidence of absence.” This was a very useful concept to point out to those people who believed that because nominal national US housing prices had never dropped before that they wouldn’t drop in the future.
2. Correlations are assumed to be causation in many cases where it is incorrect. People make arguments all the time that point out correlations, think up some mechanic that explains why the correlation is causation but fail to even consider whether the correlation is actually coming from an underlying factor driving both conditions.
3. Politicians always like to bring out a few people who would be really helped or harmed by the policies. Obvious examples include the widow in need of medical care for her sick kids or Joe the Plumber who would be caught in the higher tax bracket.
Obviously there are places where arguments 1 through 3 work. So the main problem isn’t about the arguments themselves, but that people use these responses when they assume the person is either arguing in bad faith or is relatively ignorant when that isn’t the case. Furthermore, they often feel like 1 through 3 are enough to settle the question entirely when more argumentation is needed to make their point.
Seth pointed out that he really wants people to reply with data, not slogans. He wrote a post on how he is worried that these slogans are bad because they throw out data instead of adding new data. He then wrote a new post highlighting my use of point number one in the above comment about the housing bubble. He thought that this was illustrative of preventive stupidity, since I did not mention the response that one could have told the person who believed in housing that housing prices did fall in the Great Depression (Which is a weird point to make because while it directly contradicts the "housing never goes down" meme; it is not the best argument to make about whether or not housing prices were likely to go down in a relatively stable economic environment). Again, I responded in the comments.
It is weird that you think this is an example of preventative stupidity rather than that of an incomplete argument.
Side note, the phrase “evidence of absence” in the context of the phrase you think of as “preventive stupidity” seems to have a more specific meaning than your more literal interpretation:
http://en.wikipedia.org/wiki/Evidence_of_absence
I didn’t make the other good arguments against housing price movements never going down because it seemed off topic - but there is a whole lot of other data driven analysis that could have been made about the likely path of housing prices such as price to rent or income ratios, affordability indices, real prices historically dropping, rising interest rates, unsound lending and the financial condition of the average borrower, the prevalence of ARMs and the future path of interest rates, other regions in the US with falling housing prices such as Texas in the 80’s and CA in the 90’s and other countries with falling housing prices.
The key here is that even the above arguments are just surface arguments. I can’t just make an argument about price to income ratios without addressing whether or not there is something truly significant about that ratio. If we are having an in-depth discussion, I also have to address the counter arguments that try to give a justification for why those higher ratios are sustainable in today’s environment.
The question is whether or not the other arguments on your list of “preventive stupidity” are good in the same way as the price to income ratio is useful for learning about housing markets - they make a point but are otherwise incomplete. I still stand by my statement in the original comment - In the case of housing markets, knowing that something can still be possible or even likely despite it not happening in recent history can be very useful. It is actually more useful than knowing that housing prices went down in the Great Depression, because while many people believed that the future path of the economy was tied to the housing market, the point that housing markets fell in the Great Depression is just not that interesting unless people thought another Great Depression was likely independent of the housing market.
Now, when discussing ideas with other people we do not always give them the benefit of doubt and dig down into the data with them. The salesman/huckster and politician are two groups of people where we know they are biased so if we do engage them we might not engage them fully. It is insulting to explain that I am not weighting their ideas very much because I think their anecdotes or correlations are just convenient coincidences or carefully cherry-picked data. In these cases it is often easier to tell them that the plural of anecdote is not data and correlation is not causation rather than go into the details and accuse them of dishonest data manipulation.
It is very annoying to be on the other side of this style of confrontation because the person making the “preventive stupidity” arguments, when not followed up with the below arguments is being a little (or if it is about your own research, very) disrespectful to the person who sees themselves as trying to learn the truth in an unbiased manner.
1. The underlying cause for the correlation.
2. When the absence of evidence would be significant or what evidence would disprove the view (The housing market hasn’t fallen in recent history, but if the market is still rising when the mortgage credit to GDP ratio falls I’ll take that as evidence that the market has moved for fundamental rather than speculative reasons)
3. The reasons why the anecdotes were collected in a biased manner and how this bias skews the results
From this point in a conversation, you can either label their actions preventive stupidity and halt discourse entirely, or you can ask for more depth about what they think the underlying causal factor is, what would qualify as evidence of absence or why the anecdotes are significantly biased/unrepresentative.
Seth does point out that people rely on the phrases "correlation is not causation, "absence of evidence is not evidence of absence" and "the plural of anecdote is not data" a bit too much. The phrases by themselves only imply the real argument, and often imply that the other person failed to consider a very basic principle. Used properly, the phrases are starting points for thinking about interpreting data, not concluding arguments. In the end, it is up to both people to try to communicate, so knowing the proper response to these types of implied arguments might make for better dialogue than trying to label their phrasing as Orwellian self deception.
[Side note to explain why Seth's anecdotes about people incorrectly making these "preventive stupidity" arguments more often than they make other incorrect arguments is biased data: People might be more likely to incorrectly treat Seth Roberts as a salesman because he is the author of The Shangri-La Diet and writers of diet books are not generally seen as truth seekers]
Addendum: Also, anyone interested in making ground breaking paradigm shifts will encounter resistance to new ideas that might appear obstructionist from people who are focused on incremental progress. As for the people obstructing that person, they may feel like they are merely doing their part to protect people from giving credibility to yet another false belief. From the perspective of a person caring about maximizing society's true beliefs in aggregate, sometimes certain types of arguments need to be shot down quickly merely because of how tedious it would be to refute each person who makes that type of argument. While this approach itself has some false negatives and might unfairly reject things like the Shangri-La diet without even testing it, the people who act in this manner may believe that it is a small price to pay to protect people from the cranks and cultists. Of course, the people to take this "maximize the aggregate belief in truth" approach are likely to be insiders, with their own special set of biases.