Physics 3333 / CFB 3333 - Pseudoscience and Evidence
-- Thomas Huxley
"The plural of anecdote is not data."
pace Raymond Wolfinger (who actually said the opposite)
Pseudoscience and Evidence
Older slides in PDF Format
Pseudoscience is, as the name implies, something that claims to be scientific but is not. The modern world is LOADED with pseudoscience. You can learn to detect it because it has properties that you can compare to the real scientific method.
Page 162 of Schick and Vaughn shows one form of the scientific method. We'll summarize it here.
- Observe some phenomenon
- Develop a hypothesis (possible explanation) for it.
- Use the hypothesis to predict another phenomenon.
- Check out the prediction by observation or experiment.
It doesn't always work exactly like this; some things (like the Sun) cannot be experimented with. However, according to S & V, "any procedure that serves systematically to eliminate reasonable grounds for doubt can be considered scientific." Along the same line, in Shermer we find "Scientific progress is the cumulative growth of a system of knowledge over time, in which useful features are retained and nonuseful features are abandoned, based on the rejection or confirmation of testable knowledge." This is how we have learned everything we know about the universe.
Pseudoscience, on the other hand, ignores one or more of the steps. The true scientific method is NOT followed, although the pseudoscience is littered with scientific-sounding terminology. It's the method that counts, and that is what fails. True objective evidence analysis is usually lacking.
Properties of Pseudoscience
Always be alert for signs. Pseudoscience is identifiable.
- Heavy reliance on testimonials or anecdotal evidence.
- Built-in or ad-hoc excuses for failure.
- Non-falsifiable hypotheses.
- No serious attempt to disprove it.
- Attacks existing explanations without offering anything new.
- Patents cited as evidence.
- Evidence is used selectively; some is ignored, particularly that evidence which might disprove the hypothesis ("cherry picking").
- Misrepresentation of real scientific work.
- Claims of "It can't be _____, so it isn't."
- Makes no progress and generates no new knowledge, even over many decades.
- Claims of effects or forces never measured, or even observed.
- Claims of impossible precision of measurement.
- Makes no useful predictions.
- Tests that have VERY small samples.
This is important. Remember that Carl Sagan said "Extraordinary claims require extraordinary evidence." A skeptic wants evidence in support of a claim. A skeptic will accept a claim IF it is accompanied by impeccable evidence. So what is evidence?
Let's begin by discussing some things that are NOT evidence. These things are, unfortunately, what pseudoscience uses.
Anecdotes or testimonials: You encounter these all the time, mostly in advertisements. "I had no musical sense whatever until the day I was frightened by a tuba. Now I can belch Mary Had a Little Lamb." Try "I was slowly going bald until I started using HairBlast; now my hair is growing back, my teeth have turned green, and I smell funny." A personal story (anecdote) describes only one event (assuming it is truthful). It tells you nothing useful about any larger population. Everybody knows someone who _________ (fill in the blank). A testimonial is even more suspect. They are frequently used in ads for a good reason - they are not regulated. Advertisers are supposedly barred from making unsupported claims in their ads, but they can use all the testimonials they want. What you don't know is whether they are genuine or written by an ad copy writer. You also don't know of any other pertinent circumstances that apply to the situation.
For example, on January 4 2007 the FTC went after some weight-loss pill vendors. They had been making claims for their product that were not supportable; the manufacturers will be required to remove the unsupportable claims from their advertising. Some of the testimonials these vendors used in their ads were from people who had lost a substantial amount of weight while taking the pills. What you didn't get from the ad was the fact (found by the FTC) that those people had also been following a good diet and exercise program, which was almost certainly the real reason for the weight loss.
FTC Chairman Deborah Platt Majoras said "Testimonials from individuals are not a substitute for science, and that's what Americans need to understand." One of the marketers actually had a study that showed that people who got the placebo lost more weight than those who got the actual weight-loss pill! You can bet that study was never publicized.
Patents: Believe it or not, a patent does not give you any assurance that the patented device or process actually works! You wouldn't believe some of the stuff that gets patented.
Photographs: Given modern image processing software and hardware, almost any photograph can be faked. It is practically impossible to rule out hoaxes with photographs. An out-of-focus, fuzzy image is worthless also.
Attacks on existing explanations: Someone can fire all the arrows they want at existing theories or explanations, but that in no way constitutes evidence for the new claim. It's not evidence unless it supports the new claim.
"It can't be, so it isn't": This is a form of wishful thinking. You might also translate this as meaning "I don't like it, so it can't be true." If you are convinced that something can't be true, then it isn't. This may or may not have any connection to reality.
"I never heard of that - it can't be true" Maybe, maybe not. It is not reasonable to think that you have heard of everything. If someone says "Well, I never heard of that" you can take it as a statement of fact: they have not heard of it. What you are not allowed to do is to extrapolate that to "it can't be true." It is perfectly possible that there is some important result, mechanism, theory, or phenomenon that the person has not heard of.
"Post hoc, ergo propter hoc" arguments: This thinking involves making unjustified connections between events. Someone who lives near a power line get cancer, so obviously the power line caused the cancer. Wrong. Extensive research has showed no connection.
Coincidence: Event A and event B happened together, so they must be related. We may forget that events A and B happen frequently and not together. When the coincidence occurs, we forget about all the other times.
Correlation: Two phenomena may appear to be related. For example, the stock market and some sports team's performance may seem to track each other. Would you invest based on a ballgame score? Very important: Correlation does not mean causation. If a correlation appears, one must investigate thoroughly to see if there is an understandable linkage between the two. IT is also possible that there is some yet unidentified factor working that is responsible for both the observed phenomena.
Ok - so what does constitute evidence? This depends somewhat on the claim. Take the example of "HairBlast" that we made up earlier. What would it take to convince a skeptic that the stuff really works? The anecdote certainly won't do it. It's going to require a properly done experiment with a large enough sample and proper protocols so that the result is unambiguous - either the stuff works or it doesn't. You might do it like this.
- Gather a large sample of balding men.
- Randomly divide the sample into two groups.
- Give one group HairBlast to use along with instructions.
- Give the other group something which looks exactly like HairBlast but is completely inert.
- No participant is told what they are using.
- After the test period, someone who does not know which participant got what measures the hair growth for all.
- Compare hair growth of the two groups.
By the way, that example is what is known as a randomized, double blind, placebo-controlled trial. This is considered to be the gold standard in experiments. We'll see more of this later.
There are two experimental criteria that sometimes need to be applied.
In the case of "psychic" phenomena, such as ESP, the experiment must be
designed so that
Direct Presentation: If a friend of yours insists that they are communicating with alien visitors, have them introduce you to one. That should be interesting. See if you can wangle a spaceship ride. That would be convincing. We are looking for something concrete which, by existing, backs up the claim.
You are looking for evidence that is clear and unambiguous, not requiring tortured interpretation or assumptions.
There are, of course, cautions to be observed. It is very important to rule out errors, self-deception and hoaxes. These phenomena are major problems in most of the pseudo-scientific areas we will visit in 3333. Suppose that your friend does introduce you to an "alien." You need to check very carefully to be sure it's not some short guy in a funny suit. Any experiments must be checked very carefully for flaws and errors. Poorly done experiments have been responsible for a lot of BS. We'll explore some of these a bit later.
Media and Resources on this topic
- Epidemiologist Ben Goldacre presents a short (14') talk at the TED Conference on "Battling Bad Science": Link from the TED website - Local copy of video
- Types of BS Encountered