Arabic, Español, Korean, Français

The Baloney Detection Collection

One could easily find a large number of books on the subject of baloney detection, pseudoscience, and pathological science and deception. A complete bibliography would fill several pages. Michael Shermer, Robert Park, Carl Sagan, and others have written books and articles about these subjects. All have produced lists of things to look for and questions to ask in order to detect bogus claims or even outright fraud.

What follows is a synthesis of a number of these. All writers will say that no scheme provides completely foolproof detection of bogus claims; however, if you poke and prod the claim with the questions that follow, the odds are in your favor. The questions are in no particular order, so you'll have to mine out the applicable ones.

1. How is the claim/discovery announced?
A scientific discovery is checked and reviewed before being accepted for publication in a peer-reviewed journal. If a claim or discovery is announced by holding a press conference, beware. Such a claim may be unverified - not checked by an independent investigator. The cold fusion debacle is often cited. Robert Park's description of this affair in "Voodoo Science" is recommended reading. Also beware of "discovery" announcements that ask you to subscribe (pay) to some newsletter.

2. Does this source often make such claims?
If this particular source frequently makes claims that bear no resemblance to the best current knowledge, beware. There was someone in Dallas who, as the year 2000 approached, made a series of predictions based on the book of Revelations. He described how the sky would open up, trumpets would sound, and the world would end. Obviously, nothing happened.

3. Is anecdotal evidence cited?
Anecdotes, or individual stories, are not scientifically useful. Such a story suggests only that the individual did so-and-so (assuming no fakery). Nothing about larger populations can be inferred from it. Any description of an invention or product which uses large numbers of anecdotes or testimonials is suspect. You also must remember that testimonials can be bought; you can pay someone to do it for you. Also - the Federal Trade Commission will prosecute false claims, but testimonials are not regulated.

4. Does the source claim that "the establishment is trying to suppress this discovery?"
Such a claim does not provide any evidence that the "discovery" is really valid. You need to poke these with other questions to find out more.

5. Does the claim fit in with what we know about the world?
Does it fit into the larger context of knowledge? Shermer describes the claim that the Earth is only 6,000 years old. Accepting that claim requires that you also concede that everything learned about the Earth and solar system through astronomy, physics, geology, biology and other sciences is completely wrong.

6. Was the "discovery" made in isolation?
The reality of this world is that someone not trained in the science and its methods and working alone in a garage is not likely to make any kind of significant scientific discovery. Such a person making a great "discovery" is likely to be misinterpreting a known effect. They will likely persist with their belief in their discovery even after being shown that it is a well-known effect.

7. Has anyone tried to disprove the claim?
It is very important to find out if other investigators have tried to replicate the work. Toward this end, real scientists will publish complete information about the work, enough for another competent investigator to attempt replication. Others will repeat the work and publish the results.

8. Is this source offering a new explanation for observed phenomena, or simply attacking the existing explanation?
As with claims of suppression, this claim does not offer any evidence that the discovery is valid. Anyone attacking an existing explanation had better offer evidence. Absent evidence, all you know is that they do not like the existing explanation, for whatever reason.

9. Does the source claim that "this knowledge has been around for so long it must be good?"
A lot of interesting "discoveries" were made a long time ago. These include such gems as "the Earth is flat", "fire is a result of the phlogiston escaping from the burning material", and "the primary components of everything are Earth, air, fire and water." That they have been around a long time doesn't make them right. Again, what is the evidence?

10. Is the observed effect very small and accompanied by lack of success in increasing it?
Along this line, is the experiment a multi-trial statistical one or a direct measurement? Park's description of mind-over-matter experiments is recommended. Although balances capable of measuring micro-gram (very tiny) forces exist, mind-over-matter experiments use statistical trials, possibly because no claimant has ever been able to move a micro-balance with thoughts. The small effect observed in statistical trials is likely due to some small systemic bias in the experiment itself. The polywater affair is an example of a small effect that could not be increased.

11. Does the evidence for the "discovery" not get any better with time?
See number 10 above. If the observed affect is really a tiny bias in the experiment, nothing the investigator can do will increase the effect. Another thing to check is the results of better experiments. If, as the experimental methods improve, the observed effect gets smaller, it is likely that a perfect experiment will show no effect at all.

12. What does the bulk of evidence point to - the new claim or something else?
Is the source focusing on one small thing and ignoring a huge base of accumulated evidence that points to something else? Be sure to look at the claim in the context of current knowledge.

13. What kind of reasoning has been used?
Beware of "Post hoc, ergo propter hoc" thinking; this implies confusion about causation. This latin adage translates as "after this, therefore because of this." If A happened then B happened, then obviously A caused B. This is a very bad assumption. A claim that A caused B must be accompanied by credible evidence showing how A caused B. To put it simply, correlation does not mean causation.

Also beware of such things as questionable techniques (like hypnotic regression), anecdotal evidence (see 3 above), conspiracy theories (like government cover-ups), poor quality evidence (low-resolution photos) and simple misperceptions.

14. Does the new claim offer a new explanation for something, does it account for as many phenomena as the old one did?
Einstein's relativity explained more phenomena than did Newtonian mechanics. At velocities much less than the velocity of light, relativity reduces to Newtonian mechanics.

15. Is there any indication that the source's personal beliefs and biases are driving the conclusions?
Has contradictory evidence been ignored or pushed aside? It is necessary to consider all the evidence, not just the favorable bits. The process of science tends to sort this out.

16. Is it possible to test the claim?
Obviously, a statement that "God did it" cannot be tested in any way. Such a claim is an untestable construct. Explanations that do not make any testable predictions are of no use. They do not add to knowledge.

17. Is a chain of evidence (links) offered?
If chain of evidentiary links for a claim is presented, every link must be solid. A proposed chain of evidence fails completely if even one link fails. If someone claims that A causes B, B causes C, C causes D, and D causes E, they had better be prepared to demonstrate EVERY link. If, for example, all links are proven except that C causes D, which cannot be in any way demonstrated, then there is no proof that A results in E.

18. In extreme cases, such as claims about UFOs, can hoax be confidently ruled out?
Evidence for "flying saucers" is usually photographic. Modern image processing technology is so good that it is no longer possible to reliably detect a hoax photograph by examining it. A claim that a photograph has "passed all photographic analyses" may mean nothing more than that the hoax is very well done.

The 1967 Patterson film of a Bigfoot has long been considered the best evidence for the existence of that creature. The film was widely thought to be genuine until an investigator named Greg Long did several years work interviewing everyone in the area who knew anything about Roger Patterson. What Long turned up was evidence that was more than sufficient to show that the famous film is a hoax. Long even talked to the costume manufacturer who sold Patterson the gorilla suit!

If hoax cannot be ruled out, there remains a small probability that the evidence is fake.

19. Would a claimed invention violate the laws of thermodynamics?
Be aware of the laws of thermodynamics. The universe operates according to a set of physical laws that we understand reasonably well. Those laws govern what can and cannot be done. No amount of cleverness or piety will allow anyone to violate those laws.

The First Law of Thermodynamics says, simply, that "you can't win." Energy is conserved; it is not magically created. Always remember the old phrase " Non gratuitum prandium", which means "no free lunch." No thermal energy can be extracted from a single source - a heat flow from hot to cold is required.

The Second Law of Thermodynamics says "you can't even break even." Any energy conversion process has losses - the output energy will be less than the input. The difference is wasted and contributes to increasing the entropy of the universe. No one has ever found a way to get around this.

20. Is the claim/discovery really spectacular?
Scientists will tell you that "extraordinary claims require extraordinary evidence." They mean exactly that. If the claim is that life has been found on Mars or that someone has found a way to eliminate aging, really extraordinary evidence will be required to demonstrate it. A vague observation or a few anecdotes will not do the job.

21. Beware of special pleadings.
Be especially wary of any claim or excuse that the claimed effect cannot be measured for some reason. It may be that "the presence of a skeptic contaminates the effect", or "attempting to measure the effect destroys it", or "bad vibrations interfered", or something like that. An "effect" that cannot be measured likely does not exist.

22. If the effect is measured from a sample, how was the sample obtained?
Obtaining a proper random sample for a statistical measurement is more difficult than most people realize. There are many problems that can occur with sampling, problems that can seriously bias the results.

For example, suppose you want to take a poll about something and you want a random sample of the population of your city. How to do it? You might think of randomly choosing names from the phone directory - but that will exclude all people who either have no phone or have unlisted numbers. Try going downtown and randomly picking people who pass by. That will exclude all people who do not go downtown. Getting a real random sample is not easy.

23. Beware of "It can't be, so it isn't" thinking.
In the Cottingley Fairies incident, in 1917, two young girls took photographs which supposedly showed fairies cavorting with them in the woods. The two girls who photographed the fairies were considered incapable of perpetrating such a hoax, and thus followed the conclusion that they had not. The photos were therefore fawned over as genuine pictures of fairies. Truth was that the girls had indeed cut the "fairies" out of paper and photographed them in a prank which got out of hand when others saw the photos. This was not figured out until 1978.

24. Look at spelling and grammar.
When reading any web site, flyer, or anything else, be aware of how well the stuff is written. A web site or printed item that is filled with misspellings and bad grammar is highly suspect.

A number of excellent books on the subject of pseudo-science exist. Here are some good ones. Please remember that there are many more. The book on hoaxes is included because it is a good collection of historical (and hysterical) hoaxes.

<-- Back to the Physics 3333/ CFB 3333 Home Page

This site is a member of WebRing.
To browse visit Here.