Experiment Design


There are number of important facets to an experiment for testing a paranormal claim. All of these must be accounted for to make the experiment valid.

Assume that someone presents you with a claim of paranormal ability. Your job is to test this claim with an appropriate experiment. The experiment design must meet two primary goals.

1. If the person really does possess the claimed power, he will be able to perform the feat as advertised.

2. If the person does NOT actually have the claimed power, he will NOT be able to perform the feat.

Many experiments we have studied have met the first requirement but failed badly in the second. Such an experiment does not produce valid results.

Facets of Experiments

Here are some important considerations.

1. The experiment must be feasible, reasonable in cost, and performable in a reasonable period of time.

2. The trials MUST be randomized. This is critical. Randomization need not be terribly complicated; a coin or die will suffice (note that dice of more than 6 sides are available in game stores).

3. The test must be double-blind if possible. This means that it's most desirable that the test subject and the evaluator NOT know the test condition. For example, if a dowser is trying to find water in one of several large crocks, neither the dowser or the evaluator can know which jar contains the water. That is determined and set up by someone who comes in out of view (like behind a curtain), positions the crocks, then leaves. No one else knows where the water is and the person who does know is not visible to the participants during the trial.

4. You need to have a competent magician (conjuror) present to detect trickery. The magician's art is trickery, so a magician is best equipped to detect it. Remember that Uri Geller made monkeys out of a group of scientists by using tricks that any competent conjuror would spot very quickly.

5. Keep a permanent and complete record. Videotape works well. Careful and complete written records are also valuable.

6. BEFORE the trials begin, the claimant must sign a contract specifying exactly what constitutes success and failure. The percentage of successful trials required AND the total number of trials MUST be determined in advance. Read James Randi's contract to see what one looks like.

7. The trials must be such that success or failure is easily, objectively determinable. No judgment calls allowed. For example, do not try to decide if a drawing made by a remote viewer is "close to" the real scene. Instead, provide the remote viewer with 10 pictures and ask him to pick the one that matches the remote scene. He will either pick the correct one, or he will miss it.

8. Within reason, cater to the claimants. Ensure that they are comfortable, that there are no "interfering" influences, and that they are able to perform. If the subject wants a cold beer, send someone out to get one. Preliminary tests of the claimed ability are allowed; these are informal tests which allow the claimant to see that everything is working properly before the strict experimental controls are applied. For example, a dowser can test by "finding" water in a known location. This provides assurance that the "powers" are working. These preliminary test cannot be counted as part of the experiment. Be reasonable and helpful. You want to provide a FAIR test.
Achau Nguyen Test from CSICOP.org

9. Where appropriate, use instrumentation to measure test parameters. For example, do not try to decide if a psychic can make a candle flame brighter just by looking at it; get a photometer and MEASURE the brightness when the psychic is performing and when he is not. Deal with QUANTIFIABLE differences. HOW MUCH brighter does the psychic claim he can make the flame?

10. A successful test run must be repeatable. Good results are possible by chance. Really proving the ability requires doing it again.

11. All scheduled trials will be run; the subject is not allowed to "quit while he is ahead."

12. Remember - the person making the claim must prove the claim. The burden of proof is on the claimant. The rest of us DO NOT have to prove that the claim is false.

The contract and preliminary tests are intended to ensure that the subject is comfortable with the test AND also to head off ad-hoc excuses. The goal is to set up a test under "controlled conditions," which means that a claimant REALLY possessing the power will be easily able to perform, while someone who is relying on trickery will not. This means blocking any "information leakage" that might occur through side routes.