Below you have a pseudoscientific claim and a design for an experiment to test the claim. Your task is to study the experiment protocol and determine whether it meets all requirements for a valid test. Remember to determine whether the protocol eliminates deception, ad hoc excuses, etc., so that if a subject passes the test it can only be because there is a real effect.
This is the claim:
I, Bryan Boondocker, and a friend of mine have the ability
to transmit colors telepathically. I can look at a color, any color, and transmit
it to my friend, who will receive it. We can do this with over 90% accuracy.
We do not need to be in the same room.
See Criteria for a Successful Experiment.
Procedure:
Staff: Transmit observer and recorder
Receive observer and recorder
Required: Deck of 10 various color cards (supplied by Boondocker)
plus 2 additional cards for pretest
Two adjoining rooms (common wall) in some convenient location
Table/desk for each room
Note pads for observers
Pencils
Small supply of snacks/drinks/etc
Procedure:
Prior to the test, Boondocker and the testers will meet and agree on test
conditions and procedures. A contract will be executed and signed by
all agreeing to procedures, evaluation and success criteria. A success
rate of 90%, as claimed by Boondocker, will be required.
On the day of the test, all will meet at the agreed location. All will check
to be sure all necessary supplies are ready.
1. Boondocker and the transmit observer will locate in one of the rooms.
2. The receiving friend and the receive observer will locate in the other.
3. The transmit observer will shuffle the color card deck.
4. The claimants will be asked if everything is in order and that no problems
exist. A pretest with the two supplied cards may be done. The claimants
will certify that conditions are satisfactory for the test.
5. When everyone is comfortable with the situation and is ready, then begin.
6. Boondocker will select a card from the deck and lay it on the table.
7. The recorder will write a sequence number (beginning with 1) on his pad
along with the color chosen.
8. After recording the color, the recorder will knock once on the wall to
indicate that the transmitter is ready.
9. Upon hearing the knock, the receiver will begin attempting to receive
the telepathic transmission.
10. Upon successful reception, the receiver will announce the color received.
11. The receive recorder will assign a sequence number (beginning with 1)
and write down the received color and have the receiver verify it.
12. The recorder will knock once on the wall to indicate that they are
ready for the next transmission.
13. The card just used is removed from the table by the observer/recorder.
14. This process (steps 6-12) will be repeated ten times, till all of the
colors have been transmitted.
15. Upon completion all will meet in one of the rooms. The two recorders
will compare their records and determine how many colors were received
correctly. Nine or more correct constitutes a successful test.
The whole test should take less than one hour.
The protocol given above has a serious flaw. Relatively few picked it up. The scoring involved 5 items in the protocol, shown at the top of the paper.
The huge hole involves the lack of randomization. Notice that the deck shuffling is done only once, before the trials begin. The hole is based on "Boondocker will select a card..." for each trial. With no randomization and Boondocker supplying the cards (and therefore knowing the colors in advance), the door is wide open for Boondocker and his friend to use a memorized color sequence to fool the testers.
This design fails the second primary requirement:
If the person does NOT actually have the claimed power, he will NOT be able to perform the feat.These guys could pass this test without any psychic ability whatever.
A few people designed a new experiment, although that was not what we asked for. The goal was an analysis of the stated protocol.