Physics 3333 / CFB 3333 Homework 5


Below you have a pseudoscientific claim and a design for an experiment to test the claim. Your task is to study the experiment protocol and determine whether it meets all requirements for a valid test. Remember to determine whether the protocol eliminates deception, ad hoc excuses, etc., so that if a subject passes the test it can only be because there is a real effect.

See Criteria for a Successful Experiment.

The Claim: I, Bryan Boondocker, have invented the Smellex, a patented gold detector. With this device I can detect any amount of gold at a distance of up to 50 feet. The gold can be buried in the ground up to 50 feet down and I can still find it. The Smellex is at least 90% accurate.''

Experiment:

  • Staff:
    • Test conductor
    • Result recorder
    • Professional magician to detect cheating and who will always be near Boondocker
    • Video camera operator to record all procedures.
  • Items Required:
    • Smellex device (supplied by Boondocker)
    • One 2 inch x 4 inch x 7 inch standard brick of gold (27 pounds)
    • Open field for test area
    • 10 opaque plastic buckets
    • Marker for marking buckets
    • Note pad and pencils for data recorder
    • Small tent or shelter for rest area.
    • Small supply of snacks/drinks/etc.
  • Procedure:
    • Prior to the test, Boondocker and the testers will meet and agree on test conditions and procedures. A success rate of 90%, as claimed by Boondocker, will be required. Ten trials will be performed. In the event of a success, 10 more trials will be performed in a retest. If all scheduled trials are not run, that constitutes a failure.
    • On the day of the test, all will meet at the agreed location. All will check to be sure all necessary supplies are ready. Boondocker will survey the site to ensure that there is no interference present and that his device is working on the gold brick.
    • The test conductor will use the marker to write numbers 1-10 on the bucket bottoms. The buckets will be positioned in the test area separated by a minimum distance specified by Boondocker.
    • Drinks and snacks are provided for Boondocker. The test conditions are to be as agreeable as possible.
    • The result recorder will prepare two pages: one for recording the actual locations of the gold and another for recording Boondocker's choices. Both sets of data will be keyed by the trial number. Boondocker will not see the actual location data until the test is complete.
    • When everyone agrees that conditions are satisfactory, the test will proceed. If Boondocker wishes to pause for a snack or drink, that is allowed. A relaxed atmosphere is the goal.
    • Boondocker and the magician will be in the small tent away from the test area. The tent/shelter will be set up so that the view of the test area from inside is blocked. The snack/drink supply will be in the tent.
    • The entire test will be recorded by the camera operator, who will remain in the test area for the entire test.
    • The actual test protocol is as follows, all recorded by the video camera operator:
      1. The test conductor and result recorder will enter/remain in the test area.
      2. The test conductor will pick a bucket number between 1 and 10 and announce it to the result recorder. The camera operator willl record this.
      3. The result recorder will write the trial number and the bucket number on the actual locations page.
      4. The conductor and the recorder will then put the gold brick under the bucket with its number equal to the chosen number. The camera operator willl record this.
      5. After the gold is placed, the conductor and recorder will slightly move all other buckets and retreat to a position near the tent.
      6. Boondocker will next enter the test area and use his Smellex to locate the hidden gold. The magician will accompany him. The camera operator will make a video recording of the trial.
      7. During the trial Boondocker is not permitted to touch any buckets.
      8. When Boondocker has located the gold, he will announce his choice on camera.
      9. A hand signal will call the result recorder to the test area; Boondocker will repeat his choice to the recorder, who will enter it on the page of Boondocker's choices.
      10. Boondocker and the magician will then return to the tent while the gold is moved.
      11. The test conductor will then join the camera operator and the recorder in the test area.
      12. The test conductor will retrieve the gold.
      13. These steps will be repeated for a total of 10 trials.
    • When all 10 trials are complete, Boondocker, the test conductor, and the recorder will meet at the tent. The recorder will reveal the actual locations of the gold along with Boondocker's selections. The result will be evaluated. The video can be reviewed if there are any claims of discrepancy. Boondocker must get at least 9 out of 10 correct to pass. If he passes, the retest begins after a rest period that is satisfactory to Boondocker.

    • DO NOT try to design a new experiment for this test.
    • DO NOT make recommendations for fixing any flaws.
    • Describe the flaws in THIS protocol, nothing else. In other words, do the following:
      • Apply the functional specifications to see that all are met.
      • Examine the procedure carefully looking for any opening for cheating.
      • Tell us what flaw(s) you find.

      List the flaws you found in the experiment. List ONLY flaws.