- The experiment
- See also
- References
In the design of experiments in statistics, the lady tasting tea is a randomized experiment devised by Ronald Fisher and reported in his book The Design of Experiments (1935).{{sfn|Fisher|1971|loc=II. The Principles of Experimentation, Illustrated by a Psycho-physical Experiment}} The experiment is the original exposition of Fisher's notion of a null hypothesis, which is "never proved or established, but is possibly disproved, in the course of experimentation".{{sfn|Fisher|1971|loc=Chapter II. The Principles of Experimentation, Illustrated by a Psycho-physical Experiment, Section 8. The Null Hypothesis}}[1]
The lady in question (Muriel Bristol) claimed to be able to tell whether the tea or the milk was added first to a cup. Fisher proposed to give her eight cups, four of each variety, in random order. One could then ask what the probability was for her getting the specific number of cups she identified correct, but just by chance.
Fisher's description is less than 10 pages in length and is notable for its simplicity and completeness regarding terminology, calculations and design of the experiment.[2] The example is loosely based on an event in Fisher's life. The test used was Fisher's exact test.
The experiment
The experiment provides a subject with 8 randomly ordered cups of tea – 4 prepared by first pouring the tea, then adding milk, 4 prepared by first pouring the milk, then adding the tea. The subject has to select 4 cups prepared by one method. Judging cups by direct comparison is allowed. The method employed in the experiment is fully disclosed to the subject.
The null hypothesis is that the subject has no ability to distinguish the teas. In Fisher's approach, there was no alternative hypothesis,{{sfn|Fisher|1971|loc=Chapter II. The Principles of Experimentation, Illustrated by a Psycho-physical Experiment, Section 8. The Null Hypothesis}} unlike in the Neyman–Pearson approach.
The test statistic is a simple count of the number of successes in selecting the 4 cups (the number of cups of the given type successfully selected). The distribution of possible numbers of successes, assuming the null hypothesis is true, can be computed using the number of permutations. Using the combination formula, with 
total cups and 
cups chosen, there are 
possible combinations.
| Success count | Permutations of selection | Number of permutations |
|---|---|---|
| 0 | oooo | 1 × 1 = 1 |
| 1 | ooox, ooxo, oxoo, xooo | 4 × 4 = 16 |
| 2 | ooxx, oxox, oxxo, xoxo, xxoo, xoox | 6 × 6 = 36 |
| 3 | oxxx, xoxx, xxox, xxxo | 4 × 4 = 16 |
| 4 | xxxx | 1 × 1 = 1 |
| Total | 70 | |
The frequencies of the possible numbers of successes, given in the final column of this table, are derived as follows. For 0 successes, there is clearly only one set of four choices (namely, choosing all four incorrect cups) giving this result. For one success and three failures, there are four correct cups of which one is selected, which by the combination formula can occur in 
different ways (as shown in column 2, with x denoting a correct cup that is chosen and o denoting a correct cup that is not chosen); and independently of that, there are four incorrect cups of which three are selected, which can occur in 
ways (as shown in the second column, this time with x interpreted as an incorrect cup which is not chosen, and o indicating an incorrect cup which is chosen). Thus a selection of any one correct cup and any three incorrect cups can occur in any of 4×4 = 16 ways. The frequencies of the other possible numbers of successes are calculated correspondingly. Thus the number of successes is distributed according to the hypergeometric distribution.
The critical region for rejection of the null of no ability to distinguish was the single case of 4 successes of 4 possible, based on the conventional probability criterion < 5%. This is the critical region because under the null of no ability to distinguish, 4 successes has 1 chance out of 70 (≈ 1.4% < 5%) of occurring, whereas at least 3 of 4 successes has a probability of (16+1)/70 (≈ 24.3% > 5%).
Thus, if and only if the lady properly categorized all 8 cups was Fisher willing to reject the null hypothesis – effectively acknowledging the lady's ability at a 1.4% significance level (but without quantifying her ability). Fisher later discussed the benefits of more trials and repeated tests.
David Salsburg reports that a colleague of Fisher, H. Fairfield Smith, revealed that in the actual experiment the lady succeeded in identifying all eight cups correctly.[3][3]The chance of someone who just guesses of getting all correct, assuming she guesses that any four had the tea put in first and the other four the milk, would be only 1 in 70 (the combinations of 8 taken 4 at a time).
Salsburg published a popular science book entitled The Lady Tasting Tea,[4] which describes Fisher's experiment and ideas on randomization. Deb Basu wrote that “the famous case of the ‘lady tasting tea’” was “one of the two supporting pillars [...] of the randomization analysis of experimental data.”[5]
See also
- Hypergeometric distribution
- Permutation test
- Random assignment
- Randomization test
References
2. ^{{Cite book|first=Sir Ronald A.|last=Fisher|authorlink=Ronald Fisher|chapter=Mathematics of a Lady Tasting Tea|origyear=The Design of Experiments (1935)|year=1956|title=The World of Mathematics, volume 3|editor=James Roy Newman|url=https://books.google.com/?id=oKZwtLQTmNAC&pg=PA1512&dq=%22mathematics+of+a+lady+tasting+tea%22|publisher=Courier Dover Publications|isbn=978-0-486-41151-4}}
3. ^{{Cite book | last= Box | first= Joan Fisher | title= R.A. Fisher, The Life of a Scientist | year= 1978 | publisher= Wiley | location= New York | isbn= 0-471-09300-9 | page= 134}}
4. ^1 Salsburg (2002)
5. ^Basu (1980a, p. 575; 1980b)
- {{cite book |title=The Design of Experiments |edition=9th |last=Fisher |first=Ronald A. |authorlink=Ronald Fisher |origyear=1935 |year=1971 |publisher=Macmillan |isbn=0-02-844690-9 |ref=harv }}
- {{cite journal
|doi=10.2307/2287648
|title=Randomization Analysis of Experimental Data: The Fisher Randomization Test
|first=D.
|last=Basu
|authorlink=Debabrata Basu
|journal=Journal of the American Statistical Association
|volume=75
|issue=371
|year=1980a
|pages=575–582
|jstor = 2287648
}}
- Basu, D. (1980b). "The Fisher Randomization Test", reprinted with a new preface in Statistical Information and Likelihood : A Collection of Critical Essays by Dr. D. Basu ; J. K. Ghosh, editor. Springer 1988.
- {{cite book| author=Kempthorne, Oscar|authorlink=Oscar Kempthorne|chapter=Intervention experiments, randomization and inference
|title=Current Issues in Statistical Inference – Essays in Honor of D. Basu
|url=http://projecteuclid.org/euclid.lnms/1215458836
|editor=Malay Ghosh and Pramod K. Pathak|pages=13–31
|publisher=IMS
|location=Hayward, CA.
|doi=10.1214/lnms/1215458836}}
- Salsburg, D. (2002) The Lady Tasting Tea: How Statistics Revolutionized Science in the Twentieth Century, W.H. Freeman / Owl Book. {{isbn|0-8050-7134-2}}