6: Making Decisions

Suppose there is a new way of teaching spelling in grade schools. A teacher has two classes of students, and the students were assigned randomly to the two classes. She teaches one of the classes using the new method, A. She teaches the other class  using her usual method, B. Then she gives both classes the same spelling test. These scores are obtained:

Method A: 10 10 10 12 15 17 17 19 20 22 25 26

Method B: 6 7 8 8 12 16 19 19 22

We want to decide whether method A is better than method B. That is, did the students get a higher average (median) test score with method A than with method B?

It is required that the classes have comparable spelling ability before being taught by the two methods. Here, as in actual research, the researcher achieves such comparable groups by assigning people (or "units" in general) to the two treatment groups at random, thus "averaging out" differences between the people of the two groups.

One way to compare Method A to Method B is the median test. We compare all 21 students’ performances with the median of the 21 scores (16). If the "above median" scores fall predominantly in one group, we can conclude that that method is superior. 

Method A has 7 scores above the median (16), while Method B has only 3. But is it possible that the methods are no different, and this advantage resulted just from chance assignment of the better readers to method A? Let’s test this null hypothesis.

Using the 5-step procedure:

1. A box with the scores written on 21 tickets. 
2. Draw 12 tickets at random without replacement to be the scores for group A. The remaining tickets give the score for group B. The median of combined group of 21 scores is still 16, since we still have the same 21 scores. Count the number of scores in Group A that are greater than 16. 
3. A trial is successful if group A has >= 7 scores above 16. 
4. Do, say, 1000 trials.
5. From running the  Box Sampler illustration, we found that P (7 or more) = 438/1000 = 0.438  fill in from simulation results.
   

Interpretation :Getting seven or more scores in group A that are larger than the median for the combined group happens often by chance, namely, 30% of the time. Because getting seven or more scores in group A larger than the median is  not a rare event, we decide that the model chosen in step 1 could be appropriate. The difference in scores between the two groups observed in the original test could easily have happened just by chance. Therefore we do not  have strong evidence that  method A is better, and our initial impression that method A might be better has not been proven by the data.