Design and Analysis of Experiments
Professor Daniel Houser
Noteset 6 - Repeated measures.
I. Introduction
Repeated measures designs refers to situations in which some experimental units are observed in more than one treatment condition. Complete repeated measures designs, our focus, occur when each experimental unit is observed under every treatment. It is important to point out that repeated measures designs are not meant to handle the following situations.
(a) Multiple measurements taken on the same fixed quantity. For example, the weight of a particular sub-atomic particle. In this case the value of multiple measurements arises because the effect of measurement error is averaged out over a large number of observations, thus increasing the precision of the weight’s estimate. More will be said about this below.
(b) Multiple measurements taken on each experimental unit but within a single treatment. For example, in a simple CRD fertilizer experiment, one might measure the height of each plant each day. The measurements within a single plant will likely be positively correlated across time (if it is taller than average one day it will likely be taller than average the next) but valuable information might be gained if these daily values are compared, for example, to weather patterns. It might be found that one type of fertilizer does best overall while another seems to provide the best foundation for growth during particularly dry periods.
When the sequence in which treatments are applied is unimportant, as may be the case in some industrial manufacturing experiments, repeated measures designs are often just special cases of randomized block designs. This is also the case when a particular sequence is of interest and is used by all subjects– as is the case in many learning experiments. In these cases the goal is to increase the sensitivity of the experiment to differences between treatments by controlling for variation due to inherent differences between individuals. The idea is that each subject is a block, and therefore acts as their own control. The response of a subject to each treatment is measured in relation to their overall mean response to all treatments.
Sequence and order dependencies pose special problems. If there is a fairly large number of experimental units then randomizing the order in which they experience treatments, perhaps subject to the constraint that all sequences of interest are observed the same number of times, might control for such effects. This and other ways to handle such problems are discussed below.
The main difference between completely randomized and repeated measures designs is that in the former it is often reasonable to assume independence across all observations, while in the latter case observations are likely to be dependent. If this dependence is not taken into account then inference about treatment effects may be biased or may fail to reveal the true significance of treatment contrasts.
II. Standard ANOVA for repeated measures without order dependencies.
In some cases repeated measures designs can be analyzed using the same type of ANOVA used in randomized complete block designs. This section reviews that ANOVA, provides the necessary and sufficient condition for its validity, and provides two models that, if valid, allow a standard ANOVA analysis.
A. Experiment’s data and analysis:
TreatmentPerson / 1 / … / k / Total / Mean
1 / / … / / /
… / … / … / … / … / …
n / / … / / /
Total / / … / /
G
Mean / / … / /The total sum of squares for this data is
.
The total sum of squares can be decomposed in two useful ways. One is the standard way for block designs. This is:
(1)
where
This is the standard randomized block design decomposition we saw in lecture four. A second way to decompose the total sum of squares is:
(2)
where
This is similar to the CRD sum of squares decomposition, where we saw that the total sum of squares was equal to the sum of within and between sums of squares.
From (1) and (2) it follows that
This is summarized by the following standard ANOVA.
Source of variation / SS / dfBetween people / / n-1
Within people / / n( k-1)
Between treatments / / k-1
Residual
/ / (n-1)(k-1)Total / / nk-1
We would like to test the null hypothesis that different treatments have no effect on outcomes by running the standard F-test. That is, under the null hypothesis, we would like it to be the case that:
B. Ensuring the validity of the F-test.
· A necessary and sufficient condition for the validity of the standard F-test is that the experiment’s covariance matrix is circular in form.
An experiments covariance matrix has the variance of each treatment’s observations along the main diagonal, and the covariance between the observations in the off-diagonal cells. Hence, in the case of a repeated measures design that includes k treatments and n experimental units,
We say that the covariance matrix is circular if:
Hence, a circular covariance matrix ensures that (a) the variance of the difference between any two X variates is a constant and (b) the average variance and the average covariance differ by a constant.
Notice that any scalar multiple of a k X k identity matrix is circular. In fact, this sort of matrix is called “spherical.”
C. Two models that can generate circular covariance matrices.
The strictly additive model.
Here,
Note that the distribution of the error is the same for all treatments and uncorrelated with everything else in the model.
Since the random individual effects and the error terms are assumed to be uncorrelated, it follows that:
Hence, this covariance matrix is circular and the usual ANOVA analysis applies.
A model with interaction that can generate a circular covariance matrix.
where everything is as above, except that
Because of the restriction that within each person the interaction effects must sum to zero, it follows that the interaction effects are not independent, but rather
The expected mean square statistics for this model are as follows.
Source of Variation / Df / MS / E(MS)Between people / n-1 / /
Between treatments / k-1 / /
Residual / (n-1)(k-1) / /
If the covariance matrix is circular then the usual F-statistic is exact. However, this model does not necessarily lead to a circular covariance matrix. If not then the F-test is only approximate.
D. The Box correction
In the interaction model the circularity assumption does not necessarily hold, but if it did the usual F-test would be appropriate and exact. In such situations Box (1954) has come up with a way to modify the degrees of freedom for the F-test so that even if circularity is violated in an arbitrary way the test is extremely accurate.
Continue to assume that there are k treatments in the experiment, so that the experiment’s covariance matrix is k X k. If the true covariance matrix were known, then a measure of the extent to which it departed from circularity could be constructed as follows.
Of course, indicates the entry in row i and column j in the covariance matrix.
It can be shown that with circularity being achieved when it takes value one, and the maximum departure from circularity occurring when
Typically, one will need to estimate using the sample covariance matrix. This can be done as follows:
where the s represent the elements of the estimated covariance matrix.
This estimate is biased. An alternative biased estimator that seems to perform better is
Box (1954) has shown that the F statistic in the repeated measures designs is well approximated by
Hence, in the event that the covariance matrix is not circular one can conduct the standard F-test by simply adjusting the degrees of freedom appropriately.
If the covariance matrix is not circular, but no adjustment to the degrees of freedom is made, the test will have positive bias. This means that it will be “too easy” to reject the null hypothesis of no treatment effects. Another way to think about this is that the experimenter may think they are rejecting the null at a 5% confidence level when the true level of significance is, say, 15%. Note that if the null is not rejected under this test, then one need not do any more work since adjusting the degrees of freedom only makes rejecting the null more difficult. That is, the unadjusted test has the most power.
Since it is difficult to know what the true form of the covariance matrix is, and since in small sample estimates of the covariance matrix will be very imprecise, some argue that one should take a conservative approach and test the null under the “worst case” scenario. If the covariance matrix deviates maximally from circularity, then so the F statistic should be compared to an F(1,n-1) distribution. This test is conservative in the sense that the true significance level of the test will not be greater than the significance level implied by the F(1,n-1) distribution. Thus, if we find evidence of treatment effects we can be confident that they do exist.
If the unadjusted F leads to rejection of the null, and the conservative F does not, then one has no choice but to estimate the correction factor and report the results from that test.
III. Comments on repeated measures designs that address order and sequencing effects.
Earlier we considered examples where the order in which treatments are encountered are expected to have an effect on outcomes. This problem arises particularly when we are concerned that “learning” resulting from experience with one treatment will affect outcomes on other treatments.
A “brute-force” way to handle order or sequence effects is to block them. That is, if there are k distinct treatments, then there are M=k! distinct orders in which the treatments can be encountered. One might model this by assuming a strictly additive model:
where is the effect due to sequence m, or one might specify a model that allows for interactions (modeled as random effects) between the treatments and order, such as
the ANOVA analysis of these models follows the arguments set out above directly as long as the assumptions of those models are met and the design is complete, in the sense that all possible sequences are observed and that there is an equal number of subjects within each sequence. Note that if there is only one subject within each sequence then in the strictly additive model the sequence effect is subsumed within the individual effect, while in the interactive model the sequence effect is subsumed within the residual. Finally, it should be pointed out that this same sort of argument applies to factorial experiments, where in this case each distinct sequence becomes an additional factor (called, it turns out, a “sequence factor.”)
The main drawback to this sort of blocking is that it can be very expensive – both in pecuniary terms and with respect to the experimenters time. The reason is that k! is rather large unless k is fairly small, say less than four or five. As a practical matter, one often needs to use a different approach. One such approach is based on a Latin square.
Example:
Suppose one is doing research that makes use of FMRI technology. Suppose one wants to shed light on the areas of the brain that are important in solving certain types of economic problems. Suppose there are three games of interest: a trust game, a dictator game and a punishment game, Moreover, there are three “players” against which each game might be played: a human, computer1 or computer2, denoted The difference between the two computers is that one plays a stochastic strategy, while the other plays in a known, nonstochastic way. Hence, there are a total of 9 experimental conditions of interest. If we are concerned about order effects, and if we want each subject to be measured in each condition, we must run at least 9!=362,880 subjects through the experiment.
Suppose that we are concerned that subjects will become tired or bored if they stay in the FMRI too long, so we decide that each subject will experience only three of the conditions. Still, we are also concerned that the sequence in which they experience the conditions may affect their responses (in this case, brain activation.) This still requires at least 9*8*7=504 subjects if we want to block out sequence effects. An alternative way to control for these effects using a Latin square, and that requires fewer subjects, is as follows.
One assigns n subjects to each of three groups randomly. Each group plays the same conditions. For example, group 1 plays a1b3 (trust-computer2), a2b1 (dictator-human) and a3b2 (punishment-computer1). To try to remove possible order effects, the sequence in which each subject plays the three treatments is randomized. For instance, one could assign 6 players to each group, and then randomly assign each of the six possible sequences to one member of the group. In this form, the experiment requires only 18 subjects.
Group/Game / / /G1 / b3 / b1 / b2
G2 / b1 / b2 / b3
G3 / b2 / b3 / b1
The model corresponding to this is:
where is a group effect, is an individual effect, and the others are the effects from a, b and their interaction.