Minitab Macros for
Resampling Methods
By Adam Butler
CEH Monks Wood
September 2001
SUMMARY
This report describes a library of macros for implementing a variety of statistical methods in Minitab using computationally-intensive methods of inference (randomization, bootstrapping and Monte Carlo simulation).
CONTENTS
INTRODUCTION 4
1 Resampling methods in statistics 4
What are they ? 4
When should I used them ? 4
Randomization, bootstrapping and Monte Carlo simulation 5
A note on the use of p-values 5
2 Resampling in Minitab 7
Minitab 7
Some useful Minitab commands 7
The resampling macros 8
Other sources of information 8
Arguments to the macros 8
Subcommands in the macros 8
Computing power and number of resamples 9
Speed 10
3 How to use this guide 11
Information about the macros 11
Worked examples 12
4 Literature review 13
REFERENCE MANUAL 14
1 Significance tests 14
Overview 14
ONESAMPLERAN 15
TWOSAMPLERAN 18
TWOTRAN 22
TWOTPOOLBOOT 26
TWOTUNPOOLBOOT 29
CORRELATIONRAN 32
2 Confidence intervals 35
Overview 35
An introduction to bootstrap confidence intervals 35
MEANCIBOOT 37
MEDIANCIBOOT 41
STDEVCIBOOT 45
ANYCIBOOT 49
3 Analysis of variance 53
Overview 53
ONEWAYRAN 54
TWOWAYRAN 58
TWOWAYREPRAN 62
LEVENERAN 66
4 Regression 70
Overview 70
Should we resample residuals or observations ? 70
REGRESSSIMRAN 72
REGRESSOBSRAN 76
REGRESSRESRAN 79
REGRESSBOOT 83
5 Time series 89
Overview 89
ACFRAN 90
TRENDRAN 96
6 Spatial statistics 100
Overview 100
Which procedure should I use ? 100
Using the macros for spatial statistics 101
SPATAUTORAN 102
MANTELRAN 107
MEAD4RAN 111
MEAD8RAN 114
Creating and interpreting EDF plots 117
DISTEDFMC
NEARESTMC
LOCREGULARMC
TABLE OF ALTERNATIVE DATASETS 137
REFERENCES 138
ACKNOWLEDGEMENTS AND CONTACT DETAILS 139
APPENDIX : Reference card for the macros
INTRODUCTION
1 RESAMPLING METHODS IN STATISTICS
What are they ?
· Resampling methods are a class of statistical techniques for drawing inferences based on the variability present within a dataset
· Resampling methods (sometimes known as computationally intensive methods) include :
· Bootstrapping
· Randomization tests (also known as permutation tests)
· Monte Carlo tests and related procedures
· In general, resampling methods are difficult to justify in theory, but relatively easy to apply in practice.
· The common concept underlying all resampling methods is that we can assess the variability by drawing a large number of samples, each having the same size as the original dataset, from the observed data (this is the process of resampling); we then compare the properties of the observed data to the properties of the resampled datasets.
When should I use them ?
· Resampling methods are useful for obtaining assessments of variability - this means that they are principally used to calculate confidence intervals and p-values.
· Resampling methods can be used with many different statistical methods - including comparison of two samples, ANOVA, regression, spatial statistics, time series and multivariate analysis - and can potentially be applied to any area of application; Manly (1997) discusses how resampling methods have been applied in a number of different areas of biology.
· Resampling methods have become increasingly popular in recent year, partly because of increasing computer power.
· Resampling methods are usually used instead of - or alongside - standard techniques for drawing inferences from data. Standard techniques usually rely upon statistical theory (especially asymptotic arguments) and assumptions about the distribution of the data (for example, that the data are normally distributed). Resampling methods do not make these assumptions, and so should be more reliable in those situations in which the standard assumptions are false.
· If the assumptions underlying standard theory are valid, resampling and standard techniques should give very similar results. In fact, resampling methods often give similar results to standard theory even if the assumptions underlying standard theory are not valid.
· Resampling methods also rely upon their own (fairly complicated) assumptions. It is felt that these assumptions will often be valid, or approximately valid, but it is worth noting that there are situations in which the application of resampling methods may go badly wrong.
· Resampling methods place much emphasis on the observed dataset, and so may be very susceptible to any errors or problems with the data that has been collected. It is therefore important to check data carefully, and to use graphical techniques to look for outlying points.
· Possibly the most interesting feature of resampling methods is their generality - they may be used to tackle a wide variety of practical statistical problems, including problems for which standard theory does not yet exist, in a fairly straightforward way.
Randomization, bootstrapping and Monte Carlo simulation
The macros in this library sometimes use randomization tests, and sometimes use bootstrapping. The differences between the two techniques are rather subtle. The key differences are that :
· In practice, if we are in a situation in which either method can be used, then the methods work in an almost identical fashion. Usually the only difference is that randomization tests involve resampling without replacement (i.e. we simply re-order the original data), whereas bootstrapping involves resampling with replacement (i.e. a value from the original data may occur more than once in a resampled dataset).
· Bootstrap methods are substantially more general than randomization methods, and may often be used in situations in which randomization methods are not available.
· The assumptions which justify the use of the two techniques are different.
A few of the macros for spatial statistics used Monte Carlo methods; these are a more general class of technique than either randomization or bootstrap methods (in fact, both randomization and bootstrap methods can be viewed as special cases of Monte Carlo methods). Whilst bootstrap and randomization methods involve simulating only from the observed data, Monte Carlo methods involve taking simulations using a statistical model. All of the Monte Carlo methods in this study involve simulating datapoints at random from within a fixed rectangular region, in order to examine the hypothesis of Complete Spatial Randomness (CSR).
A note on the use of p-values
The bulk of the macros in the library deal with significance tests. In general, these involve testing a null hypothesis against one or more possible alternative hypotheses. Performing a significance test involves calculating the location of the observed test-statistic value t within the probability distribution of the test-statistic. Assume that the true distribution is T. This probability distribution can often be approximated either using statistical theory, or, as is the case in the macros, by resampling.
One-sided randomization p-values
Assume for the time being that we are only interested in the alternative hypothesis which implies a large value of the test-statistic. Then the true one-sided p-value of interest is
Standard procedure
If the test-statistic is known by statistical theory to follow a particular distribution, Ta, then the standard one-sided p-value of interest is given by
.
For a continuous distribution, this value is obtained by integrating the probability density from t to infinity, whilst for a discrete distribution the probability mass function is summed from t to infinity (inclusive).
Randomization
If the resampling distribution is given by Tr, then the resampling one-sided p-value of interest is given by
.
This value is therefore the proportion of all test-statistics (the set of resampled test-statistics, plus the observed test-statistic, since under the null hypothesis this is also a realisation from T) greater than or
equal to t.
Two-sided randomization p-values
Now assume that either alternative hypothesis may be of interest.
One-sided randomization p-values for the opposite alternative hypothesis, corresponding to small values of the test-statistic, are analogous to those defined above. Calculating the two-sided p-value, i.e. the probability of the test-statistic being extreme in either direction is more complicated, because there are now two possible approaches :
1. Let the two-sided p-value be the probability of being as far from the mean of the
distribution as the observed test-statistic, in either direction
2. Let the two-sided p-value be double the smaller of the one-sided p-values
Most standard theory either uses distributions for which only one-sided p-values are relevant (e.g. F and chi-squared distributions), or uses distributions (e.g. normal or T distributions) which are symmetric. For a symmetric distribution, either method of computing the 2-sided p-value will give the same answer, because both of the one-sided p-values will be the same.
When we obtain distributions by resampling, however, there is no reason to assume that they will be symmetric. For very non-symmetric distributions, the first approach to computing two-sided p-values may give substantially misleading results. The disadvantage of using the second approach in the resampling context is that we use the data from only one tail of the distribution, so that we need a greater number of resamples to give the same accuracy in the calculation of p-values. Since resampling methods are most useful for those situations in which standard approximations are not valid - i.e. for situations in which test-statistics have highly skewed distributions - we use the second method of computing resampling two-sided p-values (where these are required) throughout the library.
2 RESAMPLING IN MINITAB
MINITAB
MINITAB is a general purpose package for data manipulation and statistical analysis.
This guide outlines a library of MINITAB macros which have been written to perform a variety of commonly-used statistical procedures using randomization and bootstrapping methods, rather than the more traditional (and less computationally-intensive) methods which involve approximations and distributional theory.
In order to implement the macros, you must work in the session window.
To open the session window, click on window on the menu bar. Move down the list, and select session. Click on the editor menu, move down the list and select enable commands.
The various macros may be invoked by typing their name at the MTB> prompt; all other MINITAB commands can also be invoked from this prompt.
Some useful MINITAB commands
The most useful MINITAB commands in the context of randomization and bootstrapping are -
statistics
Can be used to display or store a wide variety of descriptive statistics for a given column. The subcomm-ands specify the various descriptive statistics to be used.
Example:
statistics c1;
mean c2.
This takes the mean of column c1 and stores it in the first element of column c2.
sample
Can be used to draw a sample, with or without replacement, from a column.
Example:
sample 10 c1 c2
This takes a sample of size 10 from column c1, without replacement, and stores it in c2.
Randomization tests are based upon sampling without replacement.
Example:
sample 7 c1 c2 ;
replace.
This takes a sample of size 7 from column c1, with replacement, and stores it in c2.
Bootstrapping is based upon sampling with replacement.
random
Can be used to simulate random datasets from standard probability distributions.
Example:
random 50 C1;
normal 0 1.
This simulates 50 values from a standard normal [i.e. a Normal(0,1)] distribution, and stores the simulated values in c1.
Example:
random 10 c2;
poisson 6.
This simulates 10 values from a Poisson distribution with parameter 6, and stores the values in c2.
The resampling macros
The resampling macros are designed, as far as possible, to mimic standard MINITAB functions for the statistical methods in question. In some cases, there are both randomization and bootstrap versions of standard MINITAB commands; the justification for using randomization and bootstrap techniques is substantially different, but they will often (though not always) give similar answers. Much of the output from the macros will be identical to output from the standard MINITAB functions, since it is not dependent upon the randomization or bootstrapping process (for example correlation coefficients, regression parameter estimates, ANOVA tables and sample statistics will not be affected by using randomization or bootstrap techniques in place of standard techniques). However, assessments of the significance or variability of an estimate (such as p-values and confidence intervals) will be altered by the use of randomization and bootstrap techniques. It is important to realise that MINITAB functions for most standard techniques will yield the same answers again and again, regardless of how many times they are run; in contrast, p-values and confidence intervals produced using the resampling macros will be different every time the macro is run. This is an inherent feature of randomization and bootstrap techniques; so long as the number of randomizations or bootstrap samples is large (how large depends upon the particular statistical method being used), these differences should not be particularly important.
The Minitab macros are designed for release 13, but most will probably function with earlier releases.
The macro generally have a similar calling statement to the corresponding standard Minitab command, if this exists. Additionally, macro names end in a suffix, depending upon the type of resampling methodology used:
· ran : for randomization procedures
· boot : for bootstrap procedures
· mc : for Monte Carlo tests and related procedures.
Although these three classes of methods are all fairly similar to implement in practice, the theoretical justification for using them is different, so we distinguish clearly between the different forms of resampling.
Other sources of information
Along with the macros library, we provide :
· Individual descriptions of each macro (taken from the sections of this guide)
· Sample datasets, as Minitab files
· Sample datasets, as .DAT files
· .TXT files, containing output from running the macros over the sample datasets
· Minitab files, containing the final worksheet obtained after running the macros over the sample datasets (although these are missing for some macros)
Arguments to the macros
· Most of the macros require one or more columns of numeric data as input.
· For some macros, the order in which the columns are entered is crucially important. In regression, the response must be the first column, followed by one or more predictors. In two-way analysis of variance, the group must be entered before the block.
· For some of the spatial macros, the input is in the form of matrices. Consult the Minitab documentation or help menu for information on entering and reading matrices.