Goodness of Fit testing
All of the models and approaches we have discussed so far make very specific assumptions (concerning model fit) that must be tested before using MARK - thus, as a first step, you need to confirm that your starting (general) model adequately fits the data, using GOF tests.
There are two primary purposes for GOF testing.
· it is a necessary first step to insure that the most general model in your candidate model set adequately fits the data.
· Comparing the relative fit of a general model with a reduced parameter model provides good inference only if the more general model adequately fits the data.
What do you do if you run a GOF test and it doesn’t adequately fit the data?
Well it forces you to look at the data and ask why it doesn’t it and of course the answers can be quite revealing.
So what causes lack of fit and what do we mean by it? We mean that the arrangement of the data do not meet the expectations determined by the assumptions underlying the model. In the context of simple mark-recapture, these assumptions, sometimes known as the ‘CJS assumptions’ are:
1. Every marked animal present in the population at time (i) has the same probability of recapture (pi)
2. Every marked animal in the population immediately after time (i) has the same probability of surviving to time (i+1)
3. Marks are not lost or missed.
4. All samples are instantaneous, relative to the interval between occasion (i) and (i+1), and each release is made immediately after the sample.
GOF testing is a diagnostic procedure for testing the assumptions underlying the model(s) we are trying to fit to the data. To accommodate (adjust for, correct for...) lack of fit, we first need some measure of how much extra binomial ‘noise’ (variation) we have. The magnitude of this over-dispersion cannot be derived directly from the various significance tests that are available for GOF testing, and as such, we need to come up with some way to quantify the amount of over-dispersion. This measure is known as a variance inflation factor (, or phonetically, ‘c-hat’).
C-hat is the measure of the lack of fit between the general and saturated models and so as the general model gets ‘further away’ from the saturated model, c-hat 1. Now a saturated model is loosely defined as the model where the number of parameters equals the number of data points - as such, the fit of the saturated model to the data is effectively ’perfect’ (or, as good as it’s going to get).
Many different approaches to estimating this are available and you can find a full description of this in Chapter 5 of the MARK book. There are several approaches you can use including program RELEASE GOF within MARK itself (only applicable to the recapture model) and the Bootstrap, and median c-hat approaches which use simulation and re-sampling to generate the estimate of c hat. Rather than assuming that the distribution of the model deviance is in fact χ2 distributed (since it generally isn’t, for typical ’MARK data’), the bootstrap and median c-hat approaches generate the distribution of model deviances, given the data, and compare the observed value against this generated distribution. The disadvantage of the bootstrap and median c hat approaches (beyond some technical issues) is that both merely estimate c hat. While this is useful (in a practical sense), it reveals nothing about the underlying sources of lack of fit.
Program RELEASE GOF: 2 tests associated with it
Test 2 – tests the assumption that all marked animals are equally detectable
Test 3 - tests the assumption that that all marked animals alive at (i) have the same probability of surviving to (i+1) - the second CJS assumption.
It is easy to run simply highlight your most parameterised model (e.g. in the dipper case Phi(t)p(t)) and select RELEASE GOF under the “tests” tab. It will run automatically and spawn a notepad window. BUT i don’t want you to use this and if you want to know more then read pages 147-155 (Chp5-9 – Chp 5-17)
POINT TO REMEMBER – USE THESE TESTS ON YOUR MOST GENERAL MODEL I.E. THE ONE WITH THE MOST PARAMETERS ONLY.
e.g. IN THE DIPPER CASE Phi(t)p(t).
e.g. IN THE SWIFT CASE Phi(c*t)p(c*t).
BOOTSTRAP APPROACH.
The bootstrap approach simulates data based on the parameter estimates of the model and these simulated data exactly meet the assumptions of the model, i.e., no over-dispersion is included, animals are totally independent, and no violations of model assumptions are included. Data are simulated based on the number of animals released at each occasion. So basically the bootstrap will generate a whole set of deviances, c hat estimates AICc values etc, etc and you would look at where your model deviance falls in the distribution. So if you ran 100 simulations and your model fell between the 80 and 81st simulation (when sorted by deviance values from lowest to highest) you would have an approximation that your model deviance was reasonably likely to be observed P < 0.19 (19 models with a higher deviance/100). How many simulations to use? Run 100 simulations, and do a rough comparison of where the observed deviance falls on the distribution of these 100 values. If the “P value” is > 0.2, then doing more simulations is probably a waste of time - the results are unlikely to change much (although obviously the precision of your estimate of the P-value will improve). However as the value gets closer to nominal significance (say, if the observed P-value is < 0.2), then it is probably worth doing ≫ 100 simulations (say, 500 or 1000). Note that this is likely to take a long time (relatively speaking, depending on the speed of your computer).
To get the bootstrap approach running, simply select this option under the “tests” tab. You will get a window like this. Click the top option and hit OK.
You will then be asked to name the bootstrap results dbf file
Give it a unique identifier – normally in relation to the dataset and model concerned - and click save. You will then get a little confirmation window. After this you will get the following. Now select the number of simulations (start with 100 but see above) and leave the random number seed at 0. The next time you come to do this choose a different random number seed e.g. 1, 2 etc.
As stated above the time taken for this to run will vary on the model, number of simulations and your computer power – so could take a long time.
When completed you need to choose “view simulation results” under the simulations tab. Choose the appropriate file (the one you just named) and you will get a window like the following
You can sort the data using the AZ↓ key and choosing which column header to use – i would use the deviance. You can then check where your model deviance falls in the distribution. You can also generate summary data from the simulations using the little calculator key – very useful.
BUT HOW DO I GET MY ESTIMATE OF C HAT?
Either use:
1) Observed deviance /mean deviance of simulations to estimate c hat or
2) Observed model deviance /deviance df will give you an observed model c hat which you can then divide by the mean of the simulated c hats). You can obtain the observed model c hat by selecting “median c hat” under the “tests” tab (see below).
Which one should i choose? I would calculate both and use the higher value of the 2 as this will make your estimates more conservative.
Recently as an alternative the Median c hat approach has been used ( see MARK book for a full description).
MEDIAN C HAT APPROACH. This gives you an estimate of c hat. Simply select “median c hat” under the “tests” tab. You will get a window like this which gives you the observed model deviance /deviance df at the top. This is the observed model c-hat that you would need in the above.
You should always set the lower bound at 1 and the upper bound at 3. Why these values? 1 because a c hat =1 would be a perfect fit and if c hat>3 then there is probably some fundamental problems with your model. I would set the # intermediate points = 3 and then it is entirely up to you how many replicates you choose. I would use a 100. Click ok and it will run and then spawn a Notepad window which will give you an estimate of c hat at the very top + a se. It will also produce a graph within MARK which will also show the estimate of c hat.
I have my c hat estimates now what do i do?
Well you can adjust your original model estimates (which were based on c-hat =1 and now you can account for some of that over-dispersion). To do this you simply select the “c hat” option under the “Adjustments” tab and enter your value for c hat.
Once you do this your results browser will change - try typing in 1.5 for the Phi(t)p(t) model in the male dipper dataset.
This will result in QuasiAICc values and the weighting of your models will also change. The best fitting model has suddenly become a much better fitting model compared to the others in this example. If you get estimates for these new adjusted models you will notice that the parameter estimates are still the same but the standard errors of those estimates will have changed.
How big a c hat can you use ?
When should you apply a new c hat? Is 1.5 really different than the null, default value of 1.0? At what point is c hat too large to be useful? If the model fits perfectly, then c hat = 1. What about if c hat = 2, or c hat = 10? Is there a point of diminishing utility? As a working “rule of thumb”, provided c hat ≤ 3, you should feel relatively safe. Above 3 and you need to consider whether your model is adequate.