Example of Proposed Methodology

Power analysis for a proposed monitoring plan for a Toba Montrose Hydro project - update

Carl James Schwarz, P.Stat.

Department of Statistics and Actuarial Science

Simon Fraser University

8888 University Drive

Burnaby BC Canada

2010-09-29

1. Introduction

This is a follow-up report from an earlier report dated 2010-03-29 which discussed aspects of a power analysis for an Independent Power Project.

Toba Montrose Hydro, Inc is proposing an independent power production facility on Montrose Creek and East Toba River watersheds. As part of the license agreement, a monitoring study must be implemented with monitoring taking place pre- and post-startup. One of the response variables to be monitored is the density of aquatic invertebrates.

Invertebrate sampling has taken place in fall 2008, spring 2009, fall 2009, and spring 2010. All dates are pre-startup with startup proposed for later in 2010. A series of spreadsheets with the invertebrate data from the two projects was made available for analysis.

The proposed design is the paired-BACI (before-after, control-impact) designs where both the control and project sites are monitored before and after impact. The pairing indicates that sampling occurs at the same time on both sites to account for “seasonal” effects that simultaneously affect the response variable in both sites. It is necessary to monitor the control site to account for changes in the response variable that occur in the absence of an environmental impact. Evidence of an environmental impact is declared if the change in the response variable between pre- and post-project is different in the project site than the control site.

In this proposed monitoring scheme, the control and project sites will be monitored with 5 samples taken each year in both sites in a paired fashion.

2. Power analysis for biomass data.

The invertebrate biomass in a standardized sampling scheme was computed for each sampling event and plots of the standardized biomass are shown in Figures 1a and 1b. These figures show a strong seasonal effect with biomass higher in the fall than the spring, year-to-year variation, and sample-to-sample variation within each sampling event.

Figure 1a. Plot of biomass/sample measured pre-project in Montrose site.

[Points offset in plot to avoid overplotting.]

Figure 1b. Plot of biomass/sample measured pre-project in Toba site.

[Points offset in plot to avoid overplotting.]

The data from the Montrose project shows a somewhat disturbing trend – in the absence of any impact (all measurements taken pre-project) biomass shows a consistent decline over time. The causes for this should be investigated. The data from the Toba project show a season effect (biomass readings in the fall tend to be higher in the fall than in the spring) but no clear pattern across the year pre-project. Both project show substantial variation among samples taken at the same time.

The key determinants for a power analysis are estimates of the variance components (the variation attributable to sampling event variation (event-variation), the variation in the difference in the trend between the control and impact sites over the sampling events (known as the site-event variation), and the sample-to-sample variation within each site-event. These can be estimated by fitting a linear mixed model.

The baseline data were analyzed using Proc Mixed of SAS (Version 9.2). using the following statistical model:

Log(biomass) = location event-R event*location-R sample-R

where the term location measures the average difference between the proposed control and impact site, event-R is the event-to-event variation (includes seasonal and yearly effects); event*location-R measures the site-event variation; and sample-R measures the variation among samples within a particular event-location. A summary of the estimated variance components is found in Table 1.

One of the key-advantages of the paired-BACI design is that because both locations (within a project) are measured at the same time (e.g. fall or spring in year), the effects of the event-to-event variation “cancel” when testing if an environmental impact has occurred. Approximately ¼ of the total variation in the Toba project and about 70% of variation in the Montrose project will be “eliminated” in the paired-BACI design which improves the power to detect difference. The large event-to-event variation in the Montrose project is due to the consistent decline over events seen in the 4 baseline samples. Note however, that the total remaining variation (0.11+0.27=0.37 for Toba, and 0.05+0.31=0.36 for Montrose) is approximately the same.

The sample-to-sample variation effect on the power of the design is modified by the number of samples taken each sampling event. The site-event variation effect on the power of the design is modified by the number of years of sampling conducted pre- and post-startup.

A power analysis was conducted using the methods of Stroup (1999). Basically, the variance components from Table 1 are used to generate pseudo-data that reflect the underlying change of interest (e.g. a 50% decline). Then this pseudo-data is analyzed like real data, and the results of the analysis of the pseudo-data gives information on the power of the proposed design.

The power to detect a 25% and 50% step decline in invertebrate biomass, with 2 years (4 sampling events) pre-startup and 1 to 5 years (with 2 sampling events in the spring and fall of each) of monitoring; and 5 or 10 samples taken in each sampling event post-startup is presented in Table 2.

In most monitoring plans, the number of sub-samples per sampling event (5 or 10 samples in Table 2) hardly affects power. The main determinant of power is the number of independent sampling events (the number of years of monitoring). Because the Montrose project had a smaller sampling-event variance, the power to detect a change was higher than in the Toba project. In neither case, does power reach an acceptable level to detect a 25% step decline in invertebrate biomass even after 5 years post-startup. Power is initially low with only 1 year of post-start up monitoring and is approaching acceptable levels 5 years post-start up.

These results are comparable to those presented in the earlier report (Schwarz, 2010) (a portion of which is presented in Table 3). In the earlier report, the sampling cv was set to 0.2, 0.4 or 0.8. According to Table 1, the sampling variation for both projects had values around 0.3[1]. The common process error and site-specific process cv were set to 0.1 in the previous report; for this report values near 0.1 were obtain (the Site-Event cv in Table 1). We see that power in Table 2 after 5 years of sampling (0.63 for Montrose, 0.48 for Toba) is comparable to those values presented in Table 3 (between 0.56 and 0.85 - highlighted in yellow) for 5 years post-project.

4. Power analysis of species diversity data.

A common measure of species diversity is the Shannon Index ( where is the proportion of the counts in species i in the sample. This index takes into account the number of species and the evenness of the species. The index is increased either by having additional unique species, or by having a greater species evenness. Typically, the value of the index may range from 1.5 for low species richness and evenness to 3.5 for those with high species evenness and richness (McDonald, 2003, p. 409) though values beyond these limits may be encountered. Because the index gives a measure of both species numbers and the evenness of their abundance, it does not give an absolute description of a sites biodiversity. The index is particularly useful when comparing similar ecosystems or habitats, as it can highlight one example being richer or more even than another.

Plots of the diversity index from the baseline data are shown in Figures 2a and 2b. These figures show a strong sampling event variation with large changes among events. The case of this variation seems to be larger than the project scale because of the common (large) decline seen in both projects in the spring 2009 sampling event followed by a rebound in later sampling events.

Figure 2a. Plot of Shannon Index measured pre-project in Montrose site.

[Points offset in plot to avoid overplotting.]

Figure 2b. Plot of Shannon Index measured pre-project in Toba site.

[Points offset in plot to avoid overplotting.]

Similar to the analysis of the baseline biomass data, the baseline diversity data were analyzed using Proc Mixed of SAS (Version 9.2). using the following statistical model:

H = location event-R event*location-R sample-R

where the term location measures the average difference between the proposed control and impact site, event-R is the event-to-event variation (includes seasonal and yearly effects); event*location-R measures the site-event variation; and sample-R measures the variation among samples within a particular event-location. A summary of the estimated variance components is found in Table 4.

The total variation over time (event-to-event) variation is again larger in the Montrose project than the Toba project, but the Toba project is much more variable among samples within an event (the sample-to-sample variation).

A similar power analysis was performed for the ability of the project to detect changes in biodiversity. Here it is difficult to know exactly what a shift in biodiversity actually implies and so the size of biologically important changes in biodiversity are difficult to quantify. A common measure of `biological effect’ size when no suitable measures are available is a one standard deviation shift in the mean. This corresponds to roughly a drop of about 25% of the range (typically the range is about 4 standard deviations). Table 4 suggests that we examine change in diversity then of about to units or between a 20% to 25% drop in diversity from a baseline of 3.0. We estimated the power to detect a drop of 0.25 and 0.50 in the diversity index using similar methods as the power analysis for total biomass. The results are summarized in Table 5.

Power is reasonable large for a 0.50 step decline after 3 or 4 years of sampling post-startup. Power is never very larger for a 0.25 step decline (but this is a very small decline).

Halse et al (2002) suggest that ordination methods may be more suitable for examining changes in diversity, composition, and substitutions.

5. Conclusion.

This report is an update to Schwarz (2010) where a previous power analysis for changes in biomass had to make certain assumptions about the size of the process errors because of the lack of information from the pre-implementation phase of the study. Schwarz (2010) chose values of .1 for the common and site-specific process cv. In this report we found that the site-specific process cv was indeed around 0.1. Consequently, the conclusions from the earlier report can be used and our new power study is consistent with previous studies.

The pairing of the design implies that the common process error (the event process error) “cancels”. This is particularly important in the Montrose project where almost 70% of the total variation in biomass among samples can be “eliminated” by the paired design.

The proposed design also has reasonable power for detecting a step change of 0.50 in species diversity, but the lingering question is the ability to come up with a sensible measure of `biological effect’ for species diversity.

References:

Halse, S.A, Cale, D.J., Jasinska, E.J., and R.J. Shiel, R.J. (2002). Monitoring change in aquatic invertebrate biodiversity: sample size, faunal elements and analytical methods. Aquatic Ecology, 36, 395-410. DOI: 10.1023/A:1016563001530

McDonald, G. (2003) Biogeography: Space, Time and Life, Wiley.

Schwarz, C. J. (2010). Review of a proposed monitoring plan for a Toba Montrose Hydro project. Dated 2010-03-29.

Stroup, W. W. (1999). Mixed model procedures to assess power, precision, and sample size in the design of experiments. Pages 15-24 in Proc. Biopharmaceutical Section. Am. Stat. Assoc., Baltimore, MD. Available at:

1

[1] The standard deviation on the log-scale is almost exactly equal to the cv on the original (anti-log) scale.