Turner Answer Key for Chapter Seven

Using statistics in small-scale language education research: Focus on non-parametric data

Chapter Seven

Practice Problems Answer Key

Study A: Re-analysis of Jen’s data

The data are presented in Table 6.2., but a downloadable dataset can be found on the Companion Website ( Go to the For Students tab and select Data Sets; you can find the data for this problem from the Chapter Six menu (Box 6.5) or from the Chapter Seven menu (Jen’s data).Click on the dataset and save it onto your computer using the csv (DOS) format. Remember where you saved it so you can import it into R for analysis.

Remember from Chapter Six (pp. 141-154) that Jen wanted to determine whether learners’ perceived usefulness of authentic differed from their perceived usefulness of instructional materials (see p. 142 for a detailed discussion of her dependent variable and data). Because the assumption of normal distribution was not met (p. 144), we’ll now re-analyze the data using the Wilcoxon Signed Rank statistic. The formal research hypotheses below are phrased for this non-parametric analysis.

Step 1. State hypotheses

H0: There is no statistically significant difference between the rankings of the learners’ perceptions of the usefulness of the authentic versus the instructional materials.

H1: There is a statistically significant positive difference in the rankings of learners’perceptions of the usefulness of the authentic materials versus instructional materials.

H2: There is a statistically significant negative difference in therankings of the learners’ perceptions of the usefulness of the authentic materials versus instructional materials.

Step 2. Set alpha

alpha = .01

Step 3. Identify the appropriate statistic for the analysis

I propose to analyze the data using the Wilcoxon Signed Rank statistic and converting the observed value of the statistic to a z-score to interpret statistical significance because:

1) the independent variable is nominal and has two levels;

2) the comparison is of two sets of data for exactly the same set of participants;

3) there are 30 or more participants;

4) the dependent variable yields rankable data.

Step 4. Collect the data.(Retrieve from Companion Website at

Step 5. Check the assumptions

1 & 2) the two levels of the independent variable are represented by the same people---there are two “perceived usefulness” scores for each person (one score is the perceived usefulness of authentic materials and the other is the perceived usefulness score of instructional materials);

3) there are 93 participants

4) when I review the data, I see that they are rankable.

Step 6. Calculate the observed value of the statistic

I use R to calculate the observed value of the Wilcoxon Signed Rank statistic and the exact probability of the observed value. (If you’d rather do the calculations with a calculator, see pages 203-206 for guidance.)

jen.data =read.csv(file.choose (), header =T) [import the Excel dataset in csv format; I named the dataset “jen.data”]

> View (jen.data)[view the dataset]

summary(jen.data$totauth) [calculate the mean and median]

Min. 1st Qu. Median Mean 3rd Qu. Max.

4.00 23.00 28.00 26.88 33.00 36.00

summary(jen.data$totinst) [calculate the mean and median]

Min. 1st Qu. Median Mean 3rd Qu. Max.

4.00 21.00 26.00 24.76 29.00 36.00

sd(jen.data$totauth) [calculate standard deviation]

[1] 7.474376

sd(jen.data$totinst) [calculate standard deviation]

[1] 6.138612

par(mfrow = c(1,2)) [set up a space for side-by-side histograms]

hist(jen.data$totauth, col = "tomato", breaks =10)[make 1st histogram]

hist(jen.data$totinst, col = "sea green 1", breaks =10)[make 2ndhistogram]

shapiro.test(jen.data$totauth)[calculate Shapiro Wilk statistic—a normal distribution isn’t required for this analysis, but check it anyway!]

Shapiro-Wilk normality test

data: jen.data$totauth

W = 0.9195, p-value = 2.542e-05

shapiro.test(jen.data$totinst)[calculate Shapiro Wilk statistic—a normal distribution isn’t required for this analysis, but check it anyway!]

Shapiro-Wilk normality test

data: jen.data$totinst

W = 0.9519, p-value = 0.001797

wilcox.test(jen.data$totauth, jen.data$totinst, paired = T) [calculate the observed value of the Wilcoxon Signed Rank statistic; perception for authentic materials entered first]

Wilcoxon signed rank test with continuity correction

data: jen.data$totauth and jen.data$totinst

V = 2517.5, p-value = 0.005327

alternative hypothesis: true location shift is not equal to 0

wilcox.test(jen.data$totinst, jen.data$totauth, paired = T) [calculate the observed value of the Wilcoxon Signed Rank statistic; perception for instructional materials entered first]

Wilcoxon signed rank test with continuity correction

data: jen.data$totinst and jen.data$totauth

V = 1223.5, p-value = 0.005327

alternative hypothesis: true location shift is not equal to 0

The observed value of the Wilcoxon Signed Rank statistic is the smaller of the two observed values of V, V = 1223.5.

Step 7.Calculate the exact probability of the statistic

I simply retrieve the exact probability from the R output, so exact p =.005327.

Step 8.Compare the exact probability to alpha

The rules for interpreting exact probability are:

If exact probability ≥ alpha → accept the null hypothesis

If exact probability < alpha → reject the null hypothesis

The exact probability, p = .005227, is less than alpha, .01, so reject the null hypothesis. I accept the first alternative hypothesis because I see that the mean perceived usefulness of the authentic materials is higher than the mean perceived usefulness of the instructional materials.

H1: There is a statistically significant positive difference in therankings of the learners’ perceptions of the usefulness of the authentic materials versus instructional materials.

Step 9. Make the probability statement

There’s 99% certainty the rankings of the learners’ perceptions of the usefulness of the authentic materials are systematically higher than their perceptions of the usefulness of instructional materials.

Step 10. Interpret the meaningfulness

There are two avenues for interpreting meaningfulness: 1) with reference to the research question, and 2) by calculating effect size.

The research question that guided this part of Jen’s research was: What are language learners’ perceptions of the usefulness of authentic versus instructional materials?” (p. 142).

To calculate effect size, the observed value of the Wilcoxon Signed Rank statistic, V, is converted to a z-score. The R commands for doing so are presented below—I use the command with the values for “totinst” because it yields the lower value of V observed:

study.model = wilcox.test (jen.data$totinst, jen.data$totauth, paired =T)

study.model

Wilcoxon signed rank test with continuity correction

data: jen.data$totinst and jen.data$totauth

V = 1223.5, p-value = 0.005327

alternative hypothesis: true location shift is not equal to 0

> z=qnorm(study.model$p.value/2)[This command retrieves the value of z from the calculation.]

z[Type z to see the value of z.]

[1] -2.78657

effect.size = abs(z)/sqrt(93)[Calculate the effect size using this formula; 93 is the number of participants.

effect.size[Type effect.size to see the effect size itself.]

[1] 0.2889538

Here is my interpretation of the meaningfulness of the findings:

For these 93 language learners, there is 99% certainty of a statistically significant difference in their perception of the usefulness of authentic versus instructional materials, with the learners reporting that authentic materials are more useful (Wilcoxon Signed rank value = 1223.5, p = .005327; effect size = .29). The effect size is moderately strong. The ex post facto design does not support causal interpretation of the findings, but indicates a need for further research on learners’ perception of the usefulness of these two types of learning materials.

Study B:

1. The independent variable is +/- instruction in reading strategies.

2. The dependent variable is reading comprehension.

3. Student level of proficiency is a control variable—only high beginners are included in the study. There is no explicit moderator variable.

4. The instruction on reading strategies is a new component added to the course; if the teacher previously taught the course, she may have some difficulty fitting the new component into the program. Perhaps the two teachers could meet regularly to discuss how their classes are progressing and work together to find solutions to how to fit the new info into the existing course.

5. External validity is threatened by the fact that only high beginners participated in the study. This problem could be addressed by including intermediate level participants, too.

6. Descriptive statistics are presented in the chart below. The R commands are presented below the chart.

Non-experimental
Group / Experimental Group
1 / 47 / 45
2 / 42 / 44
3 / 39 / 42
4 / 37 / 42
5 / 37 / 42
6 / 36 / 37
7 / 32 / 34
mean / 38.57 / 40.86
mode / 37.00 / 42.00
median / 37.00 / 42.00
sd / 4.79 / 3.93
range / 15 points / 11 points

non.exp = c(47, 42, 39, 37, 37, 36, 32) [enter data for the non-experimental group]

exp = c(45, 44, 42, 42, 42, 37, 34)[enter data for the experimental group]

summary (non.exp)

Min. 1st Qu. Median Mean 3rd Qu. Max. [calculate mean, median, and minimum & maximum scores]

32.00 36.50 37.00 38.57 40.50 47.00

summary (exp)

Min. 1st Qu. Median Mean 3rd Qu. Max. [calculate mean, median, and minimum & maximum scores]

34.00 39.50 42.00 40.86 43.00 45.00

subset(table(non.exp), (table(non.exp)==max(table(non.exp)))) [find mode]

sd(non.exp)[calculate standard deviation]

[1] 4.790864

> 47-32[use minimum & maximum to find range]

[1] 15

subset(table(exp), (table(exp)==max(table(exp)))) [find mode]

sd(exp)[calculate standard deviation]

[1] 3.933979

> 45-34[use minimum & maximum to find range]

[1] 11

hist (non.exp, col = "dark sea green", breaks = 10)

hist (exp, col = "medium spring green", breaks = 10)

7. Follow the steps in statistical logic....

Step 1. State hypotheses

H0: There is no statistically significant difference in the rankings of the reading comprehension scores for the group that received reading strategies and the group that did not.

H1: There is a statistically significant positive difference in the rankings of the reading comprehension scores for the group that received reading strategies and the group that did not.

H2: There is a statistically significant negative difference in the rankings of the reading comprehension scores for the group that received reading strategies and the group that did not.

Step 2. Set alpha

I set alpha at .05.

Step 3. Identify the appropriate statistic for the analysis

I propose to the Wilcoxon Rank Sum statistic to analyze the data because:

1) the independent variable is nominal and has two levels;

2) the two groups include different participants;

3) the dependent variable yields rankable data.

Step 4. Collect the data

The data are presented in the table above.

Step 5. Check the assumptions

1 & 2) there are two levels of the independent variable and they are represented by different people;

3) review of the data shows that the values are rankable.

Step 6. Calculate the observed value of the statistic.

The R commands are presented below:

wilcox.test (non.exp, exp)

Wilcoxon rank sum test with continuity correction [The highlighted warning indicates that “exact = FALSE” should be added so the appropriate formula is used to calculate the exact p-value.]

data: non.exp and exp

W = 16.5, p-value = 0.3304

alternative hypothesis: true location shift is not equal to 0

Warning message:

In wilcox.test.default(non.exp, exp) :

cannot compute exact p-value with ties

wilcox.test (non.exp, exp, exact =F)[Calculate Wilcoxon Rank Sum value]

Wilcoxon rank sum test with continuity correction

data: non.exp and exp

W = 16.5,p-value = 0.3304

alternative hypothesis: true location shift is not equal to 0

wilcox.test (exp, non.exp, exact =F)[Reverse order of datasets to identify the lower of the two observed values.

Wilcoxon rank sum test with continuity correction

data: exp and non.exp

W = 32.5, p-value = 0.3304

alternative hypothesis: true location shift is not equal to 0

Step 7. Retrieve the exact probability to consider the formal research hypotheses

Using the exact probability approach—

Retrieve the exact probability from the R output; p = .3304.

Step 8. Consider the formal research hypotheses

Using the exact probability approach—

Use the rules for comparing exact probability to alpha:

If exact probability ≥ alpha → accept the null hypothesis

If exact probability < alpha → reject the null hypothesis

The exact probability, p = .3304 is greater than alpha, .05, so accept the null hypothesis.

H0: There is no statistically significant difference in the rankings of the reading comprehension scores for the group that received reading strategies and the group that did not.

Step 9. Make the appropriate probability statement

There’s 95% certainty of no statistically significant difference in the rankings of the reading comprehension scores for the group that received reading strategies and the group that did not.

Step 10. Interpret meaningfulness

The researchers found that for this small group of participants there is no statistically significant in the reading comprehension of learners who received instruction on reading strategies and learners who did not receive instruction on these strategies (Wilcoxon Rank Sum W = 16.5, p = .3304). Effect size cannot be calculated due to the small number of participants (Turner, 2014, p. 179).

Study C:

8. The independent variable is +/- study in the intermediate level Summer Intensive Arabic Program

9. The dependent variable is Arabic oral abilities.

10. The design is ex post facto, or if one considers participation in the Summer Intensive Language Program to be a treatment, the study is pre-experimental (there is no legitimate comparison group; there’s just the one group of students before and after participation in the program).

11. Descriptive statistics & R commands

Pretest / Posttest / R commands
1 / 62 / 80
2 / 48 / 60
3 / 50 / 50
4 / 30 / 48
5 / 30 / 44
6 / 28 / 26
7 / 26 / 30
mean / 39.14 / 48.29 / > summary (pretest)
Min. 1st Qu. Median Mean 3rd Qu. Max.
26.00 29.00 30.00 39.14 49.00 62.00
> summary (posttest)
Min. 1st Qu. Median Mean 3rd Qu. Max.
26.00 37.00 48.00 48.29 55.00 80.00
mode / > subset(table(pretest), (table(pretest)==max(table(pretest))))
30
2
> subset(table(posttest), (table(posttest)==max(table(posttest))))
posttest
26 30 44 48 50 60 80
1 1 1 1 1 1 1
median / 30.00 / 48.00 / See above (summary command for mean)
sd / 14.04 / 18.24 / sd (pretest)
[1] 14.04076
sd(posttest)
[1] 18.23654
range / 36 points / 54 points / > 62-26
[1] 36
> 80-26
[1] 54

12. Steps in statistical logic

Step 1. State hypotheses

H0: There is no statistically significant difference in the rankings of the Arabic oral skills of the group before they received the SILP instruction and after they had completed the SILP instruction.

H1: There is statistically significant positive difference in the rankings of the Arabic oral skills of the group before they received the SILP instruction and after they had completed the SILP instruction.

H2: There is statistically significant negative difference in the rankings of the Arabic oral skills of the group before they received the SILP instruction and after they had completed the SILP instruction.

Step 2. Set alpha

I set alpha at .05.

Step 3. Identify the appropriate statistic for the analysis

I propose to the Wilcoxon Signed Rank statistic to analyze the data because:

1) the independent variable is nominal and has two levels, represented by the same people;

2) there are two sets of dependent variable data from the same group of people;

3) the dependent variable yields rankable data.

Step 4. Collect the data

The data are presented in the table above.

Step 5. Check the assumptions

1 & 2) there are two levels of the independent variable and they are represented by same people;

3) review of the data shows that the values are rankable.

Step 6. Calculate the observed value of the statistic.

The R commands are presented below:

wilcox.test(pretest, posttest, paired=T, exact = F)

Wilcoxon signed rank test with continuity correction

data: pretest and posttest

V = 1, p-value = 0.05848

alternative hypothesis: true location shift is not equal to 0

wilcox.test(posttest,pretest, paired =T, exact =F)

Wilcoxon signed rank test with continuity correction

data: posttest and pretest

V = 20, p-value = 0.05848

alternative hypothesis: true location shift is not equal to 0

Step 7. Retrieve the exact probability to consider the formal research hypotheses

Using the exact probability approach—

Retrieve the exact probability from the R output; p = .05848

Step 8. Consider the formal research hypotheses

Using the exact probability approach—

Use the rules for comparing exact probability to alpha:

If exact probability ≥ alpha → accept the null hypothesis

If exact probability < alpha → reject the null hypothesis

The exact probability, p = .05848 is greater than alpha, .05, so accept the null hypothesis.

Step 9. Make the appropriate probability statement

There’s 95% certainty of no statistically significant difference in the rankings of the group’s oral skills before and at the end of the SILP instruction.

Step 10. Interpret meaningfulness

The researchers found that for this small group of participants there is no statistically significant in the oral skills of learners before and after they participated in the Summer Intensive Language Program (fabricated data) (Wilcoxon Signed Rank V = 1, p = .05848). Effect size cannot be calculated due to the small number of participants (Turner, 2014, p. 179).