Ch. 11: Tasks & Quizzes

TipYour turn!

Q11.1 Which of the following statements best describes random sampling?

Participants are selected because they are easy to reach (e.g. university students).
Each individual in the population has the same known probability of being chosen (e.g. 0.001%).
Every member of the target population is guaranteed to take part in the study.
The sample is built so that its composition (age, gender, socioeconomic status, etc.) matches known census figures.

Q11.2 True or false: In a convenience sample, the researcher can ensure that the sample is representative of the entire population if they are careful to match the sample to census demographics (age, gender, region, etc.).

True
False

TipYour turn!

Consider the following research question:

• Across all adult English speakers in the UK, do L1 speakers, on average, achieve higher scores than L2 speakers on the English receptive vocabulary test?

Q11.3 Which null hypothesis could you formulate for this research question?

Across the entire adult UK population, there is no difference in the mean scores of L1 vs. L2 speakers on the English receptive vocabulary test.
Across the entire adult UK population, the mean scores of both L1 and L2 speakers on the English receptive vocabulary test are null.
Across the entire adult UK population, there is no statistically significant difference between the mean scores of L1 speakers and those of L2 speakers on the English receptive vocabulary test.

Q11.4 Run a t-test on the Dąbrowska (2019) data to test the null hypothesis that you selected in Q11.3 above. What is the value of the t-statistic?

133.83
23.240497
4.6768
9.425801
0.000007032

🐭 Click on the mouse for a hint.

# It is always a good idea to visualise the data before running a statistical test:
Dabrowska.data |> 
  ggplot(mapping = aes(y = Vocab,
                       x = Group)) +
  geom_boxplot(alpha = 0.7) +
  theme_bw()

# Running the t-test:
t.test(formula = Vocab ~ Group, 
       data = Dabrowska.data)

Q11.5 What is the p-value associated with the t-test you ran above?

0.0000007032
0.000007032
7.03206
7.032 multiplied by minus six
7.032 to the power of minus six

Q11.6 The p-value is very, very small. What does this mean? Select all that apply.

Given the sample size, the results of this t-test are highly precise.
The true difference in the mean receptive English vocabulary scores of English native and non-native speakers is not equal to zero.
The true difference (across the full population) in the mean receptive English vocabulary scores of English native speakers is only very slightly higher than that of non-native speakers.
The difference in the mean receptive English vocabulary scores of English native and non-native speakers is highly relevant.
If there were no association between English native-speaker status and receptive English vocabulary scores, it is very unlikely that we would observe this difference in mean values by chance alone.

🐭 Click on the mouse for a hint.

TipYour turn!

Compute three Cohen’s d values to capture the magnitude of the difference between L1 and L2 speakers:

1. English grammar (Grammar) test scores
2. English receptive vocabulary (Vocab) test scores, and
3. English collocation (Colloc) test scores.

As these are three standardised effect sizes, we can compare them.

Q11.7 Comparing the three Cohen’s d that you computed, which statement(s) is/are true?

The smallest effect size is observed in the collocation test.
The difference in all three effect sizes is statistically significant.
By far the largest difference between L1 and L2 speakers is in the English collocation test.
The largest difference between L1 and L2 speakers is in the receptive English vocabulary test.
The effect sizes across the three English tests are practically equal.

🐭 Click on the mouse for a hint.

cohens_d(Grammar ~ Group, 
       data = Dabrowska.data)

cohens_d(Vocab ~ Group, 
       data = Dabrowska.data)

cohens_d(Colloc ~ Group, 
       data = Dabrowska.data)

Q11.8 Now compute 99% confidence intervals (CI) around the same three Cohen’s d values. Which statement(s) is/are true?

The most relevant difference can be found in the collocation test data.
All three differences in means are statistically confident at the α-level of 0.01.
For all three effect sizes, the 99% CIs are wider than the 95% CI.
In 99% of cases, L1 speakers perform better than L2 speakers on all three tests.
All three differences in means are statistically significant at the α-level of 0.01.
We can be 99% confident that all three differences in means are real.

🐭 Click on the mouse for a hint.

cohens_d(Grammar ~ Group, 
       data = Dabrowska.data,
       ci = 0.99)

cohens_d(Vocab ~ Group, 
       data = Dabrowska.data,
       ci = 0.99)

cohens_d(Colloc ~ Group, 
       data = Dabrowska.data,
       ci = 0.99)

Q11.9 Consider the code and its outputs below. We are now comparing the English grammar comprehension test scores of male and female native speakers of Romance languages only. Hence, we are now looking at a very small sample size comprising just five female and one male participants.

Dabrowska.Romance <- Dabrowska.data |> 
  filter(NativeLgFamily == "Romance")

table(Dabrowska.Romance$Gender)

F M 
5 1 

cohens_d(Grammar ~ Gender,
         data = Dabrowska.Romance)

Cohen's d |        95% CI
-------------------------
-1.04     | [-3.24, 1.27]

- Estimated using pooled SD.

Which statement(s) is/are true about the standardised effect size computed for the difference in the English comprehension grammar scores of male and female Romance L1 speakers in this very small dataset (n = 6)?

The difference in mean grammar test scores across the two genders is not statistically significant at the α-level of 0.05.
We cannot reject the null hypothesis that there is no difference between male and female performance on the grammar test.
The effect size is negative, which means that the single male participant performed better on the grammar test than the five female participants.
According to Cohen's rule of thumb, the effect size is very small.
All six participants did equally well at the α-level of 0.05.
According to Cohen's rule of thumb, the effect size is very large.
The 95% CI around Cohen's d includes zero, which means that there is no practical difference between the performance of male and female participants on the grammar test.

🦉 Hover over the owl for a first hint.

🐭 Click on the mouse for a second hint.

TipYour turn!

Generate a facetted scatter plot (similar to Figure 11.4) visualising the correlation between age (Age) and English receptive vocabulary test scores (Vocab) for L1 and L2 participants in two separate panels. Add a grey band visualising 95% confidence intervals around the regression lines.

Q11.10 Looking at your plot only, which of these statements can you confidently say is/are true?

There is more variability around the correlation estimate of the L2 than the L1 group.
For both the L1 and L2 groups, the correlations are positive.
Even though one correlation is positive and the other negative, in absolute values, the correlations are equally strong.
There is more variability around the correlation estimate of the L1 than the L2 group.
In the L1 group, the correlation is positive, whereas in the L2 group it is negative.

🐭 Click on the mouse for a hint.

Dabrowska.data |> 
   ggplot(mapping = aes(x = Age, 
                        y = Vocab)) +
     geom_point() +
     facet_wrap(~ Group) +
     geom_smooth(method = "lm", 
                 se = TRUE) +
  theme_bw()

Now use the cor.test() function to find out how strong the correlations are in both the L1 and the L2 groups. Use the default α-level of 0.05.

Q11.11 Based on the outputs of the cor.test() function, which of these statements can you confidently say is/are true? Note that, in this question and the next one, values are reported to two decimal places.

In absolute values, the correlation between Age and Vocab scores is stronger in the L1 than in the L2 sample.
Based on the data available to us and an α-level of 0.05, we cannot reject the null hypothesis of no correlation between Age and Vocab scores for the L2 population.
The correlation of Age and Vocab scores in the L1 sample is 0.37.
The correlation of Age and Vocab scores in the L2 population is -0.20.
Assuming that the null hypothesis of no correlation between Age and Vocab scores in the L1 population is true, there is only a 0.03% chance of observing a correlation of 0.37 or larger in a sample of this size.

🐭 Click on the mouse for a hint.

cor.test(formula = ~ Age + Vocab,
         data = L1.data)

cor.test(formula = ~ Age + Vocab,
         data = L2.data)

Consider the code and its output below. Figure 1 visualises the correlation between L2 speakers’ English receptive vocabulary test scores (Vocab) and their age of arrival in the UK (Arrival).

L2.data |> 
   ggplot(mapping = aes(x = Arrival, 
                        y = Vocab)) +
     geom_point() +
     geom_smooth(method = "lm", 
                 se = TRUE) +
  theme_bw()

Figure 1: Correlation between L2 speakers’ English vocabulary test scores and their age of arrival

Q11.12 Run a correlation test on the correlation visualised in Figure 1. How strong is the correlation (to two decimal places)?

🐭 Click on the mouse for a hint.

cor.test(formula = ~ Vocab + Arrival,
         data = L2.data)

Q11.13 How likely are we to observe such a strong correlation or an even stronger one in a sample of this size, if there is actually no correlation in the full L2 population?

about 1%
about 0.01%
about 10%
It's impossible to tell because the correlation is not statistically significant.

🐭 Click on the mouse for a hint.

CautionOptional (fun) task 🦖

Inspired by the Anscombe Quartet, Matejka & Fitzmaurice (2017) created a set of 12 pairs of variables that have the same descriptive statistics as the data that produce a scatter plot representing a tyrannosaurus (see Figure 2).

Figure 2: Scatter plot depicting a tyrannosaurus (originally created by Alberto Cairo)

1. Install and load the datasauRus package to access the R data object datasaurus_dozen:

install.packages("datasauRus")
library(datasauRus)

datasaurus_dozen

2. Run the following code to compare the descriptive statistics of all three datasets within the datasaurus_dozen. The fourth set, dino, is the one visualised in Figure 2. Note that there is a very small, negative correlation between all pairs of variables of -0.0656.

datasaurus_dozen |> 
  group_by(dataset) |> 
  summarise(
    mean_x = mean(x), 
    mean_y = mean(y), 
    std_dev_x = sd(x), 
    std_dev_y = sd(y), 
    corr_x_y = cor(x, y))

# A tibble: 13 × 6
   dataset    mean_x mean_y std_dev_x std_dev_y corr_x_y
   <chr>       <dbl>  <dbl>     <dbl>     <dbl>    <dbl>
 1 away         54.3   47.8      16.8      26.9  -0.0641
 2 bullseye     54.3   47.8      16.8      26.9  -0.0686
 3 circle       54.3   47.8      16.8      26.9  -0.0683
 4 dino         54.3   47.8      16.8      26.9  -0.0645
 5 dots         54.3   47.8      16.8      26.9  -0.0603
 6 h_lines      54.3   47.8      16.8      26.9  -0.0617
 7 high_lines   54.3   47.8      16.8      26.9  -0.0685
 8 slant_down   54.3   47.8      16.8      26.9  -0.0690
 9 slant_up     54.3   47.8      16.8      26.9  -0.0686
10 star         54.3   47.8      16.8      26.9  -0.0630
11 v_lines      54.3   47.8      16.8      26.9  -0.0694
12 wide_lines   54.3   47.8      16.8      26.9  -0.0666
13 x_shape      54.3   47.8      16.8      26.9  -0.0656

3. Run the following code to visualise the relationships between the other 12 pairs of variables.

ggplot(datasaurus_dozen, 
       aes(x = x, y = y, colour = dataset)) + 
  geom_point() + 
  facet_wrap(~ dataset, ncol = 3) +
  theme_bw() + 
  theme(legend.position = "none") + 
  labs(x = NULL,
       y = NULL)

4. Add a geom_ layer (see Geometries) to the following {ggplot2} code to add a blue linear regression line in all 13 panels.

ggplot(datasaurus_dozen, 
       aes(x = x, y = y, colour = dataset)) + 
  geom_point() + 
  facet_wrap(~ dataset, ncol = 3) +
  geom_smooth(method = "lm",
              se = FALSE,
              colour = "blue") +
  theme_bw() + 
  theme(legend.position = "none") + 
  labs(x = NULL,
       y = NULL)  

Clearly, if we had only used correlation statistics to describe the relationships between these 13 pairs of variables, we would have missed some literally dinosaur-sized patterns! 🤯

TipYour turn!

The dataset from Dąbrowska (2019) includes the results of numerous additional tests that we have not yet examined.

In this task, we imagine that a friend of yours has decided to test whether, on average, male and female speakers performed equally well on four additional English grammar tests. The four variables that he selected correspond to participants’ scores on English tests assessing participants’ mastery of:

• the active voice (Active)
• the passive voice (Passive)
• post-modified subjects (Postmod)
• subject relatives (SubRel)

Your friend conducted the following four t-tests to find out whether he could reject the the null hypothesis of no difference in the average performance of male and female English speakers in these four grammar tests. He chose 0.05 as his significance threshold and, on the basis of these four tests, concluded that he could reject the null hypothesis in two out of four tests: the one concerning the active voice and the one about subject relatives.

t.test(formula = Active ~ Gender, 
       data = Dabrowska.data)

t.test(formula = Passive ~ Gender, 
       data = Dabrowska.data)

t.test(formula = Postmod ~ Gender, 
       data = Dabrowska.data)

t.test(formula = SubRel ~ Gender, 
       data = Dabrowska.data)

Q11.14 Your friend tells you that, based on the results of the two significant t-tests, he has decided to write a term paper focussing on how men have a better understanding of the active voice and subject relative constructions than women. You warn him that this approach reminds you of a questionable research practice (QRP) that you read about in the FORRT glossary. Which one?

Salami slicing
Optional stopping
HARKing
Creative use of outliers

🐭 Click on the mouse for a hint.

Q11.15 You explain to your friend that, if he wants to conduct four separate tests on the same dataset, he needs to correct the p-values for multiple testing. Which reason(s) can you use to explain this to your friend?

Even though you chose an α-level of 0.05 for each test, your overall risk of erroneously rejecting the null hypothesis in at least one of these four tests is actually much higher than 5%.
Running multiple tests on the same dataset makes each individual test weaker and less reliable.
The sample size gets split up when you conduct multiple tests, which results in less accurate results.
Multiple tests on the same dataset make it harder to detect real differences when they actually exist.
Running multiple tests on the same dataset increases your chances of obtaining a false positive result.

🐭 Click on the mouse for a hint.

Q11.16 How high is your friend’s risk of erroneously rejecting the null hypothesis in at least one of his four tests?

5%
7%
19%
20%
almost 100%

🐭 Click on the mouse for a hint.

1 - (1 - 0.05)^4 

Q11.17 Use Holm’s correction to correct the four p-values that your friend obtained to account for the fact that he conducted four tests on the same dataset. After correction, which null hypotheses can be rejected at α-level = 0.05?

All four of them.
None of them.
Only the one concerning the active voice.
Only the one concerning the passive voice.
The one concerning the active voice and the one concerning subject relatives.

🐭 Click on the mouse for a hint.

# Running the four t-tests and saving the output as four R objects to the local environment:
active.ttest <- t.test(formula = Active ~ Gender, 
       data = Dabrowska.data)

passive.ttest <- t.test(formula = Passive ~ Gender, 
       data = Dabrowska.data)

postmod.ttest <- t.test(formula = Postmod ~ Gender, 
       data = Dabrowska.data)

subjrel.ttest <- t.test(formula = SubRel ~ Gender, 
       data = Dabrowska.data)

# Extracting just the p-values from the test outputs
p.values <- c(active.ttest$p.value, passive.ttest$p.value, postmod.ttest$p.value, subjrel.ttest$p.value)

# The original p-values:
p.values

# Applying Holm's correction to the p-values
p_values.adjusted <- p.adjust(p.values, method = "holm")

# The adjusted p-values in scientific notation:
p_values.adjusted

# Now only the first p-value is below 0.05:
p_values.adjusted < 0.05

Check your progress 🌟

Well done! You have successfully completed this chapter introducing the complex topic of inferential statistics. You have answered 0 out of 17 questions correctly.

Dąbrowska, Ewa. 2019. Experience, aptitude, and individual differences in linguistic attainment: A comparison of native and nonnative speakers. Language Learning 69(S1). 72–100. https://doi.org/10.1111/lang.12323.

Matejka, Justin & George Fitzmaurice. 2017. Same stats, different graphs: Generating datasets with varied appearance and identical statistics through simulated annealing. In Proceedings of the 2017 CHI conference on human factors in computing systems (CHI ’17), 1290–1294. New York, NY, USA. https://doi.org/10.1145/3025453.3025912.