11.3 Sampling Distribution of F

So far we have constructed and examined the sampling distribution of PRE to examine the variation in PREs that could be generated by the empty model in the context of the tipping experiment. We can use the same method to construct the sampling distribution of F under the empty model.

In fact, the sampling distribution of F is one of the most common tools used to compare a complex model to the empty model (i.e., to do a Null Hypothesis Significance Test, or NHST). It is so popular, in fact, that it has its own name: the F-test. For this reason, we will spend a little time investigating the sampling distribution of F.

Review of the F Ratio

First let’s review how to calculate the F ratio for a model (which is a sample statistic). It appears in the supernova() table. But just as we have an R function to directly calculate the PRE for a model, we also have one to calculate F: f(). The following line of code calculates the sample F ratio that results from fitting the Condition model to the tipping study data.

f(Tip ~ Condition, data = TipExperiment)

3.3049725526482

The F ratio, like PRE, is a sample statistic that tells us something about how well our model fits the data. In general, the higher the PRE and the higher the F, the better job our model does in explaining variation in the outcome variable. But whereas PRE keeps going up and up as more predictors are added to the model, F adjusts based on the number of degrees of freedom used to explain more variation.

As just a reminder, we have learned two formulas for computing the F ratio:

\[F = \frac{\text{MS}_\text{Model}}{\text{MS}_\text{Error}} = \frac{\text{PRE}/\text{df}_\text{model}}{(1-\text{PRE})/\text{df}_\text{error}}\]

The first formula (a ratio of two variances, or \(\text{MS}\)s) is the one most commonly used to compute F. But the second one (with \(\text{PRE}\)s) helps us understand the relationship between PRE and F.

The numerator (\(\text{PRE}/\text{df}_\text{model}\)) is the PRE (proportion of variation explained) per degree of freedom used in the model; the denominator (\((1-\text{PRE})/\text{df}_\text{error}\)) is the proportion of variation left unexplained per remaining degree of freedom. The F of 3.30 could be thought of, therefore, as the number of times more powerful the parameter in our model is as a predictor compared to any other parameter that could have been added to the model.

Using shuffle() to Construct a Sampling Distribution of F

Armed with the f() function, we can use the same approach we used for PRE to construct the sampling distribution of F under the empty model. We will use shuffle() to simulate a DGP in which the only difference between the groups is due to randomization, and then use f() to find the F for the shuffled data. We will then repeat this process many times to create the sampling distribution.

Use the code block below to save 1000 randomly generated F ratios in a data frame called sdof (an acronym for the sampling distribution of f). We’ve already provided some code that will display this distribution in a histogram.

Below on the right we have plotted a histogram of the randomized sampling distribution of F. On the left we have plotted the sampling distribution of PRE for purposes of comparison.

You may be happy to find that the shapes of the sampling distributions of PRE and F are very similar. Neither of these sample statistics can be negative. And, the result of either a large positive or large negative effect of smiley face on Tip would both yield extreme Fs in the upper tail of the distribution.

Both of these sampling distributions are based on the assumption that the empty model is true in the DGP. We have previously developed the idea that if there is no effect of smiley face in the DGP (i.e., the empty model is true) then PRE in the DGP would be equal to 0. This means that knowing which condition a table is in explains literally 0% of the variation in Tip, which is the same as saying \(\beta_1=0\).

But what would the expected value of F be if the empty model were true? F is a more difficult concept to understand, and so we won’t develop this fully here. But if the empty model were true, meaning that PRE were literally 0, then the expected value of F would be 1. The variance estimated using the model predictions would be roughly equal to the variance estimated based on the error within groups.

To confirm that this is true, you can use the code window below to calculate the mean of f for our sampling distribution of F. Because our sampling distribution, which we created using shuffle(), assumes that the empty model is true, the average of all the Fs we generated should be roughly equal to 1.

Because the sampling distribution of F is more common and highly similar to the sampling distribution of PRE, we will focus on using the distribution made of Fs. However, just know that everything we say in the next few sections will also apply to PREs.