Course Outline

segmentGetting Started (Don't Skip This Part)

segmentStatistics and Data Science: A Modeling Approach

segmentPART I: EXPLORING VARIATION

segmentChapter 1  Welcome to Statistics: A Modeling Approach

segmentChapter 2  Understanding Data

segmentChapter 3  Examining Distributions

segmentChapter 4  Explaining Variation

segmentPART II: MODELING VARIATION

segmentChapter 5  A Simple Model

segmentChapter 6  Quantifying Error

segmentChapter 7  Adding an Explanatory Variable to the Model

segmentChapter 8  Digging Deeper into Group Models

segmentChapter 9  Models with a Quantitative Explanatory Variable

segmentPART III: EVALUATING MODELS

segmentChapter 10  The Logic of Inference

segmentChapter 11  Model Comparison with F

11.3 Sampling Distribution of F

segmentChapter 12  Parameter Estimation and Confidence Intervals

segmentChapter 13  What You Have Learned

segmentFinishing Up (Don't Skip This Part!)

segmentResources
list High School / Advanced Statistics and Data Science I (ABC)
11.3 Sampling Distribution of F
So far we have constructed and examined the sampling distribution of PRE to examine the variation in PREs that could be generated by the empty model in the context of the tipping experiment. We can use the same method to construct the sampling distribution of F under the empty model.
In fact, the sampling distribution of F is one of the most common tools used to compare a complex model to the empty model (i.e., to do a Null Hypothesis Significance Test, or NHST). It is so popular, in fact, that it has its own name: the Ftest. For this reason, we will spend a little time investigating the sampling distribution of F.
Review of the F Ratio
First let’s review how to calculate the F ratio for a model (which is a sample statistic). It appears in the supernova()
table. But just as we have an R function to directly calculate the PRE for a model, we also have one to calculate F: f()
. The following line of code calculates the sample F ratio that results from fitting the Condition
model to the tipping study data.
f(Tip ~ Condition, data = TipExperiment)
3.3049725526482
The F ratio, like PRE, is a sample statistic that tells us something about how well our model fits the data. In general, the higher the PRE and the higher the F, the better job our model does in explaining variation in the outcome variable. But whereas PRE keeps going up and up as more predictors are added to the model, F adjusts based on the number of degrees of freedom used to explain more variation.
As just a reminder, we have learned two formulas for computing the F ratio:
\[F = \frac{\text{MS}_\text{Model}}{\text{MS}_\text{Error}} = \frac{\text{PRE}/\text{df}_\text{model}}{(1\text{PRE})/\text{df}_\text{error}}\]
The first formula (a ratio of two variances, or \(\text{MS}\)s) is the one most commonly used to compute F. But the second one (with \(\text{PRE}\)s) helps us understand the relationship between PRE and F.
The numerator (\(\text{PRE}/\text{df}_\text{model}\)) is the PRE (proportion of variation explained) per degree of freedom used in the model; the denominator (\((1\text{PRE})/\text{df}_\text{error}\)) is the proportion of variation left unexplained per remaining degree of freedom. The F of 3.30 could be thought of, therefore, as the number of times more powerful the parameter in our model is as a predictor compared to any other parameter that could have been added to the model.
Using shuffle()
to Construct a Sampling Distribution of F
Armed with the f()
function, we can use the same approach we used for PRE to construct the sampling distribution of F under the empty model. We will use shuffle()
to simulate a DGP in which the only difference between the groups is due to randomization, and then use f()
to find the F for the shuffled data. We will then repeat this process many times to create the sampling distribution.
Use the code block below to save 1000 randomly generated F ratios in a data frame called sdof
(an acronym for the sampling distribution of f). We’ve already provided some code that will display this distribution in a histogram.
require(coursekata)
# save 1000 randomly generated F ratios in a data frame called sdof
sdof <
gf_histogram(~ f, data = sdof, fill = "darkgoldenrod1")
# save 1000 randomly generated F ratios in a data frame called sdof
sdof < do(1000) * f(shuffle(Tip) ~ Condition, data = TipExperiment)
gf_histogram(~ f, data = sdof, fill = "darkgoldenrod1")
ex() %>%
check_object("sdof") %>%
check_equal()
Below on the right we have plotted a histogram of the randomized sampling distribution of F. On the left we have plotted the sampling distribution of PRE for purposes of comparison.
You may be happy to find that the shapes of the sampling distributions of PRE and F are very similar. Neither of these sample statistics can be negative. And, the result of either a large positive or large negative effect of smiley face on Tip
would both yield extreme Fs in the upper tail of the distribution.
Both of these sampling distributions are based on the assumption that the empty model is true in the DGP. We have previously developed the idea that if there is no effect of smiley face in the DGP (i.e., the empty model is true) then PRE in the DGP would be equal to 0. This means that knowing which condition a table is in explains literally 0% of the variation in Tip
, which is the same as saying \(\beta_1=0\).
But what would the expected value of F be if the empty model were true? F is a more difficult concept to understand, and so we won’t develop this fully here. But if the empty model were true, meaning that PRE were literally 0, then the expected value of F would be 1. The variance estimated using the model predictions would be roughly equal to the variance estimated based on the error within groups.
To confirm that this is true, you can use the code window below to calculate the mean of f
for our sampling distribution of F. Because our sampling distribution, which we created using shuffle()
, assumes that the empty model is true, the average of all the Fs we generated should be roughly equal to 1.
require(coursekata)
# we have created the sdof for you
sdof < do(1000) * f(shuffle(Tip) ~ Condition, data = TipExperiment)
# calculate the mean of f
favstats( )
# we have created the sdof for you
sdof < do(1000) * f(shuffle(Tip) ~ Condition, data = TipExperiment)
# calculate the mean of f
favstats(~ f, data = sdof)
ex() %>%
check_function("favstats") %>%
check_result() %>%
check_equal()
Because the sampling distribution of F is more common and highly similar to the sampling distribution of PRE, we will focus on using the distribution made of Fs. However, just know that everything we say in the next few sections will also apply to PREs.