Course Outline
-
segmentGetting Started (Don't Skip This Part)
-
segmentStatistics and Data Science: A Modeling Approach
-
segmentPART I: EXPLORING VARIATION
-
segmentChapter 1 - Welcome to Statistics: A Modeling Approach
-
segmentChapter 2 - Understanding Data
-
segmentChapter 3 - Examining Distributions
-
segmentChapter 4 - Explaining Variation
-
segmentPART II: MODELING VARIATION
-
segmentChapter 5 - A Simple Model
-
segmentChapter 6 - Quantifying Error
-
segmentChapter 7 - Adding an Explanatory Variable to the Model
-
segmentChapter 8 - Digging Deeper into Group Models
-
segmentChapter 9 - Models with a Quantitative Explanatory Variable
-
segmentPART III: EVALUATING MODELS
-
segmentChapter 10 - The Logic of Inference
-
segmentChapter 11 - Model Comparison with F
-
segmentChapter 12 - Parameter Estimation and Confidence Intervals
-
12.9 Confidence Intervals and Model Comparison
-
segmentChapter 13 - What You Have Learned
-
segmentFinishing Up (Don't Skip This Part!)
-
segmentResources
list High School / Advanced Statistics and Data Science I (ABC)
12.9 Confidence Intervals and Model Comparison
We have now used the sampling distribution of
The confidence interval provides us with a range of models of the DGP (i.e., a range of possible
We would reject any values of
Using A Confidence Interval to Evaluate the Empty Model |
Using a Hypothesis Test to Evaluate the Empty Model |
---|---|
In the right panel of the figure above, the model comparison (or hypothesis testing) approach considers just one particular model of the DGP, not a range of models. In this model, in which shuffle()
to mimic such a DGP, and built a sampling distribution centered at 0. We can see in the picture above that if such a DGP were true, our sample
We then used the sampling distribution as a probability distribution to calculate the probability of getting a sample
These two approaches – null hypothesis testing and confidence intervals – both provide ways of evaluating the empty model, and both lead us to the same conclusion in the tipping study: the empty model, where
If the 95% confidence interval does not include 0, then we would reject the empty model because we are not confident that
As another example, let’s consider a second tipping study done by another team of researchers. They got very similar results but this time, their
Original Study ( |
Second Study ( |
---|---|
We don’t really believe that the DGP has changed, so we wouldn’t say the
Let’s take a look at how the confidence interval might be different across these two studies.
Original Study ( |
Second Study ( |
---|---|
In the left panel of the figure the confidence interval (marked by the two red boxes) is centered around an assumed
In the right panel of the figure, we see what happened in the second study where the observed
It is also worth noting that we get a lot more information from the confidence interval than we do from the p-value. For example, in the original tipping study (where