Course Outline
-
segmentGetting Started (Don't Skip This Part)
-
segmentStatistics and Data Science: A Modeling Approach
-
segmentPART I: EXPLORING VARIATION
-
segmentChapter 1 - Welcome to Statistics: A Modeling Approach
-
segmentChapter 2 - Understanding Data
-
segmentChapter 3 - Examining Distributions
-
segmentChapter 4 - Explaining Variation
-
segmentPART II: MODELING VARIATION
-
segmentChapter 5 - A Simple Model
-
segmentChapter 6 - Quantifying Error
-
segmentChapter 7 - Adding an Explanatory Variable to the Model
-
7.9 Modeling the DGP
-
segmentChapter 8 - Models with a Quantitative Explanatory Variable
-
segmentPART III: EVALUATING MODELS
-
segmentChapter 9 - The Logic of Inference
-
segmentChapter 10 - Model Comparison with F
-
segmentChapter 11 - Parameter Estimation and Confidence Intervals
-
segmentChapter 12 - What You Have Learned
-
segmentFinishing Up (Don't Skip This Part!)
-
segmentResources
list High School / Advanced Statistics and Data Science I (ABC)
7.9 Modeling the DGP
We now know how to fit a two-group model to data and to see how much error we can reduce with the two-group model compared to the empty model. But let’s pause for a moment to remember: Our main interest is not in a particular data set, but in the Data Generating Process that generated the data. Even if we have a model that fits our data well, it may not be a good model of the DGP. In fact, it might not even be better than the empty model. Let’s spend some time thinking about this possibility.
Revisiting the Tipping Study
Back in Chapter 4 we learned about an experiment that investigated whether drawing smiley faces on the back of a check would cause restaurant servers to receive higher tips (Rind, B., & Bordia, P. (1996). Effect on restaurant tipping of male and female servers drawing a happy, smiling face on the backs of customers’ checks. Journal of Applied Social Psychology, 26(3), 218-225).
The study was a randomized experiment. A female server in a particular restaurant was asked to either draw a smiley face or not for each table she served following a predetermined random sequence. The outcome variable was the amount of tip left by each table (Tip
). Distributions of tip amounts in the two groups (n=22 tables per condition) are shown below.
Use the code window below to fit this two-group model to the data:
In this model,
supernova(Condition_model)
Analysis of Variance Table (Type III SS)
Model: Tip ~ Condition
SS df MS F PRE p
----- --------------- | -------- -- ------- ----- ------ -----
Model (error reduced) | 402.023 1 402.023 3.305 0.0729 .0762
Error (from model) | 5108.955 42 121.642
----- --------------- | -------- -- ------- ----- ------ -----
Total (empty model) | 5510.977 43 128.162
Keep in mind, though, that both
$6 is our best estimate of the true benefit, but if we had taken a different sample we would have gotten a different estimate. How much uncertainty is there in our estimate? What might be the true DGP that generated the $6 benefit in our sample?
Although we can calculate the value of
We can, however, imagine different possible values of
If there were no effect of smiley face in the DGP then
The question is: how much would they vary? Is it possible that a
We used the shuffle()
function in Chapter 4 to simulate random samples from a DGP with a shuffle()
again, but this time in a more sophisticated way now that we’ve learned how to create a model and calculate
Hypothetical Thinking
When we use the shuffle()
function to simulate the empty model, we are getting our brain to do something it doesn’t naturally do: think hypothetically. We are asking: if the empty model is true, what might the distribution of data look like? What might it look like for one random sample? Or many random samples? If you practice thinking like this, you will find that it can help you draw conclusions about the DGP. But it will take practice, and we will do a lot more of it as we get further through the course.
Let’s look again at the data from the tipping experiment (represented in the jitter plot below). We used black dots to plot the predictions of the two-group model (i.e., the two group means). We also colored one of the tables (Table 1 in the data frame), blue.
We know that in the actual experiment Table 1 did not get a drawn smiley face and they tipped $39. Let’s think hypothetically for a moment about this: If Table 1 were in the smiley face condition, what would they have tipped? (We know that Table 1 wasn’t in the smiley face condition, but remember, we are thinking hypothetically.) It turns out that the answer to this question depends on which model we believe to be true.
If we believe that smiley face does have an impact on tip amount (TIP = SMILEY FACE + OTHER STUFF), then we would predict, based on our estimate of
What if we believe that smiley face does not have an impact on tipping in the DGP? This would mean that you believe the empty model is true.
If the empty model is true,
Remember: if we assume that the empty model is true (really true in the DGP) then whether or not someone got a smiley face on their check is going to make no difference to how much they tip. Someone who tipped $39 will still tip $39. This is what makes the shuffle()
function so powerful, as we will see in the next section.
Simulating the Empty Model ( )
The shuffle()
command simulates the case where the empty model is true, i.e.,
When tables were initially randomly assigned, in the actual experiment, to either get a smiley face or not, the result was only one of the possible random assignments that could have occured. The function shuffle()
let’s us do it again (and again), and see what other randomizations would have been possible.
In the figure below, we randomly shuffled all the tables. This time Table 1 happened to be assigned to the smiley face condition. Because we are simulating a random DGP in which smiley faces do not affect tipping, Table 1 still tips the same amount ($39).
When we shuffle, we are simulating a world in which the only difference between the two groups is due to randomness. Although it’s possible, it won’t usually happen that all the big tipping tables would get randomly shuffled into one group and the smaller tippers into the other group. What’s more likely is that some of the big tipping tables would end up in both groups. For this reason, we would expect the group difference after the shuffle to be close to 0. But because the shuffle is random, it won’t necessarily be exactly 0.
In the plot above, we will still get group differences (the black lines do not have exactly the same value on the y-axis; the two groups don’t have exactly the same means). But the difference is much smaller than in the actual experiment – much closer to 0.
Let’s shuffle the data again, but this time get right to an estimate of what b1()
function, returns the difference in group means (
b1(shuffle(Tip) ~ Condition, data = TipExperiment)
Go ahead and add shuffle()
to the code in the window below.
Adding the do()
function in front of the code above lets us repeat this random process multiple times, generating a new
Try modifying the code in the window below to produce 10 shuffles of the tips across conditions, and 10 estimates of
b1
1 -3.7727273
2 -2.6818182
3 -1.4090909
4 -2.6818182
5 -4.6818182
6 2.0454545
7 -0.1363636
8 -1.1363636
9 0.5909091
10 4.6818182
The ten
If we continue to generate random
Bear in mind: Each of these shuffle()
function mimics a DGP where
Using Simulated s to Understand Our Sample Better
Now let’s see how we can use the
By simulating the range of
To see how this works, let’s look again at the 10
b1
1 -3.7727273
2 -3.2272727
3 -1.6818182
4 -1.6818182
5 -1.5000000
6 -0.5000000
7 0.1363636
8 2.7727273
9 3.6818182
10 6.9545455
One of the 10 simulated
Based on this, would we want to rule out the empty model as being the true model of the DGP? That’s a hard call, and one we will return to in Chapter 9. For now, just note that seeing the observed
Connecting to
We have seen that when we use shuffle
to randomly generate
This same logic can be applied to other measures of effect size, for example, PRE. If we want, we can shuffle the data again, but instead of calculating
We can apply the same logic we used above for interpreting
bunch_of_PREs <- do(10) * PRE(shuffle(Tip) ~ Condition, data = TipExperiment)
arrange(bunch_of_PREs, PRE)
PRE
1 0.005645757
2 0.005645757
3 0.008351101
4 0.014355646
5 0.015345406
6 0.016368158
7 0.016368158
8 0.040419328
9 0.047215681
10 0.078949444
Similar to when we simulated
Does this mean we can reject the empty model? Maybe. We will address this question in much more detail in Chapter 9. For now, just keep in mind that in order to think about what our data say about what’s really true about the DGP, we will need to consider (and eventually quantify) sampling variation.