How the Vaccine Cured Polling
Predicting 2022, Part 4: In our latest battleground polling our team noticed boosted individuals are participating in polls at much higher rates than their counterparts which resulted in a 3% to 5.5% error in polls before our team started correcting for it. This discovery is a symptom of an amplified selection bias that we believe is has been the hidden issue with polling for the last 10 years.
We know... you probably came to this page looking for the polling toplines (and an explanation of how we have Dr. Oz up 4.5 points in our polling from the week of October 8th). Click here to scroll directly to the results of our battleground polls. These polls are the result of some, but definitely not all, of the methodology changes needed to predict 2022 accurately. We still haven't fully taken into consideration the Shifts in Party Affiliation that we we write about in Part 1 of this series nor have we really explored different turnout scenarios that we discuss in Part 3 of this series. For now, we want to stay focused on a key symptom of selection bias from our latest round of polling and how adding a question on vaccine status helped treat the symptom in a way that changed candidate ballot measures by up to 5.5%.
INTRODUCTION
As we write this article in mid-October, political junkies and astute election monitors may be thinking: “Public opinion is shifting in favor of Republicans at a pretty fast rate. Just 4 weeks ago, Senate candidate Whitmer was up by 15 points and now it’s only 5 points. This looks like 2020 all over again!” You might even be wondering what’s driving the shift in public opinion. The answer is: Nothing. Public opinion hasn’t shifted anywhere close to that amount.
To put it simply, for most of an election cycle (about 101 out of 104 weeks) polling is not giving us an accurate depiction of the actual population of voters because it does not include a representative sample of voters- it’s just hard to see the representation problems at a surface level. This lack of representation gets a little better in the final four weeks leading up to the election when energy is high and all groups of voters start feeling inspired to answer polls at higher rates. The fact that this hidden selection bias issue starts to self-correct is nice for the polling industry because we get to write articles after the election entitled, “Actually, polling wasn’t all that bad this cycle.” Typically, these articles conveniently ignore polls conducted in August and September that were 15 points wrong when compared to election day results.
So what do we mean by a hidden selection bias? We go into great detail about what is driving this selection in Part 2 of this Series: A new era of selection bias (see image below), but the short version is this: Poll results are being woven into political narratives constantly by the media. These poll-buttressed political narratives are ubiquitous, exposing hundreds of millions of possible voters via social media channels that did not exist in decades past (or before technology made polling easier, faster, and cheaper to execute). If you agree with these poll-buttressed political narratives, you’re likely to feel validated… you feel hope. For people with a certain psychological profile, this feeling of validation serves as enough of an incentive to take polls and answer in a way that supports those narratives. We call it the Flywheel of Expression theory.
The challenge for pollsters, however, is that we can’t easily profile or set quotas on people based on how they respond to the media narratives they see or hear, so the best we can do is identify choices people make that may have a strong correlation with their psychology or personality (choices on which we have good data). For example, did an individual choose to go to college, work for the public or private sector, get married, be a parent, go to church, etc? What we are looking for is any group that is answering at much higher rates than another. When we asked these questions in our most recent battleground polling, we saw a lot of symptoms that need to be addressed, but the loudest symptom by far was how different the set of respondents’ vaccination status was from the general population. Before correcting for the non-response bias, the percentage of people who answered they were boosted was over 60% in every battleground survey. It is very important to keep in mind that, though we used probability sampling methods, we only used IVR and Text-to-Web data collection methods and can’t speak to other methods of probability sampling like live agent polling. The next three tables below demonstrate the size of this selection bias problem in our most recent round of battleground polling.
Vaccination Status of battleground respondents before corrections
| MI | AZ | GA | NC | PA | |
| Vaccinated, boosted | 61% | 62% | 60% | 65% | 65% |
| Vaccinated, not boosted | 10% | 11% | 14% | 14% | 13% |
| Unvaccinated | 29% | 27% | 26% | 29% | 23% |
Let’s look at the actual percentages of people who are unvaccinated, vaccinated, and boosted as reported by the CDC (data varies widely on this depending on the source):
Vaccinated Status of all 18+ in the United States
| MI | AZ | GA | NC* | PA | |
| Vaccinated, boosted | 42% | 35% | 29% | 38% | 38% |
| Vaccinated, not boosted | 29% | 37% | 37% | 36% | 42% |
| Unvaccinated | 29% | 27% | 33% | 25% | 19% |
To get an idea of how unrepresentative our polling is (and we would wager a lot of pollster’s probability polling is) compared to the actual population, we have created the following table that shows the spread between the polling sample and the actual 18+ population of the United States.
Difference between the survey sample population and CDC reported population
| MI | AZ | GA | NC | PA | |
| Vaccinated, boosted | +19% | +27% | +31% | +27% | +27% |
| Vaccinated, not boosted | -18% | -26% | -23% | -22% | -29% |
| Unvaccinated | 0% | 0% | -7% | -4% | +4% |
It’s humbling that we didn’t discover this earlier because a person’s vaccination status, especially within his or her respective age group, may have a stronger correlation to how he or she sees the world than race, gender, education, urbanicity, etc… Whether you agree about the relationship between vaccine status and psychological profile, our Flywheel of Expression theory concerning increased participation rates hinges on the idea that certain groups respond differently to information they see online. To us, it’s hard to argue individuals who made the choice to remain unvaccinated vs get boosted are partially a result of how they responded to information online.
KEY FINDINGS FROM TREATING THE SYMPTOMS
In mid-October, we went into the field not knowing that boosted individuals were answering polls at much higher rates than vaxxed-not-boosted or not-vaxxed. When we recognized this imbalance, we made adjustments and then relied on weighting to correct the rest. Although we do not know the exact percent of 2022 voters that are vaccinated, we are confident that every polling firm using probability sampling techniques has been over-sampling boosted voters for the last 12 months. So at the end of the day, compared to the other polls out there I would give this research top marks, but compared to our next round, in which we will design the polling methodology with this knowledge, I would give it a C+. Think of this article more as evidence proving that there is a selection bias that is impacting polling rather than a perfect example of how to treat that bias.
3 KEY FINDINGS
- How does this selection bias impact the results of polling? The simple way to explain it is that you may have the right number of Independents but the wrong diversity of Independents. The table below shows Tudor Dixon support at 23% with boosted Independents and 52% with vax-not-boosted Independents, a 29-point difference.
Candidate Preference by Vaccine Status with Michigan Independents
| Values | Not vaccinated | Vaxxed, not boosted | Vaxxed, boosted | Total |
| Gretchen Whitmer | 10% | 36% | 72% | 49% |
| Tudor Dixon | 80% | 52% | 23% | 43% |
| Someone else | 4% | 6% | 2% | 3% |
| Undecided | 6% | 6% | 3% | 4% |
| 100% | 100% | 100% | 100% |
- How big was the impact? In each state, correcting for this variable changed the results anywhere from 2.5% – 5.5%. I would argue that setting quotas on this variable would have had even a larger impact in August and September. However, in our next round of polling in late October we expect the impact to be lower because election day serves as a new incentive to take polls and it serves as somewhat of an equalizer that waters down selection bias.
- What else did we find? We wrote in Part 1 of this series about a possible party affiliation shift and the difficulties measuring that shift because of selection bias issues and other modern-era issues with probability polling. We did a recontact study where we resurveyed the exact same respondents from our 2020 battleground polling and measured how many of the respondents answered that they identify with a different party now in 2022. We ended up seeing approximately a 5-point shift in the R/D spread in favor of the Republican party. When we corrected for the over-representation of boosted individuals, we saw another shift in affiliation in favor of the Republican party.
Here are the results of our polls with public viewing links courtesy of Wick, our parent SaaS company that helps our agency shift the focus from day-to-day tasks to breakthrough insights like the ones you are currently reading about:
North Carolina
Oct 8-14
1,009
Text to Web & IVR
Governor
NO RACE
US Senate
BUDD +4.5
Pennsylvania
Oct 8-14
1,013
Text to Web & IVR
Governor
SHAPIRO +2
US Senate
OZ +4.5
Get The Whole Truth
Subscribe to The Whole Truth, our email newsletter that makes sure you don’t miss a single insight and gets you special access to deeper dives and exclusive content.
You may opt out any time. Terms and Conditions and Privacy Policy
Let’s get in touch.
Are you a media representative wishing to interview our CEO and Chief Pollster, David Burrell? Let’s get something scheduled. Email: stephanie@wick.io
Schedule now