A representative sample
Pollsters forced to pursue new methods
October 11, 2012
as we approach election day, one can scarcely turn on the news or pick up a newspaper without reading about polls. Polls in the run-up to election day tend to focus on which politician is up and which one is down, what their “favorability rating” is, and where their support is coming from. The prevalence of polls this year is similar to what it was in other recent presidential election years, making it appear that polling today is a stable industry.
| The abandonment of landlines is giving pollsters |
In truth, however, the polling industry is going through rapid change, and high quality polling is a more difficult proposition today than ever before. The challenges pollsters face are immense, including the shift to cell phones, a public less willing to spend time on the phone answering questions, shrinking media budgets leaving less resources for high quality polling, and less rigorous scrutiny applied to poll results. The challenges are significant enough that many within the polling industry are looking for new ways to reach voters.
Domestic polling companies have historically relied on dialing a carefully constructed sample of landline phones within the target geographic area to achieve a representative sample. That was relatively easy 15 years ago, when neighbors had the same area code and the same three-digit prefix in their telephone number. Pollsters used that information to ensure the sample was geographically balanced.
In recent years, however, more and more households have given up their landlines or stopped answering them unless a recognized number appears on their caller ID. A recent Pew study, Assessing the Representativeness of Public Opinion Surveys, showed response rates declining to single digits in recent years, even for polls with many attempts to reach specific respondents.
When the decline in landline usage first began, it was generally possible to achieve good poll results calling those households that did still use landlines, and using those households to represent the general population. Recently, that approach has become a much more dicey proposition because those households that still have landlines look less and less like the overall population. Those sticking with landlines are more likely to be old, white, and live in either suburban or rural areas, while those accessible only by cell phone tend to be disproportionately young, non-white, lower-income urban dwellers.
Given these demographic differences, comparing the opinions of those reached by a cell phone and those reached by a landline in a given poll often shows very different results. The MassINC Polling Group’s quarterly poll in July of this year showed US Senator Scott Brown leading by 3 points among respondents who answered by landline, compared to a 13-point edge for Democrat Elizabeth Warren among cell phone respondents, a demonstration of the increasing hazards of excluding cell phone users from a poll sample.
Some pollsters are still clinging to “landline-only” methods, though the risks of doing so are significant and increasing. Others have started supplementing their landline samples with cell phone users or with Internet surveys, while others are offering multiple modes of response, such as mailed invitations with web links for online responses to dial-in telephone numbers allowing respondents to take a survey at a time of their choosing.
More and more pollsters (including all of the public pollsters located in Massachusetts) are now calling both landlines and cell phones, although the approach presents a number of challenges. Calling cell phone users is two to three times more expensive than calling a landline, largely because of regulatory restrictions on using dialing machines to reach cell phone users. Accounting for the fact that some respondents own both phone types and could be reached on either requires more complicated weighting of responses once the survey is completed to ensure accuracy. There are many ways of doing the weighting, but the best method is still a matter of considerable debate.
Cell phones present other challenges as well. Cell phones are mobile, so it’s difficult to know whether the user resides in the targeted area or not. For example, college students from out of state often bring their cell phone from home with a non-Massachusetts area code. This group’s opinion will not be counted in many telephone opinion polls, since most polls rely on randomly generated phone numbers beginning with Massachusetts area codes. The reverse is also true: former Massachusetts residents will be called to participate in Massachusetts polls if they kept their number when they moved out of state. Fortunately, these geographical challenges have not yet risen to the level of frequency to cause a serious problem, but they will only become more common as cell phones proliferate.
Surprisingly, some studies are showing that low-response-rate polls are still able to provide accurate estimates of election outcomes, though the explanation for why is unclear. Even though the percentage of people who respond to polls has fallen into the single digits, the aggregated views of the respondents still provide an accurate picture of the overall population. Put simply, those who answer the phone can still be relied on to represent the population—so far.
Anticipating a day when this is no longer true, many in the industry are considering how to either improve or replace telephone polling before falling response rates do undermine reliability. Many companies are also looking to the Internet for answers. Companies such as Knowledge Networks have created scientifically representative panels of respondents, but do so at huge expense by supplying Internet service and laptop computers to the portion of their panelists that do not have home Internet access.
Other companies, such as YouGov, are exploring innovative ways of composing reliable online samples without breaking the bank. Their methods rely on panelists who “opt in” to participate in such surveys, which for most pollsters has traditionally been a red flag since there is no way to determine whether those who opt-in are truly representative. YouGov has ditched the assumption that the panelists are randomly recruited and instead uses a matching algorithm where a representative sample of residents is essentially replicated using YouGov panelists. Underscoring the challenges offered by the new methods, a recent University of Massachusetts Amherst poll using YouGov generated a heated exchange among pollsters as to whether such a poll could be described using traditional terminology, such as a margin of error.
UMass Amherst professor Brian Schaffner, who has produced several papers on the YouGov methods, says
his research shows that the online polls conducted by YouGov produce results that are just as accurate as those that can be obtained from a modern telephone poll. “The fact is that in the modern communications environment, where telephone pollsters struggle to achieve response rates of even 10 percent, there is no single ideal way to reach individuals,” he says in an email. “Thus, innovative statistical techniques, like those used by YouGov, are a viable solution to ensure that our samples are as representative as possible.”
The American Association for Public Opinion Research’s 2012 conference included many presentations on different approaches to non-probability methods, a subject matter that would have been mostly unthinkable as recently as 10 years ago. Non-probability methods depart from the underpinnings of traditional surveys, which have had as their foundation the idea that anyone in the group being targeted has some measurable chance of being selected for a survey. For example, if each Massachusetts resident had an equal chance of being selected, their probability would be about 1 in about 6 million, which makes creating a representative sample a very straightforward proposition. With opt-in panels, one of the non-probability methods, if only 1 million people sign up for the panel, the rest have a zero percent chance of being selected, which challenges the principles of probability on which survey research has historically relied. It is these very principles that allow a survey researcher to talk to only 400 or 500 people and say with a measureable level of confidence that their opinions represent the entire population.
Included among these non-probability methods are Google Surveys, conventional online panels, and analysis of social network data in lieu of conducting a survey, among others. Each has its uses, but their common challenge is assessing the degree to which the respondents represent the general public. Abandoning traditional probability methods, where the opinion of each person is treated as equally important, is not without significant risk, particularly the risk of no longer representing the totality of the population. The people who sign up for online survey panels, who post their everyday life on Facebook, or share their opinion on Twitter are not the same as the population as a whole, and there is no guarantee that analyzing the data from these sources can reliably teach us about the opinions of the population as a whole.
What is also unclear is how much longer traditional telephone research will be able to do so.
Steve Koczela is president of The MassINC Polling Group. Paul Braun is CEO of Braun Research Inc., a call center.