Gallup explains what went wrong with its 2012 polls
June 4th, 2013
03:23 PM ET
10 years ago

Gallup explains what went wrong with its 2012 polls

Washington (CNN) - Saying "our intent is to do a better job," Gallup Editor-in-chief Frank Newport spelled out four main reasons why his organization's surveys understated President Barack Obama's support in the 2012 presidential election.

And at a news conference at Gallup's offices in the nation's capital Tuesday, Newport outlined the experiments the country's oldest and best known polling organization has already taken - and steps it will continue to take later this year - to understand why its surveys had a pro-Republican bias and how to correct the situation.

"We think there were issues related to the way in which we conduct surveys which needed to be corrected, which were responsible for some of the fact that we were too Mitt Romney. We've corrected those," said Newport.

Gallup's final national poll prior to last November's election indicated GOP nominee Mitt Romney with the support of 49% of likely voters and the president at 48%. Obama ended up winning the national popular vote 51%-47%.

Of the four reasons Newport outlined, Gallup's likely voter model received the most attention. The organization used a series of seven questions to determine which voters are likely to cast ballots in an election, a system first developed in the 1950s.

"I do believe in terms of Gallup and the industry, underrepresentation of President Obama's four point margin over Romney, a lot of that is due to the estimated likely voters, which we're going to have to work on," said Newport.

The Gallup report state's that their "procedures are broadly similar to those of other survey organizations, most of which sort respondents into likely voters and non-likely voters based on self-reported turnout questions," but it goes on to state that "Gallup's likely voter questions are more heavily weighted toward past voting behavior than other firms' questions."

Gallup's post mortem points to a question which measures how much thought respondents were giving to the election as a significant culprit. According to Gallup, that variable swung the advantage to Romney.

Concerns over Gallup's tracking polls were not limited to the period when the firm was using its likely voter model. In the week before the Republican Convention, for example, Gallup's poll showed Obama two points behind Romney or in a tie with him while all other major polls showed Obama ahead by one to four points. Gallup was not reporting results for likely voters at that point, indicating that there were other factors.

Gallup identified three other areas of concern that may have led to higher estimates of Romney's strength throughout the 2012 season.

One likely culprit was the way Gallup asked respondents for data on their racial background, and then used that date to adjust their surveys, known as 'weighting.' The result was that in Gallup's weighted data throughout 2012, white respondents probably had a greater impact on the horse race number than they should have.

"In early 2013, Gallup addressed these issues by changing the way it collects race and ethnicity data; respondents are now read a list of all racial and ethnic categories at once, and respondents can select up to five categories," states the Gallup report.

The post mortem also lists Gallup's "regional controls on interviews" as a factor. Newport said his organization interviewed too many people in the Central and Mountain time zones and not enough in the Eastern and Pacific time zones, where the president performed extremely well.

A third area of concern was how Gallup chose the telephone numbers that its interviewers called. In order to compensate for higher costs associated with calling more cell phones, Gallup changed from a standard technique called 'random digit dialing' (or 'RDD') to a method known as 'listed sampling' – a technique used by few other major polls – produced a sample that "consisted of older and more Republican respondents," according to the Gallup report.

"These differences likely contributed to Gallup's less accurate vote estimate," the report concluded. "Gallup has addressed this issue by making the transition back to an RDD list – assisted landline sample, which is now underway."

A couple of times during his presentation, Newport emphasized that what happened at Gallup was something "that affected the entire industry."

CNN's final poll of polls compiled prior to the November election, which was an average of the eight live operator, non-partisan national surveys conducted October 29 through November 4 (a CNN/ORC poll was one of the eight surveys) put Obama at 49% and Romney at 48% among likely voters.

But Gallup exhibited a pro-GOP bias throughout 2012, and since it is the most well-known polling organization in the country, it received the lion's share of blame.

"Because we've been around longer and are larger and more visible, we probably do get more attention from the press than other organizations. There's no doubt about that. I wouldn't say unfairly, it comes with the territory," said Newport.

Gallup's very public review appears to be a first of its kind, and the research its conducting as part of the review, could have an impact on polling in the future. University of Michigan professor Michael Traugott, who teamed up with Gallup to conduct the project, stressed that a lot is on the line.

"Political polling is the public face of survey research," said Traugott. "We know that confidence in the method and the image of the entire industry are related to how well the pre-election pollsters do."

Filed under: Mitt Romney • Polls • President Obama
soundoff (78 Responses)
  1. Ol' Yeller

    I went back and read your posts Tom... I understand perfectly now. You are one of those folks who feel superior to Dems and reps because you vote for the Libertarian whacko guy. Nothing personal against Paul, I kinda' like some of his ideas, but then again some of his ideas are incredibly insane. Voting for him and calling everyone else crooks does not make you superior or smarter than sniffit or anyone else on here (okay, it would make you smarter than just sayin', but that's like showing you are a smarter than styrofoam cup... no big feat) it just shows you have an opinion and that's great. Why. other than your obvious jealousy, are you singling out sniffit?
    BTW- I think sniffit can more than hold his own against the likes of you, but I kinda' like him... but then again, I'm as partisan as they come 🙂

    June 4, 2013 05:50 pm at 5:50 pm |
  2. Lynda/Minnesota

    Ol' Yeller

    Gosh Tom... get off sniffit. He has the same right to be partisan as the hacks on here who would attack Obama if he say.... ordered the Seals to kill OBL. After 8 years of being verbally assaulted and having your patriotism questioned simply because you didn't fall in lock step with the most corrupt bunch of fascists ever to steal office, I can relate to sniffit's partisan posts, especially when these inane morons get on here and accuse President Obama of being Satan's daddy or whatever...

    Well said, Ol' Yeller. Well said.

    June 4, 2013 05:51 pm at 5:51 pm |
  3. Sniffit

    " But he signed a search warrant."

    Again, you're conflating the affidavit and the search warrant with potential prosecution of Rosen. What's hard to understand about them not being the same thing when legal experts are telling you they're not the same thing? Hmmm? What's hard to understand about the intent all along not being to drag Rosen into court but to get the person who committed the leak? Your desperation to find a toe-hold for your conspiracy theories makes it not worth trying to talk to you anymore. Good day, sir.

    June 4, 2013 05:51 pm at 5:51 pm |
1 2 3 4