Gallup explains what went wrong with its 2012 polls
June 4th, 2013
03:23 PM ET
2 years ago

Gallup explains what went wrong with its 2012 polls

Washington (CNN) - Saying "our intent is to do a better job," Gallup Editor-in-chief Frank Newport spelled out four main reasons why his organization's surveys understated President Barack Obama's support in the 2012 presidential election.

And at a news conference at Gallup's offices in the nation's capital Tuesday, Newport outlined the experiments the country's oldest and best known polling organization has already taken - and steps it will continue to take later this year - to understand why its surveys had a pro-Republican bias and how to correct the situation.

"We think there were issues related to the way in which we conduct surveys which needed to be corrected, which were responsible for some of the fact that we were too Mitt Romney. We've corrected those," said Newport.

Gallup's final national poll prior to last November's election indicated GOP nominee Mitt Romney with the support of 49% of likely voters and the president at 48%. Obama ended up winning the national popular vote 51%-47%.

Of the four reasons Newport outlined, Gallup's likely voter model received the most attention. The organization used a series of seven questions to determine which voters are likely to cast ballots in an election, a system first developed in the 1950s.

"I do believe in terms of Gallup and the industry, underrepresentation of President Obama's four point margin over Romney, a lot of that is due to the estimated likely voters, which we're going to have to work on," said Newport.

The Gallup report state's that their "procedures are broadly similar to those of other survey organizations, most of which sort respondents into likely voters and non-likely voters based on self-reported turnout questions," but it goes on to state that "Gallup's likely voter questions are more heavily weighted toward past voting behavior than other firms' questions."

Gallup's post mortem points to a question which measures how much thought respondents were giving to the election as a significant culprit. According to Gallup, that variable swung the advantage to Romney.

Concerns over Gallup's tracking polls were not limited to the period when the firm was using its likely voter model. In the week before the Republican Convention, for example, Gallup's poll showed Obama two points behind Romney or in a tie with him while all other major polls showed Obama ahead by one to four points. Gallup was not reporting results for likely voters at that point, indicating that there were other factors.

Gallup identified three other areas of concern that may have led to higher estimates of Romney's strength throughout the 2012 season.

One likely culprit was the way Gallup asked respondents for data on their racial background, and then used that date to adjust their surveys, known as 'weighting.' The result was that in Gallup's weighted data throughout 2012, white respondents probably had a greater impact on the horse race number than they should have.

"In early 2013, Gallup addressed these issues by changing the way it collects race and ethnicity data; respondents are now read a list of all racial and ethnic categories at once, and respondents can select up to five categories," states the Gallup report.

The post mortem also lists Gallup's "regional controls on interviews" as a factor. Newport said his organization interviewed too many people in the Central and Mountain time zones and not enough in the Eastern and Pacific time zones, where the president performed extremely well.

A third area of concern was how Gallup chose the telephone numbers that its interviewers called. In order to compensate for higher costs associated with calling more cell phones, Gallup changed from a standard technique called 'random digit dialing' (or 'RDD') to a method known as 'listed sampling' – a technique used by few other major polls – produced a sample that "consisted of older and more Republican respondents," according to the Gallup report.

"These differences likely contributed to Gallup's less accurate vote estimate," the report concluded. "Gallup has addressed this issue by making the transition back to an RDD list – assisted landline sample, which is now underway."

A couple of times during his presentation, Newport emphasized that what happened at Gallup was something "that affected the entire industry."

CNN's final poll of polls compiled prior to the November election, which was an average of the eight live operator, non-partisan national surveys conducted October 29 through November 4 (a CNN/ORC poll was one of the eight surveys) put Obama at 49% and Romney at 48% among likely voters.

But Gallup exhibited a pro-GOP bias throughout 2012, and since it is the most well-known polling organization in the country, it received the lion's share of blame.

"Because we've been around longer and are larger and more visible, we probably do get more attention from the press than other organizations. There's no doubt about that. I wouldn't say unfairly, it comes with the territory," said Newport.

Gallup's very public review appears to be a first of its kind, and the research its conducting as part of the review, could have an impact on polling in the future. University of Michigan professor Michael Traugott, who teamed up with Gallup to conduct the project, stressed that a lot is on the line.

"Political polling is the public face of survey research," said Traugott. "We know that confidence in the method and the image of the entire industry are related to how well the pre-election pollsters do."


Filed under: Mitt Romney • Polls • President Obama
soundoff (78 Responses)
  1. Thomas

    Karl Rove maybe ?

    June 4, 2013 03:31 pm at 3:31 pm |
  2. Kikaha Wolfe

    Excuses, excuses, excuses

    June 4, 2013 03:31 pm at 3:31 pm |
  3. Dutch/Bad Newz, VA -aka- Take Back The House -aka- No Redemption Votes

    What went wrong was the voter surpression tactics the GOP were trying to implement backfired.

    June 4, 2013 03:32 pm at 3:32 pm |
  4. just sayin

    well, besides flawed polling, now we know that the democrats were using the irs to suppress the tea party and conservative vote. the largest election tampering case in history.

    and of course, the lsm was a co-conspirator with the obama administration in burying and concealing the facts concerning benghazi.

    June 4, 2013 03:32 pm at 3:32 pm |
  5. Sniffit

    17 pages of "how to euphemize 'we farked up royally'."

    June 4, 2013 03:32 pm at 3:32 pm |
  6. The Real Tom Paine

    Biased? Really? You mean, like Rasmussen/Dick Morris biased? Gallup? How could that be?

    June 4, 2013 03:34 pm at 3:34 pm |
  7. Babyboomer

    It doesn't take a rocket scientist to figure out what went wrong here. Gallup is old and antiquated just like the Republican party. They heard what they wanted to hear and reported it so. Not only did they go by "Likely Voters" from years other than 2008 but they contacted too many land lines which are no longer used by young people who are now the real "Likely Voters". This is a no brainer..

    June 4, 2013 03:35 pm at 3:35 pm |
  8. Dutch/Bad Newz, VA -aka- Take Back The House -aka- No Redemption Votes

    and of course, the lsm was a co-conspirator with the obama administration in burying and concealing the facts concerning benghazi.
    ----------------------------------------------
    "Proceed Governor."

    June 4, 2013 03:37 pm at 3:37 pm |
  9. The Real Tom Paine

    -just sayin

    well, besides flawed polling, now we know that the democrats were using the irs to suppress the tea party and conservative vote. the largest election tampering case in history.

    and of course, the lsm was a co-conspirator with the obama administration in burying and concealing the facts concerning benghazi.
    *******************
    Right. The vast liberal conspiracy defeated you, not a whack-job platform half-heartedly pushed by a Mormon aristocrat that no one was enthusistic about, coupled with voter supression laws that targeted far more people than the IRS did with the Tea Party. The only thing you are missing is the Freemasons.

    June 4, 2013 03:37 pm at 3:37 pm |
  10. just askin

    gallup failed to account adequately for the massive illegal and dead person vote that the democrats created.

    June 4, 2013 03:37 pm at 3:37 pm |
  11. Sniffit

    "well, besides flawed polling, now we know that the democrats were using the irs to suppress the tea party and conservative vote. "

    Scrutinizing groups who were going to use secretly donated money to buy advertising is not "voter suppression." Your Teatroll psychological projectionator is malfunctioning. Get it a tune up and maybe next time it will be a better attempt.

    June 4, 2013 03:39 pm at 3:39 pm |
  12. Sniffit

    "and of course, the lsm was a co-conspirator with the obama administration in burying and concealing the facts concerning benghazi."

    Please point to the page in Gallup's report that says that.

    June 4, 2013 03:39 pm at 3:39 pm |
  13. Lynda/Minnesota

    "We think there were issues related to the way in which we conduct surveys which needed to be corrected, which were responsible for some of the fact that we were too Mitt Romney. We've corrected those," said Newport."

    Seeing that Mitt Romney won't be on the 2016 ticket, there shouldn't be a worry as to whether or not Gallup is too Mitt Romney come 2016.

    Considering that Gallup was off on its polling data from the get go ... yeah, it did get a lot of deserved negative attention ... regardless of how long its been polling.

    This is what happens when pollsters try to lead the conversation in a direction the majority of Americans aren't willing to go. No doubt Gallup employees spent more time crossing their fingers hoping for a certain result then they spent actually polling for truth.

    For the most part, other "pollsters" went ahead and followed Gallup's cross your fingers approach as well. One might even suggest that they were hoping to sway the public into a false narrative.

    Apparently, Mitt Romney bought the false narrative. The rest of us? Not so much.

    June 4, 2013 03:40 pm at 3:40 pm |
  14. Wake up People!

    Who gives a ish? Mitt lost. It's over, move the hell on!!

    June 4, 2013 03:42 pm at 3:42 pm |
  15. Sniffit

    "I don't know what the costs are, and frankly I don't care," Christie said."

    Bwahahahahaha....2016 campaign gold for the Dems. can't wait to see the ad.

    June 4, 2013 03:43 pm at 3:43 pm |
  16. Kevin

    I think it's interesting and very sad that people are coming out with conspiracy theories when, in fact, it's actually very hard to predict how the country will vote and subtle ways in which polls are collected – like what's described in the article – could easily account for the discrepancy between the predicted and actual outcomes.

    June 4, 2013 03:43 pm at 3:43 pm |
  17. Gurgyl

    Question this after FIFTY YEARS. Idiots. If you have to fill–fill it with some trash–not this worth. Idiots. Go to journalism school. Or go to some NC beaches.

    June 4, 2013 03:50 pm at 3:50 pm |
  18. Wake up People!

    I don't think I've ever seen so much talk about an election defeat almost a year after. I don't remember any articles of this type after the 2000 election. Remember THAT one MSM?? That is the one you should have been debating. And don't worry Gallup, apparently you weren't the only sad people on election night......

    June 4, 2013 03:51 pm at 3:51 pm |
  19. Al-NY,NY

    just sayin

    and of course, the lsm was a co-conspirator with
    ---–

    the use of that term "lamestream media" automatically makes anything you say to be void of logic, thought, or originality

    June 4, 2013 03:51 pm at 3:51 pm |
  20. GOP = Greed Over People

    Two words:

    Nate Silver.

    Next time, listen to him, instead of ridiculing him.

    And yes, Joe Scarborough, I'm talking to you.

    June 4, 2013 03:52 pm at 3:52 pm |
  21. Rudy NYC

    I heard an explanation of the problems from the Gallup chief this morning. In each case, he basically said was, "What Gallup did in 2012 did not work, and we will do it differently next time. Gallup apologizes." He guess his trying to repair their image in the eyes of the press. The general public could care less.

    June 4, 2013 03:56 pm at 3:56 pm |
  22. Ratt

    I didn't see anything go wrong, it went just like I predicted.

    72 Million on Medicaid
    38 Million on Medicare
    49 Million on Food Stamps
    12 Million on Disability
    9 Million Free Cell Phones
    12 Million Absentee ballots submitted by ACORN
    15 Million Illegals
    Throw in another 10 – 15 Million white liberals

    And, Bada Bing Bada Boom, you got yourself an election.

    June 4, 2013 03:57 pm at 3:57 pm |
  23. Tom

    How is it possible that a liberal after 8 years of George Bush cannot understand the skepticism of conservatives with Barack Obama.

    And if you think Barack Obama is this good, honest man then you are just to blinded by partisanship.

    June 4, 2013 03:59 pm at 3:59 pm |
  24. Guess

    Just sayin, thanks man, i have not laughed that hard for a while, wow, your in deep denial bub, must be a real interesting place you live. seek help immediatly. i remember the gop getting caught with the voter fraud thing, two election cycles in a row, same guy. all the voter supression and you still lost badly, must be a messaging thing huh?.. bwaa haa haa

    June 4, 2013 04:00 pm at 4:00 pm |
  25. Bahska

    That said most polls on other sites like Huffington had obama in the lead by a large margin. It was mostly ignorance and self defeating comments like the Pregnancy during rape isn't real" and wanting to cut money for the poor (which is most of this country sadly) and projects like FEMA right after a major hurricane by the republican party for the "surprise" loss

    June 4, 2013 04:02 pm at 4:02 pm |
1 2 3 4