Gallup explains what went wrong with its 2012 polls
June 4th, 2013
03:23 PM ET
2 years ago

Gallup explains what went wrong with its 2012 polls

Washington (CNN) - Saying "our intent is to do a better job," Gallup Editor-in-chief Frank Newport spelled out four main reasons why his organization's surveys understated President Barack Obama's support in the 2012 presidential election.

And at a news conference at Gallup's offices in the nation's capital Tuesday, Newport outlined the experiments the country's oldest and best known polling organization has already taken - and steps it will continue to take later this year - to understand why its surveys had a pro-Republican bias and how to correct the situation.

"We think there were issues related to the way in which we conduct surveys which needed to be corrected, which were responsible for some of the fact that we were too Mitt Romney. We've corrected those," said Newport.

Gallup's final national poll prior to last November's election indicated GOP nominee Mitt Romney with the support of 49% of likely voters and the president at 48%. Obama ended up winning the national popular vote 51%-47%.

Of the four reasons Newport outlined, Gallup's likely voter model received the most attention. The organization used a series of seven questions to determine which voters are likely to cast ballots in an election, a system first developed in the 1950s.

"I do believe in terms of Gallup and the industry, underrepresentation of President Obama's four point margin over Romney, a lot of that is due to the estimated likely voters, which we're going to have to work on," said Newport.

The Gallup report state's that their "procedures are broadly similar to those of other survey organizations, most of which sort respondents into likely voters and non-likely voters based on self-reported turnout questions," but it goes on to state that "Gallup's likely voter questions are more heavily weighted toward past voting behavior than other firms' questions."

Gallup's post mortem points to a question which measures how much thought respondents were giving to the election as a significant culprit. According to Gallup, that variable swung the advantage to Romney.

Concerns over Gallup's tracking polls were not limited to the period when the firm was using its likely voter model. In the week before the Republican Convention, for example, Gallup's poll showed Obama two points behind Romney or in a tie with him while all other major polls showed Obama ahead by one to four points. Gallup was not reporting results for likely voters at that point, indicating that there were other factors.

Gallup identified three other areas of concern that may have led to higher estimates of Romney's strength throughout the 2012 season.

One likely culprit was the way Gallup asked respondents for data on their racial background, and then used that date to adjust their surveys, known as 'weighting.' The result was that in Gallup's weighted data throughout 2012, white respondents probably had a greater impact on the horse race number than they should have.

"In early 2013, Gallup addressed these issues by changing the way it collects race and ethnicity data; respondents are now read a list of all racial and ethnic categories at once, and respondents can select up to five categories," states the Gallup report.

The post mortem also lists Gallup's "regional controls on interviews" as a factor. Newport said his organization interviewed too many people in the Central and Mountain time zones and not enough in the Eastern and Pacific time zones, where the president performed extremely well.

A third area of concern was how Gallup chose the telephone numbers that its interviewers called. In order to compensate for higher costs associated with calling more cell phones, Gallup changed from a standard technique called 'random digit dialing' (or 'RDD') to a method known as 'listed sampling' – a technique used by few other major polls – produced a sample that "consisted of older and more Republican respondents," according to the Gallup report.

"These differences likely contributed to Gallup's less accurate vote estimate," the report concluded. "Gallup has addressed this issue by making the transition back to an RDD list – assisted landline sample, which is now underway."

A couple of times during his presentation, Newport emphasized that what happened at Gallup was something "that affected the entire industry."

CNN's final poll of polls compiled prior to the November election, which was an average of the eight live operator, non-partisan national surveys conducted October 29 through November 4 (a CNN/ORC poll was one of the eight surveys) put Obama at 49% and Romney at 48% among likely voters.

But Gallup exhibited a pro-GOP bias throughout 2012, and since it is the most well-known polling organization in the country, it received the lion's share of blame.

"Because we've been around longer and are larger and more visible, we probably do get more attention from the press than other organizations. There's no doubt about that. I wouldn't say unfairly, it comes with the territory," said Newport.

Gallup's very public review appears to be a first of its kind, and the research its conducting as part of the review, could have an impact on polling in the future. University of Michigan professor Michael Traugott, who teamed up with Gallup to conduct the project, stressed that a lot is on the line.

"Political polling is the public face of survey research," said Traugott. "We know that confidence in the method and the image of the entire industry are related to how well the pre-election pollsters do."


Filed under: Mitt Romney • Polls • President Obama
soundoff (78 Responses)
  1. ladeeda

    Looks like the results still came within the standard margin of error + – 3%. I guess it is simply annoying when it goes nearly the full error + 3% for Obama and down 2% for Romney. I don't really know what they are concerned about. Looks like they did OK.

    June 4, 2013 04:53 pm at 4:53 pm |
  2. Mark

    You don't think the major news networks wanted to make it sound closer than it was to keep people glued to the TV, do you? Naaaaaah. They wouldn't do that.

    Even with the stacked stats, Mitt looked like he was going down. So, all the GOP talkers were out there trying to claim that the pollsters were all stacking the deck for the liberals... when in fact, Mitt was losing much worse than he was.

    I remember Newt Gingrich talking about the polls only mattering if you live in a "New York Times fantasy world," and that we would get the real figures when REAL voters got out there (because Democratic voters aren't REAL... apparently). In fact, the polls were wrong Newt! Obama won every swing state by larger margins than they predicted. The only poll that WAS wrong was the poll showing Romney would probably win Florida.

    June 4, 2013 04:58 pm at 4:58 pm |
  3. Tom

    Sniffit,
    I will repeat myself, to an extent. For such a smart man you completely 100% back everything that this administration has done. You have zero skepticism towards Pres Obama, always believe everything he has said, and then have the audacity to mischaracterize a group of Americans with wild assertions that only a person of extreme partisanship would do; very similar to our president.
    You need to gain some perspective. I have read probably hundreds of your comments. All are snide towards republicans and all speak glowingly of democrats. You just don't see it. That's because you are blindly following one party.
    An honest person would be disgusted by the IRS without qualification. You qualify everything.
    Again, for a smart person, I don't understand why you don't think for yourself.
    The way you have spoken about the TEA Party speaks volumes for the partisan democrat person that you are.

    June 4, 2013 04:59 pm at 4:59 pm |
  4. The NRA is a T e r r o r i st organisation....and MORE JOBS has been created under President Obama 2times over than under GW Bush!!! ***FACT***

    They have TRIED every name & trick in the smear campaign book to defeat the BLACK MAN..........lied, called him names, insulted his mama, forged the poll numbers, called him anti-American, no-birth certificate Kenya muslim, but he still DEFEATED the GOP-TEAPOARTY TWICE....and WON IN A LANDSLIDE!!!

    Obama is STILL the President!!!.....GET USED TO IT ALREADY!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

    June 4, 2013 05:01 pm at 5:01 pm |
  5. Sniffit

    "are you serious??? no group should be targetted, period, end of story."

    They ALL should be "targeted" in the sense of being scrutinized like these groups were, particularly when they have overt intentions to engage in political activity. Of course, at that point it's not "targeting" anymore because you've erased the unequal treatment....i.e., it becomes "policy" not "targeting."

    June 4, 2013 05:02 pm at 5:02 pm |
  6. Dominican mama 4 Obama

    Lynda/Minnesota
    I noticed the same thing myself.
    Curious, yet the unconnected ramblings of the usual suspects are posted and re-posted in their entirety. It matters not if their oral regurgitations have anything at all to do with the story. As long as they get their prerequisite "union toady", "Benghazi cover-up", and "corrupt administration" in they get paid, er, posted.
    Is Gallup affiliated with CNN perhaps?
    Wink!

    June 4, 2013 05:04 pm at 5:04 pm |
  7. The NRA is a T e r r o r i st organisation....and MORE JOBS has been created under President Obama 2times over than under GW Bush!!! ***FACT***

    They have TRIED every name & trick in the smear campaign book to defeat the BLACK MAN..........lied, called him names, insulted his mama, forged the poll numbers, called him anti-American, no-birth certificate Kenya muslim, but he still DEFEATED the GOP-TEA-PARTY TWICE....and WON IN A LANDSLIDE!!!

    Obama is STILL the President!!!.....GET USED TO IT ALREADY PEOPLE!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

    June 4, 2013 05:05 pm at 5:05 pm |
  8. Tom

    Sniffit,
    A perfect example of your blindness is Holder. He clearly lied. He clearly knew about the potential prosecution and signed a search warrant.
    If you were not a blind partisan, you would be skeptical. You would wonder what was going on with that, too. Just because he went after a Fox guy is irrelevent, and really should concern you even more because that is an indication that someone from the Obama administration is going after his "enemies" – you do remember when Obama said that you reward your friends and go after your enemies? He said that
    But, alas, nothing from Sniffit. Just accusations about republicans and rationalizing away based on some technicality.
    I could go on an on with you Sniffit.
    Again, as a smart person, I can't believe how much you are swayed by one political party.

    June 4, 2013 05:07 pm at 5:07 pm |
  9. Chaerlotte

    What they're concerned about is loss of reveneue for the next time. Whoever pays for these polls might not be willing to pony up as much if they suspect the polling is at the fringes of uncertainty tolerance.

    June 4, 2013 05:08 pm at 5:08 pm |
  10. Whartonwizkid

    Gallup is like the GOP – Stuck in the 50's. The world has changed and so has survey methodology. The sole objective of conducting a survey is to get accurate information, something Nate Silver has excelled in.

    June 4, 2013 05:10 pm at 5:10 pm |
  11. The NRA is a T e r r o r i st organisation....and MORE JOBS has been created under President Obama 2times over than under GW Bush!!! ***FACT***

    DICK MORRIS who predicted on FAUX NEWS that Mitt Romney was gonna win in A MASSIVE LANDSLIDE with over 300 ELECTORAL VOTES is the idiot behind the GALLUP POLL NUMBERS!

    LOL

    June 4, 2013 05:11 pm at 5:11 pm |
  12. DaffyDuck

    I'm a cartoon character and neither me nor my cartoon friends got a call from Gallup. And we vote 100% for Democrats. Not just cartoon characters like me, but also other fictional characters.

    This is reality racism. Cartoons should not be discriminated against!

    June 4, 2013 05:19 pm at 5:19 pm |
  13. Dominican mama 4 Obama

    @ Tom
    Give it up Tom.
    If Holder lied the Rethugs would be all over him.
    Here's what you're doing and we all see it: you're trying to inject yourself into the limelight by trying to discredit Sniffit.
    I'm here to tell you that you are being as transparent in your attempt and as successful as your boy Issa.
    Shut up and sit down. You're not at Sniffit's level.
    Thanks for playing though!

    June 4, 2013 05:20 pm at 5:20 pm |
  14. The NRA is a T e r r o r i st organisation....and MORE JOBS has been created under President Obama 2times over than under GW Bush!!! ***FACT***

    No wonder Karl Rove HUMONGOUS HEAD exploded on the anti-Obama FAUX NEWS on election night...because he believed Gallup Poll!!!!

    June 4, 2013 05:21 pm at 5:21 pm |
  15. DaffyDuck

    I am a cartoon character, and I never got polled by Gallup. And neither did any of my cartoon friends. This is reality racism!!! 99.9% of cartoon characters vote for Democrats! I even registered in seven states!

    June 4, 2013 05:22 pm at 5:22 pm |
  16. Dominican mama 4 Obama

    @Wake up People!
    Right back at you babe!!
    Wink! Wink!

    June 4, 2013 05:23 pm at 5:23 pm |
  17. Sniffit

    "A perfect example of your blindness is Holder. He clearly lied. He clearly knew about the potential prosecution and signed a search warrant."

    What potential prosecution? Holder has consistently testified that there was never any intent or plan to prosecute Rosen and Holder would ultimately be the one deciding whether the guy was prosecuted. Are you seriously going to sit here and tell us that he intended to prosecute Rosen at the time he signed of on the search warrant and that Rosen was never prosecuted simply because "he never got around to it"? Do you have any idea how ridiculous that sounds without concrete evidence to back it up. And AGAIN, an affidavit calling Rosen a "potential co-conspirator" is not evidence that they intended to haul Rosen before a jury and throw him in jail. As I said before, perjury is a very technical charge, so complaining about "technicalities" is silly. You have to show not only an intent to mislead but also the untruth and deception of the statement. Neither have been shown with any concrete evidence whatsoever, just your misunderstanding of what an affidavit really means, your lack of understanding of the law and your willingness to make assumptions.

    June 4, 2013 05:25 pm at 5:25 pm |
  18. Ol' Yeller

    Gosh Tom... get off sniffit. He has the same right to be partisan as the hacks on here who would attack Obama if he say.... ordered the Seals to kill OBL. After 8 years of being verbally assaulted and having your patriotism questioned simply because you didn't fall in lock step with the most corrupt bunch of fascists ever to steal office, I can relate to sniffit's partisan posts, especially when these inane morons get on here and accuse President Obama of being Satan's daddy or whatever...
    Let's talk about this article. Oh, the fond memories I have of election night. The mainstream media was even fooled by the altered polls, which I suspect had a lot to do with FOX trying to skew everything to look better for their guy. The coverage I was watching had the commentators looking quite shocked when it was over so soon. I quickly switched over to FOX and RELISHED the meltdown of Karl Rove, whose efforts to steal this election fell flat despite spending more money per vote than any group in history. It was the BEST moment I've experienced in national politics in some time. Maybe more so than Obama's first win. Gives me hope for America and our youth, because one thing is for sure... until the republican party can run some sane candidates, put forth some reasonable ideas, and broaden their tent, they will continue to lose election after election and fall by the wayside of history.
    Which is absolutely hunky dory with me.

    June 4, 2013 05:26 pm at 5:26 pm |
  19. smith

    "What went wrong" that question should be directed at President Obama. Where`s all the hope and change? Well, in 2016 it will end and we can begin to fix this country.

    June 4, 2013 05:35 pm at 5:35 pm |
  20. Tom

    He said he had no knowledge of any potential prosecution. But he signed a search warrant. That would indicate that he clearly had knowledge.

    June 4, 2013 05:39 pm at 5:39 pm |
  21. Tom

    @Ol Yeller,
    So if you watched Karl Rove's meltdown then you would know that they basically mocked him. They told him he was wrong. If the best moment of your politcal life is that, that should indicate to you what a huge disappointment Obama must be to you.

    June 4, 2013 05:42 pm at 5:42 pm |
  22. Tom

    @Sniffit

    Just read this article. So now they have secret emails. This was not on Fox. Thoughts?

    "There's nothing secret," Carney said.

    The AP reviewed hundreds of pages of government emails released under the federal open records law and couldn't independently find instances when material from any of the secret accounts it identified was turned over. Congressional oversight committees told the AP they were unfamiliar with the few nonpublic government addresses that AP identified so far, including one for Secretary Kathleen Sebelius of the Health and Human Services Department.

    Again, a perfect opportunity for you to demonstrate that you are not a blinded by partisanship. I'm excited to see your reaction.

    June 4, 2013 05:44 pm at 5:44 pm |
  23. Sniffit

    "you completely 100% back everything that this administration has done."

    Untrue. I can't stand the treatment of Bradley Manning, I'm no fan of the drone program, I wish he had pushed harder for single-payer, I'm not fan at all of his resigning of the Patriot Act, not particularly fond of his offer to use chained CPI for the Social Security adjustments, I disagree with him backing down on assault weapons bans...and many other things.

    What your problem is, "my friend," is that you confuse my opposition to the GOP/Teatrolls' attempts to turn everything into a giant sh-tstorm of mischaracterizations and demagoguery as somehow defending Obama, but that's a mistake. Just because I disagree with the idiots claiming that drones are going to be used to bomb Americans while taking their morning dump doesn't mean I support drones being used to murder American citizens abroad without some form of due process or even that I agree with the innocent "collateral damage" involved in taking out a known terrorist. What I won't do, however, is sit idly by while the GOP/Teatrolls push their plutocratic, racist, bigoted mess on this country while others are at least trying to do an honest job. You want to call me partisan, that's fine, because I'll spit in GOPers/Teatrolls' eyes until the day I die, but it's not because I blindly follow the other guys.

    June 4, 2013 05:46 pm at 5:46 pm |
  24. Dominican mama 4 Obama

    @ Ol' Yeller who eloquently wrote:
    Let's talk about this article. Oh, the fond memories I have of election night. The mainstream media was even fooled by the altered polls, which I suspect had a lot to do with FOX trying to skew everything to look better for their guy. The coverage I was watching had the commentators looking quite shocked when it was over so soon. I quickly switched over to FOX and RELISHED the meltdown of Karl Rove, whose efforts to steal this election fell flat despite spending more money per vote than any group in history. It was the BEST moment I've experienced in national politics in some time. Maybe more so than Obama's first win. Gives me hope for America and our youth, because one thing is for sure... until the republican party can run some sane candidates, put forth some reasonable ideas, and broaden their tent, they will continue to lose election after election and fall by the wayside of history.
    Which is absolutely hunky dory with me.
    -------------------------------------------------
    Ol' Yeller you made my day!!!!!

    June 4, 2013 05:48 pm at 5:48 pm |
  25. Wake up People!

    Ol Yellar.

    I'm right there with you. Karl Rove and Ann Romney made my night!!

    June 4, 2013 05:50 pm at 5:50 pm |
1 2 3 4