Gallup explains what went wrong with its 2012 polls
June 4th, 2013
03:23 PM ET
11 months ago

Gallup explains what went wrong with its 2012 polls

Washington (CNN) - Saying "our intent is to do a better job," Gallup Editor-in-chief Frank Newport spelled out four main reasons why his organization's surveys understated President Barack Obama's support in the 2012 presidential election.

And at a news conference at Gallup's offices in the nation's capital Tuesday, Newport outlined the experiments the country's oldest and best known polling organization has already taken - and steps it will continue to take later this year - to understand why its surveys had a pro-Republican bias and how to correct the situation.

"We think there were issues related to the way in which we conduct surveys which needed to be corrected, which were responsible for some of the fact that we were too Mitt Romney. We've corrected those," said Newport.

Gallup's final national poll prior to last November's election indicated GOP nominee Mitt Romney with the support of 49% of likely voters and the president at 48%. Obama ended up winning the national popular vote 51%-47%.

Of the four reasons Newport outlined, Gallup's likely voter model received the most attention. The organization used a series of seven questions to determine which voters are likely to cast ballots in an election, a system first developed in the 1950s.

"I do believe in terms of Gallup and the industry, underrepresentation of President Obama's four point margin over Romney, a lot of that is due to the estimated likely voters, which we're going to have to work on," said Newport.

The Gallup report state's that their "procedures are broadly similar to those of other survey organizations, most of which sort respondents into likely voters and non-likely voters based on self-reported turnout questions," but it goes on to state that "Gallup's likely voter questions are more heavily weighted toward past voting behavior than other firms' questions."

Gallup's post mortem points to a question which measures how much thought respondents were giving to the election as a significant culprit. According to Gallup, that variable swung the advantage to Romney.

Concerns over Gallup's tracking polls were not limited to the period when the firm was using its likely voter model. In the week before the Republican Convention, for example, Gallup's poll showed Obama two points behind Romney or in a tie with him while all other major polls showed Obama ahead by one to four points. Gallup was not reporting results for likely voters at that point, indicating that there were other factors.

Gallup identified three other areas of concern that may have led to higher estimates of Romney's strength throughout the 2012 season.

One likely culprit was the way Gallup asked respondents for data on their racial background, and then used that date to adjust their surveys, known as 'weighting.' The result was that in Gallup's weighted data throughout 2012, white respondents probably had a greater impact on the horse race number than they should have.

"In early 2013, Gallup addressed these issues by changing the way it collects race and ethnicity data; respondents are now read a list of all racial and ethnic categories at once, and respondents can select up to five categories," states the Gallup report.

The post mortem also lists Gallup's "regional controls on interviews" as a factor. Newport said his organization interviewed too many people in the Central and Mountain time zones and not enough in the Eastern and Pacific time zones, where the president performed extremely well.

A third area of concern was how Gallup chose the telephone numbers that its interviewers called. In order to compensate for higher costs associated with calling more cell phones, Gallup changed from a standard technique called 'random digit dialing' (or 'RDD') to a method known as 'listed sampling' – a technique used by few other major polls – produced a sample that "consisted of older and more Republican respondents," according to the Gallup report.

"These differences likely contributed to Gallup's less accurate vote estimate," the report concluded. "Gallup has addressed this issue by making the transition back to an RDD list – assisted landline sample, which is now underway."

A couple of times during his presentation, Newport emphasized that what happened at Gallup was something "that affected the entire industry."

CNN's final poll of polls compiled prior to the November election, which was an average of the eight live operator, non-partisan national surveys conducted October 29 through November 4 (a CNN/ORC poll was one of the eight surveys) put Obama at 49% and Romney at 48% among likely voters.

But Gallup exhibited a pro-GOP bias throughout 2012, and since it is the most well-known polling organization in the country, it received the lion's share of blame.

"Because we've been around longer and are larger and more visible, we probably do get more attention from the press than other organizations. There's no doubt about that. I wouldn't say unfairly, it comes with the territory," said Newport.

Gallup's very public review appears to be a first of its kind, and the research its conducting as part of the review, could have an impact on polling in the future. University of Michigan professor Michael Traugott, who teamed up with Gallup to conduct the project, stressed that a lot is on the line.

"Political polling is the public face of survey research," said Traugott. "We know that confidence in the method and the image of the entire industry are related to how well the pre-election pollsters do."


Filed under: Mitt Romney • Polls • President Obama
soundoff (78 Responses)
  1. Dominican mama 4 Obama

    and steps it will continue to take later this year – to understand why its surveys had a pro-Republican bias
    -------------------------------------------------
    Oh please.
    You LIED.
    You were hellbent on making people think that Mittens had it in the bag.
    You did your part in continuing the narrative that you wanted out there, regardless of the truth.,
    You lied, like you always do but we waited for HOURS to vote.
    You didn't count on that.
    You LIED but we VOTED.

    June 4, 2013 04:03 pm at 4:03 pm |
  2. Tom

    Sniffit,
    For such a smart man I can't believe how swayed you are by one political party. It's amazing to me.

    The fact that you don't have the intellectual honesty to recognize that the IRS was wrong (which they admitted to) shows an incredible lack of perspective. They targeted one group while not targeting the other. They have admitted this. There is testimony to that fact. And you still blindly back the IRS which demonstrates your inability to be fair or think for yourself. That is sad for such an intelligent person. You should start to think for yourself.

    June 4, 2013 04:03 pm at 4:03 pm |
  3. Guess

    What went wrong, you thought you could get people to vote for romney if the polls leaned that way, idiots

    June 4, 2013 04:03 pm at 4:03 pm |
  4. Rudy NYC

    just sayin wrote:

    well, besides flawed polling, now we know that the democrats were using the irs to suppress the tea party and conservative vote. the largest election tampering case in history.
    -----------
    Please explain how that works. Please, explain how paying your taxes affects your participation at the voting booth. Please, explain why the IRS should not take a closer look at organizations with 'tea party" in their names?

    There are a number of caucus groups in the House. One among them is the Tea Party Caucus, while another is Congressional Black Caucus. It the IRS was suddenly hit by a large number of groups with the phrase "congressional black" as part of their name, don't think the IRS should take a closer look at the group?

    June 4, 2013 04:05 pm at 4:05 pm |
  5. cnnife

    Gallup really think most people believe their poll? How can a candidate be soooo unpopular among women, blacks, Hispanics, young people and educated American and still predict that he will win a presidential election!

    June 4, 2013 04:09 pm at 4:09 pm |
  6. tony h

    finally admitted errors. would have thought when Gallup polls consistently were at odds with almost all other polls and forecasts, an adjustment would have been done THEN in 2012. This way wouldn't have looked like the 2012 version of the 1936 "Literary Digest" which predicted Alf Landon was going to beat FDR by a landslide.

    June 4, 2013 04:11 pm at 4:11 pm |
  7. The Real Tom Paine

    -Tom

    Sniffit,
    For such a smart man I can't believe how swayed you are by one political party. It's amazing to me.

    The fact that you don't have the intellectual honesty to recognize that the IRS was wrong (which they admitted to) shows an incredible lack of perspective. They targeted one group while not targeting the other. They have admitted this. There is testimony to that fact. And you still blindly back the IRS which demonstrates your inability to be fair or think for yourself. That is sad for such an intelligent person. You should start to think for yourself.
    ***************
    Tom, its not blindly backing the IRS when you prefer to get all the facts before rendering judgement. Yes, the IRS admitted wrongdoing, but whether it a few rogues or a systematic attempt at targeting by the White House is what some people are trying to determine. The people like Darrell Issa who inhale the reality that FOX and other Right-wing organizations create for themselves ( like Gallup), end up contributing nothing of value to the conversation. Concluding that this was orchestrated by the White House before a shred of real proof presents itself is nothing more than an idication of how easily you are swayed, not Sniffit.

    June 4, 2013 04:14 pm at 4:14 pm |
  8. Wake up People!

    @Dominican Mama!!

    My girl!! Right on the money lady!!

    June 4, 2013 04:14 pm at 4:14 pm |
  9. ghostriter

    Conservative thought process:

    Holding up folks tax exempt status (which in no way impeded them for doing whatever it is they wanted to do) is voter suppression.

    Creating new ID laws months before an election that had to be shot down by the SC, changing ID laws to make it easier for gun owners to vote, but not students (Texas), reducing voting times and days in democratic districts while increasing them in republican districts (ohio)

    Business as usual......

    June 4, 2013 04:15 pm at 4:15 pm |
  10. Dominican mama 4 Obama

    @Dutch Bad Newz/VA

    "Proceed Governor."
    -------------------------------------------------
    Good one Dutch!!!
    I will never forget the delivery of that line and the blithering idiot he was delivering it to, much like your delivery today!

    June 4, 2013 04:15 pm at 4:15 pm |
  11. Donna

    Isn't it curious how the IRS scandal was known well ahead of the 2012 election, yet it was never disclosed to the American people?

    Isn't it curious how the Justice Dept. and Eric Holder was targeting reporters before the 2012 election, yet it was never disclosed to the American people?

    Isn't it curious how the facts aroubd the Benghazi attack were known before the 2012 election, yet it was never disclosed to the American people? (and to this day, the Obama administration has not come clean)

    2012 will mark the official DEATH of journalism in this country. It is the year that the news media removed the veil and revealed themselves as the left wing Democrat tool we all suspected they were.

    June 4, 2013 04:19 pm at 4:19 pm |
  12. Rudy NYC

    Ratt wrote:

    I didn't see anything go wrong, it went just like I predicted.

    72 Million on Medicaid
    38 Million on Medicare
    49 Million on Food Stamps
    12 Million on Disability
    9 Million Free Cell Phones
    12 Million Absentee ballots submitted by ACORN
    15 Million Illegals
    Throw in another 10 – 15 Million white liberals
    ---------------------
    70% of Medicaid/Medicaid dollars spent went to people who voted for Romney, old folks in nursing homes.
    50% of those on food stamps signed up after The Great Recession began in 2007.
    12 million on disability. And?
    9 million free cell phones is a program created by a Republican POTUS.
    12 million ballots from ACORN, despite the fact that ACORN was dismantled back in 2009.
    15 million illegals. And?
    I think there are a lot more than 10-15 liberals that one could describe as "white".

    All of this adds up to a ton of sour grapes per Republican voter. Get over it. It's called democracy.

    June 4, 2013 04:20 pm at 4:20 pm |
  13. Larry in Houston

    LOL – CNN – what do u mean, "what went wrong" ?? do u think it could have been "rove math" ?? first of all, forget all of the so called "polls" – - Let me give you a hint : Obama got elected because the typical average Voter didn't want to go back to the 1950's – where the women didn't have the "rights" they have today, & I can go on & on. (especially on that one)
    # 2 ) Obama got "elected" (from the typical average Voter, who has an income of 150 K ( per household) of the "40 & under" Crowd – ( not the "over 40, or over 60 crowd)
    Hope this clears things up.

    June 4, 2013 04:20 pm at 4:20 pm |
  14. Sniffit

    "For such a smart man I can't believe how swayed you are by one political party. It's amazing to me."

    I'm not partisan other than to find the GOP/Teatrolls to be heinous and the most dangerous thing to our country our economy and our middle-class and "least among us" in existence. I do not declare party, nor do I support Dems solely because they are Dems. I know it may seem that way because I spend my time attacking the ridiculous BS we have to put up with from the GOP/Teatrolls, but it's not the case. I've spoken out for them when they behave reasonably too...but that's the extreme exception to the rule lately. They went off the deep-end and refuse to return.

    As for intellectual honesty, I am being honest. I don't back the IRS and I never said what they did wasn't wrong. What I did was disagree with the characterization of WHY it was wrong. I sincerely believe that what they did was wrong SOLELY because it wasn't done to everyone, BUT SHOULD HAVE BEEN DONE TO EVERYONE. That's not intellectually dishonest. That's taking a different position than the MSM's desperate narrative that is designed to protect the 501(c)(4) ad revenue gravy train....all by pretending that scrutinizing the applications from obvious political groups was wrong in and of itself.

    June 4, 2013 04:23 pm at 4:23 pm |
  15. Sniffit

    "For such a smart man I can't believe how swayed you are by one political party. It's amazing to me."

    I'm not partisan other than to find the GOP/Teatrolls to be heinous and the most dangerous thing to our country our economy and our middle-class and "least among us" in existence. I do not declare party, nor do I support Dems solely because they are Dems. I know it may seem that way because I spend my time attacking the ridiculous BS we have to put up with from the GOP/Teatrolls, but it's not the case. I've spoken out for them when they behave reasonably too...but that's the extreme exception to the rule lately. They went off the deep-end and refuse to return.

    June 4, 2013 04:24 pm at 4:24 pm |
  16. Sniffit

    As for intellectual honesty, I am being honest. I don't back the IRS and I never said what they did wasn't wrong. What I did was disagree with the characterization of WHY it was wrong. I sincerely believe that what they did was wrong SOLELY because it wasn't done to everyone, BUT SHOULD HAVE BEEN DONE TO EVERYONE. That's not intellectually dishonest. That's taking a different position than the MSM's desperate narrative that is designed to protect the 501(c)(4) ad revenue gravy train....all by pretending that scrutinizing the applications from obvious political groups was wrong in and of itself.

    June 4, 2013 04:25 pm at 4:25 pm |
  17. Lynda/Minnesota

    Dominican mama 4 Obama: "Oh please. You LIED. You were hellbent on making people think that Mittens had it in the bag."

    Much to our amusement, it didn't quite work out the way the pollsters hoped for, did it? On a side note, I kind of hate to comment on your comments. For whatever reason, not only is my comment not printed, but your comment disappears entirely.

    We'll see how this one works out for me (and you).

    June 4, 2013 04:26 pm at 4:26 pm |
  18. Sniffit

    "Isn't it curious how the IRS scandal was known well ahead of the 2012 election, yet it was never disclosed to the American people?"

    Yeah, Issa knew and said nothing.

    Next.

    June 4, 2013 04:26 pm at 4:26 pm |
  19. don in albuquerque

    Yeah, but you gotta admit....the look on Roves face that night on Fox........Priceless.

    June 4, 2013 04:27 pm at 4:27 pm |
  20. Sniffit

    "2012 will mark the official DEATH of journalism in this country. It is the year that the news media removed the veil and revealed themselves as the left wing Democrat tool we all suspected they were."

    Can't kill something that was already dead.

    Isn't it curious that you're so desperate to find some excuse for why the GOP/Teatrolls/Mittens lost the election to avoid having to admit the majority of the country is rejecting the GOP/Teatrolls' ideology, policy positions and repeatedly racist/bigoted/religio-fascist/misognynistic behavior?

    June 4, 2013 04:28 pm at 4:28 pm |
  21. Joan

    It is proving over and over that polls are completely out to lunch. Maybe in days gone by when everyone just had land lines and information was limited to nightly news shows and newspapers they had some relevance. But now with internet information flowing at lightning speed, blogs and texting there is no possible way that 1000 people called can represent over 3000,000,000 people. Why don't you rely on internet polls? You would get tons more responses. CNN had online polls during the debates but totally ignored the ones that showed Pres. Obama did way better on most questions and instead went with all of the pro-Romney polls. Why? The people spoke in November and they rejected old fashioned polls, mindless spin by pundits on TV and the many lies bombarding them in expensive negative ads. The only polls that mean anything are the ones on election day where average Americans vote.

    June 4, 2013 04:33 pm at 4:33 pm |
  22. just sayin

    Rudy NYC
    Please explain how that works. Please, explain how paying your taxes affects your participation at the voting booth. Please, explain why the IRS should not take a closer look at organizations with 'tea party" in their names?
    ------
    get a clue please. this has to do with tax exempt organizations, not paying ones taxes. these organizations were sidelined for two election cycles and they were organizations that held view points which were not favorable to the obama administration. well if you have no problem with well known political phrases being used to target and deny organizations tax exempt status, then the irs needs to now shift their targeting to groups with the words progress, progressive, black, hispanic, etc. no problems there right?

    There are a number of caucus groups in the House. One among them is the Tea Party Caucus, while another is Congressional Black Caucus. It the IRS was suddenly hit by a large number of groups with the phrase "congressional black" as part of their name, don't think the IRS should take a closer look at the group?
    -–

    are you serious??? no group should be targetted, period, end of story.

    June 4, 2013 04:38 pm at 4:38 pm |
  23. California Gary

    I think Gallup, as well as some other pollsters, used their "tried and true" old models for an election in which the dynamics had changed completely. Failing to recognize the electorate had changed dramatically doomed their polls right from the start. They need to accept the fact that it is no longer "business as usual" if they hope to be more accurate in the future.

    June 4, 2013 04:38 pm at 4:38 pm |
  24. don in albuquerque

    @Donna---What is even more curious is that you cling to "facts" which so far have proven to be nothing more then opinions not based on hard evidence. As an old cop, one of the first things you learn about "investigation" is that "you go where the investigation leads you", even if you personally don't like it. So far the investigation seems to go more to the Right, then the Left, too include evidence tampering, but the Repubs refuse to go there. Apparently the Right has no use for the truth, or hard evidence.

    June 4, 2013 04:47 pm at 4:47 pm |
  25. Tom

    @The Real Tom Paine,
    They admitted they were wrong. I don't care about Republicans. I voted for Ron Paul. The fact is that it is clear to anyone that is not blinded by the D vs. R thing that they clearly targeted one political group. They have admitted this. I am for freedom. I am not for stifling one based on the fact they have a view different from mine.
    I can think for myself, recognize that both parties are bad and only out for their own power/money and am not beholden to one side.
    You're right, the people on the far right wing do contribute very little to the conversation. And I am able to have the perspective that people on the far left wing contibute very little to the conversation, as well.
    Sniffit is one of those people that you have identified as contrbuting very little to the conversation.
    Whether you are from the left wing or the right wing, if you are consistent then you will agree.
    But you also are covered in partisanship because you talk about Fox but don't talk about MSNBC.
    Gain some perspective, think for yourself and recognize that similar to Bush, you have a very crooked man in the oval office right now.

    June 4, 2013 04:53 pm at 4:53 pm |
1 2 3 4