The Importance of the Question

A number of people have wondered why the numbers from Rasmussen Reports look different from polling conducted by other outlets, specifically why Barack Obama's approval rating in their daily tracking poll comes in lower than most other surveys. Central to the answer of this question is the fact that Rasmussen surveys only those it believes to be "likely voters" -- a value judgment, particularly this far away from the 2010 and 2012 elections -- and, additionally, that Rasmussen is using recorded interviews rather than live interviews. But that's not all.

As Pollster.com noted last week, and as I have been meaning to write about for some time (but have been waylaid with preparing for finals), the question being asked in the survey matter, too. Rasmussen was good enough to run three simultaneous polls last month asking three variations of approval rating questions: (1) Asking respondents whether they approve or disapprove; (2) Giving respondents four choices rather than two, asking them if they strongly approve, somewhat approve, somewhat disapprove or strongly disapprove; and (3) Giving respondents four choices, but four unbalanced ones, asking respondents if they think the President is doing an excellent, good, fair or poor job (unweighted because it lumps the ambivalent answer of "fair" into the disapprove category). For reference, Rasmussen traditionally uses question (2). Here were the numbers they found:

The differences are quite apparent: While the traditional question yields numbers that look a lot like the national trend, the question normally used by Rasmussen yields numbers that are flipped on their head. The third question, used by some pollsters like Harris, yield even more divergent numbers, data that look nothing like other polling.

Why does this matter? Many aggregating the polls are lumping together every survey regardless of the question being asked. While this is effective in gauging the overall trend -- which, don't get me wrong, is extremely important -- it obscures the picture of where the President currently stands by including with the traditional approval/disapproval question survey questions that tend to yield much lower "approval" ratings. In other words, it lumps in three different things as if they were the same. It would be like putting in a basket oranges, mandarins and pummelos but simply calling them all "oranges" when it used to be that you only placed oranges (but not mandarins or pummelos) in the basket.

So it's something to consider when looking at Rasmussen surveys, as well as trend estimates of the President's standing among the American people.

Tags: Approval Rating, Barack Obama, Rasmussen Reports (all tags)

Comments

10 Comments

Thanks for this post

Disapprove includes "Fair?". Wow, that's harsh.

Sadly, I've seen Ras numbers touted in "progressive" establishments, usually when someone has an axe to grind with Obama.

by NoFortunateSon 2009-12-10 09:38AM | 0 recs
NotFortunateSon nails the fatal flaw...

...in the 3rd type of "approval" poll.

The meaning of "fair" is in the eye of the beholder, and means very different things to different people.  And that renders it useless.

I always disregard any poll that includes "fair" as a choice for respondents to measure their sentiment.  In the case of Obama, "fair" respondents include both people who think he's doing a good job and people who think he's doing a bad job.  Thus, as I said above, it's useless.

by DCCyclone 2009-12-11 04:24AM | 0 recs
Can we see the actual numbers?

None given in the post and no link provided.

by Davidsfr 2009-12-10 10:00AM | 0 recs
Re: Can we see the actual numbers?

Got them from the Pollster.com post, but here's the topline if you want.

by Jonathan Singer 2009-12-10 10:15AM | 0 recs
Re: The Importance of the Question

Check the Pollster.com link. I recall that Rasmussen was an outlier during the Bush admin as well (going the other direction) I assume that they are biased towards repubs.

by MichiganMark 2009-12-10 10:16AM | 0 recs
Re: The Importance of the Question

Yes, they regularly showed Bush with a 30+ approval rating when the others had him in the 20s...

However they were reasonably accurate in 2004. Personally I think they are reasonably accurate if the GOP base is energized and enthusiastic to vote. Which might be the case in 2010.

by vecky 2009-12-10 11:22AM | 0 recs
Rasmussen does better close to election day......

Their pre-election polling the last month or two before an election is usually pretty good.

But that also coincides with Rasmussen's results simply lining up with other polling outfits' results.  Rasmussen is no better than them, just on par.

Further from an election, or any time on issue polling and other polling, Rasmussen is a consistent outlier.  That happens for multiple reasons, but to be blunt I don't doubt for a second that it's deliberate.  Scott Rasmssen is a right-wing Christian fundamentalist, and some of his polling techniques, particularly this year, are transparently biased as well as stupid, ranging from his "presidential approval index" that he highlights over topline job approval, to his fictitious "political class" of respondents who he segregates from other poll respondents.  Also transparently bad is that Rasmussen's election polls include internals that show candidate known to have extremely low name recognition producing favorable/unfavorable numbers that show very few "no opinions." For just one of many examples, Rasmussen's last NC-Sen 2010 poll showed two-thirds of respondents having either a favorable or unfavorable opinion of lawyer Ken Lewis, with only one-third of respondents claiming to have no opinion of him, even though he has never run for anything in his life and has virtually zero name recognition.  You see this repeated over and over again.

Not to mention I've never seen exactly what's Rasmussen's likely voter screen, beyond a vague description that it includes, perhaps among other things, questions in recent past voting history.  Who is a "likely voter" when it comes to Obama's job approval?  Simply people who say they actually voted last year?  Or people planning to vote in the midterms?  Or people planning to vote in 2012?

It's well-placed that virtually no one but Rasmussen uses a likely voter screen far from an election.  Such a screen just doesn't make any sense.

by DCCyclone 2009-12-11 04:34AM | 0 recs
Re: Rasmussen does better close to election day

I think a likely voter screen far from an election makes sense.  If you want to show trends over time, why would you suddenly switch your sample space?

by Anthony de Jesus 2009-12-11 07:34AM | 0 recs
Re: Rasmussen does better

The fact that the outgoing CEO of North Carolina's largest company (Bank of America) is also named "Ken Lewis" might have something to do with a lot of voters thinking they recognize his name.

by Steve M 2009-12-11 02:36PM | 0 recs
Re: The Importance of the Question

As noted in the link you mentioned, Rasmussen uses a likely voter screen.

In terms of results, I believe Rasmussen tends to be more accurate than other groups.  Nate Silver, for example, considered them one of the better pollsters during the 2008 primaries.

by Anthony de Jesus 2009-12-10 07:31PM | 0 recs

Diaries

Advertise Blogads