Research 2000 polling questioned

Markos Moulitsas fired Research 2000 as the pollster retained by Daily Kos a few weeks ago after R2K fared poorly in "pollster ratings" compiled by FiveThirtyEight's Nate Silver. At the time I wondered whether Markos reacted a bit harshly, since Silver himself admitted, "The absolute difference in the pollster ratings is not very great." In addition, some polling experts had raised questions about Silver's rating system (see also here).

Today Markos published a remarkable analysis of "problems in plain sight" with Research 2000's polling. Three researchers uncovered "extreme anomalies" in certain results and concluded, "We do not know exactly how the weekly R2K results were created, but we are confident they could not accurately describe random polls." You should click over and read the whole thing, but here are the anomalies in question:

1. A large set of number pairs which should be independent of each other in detail, yet almost always are either both even or both odd.

2. A set of polls on separate groups which track each other far too closely, given the statistical uncertainties.

3. The collection of week-to-week changes, in which one particular small change (zero) occurs far too rarely. This test is particularly valuable because the reports exhibit a property known to show up when people try to make up random sequences.

Markos has renounced "any post we've written based exclusively on Research 2000 polling" and asked polling sites to "remove any Research 2000 polls commissioned by us from their databases."

Based on the report of the statisticians, it's clear that we did not get what we paid for. We were defrauded by Research 2000, and while we don't know if some or all of the data was fabricated or manipulated beyond recognition, we know we can't trust it. Meanwhile, Research 2000 has refused to offer any explanation.

Last year the Strategic Vision polling firm was brought down by convincing allegations that at least some of its polling results had been fabricated. Research 2000 had a much better reputation than Strategic Vision, though. Markos listed some of the news organizations that have commissioned R2K polls. I am seeking comment from KCCI-TV (the CBS affiliate in Des Moines), which has used R2K in the past. I will update this post if I hear back about their future plans regarding commissioned polls.

Share any relevant thoughts in this thread.

Has Strategic Vision been faking polls?

I've never paid much attention to Strategic Vision polls, because I assumed that a Republican PR firm would produce results slanted toward Republican candidates. Even so, it didn't occur to me that Strategic Vision might be fabricating polls.

The American Association for Public Opinion Research has criticized Strategic Vision for refusing "to release essential facts about polls it published prior to the 2008 presidential primaries in New Hampshire and Wisconsin."

Strategic Vision also refuses to release cross-tabs for any of its polls and doesn't seem to have any physical offices. (The addresses listed on its website are UPS offices.)

Nate Silver of FiveThirtyEight.com came right out and said it (emphasis in original):

One of the questions, in light of Strategic Vision LLC's repeated failure to disclose even basic details about its polling methodology, is whether the firm is in fact conducting polling at all, or rather, is creating fake but plausible-looking results in order to increase traffic and attention to its core business as a PR and literary firm.

I posed that question largely as a hypothetical yesterday. But today, I pose it much more literally. Certain statistical properties of the results reported by Strategic Vision, LLC suggest, perhaps strongly, the possibility of fraud, although they certainly do not prove it and further investigation will be required.

Silver looked at results for all questions in more than 100 Strategic Vision polls and found that some numbers were much more likely to be the final digit than other numbers. That is, the percentage of respondents answering any given question a certain way was much more likely to end in a 7, 8 or 9 than a 1 or 2.

Even without considering Silver's "trailing digit" analysis, Mark Blumenthal of Pollster.com finds enough conflicting statements and troubling facts to question whether Strategic Vision's polls "deserve our trust."

Ben Smith's piece for Politico on this controversy is also worth reading.

Whether or not Strategic Vision has been faking its polls, it's disturbing that so little is known about their methodology. Blumenthal is right to call for scoring "the quality of disclosure of every public survey release" and "publish[ing] the scores alongside polling results."

If you've been a respondent for any Strategic Vision poll during the last four years, Nate Silver wants to hear from you. For future reference, always ask who's conducting the survey if you agree to participate in any telephone poll.

Update [2009-9-28 10:50:12 by desmoinesdem]: Nate Silver followed up here, comparing patterns in Strategic Vision poll findings to those from Quinnipac.

There's more...

And it's a tie -- again!

A little break from the sad news from Pakistan today.

A new Strategic Vision poll for the Iowa Caucus rebuts the "Hillary might be pulling away" narrative the media has been pushing the past few days due to the outlier Christmas ARG poll.  It shows Obama with 30%, Clinton with 29%, and Edwards with 28%.  In other words, it's a three-way tie.

It's beginning to look more and more like this thing will go down to the wire.  I think I'm going to have to find someone to babysit my kids on the 3rd so I can follow the returns since I suspect it's going to be a late night even here in California.

As an Edwards supporter, I'm heartened to see that despite things remaining tied, Edwards support keeps creeping up.  I'm betting that he pops that 30% ceiling very, very soon.

Yikes, what a nail-biter.

There's more...

2008 Poll digest: Nov 15, 2006

This is a "digest" of 2008 polls that I have come across since Obama entered the landscape.

The general trend is that, Obama has essentially risen to the number two spot for the nomination in national polls, but in strategic vision state polls (covering some key states), Gore is in a strong second place, somewhat ahead of Obama. Edwards is hovering around 10% in most polls, and hence remains a factor.

There's more...

Sorry, Bush and Rove, fear has declined since September

Are you scared?

Pollsters don't ask that basic question. Actually, they do, with slightly more finesse.

I realize we don't like Strategic Vision (R) around here, but they've had one question that has fascinated me all year:

"Do you expect another terrorist attack within the United States in the next six months?"

There's more...

Diaries

Advertise Blogads