Research 2000 polling questioned

Markos Moulitsas fired Research 2000 as the pollster retained by Daily Kos a few weeks ago after R2K fared poorly in "pollster ratings" compiled by FiveThirtyEight's Nate Silver. At the time I wondered whether Markos reacted a bit harshly, since Silver himself admitted, "The absolute difference in the pollster ratings is not very great." In addition, some polling experts had raised questions about Silver's rating system (see also here).

Today Markos published a remarkable analysis of "problems in plain sight" with Research 2000's polling. Three researchers uncovered "extreme anomalies" in certain results and concluded, "We do not know exactly how the weekly R2K results were created, but we are confident they could not accurately describe random polls." You should click over and read the whole thing, but here are the anomalies in question:

1. A large set of number pairs which should be independent of each other in detail, yet almost always are either both even or both odd.

2. A set of polls on separate groups which track each other far too closely, given the statistical uncertainties.

3. The collection of week-to-week changes, in which one particular small change (zero) occurs far too rarely. This test is particularly valuable because the reports exhibit a property known to show up when people try to make up random sequences.

Markos has renounced "any post we've written based exclusively on Research 2000 polling" and asked polling sites to "remove any Research 2000 polls commissioned by us from their databases."

Based on the report of the statisticians, it's clear that we did not get what we paid for. We were defrauded by Research 2000, and while we don't know if some or all of the data was fabricated or manipulated beyond recognition, we know we can't trust it. Meanwhile, Research 2000 has refused to offer any explanation.

Last year the Strategic Vision polling firm was brought down by convincing allegations that at least some of its polling results had been fabricated. Research 2000 had a much better reputation than Strategic Vision, though. Markos listed some of the news organizations that have commissioned R2K polls. I am seeking comment from KCCI-TV (the CBS affiliate in Des Moines), which has used R2K in the past. I will update this post if I hear back about their future plans regarding commissioned polls.

Share any relevant thoughts in this thread.

Tags: polls, Research 2000, Strategic Vision (all tags)

Comments

6 Comments

Yet another bogus polling firm gets caught

R2K could easily refute these claims by ponying up the relevant data and processes they used to arrive at their poll results. That they are unwilling to do so, speaks volumes about their level of mendacity. KOS gave them several opportunities over several weeks to answer the charges and come clean or show their methodology to refute his accusations. Their silence says they cannot do that and/or will not do it. In either case, it makes me believe KOS and distrust R2K. Markos is right to file a lawsuit. We cannot afford to permit fraudulent polling companies to foist themselves on the public as honest, trustworthy pollsters if in fact they are just making Sh*t up as they go along, or worse, are creating polls to fit with their agenda and political perspective. Kudos to Markos and Daily KOS!

by mcarnes 2010-06-29 04:11PM | 1 recs
I doubt it

I really doubt this is fraud.  R2K has been around for a long time, and it seems inconceivable to me that they would risk their reputation in the manner described. 

The article that alledges fraud compares R2K to Gallup, but Gallup is laughable in their swings.  I wrote about this in '08 at Openleft. The Gallup voter screens produce large voter swings which are simply indefensible, and have been for a number of cycles (in 2000 they were laughable).  Gallup is hardly the gold standard in polling.

Bottom line, what we know about the electorate is based on data from a few pollsters until September and that should everyone hesitate leaping to conclusions.

by fladem 2010-06-29 04:53PM | 0 recs
R2K

I wrote about this multiple times in '08, in one post Markos even claimed that I was saying that R2K was fraudelent in their polling, which is laughable today in my poking emails.

But basically, all R2K did was cater their numbers to the crowd, and they ate it up over there too. They got what they paid for imo.

by Jerome Armstrong 2010-06-29 05:07PM | 0 recs
Got what they paid for?

KOS paid R2K for real polls, not bogus crap that R2K thought would make them happy. The fact that they failed to provide real, serious, truthful polls indicates just how fraudulent they were.

by mcarnes 2010-06-29 08:46PM | 0 recs
RE: Got what they paid for?

But did they make them happy?

by Jerome Armstrong 2010-06-29 08:55PM | 1 recs
RE: Got what they paid for?

I'm not sure how that is material here?

As Nate Silver pointed out, the margin of error for R2k was not that great. So the argument that R2k was kept on for too long because they were providing friendly results doesn't hold much merit -- the results weren't that friendly.

This appears to be a case of fraud not lean.

by NoFortunateSon 2010-06-29 10:09PM | 0 recs

Diaries

Advertise Blogads