Has Strategic Vision been faking polls?

I've never paid much attention to Strategic Vision polls, because I assumed that a Republican PR firm would produce results slanted toward Republican candidates. Even so, it didn't occur to me that Strategic Vision might be fabricating polls.

The American Association for Public Opinion Research has criticized Strategic Vision for refusing "to release essential facts about polls it published prior to the 2008 presidential primaries in New Hampshire and Wisconsin."

Strategic Vision also refuses to release cross-tabs for any of its polls and doesn't seem to have any physical offices. (The addresses listed on its website are UPS offices.)

Nate Silver of FiveThirtyEight.com came right out and said it (emphasis in original):

One of the questions, in light of Strategic Vision LLC's repeated failure to disclose even basic details about its polling methodology, is whether the firm is in fact conducting polling at all, or rather, is creating fake but plausible-looking results in order to increase traffic and attention to its core business as a PR and literary firm.

I posed that question largely as a hypothetical yesterday. But today, I pose it much more literally. Certain statistical properties of the results reported by Strategic Vision, LLC suggest, perhaps strongly, the possibility of fraud, although they certainly do not prove it and further investigation will be required.

Silver looked at results for all questions in more than 100 Strategic Vision polls and found that some numbers were much more likely to be the final digit than other numbers. That is, the percentage of respondents answering any given question a certain way was much more likely to end in a 7, 8 or 9 than a 1 or 2.

Even without considering Silver's "trailing digit" analysis, Mark Blumenthal of Pollster.com finds enough conflicting statements and troubling facts to question whether Strategic Vision's polls "deserve our trust."

Ben Smith's piece for Politico on this controversy is also worth reading.

Whether or not Strategic Vision has been faking its polls, it's disturbing that so little is known about their methodology. Blumenthal is right to call for scoring "the quality of disclosure of every public survey release" and "publish[ing] the scores alongside polling results."

If you've been a respondent for any Strategic Vision poll during the last four years, Nate Silver wants to hear from you. For future reference, always ask who's conducting the survey if you agree to participate in any telephone poll.

Update [2009-9-28 10:50:12 by desmoinesdem]: Nate Silver followed up here, comparing patterns in Strategic Vision poll findings to those from Quinnipac.

Tags: Fraud, polls, Strategic Vision (all tags)

Comments

5 Comments

Re: Has Strategic Vision been faking polls?

Wow, if anyone wondered what sort of clout Nate Silver has earned himself in the past year, this story is it... it began in the blogosphere, but has been picked up by a whole lot of the MSM, including the Atlanta Constitution-Journal.

You know you've made it bigtime as a political blogger when you publish work that has the power to destroy a polling company.

But it looks like Strategic Vision isn't gonna go down without a fight... The Hill is reporting that the firm's CEO is seriously considering filing a lawsuit against Nate.

I hope Silver buries these clowns.

by Obamaphile 2009-09-27 08:57AM | 0 recs
Re: Has Strategic Vision been faking polls?

One of the weirder things about this is that Nate's accusations are so basic and crazy that making them go away would be almost effortless. Just release information-- any information-- that would demonstrate that they have, at some point in the past, conducted a poll. For some reason they don't want to do this?

by mcc 2009-09-27 10:26AM | 0 recs
statistical thoughts

Nate's analysis is certainly interesting, but I'd be careful.  If the data are random, then the trailing digits follow a multinomial distribution of dimension 10 (0...9), with a parameter vector consisting of 10 values of .1 each and n=5544 (the size of the "sample" of trailing digits he shows).  I assume Nate made this assumption and did some simulation.

I just did this 1 million times and found that the 95% "confidence interval" for ANY trailing digit to occur is [493,526].  The data that Nate presented had a minimum of 431 and a maximum of 676, so it lends credence to his falsification argument.  

HOWEVER, if the multinomial parameters deviate from .1, i.e., some digits are simply more likely to occur than others because polling data aren't simply random, it wouldn't take much to make their data plausible.  

An appropriate analysis would involve putting a Dirichlet prior over the multinomial parameter vector (perhaps using Nate's own data as a prior), incorporating the data, and examining the posterior distribution for the Dirichlet parameters.  Then, would could look at how much those parameters would have to deviate from .1 each to obtain the data.  If .1 were within the credible interval for each of the Dirichlet parameters, then the data could've happened via chance.  I'm guessing, given that Nate's own data are not uniform, such an analysis would show that their data are plausible. If I have time today, I'll write a diary on this, but, unfortunately, I have a test to write as well as an overdue book chapter!

by slynch 2009-09-27 11:33AM | 0 recs
Re: statistical thoughts
Without diving too deeply into the subject, I believe the analysis being applied is based on Benford's Law. ( http://en.wikipedia.org/wiki/Benford's_l aw ) ( http://mathworld.wolfram.com/BenfordsLaw .html ) Benford's Law is based on the observation that data collected from the "real world" tends to have a bias in digits. One might assume that so-called random data, like say the square acreage of lakes in Michigan, should have as many leading '9's as '8's as '7's, so-on and so-forth (every digit should have a probability of 0.1, right?). This, however, is not the case (see web pages, google).
Benford's Law has appeared in several popular press articles over the past few years, one I recall was in the New York Times and detailed the use of the analysis to catch tax cheats. It turns out people just aren't good at making up "random" numbers!
d
by dentearl 2009-09-28 09:35AM | 0 recs
Re: statistical thoughts

right--Benford's law says many natural phenomena have trailing digits that follow, essentially, an exponential decline in their frequency of occurrence (from 0 to 9).  So, 1 occurs quite frequently, followed by 2 a little less often, and so on.

But, Benford's law doesn't apply to everything, including, potentially, polling percentages.  Clearly, the SV polling data doesn't follow this pattern--it's peak in trailing digits is around 7/8.  But 538's data doesn't follow Benford's law either.  Nate has an argument for why, but I thought it was somewhat ad hoc.  And, comparing the patterns of trailing digit frequency across the two sets of data, I sure wouldn't hang my hat on a fraud accusation that could end up as a lawsuit.

by slynch 2009-09-28 11:09AM | 0 recs

Diaries

Advertise Blogads