Has Strategic Vision been faking polls?
by desmoinesdem, Sun Sep 27, 2009 at 12:05:17 AM EDT
I've never paid much attention to Strategic Vision polls, because I assumed that a Republican PR firm would produce results slanted toward Republican candidates. Even so, it didn't occur to me that Strategic Vision might be fabricating polls.
The American Association for Public Opinion Research has criticized Strategic Vision for refusing "to release essential facts about polls it published prior to the 2008 presidential primaries in New Hampshire and Wisconsin."
Strategic Vision also refuses to release cross-tabs for any of its polls and doesn't seem to have any physical offices. (The addresses listed on its website are UPS offices.)
Nate Silver of FiveThirtyEight.com came right out and said it (emphasis in original):
One of the questions, in light of Strategic Vision LLC's repeated failure to disclose even basic details about its polling methodology, is whether the firm is in fact conducting polling at all, or rather, is creating fake but plausible-looking results in order to increase traffic and attention to its core business as a PR and literary firm.
I posed that question largely as a hypothetical yesterday. But today, I pose it much more literally. Certain statistical properties of the results reported by Strategic Vision, LLC suggest, perhaps strongly, the possibility of fraud, although they certainly do not prove it and further investigation will be required.
Silver looked at results for all questions in more than 100 Strategic Vision polls and found that some numbers were much more likely to be the final digit than other numbers. That is, the percentage of respondents answering any given question a certain way was much more likely to end in a 7, 8 or 9 than a 1 or 2.
Even without considering Silver's "trailing digit" analysis, Mark Blumenthal of Pollster.com finds enough conflicting statements and troubling facts to question whether Strategic Vision's polls "deserve our trust."
Ben Smith's piece for Politico on this controversy is also worth reading.
Whether or not Strategic Vision has been faking its polls, it's disturbing that so little is known about their methodology. Blumenthal is right to call for scoring "the quality of disclosure of every public survey release" and "publish[ing] the scores alongside polling results."
If you've been a respondent for any Strategic Vision poll during the last four years, Nate Silver wants to hear from you. For future reference, always ask who's conducting the survey if you agree to participate in any telephone poll.Update [2009-9-28 10:50:12 by desmoinesdem]: Nate Silver followed up here, comparing patterns in Strategic Vision poll findings to those from Quinnipac.