Rapid Poll Movement is a General Election Myth
by Chris Bowers, Sat Sep 18, 2004 at 12:05:04 AM EDT
This week, the CBS poll, conducted from September 12th until September 16th, showed the exact same overall result: Bush leading Kerry 50-42 among registered voters in a two-way trial heat. On the surface, this may not seem like good news for Kerry. It may even appear as though Bush's convention bounce is sticking. However, this poll was actually very good for Kerry and, as we shall see, highly unusual because it showed real movement. While Bush leads Kerry by exactly the same 50-42 margin among registered voters, among all three different Party ID groups, Kerry actually improved his position. Among Republicans, Bush fell from 90 to 87. Among Democrats, Kerry moved up to 83 from 80. Among independents, Kerry went from nine points down to being down by only a single point, 43-42. While the overall results look exactly the same as those form last week, CBS admits to having a potentially spoiled sample:
In most CBS News Polls, Democrats outnumber Republicans among registered voters. Personal identification with a party can change (temporarily or permanently) with events and voting preferences. In many states voters do not register with a party, and individual identification is a matter of choice. In this poll, where the Republicans hold a significant lead in voter preference and more voters hold negative views about Democratic nominee John Kerry, when voters are asked about their partisan identification at the end of the questionnaire, more identify themselves as Republicans. 36 percent say they are Republican, 32 percent Democrats. The percentage that identifies themselves as Democrats in this poll is lower than it has been in CBS News Polls conducted earlier in the year. In this case, it is impossible to determine a trend in the CBS poll simply by looking at the top-sheet results. The overall results of the two polls are exactly the same, even though Kerry gained among all three Party ID groups. Had the later poll been weighted according to either 1996 or 2000 Party ID turnout, Kerry would have completely erased Bush's deficit. If the two polls had the same Party ID, not matter what that Party ID was Kerry would have gained considerably from the first poll to the second. What happened was that the 33.5% Democrat, 32.1% Republican, 34.5% Independent Party ID sample changed to a 36% Republican, 32% Democrat, 32% Independent Party ID sample. Kerry's gains within the Party ID groups were wiped out by a shift of the relative importance of the Party ID groups within the poll as a whole.In the block quote above, CBS openly admitted that the September 12-16 poll might be inaccurate because of its Party ID structure. However, as a means of defending their current results, CBS also claimed that "personal identification with a party can change (temporarily or permanently) with events and voting preferences." Considering this, it would be useful to examine just how valid that proposition is.
Two polling firms, Harris and Pew, have conducted Party ID surveys over a span of thirty-five and seventeen years respectively. Here are Pew's results:
Pew's 2004 survey was conducted in June and July, and contained over 19,000 people in their sample (under 0.5 MoE). Although I do not know for certain, it is reasonable to assume that their previous surveys were conducted among similar sample sizes. Since 1990, Republican self-identifiers have varied between 27 and 30 percent of the population, except once in the immediate post-9/11 survey when they rose to 31%. Democratic self-identifiers have varied between 30 and 34% of the population, except once, immediately before 9/11, when it rose to 35%. In eight of the sixteen surveys since 1990, self-identifying Democrats made up 33% of the population. This chart does not just show slow movement in Party ID--it shows glacial movement.
Here are the Harris results:
Year Rep Dem Gap 2003 28 33 5 2002 31 34 3 2001 29 36 5 2000 29 37 8 1999 29 36 7 1998 28 37 9 1997 29 37 8 1996 30 38 8 1995 31 36 5 1994 32 37 5 1993 29 38 9 1992 30 36 6 1991 32 37 5 1990 33 38 5In 2003, this poll included more than 6,000 registered voters, and thus has a slightly higher Margin of Error than Pew, but the MoE is still under plus or minus one. According to Harris, since 1990 Republicans have varied from between 28 and 33% of the population, while Democrats varied from between 36 and 38% of the population before dropping to 34 and 33 the last two years respectively. In those two years, Republicans also dropped while Independents / Not Affiliated / Other saw an increase. Further, with the exception of 2002, the Democratic edge in Party ID ranged between 5 and 9 points--a mere four-point swing. Overall, this chart also shows extremely gradual movement, never once in a year seeing the same kind of movement CBS recorded in a single week. Considering this, there is simply no circumstance under which the shift recorded by CBS, a poll with a Margin of Error of plus or minus, 3.5 can be understood as accurate.
Party ID is indeed a variable rather than an absolute constant. However, even over periods of years in surveys with miniscule margins of error, it is a slow moving, incrementally shifting variable. Considering this, from the perspective of week to week polling in the final two months of a Presidential election, in order for such polling to truly gauge the state of the campaign, it should be treated as a constant. Given this, it is my position that polls that weight their results according to a fixed Party ID model offer a more accurate picture of the campaign than those that do not weight their results by Party ID (or other demographics).
* * * * *
In a forthcoming paper concerning Gallup's 2000 tracking poll to be published in Public Opinion Quarterly, Robert Erikson, Costas Panagopoulos and Christopher Wlezien put forth the following thesis:
[E]stimates of who may be likely voters in the weeks and months prior to Election Day in large part reflect transient political interest on the day of the poll, which might have little bearing on voter interests on the day of the election. Likely voters early in the campaign do not necessarily represent likely voters on Election Day. Early likely voter samples might well represent the pool of potential voters sufficiently excited to vote if a snap election were to be called on the day of the poll. But these are not necessarily the same people motivated to vote on Election Day..... [S]hifts in voter classification as likely or unlikely account for more observed change in the preferences of likely voters than do actual changes in voters' candidate preferences While I do not have access to the entire paper, I find it quite interesting that changes within the demographic composition of the likely voter sample account for a greater proportion of Gallup's day-to-say tracking variation than do actual changes in voter preferences. Based on my research, I am prepared (at least blog-prepared) to submit a similar thesis. Differences in overall trial heat results between polling firms, and even between polls conducted by a single polling firm that does not weight by Party ID, are more a result of differences in the Party ID composition of the polls than in actual changes in voter preference. Further, such differences present a false picture of the election since significant shifts in Party ID among the electorate do not take place.Including the CBS poll, I have gathered together Party ID data from twelve recent polls. In six of the polls, the Economist, Harris, ICR, Pew, Rasmussen and Zogby, the samples were weighted to fit demographic and / or previous turnout models. The results of these six polls poll as follows:
Table One * = likely voters Bush Kerry Date Econ 47 46 9/15 Harris* 47 48 9/13 ICR 48 44 9/12 Pew 46 46 9/14 Rasm* 49 45 9/16 Zogby* 47 45 9/9The similarity between the results in these six polls is remarkable. The race varies from Bush up four to Kerry up one, with no two polls disagreeing about Bush's raw score by more than three points or Kerry's raw score by more than four points. On average, Bush leads by less than two points (47.3-45.5).
For the sake of comparison, I have also been able to track down the internals of six recent polls that do not weight their results according to Party ID or other demographics: ABC, CBS, Fox, Gallup, IBD / CSM and Newsweek. Here is how these six compare to each other:
Table Two
* = likely voters
Bush Kerry Date
ABC 50 44 9/8
CBS 50 42 9/16
Fox* 47 45 9/8
Gallup 52 44 9/14
IDB 44 46 9/12
News 49 43 9/10
These polls show significantly more variance than the other six. But what is perhaps even more remarkable is the variance they show among themselves:
Table Three
* = likely voters
Bush Kerry Date Shift
ABC 48 47 8/29 5
CBS 44 47 8/18 11
Fox* 44 45 8/25 3
Gallup 49 48 9/5 7
IDB 44 44 8/23 2
News 54 43 9/3 5
With the exception of Fox and IBD / CSM, which are the outliers among the most recent set of polls, the shift from poll to poll within each polling firm has been enormous. Compare this to the shifts experienced by the six polls that weight their results:
Table Four
* = likely voters
Bush Kerry Date Shift
Econ 46 45 9/8 0
Harris* 47 47 8/15 1
ICR 46 47 9/5 5
Pew 52 40 9/10 12
Rasm* 48 46 9/9 2
Zogby* 46 44 9/2 0
With the exception of Pew, which showed an enormous shift, there has been virtually no movement from poll to poll within polling firms that conduct sample weighting. In four of out six cases, the shift in the gap between Bush and Kerry was two points or less. Among the unweighted polls, in four our of six cases, there was a shift of five points or more. In fact, among the unweighted polls, two points was the smallest shift.This provides evidence to support both aspects of my hypothesis. First, polls that weight are more similar to one another than polls that do not weight. Second, polling firms that weight show less movement from poll to poll than polling firms that do not weight. The reason for the difference between the two groups of polls is that Party ID varies in one set, but not the other.
Interestingly, had Party been weighted in the six most recent unweighted polls, they would look almost exactly like the six recent weighted polls:
Table Five
* = likely voters
Bush Kerry Date
ABC 48 47 9/8
CBS 47 46 9/16
Fox* 46 47 9/8
Gallup 48 48 9/14
IDB 47 47 9/12
News 46 47 9/10
Yikes!* * * * *
People typically dismiss Rasmussen because of their horrible results in 2000 (easily the least accurate pollster). However, Scott Rasmussen seems to have learned from his mistakes, and now he weights by Party ID, something that he did not do in 2000. Because they have done a daily tracking poll for eight months now, Rasmussen is easily the polling outfit with the most polling data on this election. Despite this, they have shown little movement in the election during the entire eight months of their tracking poll:
Bush vs. Kerry, Rasmussen Tracking, January 19-July 15
This graph only runs from mid-January until mid-July, but even if the latest results were incorporated, they would look much the same (with a slight Bush uptick over the past three weeks). Because Rasmussen weights by Party ID, no matter what cataclysmic events have occurred in the election that have affected other polls, they have shown occasional blips only briefly and slightly altering a long-term deadlock. Now, compare this to polls that do not weight by Party ID during the same period:
Bush vs. Kerry, all polls, January 19-July 15
The Economist and ICR started polling post-July 15. Harris and Pew poll very infrequently. Thus, among polls that weight, only Zogby plays a significant role in the second graph, which shows far more variation than the first graph. I do not have all of the internals to prove this conclusively, but I am pretty damn sure the above variations are more due to Party ID than anything else.
Party ID does change over time, but slowly, even glacially. For national results, right now I trust Rasmussen (along with the Economist, Harris, ICR, Pew and Zogby) as much as I trust any other poll in the field. By weighting Party ID, these six polls limit randomness, maintain a consistent sample, and prevent erroneous polls that, for the past several months, have been unfortunately reported as evidence of a "bounce" for one candidate or the other. There have been no significant bounces for either candidate this entire election. For four months, Kerry was on a very slow upward slope. For the past month, Bush has been on a slow upward slope. Recent polls suggest that Bush's rise has stalled, and even reversed. However, whatever is happening, it is all happening very, very slowly. Rapid poll movement is a mirage in the desert of the horserace generated by poor polling methodology. On November 2nd, I could end up getting burned, but I have cast my lot with the polling firms that weight their samples. My new methodology in projecting the popular vote in the Presidential election is almost entirely based on the theory I have put forth in this post. It won't be long before we find out who is right.
This is going to be a very close leection. GOTV
Tags: General 2008 (all tags)
29 Comments