I live in a neighborhood in Philadelphia called University City. This past week, most of the students who live in the area have moved out, as the semester is over. After yesterday, I kind of feel like our little investigation into national Democratic nomination polling came to an end. Like a semester, it lasted around four months, and I think was quite informative. Here are the conclusions I have drawn from our studies:

1. National Democratic nomination preference polls include too wide a net of people in their sample, typically somewhere between 35%-50% of the Voting Age Population. Typically, outside of New Hampshire, only about 10-15% of the Voting Age Population participates in Democratic Presidential primaries. However, at this early date, it would not be wise to significantly narrow the sample universe, as it is too early to know who will actually form the electorate in the Democratic primary / caucus season. That might change, come January 2008.
   
   
2. Some early indications of voter turnout favor Clinton and Edwards, while others favor Obama. Specifically, Clinton and Edwards do well among older poll respondents and Clinton does better among self-identified Democrats than among Independents who lean Democratic. However, Obama does better among poll respondents who are paying more attention to the campaign. When averaged together, these effects might very well cancel each other out / compliment each other.
   
   
3. Clinton does better in polls where undecided respondents are pushed to make a decision, thus emphasizing her advantage among voters who are not paying close attention to the campaign. However, Obama does better in automated IVR polls like Rasmussen that have a history of including more young voters in their samples. Once again, when combined, these skews might cancel each other out / compliment each other.
   
   
4. Al Gore draws a significant percentage of support (roughly 10-15%) from all three "top tier" candidates) simply by being included in the question. This usefully shows, once again, that there is a significant amount of "soft" support for all candidates. However, Al Gore is also currently not running, thus making it quite difficult to justify including him in polls that are meant to be an accurate snapshot of public opinion on the current campaign. The solution here is probably for polls to ask "someone else" as an option for respondents, rather than to name specific candidates who have not announced. Overall, until polls settle on a consistent list of candidates to include in their questions, it will be necessary to collect two different polling averages, one with Gore, and one without.
   
   
5. As demonstrated by the soft support of undecideds, the still large number of potential Gore supporters, the varying movement in the national campaign over the past couple of months, and the wide difference in results between different polls conducted at the same time, there is a lot of movement yet to be had in the Democratic primary season. However, it is probably wrong to assume that said movement is on the level of 2004, either to the degree to which early Lieberman "supporters" abandoned him before Iowa throughout 2003, or to the degree that Democrats flocked to Kerry after the 2004 Iowa caucuses. Increased star power in the field, a higher level of voter engagement, increased Democratic satisfaction with the field, and the lack of a 2004 "perfect momentum storm" are among the reasons that will probably reduce poll movement compared to the 2004 primary season.
   
   
6. A few of side notes. First, there does not appear to be a large "anti-Hillary" vote in the Democratic electorate. Second, social pressure to say you are voting for a woman or an African-American does not appear to be artificially inflating either Clinton or Obama's poll numbers. Third, while Clinton performs slightly worse than Edwards or Obama in general election trial heats, the gap is not massive (currently between 2.8% and 6.9% depending on the matchup). While this is not currently indicative of an "electability" problem, and is more indicative of Clinton's longer exposure to the Republican Noise Machine, if these numbers hold, or even increase, through January of 2008, that could change.

In the end, this leaves us roughly where we were back in January: averaging polls. However, I think we now have a much better idea as to why national polls can be so different from each other, and yet all still be valid. It has also left me with a methodology to measure the current state of the national campaign in which I have a decent amount of confidence. Having a way to accurately measure the campaign is an important first step toward developing a means to influence it. To this end, it would be particularly useful if more national polls had larger sample sizes and released detailed crosstabs from within those sample sizes. It is in this way that live-interview polls commissioned by large media outlets, which are invariably have smaller sample sizes and are more hush-hush about their methodologies, remain our least useful measures of the national campaign. However, despite this, we do have some good info now, and as such we can move forward. From now on, my discussions of polls will probably be restricted to updates on the state of the national campaign, and not spill over into meta discussions on polling itself. I hope you got as much out of our polling seminar as I did, and are now excited to moving forward onto other, more qualitative topics.

The seminar's syllabus can be found in the extended entry.

There's more...

Pursuant to my latest Inflated Clinton poll theory post below, it needs to be pointed out that Internet polling conducted by YouGov (formerly Polimetrix), has a perfectly fine track record. I think Zogby's poor performance in 2004 has scared people away from Internet polling in general. However, if one compares the final Polimetrix polls in the thirteen closest Senate races in 2006 (PDF) with the IVR poll averages and live interview poll averages in the final week of those thirteen states (source), you can see that Polimetrix did just fine.

If you don't believe me, just check the numbers in the extended entry. For those not interested in looking at the numbers right now, the basic lesson is this: the three methods (IVR, Polimetrix, and live interview) performed identically well in the thirteen Senate races decided by less than 20%. In terms of overall accuracy, there were four states where Polimetrix beat the live interview average by more than 1%, four states where the live interview average beat Polimetrix by more than 1%, and five states where they were within 1% of each other. This is pretty impressive for Polimetrix, since they are a single polling firm, and they were up against poll averages in every state listed below. Clearly, Polimetrix didn't do that bad. Internet polling is coming along nicely, and Polimetrix (YouGov) is leading the way.

It seems that automated IVR telephone polling (Survey USA, Rasmussen), properly conducted Internet polling (YouGov, maybe Harris and Zogby), and live interview telephone polling (virtually everyone else) are all about as accurate as one another at this point. It makes sense, since the percentage of Americans who have email and the percentage of Americans who have landlines are almost identical right now (about 80% each). However, by the 2008 election, the number of people who have email will exceed the number of people who have landlines, and by 2012, it could be a significant difference. In order to compensate for this, Internet polling needs to by done correctly. So, we should be grateful that YouGov is making such large strides in the field, and not be so quick to dismiss Internet polls.

But what is to be done when IVR polls (Rasmussen), Internet polls (YouGov) and live interview polls (everyone else) contradict each other on individual campaigns, as they appear to be doing in the national Democratic primary? The best answer is probably to just average out the polls, and look for the causes of their difference in other locations. For example, it is entirely possible that Rasmussen, as is the case with many IVR polls, is simply sampling a higher percentage of younger voters, and younger voters tend to be more pro-Obama. In the case of YouGov, the difference is probably connected to the prominent "Undecided" option on their questionnaire, something that most live interview polls lack. This actually works quite well with the original Inflated Clinton Poll Theory. There are probably a lot of people right now who are not paying much attention, and as such don't know much about Obama or other less well known candidates. Thus, these people probably lean Clinton, but will only choose Clinton when pushed. Makes sense to me. Clinton still has the advantage among these voters, although that is the sort of lead that could quickly evaporate once more people start paying attention. I have actually written in the past about how polls that push undecideds favor Clinton more than other polls. Until that changes, her lead can be considered somewhat soft (though certainly not Lieberman-like soft).

I spent most of the last four hours thinking about my post from earlier today, and about the Inflated Clinton Poll Theory in general. Basically, I have concluded that averaging all polls is the best way to go. With so many polls, it just seems unlikely to me that one extreme Clinton-Obama margin or the other is absolutely correct, or that one methodology or the other is absolutely correct. In a heavily polled race, when has there ever been a large, hidden vote out that that most pollsters were missing? Outside of the Iowa caucuses and post-Katrina New Orleans, the answer over the last thirty years has been "basically never." These days, the worst-case scenario is for poll averages to be about six points off the final margin, which isn't that bad and can be accounted for in margin of error and turnout programs. As such, I just don't feel comfortable throwing my lot with one extreme or another, or with treating some polls as more accurate than others. I just don't want to go out on a limb like that right now, considering how I was burned by going with one extreme theory in my 2004 projections (the incumbent rule theory) and vindicated for going with poll averages in 2006 (thus opposing the wave theory).

Anyone who discounts Rasmussen and YouGov polls because they are one extreme in current national Democratic polling, or who discounts Gallup and ABC-WaPo because they are another extreme, is probably making a mistake. At this point, with so many different polls floating around, with so many different methodologies, with about half of the primary and caucus electorate not even paying "somewhat" close attention, and with an ever-changing and developing campaign, the simple fact is that widely varying results among polls is unavoidable. From time to time, I can become obsessed with trying to solve a problem in a way that will "scoop the world" that I can forget the problem probably has no clear answer. In many ways, I still live as though I am in graduate school, and as such I can still suffer from Smartest Kid In The Class syndrome, where showing up everyone else is more important than even being right. When it came to the Inflated Clinton Poll Theory, I think I just let that a case of that syndrome get out of hand.

Average the polls--all of the polls--and don't dismiss any of them just because they seem odd or you don't like the results for your candidate. Right now, that would indicate that Clinton is probably up by 10-12 points. And so she probably is. However, as the differences between the varying polls shows, there is still a lot of movement left in this electorate. It ain't over until February 6th.

There's more...

This is the post I had been hinting at for three days--Chris

Are most live-interviewer, national polls inflating Hillary Clinton's national poll advantage? For two weeks in April, I spent a lot of time trying to answer that question through a series of posts on MyDD that I referred to as the Inflated Clinton Poll Theory. Even though, at the end of the series, I concluded that there was no clear evidence to support the theory, there was no official test of the theory at that time and there were still reasons to think that there might be serious problems with national Democratic nomination preference polls. For one thing, the number of households shifting to wireless only continues to rise, which poses a real danger to traditional, live-interviewer telephone polls. Also, that Rasmussen Reports, which has proven to be an accurate polling firm over the past few years and which employs an automated, IVR polling technique, repeatedly shows different results from more traditional methods, and as such continues to raise eyebrows. Third, all national polls on Democratic primary preferences, no matter who those polls favor, are including an extremely wide net of Democrats and Democratic-leaning independents in their samples that is unrepresentative of the generally narrow Democratic primary and caucus electorate. Finally, there remains the possibility that live-interviewer telephone polls might create a sort of social pressure that alters results. Do people tell machines different things about their political preference than they tell live humans?

Taken together, do all of these concerns inflate Clinton's national poll lead? For over a month, I have sought out a polling firm that would test this theory. In their latest poll, Cook / RT strategies conducted just such a test, and the results are quite interesting. In fact, they support the Inflated Clinton Poll Theory. The following passage from a memo produced by Thomas Riehle, which I just received over email:

Lots of experiments will take place in 2007 testing new methodologies to see if they can meet or exceed the standards for accuracy of the tried-and-true live interviewer methodology using alternative polling methodologies. One important test will aim to determine whether the internal breakdowns of alternative methodologies yield similar results and similar conclusions to what we learn from live interviewer polls. RT Strategies, which conducts live interviewer polls for Cook Political Report, cooperated with a leading online polling firm, YouGov America (formerly Polimetrix) to investigate.

YouGov America is the best choice for an online pollster with whom to compare results, because all "online polls" are not the same. YouGov America employs a sophisticated sampling and respondent-matching methodology, in concert with superior panel recruitment, to make it the only online poll that can deliver a true, randomly selected sample without resorting to the crushing expense of recruiting and then providing internet access to communities that are otherwise under-represented online, or by employing the statistical gymnastics of "propensity weighting" to mask that under-representation.

Two recent Cook Political Report/RT Strategies polls were combined to yield a healthy sample size of 790 Democrats and Democratic leaners from live interviewer polls conducted April 27-29 and May 11-13. During that same time span, and using a questionnaire identical to the questionnaire used in the Cook Political Report / RT Strategies polls, YouGov America interviewed a random selection of its online panelists that included 750 Democrats and Democratic leaners. The results follow.

• Hillary Clinton scores less well online than she does in live interviewer polling. Overall, Clinton gets 32% in the combined Cook Political Report/RT Strategies polls, just 24% in the YouGov America online poll. The big differences: Among Baby-boomers and in the West, the online panelists were less likely than respondents to a live interviewer poll to make Clinton their first choice for the nomination. Hispanics online are much less likely than respondents on the phone to support Clinton.
  
  
• Obama's scores are very similar online to what he scores in live interviewer polls. Overall, Obama gets 24% with live interviewers, 23% online. Obama tends to score as well or better online among younger voters, much worse among the oldest voters, and not as well online among African-Americans as he does under the live interviewer methodology.

You can read the entire memo here (PDF), and examine the full set of comparative crosstabs between the two polls here (large PDF). While I have not had the time to go over it in full detail yet, on the surface it seems to support the idea that Clinton is not ahead nationally by the amount that traditional, live-interviewer polls suggest. In the live interview poll, she led 32-24-12 over Obama and Edwards respectively, but in the YouGov online poll, her lead was significantly reduced to 24-23-15.

Could the difference be social pressure, where Democrats don't tell live-interviewers that they are currently leaning against Clinton? Rasmussen's numbers consistently back up that theory, but those produced by Harris do not. Could it be that traditional live-interview polls and newer polling methodologies sample different universes of voters, thus producing different results? Possibly, but even if that is the case, it is extremely difficult to say which group of polls is sampling a more representative universe right now, both because we don't know who will vote in the 2008 primaries and because few polling firms release comprehensive crosstabs and methodologies. Could it simply be that when it comes to the 2008 Democratic nomination, live-interview polls are growing less useful due to the rising wireless-only population and social pressure, or that newer techniques are not yet able to achieve the same level of accuracy as traditional methods? Both are possible, but neither can be confirmed at this time.

What I do know is that this poll was conducted using the best online polling techniques available, and took place simultaneously with a live-interview poll under the supervision of a single polling firm. In the end, the two polls showed differing results that were statistically significant, and that difference does not favor Clinton. While this is not enough evidence to clearly demonstrate the Inflated Clinton Poll Theory, right now it is pretty much the only up to date evidence available to test the theory, and that evidence suggests that the theory is correct. More testing is obviously needed in order to draw a more definitive conclusion. Still, until that time comes, I don't think that the theory can simply be dismissed anymore, and an asterisk might be needed when it comes to all national Democratic nomination preference polls, both live interview and non-live interview. The current state of public opinion among the Democratic primary and caucus electorate is by no means clear, and it cannot be definitively stated that Clinton holds a large lead. The campaign might very well be tied right now. That, certainly, is big news.

I am actually in Madison, Wisconsin, right now, where I am about to speak at a regional online journalism conference. However, I just wanted to post something really quick about one reason why Rasmussen might be showing different numbers than other polling firms in the national Democratic horserace.

In Breaking Blue, Jerome argued recently that Rasmussen was including too many independents in their model:

Among all the recent polling, it's only in Rasmussen that Obama has showed competitiveness that makes it a single-digit race. That's made for a lot of puzzlement, about how Scott Rasmussen is arriving at results which are the direct opposite of every other polling outfit in the past two weeks. And it turns out, not surprisingly, to be the voter turnout model that Rasmussen is projecting.

He says, "Senator Hillary Clinton holds an eight-point lead among Democrats while Senator Barack Obama has a substantial lead among Independents who say they will vote in a Democratic Primary."

Any polling model that banks on Independents showing up to vote in a Democratic Party is a speculative turnout model, particularly in consideration of caucus states like Iowa and Nevada. In New Hampshire and South Carolina, it makes a bit more sense (but they can just as likely vote for their favorite Republican), but at least we know now where he is finding his numbers.

I agree that too many independents are included in Rasmussen's voter sample. However, I can't accept that is why Rasmussen is showing different results, simply because the entire reason I started the Inflated Clinton Poll Theory is that every polling firm is including too many people, including independents, in their survey methodology. Gallup, for example, includes about 49-50% of all adults in their samples for the Democratic primary. This is absurdly high, and obviously includes a lot of independents as well.

Without giving away tomorrow's announcement, let's consider a couple of different possibilities. First, look at two polls conducted for the Philly mayoral election at the end of April, one using automated IVR polling by Survey USA, and another using traditional, live-interviewers conducted by Susquehana (PDF). If you look at the age crosstabs of the two polls, you will note that Survey USA projected 31% of the electorate to be under the age of 35, while Susquehana projected only 17% of the electorate to be under the age of 45. That is a huge difference in the number of young voters the two polls predicted, and is generally keeping with a pattern where IVR polls pick up more young voters than do live interviewer polls (or, at least that is what Mystery Pollster told me about IVR polls the last time we talked). Obviously, any poll that picks up more young people will be more favorable to Obama, since he has consistently polled either even or ahead of Clinton among younger voters. Since Rasmussen is an IVR poll, it stands to reason that they are showing a closer campaign, at least in part, because they are picking up a higher percentage of younger voters who are generally pro-Obama. Now, I would be all ready to run with this theory, but the thing is that so few national polls on the Democratic primary release crosstabs, that it is difficult to say for certain right now.

Anyway, there are other possibilities too, including that people might say different things to machines than they do to live humans. Pollster.com's original post on this subject is still worth a read. For now, I have to talk in about fifteen minutes, before catching a plane back to Philly. More tomorrow

I am on the road today, but I will be back tomorrow morning with a huge piece of info that could shake up perceptions of the 2008 campaign. Stay tuned...

In the meantime, tell the world what is on your mind.

Update [2007-5-17 10:40:48 by Matt Stoller]: I am also on the road today, but I wanted to leave you with this post from Carl Pope of the Sierra Club on the trade deal. I basically agree with him.