Now more than ever our profession needs to embrace a spirit of innovation tempered by experimentation and evaluation within the context of the Total Survey Error framework (see Groves 1989). Yet all too often we respond to the introduction of a new technology with a reflexive fear. As Mick Couper put it after reviewing responses to advances such as computer-assisted telephone interviewing and computer-assisted personal interviewing, "Each such innovation is viewed as a portend of the end of surveys as we know them, as a threat to all we hold near and dear as survey researchers" (2002, p. 1). But, as Couper also notes in the same paper,

the reality is often neither as wonderful as the proponents of the technologies argue, nor as dire as the major detractors fear. Each new technology enhances and extends the range of possibilities and opportunities for survey research, but also often introduces new challenges and issues for further research. (2002, p. 2)

[...]

Yet unlike our consumers on the Web, we often shun arguments for newer technologies based on crude efforts to compare preelection survey results against election returns. What makes for a good sample, we often say, is not what comes out (the results) but what goes in (probability sampling, rigorous field practices, and standardized questionnaires). That is sound advice, and no one can deny that in any single application, chance alone can determine which preelection survey comes closest to the final result. However, with a meta-analysis involving enough cases, we can at some point overcome the limitations of sampling error and conduct comparisons that add up to "validation." If we can include and control for other variables that might affect survey accuracy, such meta-analyses can become powerful tools for assessing the efficacy of newer methods like IVR.

In the world of academic survey research, a form of "open-source" methodology reigns. Survey data and methodology are fully disclosed and shared. While we certainly lack a single unified survey methodology analogous to the Mozilla browser or Unix operating system, academic survey researchers have reached broad consensus around a set of "best practices" on many issues. Moreover, the "source code" of academic survey research is free and available to all practitioners to modify as they see fit.

In the media and commercial world, however, a more proprietary model still applies. The bigger survey research companies conduct their own research and development. Yes, many of the well-known national pollsters deposit raw data with the archives of the Roper Center for Public Opinion Research, the Inter-University Consortium for Political and Social Research, and the Odum Institute for Research in Social Science (University of North Carolina). While many of these same pollsters will share aspects of their internal research and procedures at AAPOR conferences and for studies that appear in this journal, much methodology remains closely guarded or inaccessible, including details regarding the selection of "likely voters" and specific procedures for weighting data.8

It is worth noting that even among media pollsters, an "open-source" philosophy exists with respect to the dissemination and sharing of question text. Spurred in part by the standards propagated by the National Council on Public Polls (NCPP), most survey organizations routinely release the full text of the questions they ask. As a result, pollsters in the process of designing a new questionnaire need not contact colleagues to peruse the language of previous studies. They can go online to resources like the Polling Report or the I-Poll online databases of the Roper Center for Public Opinion Research to search and compare numerous questions from comparable past studies.9 As with "open-source" software, they can replicate the language used in previous studies or modify it as they see fit. By putting our question language out into the public domain, we collectively strengthen the quality of all opinion research.

Moving toward such an "open-source" collaboration on methodology could take many forms, but it would generally call for more experiments by practitioners and more disclosure to facilitate more and better meta-analyses across surveys. Few practitioners regularly conduct controlled experiments--they lack either the budget, the training, or both. Most of us choose to "tinker" in an evolving way. But if one can imagine an environment in which practitioners were willing to collaborate on methodological studies, the possibilities are unlimited.

Further, simple disclosure of more methodological information on studies that are already being released into the public domain could facilitate a far richer and more useful brand of meta-analysis. For example, the existing studies that compare preelection surveys to vote results could be that much stronger if we could include variables for sample frame (RDD or list), whether the pollster weights, and by what (demographics? regional breaks? party ID or other attitudes? weighted and unweighted frequencies for common demographic variables?).

Is this a pipe dream? Perhaps. But the growing popularity of sites that provide access to polling data on the Internet creates an opportunity to set a new standard of disclosure for that medium. The NCPP standards have had tremendous reach far beyond the NCPP membership. Nonmembers and private pollsters routinely include field dates, number of interviews, information about the "margin of error," and a description of who was interviewed largely because consumers demand it. They demand it because the most prominent public polls provide it. The original NCPP standards make sense in a world of very limited column inches and airtime. But the Internet has few such limits. Public polling organizations now routinely provide long detailed reports on their websites with no such constraints. The opportunity for a new standard is ripe.