We've been watching the results of our reader survey [1] roll in, with over 2,800 responses so far. You can still vote, but I can likely predict how you'll vote, based on current responses. In fact, the percentages of certain answers have been stable since the first few hours of the survey.
This fact - that not much data is necessary to draw accurate conclusions - goes against the strongly held belief among many survey professionals that a high response rate is necessary. In fact, for a proposed survey to win a federally funded grant, one of the most important criteria is a predicted high response rate, and media pollsters performing quick surveys seldom report their response rates because they're so low. But according to Dr. Jon Krosnick [2] of Stanford University, that belief turns out to be wrong, something that researchers are just coming to realize.
Dr. Krosnick spoke last week as part of a speaker series organized by the Cornell University Survey Research Institute [3], and although Tonya and I felt somewhat out of place in a room of academics, we were pleasantly surprised to find Dr. Krosnick's talk engaging and accessible even to those of us who have no formal training in surveying or statistics. If you're extremely interested in the topic, I encourage you to listen to the talk [4] (26.6 MB MP3); for the rest of us, I thought I'd offer a quick summary of the non-intuitive lessons Dr. Krosnick imparted and a few other facts of interest to anyone who has been asked to complete a survey in person, over the telephone, or on the Internet.
A Novice's Conclusion -- We found Dr. Krosnick's talk utterly fascinating, and although we didn't have time to chat with him beyond the Q&A session at the end, it would seem that some conclusions could be drawn from his lessons about the kind of Internet polls and surveys we see so frequently.
First, although there is absolutely no disagreement that a random sample is ideal, the difference in accuracy was not huge. When applied to a question that is likely to have relatively divergent answers (such as the age of TidBITS readers), useful conclusions can easily be drawn without worrying that a self-selected sample would be horribly biased in one direction. Attempting to distinguish between answers separated only by a percentage point or two wouldn't be possible, though.
Second, even if the response rate isn't huge, that wouldn't seem to make much of a difference. We might end up with a response rate of less than 10 percent in our survey, but the added accuracy gained by a larger response rate certainly wouldn't be worth harassing you all multiple times to answer our questions. Just how small that rate can be isn't entirely clear, but single digits don't appear to be a major problem.
Third and finally, unlike a survey gauging national voting plans, most Internet polls don't attempt to use the results to predict the future, nor are the results likely to affect the future actions of other people. I can't quite put in words why this seems like a relevant difference, but it's related to the goal of the survey. If I learn what percentage of TidBITS readers regularly play computer games (28 percent), I can use that information when considering what articles to write, but I can't see the publication of this fact causing people either to start or stop playing games. However, compare that to surveys that ask who you plan to vote for in the next election; your answer has the power to help sway the opinions of other voters.
And in that thought is where I think the answer to decreasing response rates lies. Surveys can be intrusive and badly timed, but if it's reasonable to complete them, they should be seen as a way of spreading your opinions to the rest of the world. It's the same reason I don't mind using grocery store shopper cards; I know they're tracking my purchases, and I want the fact that I'm buying more organic and less processed food to be recorded prominently. So the next time you're polled, consider it a chance not just to be counted, but also perhaps to nudge the world in the direction you want.
[1]: http://emperor.tidbits.com/webx/?displaySurvey@@.3c8dc29b
[2]: http://communication.stanford.edu/faculty/krosnick.html
[3]: http://www.sri.cornell.edu/
[4]: http://www.sri.cornell.edu/sri/files/Krosnick_Lecture.mp3
[5]: http://www.knowledgenetworks.com/