Annabel Kelly is a veteran opinion researcher who has designed, managed, and reported on hundreds of surveys for clients such as PwC, the Clinton Foundation, Shell, Sallie Mae, ING, AstraZeneca, Geico, Unilever, and the US Embassy. Her published surveys have been covered by USA Today, WSJ, Family Circle, and many more.
What are the telltale signs of a credible survey?
The first sign of a survey’s credibility is the amount of disclosure that comes with the reporting of the survey results. When evaluating survey data, journalists expect complete transparency. Things like the survey sponsor, the survey method used, the number of interviews conducted, and what population the survey was trying to reflect are all valuable pieces of information. Journalists expect to have access to the survey questions and analyses as well.
Professional journalists will also want to see that the sponsor has worked with an experienced and impartial fieldwork partner that is invested in maintaining a reputation for producing quality research. The big survey firms (Ipsos, GfK, YouGov, and others) are constantly fine-tuning their methods to improve the quality of their samples, as much of survey work has moved from telephone-based research to more online panel surveys. Ipsos, for example, continuously supplements its online panel with respondents recruited from high traffic websites and GfK actively recruits members for its online panels based on their physical addresses rather than allowing them to opt in.
What are your bare minimum criteria that constitute usable data?
For a general public audience, most journalists would expect to see at least 1,000 respondents (representative of the general population). A sample of this size would enable interesting cross-group comparisons (e.g. men vs. women; young vs. old; parents vs. non parents etc.).
For surveys of elite audiences, however, much smaller samples are totally acceptable. The Associated Press, for example, only polls around 65 sportswriters and broadcasters to produce its weekly rankings of the top 25 NCAA teams. The key, again, is to be completely transparent about the respondents. When the University of Chicago publishes the results from its economic expert polls, it not only lists its forty or so respondents, it also provides their actual responses to the questions posed.
Make sure your survey has a watertight methodology, is not too-self serving, and offers a new perspective on an issue
What about Survey Monkey?
With the advent of Survey Monkey, Google Surveys and the like, it’s now possible for anyone to conduct a survey for very little investment. While these tools offer access to representative samples, they can also be used to survey an email list or a group of social media followers. Anyone with little to no grounding in survey research runs the risk of asking biased questions of a biased sample; messing up the analyses; and/or misrepresenting the findings in a press release.
PR people sometimes concoct a dream headline, then backwards engineer survey questions from there. Does that make sense?
Absolutely! It’s comparable to forming a hypothesis and helps the research team to understand the objectives for the research so that all bases can be covered in the design. I also recommend that the survey team think through some of the graphics they might want to produce from the data before finalizing the questions. Obviously results can’t be guaranteed!
What are your tips for making news with survey results?
So many! First and foremost, get a journalist onboard at the outset, offer them an exclusive, and invite them to contribute to the survey design. I’m seeing more and more surveys conducted as joint ventures between brands and media outlets.
It is also important to make sure your survey has a watertight methodology (as discussed earlier); is not too-self serving; and offers a new perspective on an issue either through the questioning angle or the audience being surveyed.
A rewarding strategy would be to invest in a longitudinal rather than a one-off survey. Providing regular updates on at least one of your key metrics will show which way attitudes or behaviors are moving and help your brand command ownership of a topic.
When releasing your results, giving your survey a brand will add gravitas and encourage recognition. Some examples of strong survey brands include IBD/TIPP’s “Economic Optimism Index”; Sallie Mae’s “How America Pays for College”; and Pew’s “Internet & American Life Project”.
Do you suggest creating a report from the survey? A press release? What should we do with it?
Definitely a press release and, maybe, an infographic for a short survey. For a more involved survey, I would also recommend producing a report that is downloadable from your website.
You can also add color and draw attention to your survey by inviting relevant personalities to respond to some of the questions and report their point of view alongside the main findings.
To boost engagement and the shelf life of a survey, some sponsors are embracing the quantified self movement and inviting their audiences to respond to the same survey questions before presenting their answers in the context of the population as a whole. For examples, see the quizzes developed by the Pew Research Center where readers can compare their knowledge of and attitudes towards different topics with the general population.