Annabel Kelly is a veteran opinion researcher who has designed, managed, and reported on hundreds of surveys for clients such as AstraZeneca, the Clinton Foundation, GEICO, ING, PwC, Sallie Mae, Shell, and Unilever. Her surveys have been covered by Family Circle, USA Today, the Wall Street Journal, and many other media outlets.
As a PR professional who often leverages surveys to secure media coverage for my clients, I sat down with Annabel to learn a little more about the science behind surveys and how best to utilize survey data. Here’s what Annabel had to say.
What are the telltale signs of a credible survey?
The first sign of a survey’s credibility is the amount of disclosure that comes with the reporting of the results. When evaluating survey data, journalists expect complete transparency. Things like the survey sponsor, the survey method, the number of respondents, and the population the survey respondents are supposed to reflect are all valuable pieces of information. Journalists also expect to have access to the survey questions.
Professional journalists will want to see that the sponsor has worked with an experienced and impartial survey company that is invested in maintaining a reputation for producing quality research. The big survey firms (Ipsos, GfK, YouGov, and others) are constantly fine-tuning their methods to improve the quality of their samples. Ipsos, for example, continuously supplements its online panel with respondents recruited from websites, and GfK recruits members for its online panels based on their physical addresses rather than soliciting general opt-ins.
What are the minimum criteria that constitute usable survey data?
For a general public audience, most journalists would expect to see at least 1,000 respondents. A sample of this size would enable interesting cross-group comparisons (e.g., men vs. women; young vs. old; parents vs. non-parents).
For surveys of targeted audiences, however, much smaller samples are acceptable. The Associated Press, for example, only polls around 65 sportswriters and broadcasters to produce its weekly rankings of the top 25 NCAA basketball and football teams. The key, again, is to be completely transparent about the respondents. When the University of Chicago publishes the results of its economic expert polls, it not only lists its forty or so respondents, but also provides their actual responses to the questions posed.
The data will ultimately speak for itself.
What do you think about Survey Monkey?
With the advent of Survey Monkey, Google Surveys and the like, it’s now possible for anyone to conduct a survey quickly and inexpensively. While these tools offer access to representative samples they can also be used to survey an email list or a group of social media followers. Anyone with little to no grounding in survey research runs the risk of asking biased questions of a biased sample. This would make the results almost meaningless to the media. Great care must be taken to produce useable data from these survey tools.
PR people sometimes concoct the dream headline, then reverse engineer survey questions from there. Does that make sense?
Absolutely! It’s comparable to forming a hypothesis that helps a research team to understand the objectives for a project. This can inform the design of the research. I also recommend that the survey team think through some of the graphics they might want to produce from the data before finalizing the questions. Obviously results can’t be guaranteed. The data will ultimately speak for itself.
What are your tips for making news with survey results?
First and foremost, get a journalist onboard at the outset, offer them an exclusive, and invite them to contribute to the survey design. I’m seeing more and more surveys conducted as joint ventures between brands and media outlets.
It is also important to make sure your survey method will withstand media scrutiny, is not too-self serving; and offers a new perspective on an issue.
A rewarding strategy would be to invest in a longitudinal rather than a one-off survey. Providing regular updates on at least one of your key metrics will show which way attitudes or behaviors are trending and help your brand command ownership of a topic.
When releasing your results, giving your survey a brand will add gravitas and encourage recognition. Some examples of strong survey brands include IBD/TIPP’s “Economic Optimism Index”; and Pew’s “Internet & American Life Project.”
Do you suggest creating a report from the survey? A press release? What should we do with it?
Definitely issue a press release and, maybe, an infographic for a short survey. For a more involved survey, I would also recommend producing a report that is downloadable from your website.
You can also add color and draw attention to your survey by inviting relevant personalities to respond to some of the questions and report their points of view alongside the main findings.