“Hey DRR, we did a study and the conclusion is that you are incompetent.”

“While this may be true, can I enquire as to your study design?”

“We did a survey study and 67% of respondents agree that you aren’t fit to be the editor of a journal. We did all the statistics and the P value is < 0.001.”

“I’m curious how you decided who to survey?”

“Well, Bob and I don’t like you and my wife thinks you are okay, mostly because she doesn’t really know you.”

At the BCMJ we review all sorts of submissions for publication and we appreciate all the work that goes into the process of designing and carrying out a scientific study. That being said, one thing that drives us a little crazy (particularly the editor) is low-response survey studies. Surveys are handed out, collected, tabulated, and subjected to rigorous statistical analysis including P values, which all looks very impressive. The problem: many of these surveys have response rates of less than 20% from which no meaningful information can be obtained. The assumption that the greater than 80% of people who didn’t respond would have completed the survey the same way as the respondents is just that—an assumption. What if that 80% couldn’t be bothered to complete the survey because they really disliked something about it? Good survey studies are easy to spot. The target population is clearly defined and follow-up contact is done on numerous occasions in an attempt to increase the response rate. The authors also include a discussion in their paper of the limits of their survey study. Here at the BCMJ we don’t really look at a survey study unless the response rate is well over 50%.

Now, I don’t want to discourage prospective authors, only to give advice on how to increase the chance of publication. Handing out program evaluation surveys in a haphazard fashion without regard to random sampling techniques or total number of potential respondents is really a waste of everyone’s time and doesn’t lead to conclusions that can be acted upon.

Okay, I’ve said my piece and have ranted enough.
—DRR