If you work in international development and if you commission research in developing countries, then this series of blog posts is for you! This is a call to ask you to take the lead in rethinking from an ethical and human perspective what is acceptable and what is not in development research and to tackle some of the unconscious biases in our field. We follow our call for respecting local approval procedures with a plea for researchers to consider the respondents’ time.
We often find ourselves nudging clients to shorten extremely long surveys, full of unnecessary questions that take too much of people’s time. These surveys are typically the result of overly ambitious research objectives and too many cooks in the kitchen. When researchers collaborate on a set of survey instruments, it’s easier to add questions and ideas than to remove them, especially if the questions come from colleagues. Surveys often end up with too many elements that are merely glanced over in the analysis phase. Each question that is ‘discarded’ during the analysis phase comes at the cost of the time it took hundreds of respondents to answer that question.
When we argue to reduce the length of surveys, the counterpoints we hear are often the same. One is that receiving survey consent is sufficient, as long as we specify that the survey is going to take up several hours. This can be disingenuous since respondents might feel obligated to consent. The second argument is that it’s ok to take up people’s time if we can provide a small compensation. And the third is that respondents are “benefiting” from a particular intervention, so taking up their time is justifiable. These arguments all have one thing in common: they touch on the sort of determinations that should be made by a local research review process, not by the organization designing or funding the survey.
We once asked the ethics board of a large research funder whether they had any guidelines on survey duration. They said it was not within their competence to answer that question, and that it was up to their project teams to determine if a 4-hour long survey with farmers was ethical. The project team saw no harm in moving forward with the survey, without compensation for participants, on the basis that respondents were receiving support from a large-scale program that their organization was funding with government support.
Surveys with fewer, more targeted, questions are much cheaper to run, give clearer results and, most importantly, are more considerate of those we want to empower and support.
Shorter surveys are cheaper and yield better results
The truth is that major development organizations have no qualms running extremely long surveys. Even if people consent to participate, and even if the consent form specifies the expected duration, long surveys are disruptive of people’s time. This is especially true if the survey firm just shows up at someone’s door or calls over the phone with no warning (which, by the way, is not unusual).
And let’s be frank. Would anyone working in our sector take 2, or 3, or more hours to respond to a survey without significant compensation? If you were to commission a similar survey in the US, Canada, UK, or Germany for example, would you consider making that survey several hours long?
In our experience, we get better results when we take the time to obtain consent first and make an appointment with households before the data collection begins. This is not only more respectful of people’s time but allows us to achieve significantly higher survey success rates and better data quality. We admit we could do more of this ourselves at Laterite, as we are often limited by budget ceilings that are incompatible with an approach involving multiple touch-points with respondents.
To summarize: we need a paradigm shift away from the mental models we have developed as researchers and funders. We need to think twice about what is acceptable when designing research instruments. A good place to start is with a ‘less is more’ approach to survey design, and to use the budget savings from shorter surveys to fund an approach that factors in several household visits in order to get consent, book an appointment, and survey respondents.
Surveys with fewer, more targeted, questions are much cheaper to run, give clearer results and, most importantly, are more considerate of those we want to empower and support. Humans, not subjects.
—
This post was contributed by Dimitri Stoelinga and Sachin Gathani, Managing Partners at Laterite