Evaluation, Time, Money and Methods

16 Sep

Whenever you are tasked or contracted to carry out an evaluation of a project, particularly in health, there are nearly always 2 immediate challenges

  • Investment- the amount of money and time available to carry the evaluation out
  • Access- the availability of relevant people and populations to provide data

These issues immediately restrict the capacity of a researcher to conduct fieldwork. Time spent out of the office interviewing respondents is time consuming. Factor in the time taken to organise interviews and other data collection points, and very quickly you have either burned through the budget or are left with an inadequate amount of data as time runs out. Designing and conducting research methods to scale is always a problem, the gap between the “theoretically desirable and the practically possible”. But funders pay for results so it’s a problem that requires solving.

A method of scaling projects is to train non-researchers from within target populations, or people who have easy access to the target populations, to collect the data. This works best with larger scale projects, so you avoid skews with small numbers (for example, only 10 total respondents all interviewed together in the same room). And for similar reasons the approach works best across multiple independent sites, because multiple sites (contexts) stand a better chance of locating genuine reoccurring themes and also insight specific to local context. Below is a recent example from the method angle.

I’ve recently completed a project where I was part of a team where we collected qualitative data from over 250 respondents. The data was collected by non-researchers from across multiple sites and the project was focused on assessing priority areas for strategic decision making. The design of the data collection helped us deliver to budget and on time, and importantly, produce analysis the funder was happy with.

When designing the methods for use by the non-researchers, we looked at other domains where good quality data had to be collected rapidly, and, was not collected by research professionals. The domain which we used was wild firefighting, which I examined through the work of Weick and Sutcliffe (2007). The chaotic nature of a wild fire means firefighters have to debrief rapidly, pass on learning, identify vital cues and focus attention on what is most important. These outcomes were exactly what we were looking to achieve through our data collection. To ensure these conditions are met there exist interview schedules which are designed to collect data from firefighters as they leave the field.

I adapted one of the fire fighter interview schedules (Weick and Sutcliffe, 2007) for the non-researchers in our project to use in both interviews and focus groups. The questions were structured to collect anticipation-what people think could and should happen next, and also reflection-what has gone wrong specifically in the past. These type of questions are useful for separating out needs from wants and setting priorities. Most of all, these type of questions enable strategic decision making to be improved via insight.

For our team and project, the adapted interview schedule worked very well, better than we could have hoped. We had piloted it and stress tested it before it went “live”, and this allowed us to make further adjustments. But most significantly, it was very easy for non-researchers to understand the underlying logic of the schedule and so make adjustments independently, depending on the context and flow, and for them to use practically in the field. This should have been no surprise, the interview schedule which we adapted was designed specifically to achieve these outcomes.

Reading

Weick, K. E., & Sutcliffe, K. M. (2007). Managing the Unexpected: Resilient Performance in and Age of Uncertainty, Second Edition. San Francisco, CA: Jossey-Bass

One Response to “Evaluation, Time, Money and Methods”

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: