Evaluator Competencies Series: Independent & Balanced Perspective

1.5 Provides an independent and balanced perspective in all aspects of the evaluation.

This is one competency that I have done a fair bit of thinking about. I believe that ito be a good evaluator, you need to bring a balanced perspective. I interpret this to mean that, as much as possible, you:

• go into an evaluation without a preconceived notion of whether the program/initiative that you are evaluating is achieving its goals/is having good effects/is having bad effects/is a success/is a failure. An evaluator should not be setting out to prove that the program works – they are seeking to find out if it “works” or to what extent it does or does not “work” or in what ways is does or does not work
• go looking for both good and bad effects of a program
  • bring in as many perspectives as possible into the evaluation – what is considered “good” for one person or group might be considered “bad” by another person or group. Or which outcomes are most important might differ depending on who you ask.

When I first started working on the project I’m currently evaluating, people were referring to me as doing “benefits evaluation” and would say things like “Beth is going to be measuring the benefits of the project.” To which I would reply, “No, I’m going to be measuring outcomes. I’ll let you know if they are “benefits” or not.” (OK, so I may have said that last part in a little less snarky of a way, but you get my point. I was confused at first as to where this phrase of “benefits evaluation” came from, as I was new to the world of healthcare technology. After doing some digging, it appears that the phrase is often used by software vendors, who try to sell their products by highlighting all the purported “benefits” they are going to provide. (Sometimes they would also use the phrase “benefits realization” – as in, “Beth is going to be measuring the benefits realization” (to which I’d have a similar reply to the one I mention above)).

Also, there is an organization called Canada Health Infoway (CHI) that has a “Benefits Evaluation Framework” that is intended to be used by projects that are implementing healthcare technology, but it’s always bothered me because it makes the assumption that the technology will result in benefits and the evaluator just needs to measure them. (Or that if the expected benefits aren’t seen, you just identify what’s preventing it – such as people aren’t adopting the technology, so you increase adoption and then the benefits will happen). But we know that technology does not necessarily deliver the benefits that those selling those technologies claim they will deliver. Plus, the framework has nothing in it to allow you to look for unintended consequences, but we know that implementing technology is a complex endeavour and complexity is characterized by, among other things, the fact that we cannot predict what will come out of them. I saw a presentation by some people from CHI at the most recent CES conference and when I asked them about these issues, they conceded that their framework is missing these things.

Anyway, early on at the organization I am working in, I went on a bit of a campaign against the word “benefits” – and when I explained to people why I was opposed to the word, everyone I talked to saw my points as valid and I rarely hear the word benefits used on the project anymore (Every once in a while, when a new contractor starts on the project, they’ll start saying “benefits realization” and inevitably someone will say “You better not let Beth hear you say that word!”). It might seem overly pedantic to be so concerned about a word, but words have power and using a work that frames an evaluation in a biased way can set up the evaluation to be designed with that bias. And even if you set up the evaluation to be more balanced than the word implies, people saying that they are “measuring the benefits” is not conveying the right message about what evaluation is all about.

I struggle a bit with the phrase “independent perspective” in this competency. I think maybe they are going for “unbiased” in the sense of “doesn’t have a vested interest in the program being seen as a success”. I know that I’ve had the experience of having people who are interested in the results of an evaluation (e.g., participants in focus groups or interviews, others that are affected by a program) assume that an evaluator (who is hired by management) is going to just try to make a program look good. The way I see it, while I may want a program to be successful – I wouldn’t be working on an evaluation for an organization or program whose goals I opposed – I don’t think that saying a program is successful when it is not would be helpful. Because wanting a program to actually be successful is different from wanting the program to look like it’s successful. If I, as an evaluator, put out an evaluation that says “this program is successful” (whether by cherry picking data that makes the program looks good or by designing the evaluation in a way that it only looks at the good stuff and ignores bad things that are happening), then not only is that being dishonest, but it’s also not helpful as then the program will continue to operate in a way that is not getting the promised results and/or is causing harm. The way that I, as an evaluator, can help to contribute to the goals of a program being achieved is by providing an honest and thorough assessment of the what’s going well (if anything) and what’s not (if anything).

But perhaps the word “independent” in this competency is more about “is not influenced by the stakeholders to change the results of the evaluation.” We’ve all heard stories of, and/or have experienced for ourselves, situations in which stakeholders don’t like the results of an evaluation and want them changed or suppressed. And I agree that it’s important for evaluators not to be influenced to change results. And I can also see that it can be a challenging situation, especially when there is a power differential between the stakeholder and the evaluator (and if the stakeholder is the one paying the evaluator (whether internal or external) and/or can influence future work opportunities for the evaluator, there is a power imbalance. Some strategies that I use to help prevent a situation like that include:

• agreeing at the start of the evaluation about how widely the results of the evaluation will be shared. Getting this in writing at the start can give you something to go back to to say “we agreed to this”.
• discussing the possibility that “negative” findings might come up with stakeholders and how you will deal with them (e.g., “negative” findings won’t be suppressed, but perhaps management would want to include a response to the findings about what they are doing to make improvements) – it’s much easier to discuss this and come to an agreement when the idea of negative findings is just an idea, rather than waiting until there is a real negative finding about the program that might bring up an emotional or defensive reaction
• discussing the importance of transparency and credibility – I often point out that participants in the evaluation know what they said in interviews/focus groups/surveys, so if you remove findings that management doesn’t like (again, not only would that be dishonest) they will know that this evaluation report does not reflect the data they provided and you’ll lose all credibility.

The other challenge that I have with the word “independent’ is that it suggests that the evaluator is separate from the evaluand, but again, going back to complexity, we know that things are connected and the evaluator is just another agent in the system, interacting with all of the other agents. Evaluations have affects on their evaluands – for example, when people know something is being measured, they change their behaviour. When people know they are being watched (such as when an evaluator is conducting observations), they change their behaviour. Evaluators also often do things like helping programs clarify their goals – and thus the program is changed by its interaction with the evaluator. I don’t think this is a bad thing. But I do think it’s important to be aware of. In my work, I write reflections on the ways in which I see the evaluation work affecting the program and I try to include that in my interpretation/sensemaking.

Images Sources:

• The image of the mutlicolour ovals and circles was posted on Flickr by Anne Adrian with a Creative Commons license.
• The photo of a computer was posted on Flickr by Gareth Halfacree with a Creative Commons license.
• The photo of the leaning Jenga tower was posted on Flickr by Ed Garcia with a Creative Commons license.