It’s that busy time in the semester when the marking for the courses that I’m teaching is piling up and I am also working furiously on a couple of online courses (in addition to my day job). So I’ve decided that I’m going to take a wee break from writing my (nearly) weekly blog series on evaluator competencies until I get through this backlog of other work.
Recognizing our limitations and being about to prioritize are important competencies for an evaluator, right?
I really like helping programs figure out what their theory of change is. Early in my career as an evaluator, I was surprised how often I would work with a program that had no idea what its theory was. Like, you’d sit down with them and ask questions about what they were trying to achieve and how what they thought what they were doing was going to help them achieve it – and they didn’t know. They had never really thought about it. The program was the way it was by some combination of it having been started by someone in some way at some time for some reason and then it had been adapted over the years in response to funding cuts/new funding opportunities/new leadership/new research/[enter all sort of other possible factors here]. While talking about this with my class this weekend (I’m teaching a Program Planning & Evaluation course in a Masters of Health Administration program), one student described the programs that she’s worked on as having been MacGyvered and I absolutely love that description!
Perhaps way back when a program started there had been an idea of a program theory – or possibly not – but it’s been MacGyvered over the years and often there us no record of any original program theory. And so I discovered that an important part of work as an evaluator is often to help the program make explicit the theory of why they think the program will result in changes to achieve whatever it is trying to achieve. Because even if a program doesn’t have an explicit program theory, there is some implicit theory underneath.
And there are many benefits about making your program ‘s theory of change explicit. As an evaluator, I want to know what the program’s theory is so I can design an evaluation to test the theory. But it can also be quite helpful to the program itself – helping them to get everyone on the same page about what the program is actually trying to achieve and getting them to think about whether what their program does is likely to get them there. Also, sometimes mapping out a program theory helps a program to identify that it is doing activities that are not likely to help them achieve their goals. It’s surprising how often programs do things because “we’ve always done these things”, even though they may no longer be needed or relevant. Working through a program theory can help identify those things.
Oftentimes, I work with those involved in the program to clarify the theory by developing a logic model together. There is a debate about whether a logic model is or is not a program’s theory of change. According to Michael Quinn Patton (2012), a logic model is simply a description of a logical sequence, but “specifying the causal mechanisms transforms a logic model into a theory of change”, i.e., you need to “explicitly add the change mechanism” to make it a theory of change. I like this explanation because it reminds us that a logic model on its own isn’t quite enough to be a “theory of change” so we need to think about what is the actual mechanism that is believed to lead to the change.
Thinking about how I do the work of clarifying program theory, I think my tips would be:
This topic is also a good time to plug the free online logic modelling software that my sister, her partner, and I created: Dylomo (short for DYnamic LOgic MOdels). You can sign up for free and play around with it. Apologies in advance for any bugs – we created it off the side of our desks, so haven’t had time to add all the features we would like. If you do have any issues with it – or feedback about it – do get in touch!
Patton, M. Q., (2012). Essentials of Utilization-Focused Evaluation. Thousand Oaks, CA: Sage.
Photo of leaves was opsted on Flickr by Mehul Antani with a Creative Commons licence. Again, I couldn’t find a good free-to-use image for what I was searching for (program theory, theory of change, logic model), but while searching for “change” I found that image of leaves changing colour and thought it was beautiful.
Dylomo logo was designed by my amazing sister, Nancy Snow.
I can’t remember exactly when or where this was, but at some point during my career as an evaluator, I saw the phrase “Evaluability Assessment” and thought “what’s that?” I know I wasn’t a brand new evaluator, as when I looked it up and learned what it was I thought, “Oh! I’ve been doing that at the start of every evaluation that I’ve done. I didn’t know there was a name for it!”
An evaluability assessment is, much like the name suggests, assessing “the extent to which an activity or project can be evaluated in a reliable and credible fashion” (OECD-DAC 2010; p.21 cited on Better Evaluation). As I learned to be an evaluator, this seemed to be a thing that I naturally needed to do.
An example would be where a client says “I want an evaluation that tells me if the program is achieving its goals”. The first question one would ask is: “What are the program goals?” because I certainty can’t tell you to what extent you are achieving your goals if you don’t have any goals. Similarly, a program may ask an evaluator to conduct an evaluation on whether the program has improved some particular thing for their program participants (e.g., their health, their knowledge of a topic, their social connectedness – whatever thing the program is trying to help its participants improve). In that case, one would naturally ask “how were the clients doing on that something before they started the program?” (i.e., do you have any baseline data we can use to compare to?). Or perhaps it’s a case where the client says “I want to know if my program is working?”, in which case I would ask “What does the program “working” mean to you?”. And that might lead to some work around developing a program theory, or figuring out if they want to know what outcomes are achieved, or if they want to know if their processes are efficient, or whether they are concerned about negative unintended consequences (or maybe all of the above). In my experience as an evaluator, when I ran into situations like this, my first course of action would be to work with the clients to figure out how to get their program into a state in which they are evaluable.
What I didn’t realize in my early years as an evalulator is that in some cases, an evaluability assessment could be a project unto itself. (Check out the Better Evaluation page on Evaluability Assessment if you want to read more about it.)
This may be due to the fact that I’ve always been an internal evaluator (or, as I think of myself on the program I’m currently working on, an external evaluator who is embedded in the program for the long term). So I’ve always had the luxury to be able to work with my “clients” to get them into an evaluable state as part of the work I do with them. Perhaps if I were an external evaluator, I may have come across stand alone evaluability assessments as potential projects.
I couldn’t find any images online that I felt represented “evaluability assessment” (probably not surprising… spell check doesn’t even believe that “evaluability” is a word!). So instead I give you this picture of my cats:
The next domain of competence is technical practice.
2. Technical Practice competencies focus on the strategic, methodological, and interpretive decisions required to conduct an evaluation.
And the first competency in this domain is:
Let’s begin at the beginning. That may seem trite, but I think it’s such a common saying because so often, people want to start somewhere other than the beginning. I cannot tell you how many times I’ve been consulted about an evaluation and the person seeking my advice starts with something like:
These are all examples of not beginning at the beginning. Many people seem to think that an evaluation requires a specific method (e.g., a survey) or a specific design (e.g., an RCT). Or they think that whatever the latest trend in evaluation must be the best approach, because it’s new. Or they have data already and they want to use it 1I just noticed that I‘ve written about this before, more than 4 years ago! Past Beth would be sad to hear that I’m still experiencing this!. But where an evaluation needs to start is with its purpose. Why, exactly, do you want an evaluation? What will you use the findings of the evaluation for? These are the types of questions that I will ask (usually preceded by me saying “Let’s back up a second!”). Because the purpose of the evaluation will guide the choice of approach, design, and methods. For example, if you are interested in an evaluation that will help you to determine to what extent you’ve achieved your goals, and none of your current indicators relate to your goals, then starting with “I have a set of indicators and I need to do an evaluation using them” is not going to get you where you want to be. Similarly, if you want an evaluation that will help surface unanticipated consequences (and I tend to think that evaluations should usually be on the look out for them), then having a set of pre-defined indicators is not going to be what you need (after all, to create an indicator, you have to have been anticipating that it might be affected by the program!). If the purpose of your evaluation is not a developmental one, then developmental evaluation might not be the best approach for you. So clarifying the purpose (or purposes) of an evaluation is something that I do at the start of every evaluation – and something that I check in on during the evaluation, both to see if what we are doing in the evaluation is helping to meet its purpose and to see if the purpose changes (or new purposes emerge) along the way.
Clarifying the scope of an evaluation is also really important, and something that I struggle with. I’m am an infinitely curious person and I want to know all the things! But there just isn’t enough time and resources to look at every possible thing in any given evaluation, so it’s important to be able to clarify what the scope of any given project is. Like purpose, it’s important to clarify the scope of the evaluation with your client at the start, and to keep tabs on it throughout the evaluation. If you don’t have a clear scope, it’s very easy to fall into the trap of the dreaded “scope creep” – where extra things get added to the project that weren’t initially agreed to and then either the costs go up – or the timeline gets extended. It’s not to say that the scope can’t change during an evaluation, but just that any changes to scope should be done mindfully and in agreement between the client and the evaluator.
Working in a large organization like I do, I also find it useful to understand the scope of other departments that do similar work to evaluation (like quality improvement and performance management). This is helpful in ensuring that we aren’t duplicating efforts of other teams, and also that we aren’t stepping on anyone else’s toes. Also, I’ve had the experience of taking on work that really should have been done by another team (i.e., the dreaded scope creep!) and had we not figured this out by clarifying scope, it would have really impaired our ability to deliver on the work that we needed to deliver on.
My team and I have done some work on clarifying what the scope of evaluation is relative to these other groups and I was about to say “and that’s a topic for another blog posting”, but then I remembered that I’m presenting a webinar (based on a conference presentation I gave last year) on that in a couple of weeks! So here’s my shameless plug: if you want to hear me pontificate on the similarities, differences, and overlaps between evaluation and other related approaches to assessing programs and services, register for my webinar, hosted by the Canadian Evaluation Society’s BC Chapter on Friday, September 13 (that’s right, Friday the 13th!) at 12 pm Pacific Time.
Footnotes
| ↑1 | I just noticed that I‘ve written about this before, more than 4 years ago! Past Beth would be sad to hear that I’m still experiencing this! |
|---|
There have been a few times in my life when I decided to do something and then, as the thing approached I thought “What have I done? This is too big and too scary and too hard and I’m totally not going to be able to handle this!”. Moving across the country to do a PhD. Play in a hockey game that lasts for 10 days. Do an MBA part-time while still working full-time. Accepting my current job. As it turned out, all of these were things that I could handle and are things of which, as it turns out, I’m extremely proud! It’s almost like being scared that I’ve bitten off more than I can chew is a sign that I’m about to do something awesome.
My job prior to my current job was fun and I learned a lot and I met some great people, some of whom I’m still good friends with (Hi Heather!). But after 5 years in that job, I’d hit a pay ceiling, I’d learned all that I could learn, and so I wasn’t feeling challenged any more. And then a co-worker of mine told me about a job posting she’d seen that she thought I might be interested in. It was a job doing the same type of work (evaluation in healthcare), but taking it to the next level. A leadership position where I’d get to run a team of evaluators to conduct an evaluation of a massive, multi-organization, multi-year project that has the chance to change the face of healthcare in the region. I was excited by the possibilities this job entailed, so I applied and I got the job. And a few days after I handed in my resignation at my old job I thought “Oh my god, what have I done? I know how to do my old job really well. But there’s so much I don’t know about this new job – I have to learn a whole new area of healthcare AND I’ll be the boss of people and that’s a whole new ballgame for me. What if I can’t do it?” What I should have realized then was, just like the PhD, just like the Longest Game, and just like my MBA, that fear was a sign of a great challenge and I’d shown over and over again that I can rise to a challenge.
The last five years have been really interesting. I’ve learned a tonne about health informatics, about applying complexity concepts to the evaluation of an ever changing project, about governance, about managing people, about managing data when you have a large group of people creating and using a huge dataset, and that’s not even getting into what I’ve learned in terms of the findings of the evaluation so far!
I’ve had the opportunity to collect data from 13 healthcare facilities and counting, I’ve built my team up from 2 to 11 evaluators (all of whom are pretty fantastic, I must say), and I’ve presented my work across Canada, as well as in the US and Australia.
And even after five years, I’m not bored. I honestly feel like we are just getting things rolling and we are improving our processes at every step, and I’m learning so much from all the amazing people on my team, and we are producing information that is actually getting used by decision makers. And there’s so much more still to come.
This is not to say that it’s been easy, or that I will be easy going forward. In a recent presentation I gave about the project at the Canadian Evaluation Society conference, I used this image to represent my experience:
I also often reference that MC Escher painting where the stairs are going up but also going down at the same time as representing what it’s like to work on the project I’m working on. (I can’t put the image here on the blog because I don’t have copyright permission, but here’s a link to the Wikipedia page on it where you can see the image)
But honestly, it’s kind of OK with me. The real world is messy and things don’t always work out how you planned them, but you learn a lot by going along for the ride.
Image sources:
Cross posted on my other blog.
And the final reflective practice competency is:
There are several ways that I engage in this competency:
I first got involved with CES back in 2010, when I was looking to find my way in this profession. The national conference was being held in Victoria, so I volunteered for the conference as I figured it would be a good way to meet other evaluators and learn about the field. And was I ever right – the evaluation community was so welcoming and I met people there that I’m happy to call friends and colleagues to this day.
For the next several years, I went to the CES national conference when I was able to attend, but then in 2015 the BC & Yukon chapter decided to host a one-day conference of its own, and that’s when my involvement really took off. I volunteered to be the conference program chair for that conference – and also volunteered to be a program co-chair for the national conference which was scheduled to be held in Vancouver in 2017. That role was a tonne of work, but it was also a lot of fun, as I got to work with two delightful fellow evaluators, Sandra Sellick and Wendy Rowe. I really enjoy and get a lot from conferences (both in the content I learn and in the networking opportunities they provide) and I know from experience that they take a lot of effort, so I think that volunteering for conferences is an important way that I can contribute to the profession and its community of practice.
Also in 2015, I joined the CESBCY council as a member at large, later transitioning into the VP role when the VP stepped down. In 2017, I became the chapter president. I’m really proud of the work the chapter is doing – we are hosting a lot of professional development events (e.g., one day conference, various workshops and webinars) and meetups that serve the evaluation community.
This year I also coached a student case competition team at the CES national conference – and that was a really rewarding way to support new evaluators in our community!
Another way that I feel that I contribute to the evaluation profession is by teaching evaluation. I’ve taught evaluation courses at both SFU and UBC, and I’ve supervised practicum students from SFU, UBC, and UVic. And several of my students have gone on to work in evaluation (right now, I have three of my former practicum students and two of my previous evaluation course students working as evaluators on my team!)
And speaking of my team, I currently have 10 evaluation specialists working on my team and a big part of the work that I do as the leader of the team is to mentor and support them. This is another way that I am working to contribute to the future of our profession.
Another way that I’m involved in evaluation professional networks is online. There’s the #EvalTwitter hashtag that a lot of us connect through. There’s even a monthly #EvalTwitter tweetup on the last Thursday of every month (at 5:30-6:30 pm Pacific time). And through#EvalTwitter I learned about Eval Central, an online forum that “aim[s] to encourage positive and fruitful discussion among culturally diverse evaluators from around the globe.” So I recently joined that and am eager to see what kind of conversations happen there.
Image credits:
I didn’t write a blog posting in my series last Sunday – the weekend was busy and time got away from me! But it’s now this Sunday night and I’ve got cup of tea and I’m ready to reflect on reflective thinking!
Often I do my my reflective thinking over a a cup of tea – whether sitting on my own to do some reflective writing, or chatting with colleagues (As an aside, if you want to read some brilliant thoughts on reflective practice, check out Carolyn Camman’s fabulously titled blog posting “The coffee is largely metaphorical“). I’m an external processor and I find that I tend to come up with a lot of my great “a ha!” moments when I write my thoughts down or talk to a friend or colleague. I also don’t have a great memory, so when I have an insight, I need to write it down to cement it in my brain (or the very least, so I can look it up again later.)
I write a lot of reflections as I go about my work. Whether I’m collecting data, analyzing data, in a meeting, or whatever activity I might be doing, if I have an “a ha!” moment, I write it in my reflective journal (which for the project I’m currently working on is typed up and saved on a shared drive with the rest of my team’s reflections, as these “a ha!” moments are about the content of the evaluation that we are working on together). A reflection might be about a pattern I’m noticing in the data, or a connection I’m making between different parts of the evaluation, or a surprise that I wasn’t expecting, or thoughts on some of our longer-term evaluation questions. My general rule is “if it’s interesting enough for me to want to tell my team about this cool thing I saw or thought of, it should write it down as a reflection). This improves my practice because it helps me to identify things that are important to the interpretation of the data, which allows me to develop accurate and comprehensive evaluation findings.
I also keep some separate reflections that are more for myself than as part of the evaluation data. For example, since I’m the team manager, if I have reflections that are about my work as a manager, and I might not want to share those with the team right away – especially if I’m trying to work through a challenge or figure out a way to be a better team manager. Some of those reflections might become things that I do want to talk about with my team later, but sometimes I need some time and space to work through stuff first. This helps improve my practice because being an effective leader will help my team be effective in its work.
Speaking of my team, we’ve taken to having group reflection sessions after we complete any big chunk of work where we debrief on:
These are some pretty standard evaluation type questions, but we’ve definitely been able to continually improve our practice by doing this reflection together.
For example, in our first big round of data collection, we didn’t do nearly enough documentation of our data analysis. And with having a big team of people all working on different pieces of the data analysis, it meant that we had a lot of files that we’d all named in different ways, with our spreadsheets set up in different ways and often not very well labelled. So when it came time to write up our findings, it was quite difficult to find the data we needed, and we sometimes had to reproduce some of the analyses to ensure we had the correct data. So my big lessons learned for future rounds of data collection were:
These seem like pretty basic things – and they are – but this was the first time for all of us working on a big team. We each had our own individual naming conventions and ways of setting up our analyses in our spreadsheets that had served us well working as individual and what we hadn’t realized was how many different ways people could do the same task! Since the project is being implemented in a phased approached, we are now entering a period of time where our work will be a bit cyclical (collect baseline data for a site, monitoring data at the time of implementation, collect post-implementation data 3-6 months later, and repeat for the next site). And I can see that we are getting better and better each time because we’ve been reflecting on how we do our work and finding ways to be more efficient and more effective.
Another reflection that I shared in a team reflection session recently was something that I think links to the “self-awareness” part of the competency. Working in healthcare, even as a non-clinician, you get exposed to situations and information that can be quite emotional. For example, even when doing a chart audit, you get exposed to stories of serious illnesses/injuries and deaths. Or when interviewing healthcare workers who are exposed to traumatic situations, you also get exposed to those traumatic situations. As human beings, this can bring stuff up for us (like similar illness, injuries, patient journeys, and deaths of loved ones, for example) and it’s important to be kind to ourselves when stuff like this gets to us. I am extremely lucky that I work in a large team made up of kind and caring colleagues, so we know that we can go to each other if we need to debrief, or if today is just not a good day for us to do that particular observation or interview. Being aware of situations that might bring up things for me and being aware of my emotions as I’m experiencing them can help me to manage those, ask for help when I need it, and thus help to ensure that they don’t negatively effect the work. It can also help me to be empathetic to my colleagues and the people I interact with as I do my work.
In addition to reflection with my team of evaluators at work, I am also part of a co-op inquiry group that meets monthly to reflect on a particular topic (for us, it’s “boundaries in evaluation”) and that has been an amazing experience to hear the reflections of a group of evaluators from different sectors and locations – I have left every meeting having expanded on ideas I’ve been having and having learned new ideas or perspectives from my colleagues that have resonated with me.
Teaching is a fantastic opportunity to reflect. Whenever I prepare to teach an evaluation course, I’m dedicating time to stepping back and thinking about the big picture of evaluation – what it is and how to do it well. I find it also brings me back to the basics and it gives me the opportunity to think about whether there are ways that can improve what I’m doing. I use a lot of storytelling and examples when I teach – I’ve had many students tell me that they really appreciate that I do that because I tell them “what really happens, as opposed to what the textbooks tell you it’s going to be like”. But it also helps me because, again, it gives me an opportunity to think about how I’ve done my work, how it links to concepts, theories, standards, etc. and how I might do my work in the future.
In addition to getting back to basics, I also like to tell students about whatever the “hot topics” are in the field at the time, which means that I have to keep abreast of what the hot topics are, and typically do a bit of research to be well versed enough in the topic to discuss it with the class. This is an opportunity for me to identify gaps in my knowledge and do some learning.
Another aspect of teaching that I think is reflective is that students tend to ask really great questions. And since they are coming from a different perspective, sometimes those questions are things that I haven’t thought about before, which forces to me to reflect situations from a different angle. Sometimes they ask questions that I do not know the answer to – when that happens, I tell them that I’ll go do some research and get back to them. This links to that notion of self-awareness – knowing the limits of my knowledge, having the confidence to say “I don’t know that right now, but I will find out”.
And finally, this blog is something that I’m using as part of my reflective practice now. I’m glad that I decided to write this blog series on the evaluator competencies as a way to provide some structure and timeline to get me in the habit of reflecting here on a regular basis 1Last Sunday notwithstanding.. I’m finding it quite useful to spend a bit reflecting on the extend to which I have each of the competencies and areas for each where I can continue to learn and grow.
Image Source:
Footnotes
| ↑1 | Last Sunday notwithstanding. |
|---|
transparent adjective
trans·par·enta: free from pretense or deceit : FRANK
b: easily detected or seen through : OBVIOUS
c: readily understood
d: characterized by visibility or accessibility of information especially concerning business practices
Merriam-Webster Dictionary
To reflect on this competency, I decided to see what the definition of “transparency” is. I usually think of transparency in the sense of “sharing all the information”, which is a bit more extreme than one can actually be in an evaluation. For example, we have an ethical responsibility to maintain confidentiality for participants in our evaluations when they want their identity to be kept confidential. Sometimes we are working with proprietary information that the organization requires to be kept confidential. So as with so many things, being “transparent” requires a bit of nuanced thinking.
I used to work with someone who talked about her role in a communication chain in a hierarchical organization, where information came from the top and was cascaded down through the org chart. Sometimes, information was only allowed to be shared to a certain level – say, it could go from the VPs to the EDs to the Directors, but the Directors were not allowed to share it with the Managers – at least not yet. And this person’s (who was in a Director role) approach to it was to tell their managers “I do know this information but I am not allowed to share it with you at this time.” And then they would give the reason (e.g., “Leadership is planning to do X, but until it is signed off by the board of directors, it’s not official and so they do not want put this information out broadly in case the board does not sign off it on, as it could cause confusion.”). And then they would make a commitment to tell their managers as soon as they were allowed to. This approach stuck with me, because it was honest (they weren’t saying “I don’t know this information” when they really did know, which is an approach I’d seen others take in these types of situations) and it was as informative as it was possible to be given the situation – giving a reason why they weren’t allowed to share the information at that time, rather than just saying “I’m not allowed to say”. I find that not giving a reason usually results in people coming up with their own theories about what information is being kept hidden – and that ends up causing rumours and confusion. So I think that this (sharing what you can and being honest about what you can’t share and why) can be a useful approach to being transparent. Of course, there can be good reasons or bad reasons for not wanting to share information, so I think we also have a responsibility to think critically about the reasons why an organization might not want to share and to push back in situations where appropriate (e.g., if an organization wants to suppress evaluation findings because they think it makes them look bad, as I talked about last week, I’d push back on that).
I was interested to see that the definition of “transparent” isn’t just about making information accessible, but also making information “readily understood” and “free from pretence or deceit”. These are things I can get behind. Obviously, a credible evaluation should not include anything deceitful, but I think making information “readily understood” is something that is sometimes overlooked. There are so many ways that we can exclude people from evaluation by not being “readily understood” – whether that be by the way we design our evaluations, the ways we recruit participants, the methods we use, or the way the report the findings. There seems to be a lot of interest in the evaluation world around data visualization – i.e., presenting data in ways that actually convey the meaning of them. This is something that my team and I are actively working to get better at. And there’s interest in alternative reporting formats – i.e., not just handing over a 200 page report, but actually thinking about ways to report evaluation findings that work for those who are interested in those findings.
Something I see spoken about less often, but that I think about a lot, is the use of language. I’m a bit fan of clear, simple language when writing 1Though I will admit that I have a tendency to be wordy, I write complicated sentences, and I overuse footnotes to an excessive degree. because I think that if I’m writing something, I want people to understand it. I mean, isn’t that why I’m writing it? I try to avoid jargon (or at the very least explain any jargon that I use) and prefer to pick a simple word over an obscure one. But I often see writing that is full of jargon, and unnecessarily large and obscure words. Part of me thinks that people write this way in an attempt to look intelligent. And I have seen situations where people use jargon as a way to try to cover up that they don’t know what they are talking about (which becomes evident as soon as you start asking questions like “What do you mean when you use the word X?”) An even more cynical part of me thinks that people write like this in order to exclude other people, by making the knowledge they are ostensibly trying to “share” non-understandable by “others” who don’t have the same training/background as them. After all, knowledge is power and keeping knowledge away from others by making it not understandable to others is a way of holding onto power. Which to me, is another reason to make the effort to make my writing as clear and easy to understand as possible.
At any rate, I hadn’t really thought about making information “readily accessible” as being part of “transparency” before, but it makes sense when I think about it.
Image Sources:
Footnotes
| ↑1 | Though I will admit that I have a tendency to be wordy, I write complicated sentences, and I overuse footnotes to an excessive degree. |
|---|
My notes from the third and final part of the American Evaluation Association (AEA) eStudy course being facilitated by Jonny Morell.
A neat example of how difficult it is to predict the future is shown in this letter from Rumsfeld to G.W. Bush.
Jonny provided an extensive reading list. Here are some that caught my eye and I’m planning to check out:
Image Sources
This is one competency that I have done a fair bit of thinking about. I believe that ito be a good evaluator, you need to bring a balanced perspective. I interpret this to mean that, as much as possible, you:
When I first started working on the project I’m currently evaluating, people were referring to me as doing “benefits evaluation” and would say things like “Beth is going to be measuring the benefits of the project.” To which I would reply, “No, I’m going to be measuring outcomes. I’ll let you know if they are “benefits” or not.” (OK, so I may have said that last part in a little less snarky of a way, but you get my point. I was confused at first as to where this phrase of “benefits evaluation” came from, as I was new to the world of healthcare technology. After doing some digging, it appears that the phrase is often used by software vendors, who try to sell their products by highlighting all the purported “benefits” they are going to provide. (Sometimes they would also use the phrase “benefits realization” – as in, “Beth is going to be measuring the benefits realization” (to which I’d have a similar reply to the one I mention above)).
Also, there is an organization called Canada Health Infoway (CHI) that has a “Benefits Evaluation Framework” that is intended to be used by projects that are implementing healthcare technology, but it’s always bothered me because it makes the assumption that the technology will result in benefits and the evaluator just needs to measure them. (Or that if the expected benefits aren’t seen, you just identify what’s preventing it – such as people aren’t adopting the technology, so you increase adoption and then the benefits will happen). But we know that technology does not necessarily deliver the benefits that those selling those technologies claim they will deliver. Plus, the framework has nothing in it to allow you to look for unintended consequences, but we know that implementing technology is a complex endeavour and complexity is characterized by, among other things, the fact that we cannot predict what will come out of them. I saw a presentation by some people from CHI at the most recent CES conference and when I asked them about these issues, they conceded that their framework is missing these things.
Anyway, early on at the organization I am working in, I went on a bit of a campaign against the word “benefits” – and when I explained to people why I was opposed to the word, everyone I talked to saw my points as valid and I rarely hear the word benefits used on the project anymore (Every once in a while, when a new contractor starts on the project, they’ll start saying “benefits realization” and inevitably someone will say “You better not let Beth hear you say that word!”). It might seem overly pedantic to be so concerned about a word, but words have power and using a work that frames an evaluation in a biased way can set up the evaluation to be designed with that bias. And even if you set up the evaluation to be more balanced than the word implies, people saying that they are “measuring the benefits” is not conveying the right message about what evaluation is all about.
I struggle a bit with the phrase “independent perspective” in this competency. I think maybe they are going for “unbiased” in the sense of “doesn’t have a vested interest in the program being seen as a success”. I know that I’ve had the experience of having people who are interested in the results of an evaluation (e.g., participants in focus groups or interviews, others that are affected by a program) assume that an evaluator (who is hired by management) is going to just try to make a program look good. The way I see it, while I may want a program to be successful – I wouldn’t be working on an evaluation for an organization or program whose goals I opposed – I don’t think that saying a program is successful when it is not would be helpful. Because wanting a program to actually be successful is different from wanting the program to look like it’s successful. If I, as an evaluator, put out an evaluation that says “this program is successful” (whether by cherry picking data that makes the program looks good or by designing the evaluation in a way that it only looks at the good stuff and ignores bad things that are happening), then not only is that being dishonest, but it’s also not helpful as then the program will continue to operate in a way that is not getting the promised results and/or is causing harm. The way that I, as an evaluator, can help to contribute to the goals of a program being achieved is by providing an honest and thorough assessment of the what’s going well (if anything) and what’s not (if anything).
But perhaps the word “independent” in this competency is more about “is not influenced by the stakeholders to change the results of the evaluation.” We’ve all heard stories of, and/or have experienced for ourselves, situations in which stakeholders don’t like the results of an evaluation and want them changed or suppressed. And I agree that it’s important for evaluators not to be influenced to change results. And I can also see that it can be a challenging situation, especially when there is a power differential between the stakeholder and the evaluator (and if the stakeholder is the one paying the evaluator (whether internal or external) and/or can influence future work opportunities for the evaluator, there is a power imbalance. Some strategies that I use to help prevent a situation like that include:
The other challenge that I have with the word “independent’ is that it suggests that the evaluator is separate from the evaluand, but again, going back to complexity, we know that things are connected and the evaluator is just another agent in the system, interacting with all of the other agents. Evaluations have affects on their evaluands – for example, when people know something is being measured, they change their behaviour. When people know they are being watched (such as when an evaluator is conducting observations), they change their behaviour. Evaluators also often do things like helping programs clarify their goals – and thus the program is changed by its interaction with the evaluator. I don’t think this is a bad thing. But I do think it’s important to be aware of. In my work, I write reflections on the ways in which I see the evaluation work affecting the program and I try to include that in my interpretation/sensemaking.
Images Sources:
