Happy International Year of Evaluation!
2015 was declared to be the International Year of Evaluation (EvalYear) at the Third International Conference on National Evaluation Capacities organized in São Paulo, Brazil, 29 Sep-2 Oct 2013, the aim of EvalYear is “to advocate and promote evaluation and evidence-based policy making at international, regional, national and local levels” (Source). As a professional evaluator and advocate for evidence-based practice/policies, I’m excited that the United Nations Evaluation Group and Eval Partners (the International Evaluation Partnership Initiative) are highlighting this important work on a global level.
So this has gotten me thinking – what can I do to celebrate EvalYear here in my little corner of the blogosphere? Mostly I’ve just been using this blog as a place to keep my notes on various things I’m learning – webinars and conferences I’m attending, books and articles that I’m reading, etc. – and not really as a blog that anyone else might want to read. But perhaps I can use this year as an impetus to add more reflection on my evaluation work and musings on evaluation practice 1Once I have enough such things, I might even start promoting it out to the world so that others might actually come and read the stuff here. I’m still planning to use it as a place to keep notes on things that I’m learning though, as I’m finding it very useful to have my notes in one easily accessible place!, 2Some potential topics I could write about include: What does an evaluator do? (which I get asked all the time when I’m asked “What do you do for work?” and I reply “I’m a program evaluator”), What skills does one need to have to become an evaluator?, Why should someone do an evaluation?, How do you evaluate an evaluation?, What’s the different between research and evaluation (and auditing and quality improvement and…)?, Evaluation standards and competencies, My favourite evaluation resources, What is a CE?
I was thinking that it would be cool to post my “evaluation philosophy” 3I developed a “teaching philosophy” when I started teaching at the post-secondary level and thought that the concept of thinking through and articulating a philosophy would be useful in my evaluation work as well. as a kick off to Eval Year. Which, of course, would require me to have developed my “evaluation philosophy”, which I actually haven’t sat down and done yet 4It’s been on my “to do” back burner for quite some time.. I often think about things that are potential elements of my evaluation philosophy as I go about, and reflect on, my work, but I’ve not documented my ideas or combined them into one coherent philosophy statement. But then I remembered that this is a blog, so things don’t have to be perfect or set in stone. This is actually a great place to do some documentation of what I’m thinking about and ultimately develop my evaluation philosophy statement… especially given that I’ve just been using this blog to take notes of various things I’ve been reading/attending and I’m pretty sure no one reads this stuff but me!
So, with that in mind, here are some musings towards an evaluation philosophy statement.
First things first.
A lot of people tend to put the cart before the horse. For example, they start a conversation about evaluation referring to the data they know they already have (or the data that is the easiest to get) and then try to retrofit an evaluation around the data. Or they have a favourite method – “I want to evaluate my program by interviewing stakeholders!” or “We need to evaluate our program with a survey!” – and want to create an evaluation based on the method.
My philosophy is that you should really put first things first – what is it that you actually want to know? Do you want to know if your program is achieving its intended outcomes? (Moreover, do you even know what outcomes you are intending to achieve?) Do you want to know if your program is being implemented as intended? Are you developing something innovative and you want to know how to evaluate it as you go along?
Once you know what your overall evaluation question is, then you can think about what approach makes sense, what method(s) are best suited to answer your question, and can think about what data you need to answer the question, and how will you collect, analyze, and interpret those data.
Bias
In my previous job, I worked as an internal evaluator – that means I was employed by an organization to do evaluation work for the programs within that organization, as opposed to being an external evaluator, which is someone who isn’t an employee of the organization, but rather works as an consultant and is hired by an organization to evaluate their program(s). In my current job, I’m pretty much an internal evaluator 5I am technically employed by a research institute to do an evaluation of a big project being implemented collaboratively by three organizations, but my research institute is hosted by one of those three organizations and I spent the lion share of my time working from the project management office for the project anyway. One thing I’m often asked is “Aren’t internal evaluators biased, since they are employed by, and are paid by, the organization?” The implication is that if you don’t give the organization the results they want (and the result they “want” is for you to say their program is effective) you might lose your job. Or at least you might think that if you don’t say the program is effective, you might lose your job. I have two points to make against this idea. First, external consultants are also generally paid for by the organization being evaluated 6Though I suppose external evaluators could be brought in by, for example, a funding agency, to do an evaluation, though I think that more often than not would be more like auditing that evaluation. And then you’d have potential bias of thinking that the evaluator/auditor needs to give the results the funding agency wants to hear., so they potentially have the same bias – they might think that if they don’t give the organization the results they want, they will have their contract ended or won’t be contracted for the next project. Second – and more importantly – I think any organization with integrity genuinely wants to know if their program is working, because if it’s not, they can then look at why it isn’t working and change things to make them work better. My role as an evaluator is to try to figure out what’s working and what’s not working and for whom and in what circumstances and why/why not, as well as to help people figure out what to do with this information to make things work even better. I can be genuinely excited about the vision an organization is working towards without thinking that I have to bias my results to show that they are achieving it. The way I, as an evaluator, can support them in their work is to do an honest job of evaluating. If an organization ever asked me to do otherwise, I wouldn’t want to work for them.
Integrity
This brings me to another point – integrity is a critical piece of being an evaluator. If I don’t demonstrate integrity, why would anyone trust my evaluations? If I’m not trustworthy, why would people that I’m interviewing be honest with me? I’m sure I could say a whole lot more on this topic, but this posting is getting quite long and I’ve got evaluating to do. To be continued!
Footnotes
↑1 | Once I have enough such things, I might even start promoting it out to the world so that others might actually come and read the stuff here. I’m still planning to use it as a place to keep notes on things that I’m learning though, as I’m finding it very useful to have my notes in one easily accessible place! |
---|---|
↑2 | Some potential topics I could write about include: What does an evaluator do? (which I get asked all the time when I’m asked “What do you do for work?” and I reply “I’m a program evaluator”), What skills does one need to have to become an evaluator?, Why should someone do an evaluation?, How do you evaluate an evaluation?, What’s the different between research and evaluation (and auditing and quality improvement and…)?, Evaluation standards and competencies, My favourite evaluation resources, What is a CE? |
↑3 | I developed a “teaching philosophy” when I started teaching at the post-secondary level and thought that the concept of thinking through and articulating a philosophy would be useful in my evaluation work as well. |
↑4 | It’s been on my “to do” back burner for quite some time. |
↑5 | I am technically employed by a research institute to do an evaluation of a big project being implemented collaboratively by three organizations, but my research institute is hosted by one of those three organizations and I spent the lion share of my time working from the project management office for the project anyway. |
↑6 | Though I suppose external evaluators could be brought in by, for example, a funding agency, to do an evaluation, though I think that more often than not would be more like auditing that evaluation. And then you’d have potential bias of thinking that the evaluator/auditor needs to give the results the funding agency wants to hear. |