The “Iron Law of Purpose”

I was working on a presentation about “validity” for a group that is working on developing a survey and came across the following as I was looking something up in one of my research methods textbooks:

Simply stated, the iron law of purpose declares that you cannot evaluate anything without a conception of its purpose. […]

Any act of evaluation requires an act of choosing, of deciding which alternative is preferable. The act of deciding, in turn, requires an assessment of the consequences of various alternatives. However, assessing consequences (outcomes) requires a prior conception of purpose. In short, without a conception of an object’s purpose, you cannot evaluate its consequences. If you cannot assess an object’s comparative consequences, you cannot evaluate its worth. This proposition can be labelled a ‘law’ because it applies to everything – from hammers to sweethearts, from vegetables to your life. If you are unclear on what anything is for (its purpose), you cannot assess how worthwhile it is ¹Source: : Roberts et al (2009). The Methods Coach: Learning Through Practice. Don Mills, ON: Oxford University Press Canada. p. 53.

I quite like this quotation, as it’s something that I find myself coming up against quite often. I find that often when people want to start working on an evaluation, they jump to one of the following:

• “We need to do a survey!” or “a focus group” or “a randomized controlled trial”… or just about any other method or study design. When I start to ask questions about how they decided on that method or design, it usually comes out that they think it’s the only way to evaluate or that “we can only afford (or we only have the skills to do) surveys”. Sometimes, people aren’t even clear on what question(s) they want to answer with their evaluation, but they’ve already decided on the method to answer whatever question they come up with.
• “Oh, you are going to do an evaluation? Well, we already collect data on [fill in the blank with any number of things they might be collecting data on], so you should measure that.” Again, they often haven’t even thought about what question(s) they want to answer with their evaluation, but they’ve already decided on a way to measure the outcomes ²or activities or outputs, etc. (which may or may not be the outcome that they are even trying to affect!
• Them: “I need an evaluation!”  Me: “What about your program would you like to evaluate? What would you like to know?” Them: [blank stare]… “I need an evaluation!”

All three of these situations, which I have experienced more than once, amount to the same thing – forgetting to think about what the purpose of their evaluation is and either (a) immediately jumping to how to get to an answer (despite not knowing what the question is) – and really, how can you design a good evaluation if you don’t know what its purpose is? or (b)  not even knowing where to start.

Fortunately, there is a good place to start! Figure out what you want to know, and then decide on the best way to get that information. You can think about this by asking things like:

• What is the purpose of the evaluation? What do you want to know?
  • e.g., Do you want to know if your program is achieving its objectives? ³Which, of course, requires that you know what your objectives are. You’d be surprised how often people haven’t articulated what they are intending their program to achieve. But that’s a topic for another blog posting entirely – the Iron Law of Purpose applies to your program as well as to your evaluation!.
  • e.g., Do you want to know if you are implementing your program the way it was designed?
  • e.g., Do you want to know what’s going well in your program and what’s not going well?
• What question(s) are you trying to answer with this evaluation?
• What do you want to be able to do with the results of the evaluation?
  • e.g., to improve your program? to decide if the program should be kept or stopped? to decide if the program should be expanded to other places?

So one of my big pieces of advice when it comes to evaluation really is to start with figuring out what you are doing it for, because to design a worthwhile evaluation, you have to know what you want to get out of the evaluation at the end of the day. And, as Roberts said, “If you are unclear on what anything is for (its purpose), you cannot assess how worthwhile it is.”