While researching evaluating complex programs/complex systems, I came across an evaluation approach/method that I wasn’t familiar with: Qualitative Comparative Analysis (QCA). So I’ve done a bit of reading on this to see if it might be something I can use in my work.
Background
- evaluations face the tension between contextualization (i.e., needing to understand the context in which an intervention occurs, because context (in addition to the intervention) can affect outcomes) and generalization (i.e., being able to inform future practice by identifying recurring patterns of what works and what does not work)
- “project evaluation has tended to focus on the comparison of ‘before and after’ situations, and has not adequately incorporated the influence of contextual local conditions on infrastructure project development” (Verweij & Gerrits, 2012, p. 41)
- there is a “misfit between the way infrastructure development projcets are understood and the methodologies used to evaluate them” (Verweij & Gerrits, 2012, p. 41)
- variable-oriented studies can allow the identification of generic patterns (e.g., rail projects lead to the biggest cost overruns among transportation infrastructure projects), but don’t account for local contexts that also affect outcomes, while case-based studies allow an understanding of the affect of context, but don’t allow for identifying patterns for generalization. QCA is meant to integrate these two approaches
- when we say that an infrastructure project is “complex”, “it usually means that it is perceived to be difficult” (Verweij & Gerrits, 2012, p. 42), but complexity is more than that – it is a “multi-layered concept” (p. 42):
- comprises a “mixture of generic elements [e.g., suburbanization] and local conditions [e.g., specific features of geography that affect how suburbanization could (or could not) happen at a specific site]” (p. 43)
- “becomes even more complex if its social fabric is taken into account” (Verweij & Gerrits, 2012, p. 43)
- “thus, the local built order emerges from interaction between generic and specific physical and social elements” (Verweij & Gerrits, 2012, p. 43)
- it is important that “this specific pattern of local conditions and generic developments is researched to understand ex-ante how a project should be executed, and to understand ex-post what leads to certain outcomes” (Verweij & Gerrits, 2012, p. 43)
- to understand complex infrastructure projects
- they “take place within a specific interacting mix of local conditions and generic patterns that occurs in any given location”
- “the causal relationship between site-specific conditions and generic developments are poorly known and, if known, only for that specific time and place” – so “known causal relationships specific to a certain area are by definition case-specific”
- “the emergent nature of any built area implies that it is the result of longitudinal development”, i.e., it is the result of past changes and events that are to some extent path-dependent” (Verweij & Gerrits, 2012, p. 43)
- types of complexity:
- generic complexity “focuses on the emergence of complex processes and structures from a limited set of variables. It assumes a general set or rules from which emergent complexity flows”… but it is missing that the “emergent nature of infrastructure projects is partly determined by local systems” (Verweij & Gerrits, 2012, p. 44)
- situated complexity focuses “on the explanatory value of the contextualization of a phenomenon”. Buijs et al “argue that while open systems ‘do not operate according to general rules applied in all context’, a systematic comparison can reveal differences and similarities between the operations of different systems. This approach to situated complexity focuses both on recurring patterns over multiple systems and the idiosyncratic events in particular systems, since both determine how systems develop over time” (Verweij & Gerrits, 2012, p. 44)
- how can complexity be understood, “which is basically a question of how reality can be understood”
- positivism: “primarily concerned with determining general rules by taking reality apart in discrete components” (Verweij & Gerrits, 2012, p. 44) – similar to generic complexity
- postpositivism: “has many different sub-strands that range from the extreme relativism social constructivism to the more realist thesis of negotiated subjectivism or critical realism” (“the common theme within those strands is that the contextualization is explanatory for what is being observed”) (Verweij & Gerrits, 2012, p. 44) – similar to situated complexity
- “if systems are said to be open, then it follows that their boundaries do not exist a priori, and any individual will develop a particular demarcation or set of boundary judgements about the system which includes and excludes variables (i.e., a reduction of real complexity) that may be connected but not perceived as such by the observer. Thus, there is no unambiguous separation between systems and their context, and the observer is as much part of the complexity as the system or agents that are observed. Situated complexity is therefore not confined to the presupposed demarcations of system but intersects all system representation by respondents.” Verweij & Gerrits, 2012, p. 44)
- implications of this for evaluation:
- people choose the boundaries of the system
- we should take multiple perspectives into account
- “cause and effect relations do exist and can be known through respondents’ perceptions” (Verweij & Gerrits, 2012, p. 45)
- evaluators cannot be separated from the evaluation
- implications of this for evaluation:
Criteria for a complexity-informed methodological framework
- “it balances between in-depth understanding and reductionist generalization” (Verweij & Gerrits, 2012, p. 45)
- “cased-based”, where “projects are treated holistically an not as collection par parts” (p. 46); with multiple cases that are compared “to allow for causal inference – for studying patterns across cases” (Verweij & Gerrits, 2012, p. 46)
- this “rejects the idea that variables can be disaggregated from cases and analysed separately as if it is the variables rather than the cases that are causal. […] It is the ‘case’ and the state of important conditions of each case” (they are “bundles of conditions that interact together”) (Blackman et al, 2013, p. 4)
- ” allow the observation and analysis of complex interaction between the variables” (Verweij & Gerrits, 2012, p. 45)
- consider how situated complexity came into being over time (i.e., complex dynamics)” (Verweij & Gerrits, 2012, p. 45)
Qualitative Comparative Analysis (QCA)
- an umbrella term for:
- crisp set QCA (csQCA) – where conditions are scored as binaries (present (1) or absent (0))
- multi-value QCA (mvQCA)
- fuzzy set QCA (fsQCA) – allows conditions to be scored as a gradient (e.g., could be 0, 0.5, 1, 1.5, etc.)
- “aims to integrate the case-oriented and variable-oriented approaches” (Verweij & Gerrits, 2012, p. 46)
- “can be used to achieve a systematic comparison across a smaller number of individual cases in order to preserve complexity, and yet being as parsimonious as possible and illuminating otherwise hidden causal paths on a micro level’ (Rihouse & Lobe, cited in Verweij & Gerrits, 2012, p. 46)
- in this process, you examine multiple cases:
- “to uncover the most frequent combinations of causal conditions (i.e., variables) that produce a certain outcome” (conjunctural causation)
- note that “different configurations” may produce the outcome” (equifinality)
- note that “factors can have different effects in different contexts” (multifinality)
- also looks at asymmetric causality: “the presence and absence of outcomes require different explanations” (Verweij & Gerrits, 2012, p. 46)
- uses “dialogue between theoretical and empirical evidence, which is especially important in the selection and construction of cases and variables” (Verweij & Gerrits, 2012, p. 47)
- “in QCA, variables are conceptualized as causal conditions or sets” (Verweij & Gerrits, 2012, p. 47) – but note that since “social phenomena […] are often difficult to grasp in terms of sets” … “theoretical and substantive knowledge should be used to substantiate the constructions (and membership) of sets” (Verweij & Gerrits, 2012, p. 47)
- “sets can be intersected” [with ‘logical and’] and “unified” [with ‘logical or’]
- QCA then is “able to systematically compare and analyze these set conjunctions” (Verweij & Gerrits, 2012, p. 47)
- QCA looks at “set relations instead of correlations” (Verweij & Gerrits, 2012, p. 47)
- “a condition is necessary if it has to be present for the outcome to occur” (p. 47)
- “a condition is sufficient if it can produce the outcome by itself” (p. 47)
- an INUS condition is “insufficient but non-redundant part of an unnecessary but sufficient condition” (Mackie, cited in Verweij & Gerrits, 2012, p. 48) – that is, since it is often a combination or “recipe” of conditions that are required to cause an outcome, there are “usually no purely necessary or sufficient condition” (p. 48) but rather conditions that are not sufficient on their own nor necessary on their own, but are sufficient when combined with other specific conditions
- it is important to note that “‘neither necessity nor sufficiency exists independently of theories that propose causes'” (Ragin, cited in Verweij & Gerrits, 2012, p. 48)
- once you have each case assigned to a set (i.e., you know its score for each condition, as well as its outcome), you consider a “truth table“:
- “lists all of the logically possible configurations” (Verweij & Gerrits, 2012, p. 48-9)
- fundamental unit of analysis = truth table row
| Condition A | Condition B | Condition C | Outcome | Distribution of cases |
| 1 | 1 | 1 | 1 | |
| 1 | 1 | 0 | 1 | |
| 1 | 0 | 1 | 1 | |
| 1 | 0 | 0 | 1 | |
| 0 | 1 | 1 | 1 | |
| 0 | 1 | 0 | 0 | |
| 0 | 0 | 1 | 0 | |
| 0 | 0 | 0 | 0 |
- then “the truth table can be minimized to produce a so-called solution (i.e., a statement about patterns across cases” (Verweij & Gerrits, 2012, p. 46). This is done using Boolean algebra9
- it is important that this is done not done by just applying the formula, but includes “interpreting the formula and critically assessing it in light of individual cases: does it make sense?” Thus, there should be “several iterations between data and concepts, generating increased understanding/interpretation of the cases” (Verweij & Gerrits, 2012, p. 50)
- also, “we can use the method in an exploratory rather than explanatory mode. This is particularly useful when we have ‘contradictory’ configurations (i.e., configurations without 100% of the cases having the same outcome state). In such cases, we can then return to the cases and seek additional differentiating characteristics of their previous trajectories and contexts that might help us to resolve the contradiction (i.e., generate an explanation of the difference in observed outcome state” (Byrne, 2013, p. 224)
- Steps in QCA
- define the outcome
- identify conditions thought to be relevant to the outcome
- gather data on the identified conditions for each case
- if csQCA, determine if each condition and the outcome are “present” or “absent”
- if fsQCA, determine value for each condition and the outcome
- dichotomization (or scoring) “requires judgement and discussion” (Blackman et al, 2013, p. 13)
- determine the sets (“shared configurations of conditions”
- allocate cases to sets
- create the “truth table”
- share with practitioners – allows for “them to help [researchers] develop accounts of causality in more detail”, is a “valuable knowledge exchange opportunity” and “enables [researchers] to incorporate insights from practice into our explanations so that these could be grounded in practitioners’ worlds”
Example
- Blackman et al (2013) conducted a QCA and found that for the outcome of narrowing the gap in teenage pregnancy rates in a local areas that have particularly high teenage pregnancy rates had “causal combinations” for their 27 cases in 5 sets that used different combinations of 4 conditions (they started with 9 conditions, but 5 did not show up in any of the “causal combinations”)
- these sets included:
- narrowing gap was seen in:
- cases with a high proption of black and minority ethnic groups
- cases where there was a combination of low #s in drug treatment and high #s of people under 18 years of age and a “basic” standard of commissioning
- not narrowing gap was seen when:
- lower proportion of black and minority ethnic groups and low #s of under 18s
- lower proportion of black and minority ethnic groups and higher #s in drug treatment
- lower proportion of black and minority ethnic groups and good/exemplary standard of comissioning
- ideas as to why these patterns were seen:
- having a basic standard of commissioning was seen in one ‘good’ combination, but a good/exemplary was seen in a ‘bad’ combination – the authors speculaed that this could be because getting a “good/exemplary” rating may be requiring a lot more meetings/documentation that takes people away from doing the real work
- the lower #s in drug treatment may reflect the level of substance use (which may be similar to level of risk taking generally)
- areas with the higher #s of under 18s may provide more services to under 18s, leading to lower teen pregnancy rates (but only when combined with lower #s in drug treatment (possibly = lower risk taking) and where a basic standard of comissioning is met
- narrowing gap was seen in:
Limitations of QCA
- “a static method [that] does not fully capture the dynamics of complex systems” (Verweij & Gerrits, 2012, p. 51)
- some potential workarounds for this:
- “using multiple iterations of the method (i.e., before, during, and after a certain intervention)
- interpreting he time dimension
- conceptualizing time as (part of) a set
- complementing QCA with other methods” (Verweij & Gerrits, 2012, p. 51)
- some potential workarounds for this:
- can only include a limited number of conditions because the number of possible combinations increases exponentially
- many configurations […] will have no cases […but…] this is not a problem, because, in complexity terms, such configurations can be considered as describing empty attractor states in the possibility space” (Byrne, 2013, p. 224)
- but there will still likely be lots of cases in the “occupied attractor states – configurations” (Byrne, 2013, p. 224) when use lots of conditions in our QCA
- adding a new case can lead to different solution formula – though this “is actually part of the philosophy behind QCA and its roots in systemic thinking. With QCA, the researcher does not strive to identify a single central tendency that reflects reality as more cases are added. rather, it helps researchers to examine the different causal pathways that lead to a particular outcome and how much pathways are linked to individual cases” (Verweij & Gerrits, 2012, p. 46)
Resources:
