Intro to Philosophy – Week 6 – Are Scientific Theories True?

  • this module was not what I expected – I was expecting to learn about the philosophy of science (e.g., positivism, post-positivism, etc.), but instead the whole module was about the debate between scientific realism vs. scientific anti-realism – a debate about the aims of science (rather than a debate on a specific scientific topic)
  • two main aims of science seem to be:
    • science should be accurate and provide us with a good description and analysis of the available experimental evidence in a given field of inquiry. We want “scientific theories to save the phenomenon”
    • science is not just about providing an accurate account of the available experimental evidence  and to save the phenomena, but to “tell us a story about those phenomena, how they came about , what sort of mechanisms are involved in the product of the experimental evidence, etc.
    • [I don’t fully understand what “save the phenomena” means – the instructor in the lecture just says it like we should understand it. Some further elaboration was given in the elaboration on the quiz that appeared in the first lecture, where the course instructor wrote that “saving the phenomena” is also known as “saving the appearances”: providing a good analysis of scientific phenomena as they appear to us, without any commitment to the truth of what brings about those phenomena or appearances” ]
  • Ptolemic astronomers described the motion of planets as being along small circles that were rotating along larger circles; they didn’t necessarily believe this to be what was actually happening, but rather it was a mathematical contrivance that “saved the phenomena” – that is, as long as the calculations agree with observations, it didn’t matter if they were true (or even likely)
  • Galileo, however, “replaced the view that science has to save the appearances, with the view that science should in fact tell us a true story about nature”
  • scientific realism: = “view that scientific theories once literally construed, aims to give us a literally true story of the way the world is.”
    • a semantic aspect to this idea: “once literally construed” means that “we should assume that the terms of our theory have referents in the external world” (e.g., planets are planets. Electrons are electrons.)
    • an epistemic aspect to this idea: “literally true story” – “we should believe that our best scientific theories are true, namely that whatever they say about the world, or better about those objects which are the referents of their terms, is true, or at least approximately true”
  • the “No Miracles Argument” suggests that unless we believe that scientific theories are at least approximately true, the success” of science at “making predictions later confirmed by observation, explaining phenomena, etc.” would be very unlikely
  • constructive empiricism” agrees with the semantic aspect of scientific realism (i.e., we should take the language of science at face value), but disagrees with scientific realism with the epistemic aspect (i.e., it thinks that a theory does not need to be good to be true). They think “Models must only be adequate to the observable phenomena, they are useful tools to get calculations done, but they do not deliver any truth about the unobservable entities” (e.g., atoms, protons, etc. that we cannot observe with the naked eye) – so the theory does not need to be “true” – it just needs to be “empirically adequate”. They think that science is successful because the theories that survive turned out to be the “fittest” (survival of the fittest) – the ones that best “saved the phenomena” over time.
  • Constructive empiricists view the “metaphysical commitment” necessary for scientific realism to be “risky”. If we discover later that something in our theory was non-existent, it would make scientific realism wrong, but not constructive empiricism.
  • The scientific realist would counter that the theories that survive do so because they are true (and those that fail do so because they are false).
  • Another issue is the distinction between observed vs. unobserved. E.g., observing with the “naked eye” and observing with scientific instruments. Why should we believe one more than the other?
  •  Philip Kitcher and Peter Lipton say that “we are justified to believe in atoms, electrons, DNA and other unobservable entities because the inferential path that leads to such entities is not different from the inferential path that leads to unobserved observables”
  • e.g., we know about dinosaurs from fossil evidence – we didn’t observe the dinosaurs ourselves, but can infer from the fossils. Similarly, we can infer Higgs Bosons from the evidence we get from the Large Hadron Collider.
  • “inference to the best explanation” = “we infer the hypothesis which would, if true, provide the best explanation of the available evidence”
Posted in Uncategorized | Tagged , , , , , | Leave a comment

Intro to Philosophy – Week 5 – Should You Believe What You Hear?

  • This week we are talking about “whether and in what circumstances you can believe what other people tell you”
  • will talk about the Enlightenment (1700-1800) – where reason, science, and liberal democracy were on the rise and religion and monarchy were in decline
  • intellectual autonomy was an ideal/virtue in the Enlightenment
  • David Hume is well known for his naturalistic philosophy – no appeal to God/supernatural in his philosophical explanations
  • Hume concluded “you should never believe a miracle occurred on this basis of testimony”
    • testimony = “any situation in which you believe something on the basis of what someone else asserts, either verbally or in writing”
    • a lot of what we believe is based on the testimony of others (we can’t directly experience everything – so we believe lots of things based on what others say or write)
    • “you should only trust testimony when you have evidence that the testifier is likely to be right”
    • evidentialism “a wise man… proportions his belief to the evidence”
  • miracle: “an exception to a previously exceptionless regularity” – e.g., someone rising from the dead – we’ve never seen that happen before
  • by definition, miracles are unlikely, and since we shouldn’t trust testimony unless there is evidence that the testifier is right, we shouldn’t trust a testimony of miracle
  • as well, people are often wrong when they testify (either intentionally (lying) or unintentionally (mistaken)).
  • so Hume concludes that you should never trust a testimony that a miracle occurred
  • Thomas Reid was a minister who challenged Hume’s argument
  • Hume and Reid both believed that we don’t trust our senses only when we have evidence they are likely to be right,
  • So Reid argued that trusting testimony is like trusting your senses,  so we shouldn’t demand we only trust testimony if we have evidence that it’s likely to be right (since we don’t demand that of our sense)
  • Hume & Reid both believed that we are hardwired to think in certain ways – e.g., that we are hardwired to believe our senses
  • Reid further believed that we are hardwired to trust other people’s testimony – “he thought we had an innate “principle of credulity“, which he defined as “a disposition to confide in the veracity of others and to believe what they tell us”
  • Reid noted that small children are very much disposed to believe what people tell them (even more so than adults) (so basically, he thinks it’s natural to believe people, since kids do it the most, and it gets constrained by experience) – he argues that “if our trust in testimony were based on experience (as Hume claims) it would be weakest in children”, but it’s not – “therefore, the principle of credulity is innate and not based on experience”
  • but Hume is talking about what people ought to do – he would say that children should not trust other people without evidence, whereas Reid is talking about what children actually do
  • Reid also believed in a “principle of veracity” = “a principles to speak the truth … so as to convey our real statements” and so “lying is doing violence to our nature”
    • basically, Reid are naturally trusting, naturally honest beings
  • Hume noted many ways that people testify falsely – sometimes we have motives to lie, people enjoy believing what they are told because we “find the feelings of surprise and wonder agreeable”; sometimes we lie because we get pleasure from telling of news (even if it’s not true) [also, one might testify falsely because they are mistaken – they believe what they say is true, but they are actually wrong]
  • Immanuel Kant, German philosopher, wrote: ““Enlightenment is man’s emergence from his self-incurred immaturity. Immaturity is the inability to use one’s own understanding without the guidance of another. […] The motto of the enlightenment is therefore: Have courage to use your understanding.”
    • Kant felt that not trusting another person’s testimony = a virtue; called intellectual autonomy – e.g. don’t believe something just be an authority, a religion tells you to
    • Kant said you should obey what authorities tell you to do, but not obey whai they tell you to think
  • Hume would be a fan of intellectual autonomy; it’s OK to trust other people, but it’s not OK to trust other people blindly
  • Reid held intellectual solidarity  (because we are “social thinkers: our beliefs and opinions are naturally guided by other people”) to be a virtue (rather than intellectual autonomy)
  • Kant appeals to the Latin motto “sapere aude” = “Dare to be wise” or, slightly less literally translated to “dare to know” – he argues that “if you base your beliefs on testimony, they will not amount to knowledge
  • a philosophical tradition, going back to Plato, that says that “Genuine or real knowledge requires what Plato called the ability to “give an account”: the ability to explain, or to situate that knowledge in some broader body of information.” – you can’t get that from testimony
  • so, the value of intellectual autonomy comes from the fact that knowledge/understanding/wisdom is only possible for an intellectually autonomous person
  • another way to look at it is that our beliefs/opinions tend to be passed on from parents to kids, and from people around you (your community) to you
  • Reid would view this as a good tendency, but Hume would be skeptical that this is a good thing
  • if you value progressive/innovative ideas breaking with tradition, you’ll side with Hume, but if you value conservation of your community’s beliefs and don’t like radical breaks from tradition, you’ll side with Reid
Posted in philosophy | Tagged , , , , , , , , , | Leave a comment

Intro to Philosophy – Week 4 – Morality

  • the first lecture explored the “status of morality” – not “is this moral statement correct?” but rather “what is it that we are doing when we make moral statements? are moral statements objective facts? or are they relative to cultural/personal? are they emotional?”
  • empirical judgments are things that we can discover by observations (e.g., the earth rotates around the sun; electricity has positive and negative charges; the Higgs-Boson exists; it was sunny today)
  • moral judgments are things that we judge to be right/wrong, good/bad (e.g., it is good to give to charity; parents are morally obliged to take care of their children; Pol Pot’s genocidal actions was morally abhorrent; polygamy is morally dubious)
  • 3 questions to ask to about these judgments
    1. are they the kinds of things that can be true/false or are they merely opinions? (empirical judgments can be true/false and some philosophers think that moral judgments are merely opinion, though other disagree
    2. if moral judgments can be true/false, what makes them true/false?
    3. if they are true, are they objectively true? (or only true relative to a culture/personal approach)
  • three broad approaches that philosophers have taken to these questions: objectivism, relativism, and emotivism

Objectivism

  • “our moral judgments are the sorts of things that can be true or false, and what makes them true or false are facts that are generally independent of who we are or what cultural groups we belong to – they are objective moral facts”
  • in this approach, if people disagree about morality of something, they are seen as disagreeing over some objective fact about morality
  • e.g., genocide is morally abhorrent – this seems to be something that can be true/false, seems to be objectively true (if someone disagreed, we’d probably thing they are wrong!)
  • e.g., polygamy is morally dubious – but many cultures practice it – perhaps it isn’t objectively true – so this example argues against objectivism
  • objection to objectivism: how can we determine what the empirical truth of a moral claim is? We can’t observe it like we do with empirical judgements.
    • potential responses to the objection: if you take the position that what is right is what maximizes overall happiness, then you can observe which option maximizes overall happiness to make your moral judgements. Or you can say that there are mathematically empirical facts that we can know without observing them in the physical world – instead, you reason them. So we can do the same with morals.

Relativism

  • “our moral judgments are indeed things that can true or false, but they are only true or false relative to something that can vary between people”
  • e.g., the statement “one must drive on the left side of the road” – is true in Britain, but false in the US (so it’s a statement that can be true or false, but whether it is true or false is relative to where you are)
  • e.g., polygamy is morally dubious – can be true or false, but depends on your culture
  • e.g., Oedipus sleeping with his mother was morally bad – (remember, he didn’t know it was his mother) – if you consider incest wrong, is it wrong across the board or only wrong if you know?
  • subjectivism: a form of relativism where “our moral judgments are indeed true or false, but they’re only true or false relative to the subjective feelings of the person who makes them” “X is bad” = “I dislike X”
    • subjectivism has a hard time explaining disagreements
  • cultural relativism: a form of relativism where “our moral judgments are indeed true or false, but they’re only true or false relative to the culture of the person who makes them.” “X is bad” = “X is disproved of in my culture”
  • objection to relativism: it seems like there is moral progress (e.g., people used to think that slavery was morally OK, but now we’ve progressed to say that slavery is morally wrong. However, under a relativism view, you’d say that slavery was morally acceptable relative to the time and culture. So relativism does not allow for moral progress.
    • potential answer to the objection: cultures overlap – so, for example, if you consider “America” a culture

Emotivism

  • “moral judgments are neither objectively true/false nor relatively true/false. They’re direct expression of our emotive reactions”
  • objection to emotivism: we reason our way to moral conclusions – e.g., you might say “it’s wrong for Oedipus to sleep with his mother,” but then someone says “But he didn’t know it was his mother” and then you reason “OK, he can’t be held morally responsible since he didn’t know.” But emotivism says that moral judgments are only based on emotions
    • potential answer to the objection: some evaluations are reason – e.g., if you prefer A to B and prefer B to C, but then you prefer C to A – that’s irrational. So we do use reason when it comes to emotions/preferences.
  • some people in the class discussion asked questions like “Can’t there be a universal principle that unites objectivism and relativism? E.g., a relativist might say “Women should wear headscarves in some cultures but not others, but an objective could say the principle is “When in Rome, do as the Romans do” – which would work out to “Women should wear headscarves in those cultures where that is what is expected and not in other cultures where it is not”. Another discussion point was that we could agree on a moral judgment but disagree on the reason for it (e.g., We agree that kicking dogs is morally wrong, but one might think it’s because you are causing pain to the dog, while another thinks it’s because it desensitizes the person doing it to cruelty”)
  • “Objective” can mean moral principles independent of us, or it can mean moral principles apply to everyone equally (relativists would just object to the latter).
  • Another question from the class was could objectivism be right for some moral principles, relativism is best for other moral principles, and emotivism is best for yet other principles. Philosophers talk about “agent neutrality” – the reasons that morality provide for whether something is moral are independent of the individual and they talk about morality is overriding. If this is correct, you’d expect there is a unified domain of morality.
  • Probably none of these theories are right – they all need some work to figure out which, if any, is correct.
Posted in philosophy | Tagged , , , , , , , | Leave a comment

Intro to Philosophy – Week 3 – Philosophy of the Mind

  • Cartesian dualism: the body is made of material stuff (i.e., stuff that has “extension” (i.e., takes up space)) and the mind is made of immaterial stuff (i.e., does not have extension)
  • Princess Elizabeth of Bohemia was a student of Decartes who brought up the following problem: how can an immaterial mind affect a material body? Our thoughts cause us to do things, but how does the immaterial interact with the material?
  • Another problem is how does the ingestion of a material substance (e.g., psychoactive drugs) affect an immaterial mind (i.e., hallucinations)?
  • Physicalism = “all that exists is physical stuff”
  • Identity theory = one view of physicalism in which “mental phenomena, like thoughts and emotions, etc. are identical with certain physical phenomena”
    • e.g., the mental state of “pain” is identical to a particular type of nerve cell firing
    • a reductionist view – i.e., reduces mental states to physical processes
    • token = instances of a certain type (e.g., Fido and Patches are two tokens of the type “Basset hound)
    • token identity theory = instances of mental phenomena (e.g., a particular pain that I am feeling) is identical to a particular physical state that I’m in
    • type-type identity theory = types of mental phenomena (like “pain” or “sadness”) are identical to types of physical phenomena (e.g., a particular cocktail of neurotransmitters, hormones, etc.)
      • type identity theory is a stronger claim than token identity theory
  • problem with type-type identity theory:
    • a human, an octopus, and an alien can all feel pain, but have very different brain states
    • Hilary Putnam raised this issue of “multiply realisability” in 1967 –  the same mental state can be “realized” from different physical states
    • similarly – currency can be coins & paper in one place, but shells in another place – so currency is “multiply realisable”. It doesn’t matter what they are made of – what matters is how they function.
  • Functionalism = “we should identify mental states not by what they’re made of, but  by what they  do. And what mental states do is they are caused  by sensory stimuli and  current mental states and cause behaviour and new mental states”
    • e.g. the smell of chocolate (a sensory stimulus) causes a desire for chocolate (mental state) may cause the thought (another mental state) “where is my coat?” and the behaviours of putting on coat and going to the store; but if I have a belief that there is chocolate in the fridge, the desire for chocolate could lead to the behaviour of getting the chocolate out of the fridge
    • functionalism gets away from the question of “what are mental states made of?” and instead focuses on what mental states do
  • philosophers often use the computer as a metaphor for mind – a computer is an information processing machine and it doesn’t matter what it’s made of, it only matters what it does
  • this is a computational view of the mind
  • Turing Test – you ask an entity questions and you don’t know if you are talking to a person or a computer. If we can build a computer that can fool the person asking questions into thinking they are human, we have built a computer that is sufficiently complex to say that it can “think” or it has a “mind”
    • some problems with the Turing test:
      • it’s language-based, so a being that can’t use our language couldn’t pass it
      • it’s too anthropocentric – what about animal intelligence? or aliens
      • does not take into account for the inner states of a machine – e.g., a machine that is doing a calculation of 8 + 2 = 10 is going through a process, but a machine that just has a huge database of files and just pulls the answer 10 out of its “8 + 2” file – we wouldn’t want to say that it is “thinking
  • John Searle’s Chinese Room Thought Experiment
    • You are in a room where you get slips of paper with symbols on them delivered to you through an “input” hole in the wall and you have a book that tells you what symbols to write in response to those symbols which you write down on a slip of paper and pass through the “output” slot in the wall. As it turns out, the symbols are Chinese characters and the book is written in such a way that you are giving intelligent answers to the person sending the questions to you. When they receive your “answers”, they are convinced you are a being with a mind that is answering their questions – but you have no idea that it’s questions and no idea what you are responding because you cannot read nor write in Chinese. This is how computers work – they get an input, they are programmed with a list of rules to produce a certain output. But we don’t say that they computer is “thinking” and more than the person in the room understands Chinese. There is no understanding going on within a computer – it doesn’t have a “mind” and if it passes the Turing test, it’s just a really good simulation.
    • syntactic properties = physical properties, e.g., shape
    • semantic properties = what the symbols means/represents
    • a computer only operates based on syntactic properties – it is programmed to responded to the syntactic property of a given symbol with a given response – it does not “understand” its semantic properties
    • aboutness of thought – thoughts are “about” something – they have meaning
  • some problems with the computational view of the mind
    • doesn’t allow us to understand how we can get “aboutness of thought”
    • the “gaping hole of consciousness”
    • the hard problem of consciousness: what makes some lumps of matter have consciousness and others don’t have consciousness?
  • a lot of philosophers were writing when computers were becoming a big deal, so perhaps their thinking was limited by thinking of minds as computers – perhaps we should step away from computational analysis as a metaphor for the mind because it’s limiting our thinking?

 

Follow-up discussion

  • most philosophers use the phrase “intentionality”, which the prof of this session avoided when she talked about “aboutness of thought” because it comes with a lot of philosophical “baggage” that she didn’t want to get into
  • in the discussion forum of the class, people were asking things like “do animals have mind? and how could we know if animals have mind?”
    • one school of philosophy says that you need to have language to have thoughts and since animals don’t have language (as far as we know), they don’t have thoughts
    • but others don’t think this is a fair argument – e.g., if a dog is barking at a squirrel a tree, just because it might not have as “rich” a concept of squirrel as humans do (e.g., a squirrel is a mammal with a bushy tail etc.), we can still infer from its behaviour that it is “thinking” something we can roughly describe as “the dog thinks there’s a squirrel in the tree”
    • she suggests checking out Peter Carrurthers’ work on the animal mind for more information
  • someone in the discussion said that the Turing test doesn’t test if a machine is conscious, but rather it tests at what point humans are willing to attribute conscious states to other things (similar, at what point do infants start to think of other people as having a consciousness?)

 

Posted in notes, online module notes, philosophy | Tagged , , , , , , , , , , | Leave a comment

Intro to Philosophy – Week 2 – Epistemology

Epistemology

  • studying and theorising about our knowledge of the world.
  • we have lots of information, but how do we tell good information from bad information?
  • “propositional knowledge” = knowledge that a certain proposition is the case
  • “proposition” = what is expressed by a declarative sentence, i.e., a sentence that declares that something is the case.
    • e.g., “the cat is on the mat” is a sentence about how the world is
    • it can be true or false
  • not all statements are declarative (e.g., “Shut that door” is not a sentence that declares how the world is. It cannot be true or false).
  • “ability knowledge” = know-how (e.g., knowing how to ride a bike)
  • two conditions for propositional knowledge
    • truth condition – if you know something is the case (e.g., you know that the cat is on the mat), then it has to be true (e.g., the cat really is on the mat)
      • you cannot know a falsehood
      • you can think you know a falsehood, but you cannot actually know it
        • we are interested in when you actually know something, not just when you think you know
    • belief – knowledge requires that you believe it to be true (e.g., if you don’t believe Paris is the capital of France, you cannot have the knowledge that Paris is the capital of France
      • when someone says “I don’t just believe that Paris is the capital of France, I know that Paris is the capital of France.” But this doesn’t mean that belief in a proposition is different in kind from knowledge of that proposition, just that we don’t merely believe it, but that we also take ourselves to know that proposition, and this is indicative of the fact that a knowledge claim is stronger than a belief claim. (i.e., knowledge at the very least requires belief).
  • this doesn’t mean you have to be infallible or certain, but if you are wrong about the fact, then you didn’t really know it (you just thought you did)
  • also, when we talk about propositional knowledge, we aren’t talking about knowledge that something is likely or probably true – we are talking about something that either is or is not true
    • we do sometimes “qualify” or “hedge” our knowledge claims (perhaps because we are unsure), but we are really concerned with actual true
  • knowledge isn’t just about getting it right – it also requires getting to the truth in the right kind of way
    • e.g., imagine a trial where the accused is, in fact, guilty. One juror decides that the accused is guilty based on considering the evidence/judge’s instructions/the law, while another juror decides the accused is guilty based on prejudice without listening to any of the evidence. Although they both “got it right” (i.e., what they believe is true), the first juror knows the accused is guilty, but the second juror does not know it
  • there are two intuitions about the nature of knowledge:
    • anti-luck intuition – it’s not a matter of luck that you ended up at the right answer; you actually formed your belief in the right kind of way (e.g., considering the evidence, making reasoned arguments), not that you got to the truth randomly/by chance
    • ability intuition – you get to the truth through your ability (e.g., the juror who used prejudice and happened to get the right answer did not get to the right answer by their abilities)

The Classical Account of Knowledge and the Gettier Problem

  • knowledge requires more than just truth and belief – but what is it that is required?
  • the classical account of knowledge (a.k.a., the tripartite account of knowledge):
    1. the proposition is true
    2. one believes it
    3. one’s belief is justified (i.e, you can offer good reasons in support of why you believe what you do)
  • until the mid-1960s, this classical account of knowledge was accepted by most people
  • but in 1963, Edmund Gettier published a 2.5 page paper that demolished this account – he showed some counter-examples of situations that fit the three above named criteria, but where people don’t know – they actually come to their belief by luck
  • his examples were very complicated, but here are some simple counter examples (we can call them Gettier-style cases)
  • e.g., the stopped clock example:
    • you come downstairs and see the clock says 8 am and you believe, based on the justification that this clock has always been reliable, that it is 8 am. And it happens to be 8 am. So you have a justified true belief (i.e., it satisfies the classical account of knowledge). But imagine the clock stopped 12 hours ago, but you just happened to look at the clock when it was 8 am – so you got it right by luck. So you cannot actually know the time based on looking at a stopped clock!
  • e.g., the sheep case
    • a farmer looks out his window, sees what looks like a sheep, and believes there is a sheep in the field. There is a sheep in the field, but it is hidden behind a sheep-shaped rock, which is what the farmer actually saw. So his belief is true (i.e., there is a sheep in the field) and he has a justification (he sees what looks like a sheep in the field), but he got it right only by sheer luck that there was a sheep hidden behind that rock. If there had not been a sheep hidden behind that rock, he would be believe there was a sheep in the field and he would be wrong. So he does not actually know there is a sheep in the field (he just thinks he knows and happens to be right just by luck)
  • people try to attack Gettier-style cases – e.g., asking “does the farmer really believe that there is a sheep in the field or do they believe that the rock is a sheep?” because if it is the latter, then they have a false belief (i.e., the rock is not a sheep) and thus it does not violate the classical account of knowledge – but this is just attacking a single case – to knock down Gettier-style cases in general, you’d need to think about Gettier-style cases as a whole and find a way to blow up the whole thing
  • there is a general formula for constructing Gettier-style cases
    • take a belief that is formed in such a way that it would usefully result in a false belief, but which is still justified (e.g., looking at something that looks like a sheep, or looking at a stopped clock)
    • make the belief true, for reasons hat have nothing to do with the justification (e.g., hidden sheep, happening to look at the stopped clock at the right time)
  • at first, people thought there would be some simple fix (e.g., adding a fourth condition onto the classical account), but after much trying, no one has found a way to do this
  • one example of how someone tried
    • Keith Lehrere proposed adding a fourth conditions that says the subject isn’t basing their belief on any false assumptions (a.k.a., “lemmas”)
    • this sounds like a reasonable approach
    • but what do we mean by “assumptions”?
    • a narrow definition of “assumptions” = something that the subject is actively thinking about (but you don’t look at the clock and actively think “I assume the clock is working” – you just believe it is without actively thinking about that assumption)
    • a broad definition of “assumptions” = a belief you have this is in some sense germane to the target belief in the Gettier-style case (e.g., you do believe the clock is working even though you aren’t actively thinking that) – but this is so broad that it will exclude genuine cases of knowledge because of all the things we believe, some of them may false, so then we’d exclude genuine cases of knowledge
  • two questions raised by Gettier-style cases
    1. is justification even necessary for knowledge?
    2. how does one go about eliminating knowledge-undermining luck?
  • so, it really is not that obvious what knowledge is

Do We Have Any Knowledge?

  • radical skepticism contends that we don’t know nearly as much as we think we know – and in its most extreme form suggests that we can’t know anything
  • skeptical hypotheses are scenarios that are indistinguishable from normal life, so you can’t possibly know they aren’t occurring
    • e.g. brain-in-a-vat – if you were a brain in a vat being feed the necessary nutrients to stay alive and being fed fake experiences
    • there is no way to know this isn’t true because any “evidence” you can provide against it (e.g., I can feel objects around me, I can have a conversation with you) could be explained by the situation of being a brain in a vat (e.g., your brain is being fed signals that make it appear that you can feel objects or have a conversation)
    • note that radical skepticism isn’t saying you are a brain-in-a-vat or even that it’s likely that you are a brain-in-a-vat. It’s just asking “How would you know that you aren’t a brain-in-a-vat?” And really, you can’t know.
Posted in notes, online module notes, philosophy | Tagged , , , , , | Leave a comment

Report on “Delivering the Benefits of Digital Health Care”

Delivering_the_benefits_of_digital_health_careA report on “Delivering the benefits of digital health care” from Nuffield Trust in the UK recently came across my desk. It covers a bigger scope of technology than the project I’m working on (which is a project about transforming clinical care (and implementing an electronic health record across three large health organizations to support this clinical transformation), but does not include telehealth and some of the other IT “solutions” talked about in this report), but some of the “lessons learned” that they share resonate with what we are doing.

Some highlights:

Clinically led improvement, enabled by new technology, is transforming the delivery of health care and our management of population health. Yet strategic decisions about clinical transformation and the associated investment in information and digital technology can all too often be a footnote to NHS board discussions. This needs to change. This report sets out the possibilities to transform health care offered by digital technologies, with important insight about how to grasp those possibilities and benefits from those furthest on in their digital journey” (p. 5, emphasis mine)

  • this report suggests that rather than focusing on the technology with an eye to productivity gains, “the most significant gains are to be found in more radical thinking and changes in clinical working practices” (p. 5).
    • it’s “not about replacing analogue or paper processes with digital ones. It’s about rethinking what work is done, re-engineering how it is done and capitalising on opportunities afforded by data to learn and adapt.” (p. 6)
    • This reminds me of what my IT management professor in my MBA program liked to say: “If you automate a mess, all you get is an automated mess”. It’s much better to focus on getting your processes right, and then automating them, rather than just automating what you have.
    • “”It’s fundamentally not a technology project; it’s fundamentally a culture change and a business transformation project” (Robert Wachter, UCSF)” (p. 22)
  • in a notable failure, the NHS in the UK spent 9 years and nearly £10 billion and failed to digitise the hospital and community health sectors with reasons for the failure being “multiple, complex, and overlapping” including “an attempt to force top-down change, with lack of consideration to clinical leadership, local requirements, concerns, or skills” (p. 14)
  • it is noted that implementing an electronic health record (EHR) [which is what the project I’m working on is doing) is particularly challenging
  • they also note that things take longer than you expect:
    • “The history of technology as it enters industries is that people say ‘this is going to transform everything in two years’. And then you put it in and nothing happens and people say ‘why didn’t it work the way we expected it to?… And then lo and behold after a period of 10 years, it begins working.” (Robert Wachter, University of California San Francisco (UCSF)” (p. 20)
  • and they note that “the technologies that have released the greatest immediate benefits have been carefully designed to make people’s jobs or the patient’s interaction easier, with considerable investment in the design process.” (p. 20)
  • poorly designed systems, however, can really decrease productivity
  • getting full benefit of the system “requires a sophisticated and complex interplay between the technology, the ‘thoughtflow’ (clinical decision-making) and the ‘workflow’ (the clinical pathway)” (p. 21)
  • systems with automated data entry (e.g., bedside medical device integration, where devices that monitor vital signs at the bedside automatically enter their data into the EHR, without requiring a clinician to do it manually) really help maximize the benefits

Seven Lessons Learned

  1. [Clinical] Transformation First
    • it’s a “transformation programme supported by new technology” (p. 22)
  2. Culture change is crucial
    • “many of the issues face […] are people problems, not technology problems” (p. 23)
    • you need:
      • “a culture that is receptive to change
      • a strong change management process
      • clinical champions/supporting active staff engagement” (p. 23)
  3. User-centred design
    • you need to really understand the work so that you design the system to meet the needs of the clinician
    • “the combination of a core package solution with a small number of specialist clinical systems is emerging as the norm in top-performing digital hospitals” (p. 8)
  4. Invest in analytics
    • data analytics allows you to make use of all the data you collect as a whole (in addition to using it for direct clinical care)
    • requires “analytical tools available to clinicians in real time” (p. 8)
  5. Multiple iterations and continuous learning
    • you aren’t going to get it right the first time, no matter how carefully you plan [this is something that our new Chief Clinical Information Officer is always reminding us of] and so you will need “several cycles – some quite painful – before the system reaches a tipping point where all of this investment starts to pay off” (p. 26)
  6. Support interoperability
    • to provide coordinated care,  you need to be able to share data across multiple settings
    • “high-performing digital hospitals are integrating all their systems, to as low a number as possible, across their organisation” (p. 9)
  7. Strong information governance
    • when you start to digitize patient information, the size and scope of privacy issues change (i.e., while there is risk that an authorized person could look at a patient’s paper record or paper records could be lost when being transported between places, with digitized record there is a risk that all of your patients’ record could be accessed by an unauthorized person and that it is much easier to search electronic records for a specific person, condition, etc.)
    • you need “strong data governance and security” (p. 9)

Seven Opportunities to Drive Improvement

  1. More systematic, high-quality care
    • health care “often falls short of evidence-based good practice” (p. 31)
    • “technologies that aid clinical decision-making and help clinicians to manage the exponential growth in medical knowledge and evidence offer substantial opportunities to reduce variation and improve the quality of care” (p. 31)
    • integrated clinical decision support systems and computerized provider order entry systems:
      • reduce the likelihood of med errors (they cite a review paper (Radley et al, 2013) [which I have now obtained to check out what methods the papers they reviewed used to measure med errors]
      • reduced provider resource use
      • reduced lab, pharmacy & radiology turnaround times
      • reduced need for ancillary staff (p. 32)
    • at Intermountain Healthcare, “staff are encouraged to deviated from the standardised protocol, subject to clear justification for doing so, with a view to it being refined over time” (p. 34) – “hold on to variation across patients and limit variation across clinicians” (p. 35) as “no protocol perfectly fits each patient” (p. 35)
    • need to avoid alert fatigue – by only using them sparingly (or else they will get ignored and the really important ones will be missed) and targeting them to the right time (e.g., having prescribing alerts fire while the provider is prescribing)
    • be on the lookout for over-compliance – “Intermountain Health experience problems where clinicians were too ready to adopt the default prescribing choice, leading to inappropriate care in some cases” (p. 37)
  2. More proactive and targeted care
    • “patient data can be used to predict clinical risk, enabling providers to target resources where they are needed most and spot problems that would benefit from early intervention” (p. 38)
    • drawing on patient data, computer-based algorithms “can generate risk scores, highlighting those at high risk of readmission and allowing preventative measures to be put in place” (p. 39)
    • “it may also have a role in predicting those in the community who are likely to use health care services in the near future” (p. 39)
    • “monitoring of vital signs, [which are then] electronically recorded, [can be used to] calculate early warning scores [and] automatically escalate to appropriate clinicians [and] “combine these data with laboratory tests to alert staff to risks of sepsis, acute kidney injury or diarrhoeal illness” (p. 39)
      • Cerner estimates using early warning system for sepsis “could reduce in-hospital patient mortality by 24% and reduce length of stay by 21%, saving US$5,882 per treated patient” (p. 41)
    • there’s also opportunity to “check a patient’s status from remote location within the hospital, as well as facilitating handover between staff and task prioritisation using electronic lists” (p. 39)
    • monitoring of vital signs throughout the whole hospital is best to maximize benefits
    • predictive analytics is only as good as the quality of the data you put into the system
    • lots of data is unstructured – need to find ways to use these data (e.g., natural language processing)
  3. Better coordinated care
    • coordinated care leads to a better care experience, reduces risk of duplication or neglect
    • “if all health care professionals have access to all patient information in real time, there is significant potential to reduce waste (e.g., duplication of tests). It can help make sure things are done at the right time, at the right place and not overdone” (p. 45)
    • “chasing  report or a result […is…] an inefficient use of time, effort and energy and doesn’t really give confidence to the patient and carers” (p. 47)
    • but note that “systems to share results/opinions digitally can remove the opportunity for informal exchange of views and advice across teams, which often enrich and improve clinical decision-making” (p. 48), so alternative ways of doing this may need to be provided.
  4. Improved access to specialist expertise
    • telehealth (not part of the project I’m working on)
  5. Greater patient engagement
    • this section referred to tools, like wearable tech (e.g., Fitbit) or patient portals that empower patients to take more control of their own health  (not part of the project I’m working on)
    • “patient co-production of data into a hospital EHR will redefine the interaction with care services” (e..g, questionnaires that patients fill out before they even come to the healthcare facility, tracking of long-term data (e.g., blood pressure, weight))
  6. Improved resource management
    • e-rostering (i.e., of staff), patient flow management, business process support (e.g, HR, facilities, billing) all discussed (not relevant to the project I’m working on)
    • ability of staff to remotely access health records “can transform the way hat staff in the community deliver care” (p. 66)
  7. System improvement and learning
    • “feeding learning from clinical and non-clinical data back into existing processes is essential to fully realising the benefits of digital technology” (p. 70)
    • Intermountain Healthcare:
      • captures 3 type of data:
        • intermediate & final clinical outcomes
        • cost data
        • patient satisfaction and experience
      • “clinical registries are derived directly from clinical workflows” – currently has “58 condition-specific registries – tracking a complete set of intermediate and final clinical and cost outcomes by patient” (e.g., 71)
      • remember that data collection is costly, so only collect data routinely if you are using it for some purpose that adds value (“Intermountain Healthcare does this through small individual projects, before building data collection into existings processes”) (p. 76)

What could the future look like?

  • operational improvement from:
    • combining impact of a bunch of small changes [this assumes that (a) the different elements of the system are additive, as opposed to complex, and (b) the “benefits” outweigh the unintended negative consequences]
    • getting the “full benefit” out of all the technologies (i.e., it will take time for people to implement the available technologies and to optimize their use) [this doesn’t even include technologies that are not yet available)
  • “benefits” they expect are most likely to see:
    • “reduced duplication and rework
    • removing unjustified variation in standard clinical processes
    • identifying deteriorating patients and those at risk
    • predicting the probability of an extended stay or readmission
    • cutting out unnecessary steps
    • improving communication and handoffs
    • removing administrative tasks from clinical staff
    • scheduling and improving flow
    • inventory & procurement management
    • rostering, mobile working, and staff deployment
    • patient self-service for administrative tasks such as booking
    • other automation, e.g., robotics in back office” (p. 80-1)
  • redesigning the whole pathway:
    • “reduced variation
    • ability to ensure the most appropriate level of care
    • fitting staffing skill mix to demand more effectively” (p. 81)
  • population health management
    • “early intervention & targeting
    • enabling patient self-management
    • shared decision-making
    • measuring outcomes and value rather than counting activities” (p. 82)
      • all this requires better data and analytics, learning & improvement processes, and supporting patients with self-management and supporting shared decision-making (p. 82)

“Early strategic priorities should be the areas where technology is able to facilitate some relatively easy and significant wins. Most notable are the systematic and comprehensive use of vital signs monitoring and support for mobile working. In the short to medium term, the use of EHRs, telehealth, patient portals and staff rostering apps can also generate savings and improve quality. However, these require sophisticated leadership with support for organisational development and change management to ensure that the full benefits are realised. In the longer term, the really big benefits will come from the transition to a system and ways of working premised on continual learning and self-improvement.” (p. 88, emphasis mine)

Potential intended consequences mentioned in the report:

  • decreased productivity if the system is poorly designed (e.g., time spent on data entry, time spent responding to unhelpful alerts)
  • “over-compliance” – “Intermountain Health experience problems where clinicians were too ready to adopt the default prescribing choice, leading to inappropriate care in some cases” (p. 37)
  • “systems to share results/opinions digitally can remove the opportunity for informal exchange of views and advice across teams, which often enrich and improve clinical decision-making” (p. 48),

Limitations:

  • they noted there was little evidence on this type of work in the literature, particularly in terms of return on investment
Imison, C., Castle-Clarke, S., Watson, R., & Edwards, N. (2016).Delivering the benefits of digital health care. Nuffield Trust. [Download the full report.]
Posted in healthcare, information technology, notes | Tagged , , , , , , , , , , , , , , | Leave a comment

Implementation Science

Trying to avoid falling into yet another rabbit hole of reading (this time on “Implementation Science”1There’s another rabbit hole of “Program Science” awaiting me as well!), but here are notes from a couple of papers I’ve read trying to get the lay of the land on this.

Implementation Matters

Diffusion or Technology Transfer = “the spread of new ideas, technologies, manufactured products […] or […] programs” (p. 327)

  • Phases of program diffusion:
    dissemination: “how well information about a program’s existence and value is supplied to” end users
  • adoption: whether end users “decides to try the new program”
  • implementation: “how well the program is conducted during a trial period”
  • sustainability: “whether the program is maintained over time” (p. 327)

8 aspects of implementation:

  • fidelity: “the extent to which the innovation corresponds to the originally intended program (a.k.a. adherence, compliance, integrity, faithful replication)”
  • dosage: “how much of the original program has been delivered (a.k.a. quantity, intervention strength)”quality: “how well different programs have been conducted”
  • participant responsiveness: “degree to which the program stimulates the interest or hold the attention of participants”
  • program differentiation: “extent to which a program’s theory and practices can be distinguished from other programs (a.k.a. program uniqueness)”
  • monitoring of control/comparison conditions: “describing he nature and amount of services received by members of these group (treatment contamination, usual care, alternative services)”
  • program reach: “rate of involvement and representativeness of program participants (participation rates, program scope)”
  • adaptation: “changes made in the original program during implementation (a.k.a. program modification, reinvention) (p. 329)

In order to evaluate whether a program –> outcomes, you need to monitor:

  • how the program is being implemented:
    • so you know what you are actually evaluating
    • because negative results could occur because you didn’t actually implement as planned (and if you don’t monitor what is actually implemented, you would come to the incorrect conclusion that the program doesn’t work)
    • because positive results could come from an innovation that was implemented instead of what was planned (and if you don’t monitor what is actually implemented, you would come to the incorrect conclusion that the program works and would miss out on being able to sustain/spread the innovation that actually does work)
  • what the comparator group is actually getting (so you know what you are actually comparing your program to)

It’s important to find the right mix of fidelity and adaptation. Although fidelity can –> improved outcomes, no program is implemented with 100% fidelity and some adaption to local context can also improve outcomes; so, it is important to “find the right mix of fidelity and adaption”. Importantly, you need to “specify the theoretically important components of interventions, and to determine how well these specific components are delivered or altered during implementation. This is because core program components should receive emphasis in terms of fidelity. Other less central program features can be altered to achieve a good ecological fit.” (p. 341)

Source: Durlak, J. A., & DePre, E. P. (2008) Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. Am J Community Psychol 41: 327-350.

Making sense of implementation theories, models
and frameworks

  • implementation science came from struggles with getting research into practice; often attempts to implement evidence-based practice were not based in an explicit strategy/theory and it was hard to “understand and explain how and why implementation succeeds or fails, thus restraining opportunities to identify factors that predict the likelihood of implementation success and develop better strategies to achieve more successful implementations.” (p. 1)
  • in response, researchers have created a lot of theories and used some from other disciplines and now people find it difficult to pick a theory to use
  • Implementation Science: “the scientific study of methods to promote the systematic uptake of research findings and other EBPs into routine practice to improve the quality and effectiveness of health services and care” (p. 2)
  • diffusion – dissemination – implementation continuum
    • diffusion – practices spread through passive, untargeted, unplanned mechanisms
    • dissemination – practices spread through active mechanisms/planned strategies
    • implementation – “the process of putting to use or integrating new practices within a setting” (p. 2)
  • theory – “a set of analytical principles or statements designed to structure our observation, understanding and explanation of the world” (p. 2) – usually described as “made up of definitions of variables, a domain where the theory applies, a set of relationships between the variables and specific predictions. A “good theory” provides a clear explanation of how and why specific relationships lead to specific events” (p. 2)
  • model – ” a deliberate simplification of a phenomenon or a specific aspect of a phenomenon. Models need not be completely accurate representations of reality to have value”.
    • not always easy to distinguish betwen a “model” and a “theory” – “Models can be described as theories with a more narrowly defined scope of explanation; a model is descriptive, whereas a theory is explanatory as well as descriptive” (p. 2)
  • framework – “a structure, overview, outline, system or plan consisting of various descriptive categories, e.g. concepts, constructs or variables, and the relations between them that are presumed to account for a phenomenon. Frameworks do not provide explanations; they only describe empirical phenomena by fitting them into a set of categories”
  • in implementation science:
    • “theory usually implies some predictive capacity […] and attempts to explain the causal mechanisms of implementation”
    • models “are commonly used to describe and/or guide the process of translation research into practice […] rather than to predict or analyse what factors influence implementation outcomes”
    • frameworks “often have a descriptive purpose by pointing to factors believed or found to influence implementation outcomes” (p. 3)
      • models and frameworks are typically checklist and don’t specify mechanisms of change

 

  • there is overlap among these five categories
  • “the use of a single theory that focuses only on a particular aspect of implementation will not tell the whole story. Choosing one approach often means placing weight on some aspects (e.g. certain causal factors) at the expense ofo thers, thus offering only partial understanding. Combining the merits of multiple theoretical approaches may offer more complete understanding and explanation, yet such combinations may mask contrasting assumptions regarding key issues. […] Furthermore, different approaches may require different methods, based on different epistemological and ontological assumptions.” (p. 9)
  • research is needed to determine if use of theories/models/frameworks does, in fact, improve implementation
Source: Nielsen, P. (2015). Making sense of implementation theories, models and frameworks. Implementation Science. 10:53. (full text)

Footnotes   [ + ]

1. There’s another rabbit hole of “Program Science” awaiting me as well!
Posted in evaluation | Tagged , | Leave a comment

Intro to Philosophy – Week 1 – What is Philosophy

As I’ve been doing so much reading on things like theory, complexity science, and research methodology, I’ve been reading more and more papers with words like epistemology and ontology, and it’s prompted me to do a bit of a refresher on philosophy. I’ve never actually done a Philosophy 101 type of course – my philosophy education has been pieced together from a few philosophy courses during my various university degrees (specifically, biomedical ethics and critical thinking courses during my undergrad and a business ethics course from the philosophy department during my MBA), an ethics training course I took in my previous job so that I could serve as a consultant with the organization’s ethics department 1Sadly, I never got to put that into practice, as I left that organization for my current job before an ethics consult came up that I could participate in., and reading Sophie’s World and The Matrix and Philosophy. So I figured I should go back to basics and thus have enrolled in Introduction to Philosophy, offered by the University of Edinburgh on Coursera.

What is Philosophy?

  • is both an subject and an activity – philosophy is what philosopher’s do
  • Dr. Ward’s definition “the activity of working out the right way of thinking about things”
    • all subjects try to think about their domains in the right way, but what makes philosophy different is that working in other subjects involves doing the work of, say, physics, (e.g., collecting data, developing theories, testing theories), while philosophy involves stepping back from that work and working out the right way to think about things (e.g., “what does it mean by “physical reality”? “what distinguishes a good scientific theory from a bad one?”)
    • another example: medicine involves trying to treat people’s illness based on an understanding of the best available medical theory. That used to mean trying to balance black bile and yellow bile, because it was believed that diseases were caused by imbalances in the “humours”. But philosophy of medicine involves stepping back and thinking about how we understand “health” and “illness” (e.g., perhaps noticing that balancing black bile and yellow bile didn’t actually work to heal people would lead people to think about their understanding of causes of illness).

 

  • philosophical questions can arise from anywhere (you can always step back and ask questions about how you are thinking about something)
  • similar to when children ask you “why?” and when you answer, they ask you “why?” again, etc. – in this analogy, the philosopher is in the role of the child asking “why?” and in the role of the person trying to come up with answers
  • since we can ask philosophical questions about anything, they can be trivial, but they can also be very important (e.g., if we hadn’t asked philosophical questions about the way medicine was understood, we’d still be using leeches to treat most diseases)
  • in the past, people found it acceptable to enslave others or commit genocide – but today those ideas are indefensible. We don’t know which of our current beliefs/values will be looked back on as indefensible (e.g., farming animals for food; our treatment of the planet) – philosopher’s will step back and ask questions about these things

How Do You Do Philosophy?

  • the best way to learn how to do philosophy is to do it
  • working out the best way to think about something is something that we do naturally
    • we look around for evidence, we think about what that evidence means, and draw a conclusion
  • “argument” means evidence and a line of reasoning to support a conclusion
  • “premises” = claims made to support a conclusion
  • we examine arguments to see if we think they are good arguments
    • if the argument’s conclusion follows from the premises – i.e., if the premises are true then the conclusion must be true, then the argument is valid
    • we can question the truth of the premises – so even if the conclusion follows from the premises, if one or more of the premises are not true, then the argument does not support the conclusion
  • if an argument is valid and its premises are true, then we say it is a sound argument

An argument against free will:

  1. The way the world was in the past controls exactly how it is in the present, and how it will be in the future
  2. We’re part of the world, just like everything else.
  3. We can’t control how things were in the past, or the way the past controls the present and future
  4. Therefore, we don’t control anything that happens in the world, including all the things that we think, say and do
  • we can question the premises – e.g., some people think humans aren’t a part of the world like everything else because we have “souls”; or perhaps there is some indeterminacy on the effect of the future based on what happened in the past
  • when questioning the premises, you’ll then have more work to do to support your thoughts around if the premises is true or not
  • it is hard, but useful work, to clarify our thinking, our premises, our arguments
  • it is also useful to keep in mind the “big picture” – philosophy is not just about constructing clever arguments, but also thinking about why these issues are important

Is There a “Right” Way to Think about Things?

  • David Hume thought a skeptical attitude was the appropriate way to approach philosophy
  • he felt that it was important that philosophy stay true to our sensory experience of the world, which he felt was
    • e.g., causation – Hume argued that we can’t really know causation – e.g., when we see one billiard ball hits another and the other moves off, we attribute the causation (i.e., we add the notion of causation with our mind), as all we actually see is the behaviour of the two balls
    • he also thought there wasn’t really a “self” – all we really experience is our thoughts/feelings/impressions as they pass through our minds, but our mind adds something extra that we think of as our “self” – we don’t observe a “self” above and beyond the thoughts/feelings/impressions
    • he also thought there was no reason based on our sensory experience of the world to believe an omnipotent, omniscience “God”
  • he didn’t think there was a “right” way to think about the world
  • Immanuel Kant thought the possibility of a world that didn’t conform to the rules and patterns that our mind imposes on experience was nonsensical
    • the rules that govern our thought are the same as rules that govern the world, and that we can know this just by thinking about it. So, for Kant, there is a right way of thinking about things, and we can arrive at it by the clear and careful use of reason.

Footnotes   [ + ]

1. Sadly, I never got to put that into practice, as I left that organization for my current job before an ethics consult came up that I could participate in.
Posted in notes, online module notes, philosophy | Tagged , | Leave a comment

CES Webinar: Words of Evaluation

Title: Words of Evaluation: A terminological dictionary to clarify communication in evaluation practice
Speakers: Richard Marceau, Francine Sylvain, Ghislan Arbour, Frank Hogg
Offered by: Canadian Evaluation Society

  • project at the École nationale d’administration publique (ENAP) in Quebec City to create a terminology dictionary for people working in evaluation in French (2014)
  • then they realized that they could extend this work to English (currently working on it)
  • not just a matter of translating the French dictionary into English, but rather:
    • extracting the conceptual system
    • apply the methodology for terminology dictionary
  • what are the needs of evaluators when it comes to communication?
    • evaluators sometimes run into communication issues because they use different words to refer to the same thing (or the same word to refer to different things)
    • communication between evaluators and clients – even moreso!
    • as evaluators, our product is information, which is made of words, so we need to have clear communication!
  • challenges in evaluation language:
    • inaccuracy
    • incoherence – e.g., if you are writing a paper about needs assessment, you may just define terms relevant to that, but when others do the same for other types of/aspects of evaluation, but when you try to put it all together, you don’t have a coherent system
    • jargon – not known by not experts
  • What do we currently have?
    • evaluation textbooks contain specialized knowledge (but don’t solve our jargon issue and may not solve the incoherence problem)
    • general dictionaries use general language, but don’t contain the specialized knowledge, so may not work for our purposes
  • a terminological dictionary is meant to be a blend of the two – contain specialized knowledge but attempts to provide a coherent system
    • vertical coherence
    • horizontal coherence – e.g., if you have a definition of “program” and a definition of “evaluation”, then your definition of “program evaluation” should make sense in terms of the first two definitions
  • their terminological dictionary is focused on performance of programs (not on evaluation methodology or sociology of evaluation) – not designed to do research on evaluation, but rather to support evaluation practice
  • stay tuned for the release of the English version

Very similar slide deck to the one presented is available here.

Posted in evaluation, evaluation tools, notes, webinar notes | Tagged , , , , , , | Leave a comment

Qualitative Comparative Analysis

While researching evaluating complex programs/complex systems, I came across an evaluation approach/method that I wasn’t familiar with: Qualitative Comparative Analysis (QCA). So I’ve done a bit of reading on this to see if it might be something I can use in my work.

Background

  • evaluations face the tension between contextualization (i.e., needing to understand the context in which an intervention occurs, because context (in addition to the intervention) can affect outcomes) and generalization (i.e., being able to inform future practice by identifying recurring patterns of what works and what does not work)
  • “project evaluation has tended to focus on the comparison of ‘before and after’ situations, and has not adequately incorporated the influence of contextual local conditions on infrastructure project development” (Verweij & Gerrits, 2012, p. 41)
  • there is a “misfit between the way infrastructure development projcets are understood and the methodologies used to evaluate them” (Verweij & Gerrits, 2012, p. 41)
  • variable-oriented studies can allow the identification of generic patterns (e.g., rail projects lead to the biggest cost overruns among transportation infrastructure projects), but don’t account for local contexts that also affect outcomes, while case-based studies allow an understanding of the affect of context, but don’t allow for identifying patterns for generalization. QCA is meant to integrate these two approaches
  • when we say that an infrastructure project is “complex”, “it usually means that it is perceived to be difficult”  (Verweij & Gerrits, 2012, p. 42), but complexity is more than that – it is a “multi-layered concept” (p. 42):
    • comprises a “mixture of generic elements [e.g., suburbanization] and local conditions [e.g., specific features of geography that affect how suburbanization could (or could not) happen at a specific site]” (p. 43)
    • “becomes even more complex if its social fabric is taken into account”  (Verweij & Gerrits, 2012, p. 43)
    • “thus, the local built order emerges from interaction between generic and specific physical and social elements”  (Verweij & Gerrits, 2012, p. 43)
  • it is important that “this specific pattern of local conditions and generic developments is researched to understand ex-ante how a project should be executed, and to understand ex-post what leads to certain outcomes”  (Verweij & Gerrits, 2012, p. 43)
  • to understand complex infrastructure projects
    • they “take place within a specific interacting mix of local conditions and generic patterns that occurs in any given location”
    • “the causal relationship between site-specific conditions and generic developments are poorly known and, if known, only for that specific time and place” – so “known causal relationships specific to a certain area are by definition case-specific”
    • “the emergent nature of any built area implies that it is the result of longitudinal development”, i.e., it is the result of past changes and events that are to some extent path-dependent” (Verweij & Gerrits, 2012, p. 43)
  • types of complexity:
    • generic complexity “focuses on the emergence of complex processes and structures from a limited set of variables. It assumes a general set or rules from which emergent complexity flows”… but it is missing that the “emergent nature of infrastructure projects is partly determined by local systems” (Verweij & Gerrits, 2012, p. 44)
    • situated complexity focuses “on the explanatory value of the contextualization of a phenomenon”. Buijs et al “argue that while open systems ‘do not operate according to general rules applied in all context’, a systematic comparison can reveal differences and similarities between the operations of different systems. This approach to situated complexity focuses both on recurring patterns over multiple systems and the idiosyncratic events in particular systems, since both determine how systems develop over time” (Verweij & Gerrits, 2012, p. 44)
  • how can complexity be understood, “which is basically a question of how reality can be understood”
    • positivism: “primarily concerned with determining general rules by taking reality apart in discrete components” (Verweij & Gerrits, 2012, p. 44) – similar to generic complexity
    • postpositivism: “has many different sub-strands that range from the extreme relativism social constructivism to the more realist thesis of negotiated subjectivism or critical realism” (“the common theme within those strands is that the contextualization is explanatory for what is being observed”) (Verweij & Gerrits, 2012, p. 44) – similar to situated complexity
  • “if systems are said to be open, then it follows that their boundaries do not exist a priori, and any individual will develop a particular demarcation or set of boundary judgements about the system which includes and excludes variables (i.e., a reduction of real complexity) that may be connected but not perceived as such by the observer. Thus, there is no unambiguous separation between systems and their context, and the observer is as much part of the complexity as the system or agents that are observed. Situated complexity is therefore not confined to the presupposed demarcations of system but intersects all system representation by respondents.” Verweij & Gerrits, 2012, p. 44)
    • implications of this for evaluation:
      • people choose the boundaries of the system
      • we should take multiple perspectives into account
      • “cause and effect relations do exist and can be known through respondents’ perceptions” (Verweij & Gerrits, 2012, p. 45)
      • evaluators cannot be separated from the evaluation

Criteria for a complexity-informed methodological framework

  • “it balances between in-depth understanding and reductionist generalization” (Verweij & Gerrits, 2012, p. 45)
  • “cased-based”, where “projects are treated holistically an not as collection par parts” (p. 46); with multiple cases that are compared “to allow for causal inference – for studying patterns across cases” (Verweij & Gerrits, 2012, p. 46)
    • this “rejects the idea that variables can be disaggregated from cases and analysed separately as if it is the variables rather than the cases that are causal. […] It is the ‘case’ and the state of important conditions of each case” (they are “bundles of conditions that interact together”) (Blackman et al, 2013, p. 4)
  • ” allow the observation and analysis of complex interaction between the variables” (Verweij & Gerrits, 2012, p. 45)
  • consider how situated complexity came into being over time (i.e., complex dynamics)” (Verweij & Gerrits, 2012, p. 45)

Qualitative Comparative Analysis (QCA)

  • an umbrella term for:
    • crisp set QCA (csQCA) – where conditions are scored as binaries (present (1) or absent (0))
    • multi-value QCA (mvQCA)
    • fuzzy set QCA (fsQCA) – allows conditions to be scored as a gradient (e.g., could be 0, 0.5, 1, 1.5, etc.)
  • “aims to integrate the case-oriented and variable-oriented approaches”  (Verweij & Gerrits, 2012, p. 46)
  • “can be used to achieve a systematic comparison across a smaller number of individual cases in order to preserve complexity, and yet being as parsimonious as possible and illuminating otherwise hidden causal paths on a micro level’ (Rihouse & Lobe, cited in Verweij & Gerrits, 2012, p. 46)
  • in this process, you examine multiple cases:
    • “to uncover the most frequent combinations of causal conditions (i.e., variables) that produce a certain outcome” (conjunctural causation)
    • note that “different configurations” may produce the outcome” (equifinality)
    • note that “factors can have different effects in different contexts” (multifinality)
  • also looks at asymmetric causality: “the presence and absence of outcomes require different explanations” (Verweij & Gerrits, 2012, p. 46)
  • uses “dialogue between theoretical and empirical evidence, which is especially important in the selection and construction of cases and variables”  (Verweij & Gerrits, 2012, p. 47)
  • “in QCA, variables are conceptualized as causal conditions or sets”  (Verweij & Gerrits, 2012, p. 47) – but note that since “social phenomena […] are often difficult to grasp in terms of sets” … “theoretical and substantive knowledge should be used to substantiate the constructions (and membership) of sets”  (Verweij & Gerrits, 2012, p. 47)
  • “sets can be intersected” [with ‘logical and’] and “unified” [with ‘logical or’]
  • QCA then is “able to systematically compare and analyze these set conjunctions”  (Verweij & Gerrits, 2012, p. 47)
  • QCA looks at “set relations instead of correlations”  (Verweij & Gerrits, 2012, p. 47)
    • “a condition is necessary if it has to be present for the outcome to occur” (p. 47)
    • “a condition is sufficient if it can produce the outcome by itself” (p. 47)
    • an INUS condition is “insufficient but non-redundant part of an unnecessary but sufficient condition” (Mackie, cited in Verweij & Gerrits, 2012, p. 48) – that is, since it is often a combination or “recipe” of conditions that are required to cause an outcome, there are “usually no purely necessary or sufficient condition” (p. 48) but rather conditions that are not sufficient on their own nor necessary on their own, but are sufficient when combined with other specific conditions
  • it is important to note that “‘neither necessity nor sufficiency exists independently of theories that propose causes'” (Ragin, cited in Verweij & Gerrits, 2012, p. 48)
  • once you have each case assigned to a set (i.e., you know its score for each condition, as well as its outcome), you consider a “truth table“:
    • “lists all of the logically possible configurations”  (Verweij & Gerrits, 2012, p. 48-9)
    • fundamental unit of analysis = truth table row
 Condition A Condition B Condition C Outcome           Distribution of cases
1 1 1 1
1 1 0 1
1 0 1 1
1 0 0 1
0 1 1 1
0 1 0 0
0 0 1 0
0 0 0 0
  • then “the truth table can be minimized to produce a so-called solution (i.e., a statement about patterns across cases”  (Verweij & Gerrits, 2012, p. 46). This is done using Boolean algebra9
    • it is important that this is done not done by just applying the formula, but includes “interpreting the formula and critically assessing it in light of individual cases: does it make sense?” Thus, there should be “several iterations between data and concepts, generating increased understanding/interpretation of the cases”  (Verweij & Gerrits, 2012, p. 50)
  • also, “we can use the method in an exploratory rather than explanatory mode. This is particularly useful when we have ‘contradictory’ configurations (i.e., configurations without 100% of the cases having the same outcome state). In such cases, we can then return to the cases and seek additional differentiating characteristics of their previous trajectories and contexts that might help us to resolve the contradiction (i.e., generate an explanation of the difference in observed outcome state” (Byrne, 2013, p. 224)
  • Steps in QCA
    • define the outcome
    • identify conditions thought to be relevant to the outcome
    • gather data on the identified conditions for each case
      • if csQCA, determine if each condition and the outcome are “present” or “absent”
      • if fsQCA, determine value for each condition and the outcome
        • dichotomization (or scoring) “requires judgement and discussion” (Blackman et al, 2013, p. 13)
    • determine the sets (“shared configurations of conditions”
    • allocate cases to sets
    • create the “truth table”
      • share with practitioners  – allows for “them to help [researchers] develop accounts of causality in more detail”, is a “valuable knowledge exchange opportunity” and “enables [researchers] to incorporate insights from practice into our explanations so that these could be grounded in practitioners’ worlds”

Example

  • Blackman et al (2013) conducted a QCA and found that  for the outcome of narrowing the gap in teenage pregnancy rates in a local areas that have particularly high teenage pregnancy rates had “causal combinations” for their 27 cases in 5 sets that used different combinations of 4 conditions (they started with 9 conditions, but 5 did not show up in any of the “causal combinations”)
  • these sets included:
    • narrowing gap was seen in:
      • cases with a high proption of black and minority ethnic groups
      • cases where there was a combination of low #s in drug treatment and high #s of people under 18 years of age and a “basic” standard of commissioning
    • not narrowing gap was seen when:
      • lower proportion of black and minority ethnic groups and low #s of under 18s
      • lower proportion of black and minority ethnic groups and higher #s in drug treatment
      • lower proportion of black and minority ethnic groups and good/exemplary standard of comissioning
    • ideas as to why these patterns were seen:
      • having a basic standard of commissioning was seen in one ‘good’ combination, but a good/exemplary was seen in a ‘bad’ combination – the authors speculaed that this could be because getting a “good/exemplary” rating may be requiring a lot more meetings/documentation that takes people away from doing the real work
      • the lower #s in drug treatment may reflect the level of substance use (which may be similar to level of risk taking generally)
      • areas with the higher #s of under 18s may provide more services to under 18s, leading to lower teen pregnancy rates (but only when combined with lower #s in drug treatment (possibly = lower risk taking) and where a basic standard of comissioning is met

Limitations of QCA

  • “a static method [that] does not fully capture the dynamics of complex systems”  (Verweij & Gerrits, 2012, p. 51)
    • some potential workarounds for this:
      • “using multiple iterations of the method (i.e., before, during, and after a certain intervention)
      • interpreting he time dimension
      • conceptualizing time as (part of) a set
      • complementing QCA with other methods” (Verweij & Gerrits, 2012, p. 51)
  • can only include a limited number of conditions because the number of possible combinations increases exponentially
    • many configurations […] will have no cases […but…] this is not a problem, because, in complexity terms, such configurations can be considered as describing empty attractor states in the possibility space”  (Byrne, 2013, p. 224)
    • but there will still likely be lots of cases in the “occupied attractor states – configurations” (Byrne, 2013, p. 224) when use lots of conditions in our QCA
  • adding a new case can lead to different solution formula – though this “is actually part of the philosophy behind QCA and its roots in systemic thinking. With QCA, the researcher does not strive to identify a single central tendency that reflects reality as more cases are added. rather, it helps researchers to examine the different causal pathways that lead to a particular outcome and how much pathways are linked to individual cases” (Verweij & Gerrits, 2012, p. 46)

Resources:

References
Blackman, T., Wistow, J., & Byrne, D. (2013). Using Qualitative Comparative Analysis to understanding complex policy problems. Evaluation. 19(2): 126-140. (page references in this blog posting refer to the “Accepted Manuscript” version posted at http://oro.open.ac.uk/37540/2/5C07E325.pdf)
Byrne, D. (2013). Evaluating complex social interventions in a complex world. Evaluation. 19(3): 217-228.
Verweij, S. & Gerrits, L. M. (2012). Understanding and researching complexity with Qualitative Comparative Analysis: Evaluating transportation infrastructure projects. Evaluation. 19(1): 40-55.
Posted in evaluation, evaluation tools, methods, research | Tagged , | Leave a comment