Webinar on Tests of Change: Plan-Do-Study-Act

Watched a webinar on using PDSA to do test of change. Here are the (very rough) notes I took:

Tests of Change: PDSA Webinar
Hosted by Canadian Foundation for Healthcare Improvement (CFHI)

• PDSA = Plan-Do-Study-Act
• The Model for Improvement  (Langley et al, 2008 – The Improvement Guide)
  • Aim: what are we trying to accomplish
  • Feedback: how will we know a change is an improvement?
  • Change: what change can we make that will results in improvement
  • PDSA cycles
• We do PSDA cycles everyday (but don’t often call it that) – come up with ideas for improvement, test out the idea, see how it works,
• It’s difficult to predict effectiveness (which we overestimate) and effort (which we underestimate) – lots of evidence for this from behavourial economics (Thinking Fast & Slow summarizes this research)
• history
  • Shewhart – came up with the control chart
  • Lewis – a philosopher – pragmatism
  • Kolb’s cycle (related to PDSA) “Experiental Learning cycle”: active experimentation, concrete experience, reflective observation, abstract conceptualization (and the cycle continues)
• you can’t have improvement without learning (though you can have learning without improvement)
• why use PDSA?
  • testing on a small scale can help you to:
    • predict how much improvement can be expected from the change
    • learn how to adapt the change to conditions in your local environment
    • evaluate costs and side-effects of the change
    • minimize resistance upon implementation – telling people “we are testing it and we want to know what you think about it?” has a much different effect rather than “this is the new thing we are doing”
• difference between PDSA and pilot:
  • PDSA – early cycle prototypes, proof of concept, spirit of inquiry
  • pilot – fully developed change, spirit of confirmation, few (if any) iterations
• difference between testing and implementation
  • testing: temporary, subset of population, learn what change works, adapt/modify change, may test a single factor
  • implementing: “permanent”, whole population, make change, make a change standard, if multifactorial change – all factors
• driver diagrams – have process and outcome measures – are a good place to start when doing Planning – what to change? what will you to measures your change?
• when we start, we have some belief that the change will –> improvement – and the degree of belief may change as a you plan (and learn more), and then when you due cycles to test the change (we may learn it works or we may learn it doesn’t or we may learn which parts work) (graph of this in The Improvement Guide)
• from an idea – develop the change, test the change, adapt/adjust/adopt, implement the change
• Before PDSA: “What’s the largest thing you can do to test the change next Tuesday?”
• It is possible to have a test that is too small, but usually people wait too long and make tests bigger than they need to
• Note: it’s not a small CHANGE, it’s a small TEST we are looking for
• You can use simulation to run a test
  • works well when interested in rare events or when the risk of failure is high
• You can have people “review and comment” as a PDSA
• Test with just one person, one provider, one patient, one client
  • you can’t learn something that’s generalizable from a test with one person – but you can learn
• Testing on a small scale
  • try out new ideas before implementing the
  • break down new changes into a series of small tests that you will study and modify (if needed)
  • no important change will “fit” your system perfectly
  • you want to work the bugs out before you implement
  • does need to be large enough test for it to be informative (e.g., a tiny paint chip makes it difficult to really envision your whole room that colour)
• What thinking about a PDSA cycle, challenge how much time you need to run a cycle. I fyou are thinking a month-long teset, what you could do in a day? If you are thinking of a week, what could you test in an hour?
• Over time, you take your PDSA cycles up a ramp – from theories/hunches/ideas up through increasing complexity/number of people/different situations –> changes that result in improvement
  • very small scale tests –> follow up tests –> wide scale tests of change –> implementation of change –> spread
• size of test – consider:
  • risk of failure
  • confidence in the change
  • resistance/readiness for change
• Planning:
  • objective: aim, predictions
    • can undercover potentially unintended/unwanted consequences
  • plan for change – what are you changing? who is doing what? what are you measuring?
    • how will you know that change was done as planned?
    • how will you know that the change resulted in some positive effect?
    • can you build data collection into daily work processes? (make data collection as easy as you can)
      • can you use existing sources or will you have to gather new information?
      • how will you collect qualitative observations/data?
  • 4 test designs:
    • observational
    • before-after
    • time series
• Do – run the test
• Study
  • analysis of data
  • compare theory/prediction and results
  • summarize lessons learned
  • someone needs to record observations, harvest learning
  • look for:
    • what works
    • what doesn’t work
    • what adjustments are needed
  • be open to abandon the idea if it doesn’t work
    • sometimes people get so invested in the idea that they don’t want to abandon it, even when it doesn’t work
• Act
  • not that same as “do” – it’s about consolidating ourlearning and figuring out what to do next
  • continue, modify, or re-direct efforts
  • what next?
  • any new theories or ideas?
• You can (should?) engage patients and families in:
  • planning & selecting good change ideas
  • reviewing results of PDSAs

Update: Here’s a link to a video of the webinar.