Happy International Year of Evaluation!

Posted on January 16, 2015 by Beth

Happy International Year of Evaluation!

2015 was declared to be the International Year of Evaluation (EvalYear) at the Third International Conference on National Evaluation Capacities organized in São Paulo, Brazil, 29 Sep-2 Oct 2013, the aim of EvalYear is “to advocate and promote evaluation and evidence-based policy making at international, regional, national and local levels” (Source). As a professional evaluator and advocate for evidence-based practice/policies, I’m excited that the United Nations Evaluation Group and Eval Partners (the International Evaluation Partnership Initiative) are highlighting this important work on a global level.

So this has gotten me thinking – what can I do to celebrate EvalYear here in my little corner of the blogosphere? Mostly I’ve just been using this blog as a place to keep my notes on various things I’m learning – webinars and conferences I’m attending, books and articles that I’m reading, etc. – and not really as a blog that anyone else might want to read. But perhaps I can use this year as an impetus to add more reflection on my evaluation work and musings on evaluation practice 1Once I have enough such things, I might even start promoting it out to the world so that others might actually come and read the stuff here. I’m still planning to use it as a place to keep notes on things that I’m learning though, as I’m finding it very useful to have my notes in one easily accessible place!, 2Some potential topics I could write about include: What does an evaluator do? (which I get asked all the time when I’m asked “What do you do for work?” and I reply “I’m a program evaluator”), What skills does one need to have to become an evaluator?, Why should someone do an evaluation?, How do you evaluate an evaluation?, What’s the different between research and evaluation (and auditing and quality improvement and…)?, Evaluation standards and competencies, My favourite evaluation resources, What is a CE?

I was thinking that it would be cool to post my “evaluation philosophy” 3I developed a “teaching philosophy” when I started teaching at the post-secondary level and thought that the concept of thinking through and articulating a philosophy would be useful in my evaluation work as well. as a kick off to Eval Year. Which, of course, would require me to have developed my “evaluation philosophy”, which I actually haven’t sat down and done yet 4It’s been on my “to do” back burner for quite some time.. I often think about things that are potential elements of my evaluation philosophy as I go about, and reflect on, my work, but I’ve not documented my ideas or combined them into one coherent philosophy statement. But then I remembered that this is a blog, so things don’t have to be perfect or set in stone. This is actually a great place to do some documentation of what I’m thinking about and ultimately develop my evaluation philosophy statement… especially given that I’ve just been using this blog to take notes of various things I’ve been reading/attending and I’m pretty sure no one reads this stuff but me!

So, with that in mind, here are some musings towards an evaluation philosophy statement.

First things first.

A lot of people tend to put the cart before the horse. For example, they start a conversation about evaluation referring to the data they know they already have (or the data that is the easiest to get) and then try to retrofit an evaluation around the data. Or they have a favourite method – “I want to evaluate my program by interviewing stakeholders!” or “We need to evaluate our program with a survey!” – and want to create an evaluation based on the method.

My philosophy is that you should really put first things first – what is it that you actually want to know? Do you want to know if your program is achieving its intended outcomes? (Moreover, do you even know what outcomes you are intending to achieve?) Do you want to know if your program is being implemented as intended? Are you developing something innovative and you want to know how to evaluate it as you go along?

Once you know what your overall evaluation question is, then you can think about what approach makes sense, what method(s) are best suited to answer your question, and can think about what data you need to answer the question, and how will you collect, analyze, and interpret those data.

Bias

In my previous job, I worked as an internal evaluator – that means I was employed by an organization to do evaluation work for the programs within that organization, as opposed to being an external evaluator, which is someone who isn’t an employee of the organization, but rather works as an consultant and is hired by an organization to evaluate their program(s). In my current job, I’m pretty much an internal evaluator 5I am technically employed by a research institute to do an evaluation of a big project being implemented collaboratively by three organizations, but my research institute is hosted by one of those three organizations and I spent the lion share of my time working from the project management office for the project anyway. One thing I’m often asked is “Aren’t internal evaluators biased, since they are employed by, and are paid by, the organization?” The implication is that if you don’t give the organization the results they want (and the result they “want” is for you to say their program is effective) you might lose your job. Or at least you might think that if you don’t say the program is effective, you might lose your job. I have two points to make against this idea. First, external consultants are also generally paid for by the organization being evaluated 6Though I suppose external evaluators could be brought in by, for example, a funding agency, to do an evaluation, though I think that more often than not would be more like auditing that evaluation. And then you’d have potential bias of thinking that the evaluator/auditor needs to give the results the funding agency wants to hear., so they potentially have the same bias – they might think that if they don’t give the organization the results they want, they will have their contract ended or won’t be contracted for the next project. Second – and more importantly – I think any organization with integrity genuinely wants to know if their program is working, because if it’s not, they can then look at why it isn’t working and change things to make them work better. My role as an evaluator is to try to figure out what’s working and what’s not working and for whom and in what circumstances and why/why not, as well as to help people figure out what to do with this information to make things work even better. I can be genuinely excited about the vision an organization is working towards without thinking that I have to bias my results to show that they are achieving it. The way I, as an evaluator, can support them in their work is to do an honest job of evaluating. If an organization ever asked me to do otherwise, I wouldn’t want to work for them.

Integrity

This brings me to another point – integrity is a critical piece of being an evaluator. If I don’t demonstrate integrity, why would anyone trust my evaluations? If I’m not trustworthy, why would people that I’m interviewing be honest with me? I’m sure I could say a whole lot more on this topic, but this posting is getting quite long and I’ve got evaluating to do. To be continued!

Footnotes

Footnotes
↑1 Once I have enough such things, I might even start promoting it out to the world so that others might actually come and read the stuff here. I’m still planning to use it as a place to keep notes on things that I’m learning though, as I’m finding it very useful to have my notes in one easily accessible place!
↑2 Some potential topics I could write about include: What does an evaluator do? (which I get asked all the time when I’m asked “What do you do for work?” and I reply “I’m a program evaluator”), What skills does one need to have to become an evaluator?, Why should someone do an evaluation?, How do you evaluate an evaluation?, What’s the different between research and evaluation (and auditing and quality improvement and…)?, Evaluation standards and competencies, My favourite evaluation resources, What is a CE?
↑3 I developed a “teaching philosophy” when I started teaching at the post-secondary level and thought that the concept of thinking through and articulating a philosophy would be useful in my evaluation work as well.
↑4 It’s been on my “to do” back burner for quite some time.
↑5 I am technically employed by a research institute to do an evaluation of a big project being implemented collaboratively by three organizations, but my research institute is hosted by one of those three organizations and I spent the lion share of my time working from the project management office for the project anyway.
↑6 Though I suppose external evaluators could be brought in by, for example, a funding agency, to do an evaluation, though I think that more often than not would be more like auditing that evaluation. And then you’d have potential bias of thinking that the evaluator/auditor needs to give the results the funding agency wants to hear.
Posted in evaluation | Tagged , , , , , , | Leave a comment

Notes from a Special Journal Issue on CPOE

Posted on December 19, 2014 by Beth

The journal Healthcare Quarterly published a special issue in 2006 1Source. I don’t think it’s accessible without a subscription. about the implementation of computerized provider order entry (CPOE) at the University Health Network in Toronto. Below are my notes about the papers in this issue.

Transforming Healthcare Organizations – Brian Golden

  • “healthcare organizations are the most complex form of human organization we have ever attempted to manage.” (Drucker, cited in Golden)
    • “confluence of [different types of] professionals and other stakeholders (e.g,. patients, government) often with seemingly incompatible interests, perspectives, and time horizons” (Golden)
    • there is an “insatiable demand for healthcare – without unlimited financial support”
    • medication order entry (MOE)/medication administration record (MAR) is intended to:
      • “decrease adverse events related to medication error”
      • “provide a rich data set to monitor and improve operational performance”
    • challenges related to change management
      • disparate stakeholder groups
      • multiple organizational missions (e.g., provide care, be fiscally solvent, employ people)
      • professional autonomy highly valued by clinicians, who make decisions that “influence a major portion of healthcare expenditures”
      • “information needed to managed the change process lacking”
    • stages of the change process:
      1. determine desired end state
        • the “vision” or “where we want to be” – needs to be actionable
        • at UHN, had “measurable goals [to] allow [UHN] to chart progress and determine success or failure” – “critical to focusing the attention of change leaders and those will be asked (or required) to change”
      2. assess readiness for change
        • “broad situational analysis”, including key stakeholder analysis, what new capabilities will be needed, site’s experience with previous  (good or bad) or concurrent (competing) changes
      3. broaden support & organizational redesign
        • communications:
          • consistent message about the objectives of the change and how the change will be done
          • multiple media for communicating these messages
          • messages tailored to audiences
          • messages delivered by credible source
        • management not talking about an issue until they have all the answer -> rumours and decreased credibility of the change leaders (as people believed they were keeping secrets). Better to be honest about the unknowns and explain your plan to decrease the uncertainty (e.g., say when you expect to know or how you plan to approach things once you have the data you need to make a decision)
        • recognize the most “saleable qualities” of the change. i.e., people more likely to accept changes “to they are seen to be:
          • “trialed” and revisable
          • divisible (phased implementation)
          • concrete
          • familiar
          • congruent
          • marginal
        • staff need to appreciate the benefits of the change
        • organizational environment needs to support the change
        • changing one of the points on the star may require changes on another point to realign things
        • you can’t change culture and values directly – only indirectly by changing things about the points of the star
        • people need not only to be able to implement the change, but also be motivated to implement the change – they have to see the benefit
        • benefits can be to patients, the organization, the health system, clinicians, and/or administrations
      4. reinforce and sustain change
        • monitoring performance and showcasing successes can help reinforce and sustain the project
        • those monitoring/evaluation data can also be used to fine-tune the system
        • also need to reward supporters and recognize losses
    • recipe for a successful change:
      • vision + skills + incentive + resources + action plan

The Business Case for Patient Safety – Anderson et al

  • Canadian Adverse Events Study (data from 2000 fiscal year from chart audits at hospitals in 5 provinces):
    • errors occur in 7.55 of annual hospital admissions
    • 36.9% of those errors were “entirely preventable”
    • 20.8% of those resulted in the death of a patient
  • administration of drugs was “most critical problem contributing to adverse events”
  • problem can be from any mix of:
    • illegible written orders
    • provider writing the wrong prescription
    • prescribing the wrong meds
    • transcription errors/misinterpreted orders by nurses
    • incorrect administration of meds
    • incorrect documentation of meds
  • CPOE reduces:
    • length of stay
    • repeat tests
    • turnaround times for lab/pharmacy/radiology requests
    • costs (Birkmeyer & Dimick, cited in Anderson et al)
  • UHN’s project was driven by the goal of “improving patient safety” (though realizing that other short- and long-term benefit would also be expected)
  • in addition to reducing transcription errors and improving order-to-administration turnaround time, expected alerts –> improved quality of care through identifying, at the time an order was entered:
    • potential drug-drug interactions
    • potential drug-lab interactions
    • potential drug-allergy interactions
  • also expected to see “inefficiencies from updated order sets and better compliance with drug formulary”
  • UHN didn’t collect baseline data for errors – they felt chart auidts would be too costly and take too much time; instead, they extrapolated from the Baker et al (2004) data

The Benefits of MOE/MAR Implementation: A Quantitative Approach

  • UHN’s study objective: “to measure the impact of MOE/MAR on patient safety, clinical workflow and the quality of patient care.”
  • used study data to “measure implementation progress” and use that to make any necessary adjustments
  • if successful, findings can help affect attitudes and perceptions toward future implementation
  • shared data collected weekly (or more often) “so that each group could see what improvements there were on their specific units”
  • their approach
    • steering committee to “determine objectives and research questions”
    • select key metrics & indicators
    • identify data sources (e.g,. chart audits, time-motion studies, electronic reporting)
    • collect/analyze data
    • communicate results on a regular basis
  • indicators nee to be “meaningful and applicable for clinical and hospital executive decision making”
  • collected data for a 2 year period for 8 clinical service clusters (medical and surgical patient care areas)
  • some lessons learned:
    • good communications can help clinicians understand long-term benefits (to get them through the short-term pain)
    • it’s important to manage expectations – e.g., MOE/MAR won’t reduce ordering time, but will streamline process overall;  it won’t be perfect
    • make sure indicators are useful, keep methods as simple as you can/minimize HR requirements for data collection
    • think about data collection issues up front (will system extract the data you need? do you have the skilled personnel to extract and analyze it?)
    • get executive commitment and sponsorship
  • think sustainment – the evaluation can become “a framework for how people do their work on a daily basis” – continue to use results to inform planning

Footnotes

Footnotes
↑1 Source. I don’t think it’s accessible without a subscription.
Posted in change management, evaluation, healthcare, information technology, notes | Tagged , , , , , , , , , | Leave a comment

An Alphabet Soup of Clinical Transformation: IPOCs, CPOE, BCMA, and more!

Posted on December 12, 2014 by Beth

The project I’m working on is a very complex one, with a number of things being introduced as part of the overall project to transform healthcare. This blog posting represents a quick summary of some of the reading I’ve done on the different pieces of what’s being developed

Alphabet SoupFirst, some acronyms and what they mean:

  • BCMA – bar code medication administration
  • CIS – clinical information system
  • CLMM – closed loop medication management
  • CPOE – computerized provider 1“Provider” refers to anyone who can prescribe/give orders, such as physicians, midwives, nurse practitioners, etc. order entry
  • EHR – electronic health record: “a longitudinal electronic record of patient health ifnormation, produced by encounters in one or more care settings” (cited in Staggers et al, 2008)
  • EMR – electronic medical record
  • DSS – decision support system
  • HIMSS – Health Information Management System Society
  • IPOC – interdisciplinary plans of care
  • IT – information technology
  • MAR – medication administration record

First, a definition is in order. On the Health Information Management System Society (HIMSS) blog, we find the following definition of clinical transformation:

“Clinical transformation involves assessing and continually improving the way patient care is delivered at all levels in a care delivery organization. It occurs when an organization rejects existing practice patterns that deliver inefficient or less effective results and embraces a common goal of patient safety, clinical outcomes and quality care through process redesign and IT implementation. By effectively blending people, processes and technology, clinical transformation occurs across facilities, departments and clinical fields of expertise.”

So, the goals are promoting patient safety and improving clinical outcomes for patients and this is to be accomplished through redesigning processes and using information technology (IT). There are many things that can be done to accomplish this – below are some notes from my reading on a number of different processes that can be used.

 

Medication Reconciliation

  • medication reconciliation is “a formal process for creating the most completed and accurate list possible of a patient’s current medications and comparing the list to those in the patient record or medication orders” (p. 459)
  • five steps to med rec:
    1. “develop a list of current medications
    2. develop a list of medications to be prescribed
    3. compare the medications on the two lists
    4. make clinical decisions based on the comparison
    5. communicate the new list to appropriate caregivers and to the patient” (p. 459)
  • common issues with med rec:
    • often not a standardized process used
    • at least 3 disciplines involved (medicine, pharmacy, nursing) and often not clear role/responsibility delineation
    • often duplication in data gathering, documentation (which is often not put in one place within a chart), rarely any process to compare/resolve discrepancies
    • transitions are often a problem:
      • patients generally admitted to inpatient facilities for urgent and/or specific procedure and specialty healthcare often focuses on a single issue, not holistic view of the whole patient (could result in missing a conflict between different meds they are taking)
      • patients sometime discontinue medication they take at home when admited and with a formal med rec process at discharge, the need to restart those meds might get overlooked
      • orders from one unit (e.g., ICU) might get stopped when moved to a new unit, where new orders are written, and without a formal med rec process, may overlook meds that should be continued
  • challenges with doing med rec:
    • likely take more provider time initially, though “may become more efficient once in place” (p. 462)
    • “data are only as accurate as what has been entered” (p. 463)
    • what has been prescribed is not necessarily what is being taken by the patient
    • patients are not always “accurate historians” (p. 465)/don’t always know all the details of their meds
    • electronic med rec process can –> decrease in discrepancies in med profile, but “developing and implementing an electronic reconciliation process requires technical support” (p. 463)
  • potential benefits of med rec (esp. with electronic records):
    •  >40% of med errors are due to “inadequate reconciliation in handoffs during admission, transfer, and discharge of patients” (pg. 459)
    • ~20% of those errors –> harm
    • “computer order entry system can reduce errors at the time of discharge by generating a list of medications used before and during the hospital admission” (p. 464)
    • electronic systems – easier to access med histories, but need to keep up to date and need to correlate with actual use by patients
    • electronic system – “allows for decision support such as checking for allergies, double prescribing, and counteracting medications” (p. 464)
  • research implications
    • a good area for research
    • most studies in the area: small sample sizes, single-site QI projects
    • need studies on how med rec changes workflow (e.g., can take 30-60 mins per admission initially)
    • need studies on sustainability (do people continue the process over the long haul?)
    • “studies should also address what techniques (e.g., the use of a medication card) work best to maintain an accurate list of medications” (p. 466)
(Source: Barnsteiner, J. H. (2008). Chapter 38: Medication Reconciliation in Patient Safety and Quality: An Evidence-Based Handbook for Nurses. Hughes R. G. (ed.). Rockville, MD: Agency for Healthcare Research and Quality)

Interdisciplinary Plans of Care (IPOC)

  • plan of care: “based on patient needs identified by assessment, reassessment, and findings from diagnostic tests; based on patient goals with time frames within which to meet those goals, as well as the settings and services necessary to accomplish the goals; an evaluation of patient progress toward achieving the goals; and a revision of the plan and goals based on patient needs” (Joint Commission definition).
  • potential benefits of electronic IPOCs:
    • “improved interdisciplinary communication and collaboration
    • improve care transitions and hand-off
    • opportunity for patient and family participation in the care process
    • greater care efficiency” (p. 9)
  • 5 key success factors for adoption & bedside use of IPOCs
    • “leadership
    • bedside clinician involvement
    • patient & family focus
    • usable workflow populated with coded evidence-based content that drives assessment and care planning and supports data collection regarding care provided and outcomes achieved
    • clinical, informatics, IT, evidence-based content, and EHR vendor partnerships collaboratively focused on the same goals” (p. 10)
  • “Historically, members of each discipline did their own assessment, established their own goals, evaluated the patient’s progress toward those goals, and developed their own plan of care” (p. 10) –> interdisciplinary approach “required revised thinking, collaboration, trust in each other, and a central focus – the patient” (p. 10)
  • evaluating the journey:
    • metrics they use include:
      • “% of admitted patients with an IPOC initiated
      • compliance of initiation within 4 hours (facility & unit)
      • compliance of ancillary documentation within the IPOC” (p. 12)
    • phase 2 they want to use quality metrics such as “falls prevalence, pressure ulcer rates, glycemic control outcomes, and readmissions […] nursing documentation time” (p. 12)
    • “At the end of the day, there needs to be clear demonstration that the IPOCs aren’t only being utilized, but they also have a positive impact on patient care, patient outcomes, clinical workflow, and length of stay” (p. 12)
(Source: Jones et al (2012). The journey to electronic interdisciplinary care plans. Nursing Management. 43(12): p. 9-12.)

Computerized Provider Order Entry (CPOE)

  • essentially, this means that the provider enters their orders into the computer themselves (as opposed to writing it down and having it transcribed)
  • can include other features, such as standardized order sets,  integrated alerts; integrated decision support tools; interface with pharmacy information system; clinical documentation included
  • because there are many different ways that CPOE can be implemented, in evaluation , it is important to document all the features (as results you get with CPOE without clinical decision support tools could very well be different than using CPOE with decision support tools)
  • “ordering is a process starting with entry, to communication, to processing by various recipients, and then to documenting actions against specific orders” (so need to think about which pieces of this are being affected by your CPOE and other features)
  • “CPOE can substantially reduce overall (and many serious) medication errors if: (1) electronic communication and automatic order interfacs are in place, (2) basic order checks for completeness are present, (3) decision suppport at its most basic level is available – checking for drug-drug and drug-allergy interactions and for dosing ranges”
  • benefits seen with CPOE included:
    • reduced time for order entry to electronic availability of results for lab and radiology
    • reduced time for pharmacy ordering to medication administration
  • benefits due to increased legibility of orders and availability of information
  • need to examine work processes when implementing CPOE to make sure you aren’t causing any unintended negative consequences
  • for evaluation, need to document/measure other potential external influences (if you can’t control them), since you’ll be implementing CPOE in the real world, where other things are going on that can affect your indicators/outcomes
  • “ordering is a complex, interdependent, and interactive process composed of at least these multiple, intersecting elements: systems design, interpersonal and intersystems communication, implementation processes, and organizational structure” (paper suggests qualitative/mixed methods studies needed to do it justice)
  • studies on “what designs facilitate safer orders management; […] what designs facilitate effective clinical decisionmaking; and what design work needs to be in place for successful implementation of CPOE” recommended
(Source: Staggers et al (2008). Patient safety and health information technology: role of the electronic health record in Patient Safety and Quality: An Evidence-Based Handbook for Nurses.)

Barcode Medication Administration (BCMA)

  • also known as closed loop medication management (CLMM)
  • ordering –> dispensing –> administration
  • errors can occur at any of the stages
  • 38% of “potential and preventable adverse drug events occurred at the time of administration”
  • most common errors were wrong dose, wrong route, then wrong drug
  • CPOE ~ ordering
  • BMCA ~ dispensing and administration
  • when BCMA is integrated with the EHR, you have a “seamless flow of information following every stage of the medication administration cycle”
  • process:
    • providers enters medication orders electronically via CPOE
    • pharmacist verification
    • pharmacist packs meds into a barcoded container, sends to nursing floor
    • meds stored in med cart
    • nurse scans own badge (or logs into EHR/BCMA system), scans patient’s wristband, scans medication
    • system automatically documents what drug/dose was given to what patient by what nurse and at what time into the EHR
  • system checks the “5 rights of medication administration”
    • right patient
    • right drug
    • right dose
    • right frequency
    • right route
  • BMCA benefits:
    • real-time validation at point of care (safety check)
    • reduces workload (by automatically documenting for the nurse)
  • evidence demonstrates
    • BMCA –> decreased medication errors (though may also see increase in medication errors due to better documentation of late or missed doses)
  • how BMCA “affects the dynamics of a complex hospital setting, in ways other than the reduction in medication error rate is lacking”
  • organizations need to have a “transparent environment for the reporting of errors – rather than a culture of blame”
  • “a seamless integration between the CPOE and BMCA” is important
  • areas for future research:
    • what types of adverse drug events (such as preventable and potential (a.k.a., “near misses”) does BMCA prevent?
    • ROI and economic outcomes of BMCA (e.g., affects on length of stay, decreased need for nursing FTEs for med admin, decreased litigation)
(Source: Staggers et al (2008). Patient safety and health information technology: role of the electronic health record in Patient Safety and Quality: An Evidence-Based Handbook for Nurses.)

Decision Support Systems (DSS)

  • DSS = “software designed to support or enhance clinical decisions[, including] information displays, alerts, reminders, or fully developed algorithmic computerized protocols”
  • evaluation of the system should include “measuring the implementation itself, the work process changes, and the outcomes”
(Source: Staggers et al (2008). Patient safety and health information technology: role of the electronic health record in Patient Safety and Quality: An Evidence-Based Handbook for Nurses.)

Computerized Provider Order Entry (CPOE)

(Source: Staggers et al (2008). Patient safety and health information technology: role of the electronic health record in Patient Safety and Quality: An Evidence-Based Handbook for Nurses.)
Image credit:

Footnotes

Footnotes
↑1 “Provider” refers to anyone who can prescribe/give orders, such as physicians, midwives, nurse practitioners, etc.
Posted in healthcare, notes | Leave a comment

CIHR Policy Round: Hospital Activity-Based Funding

Posted on December 10, 2014 by Beth

A few notes on today’s webinar from the Canadian Institutes of Health Research’s (CIHR) Institute of Health Services and Policy Research (IHSPR)’s Policy Round on Hospital Activity-Based Funding.

  • Activity-based funding: hospitals paid pre-determined fixed fee per episode of care, to fund bundle of services to each patient with a particular diagnosis, regardless of actual resources used on particular patient. Hospitals received a fixed amount for each episode of care. Replaces other funding (such as global budget).
  • Pay for Performance (P4P): Reward for meeting pre-established targets on quality or efficiency. Supplements other fundings.
  • Fee for service (FFS): Hospital funding for each reported activity.
  • Patient-focused funding (PFF): a BC system, combo of ABF & P4P.

Effects of ABF (from meta-analysis):

  • no effect on acute care mortality (long-term), but early on there was an increse in mortality
  • no effect on post-acute care mortality
  • no difference on volume of care (but high variability)
  • no effect on readmission, but high variability (meaning that there’s uncertainty as to whether a single jurisdiction will see an effect or not).
  • increase in discharge to post-acute care
  • severity of illness was increased in ABF (but was this because ABF increased with increased severity, so people “up code” to a higher severity to get more funding?)

Take home message of the meta-analysis: huge amount of uncertainty in predicting the the impact of ABF.

Canadian Medicare only required to fund hospitals and physician services – not home care. Since ABF –> increase in need for post-acute care – who will pay for it? This raises equity issues, as post-acute care is a mix of publicly funded and privately funded.

Policy perspective from Steven Lewis:

  • not surprised by the variability – ABF is still just a way of funding activities (bundled instead of unbundled like FFS)
  • theoretically, ABF makes you think about efficiency (i.e., if another site is treating similar patients (and getting same outcomes) more cheaply than you, you’d ask “what activities are they doing differently than us?”
  • but healthcare should be thinking about efficiency anyway
  • reducing length of stay –> unloading cost of care (e.g., home care) to people doesn’t mean we should not try to reduce length of stay – we should make our publicly funded healthcare system truly “comprehensive”
  • we should be focusing on the “appropriateness” of care – are we giving the best care (and not giving care that isn’t appropriate)?

Update (2014-Dec-15): A recording of the session, plus the slide presentation, can be viewed here.

Posted in event notes, healthcare | Tagged , , , , | Leave a comment

Evaluation of IT Projects – Some Readings

Posted on December 9, 2014 by Beth

I recently started a new job where I will be spending the next few years evaluating the implementation of a major health information system across several health organizations. So, naturally, I’ve been doing some research on how other organizations have evaluated major health IT projects. This will likely be the first of many blog postings of my notes of my literature research.

Towards an Evaluation Framework for Electronic Health Records Initiatives:
A Proposal For an Evaluation Framework (March 2004)

  • papers is ten years old now (!), but there doesn’t seem to be an updated version
  • evaluations of health information systems traditionally focused on:
    • “technical & system features that affect system use
    • cost-benefit analysis
    • user acceptance
    • patient outcomes” (p. iii)
  • pre-post design “most widely agreed upon approach” and  and randomized control trials (RCTs) are “problematic in the evaluation of complex health information systems” (p. vi)
  • “3 general types of rationale for why evaluation is conducted in the field of health information systems:
    • to insure accountability for expenditure of resources
    • to develop and strengthen performance of agencies, individuals and/or sysems
    • to develop new knowledge” (p. vi)
  • should engage stakeholders in discussion re: which of these three things you want to do, as you have limited evaluation resources to deploy – so what do you hope to learn from an evaluation?
  • seven steps to conduct [extremely generic – I would thing you would use these in any evaluation – and I disagree on a few of them – will put my thoughts inside square brackets]
    • Step 1: Identify Key Stakeholders in Each Jurisdiction
    • Step 2: Orient Key Stakeholders To the Electronic Health Record Initiative and Reach Agreement on Why an Evaluation Is Needed
      • determine stakeholders “expectations of the EHR initiative” and “views on what an evaluation plan should address” (p. 17)
      • “given the diversity of key stakeholders involved […] it is highly likely that they will identify different rationales for conducting evaluations” (p. 17)
    • Step 3: Agree on When To Evaluate 
      • should be longitudinal
      • recommend 3 or more time points:
        • baseline (pre-system implementation)
        • during implementation
        • post implementation (preferably multiple measures at 6, 12, and 18 months post-implementation)
    • Step 4: Agree on What To Evaluation [Not sure why they have “what” after “when”. I would think “what” you are evaluating would inform “when” you needed to evaluate those things]
      • endless numbers of things you could evaluate
      • “it is very important that each jurisdiction feels that it is gaining the maximum benefit it can from the investment of scarce resources in evaluation.” (p. 17)
      • “a priority setting exercise with key stakeholders is one way to (a) identify the questions that it is important to answer (verses the questions that it is easy to answer) and (b) insure that all key stakeholders have an investment in the evaluation of projects which are undertaken” (p. 17)
    • Step 5: Agree on How To Evaluate 
      • rationale for evaluating and specific evaluation questions “have implications for the methods chosen” (p. 17)
      • recommendation: “undertake an evaluation which
        • focuses on a variety of concerns
        • uses multiple methods
        • is modifiable
        • is longitudinal
        • includes both formative and summative approaches” (p. 18)
      • “the current thinking around evaluation of complex health information systems leans towards evaluation geared to performance enhancement and knowledge development, and away from accountability, particularly costing approaches to net benefits assessment. However, accountability remains a strong value in Canadian society in general and increasingly in the health and technology sector, and therefore we recommend that some type of accountability question be included int eh evaluation approaches in each jurisdiction” (p. 18)
    • Step 6: Analyze and Report [They forgot to mention actually collecting the data!]
      • given the multiple questions, “the evaluation effort will consist of several sub-components which are in fact separate evaluation projects, including different methods and disciplines. We recommend that findings from each evaluation project with the evaluation initiative be shared with those key stakeholders identified in Step 1 […] this will permit fuller discussion of the interpretation and implications of the results obtained through different projects, or through the use of multiple methods with each project.” (p. 18)
    • Step 7: Agree on Recommendations and Forward Them to Key Stakeholders [engage stakeholders in recommendations to make recommendations more feasible and useful]
      • engage the key stakeholders in “generating the recommendations which arise from the findings of the evaluation” (p. 18)
      • there may be different interpretations given the different perspectives, but there is “a greater likelihood that common stances […] will be found if those involved are:
        • familiar with the main issue from the start
        • aware of the different perspectives each team member brings to the discussion
        • comfortable that the variety of methods used in the evaluation produced the most unbiased results possible” (p. 19)

Canada Health Infoway

  • a federally funded non-profit

Canada Health Infoway’s Benefits Evaluation Framework

Canada Health Infoway's Benefits Evaluation Framework

Canada Health Infoway’s Benefits Evaluation Framework, based on the DeLone & McLean Information System Success Model (source of image: https://www.infoway-inforoute.ca/index.php/programs-services/benefits-evaluation)

Canada Health Infoway Benefits Evaluation Indicators – Technical Report (version 2.0 – April 2012)

  • a large list of potential indicators that we might want to use
  • the basic conceptual model they use is:

CHI BE Conceptual Model

  • and here’s an example:
CHI BE Conceptual Model - example

Adapted from: CHI BE Technical Report – version 2.0

  • I think this document will be a really useful resource to go back to when we are ready to decide on indicators

A Framework and Toolkit for Managing eHealth Change: People and Processes

This document is focused on change management, but I focused on the sections related specifically to evaluation.

  • “successful implementation of change is achieved when the systems, processes, tools and technology of the change initiative are embedded in the new way health care providers do their work.” (p. 3)
  • research shows that “poorly managed change” can lead to:
    • “turnover of valued employees
    • lower productivity
    • resistance in all forms
    • disinterested, unengaged, detached employees
    • increased absenteeism
    • cancellation of projects
    • slow or non-adoption of new methods and procedures
    • little or negative return on investment (ROI)” (p. 9)
  • Change Management Working Group (CMWG) assessed change projects across Canada and identified some best practices, the following of which interface with evaluation:
    • “demonstrating early results based on comprehensive data” [requires evaluation to be conducted with timely reporting of data]
    • “continuous quality improvement cycles should be applied [again, timely evaluation]
    • “initiatives that demonstrate clinical value will be supported and those that do not include clinical adoption from the beginning will struggle or fail to be adopted” [benefits/outcomes need to be relevant] (p.11)
  • “Monitoring and evaluation [M&E] provides opportunity to identify risk, […] opportunities to improves process, to identify gaps or to recognized success, […] to understand and manage progress toward the future state. Lessons learned and process improvements need to be integrated into real-time, to avoid repeated mistakes” (p. 14)
  • M&E  extends “throughout the lifecycle of eHealth projects and into the operational life of the solution” (p. 32)
  • Infoway’s System and Use Survey is an available survey tool that facilitates evaluation and analysis of use and user satisfaction” (p. 32)

A recent blog post on Harvard Business Review, “Convincing Employees to Use New Technology”

  • ” the true ROI of their digital investments: collaboration among actively engaged users, smarter decision making, increased sharing of best practices and, over time, sustained behavior change.”
  • 3 related problems as to why new tech often does not get used:
    • “CIOs and technical leaders too often take a limited “tech-implementation” view and measure success on deployment metrics like live sites or licenses. They consider business adoption someone else’s job, but in fact no one is made accountable for it.”
    • “platform vendors often oversell the promise of instant change through digital technology. They make their money by selling products and software, rarely by getting them used at scale.”
    • “user adoption programs cost money.”
  • suggestions:
    • focus on investing in technologies you believe will really offer you a benefit and which you feasible can get done
    • “plan for adoption from the start” – includes learning, communications, change management, and evaluation/having the right metrics
    • leaders need to lead by example
    • identify and support influential front-line staff who will champion the initiative
    • align incentives/reward systems with the behaviours you want to see
Posted in change management, evaluation, healthcare, information technology | Tagged , , , , , , | Leave a comment

Evaluation in the Health Sector – conference notes

Posted on November 25, 2014 by Beth

This past Friday, the BC & Yukon chapter of the Canadian Evaluation Society hosted a day long conference on “Health Evaluation: Evaluation as a Learning Process“. Being an evaluator in the health sector, this was relevant to my interests. And I didn’t just attend – I was a presenter as well. My session was a panel on three related projects from my previous job:

Title Authors
Complexity, Collaboration, and Iteration: An Evaluation Framework for Build Healthy Communities at Fraser Health Dr. M. Elizabeth Snow, Tatsiana Dudkina, Samantha Tong, Dr. Victoria Lee
Using Partnership Evaluation to Increase Impact of Healthier Community Partnerships in Fraser Health Judith Eigenbrod, Marina Irick, Christiana Wall, Lynn Nowoselski, Onyinye Adibe, Judi Mussenden, Dr. Helena Swinkels, Dr. Malcolm Steinberg
Using Developmental Evaluation in a Complex Partnership-based Health Promotion Initiative Judi Mussenden, Richelle Foulkes, Christiana Wall, Dr. M. Elizabeth Snow, Dr. Helena Swinkels

We got a really great turnout for our session and lots of great comments and questions! Since I’m no longer at the job I was at when I worked on those projects, I won’t get to continue on in this work, but I’m very interested to see where my former co-workers take it (in particular, the first of those three presentations was on an evaluation framework that I created along with a Masters of Public Health student who did her practicum placement with me this past summer, but I left that job just after we finished creating the framework, so seeing the framework turned into evaluation plans and then those plans implemented is something that I’ll have to watch from afar).

In addition to giving our presentations, I also got to attend some other great sessions. Here are my notes!

Messy Elegance: Evaluation as a Learning Process by Ben Kadel & April Struthers (Keynote Presentation)

  • wabi sabi: Japanese concept about things of beauty being “imperfect, impermanent, and incomplete” [evaluations take place in the real world, not tightly controlled lab settings, so we need to embrace the “messiness” of the world we do]
  • psychological inoculation: like a vaccine, you expose people to a small amount of something so they can prepare for exposure to a bigger amount of it. E.g., introduce stakeholders to the idea that there will be some discomfort with an evaluation (or with a change) and that this doesn’t mean something is wrong (as we naturally associate discomfort with danger), but it is just a normal part of the process. You can also tell them “I can take you where you want to go, but it’s going to cost you a little bit of discomfort”.

SMART FUND – Using Common Outcomes Measurement in Health Promotion by Jolene Landsdowne & Dr. Marina Niks, Vancouver Coastal Health

  • Smart Fund
  • created single set of indicators for all the programs they fund, so that they can aggregate their data across programs to look at overall impact of their funding, wanted indicators based on the research evidence, and wanted to provide support to non-profits who previously had to develop own indicators (even though this is not typically the skill set of non-profits)

Closing the Loop: Promoting Uptake of Evaluation Recommendations by Derek Wilson, Dr. Chris Lovato, and Tamiza Abji,  Faculty of Medicine, University of British Columbia

  • interested in ensuring that recommendations coming from evaluations get implemented (and that evaluation reports don’t just sit on a shelf)
  • each evaluation has an “accountable committee” – recommendations are presented to that committee and a formal motion to accept the recommendations is made; committee assigns the recommendations to a specific person or committee to be responsible for each of them
  • date set to follow up on the recommendations
  • built a database into which all recommendations coming from evaluations done by their unit go, including:
    • the recommendation
    • who is responsible to implement it
    • when will the evaluator follow up with them on it
    • importance of the recommendation (e.g., critical, important, would be nice)
    • status of the recommendation (e.g., completed, partially completed, not completed)
    • comments (e.g., could comment on why a recommendation has not been implemented – is there a barrier to implementing it? has the situation changed such that the recommendation is no longer relevant?)
  • heatmapping of both importance (so you can see if you have a lot of critical recommendations) and status (so you can see if you have a lot of overdue recommendations that you haven’t acted on)
  • people will know that others will see if they haven’t implemented their recommendations
  • an oversight committee (Program Evaluation and Program Improvement (PEPI) Committee) to whom issues can be escalated (but people responsible for recommendations always approached first if a recommendation is not acted on)
  • one audience member talked about a process they use where evaluation findings are used to identify “opportunities” (rather than “recommendations”) and these opportunities are what’s taken to the accountable committee to assess (costs, benefits, risks) and then decide on recommendations
  • [I think I can use a lot of these ideas in the project I’m currently working on – in my evaluation database, I will build a section for tracking (and heatmapping!) recommendations; I think I will also use the evaluation findings –> opportunities –> assess opportunities –> recommendations process]

Capturing System Transformation at Island Health: A Blueprint for Evaluating the Movement towards Integrated Community-Based Health and Care in a Complex System by Shelley Tice, Sherry Gill, Kate Harris, Island Health

  • lots of knowledge translation tools, toolkit of methods to help working groups understand how to do evaluations
  • Tiki Toki – software to create graphic timelines

Learning Across Multiple Evaluations (Panel Discussion)

  • Anne Baldwin of Canada Health Infoway (CHI) noted that they recommend doing benefits evaluation 18-24 months post-go live (as it takes longer than you’d think to see the expected benefits)
  • CHI Benefits Evaluation Technical Indicators Report, v. 2.0 – an excellent resource listed a bunch of indicators that have been used across the country
  • CHI did a “practice challenge” – asked physicians with and without electronic medical records (EMRs) to look up a type of patient (e.g., get a list of all your patients with diabetes, or all your patients on a certain drug, etc.) and saw that those with EMRs were able to do this 30x faster than those without (also, it was hard to recruit those without EMRs to even do the challenge, since it’s so time consuming with paper records)
  • Michael Smith Foundation for Health Research (MSFHR) has recently completed an evaluation strategy for their foundation
Posted in evaluation, notes | Tagged , , , , | Leave a comment

Participatory Action Research (PAR) Lecture

Posted on September 26, 2014 by Beth

I had the opportunity to attend a lecture on Participatory Action Research (PAR), which was being held as a kickoff for a three-day workshop on the same topic 1Wished I could have done the whole workshop, but I just couldn’t afford three days away from my work at the moment!, so I’m recording a few notes here, as usual when I learn something new and cool.

PAR Venn diagram

Source: Wikipedia

PAR “seeks to understand the world by trying to change it, collaboratively and following reflection” (Source: Wikipedia). This is quite a different idea than traditional research, which usually isn’t attempting to change the world through the research itself (though maybe used to change the world after the research is over, through the application of the research findings).

The first speaker gave the interesting example of how in the field of HIV, research questions traditionally focus on things like “What are the rates of condom use?” or “What are the rates of anti-retroviral drug compliance?”, but that these type of question define the intervention/strategy in the question and presuppose the answer (e.g., increase education about condom use, increase rollout of ARV drugs). As well, we know that this hasn’t always worked. By contrast, with PAR, the research question is defined collaboratively with the people affected by the problem, so you might get a question like “How do we improve the quality of life of people living with HIV/AIDS?” In her example, she talked about doing a research project this way and finding out that other things (e.g., lack of transportation) were barriers that needed to be addressed first or things like ARV compliance would not be increased.

The second speaker talked about PAR having 3 basic questions:

  1. what is the problem?
  2. who are the actors?
  3. what is the solution?

He also talked about 5 skills needed to conduct PAR:

  1. mediating (e.g., mobilizing knowledge from different perspectives)
  2. grounding (e.g., building inquiry & learning)
  3. navigating (e.g., selecting and combining forms of inquiry, planning and participation; creating new methods & tools)
  4. scaling (e.g., adjusting inquiry methods and actions to fir the dept of evidence; more complicated is not always better)
  5. sense making (e.g., co-creating meaning in complex situations)

Footnotes

Footnotes
↑1 Wished I could have done the whole workshop, but I just couldn’t afford three days away from my work at the moment!
Posted in event notes, methods, research | Tagged , , , , , | Leave a comment

Prep for “Evaluation Options for Decisions Makers” Workshop

Posted on August 21, 2014 by Beth

I recently 1OK, it was quite awhile ago now, but this blog posting has been sitting in my draft folder for a looooong time!  gave on a presentation on Developmental Evaluation for a workshop at work called “Evaluation Options for Decision Makers“. Which meant that I finally got around to reading a several articles that I’d had on my desk for *ages* with the very good intention of reading. Here are my summaries of those articles.

How Does Complexity Impact Evaluation?

  • Complex systems:
    • “uncertainty/unpredictability
    • interdependence among a large number of actors who themselves adapt and co-evolve
    • emergent outcomes created by the connections or relationships in the system
    • nonlinearity (outputs and inputs are not directly correlated)” (p., vi)
  • Complex interventions are characterized by:
    • structural complexity: more players, increased variety of relationship between them, more interdependence of their decisions
    • cognitive complexity: “increasingly more difficult to make valid or accurate predictions about the system” (p. vi) resulting from increased structural complexity
    • social  complexity: “high level of social conflict or disagreement among the many players in the system”
  • e.g., healthcare – more types of providers, decisions of one group affect the others, different approaches of different discipline can lead to disagreements – and even the clients/patients becoming more informed (or misinformed)/engaged/involved
  • “recognizing, , acknowledging, surfacing, and addressing the paradoxes inherent in complex systems” (p. vii)
  • “It is what happens “in between” that matters: between people, organizations, communities, parts of systems – “in between” relationships”… “paying more attention to relationships as the unit of analysis rather than to parts of the system” (p. vii).
  • requires “willingness to be uncertain at times and to know that being uncertain is crucial to the process” (p. viii)
  • “embracing multiple perspectives and being aware that evaluation is about understanding networks within and between organizations” (p. viii)
Reference:  Zimmerman, B., Dubois, N., Houle, J., Lloyd, S. Mercier, C., Brouselle, A., & Rey, L., (2011). How Does Complexity Impact Evaluation? The Canadian Journal of Program Evaluation  26(3), v-xii.

Practice-Based Evaluation

  • a summary paper of the articles in the journal + a conference
  • evaluators use “conceptual models to position themselves in relation to the intervention and to guide their evaluation […] methodology was secondary and followed naturally” (p. 107)
  • need to take “into account contextual variables […] evaluation of interventions should be contingent, contextualized, and embedded within temporal, administrative, social, economic, and political realities” (p. 107)
  • differences from “pure” research paradigm:
    • “knowledge produced and shared over the course of the evaluation will inevitably influence the intervention and its context” (p. 108) vs. research where you would think of this as “contamination” of the research
    • “acknowledging that evaluation is a subjective process is at odds with the desire to see it as a scientific, externally valid, reproducible process” (p. 110)
Reference:  Dubois, N.,  Lloyd, S. Houle, J., Mercier, C., Brouselle, A., & Rey, L., (2011). Discussion: Practice-based evaluation as a response to address intervention complexity. The Canadian Journal of Program Evaluation  26(3), 105-113.

The Art of the Nudge

This article talks about the experiences of a group of Developmental Evaluators who spend three years evaluating a national initiation on youth engagement, focusing on some of the things they learned that were particularly effective to “provide real-time feedback that subtly supports shifts in policies, practices, resource flow, and programming in a way that is sensitive to context and to the energy of the people involved” (p. 40) and “creative opportunities for groups to find their collective way, to recognize patterns within complex systems, to help take stock of how the team was doing, and to name design flaws or blockages in a supportive manner” (p. 46) These included:

  1. Servant Leadership
  • leading “must always be in service to the group achieving its goals and living its principles”
  • “opening pathways for new understanding and addressing program blockages”
  • “draw out data and observations that help actors realize what they collectively believe to be the best path forward at any given time” (p. 46)
  • use of an appreciative lens – “focus on strengths and promising patterns that could be leveraged to support the initiative” (p. 46) – in the “fast-paced decision making, messy collaborations, and steep learning curves […] nudges could be perceived as threatening” (p. 47)
  • listen deeply and actively – “find synergies, identify decisions inconsistent with the group’s stated intend, and detect when and how to intervene with an effective nudge” (p. 47); “use carefully crafted questions that encourage transformative group reflections” (p., 48)
  • integrate reflection into practice
  1. Sensing Program Energy
  • “a common management response to unrest, tension, and conflict is to supress it” (p. 49)
  • but a Developmental Evaluator “who is perceived credibly, without a personal or organizational agenda apart from the project, can be well-positioned to identify an issue that is blocking program energy” (p. 49)
  • open channels of communication and bring interpersonal dynamics to the surface
  1. Supporting Common Spaces
  • common spaces = “physical places, moments in time, and virtual spaces where key actors interact” P. 50)
  • as places to: (a) identify observations and (b) prioritize interventions
  1. Untying Knots Iteratively
  • knots = “a wide spectrum of problems, given the ambiguities, concerns, and interpersonal dynamics that got tied up” (p. ,52)
  • iterative approach: “(a) identify what specifically requires more clarity, (b) consider how to collect information about the challenge and its potential solutions, (c) collect the information, (d) reflect on how to gracefully bring the information back into the system, (e) put the information back into the system, and, finally, (f) follow up on the results of the intervention” (p. 53)
  1. Paying Attention to Structure
  • “both formal decision-making structure as well as the culture of decision making, that includes an organization’s written and unspoken norms, rules, routines, and procedures” (p. 53)

Some other interesting points from the article:

  • traditional evaluation “can fail to return timely data about how an unpredictable system is responding to new inputs, leaving innovator in the dark about how to adjust” (p. 40) and “offer too little information too late to be useful for innovation” (p. 56)
  • DE “supports innovation by providing timely and actionable data about how a complex system is responding to an initiative” and is about “asking evaluative questions, applying evaluation logic, and gathering real-time data to inform ongoing decision making and adaptations” (Patton, 2011, p. 1)” (p.41)
  • purpose of evaluation:
    • DE: “the exploratory development of a social change approach”
    • formative: “fine-tuning of a program”
    • summative: “definitive judgement about a program’s impact” (p. 42)
  • Developmental Evaluator:
    • is part of the development team – can be an uncomfortable role for evaluators who are usually trained to be removed from the intervention/program
    • requires skills/experience in “organizational development, whole-systems change, pattern recognition, interpersonal dynamics, conflict management, and facilitation – all skills that are crucial for helping innovators know when and how to use data and feedback to adapt strategies as they go” (p. 55)
Reference: Langlois, M., Blanchet-Cohen, N. & Beer, T. (2012). The Art of the Nudge: Five Practices for Developmental Evaluators. The Canadian Journal of Program Evaluation. 27(2), 39-59.

Footnotes

Footnotes
↑1 OK, it was quite awhile ago now, but this blog posting has been sitting in my draft folder for a looooong time!
Posted in Developmental Evaluation Community of Practice notes | Leave a comment

A Realist Synthesis Protocol Paper

Posted on July 28, 2014 by Beth

Many moons ago, I did some work with a colleague on a grant application for a realist synthesis on self-management of chronic conditions and I was very pleased to be notified recently that this colleague has now published their protocol:

Understanding how self-management interventions work for disadvantaged populations living with chronic conditions: protocol for a realist synthesis.

Here’s a link to the full-text pdf of the article.

I’m also honoured to have been acknowledged in the paper for my work on the original grant application. I remember the grant application requiring *a lot* of work and I can see from this paper that the amount of work involved in a conducting a realist synthesis is commensurate with the amount of work it took to put the grant application together. I’m really glad to see that Sue and her colleagues have undertaken this ambitious project and I’m eagerly awaiting the findings paper now!

Posted in health equity, methods, publications, research | Tagged , , , , | Leave a comment

Empathy: The Human Connection to Patient Care – A Video From the Cleveland Clinic

Posted on May 22, 2014 by Beth

This video from the Cleveland Clinic was shown at our Quarterly Business Meeting at work last week. What a powerful reminder of the humanity of all of our patients and our staff working in the healthcare system.

And then, as often happens when one is on YouTube, I found related videos. In fact, the Cleveland Clinic has a whole series on empathy. Here’s another really powerful one:

Posted in healthcare | Tagged , , , | Leave a comment