Webinar Notes: Populations and the Power of Language

Hosted by CHNET-Works

  • “Health equity exists when all people can reach their full health potential and are not disadvantaged from attaining it because of their race, ethnicity, religion, gender, age, social class, socioeconomic status, sexual orientation or other socially determined circumstance.” (source: slide deck for this presentation)
  • language has an important role in this social construction
  • NCCDH is working on a glossary document for Public Health

Conversation

Why does language about populations matter?

  • can end up defining people based on their disease (e.g., schizophrenic, alcoholic, poor people) – but a disease does not define a person (instead, put the person first – person with schizophrenia, person with mental illness, people living in poverty. “people first language”)
  • can prevent people from seeking care (e.g., if they don’t identify with the label; feel discriminated against/stigmatized)
  • e.g., Code White changed from “violent behaviour” to “disruptive behaviour” (trying to separate mental health and violence; so responders don’t go in assuming violence. Some talk about changing it to “mental health emergency” – responders go in understanding that the person needs mental health support, not necessary violence occurring

Advantage and disadvantage coexist:

  • understand relationship between advantage & disadvantage – people are more complex than just their disease/disadvantage; have thought about intersectionality; are we trying to aggregate or disaggregate groups? We want to know that we are reaching those who we are trying to reach (and not underincluding or overincluding)
  • reflect on one’s own social position – where do you come from? how do people perceive you? When you think about how complex you are, makes you realize that other people are also very complex.
  • acknowledge structural/systematic advantage and reinforcement

Diversity within population groups:

  • often this is not considered – “disadvantaged” groups often thought of as a homogenous group based on the particular disadvantage
  • there are varying levels of advantage/disadvantage within a group; also intersecting disadvantages
  • e.g., people who are homeless experience health inequities. Some of them also have low literacy/low health literacy/disability/etc., and they experience even more inequities

Language influences power dynamics:

  • words like “vulnerable”, “disadvantaged”, “marginalized”, etc:
    • can create us vs. them situations
    • lead to victim blaming, stigmatization
  • e.g., “poor people” vs. “people who live in poverty” – the latter is people-first language, highlights the situation in which they live (i.e., poverty) vs. describing “poorness” as an attribute of the person), and sounds less permanent (poverty is a situation that they are in that can change rather than being an attribute ascribed to the person)
  • One participant noted that she grew up in poverty but didn’t think of it as that at the time and wondered if she would have internalized it if she’d heard either of those phrases used to describe her.

The language we use focuses our attention:

  • “the homeless” focused on individuals, whereas “housing” focuses us on the structures, and “racism” focuses us on society (an issue underlying the issue, driving the structure of “housing”) – looking at different levels (individual, structures, society)
  • “priority populations” is neutral language – but does this ignore the situations/structures/systems that are putting people at a disadvantage? Also, “priorities” are always shifting (and who is setting the priorities? And why?). Also, “priority populations” creates that group as separate from the “general population” (“they” vs. “us”).
  • so the language can drive action by what it causes us to focus on

How could you use language to advance your agenda in different settings?

  • not everyone has an understanding of health equities, so sometimes describe “differences in health outcomes based on social determinants of health” rather than saying “health inequities”
  • insiders (member of the community) vs. outsiders (not members) – e.g., sometimes insiders use words to describe themselves that are not deemed appropriate for outsiders (e.g., in LGBT community)
  • language is always changing, so need to keep up
  • know your audience (e.g., telling the business community that you need an intervention for mental health probably won’t get them excited, but talking about improving the productivity and creativity of your workforce might be better received; similarly, with education sector, you’d talk about scholarship achievement, high school completion – these also speak to *why* promoting mental well-being is important)
  • also, a shift to “mental well-being” – not just focusing on an absence of mental illness (similar to the definition of “health” (often thought of as specifically physical health) as not just being the absence of disease)
  • so not just language when it comes to what do we call specific population groups, but how do we use language more broadly

Take Home Message:

  • “Being intentional about the use of language can…
    • engage and empower groups (focus on strengths)
    • recognize and alter discriminatory beliefs
    • address unequal power imbalances
    • tailor programs to local context
    • maintain attention “upstream” and be systems-oriented” (Source: Slide Deck for this presentation)
  • No perfect answer to what language to use and not everyone will agree. Be intentional of what language you use, think about who it is serving and what unintentional consequences it might have.
Image Source (Creative Commons licensed): Word bubbles: https://www.flickr.com/photos/ajc1/6924223634/
Posted in health equity, Public Health, webinar notes | Leave a comment

Online Course notes: E is Epi, 1.1 – Epidemiology: A Basic Public Health Science

I am not an epidemiologist…but I work with a lot of them. In an effort to round out my knowledge of the field 1I’ve only briefly covered epidemiological concepts in my research methods class back in PhD school, and that was several years ago now., I’ve decided it’s worthwhile to brush up on my epidemiology.

I searched the CDC Train list of courses (of which there are *tonnes*) and chose the highest ranked intro epidemiology courses, which are from a series called “E is for Epi“). These courses are designed for non-epidemiologists, which I am. I have the t-shirt to prove it!

The first course in this series is actually one on the basics of the field of public health:

Epidemiology: A Basic Public Health Science (E is for Epi, Session 1.1)
North Carolina Institute for Public Health

Now, since this is a course from an American university, it’s teaching the American Public Health framework. I thought it would be interested to check it out, and then compare it to the Canadian situation. (Some colleagues of mine are involved in a project comparing the the BC Core Public Health Functions framework and the Ontario Public Health Standards, so I know that even within Canada, things are not done exactly in the same way.) To be able to make that comparison, I’ve just signed up for the Public Health Agency of Canada (PHAC)’s Skills Online module “Introduction to Public Health in Canada” 2An added incentive for me to do these online courses is that I’m currently developing an online training module, so it’s helpful to see what works – and what doesn’t – from the perspective of a learner.. It takes a week to activate one’s registration, so I won’t be able to go through that module until next week. In the meantime, I completed the E is for Epi, 1.1 module. Basically, it just covered the 3 core functions and 10 essential services of Public Health in the US:

The 10 Public Health Essential Services, surrounded by the 3 Core Functions (this is US-based):

3 core functions(shown on the outside of the wheel):

  • assessment – surveillance
  • policy development – taking surveillance data to develop policy
  • assurance – make sure policies improve health outcomes

10 essential services:

  1. Monitor health status to identify and solve community health problems.
  2. Diagnose and investigate health problems and health hazards in the community. (e.g., outbreaks, labs, infectious disease epi)
  3. Inform, education, and empower people about health issues.
  4. Mobilize community partnerships and action to identify and solve health problems.
  5. Develop policies and plans that support individual and community health efforts.
  6. Enforce laws and regulations the protect health and ensure safety. (e.g., clean water, air, alcohol & tobacco sales)
  7. Link people to needed personal health services and assure provision of health care when otherwise unavailable.
  8. Assure a competent public and personal healthcare workforce
  9. Evaluate effectiveness, accessibility, and quality of personal and population-based health services
  10. Research for new insights and innovative solutions to health problems

And here’s my certificate to prove I pass the test!

Certificate - E is for Epi 1.1

Because you can never have too many certificates

Footnotes

Footnotes
1 I’ve only briefly covered epidemiological concepts in my research methods class back in PhD school, and that was several years ago now.
2 An added incentive for me to do these online courses is that I’m currently developing an online training module, so it’s helpful to see what works – and what doesn’t – from the perspective of a learner.
Posted in online module notes, Public Health | Tagged , , , | 2 Comments

Webinar Notes: Using Randomized Controlled Trial Designs in Community Settings

Using Randomized Controlled Trial Designs in Community Settings

Presenter: Tim Aubry, U of Ottawa
Date: 26 March 2014
CES Webinar

  • RCTs often referred to as the “gold standard”, but not everyone believes it
  • “methodological pluralist pragmatists” – program development in various stages, taking context into consideration  vs. RCTs as gold standards
  • evaluation is messy – doesn’t lend itself to RCTs – need to be nimble and flexible

The “At Home/Chez Soi” Demonstration Project

  • “action research on how to support people with severe mental illness to exit homelessness”
  • funded by Mental Health Commission of Canada
  • 85% funding into services, 15% into research
  • intervention = subsidized housing + support (ACT or ICM) – provided separately
    • housing provided immediately, private market units, people hold their own lease, max of 30% income on rent (rest subsidized), funding not tied to particular housing unit
    • Assertive Community Treatment (ACT) or Intensive Case management (ICM) teams
    • both well known in community mental health

Methods:

  • protocol paper published in BMJ Open
  • multi-site (Vancouver, Winnipeg, Toronto, Montreal, Moncton)
  • non-blind parallel group RCT
  • effectiveness and cost-effectiveness
  • 2 fidelity assessments & 2 implementation evaluations
  • high needs (ACT) & moderate needs (ICM) vs. usual care
  • inclusion: mental health (with or without co-existing substance use), homeless or “precariously housed”, adults
  • intent to treat analysis

Challenges

  • top-down initiation of the project, initial resistance from the community
    • each city involved in adaptions for their “version” (e.g., Aborigianl group in Winnipeg, cultural community group in Toronto, rural project in Moncton)
    • presented as “new & innovation” addition to services, not taking away
    • emphasis on credibility of research for Housing First
  • ethical concern: people feel that those people in the intervention group are getting something better and withholding it from the control group; compounded when working with marginalized groups
    • emphasize that we don’t actually know that the intervention works (there may be studies in other places that suggest it works, but those studies are limited/conducted in different places that might not be relevant to your context)
    • adding service to some, but not withdrawing anything from anyone (e.g., if this project was not happening, those in the control group would be in the same situation as they are in the project)
    • you can argue that it would be unethical to provide services that haven’t been evaluated
    • would like to have done a “waiting list” – where those in control group are first in line for the services if the intervention is found to be significantly better
  • integrity of randomization – needs to be truly random assignment, avoid selection bias
    • referrals went to research personnel to determine eligibility and do randomization (so service providers can’t be tempted for selection bias – e.g., if service provider thinks the intervention really works, might want to assign particularly vulnerable clients to intervention group)
    • randomization occurred after first interview
    • look at characteristics of groups to see if they equal
  • sample attrition
    • especially in a transient population
    • could get differential attrition when one group connected to a service and the other not
    • oversampled to get needed power
    • actively tracked (data collection check ins every 3 months; interviews every 6 months), get permission of participant to be able to contact family members/service providers to track them down if you lose contact, rely on personal connections, honorarium for interviews
  • ensuring fidelity of the intervention
    • complex psychosocial intervention – worry about it being implemented correctly to start, but also worry about program “drift” over time
    • significant training & ongoing technical support throughout the project
    • 38 fidelity standards – did fidelity assessments

Outcomes

  • bigger improvement in housing, quality of life, community ability, and substance use in intervention vs. control (both groups improved, but intervention resulted in bigger gains)
  • there is some regression to the mean (as people are recruited when they are having a rough time), so some natural improvement (so, if didn’t have an RCT, could end up concluding that intervention causes all of the improvement, even though some of it would have happened anyway).
  • cost analysis – $10 invested in intervention –> $9.60 savings (comprehensive costs – health care, justice, etc.)
  • influenced budgeting policy – big cities have to put 2/3 of funding towards Housing First

 

Posted in evaluation, methods, webinar notes | Tagged , , , , , , , , , , , , , | Leave a comment

Online Module Notes: RE-AIM

I recently completed an online learning module about the RE-AIM framework for program planning and evaluation offered by the Centre for Training and Research Translation (TRT) at  Center for Health Promotion and Disease Prevention at the University of North Carolina at Chapel Hill. Here are some notes (with my more extensive raw notes after the jump).

Re-AIM is a framework that can be used for program planning and evaluation that goes beyond just looking at “is a program effective?” and considers who the program reaches, how well is it adopted, the fidelity of implemented, and sustainability of the program over the long-term.

The elements of the framework are:

RE-AIM

Collectively, each of these elements has an effect on the overall Public Health Impact of the initiative.

Reach =         # of people actually exposed to/served by the initiative     
# of people who could be exposed/served in an ideal world

  •  we are especially interested in whether those who are being exposed to/served by the initiative are those with the highest risk/most in need of the service, so we also want to compare those who are actually reached by the program with the overall group of people who could be reached on  any relevant characteristics (e.g., does the program reach all genders? does it reach people of the different cultural backgrounds in the population? people with different levels of ability? people who don’t speak English? people of differing income levels?)

Effectiveness = “how well your initiative affects a change in intended outcomes and whether or not there are any unanticipated outcomes”

  • consider effects on the primary outcomes of interest, but also other outcomes
  • unanticipated outcomes can be positive or negative
  • are outcomes consistent across different subgroups?
  • how confident are you that the benefits outweigh the adverse consequences?
  • when planning – look for existing evidence (research, evaluations), be clear on the outcomes you are trying to achieve (e.g., create a logic model!), and think about how the evidence relates to your specific context
  • for evaluations – looking at the outcomes of your program

Adoption =               # of settings that actually participate              
# of settings that could participate in an ideal world

  • like Reach, but at the organizational level
  • “setting” can be things like schools, daycares, health units, community-based agencies, etc.
  • may be multiple levels of settings (e.g., school districts, then schools)
  • are there differences between the settings that participate and those that don’t?
  • setting and reach word together (e.g,  if you have 5 settings that each serve 50 people: reach = 250 people; if you have 1 setting that serves 150 people: reach = 250 people)

Implementation= “extent to which your intervention is delivered as intended or designed”

  • implementation fidelity
  • “core elements” = “components of the intervention that are critical tot he effectiveness of the intervention” – based on theory/logic/main strategies; can be adapted, but not changed
  • are all the components being implemented as planned?
  • is it implemented the same way by all staff?
  • do staff change the way they implement over time?
  • what is the time and money cost of delivery?
  • design process measures for each core element – if your evaluation does not describe implementation, then you don’t really know what you are evaluating

Maintenance = “what are the long-term effects of the initiative and is it sustainable?”

  • both individual and organizational level
  • does your initiative produce lasting effects? (at individual and/or setting level)
    • individual: “the long-term effects of the intervention on both targeted outcomes and quality-of-life indicators”
    • setting/org: “extent to which a program is sustained (or modified or discontinued) over time”

RE-AIM 2

Putting it all together:

  •  our goal is to :
    • reach more people
    • more settings adopt
    • implemented as intended
    • resulting in effective
    • over the long-term (maintained)
  • all together –> Public Health Impact
  • need to attend to all 5 dimensions

Here’s my certificate of completion for the course:

Certificate+for+RE-AIM - Beth Snow

 

My more extensive raw notes that I took while going through the module (which I’m sure no one other than me would ever be interested in) are after the jump. Continue reading

Posted in evaluation tools, knowledge translation, online module notes, program planning, Public Health | Tagged , , , | Leave a comment

Webinar Notes: Evaluating FASD Prevention & Support Programs

This webinar caught my eye as it is the at the intersection of my former academic world – Fetal Alcohol Spectrum Disorder (FASD) 1My PhD dissertation was on FASD. and my current world – evaluation. Combine that with the fact that one of the presenters is colleague with whom I previously worked (and who I know *always* gives great presentations from which I learn a lot!) and I knew this webinar was one that I didn’t want to miss!

Evaluating FASD Prevention & Support Programs: Tools to Support Planning & Evaluation

Website containing resources from the project: www.fasd-evaluation.ca

  • supports capacity of community-based organizations to conduct evaluation
  • evaluation is about:
    • learning how a model works
    • learning how to improve a program
    • learning new outcomes can be identified
    • learning what measures are appropriate for those outcomes
    • learning what difference the program is making
    • informing evidence-based decision making
  • they mapped FASD prevention programs
    • levels in a cirlce (from inside to outside): program’s philosophy/theoretical framework, program activities/approaches, program outcomes, participant outcomes, community outcomes
    • not suggesting that any given program needs to have *all* the things at the map – these are a collective list of all the things they saw in the programs they looked at

Resource guides:

  • using the program philosophy as a foundation for evaluation is key (not just for FASD programs, but other programs working with children, women, and families)
    • why are we doing things in a particular way?
    • are we doing the right things?
  • program activities – important to uncover the hidden activities
  • link between activities to outcomes
  • participant outcomes – short, intermediate and long-term outcomes, along with indicators of those outcomes and measurement tools are provided on the website

Here’s a link to the recording of the webinar.

Footnotes

Footnotes
1 My PhD dissertation was on FASD.
Posted in evaluation, webinar notes | Tagged , , , , , , | Leave a comment

Webcast Forum Notes: Healthcare Leadership Forum

The Healthcare Leadership Forum in Montreal, was hosted by the Canadian Foundation for Healthcare Improvement (CFHI), the Canadian Association for Health Services and Policy Research (CAHSPR), and the Canadian Health Leadership Network(CHLNET) and was available via live webcast, which I watched. Here are the notes that I kept.

From the welcoming remarks:

The Triple Aim

  • Improve patients’ experience (including satisfication and quality)
  • Improve population health
  • Reduce costs
    • in Canada, many feel we should add “equity” as a goal

We need leadership:

  • clinical leadership – those who understand the clinicial issues
  • policy leadership – making decisions about financial, etc.
  • executive leadership – running their organizations, often have to work in policy arena too
  • patient leadership – growing in Canada
  • leadership without position – those who don’t formal leadership positions, e.g., policy researchers, health services researchers, patient advocates, journalists, etc.

Leadership is about:

  • managing people
  • strategy
  • execution
    • it’s all about getting results

We need to build leadership capacity.

Panel Discussion: Setting the Stage: Why Leadership Matters

Facilitated by TERRY SULLIVAN, University of Toronto
Panelists include:
  • JOHN ABBOTT, Chief Executive Officer, Health Council of Canada
  • JEAN LOUIS DENIS, University of Montreal
  • GRAHAM DICKSON, Royal Roads University and CHLNet

John Abbott

  • we aren’t seeing the results we want in health in Canada (e.g., relative to other countries)
  • equity is a big concern
  • we need leadership
  • with the knowledge and skills we have, can we not have the best health system to give Canadians the care they deserve?

Jean Louis Denis

  • what is leadership?
  • traditionally, position leaders as individuals – leader attracts attention to self, chariasmatic individual
  • one of the big changes in health today = seeing leadership as a collective, as a system = plural leadership
  • Baker & Denis, 2011 – 10 themes of highly performing health systems
    • quality & system improvement as a core strategy
    • leadership embraces common goals & permeates the organization
    • ?
  • different types of plural leadership (Denis et al, 2012, Academy of Management Annals)
    • sharing leadership – sharing leadership and collaborating
    • pooling  leadership – pooling at the top to lead others
    • spreading leadership – networks, interorganizational – the more we talk about chronic disease management, the more we talk about this
    • producing  leadership – leadership as emergent property of relations

Graham Dickson

  • physician engagement is key
  • very challenging to do long-term change with the high leadership turnover that we have (whereas clinicians tend to be in the system for a long-time, and they see this churn of leaders and question the commitment)
  • leadership has to be collective (instead of the traditional notions of individualism in leadership) and patients need to be at the forefront
  • we need national standards for what good leaders do
  • “Canada has a succession planning model called “I quit!”” – we need concerted leadership development and succession planning

Some questions raised during the Q&A:

  • We know it’s valuable to engage patients, so why don’t we do it?
  • How do we change culture?
    • we need to surface culture so we can see the”rational and irrational aspects” of how we work and talk about how to change culture
  • physicians often seen as leaders by virtue of their role as a physician but they often aren’t trained to be collaborative leaders and they aren’t engaged with the system in which they are working

What I have Learned About Leadership

Interview conducted by TERRY SULLIVAN with MARY JO HADDAD, Past President, Hospital for Sick Children (Sick Kids) Toronto
 
  • after 40 years in the field of child health, seeing kids who come to the acute care system – 70% of whom shouldn’t need to be there – you learn that we have to be concerned about population health, about what’s going on at the community level
  • we have evidence of what we need to do for child health – if we do those things, then in 20 years we won’t be having these same conversations we’ve been having for the past 30 years
  • we need to be OK with leaving our silos – I’ll learn from you and you’ll learn from me
  • we have to be bold and courageous and be clear on the priorities for Canada
  • are we measuring the right outcomes?
  • it costs much to train people in leadership and it needs to be sustained – she found a donor and created a leadership endowment to ensure there will be sustained funding for leadership training
  • we need a culture where everyone is empowered to lead in their practice, wherever they are in the system, and that leadership is respected and valued

Clinical Leadership – The Challenge and Promise of the Next Generation

Presented by SAMIR SINHA, Director of Geriatrics, Mount Sinai Hospital, Toronto
Response panel of emerging leaders:
ROB FRASER and BOBBIE JO HAWKES, Emerging Health Leaders Canada
 
  • aging population: multiple chronic diseases, social frailty, functional impairments – are we set up for these
  • but remember that many of our older adults are actually quite healthy and not using lots of healthcare resources
  • less expensive to care for people in the community than in hospitals
  • how we set out our healthcare budgets (e.g., acute care vs. community) is an expression of our values
  • we need to understand each others’ perspectives (e.g., clinicians learning about financial statements and the issues that administrators have to deal with)
  • clinical leadership and administrative leadership need to work together – “how can the head do anything if it doesn’t know if the feet are ready to walk?”
  • succession planning is an issue that’s coming to the fore because there are mass retirements looming – some quick wins: current leaders can think about what helped (and hindered) them to become leaders and pay that forward; do some job shadowing
  • then an audience member pointed out that we need to hear from emerging leaders about what they need (don’t want to be locked into old biases) – mentors learn from their mentees

Leaders without Borders

Moderated by HUGH MACLEOD, co-chair CHLNet; CEO, Canadian Patient Safety Institute
Panelists include:
  • JOE GALLAGHER, Chief Executive Officer, First Nations Health Authority
  • GRAEME ROCKER, Clinical Improvement Advisor, CFHI
  • LESLEE THOMPSON, President and CEO, Kingston General Hospital

How to spot great leaders outside borders (from Forbes article):

  • focus on people
  • the art of “not or” – don’t fall into either/or thinking
  • ubiquity – every person must lead
  • not tone deaf – tuned into emotional & cultural aspects
  • willing to take the hit – about courage and accountability
  • understand compromise – moves the needle forward
  • no paralysis – live outside the comfort zone – leadership without results is meaningless
  • alignment – best leaders operationalize values, vision, mission, and strategy – understanding shared purpose. With no purpose, there’s no passion. With no passion, there’s no leadership

Joe Gallagher

  • differences in perspectives on leadership
  • have put together a vision of what health and wellness means for First Nations people – holistic, including spiritual and emotional, connection with land and resources
  • leaders need to live a healthy life too
  • BC is moving towards a more preventative system instead of just a “sickness system”

Graeme Rocker

  • traditional medical system has not considered context of patients’ lives
  • e.g., COPD – doing poorly in this area of care
  • people don’t know about their own disease, don’t want to burden anyone, no plan in place about how to deal with it – end up in ER in much worse state than if they’d had support before this, and then we give some emerg care and discharge them back into the broken situation from which they came
  • need conceptual change – asset thinking (we are all in this together, what can we do to improve things) instead of deficit thinking (problems)
  • change from thinking about volumes to
  • high impact leadership (person centred, relentless focus, boundariless, etc.)

Leslee Thompson

  • we often focus on the barriers and we were taught to “colour within the lines”
  • instead of “burning platforms” – foster “burning ambition towards a common purpose” – learning about where everyone is coming from
  • moving from an “ego system” to an “ecosystem” of healthcare
  • leadership is not about you (as the leader)

Patients Canada representative noted that the experts in crossing borders are the patients/families. Also noted a speaker talked about borders not as “barriers”, but as “seams”.

What Policy Leadership is required to achieve the Triple Aim?

Moderated by FRANK MARKEL, Executive Director, CAHSPR
Panelists include:
  • LUC BOILEAU, President, National Institute of Public Health of Québec
  • LINDA MATTERN, Health System Accountability and Performance Division, Government of Alberta
  • SHELLY JAMIESON, CEO, Canadian Partnership Against Cancer

Linda Mattern

5 conditions to achieve social change

    • common agenda- takes time to learn each others languages, etc. to develop a common agenda
    • shared measurement systems
    • mutually reinforcing activities
    • continuous communications – not getting a report every 6 months to the funder; it’s communications in a timely manner to the organization so we know where things are going
    • backbone support organizations
  • we’ve had a business approach to leadership, but need more of a social change approach

Shelly Jamieson

  • as civil servants, we often wait for politicians to ask us to look at things and when politicians do ask for it, we say “we’ll look at it and get back to you in 2 years” – but that’s too long, so they go get an “answer” from someone else and tell civil servants to implement that
  • many stakeholders just stand in line asking for money (saying “people will die if we don’t do X”); a better approach is to know what you want to do and know what the government wants to do. Can tell government “This is your problem. This is how I’m going to solve it. And this is what I need to be able to do that.”

Luc Boileau

  • Every day, Canadians life expectancy increases by 6 hours. The only country that does better than us is France. But this improvement is coming from things other than healthcare (i.e., social determinants of health)
  • Used to think the Deputy Minister had the most power, but the Minister really does have a lot of power

What Do We Need To Do To Build Stronger Leadership For System
Transformation in Canada? Bringing it All Together

Open Forum: Moderated by TERRy SULLiVAN, University of Toronto
 
  • how do we build, spread, diffuse the many great things going on in Canada? Things can’t just be handed over – needs to be adapted to context.
  • we need to understand each other’s complex systems in which we are working so we can work together

Link to the program for the forum.

Posted in leadership, webinar notes | Tagged , , | Leave a comment

Webinar Notes: Knowledge Translation (KT) Planning Primer

NCCMT Spotlight on KT Methods & Tools: KT Planning Primar

hosted by: CH-NET webinar (link)
date: 11 Feb 2014
presenters: Nira Lalji, Andrea Simpson, Ann Coombs (PHAC)

  • KT = work to convert knowledge into action
  • “In an age where we know so much, why are we applying so little of it”  – the “Know-Do Gap”
  • Are we making a  difference  – how much? for whom? and how do we know? = the “So-What Factor”
  • the challenges facing Public Health are complex – need to be thoughtful in how we get knowledge into action

Knowledge Cycle Framework. Source: PHAC

The KT Planning Primer (link to download primer)

  • PHAC wanted to make a KT guide for community-based organizations (as much of what was available when they started this project was directed at academics/researchers)
  • they have a user guide and a worksheet [similar set up to the handbook/workbook the AWESOME project] and have found that it’s difficult to get people to read the user guide – they want to jump right into the worksheet!
  • cascading design to the tool
    • “questions to consider” and “tips” for each section
    • included more detailed information in appendices
Posted in knowledge translation, webinar notes | Leave a comment

Webinar Notes: “Issues in Qualitative Analysis and Reporting”

Presenter: Christine Frank, PhD, CE
Hosted by: Canadian Evaluation Society
Date: 18 Nov 2013

  • qualitative data are “words, actions, interactions, images”
  • objective” exploring in-depth, finding out why….”
  • research method:
    • qual (explore & define concepts & issues) –> quant (measure, generalize, test theory) –> qual (ask why?)
  • triangulation:
    • methods
    • data sources (different times, situations, roles, perspectives)
    • investigator (data collection & analysis)
    • theory
  • trustworthiness (Lincoln & Guba, 1986)
    • truth value: credibility
    • applicability: transferrability
    • consistency: dependability (how systematically did you collect your data? your competence as an evaluator)
    • neutrality: confirmability (would someone else get similar findings?)
  • how is a coding structure created?
    • code references in sources into categories (themes)
    • emergent/open coding: maximizes value of qual over quant; does not restrict thinking to pre-conceived notions, can use first to establish codes for a team; use after broad category coding)
    • pre-determined: ensures focus on key research question; tends to be faster; greatly assists team analysis; very often start with coding by interview question and then emergent coding from there
    • Types of codes:
      • in vivo codes: participants’ actual words
      • researcher-framed codes: may be from literature or program theory; may be term that best captures breadth of a category (e.g., other health problems)
    • Challenges:
      • too many codes – does not help reduce the data, you forget what the codes mean
      • choosing how much text to include in a reference
      • contradictory statements (could be one person contradicting self (might be code “mixed feelings”, but could be different participants or different stakeholder groups disagreeing – that’s good to surface)
      • moderator feeds ideas to participants as they moderate
      • cannot make sense of data
    • NVivo software can do “reliability check” – but time consuming (evaluators often under very tight timelines compared with academics!)
    • how can you quantify qualitative data?
      • she resists it strongly – typically convinces people that it’s the wrong thing to do; reasons include:
        • nature of qual data: people don’t talk in countable discreet ideas (if you want countable, you’d do a survey or other quant method – may do a two phase – qualitative to identify concepts, then survey to test theory); purpose of qual is to go deeper!; ideas evolved as people speak (how do you count that?)
        • coding is subjective
        • interviewer/moderator variances; timing issues
        • with focus groups – people may just nod to what other said, may agree/disagree but not express, may be unequal numbers in groups; one person may dominate or make several points
        • sample not representative (usually small & non-random)
      • possible approaches to limited quantification
        • she reports themes in order of frequency, but explains in report what that means (and does not provide numbers because of above reasons)
        • create a scale for categorization responses (still depends on analyst’s judgement)
        • report presence/non-presence of themes in sub-groups
        • include survey component in focus group or community meetings (still non-random, small sample)
        • for survey with open-ended questions, many researchers code & report frequencies
        • systems such as Trochim’s concept mapping

 

 

Posted in data analysis, evaluation tools, webinar notes | Tagged , , , , | Leave a comment

Webinar Notes: Making Program Evaluation Standards Practical

Presenter: Lyn Shulha, PhD
Canadian Evaluation Society webinar
date: 11 Dec 2013

  • editor of latest edition evaluation standards book
  • evaluation vs. research
    •  evaluation: systematic inquiry
      • systematic in that it’s logical
      • inquiry – asking questions, resolving doubt
    • evaluation is often perceived as risky & threatening to participants
    • research – how is my question situated within what is already known
    • most evaluators have their routes in research (they are “accidental evaluators”)
    • both research & evaluation lead to the creation of knowledge
    • “research & evaluation are cousins” – same decision making processes (data collection & analysis)
    • who “owns the question? and how will the findings be used? are two questions that help differentiate evaluation and research [though I think the line blurs when you start talking about participatory/community-based research]
    • but create different kinds of knowledge: research is about uncovering previously unknown information, but evaluation is about “making judgments about the merit (intrinsic value of the program), worth (getting money’s worth?) and signficance (valued by users) of a program, project or proram component” – ,
  • 3 indicators of high quality research
    1. fidelity to methodology (reliable & valid for quant and explicit, dependability, transparency, adequacy and trustworthiness for qual)
    2. positive peer review (publications) (do others judge your research to be high quality?)
    3. history of continued funding (do people judge your proposed research to be worthwhile?)
  • evaluation standards (currently a book in its 3rd edition) are intended to support design and accountability in evaluative inquiry
  • also there are CES evaluator designation of skill set and other principles for evaluations
  • Joint Committee on Standards for Educational Evaluation (JCSEE) – coalition of stakeholders
  • sponsoring members include CES and AEA (were in fact founding members)
  • standards have evolved over the 3 editions
  • first one was focused on methods, now includes culture, context, and attention to and communication with stakeholders – also evaluation accountability was added in this edition
  • intended to inform those who commission, conduct or use evaluations (helps to demonstrate rigour of evaluation process)
  • 5 dimensions of quality:
    • utility – 8 standards associated
    • feasibility – 4  standards associated
    • propriety – 7 standards associated
    • accuracy – 8 standards associated
    • accountability – 3 standards associated
  • standards are:
    • consensus statement (norms for professional practice)
    • operationalize the attribute so quality
    • provide guidance in evaluation decision making
    • suitable for use across program contexts
  • standards are available online for free – in the book, there’s more detail and background
  • establishing and maintaining the evaluator’s credibility is important
  • note that just having experience in a program or field does not make you able to evaluate it – you need to have the skills and competencies of evaluation
  • there is overlap between the attributes (e.g., work we do to promote accuracy can also promote utility)
  • some of the standards are at odds with each other (need to figure out how to balance them)
  • standards are not a checklist (using more standards is not necessarily better)
  • optimizing one standard may reduce emphasis on another
  • quality evaluations develop an argumentfor the use of the standards
  • you need to consider which standards are most appropriate to do quality work for a given project in its given context
  • the functional table of standards
    • evalaution is not a linear process – sometimes you have to revisit previous decisions as you go along (e.g., negotiating and developing evaluation purposes and questions) – many programs are working in adaptive environments – what they are doing changes as the evaluation goes along
    • growing use of participatory methods suggests there might need to be a new standard around “meaning making”
    • many ways to describe a program (and the functional table illustrates which standards help with that) – e.g., logic model, systems map, fuzzy logic model
    • can use the functional table for meta-evaluation (evaluating your evaluation)
  • takeaways:
    • clients owns the evaluation question
    • focuses on information needs of clients
    • uses research methods most appropriate to clients’ questions (don’t have to be an expert in all methods – working in teams allows you to put the right people with methods skills on a given project)
    • focuses on generating processes, findings, and judgments that are useful to clients
    • standards are practical – communicate the rigour associated with evaluation practice; makes our criteria for professional practice explicit (allows us to be accountable), encourage us to monitor and be accountable for our decision making)
  • Information from Question Period:
  • Katherine Donnelly just did a PhD dissertation related to evaluation & knowledge translation
  • Joint Committee’s mandate is to review standards every 5 years, updated version every 10 years
Posted in evaluation, webinar notes | Tagged , , | Leave a comment

Webinar Notes: Stories of Using Evidence to Inform Health Promotion and Chronic Disease Prevention Practice & Programs

Stories of Using Evidence to Inform Health Promotion and Chronic Disease Prevention Practice & Programs 

Webinar hosted by CHNET-Works on 15 January 2014.

You can download a recording of this webinar.

  • Lack of evidence ≠ lack of action – but lack of evidencet means a chance to add to the evidence base!
  • Evidence-Informed Decision Making (EIDM)- NCCMT definition:
    • use best available evidence
    • sources of evidence (all are legitimate): research, practice, & experience

7 Step rapid review process used by Peel Public Health:

  1. develop conceptual model
  2. search the evidence (use PICO; use synthesized data where possible (e.g., NICE, Health Evidence, CDC, etc.), use as high up on the pyramid of evidence as possible; use grey literature as well as published academic literature)
  3. critical appraisal tool (e.g., http://www.peelregion.ca/health/library/tools-for-rapid-reviews.asp)
  4. data extraction for systematic reviews
  5. assess applicability & transferability
  6. rapid review report structure (1(main messages)-2 (exec summary)-20 (full report))
  7. manager checklist – make sure everything has been done well

 

Posted in webinar notes | Tagged , , | Leave a comment