Evaluating informatics applications – some alterative approaches: theory, social interactionism, and call for methodological pluralism
- evaluating informatics tools, such as clinical decision support tools, under controlled conditions doesn’t provide information about how context (including human and cultural factors) affect whether those tools will actually be adopted in the real world
- as well, focusing exclusively on pre-determined outcome measures means you will miss out on learning about the processes by which the system is actually used, as well as unanticipated and emergent effects
- evaluations need to take into account “social, organizational, professional, and other considerations” (p. 40)… such as “power, culture, group relationships, work routines, stakeholders, professional values, social networks, institutional organizations, and judgement” (p. 40), but note that these things “elude quantitative and RCT-type evaluation approaches” (p. 40)
- reports of evaluations conducted in a real-world setting often fail to include information about the settings in which they were conducted, making it difficult to interpret the findings.
- when an RCT/experiment is conducted and finds a clinical decision support system to be ineffective, it does not generally provide answers to the question “why not?” so it does not “help in building better systems or in preventing decisions that may result in abandoning technologies that could potentially be useful” (p. 43)
- cites Davis & Taylor-Vaisey: “The adoption of any innovation or the dissemination of new medical knowledge should be considered in a holistic, contextual manner” (p. 43)
- “information technologies are embedded within a complex social and organisational context” (Heathfield & Buchan, 1996 cited on p. 46)
- “artifacts cannot be understood independently from how they are use in actual practice” (p. 46)
- “in sociotechnical theory, a change in technology, people, task, or structure is seen to result in adjustments by the other three components in order to maintain organizational stability. Berg focuses on work practices and how individuals, tools, documents, and machine are cooperative elements in emergent networks that make work practices function smoothly” (p. 47)
- “neither the environment nor the system itself is stable […] there is something of a co-evolution of the environment and the system” (p. 47)
- “a social influence or social interactionist approach […] takes account of the kinds of social, political, cultural, historical, institutional, cognitive, and other contextual constituents of the change process” (p. 47)
- with medical information applications: “how the technology is used and what changes occur are thought to result from coplex social interactions. Because users may modify information systems during design, implementation, and sue, they are views as active participants in what occurs” (p. 47)
- “characteristics of the technology, of the developers and potential users, and of the organizations into which they are introduced, are seen to interact with each other and may themselves chance through these interactions. The participants, the setting, and the technology are treated as dynamic emergency processes rather than as variables that can be held constant, and causality is seen to be multi-directional rather than uni-directional. Social interactionist evaluation involves studying social, political, organizational and related processes as they unfold over time.” (p. 47)
- researcher draw on:
- theories of change
- social science theories
- may use “an interpretivist approach and study what meanings individuals ascribe to the technologies and processes under study” (p. 48)
- Kaplan suggested guidelines for such studies:
- “focus on a variety of concerns
- use multiple methods
- be modifiable in study design
- employ longitudinal designs
- conduct formative as well as summative evaluations” (p. 48-49)
Evaluating information technology in health care: barriers and challenges
- “evaluation is not just for accountability: but for development and knowledge building in order to improve our understanding of the role of information technology in health care and our ability to deliver high quality systems that offer a wide range of clinical and economic benefits.”
- basing decisions on healthcare IT on failure of RCT’s to show improved outcomes–> may cause prosing technology to be abandoned prematurely
- basing decisions on healthcare IT on unsubstantiated reports (e.g., no actual evaluation) –> may cause resources to be wasted on ineffective IT and/or inappropriate application of IT
- “many evaluation studies ask inappropriate questions, apply unsuitable methods, and incorrectly interpret results. The evaluation questions most often asked include those concerning economic benefits and clinical outcomes, despite lack of strong evidence of such and the recognition of the difficulty of applying results in other context”
- RCTs
- are “vulnerable with respect to external validity: trial results may not be relevant to the full range of subjects (that is, specific implementations of a healthcare application) or typical uses of a system in day to day practice”
- “negative results from[RCTs] cannot help us understand the effects of clinical systems or build better ones in the future.”
- even if an RCT demonstrated benefits, it “does not necessary mean that end users will accept a system into their working practices”
“As pointed out by McManus, “Can we imagine how randomised controlled trials would ensure the quality and safety of modern air travel …? Whenever aeroplane manufacturers wanted to change a design feature … they would make a new batch of planes, half with the feature and half without, taking care not to let the pilot know which features were present.”
Evaluation and Implementation: A Call for Action
- evaluation of information systems is moving towards “more holistic view of information systems and their evaluation” (p. 12)
- can apply “the socio-technical approach towards evaluation: try to understand why information systems are a success or a failure, taking into acount the social context in which the systems are used” (p. 12)
- the “sociotechnical approach should not only try to determine whether an implementation was successful or not. It should also contribute to developing theory and good practice for successful implementations” (p. 14) – requires “close observation and evalaution of implementation processes” (p. 14)
- in the “declearation of Innsbruck, “an information system is defined as the techincal artifact and the environment (social, organizational) in which it is used” (p. 12)
- existing studies trying to evaluate whether IT systems provide some sort of “benefit” or “value” tend to be inadequate because:
- too little time allocated for evlaution
- too few resources allocated for evaluation
- unclear objectives of the evaluation
- weak evaluation methods (including starting too late to get baseline data)
- poor reporting (e.g., drawing conclusions not based on evaluation data (but on personal opinion instead); not clearly distinguishing between expected benefits and actual benefits (i.e., not measuring whether benefits occurred, but just assuming they did)
- there is a need for:
- “an evidence base of good evaluation practice”
- “improvement in the reporting of evaluation studies” (p. 14)
References:
Healthfield, H., Pitty, D., & Hanka, R. (1998). Evaluating information technology in health care: barriers and challenges. British Medical Journal. 316(7149):1959-61 (full text)
Kaplan, B. (2001). Evaluating informatics applications–some alternative approaches: theory, social interactionism, and call for methodological pluralism. Int J Med Inform 64(1):15-37 (abstract)
Talmon, J.L. (2006). Evaluation and implementation: A call for action. IMIA Yearbook of Medical Informatics. 11-15.
