Category: evaluation

Notes from the academic paper: A literature review for large-scale health information system project planning, implementation and evaluation

Health information systems (HIS) have potential to increase efficiency and save considerable amounts of health expenditure.

Current rates of adoption of health information technology are low and health information systems are under-utilized.

Kaplan notes that “there has been a long history of difficulties in achieving clinical use of some kinds of clinical informatics applications. ” Within this context, it is imperative that HIS implementation is evaluated and features of successful implementation identified.

Eysenbach summarizes the goals of technology in healthcare, suggesting that eHealth should be: (1) Efficient, thereby decreasing costs, (2) Enhance quality of care, (3) Evidence based, proven by rigorous scientific evaluation, (4) Empowering for consumers and patients, (5) Encouraging a partnership relationship between patient and health professional, (6) Educate physicians and consumers, (7) Enabling information exchange and communication in a standardized way between health care establishments, (8) Extending the scope of health care beyond its geographical and conceptual boundaries, (9) Ethical – e-health involves new forms of patient-physician interaction and poses new challenges and threats to ethical issues, and (10) Equitable. Haux identified seven general tasks of HIS over time. These are [59]: (1) to move paper-based processing and storage to computer-based; (2) to move from local to national and global HIS; (3) to include patients as HIS users; (4) to use HIS data for healthcare planning, clinical and epidemiological research (aside from patient care and administration); (5) to change the focus from technical aspects of HIS to management change and strategic information management; (6) to place more emphasis on image and molecular data; and, (7) to acknowledge the steady increase of new types of technologies, perhaps as yet un-imagined.

The reasons cited in favor of the implementation of HIS are primarily around efficiency, cost, quality and safety.

No matter how success or failure is defined (if it is) the evidence of effectiveness is generally weak and inconsistent. Information systems of all types notoriously fall short of their expectations and fail to deliver benefits (see for example, Gauld and Goldfinch. Shpilberg et al.  reported that only 15% of business executives surveyed believed that their company’s IT capability was highly effective, ran reliably, and delivered projects with promised functionality, timing, and cost. Systematic reviews of healthcare settings consistently find that there is little evidence that care provided by technological innovations is any better than traditional methods. Whitten’s systematic review of HIS cost-effectiveness found that there is no conclusive evidence that telemedicine is a cost effective way of delivering healthcare. Mistry reviewed the cost-effectiveness literature ten years later and concluded that the results of their review were consistent with previous findings: there is no further conclusive evidence that technological interventions are cost effective compared to conventional healthcare. However, it is also the case that these reviews noted methodological shortcomings in studies evaluating cost effectiveness. These were particularly around the amount of methodological detail provided and the methods used to measure cost effectiveness.

Effective evaluation allows us to understand how and under what conditions HIS work, and determine the safety and effectiveness of the system. Evaluation can provide guidance to the implementation process and mitigate unplanned negative outcomes. Ammenwerth defines evaluation as “the act of measuring or exploring attributes of a health information system (in planning, development, implementation, or operation), the result of which informs a decision to be made concerning that system in a specific context.” This should include the inevitable organisational change which accompanies the implementation of HIS. Early approaches to evaluation focused on the “measurement of changes in processes and of the consequences of these changes” while more recently attention has been paid to the complex, iterative and multidimensional implementation process. Effective evaluation accompanies the whole life cycle of HIS, evaluating technology against a comprehensive set of measures throughout all stages. Measuring the success of HIS is not straightforward and the challenges in the organisation and setting of HIS make both implementation and evaluation of the HIS difficult. Evaluation processes are also often flawed.

Notes from academic paper: Review: On the problems of mixing RCTs with qualitative research: the case of the MRC framework and the evaluation of complex healthcare interventions.

The ontology of critical realism, with its identification of numerous causal mechanisms interacting in different ways in different contexts to produce different outcomes, has influenced the development of realistic evaluation, as advanced by Pawson and Tilley (1997). The aim of realistic evaluation is to explain the processes involved between the introduction of an intervention and the outcomes that are produced. In other words, it assumes that the characteristics of the intervention itself are only part of the story, and that the social processes involved in its implementation have to be understood as well if we are going to have an adequate understanding of why observed outcomes come about. In contrast to the assumptions of constant conjunction, realistic evaluation posits the alternative formula of:

Mechanism + contex = outcome

In any given context, there will in all likelihood be a number of causal mechanisms in operation, their relationship differing from context to context. The aim of realistic evaluation is to discover if, how and why interventions have the potential to cause beneficial change. To do this, it is necessary to penetrate beneath the surface of observable inputs and outputs in order to uncover how mechanisms which cause problems are removed or countered by alternative mechanisms introduced in the intervention. In turn, this requires an understanding of the contexts within which problem mechanisms operate, and in which intervention mechanisms can be successfully fired. In other words, realistic evaluators take the middle line between positivism and relativism, in that positivism’s search for a single cause is seen as too simplistic, while relativism’s abandonment of any sort of generalisable explanation is seen as needlessly pessimistic. In contrast to these two poles, realists argue that it is possible to identify tendencies in outcomes that are the result of combinations of causal mechanisms, and to make reasonable predictions as to the sorts of contexts that will be most auspicious for the success of health-promoting mechanisms. The confidence of prediction can be increased through comparison of different cases (i.e. different contexts) in that concentration on context–mechanism–outcome configurations allows for the development of transferable and cumulative lessons about the nature of these configurations.