Uncertain Health Informatics Decisions: How Should We Address Them?
Jenna L. Marquard, (panel co-organizer, panelist)
MS, College of Engineering,
University of Wisconsin-Madison, WI
Patricia Flatley Brennan, (panel co-organizer, moderator)
RN, PhD, FAAN, College of Engineering and School of Nursing,
University of Wisconsin-Madison, WI
Stephen M. Robinson, (panelist)
PhD, College of Engineering,
University of Wisconsin-Madison, WI
J. Marc Overhage, (panelist)
MD, PhD, Director, Medical Informatics
Regenstrief Institute Inc, Indianapolis, IN
Kenneth W. Goodman, (panelist)
PhD, Ethics Program,
University of Miami, Miami, FL
Abstract
Decision makers in the field of health informatics face significant challenges in the complex and uncertain nature of the problems they address. In other fields, such as organ transplantation, formalized modeling has proven useful as a tool to aid in addressing difficult problems. While the use of modeling tools holds promise in health informatics decision making, issues and challenges surrounding their successful use must be addressed. This panel will attend to these issues and challenges using the uncertainties faced by regional health information organizations (RHIOs) as exemplars. The panel will present some of the major decisions RHIOs are likely to make, how these decisions are currently made, and the promise of models in aiding these decisions. Panelists will discuss how business problems such as those faced by RHIOs can be translated into formalized models and how these models can be conveyed in a way that is understandable and useful to decision makers. Panelists will also discuss the ethical considerations of using formalized models in health informatics decision making.
General Description
This interdisciplinary panel will address the potential use of formalized models in health informatics decision making, using regional health information organization (RHIO) decisions as exemplars of health informatics decisions. Dr. Overhage, an expert in strategic and operational aspects of electronic health information exchanges, or RHIOs, will identify major decisions that emerging and operational RHIOs are likely to face. Dr. Robinson, an expert in operations research, will focus on how business questions such as those proposed by Dr. Overhage are crafted into useful models. He will also address two specific modeling strategies, stochastic programming and real options modeling, that appear promising in addressing the decisions that RHIOs face. Ms. Marquard’s graduate work focuses on conveying complex models to decision makers in understandable and meaningful ways. She will outline issues surrounding how the modeler, an expert in tools to aid in decision making, can convey developed models to a decision maker, an expert in the system of interest. Dr. Goodman, an expert in the area of ethics and health informatics, will discuss the ethical implications of use and non-use of models in health informatics decision making. This panel brings together experts from complimentary disciplines, whose insights will provide a harmonized view into the use of models in health informatics decision making.
J. Marc Overhage, MD, PhD
Hard Decisions RHIOs Make
The Indiana Network for Patient Care (INPC), a currently operational regional health information organization (RHIO) in Central Indiana, has encountered many of the complex, but common, decisions emerging and operational RHIOs face.
As with many information technology changes in health care, RHIOs must determine the pace at which their network will best develop. One option allows RHIOs to implement their network in a ‘big bang’ style, creating an entire network infrastructure en masse. The complete vision of the network is then operationalized in a single implementation. This may allow RHIO participants to see early gains in the number of sharing partners and types of information shared, but may require significant changes to be made by sharing partners. Another approach to RHIO development is incremental infrastructure implementation. While participants may not see significant early gains, this development scheme allows for greater testing and refinement of the system. As RHIOs approach this development decision, the choice between these schemes is complex and uncertain. In a decision such as this, it would be helpful to understand the potential ramifications of each implementation strategy.
RHIOs also struggle to understand the critical mass of users and shared data needed to create a successful and sustainable network. For instance, if a network is implemented and only 5% of relevant information is shared electronically, there is difficulty knowing whether the system has added enough benefit to users to become sustainable. Likewise, forming RHIOs have difficulty understanding the number and types of network participants needed to create benefit across all users. In this area, it would be helpful to understand the interactions between factors such as the volume and type(s) of data shared, number and type(s) of participants, various pricing strategies, and growth patterns of the network.
As RHIOs develop, the level of information technology currently employed in physician offices continues to be of concern. These physician offices are seen as important sources of patient information, and their lack of electronic health information affects their ability to easily participate in RHIOs. Emerging RHIOs must then understand how necessary physician offices are in reaching the required critical mass of participants in the network. These RHIOs must also understand the level of HIT required by participating physician offices for the RHIO to be successful. Regarding this issue, it would be useful to explore how incentives made to physician offices for the implementation of information technology might affect the success and sustainability of the RHIO.
Finally, network growth must occur in a way that is both scalable and sustainable. Tools that would allow RHIO decision makers to understand the impact of various market structures over time would be valuable to operational RHIOs.
Stephen M. Robinson, PhD
Transforming Business Questions into Useful Models
Operations research-based modeling tools, as described by the professional organization INFORMS, help “understand and structure complex situations and … use this understanding to predict system behavior and improve system performance …using analytical and numerical techniques to develop mathematical and computer models of organizational systems composed of people, machines, and procedures.” While this discipline offers a variety of tools that may be helpful in making complex and uncertain health informatics decisions, two related approaches, stochastic programming and real options modeling, seem to apply particularly well to the RHIO problems described by Dr. Overhage.
Stochastic programming is a modeling tool for finding decisions that are best, according to some specified criterion, in problems containing significant uncertainties. It lets us examine both the array of possible future events and the system’s responses to them. We can then build multi-stage decision models to evaluate present courses of action against uncertain possible futures, often described by scenarios. In the context of the RHIO problems, stochastic programming could be used to examine the uncertain consequences of various network implementation strategies.
Real options models also have potential in evaluating economic decisions over time in an uncertain environment. In addition to modeling somewhat different kinds of uncertainty than stochastic programming usually does, these models impose an assumption that there are no arbitrage opportunities in the system. In other words, there is no opportunity to make profit by trading any valuable resources in the system at no risk. This assumption is important because it eliminates mispricing of resources and arbitrarily chosen discount rates. This method also allows for the quantitative evaluation of important qualitative concepts such as flexibility. This modeling method may be useful in looking at decisions such as those Dr. Overhage pointed out concerning single vs. incremental infrastructure implementation problem, or incentives to physician offices for implementation of IT.
Jenna L. Marquard, MS
Making Sense of Models
The modeling tools described by Dr. Robinson have the potential to be useful in addressing uncertain decisions, but this usefulness only comes when a decision maker appropriately integrates the model into their decision making. Appropriate integration of a model into decision making means neither discarding the model without examining it, nor trusting the model completely. Decision makers must instead understand that models are a lens placed on a given system, allowing the modeler to structure the system in an explicit way. As George Box stated, “all models are wrong, some are useful”.
Given that a model is a lens placed on a system to give it structure, it becomes important for a decision maker to not only understand the outputs of the model, but also the characteristics of this lens. Such characteristics may include uncertainties embedded in the model, overt or latent biases and assumptions made by the modeler. If a decision maker has a clear understanding of these characteristics, they may be better able to assess the usefulness of the model in their decision making.
If decision makers are to make sense of models and use them appropriately in decision making, both model outputs and the lens used by the modeler must be conveyed in a way that is understandable and useful to the decision maker. In this way, the decision maker needs both a heightened understanding of the system of interest, via the model, and an understanding of the lens that the modeler has used when structuring the system.
Decision makers with a clear understanding of the model and the basis on which it was formed can play a more active role in the process of model validation. While modelers understand how to build tools to aid in decision making, they often do not have the deep situational understanding of the system that decision makers do. These complementary skill sets can join together to create models that are more valid, useful and actionable.
Kenneth W. Goodman, PhD
Consequences of Using, and Not Using, Models
Health informatics decision makers facing uncertain situations must attend to not only the decision at hand, but also to their approach to decision making. Given the aforementioned modeling approaches and their potential usefulness, examining the implications of use and non-use of these models is important.
At one extreme, a decision maker might trust a model of a system without question. This may signify a belief by the decision maker that models are replacements for, instead of supplements to, intuitive decision making. Without understanding the model or analysis, along with its biases, uncertainties and assumptions, the decision maker has placed stock in a tool that may or may not be appropriate for a given decision. This action can be likened to an Alpha (Type I) error, or allowing an ineffective drug onto the market. In this way, there are ethical ramifications to a decision maker utilizing a model without understanding its capabilities and limitations.
Another extreme scenario exists when a decision maker dismisses the use of a model when that model could provide significant insight into the system in question. This action can be likened to a Beta (Type II) error, or keeping an effective drug off the market. Just as a surgeon may face scrutiny for dismissing an alternative technique shown to provide added benefit to a patient, decision makers in health informatics must understand the potential ramifications of not using a promising tool or method in their decision making.
These two scenarios appear extreme, but are common reactions to the use of formalized models in decision making. The scenarios have different, but equally significant ramifications. As health informatics decision making moves forward, decision makers must calibrate the usefulness of formalized models in their decision making process through heightened understanding of the models. While the consequences of replacing decision maker intuition with modeling techniques must be attended to, so must the consequences of not considering the use of these models.
Statement from the Co-Organizers
All panelists agree to participate in the panel, given that it is accepted by the AMIA 2006 reviewers.