By Bruce Ramshaw, MD, FACS
Controlling All of the Variables?
In December 2003, Gordon Smith and Jill Pell published an article in the British Journal of Medicine entitled, “Parachute Use to Prevent Death and Major Trauma Related to Gravitational Challenge: Systemic Review of Randomized Controlled Trials” (BMJ 2003;327:1459-1461). It was a tongue-in-cheek demonstration of the lack of common sense sometimes exhibited when groups attempt to apply mechanical tools, like prospective, randomized, controlled trials (PRCTs) to health care. It is quite ironic that the study they described, testing the benefits of using a parachute when jumping out of an airplane is actually a much better application for the use of a PRCT than most of the treatments and tests we have subjected to PRCTs in our complex health care system. Let me try to explain.
PRCTs are designed to test a hypothesis in a mechanical (simple) or isolated system. It requires that all-important variables be controlled so that the intervention can be tested against a control group under like circumstances. For the parachute study described in the BMJ article, this could actually be done fairly well: The height of the plane and the landing surface could both be reasonably well controlled. Other factors, often important in other medical studies, such as DNA variation, body mass index, medications, age, etc., probably don’t matter to the primary outcome in this trial. A 65-year-old diabetic woman on chemotherapy will probably have a result similar to that of a healthy 30-year-old man.
So, the test would actually result in an outcome that can prove to be clinically significant and likely provide a research conclusion that is generalizable to other populations. This is not the case for the great majority of clinical research in health care, where the variables are much more important and much more difficult—actually impossible—to control. Also, any attempts to control the population being tested (inclusion and exclusion criteria) lead to outcomes that are less likely to apply to the real world. Even traditional techniques of controlling variables, such as using identical twins to limit DNA variation are now known to be insufficient and ineffective. By the time they are adults, “identical” twins can have more DNA differences than similarities, and these changes may start in utero.
Unfortunately, most of our health care treatments are not so simple, and the more complex the problem is, the less useful a mechanical tool like a PRCT is. The outcomes of the parachute study would likely be clear and generalizable: Wearing a parachute when jumping out of a plane (at a certain height and speed) would not be wasteful, would not be harmful and would save lives. Almost all of our health care treatments and tests are different; they are complex. Even for the most clearly supported treatments, such as the vaccine and screening mammography examples I used in part 1 of this article (March 2014, page 1), there is a significant amount of waste (the person receiving the treatment or screening test did not really need it) and some real harm (death and harm done to children and adults from the vaccines and over diagnosis and overtreatment in women after mammography).
This reality challenges our beliefs and our thinking. If there were no alternative, then I imagine this reality would be a moot point. However, complexity science has evolved over 100 years and can contribute tools, such as clinical quality improvement principles and complex systems (nonlinear) data analytics and predictive analytics, that will allow us to better define patient groups that are helped, those that are harmed and those for which a treatment or test is wasteful. Over time, this will allow us to improve the value of patient care for all care processes in which we apply these principles.
The research is conclusive and the science of complex adaptive systems is clear: We are using research tools designed for static, isolated, linear and mechanical systems, but humans beings are nonlinear, adaptive, biologic and heavily influenced by interactions with the rest of our constantly changing world. Because applying the principles of complexity science to health care is in its infancy, there are only early, small examples to show the benefits, but in other industries and applications, these principles have led to impressive results. Two examples come from the study of spaghetti sauce and the game of Jeopardy.
Defining Clusters and Patterns
In one of his TED talks, Malcolm Gladwell tells a story about the brilliance of Howard Moscowitz and how we have all benefitted from his enlightened understanding of the consumer food industry. Moscowitz is a psychophysicist, and over the past several decades, he has been frequently hired by food companies to consult and help them determine the best tasting ... whatever product they were focusing on at the time. It could be how much artificial sweetener to put into a diet soft drink, the best blends of coffee or whatever. He would set up tasting focus groups and collect data from sometimes thousands of people to determine the best next product. But he struggled with the data because as much as he tried, it did not fit well on a bell curve to give him that one best answer.
Moscowitz knew a direct study comparing two variables would not work, so he did observational testing with a variety of options and collected a vast amount of data. It still did not lead to a single best product. While working for Prego, he had his “aha” moment. Moscowitz had collected his data and it hit him: People’s tastes did not exist on a bell curve, but they tended to cluster around a variety of choices. It was not a simple bell curve, but it was also not completely chaotic or random; there were patterns and clusters that could be observed. And this led to a revolution in the consumer food industry: many more varieties of spaghetti sauce and many choices of variations in taste for almost everything else we eat and drink.
Interpreting data from a complex and changing system is different than collecting data from a simple, static and isolated system. These complex systems data analytics will be necessary to improve the value-based outcomes in our health care system. But, how do we get the data? There are plenty of data in health care. Why can’t we analyze the massive amount of data that already exists? In a simple answer, it is the lack of context that produced the data. There are also problems with the accuracy and relevance of the data as well as how the data is analyzed and used in our current system for patient care. Understanding these concepts and an example of predictive analytics will hopefully help to explain how to apply these principles to health care.
Predictive Analytics: Context and Collaboration
One of the most successful, and famous, applications of predictive analytics was demonstrated for three days of competition on the long-running game show Jeopardy. Ken Jennings and Brad Rutter, the two most successful champions in the history of the game, were pitted against a computer, IBM’s Watson. Knowing how Watson was programmed to attempt to beat the best of the best Jeopardy contestants can help us understand how to apply predictive analytics to health care. The key to Watson’s somewhat surprising victory was not just filling the computer with searchable facts (a la Google or Wikipedia), the key was to provide context for the knowledge and generate the knowledge of learning language from multiple collaborations. Instead of asking the Jeopardy champions to recommend what knowledge should be put into the computer (which would be like asking physician experts to enter knowledge about diseases and treatments), the programmers realized that the real context would come from Jeopardy’s question writers, so they loaded all prior questions and answers ever written for Jeopardy going all the way back to the first episode on March 30, 1964. They also needed to have the best computer understanding of the English language, termed natural language processing. Rather than just entering the “best practice” from one natural language programming laboratory, they entered the research and findings from many of the laboratories, with their various findings and strategies and pooled them together in Watson. In a multiyear, iterative process—many testing phases were done almost continuously—Watson’s ability to predict the correct answer improved over time. There were failures along the way and the outcome of the competition was not guaranteed, but the result was impressive, with Watson scoring $77,147 compared with Ken Jennings’ $24,000 and Brad Rutter’s $21,600.
The New Paradigm
And now, this application of complexity science, or the ability to apply complex systems data analytics, is just beginning to be applied to health care. In fact, Watson has been contracted to perform data analytics services with several health care organizations. But, the ability to use health care data to generate accurate predictive analytics is in question. So far, the application of data analytics to health care has omitted consideration of the essential contribution of context and collaboration. Instead of defining the true context of the patient’s cycle of care, either poor data (payer claims data) or misunderstanding of context (asking doctors what information to put into the computer), or both, are used.
Asking the doctor, the expert, is like asking the Jeopardy champions what information to put into Watson, instead of the question writers. Since we don’t have question writers for our patients’ health care problems, we will need to define their processes of care (the questions) and their value-based outcomes (the answers). This context will allow us to iteratively improve over time and lead to ongoing improved patient value. The other necessary component is collaboration. We will need many teams defining patient processes and improving value-based outcomes. Otherwise we will have limited ability to continue improving, termed the law of diminishing returns.
For Watson and for Howard Moscowitz, using one “best practice,” one natural language processing lab or one person’s taste, was inadequate. It is the same in our complex system of health care. It is true that when we try to implement ideas for improvement, we can make mistakes and have bad outcomes and unintended consequences. In fact when the programmers loaded the Urban Dictionary in Watson in an attempt to have an improved range of language understanding there was an unexpected consequence. In tests, Watson answered “Bulls—t” to a researcher’s query. The team decided to remove the Urban Dictionary from Watson’s memory. We already have a massive amount of waste and harm in health care, and almost none of it is intentional. But, to implement a new way of thinking and evolve our system structure for patient care we will need to be prepared to acknowledge and deal with unintended consequences. We will all need to be accountable to the patient problems that we choose to diagnose and treat.
In his book “Predictive Analytics: The Power to Predict Who Will Click, Buy, Lie or Die,” Eric Siegel describes the story of IBM Watson’s Jeopardy victory in more detail. He said it is one of the best examples of machine learning and the potential for wonderful benefits if applied appropriately to health care. But, he is clear in the need for an accurate context that produces the data and the need for multiple collaborations to share what is learned from the data. In health care, we have a very poor understanding of the context that produces the data available that we see in fragmented care, often primarily from coding and billing data. This data are incredibly inaccurate. Some estimate that 30% to 40% of billing data is wrong, with the great majority from human and systems error, not fraud. But when predictive analytics are done well, and Siegel gives many examples in his book, then the use of a well-defined context and meaningful collaboration can prevent the law of diminishing returns, or as Siegel calls it in his book, “overlearning,” which can allow for continuous improvement of value.
We are going through a paradigm shift in the scientific understanding of our world, from the machine as a metaphor for human beings to an understanding that we are complex, adaptive and dynamic systems—yes, imperfect and nonlinear, but able to adapt in many various and wonderful ways. We can deny this shift, we can try to ignore it, we can even be defensive and aggressively argue against it, but what no one can do is prevent it from occurring. On the other hand, if we decide to accept it (and it is a conscious decision) and go through the discomfort of opening our minds and learning about the science behind it, we will begin the adventure of our lives. This journey is never over and it can be uncomfortable at times, but as we truly learn about the complexity of our world, the hope and potential for improvement becomes real and palpable.
The simple rule that emerges to achieve this potential is clear and consistent. To transform our health care system to one that is sustainable and always improving the value for the patient, we will need to treat each other and our patients with empathy, compassion and love. In the final article in this series I will present the human potential that is in all of us to drop our fears and defenses and allow our authentic selves to be present as we do our work in caring for each other.
Dr. Ramshaw is chairman and chief medical officer, Transformative Care Institute (nonprofit) and Surgical Momentum LLC (for profit), and co-director, Advanced Hernia Solutions, Daytona Beach, Fla.