The Individual Evidence Pyramid
Evidence-based medicine is increasingly the primary way in which individuals are receiving care and support. Advocates for the approach (Sackets et al 1996) defined evidence-based medicine as:
This definition sets up two sides to evidence-based medicine. One is focused on “external evidence” gathered from systematic research and the other is from “individual clinical expertise.” Based on this, there is an implied balancing of strengths and limitations between these two approaches for making decisions about the care of individual patients. While this definition is laudable, it, unfortunately, suffers from lack of clarity on how the individual patient’s expertise can be taken into account when making decisions for themselves. There is an opportunity to complement these initial two elements of evidence-based medicine with a third one that is based on the expertise that individuals themselves can bring to this equation. To this end, we propose an “individual evidence” pyramid, which is complementary to the classic “external evidence” pyramid and clinical expertise. We discuss how this new type of evidence can become increasingly valuable to individuals by helping them learn about themselves and inform their own decision-making. In subsequent posts, we will also discuss its utility for clinicians.
Informing Initial Decisions: The External Evidence Pyramid
Most evidence-based guidelines are based on some variation of an “evidence-based pyramid.” While there are many variations on this general idea, Figure 1 provides one example.
At the bottom of this pyramid are “anecdotal” evidence and individual “case studies.” Implicit to this model is that, while there is a lot of information and a lot of “case studies,” from an evidence-based medicine standpoint, this sort of information is highly suspect until the evidence can be translated and “moved up” the pyramid. At the top of this pyramid is the “gold standard” approach of randomized controlled trials (RCTs). The idea here is that, if an effect remains true across multiple RCTs, then the effect is likely to be “true,” across individuals and, thus, can be used to support decision-making for individuals as a primary source of external evidence. Strengths of this approach are that it provides a “warm start” for decision-making when little information is available about any given individual. Because there is a rich history of conducting RCTs there is good reason for this to be a “gold standard” for “external evidence.”
While there are strengths to this approach, there are also limitations. First and foremost is the possibility that insights that are gleaned from this type of evidence are not actually relevant to the individual who is making decisions. This is particularly the case because the key strategy of an RCT is the use of randomly assigning individuals to one treatment or another (often a placebo control). This approach, thus, systematically averages out any differences between people, context, or changing circumstances of the individual. This is OK if an intervention works all of the time for all people. If, however, differences between people, place, and trajectory of change of the disease matter when deciding what intervention to use over another, then this approach becomes increasingly fragile.
A key indicator on the likelihood that differences in people, place and time matter is what is called the “heterogeneity of response” that is observed within a given trial. This is a fancy way of saying that different people responded differently to an intervention. For example, one could imagine a weight loss trial that found, on average, people in the intervention lost weight compared to the control group (see below, horizontal lines are the average, vertical lines are individuals). Often hidden under that average insight is the fact that SOME people lost a lot of weight using the intervention but some people didn’t lose all that much weight and some people actually gained weight using the intervention. Similarly, some people in the control lost weight, some maintained their weight, and some gained weight as well. This heterogeneity of response is often lost, particularly when RCTs are then aggregated into systematic reviews and meta-analyses (which is basically like averaging out averaged effects, thus further removing any possible individual differences out of the evidence-base). A cursory scan of RCTs in just about any domain indicates heterogeneity of response (just look for how big standard deviations are in the intervention and control groups… the bigger it is, the more heterogeneity of response there is). Indeed, that is why the definition of evidence-based medicine includes the insights of the clinician as there is recognition that, often, what works “on average” doesn’t actually help some individuals. This point only becomes further magnified when more information about the individual is gathered and decision-making beyond the warm start might be possible.
Informing Later Decisions: The Individual Evidence Pyramid
The individual evidence pyramid (Figure 3) reflects the concept that an individual patient can make conscientious, explicit, and judicious use of their own data to make health-related decisions for themselves. After traditional forms of external evidence are used to provide a warm start when little personal information is available, and after a clinician provides their experience-based insights about the external evidence, the individual evidence pyramid depicts how individual-driven data and processes can become highly complementary to current evidence-based practice.
The foundation of this pyramid is self-tracked data, such as wearables, CGMs, and the like. These data provide the personal evidence one needs, beyond anecdote, to start to gauge and understand a person’s experience. For example, continuous monitoring of glucose data (using a CGM) is highly valuable for understanding just how dynamic glucose levels really are throughout the day in a person with type I diabetes. With those data, this person can start to make guesses on the factors that influence when their glucose levels spike, drop, or are stable. These guesses can then be formulated into hypotheses (moving one level up the pyramid) that could then, ideally, be examined with the data. This work is much like the sort of correlational analyses often done within the other pyramid but, instead of the correlations being generated across individuals, they are generated across time within an individual. Continuing with the example, one could imagine a patient with diabetes hypothesizing that when they get a poor night of sleep, their glucose levels are much more variable. With the addition of a wearable device that tracks their sleep, they could examine the correlation between nights when they slept well (e.g., 8+ hours of sleep) vs. did not sleep well (e.g., <6 hours of sleep) and see if there are differences in how variable their blood glucose levels are on days afterward.
If the individuals aren’t sure if there is a relationship from this type of study, they could consider engaging in a type of N-of-1 study design (if it would be safe to do – more on this in a moment). There are many forms to this but, in brief, it involves systematically activating and then deactivating the “intervention” that the person thinks is impacting the outcome they care about, such as variation in blood glucose levels. This can be done in a quasi-random form, whereby one monitors the outcome of interest under “normal” circumstances, then tries the intervention for a few days, and then goes back to normal (also called an ABA or reversal design). It can also be done using randomization whereby, for a given time period (e.g., one day, one week), a person tries the “intervention” based on random assignment.
Returning to our CGM example, this patient could, during a few weeks when they could easily get a good night’s sleep, consciously decide, on some nights, to get poor sleep. They could then monitor their blood glucose for a few days while getting good sleep, consciously get poor sleep for the next few days (e.g., stay up late watching videos or playing video games), and then go back to good sleep. The decision on when to get a good night sleep could be done pseudo-randomly (e.g., 3 days good sleep, 3 days bad sleep, 3 days good sleep, with order defined beforehand) or it could be done randomly (e.g., each day good or bad sleep is chosen by flipping a coin). This work helps to reduce the likelihood that any relationships are spurious/being caused by something other than what the person had in mind.
It’s important to note here that, sometimes, randomization isn’t possible because of safety concerns or other issues, such as high confidence, already, that an intervention is helpful. As such, another key tradeoff is not only working towards increasing confidence in if X causes Y, but also whether that can be done safely, a point we will return to in future blog posts.
A unique aspect of this pyramid, which is not found in traditional evidence-based medicine, is the use of system identification (system ID). System ID is a method that is used in control systems engineering that was explicitly developed for modeling and understanding the dynamics of individual “systems,” in particular over time . The basic idea involves systematically trying out different types of safe strategies that could feasibly impact the response to a key variable (or variables). The goal is to figure out how a person responds to different strategies in different situations. With that information, dynamical systems modeling can then be used to generate a personalized computational model that is explicitly designed to make future-oriented predictions for that individual on what will happen if various decisions were made in different states of the individual or different circumstances.
Returning to the patient with type I diabetes, if the patient believes that the factors that influence their glucose levels are both highly dynamic and context-dependent, then they could turn to system identification. For example, after establishing constraints for safety (e.g., boundaries on how much or how little to eat), they could systematically vary what they eat (e.g., ratio of carbs to protein and fat), when they eat (e.g., inclusion of fasting), how much they eat (e.g., total calories per day), how many steps they take in a given day, and/or engaging in vigorous physical activity. They would do this is a psuedo-random repeatable fashion, if possible or, at least, document these changes systematically, with the goal of trying out these various activities, particularly in different situations, such as days of the week, different weather patterns, or even in different locations, and states of the individual, such as when they are very stressed vs. not stressed. With all of these factors measured systematically, data analyses, such as dynamical system modeling, can then be run to understand the dynamic inter-relationships between these variables, particularly as they impact glucose measured via CGM. Central to this is that, at the end of a system ID study, the person has developed a mathematical model that, ideally, can be used to predict their personal future blood glucose levels in different situations when different actions are taken. These models are very much like weather prediction for a city but instead of predicting if it will rain at a given point in time, it provides predictions about a person’s blood glucose levels in different states and when different behaviors are enacted. These forecasts can then be used to improve decision-making (which is indeed, what occurs within controllers like the OpenAPS algorithm). As this example illustrates, system ID is a highly conscientious, explicit, and judicious way to use personal evidence to inform a person’s decision-making process.
The Two Pyramids are Complementary
This individual evidence pyramid is a logical complement to the external evidence pyramid and, in an era of increasing emphasis on self-care and patient empowerment, directly supports the inclusion of individual patient expertise in evidence-based medicine. Their complementary nature related to issues like quality, types of insights gleaned, and how they inform decisions is shown in Table 1.
| External evidence pyramid | Individual evidence pyramid | |
|---|---|---|
| How is quality determined? | Predicting the response of groups | Predicting the future response of an individual |
| What are the targeted insights? | Insights about what happens in general | Insights about what happens for an individual |
| How does it inform decision-making? | Supports initial decisions when little to no information is available about a specific person (a warm start) | Supports later decisions when some to a lot of information is available about a person (beyond the warm start) |
Table 1. Illustration on the complementary nature of the two pyramids
Conclusion
We have created this individual evidence pyramid as a frame to help support the conscientious, explicit, and judicious use of evidence created by and for the individual to inform the decision-making process of that individual. The external evidence pyramid provides a structure to use aggregated evidence from group-based studies to make increasingly better initial decisions for an individual, when little or no information about that individual is available. The individual evidence pyramid provides a structure for using information gathered about an individual to support decision-making, particularly over time for that individual (beyond the warm start). As discussed elsewhere, this individual evidence can be aggregated to feedback into the external evidence pyramid, thus further demonstrating the value of the individual evidence pyramid. Further, the individual evidence pyramid could likely be valuable for improving rigor in cultivating robust clinical expertise. Both of these points will be discussed in subsequent blog posts. We welcome further thoughts on how we can help individuals thoughtfully help themselves.