When Population Health Science Goes Wrong, Part 1.

I read with interest recently the New York Times editorial by Nina Teicholz, titled “The Government’s Bad Diet Advice.” The Times also published some comments on the Teicholz piece, including my brief thoughts. In the piece, Teicholz takes issue with dietary guidelines, many of which are based on nutritional epidemiology studies that have been reconsidered in the 2015 Dietary Guidelines Advisory Committee recommendations (the committee chair is Barbara Millen, a longtime Boston University faculty member). In her piece, Teicholz suggests that “the primary problem is that nutrition policy has long relied on a very weak kind of science: epidemiological, or ‘observational,’ studies in which researchers follow large groups of people over many years.” I am on Teicholz’s side in lamenting the challenges posed by policies that turn out to be wrong. In my assessment, however, her general diagnosis—and her fingering of observational epidemiology as the particular culprit—is wrong and a rather narrow diagnosis for a set of complex problems.

However, what is most striking is the impunity with which Teicholz fingers our quantitative methods in an op-ed page in the “newspaper of record.” That suggests, of course, that the opinion expressed is, to some extent, received wisdom and the articulation of a view that the editors of the New York Times find meritorious. What, then, do we make of the Times allowing the expression of a core, foundational challenge to the quantitative methods of population health science? Is Teicholz right, or is she wildly off the mark? And what has brought us to a place where this perspective has such high-profile currency?

I think there are multiple ways to approach these questions. The easiest might be to dismiss Teicholz’s comments. Harder, but perhaps more productive, would be to see them as a real and very public challenge to the foundational approaches we take in population health science and to consider what it is we are doing that might be contributing to this perception of our approaches. I would suggest that there are five misdirections in population health that contribute, at least in part, to the perception of population health science as being unhelpful in its contributions to an improved health landscape.

I will comment on the first two of these misdirections in this Dean’s Note and the other three in a coming week in the interest of keeping the Dean’s Notes short(er) and more readable.

A Focus on Individual Prediction

Almost our entire investment in health as a society is focused on improving health in the individual—on making sure we develop approaches and initiatives to improve the health of the individual. Although it is notoriously difficult to estimate this, by some calculations more than 90 percent of health spending in the United States is linked to individual health care, and less then 10 percent is devoted to efforts that aim to improve population health. In some respects, the challenge of spending priorities reflects a larger political challenge of resource allocation and of the identification of the best return on society’s investments. But central to the issue at hand is that our fascination with individual health is accompanied by a growing confidence in our capacity to improve and in many ways predict it. That is, of course, the underlying principle behind the dietary recommendations that are targeted by Teicholz, suggesting that if I, the individual, can eat less of X and more of Y, I can improve my health by an appreciable Z increment. This is also the foundation of a plethora of risk prediction calculators, starting with the Friedewald formula that contributed substantially to putting dietary cholesterol “on the map.” Perhaps the apotheosis of this approach can be found in the surge of interest in personalized/precision medicine, which stands to be the foundational and organizing principle behind much of federal health research in the coming decades. But forgotten in this approach is one simple mathematical observation: Our capacity to predict the health of individuals is not very good at all. The data that inform risk prediction models come from population-based studies, and these data are useful indeed for models fit to population-level inference. However, these data fall dramatically short when it comes to individual-level prediction. Although the guidelines are now catching up to this, it has long been recognized, for example, that while cholesterol intake at the population level is associated with higher cardiovascular disease, knowing someone’s cholesterol level does very little to discriminate between whether s/he will have heart disease or not. This fact is perhaps most succinctly illustrated by Figure 1 below.

Why is this? In many ways, this is a mathematical extension of the observation that relative measures of association derived from population level data have very limited discriminant ability to identify which population—one with disease, one without—any individual with a particular characteristic belongs to. This point is simply illustrated in Figure 2 below, showing that only when one reaches population-derived relative measures of association in the order of 300 or so do we achieve population “separation” and confidence that the person with such a characteristic is more likely than not to have disease.

This, then, bodes terribly for individual prediction and directly results in our failure to provide the world with useful estimates of individual health linked to particular exposures—the core challenge posed by Teicholz. We could argue that this suggests a need for a renewed focus on our methods in order to provide us with approaches that are indeed better at prediction than our current approaches. I endorse that. But, at core and absent such methods, we need to make sure that we are clear in our own minds about what we are able to predict, in which group, and that we are informed by suitable humility about the capacities of our methods to move us anywhere near individual prediction. In this context, I worry most about the ascent of precision medicine and its dominance of the biomedical research agenda, predicated on this focus on individual prediction. Absent an appreciation of our limits on this front, we are, societally, investing in an approach that is perhaps compelling but is not grounded in methods that can deliver on its promise.

What Matters and Why

Over the past 30 years, population health science has been centrally concerned with understanding how particular determinants are associated with health, so that we may then work to minimize those determinants and their impact. That is ultimately what has been animating the corpus of nutritional epidemiology challenged by Teicholz, and also what has animated much of quantitative population health science over the past several decades. In defense of the approach, we can note that it has yielded public health successes. However, this approach has also resulted in a baffling array of suggestions about what may or may not be harmful or helpful to health. Building on the referent article: Is butter good or bad for you? One can find evidence that points in both directions. Does green tea prevent stomach cancer, or does it not? This list could go on and on. Why do we have such a proliferation of science that points, maddeningly, in different directions? Although part of this answer rests on the very nature of science itself—we learn through trial and error, we move forward in one direction and then realize we are wrong and therefore correct ourselves—I would suggest that much of the challenge lies in our methods and approaches.

We owe this particular pickle in no small part to a “risk factor” approach to etiologic thinking in population health science. This approach has had us, for the past several decades, working ever harder to isolate causes that produce disease in an effort to intervene and change those causes. I commented on this in my Public Health Forum a few weeks ago. This effort has resulted in substantial and impressive attempts on our part to create ever better methods that control extraneous influences in an effort to identify and isolate single causes. This approach shifts us away from recognizing that rather than thinking about lone causes, we should be thinking about relative weights of health influencers so that we may intervene on that which matters most.

How may we think of what matters most? The core concept here is that the magnitude of the import of any given population health factor is dependent as much on that factor as it is on the prevalence of those other factors that interact with the cause of interest. This might not matter very much if we believed that any one cause works in isolation. But, apart from a few rare genetic germline mutations (e.g. cystic fibrosis), vanishingly few causes do, suggesting that we must understand the prevalence of the other factors in order to determine when intervening on a particular factor might be important. Recognizing that causation of complex diseases rests on a complex network of co-occurring causes necessitates an approach that considers these other factors if we are to determine the relative weights to give our actions.

This is particularly resonant in population health science simply because we are by definition concerned with the determination of health of populations—and, as I commented before, that determination is ineluctably linked to context. Therefore, if we are to understand what matters most, we must understand how a particular factor matters in context. This will require a dramatically different causal inferential approach as well as a mathematical appreciation of the role of context in the production of population health. This all suggests that there are mathematical limits to the extent to which any single individual cause can matter, explaining why, for example, the import of butter or fat to individual diet is inconsistent across studies— studies that vary across contexts—and explaining why authors like Teicholz despair at the observations of population health science. If anyone is interested in a formal mathematical basis for this argument, send me a note and I can send a paper, currently on press at Annals of Epidemiology, that presents this case more thoroughly.

I shall discuss the three other misdirections in an upcoming Dean’s Note.

I hope everyone has a terrific week. Until next week.

Warm regards,

Sandro

Sandro Galea, MD, DrPH
Dean and Professor, Boston University School of Public Health
@sandrogalea

Acknowledgement. This Dean’s Note and the second Dean’s Note in this series are based in part on a body of work developed in partnership with Katherine Keyes.

Previous Dean’s Notes are archived at: https://www.bu.edu/sph/category/news/deans-notes/