BY JEFF LAMBART
Every year in the United States, roughly $4 billion are spent performing 700,000 surgeries that we know don’t work.[1],[2] Why? In part because, on a superficial level, it sounds so plausible. This procedure, an arthroscopic partial meniscectomy, is a knee surgery that purports to relieve pain by trimming a frayed meniscus to return it to its normal shape. The simplicity of “fixing” the meniscus in this way appeals to patients, and doctors are encouraged by seeing patients improve after the surgery. But the preponderance of rigorous studies, including a randomized control trial where placebo surgeries were performed, indicate that, for most patients, this surgery is no better than physical therapy alone.
The fact that this surgery remains so common despite lacking proof of its efficacy should be a stark warning for proponents of evidence-based policy making. Investing in more rigorous evaluations alone isn’t enough.
There are clearly many barriers to aligning policy making with the best available evidence. This article will focus on how it can be particularly difficult for evidence-backed ideas to win out over ideas that make intuitive sense. As examples ranging from auto-traffic congestion to addressing nutritional inequality will show, there is rarely any feedback mechanism to hold policy makers accountable for the true impact of their decisions. The political rewards of doing something that constituents believe will work are often equal to if not greater than the payoff of taking action that actually does work. This dynamic is a serious threat to effective evidence-based policy making and represents a major missed opportunity to do more to improve lives.
Through a series of policy cases across a variety of domains, it is clear that common-sense solutions are not always what they seem. We need systems to separate what works from what does not in order to hold policy makers accountable for real results, rather than relying on intuition to guide ultimately ineffective action.
New Lanes, Same Congestion-Induced Demand
The canonical example of a policy idea that continues to be implemented despite considerable opposing evidence is expanding highway capacity to reduce congestion. The underlying logic is undeniably appealing to anyone who has ever wished there was another lane while stuck in rush-hour traffic. However, dating back to a seminal 1960s study by economist Anthony Downs, evidence has accumulated that expanding highway capacity has minimal positive impact on traffic. According to Downs’ “Law of Peak Hour Traffic Congestion,” demand for travel during rush hour tends to rise to meet the maximum capacity.[3] An American Economic Review article from 2011 confirmed this finding, explaining that new road capacity encourages existing residents and businesses to make trips they previously would have foregone. This effect can lead to new commercial and housing developments that add cars to the road until congestion once again becomes the norm.[4]
While there are benefits to allowing more residents to travel on the highway, these projects are routinely sold to taxpayers as a remedy for congestion, despite strong evidence that new lanes do not reduce travel times. In 2003, then-Governor Rick Perry promised Houston area commuters “relief from congestion” and “solutions to combat the gridlock grind” when announcing what eventually became a $3 billion expansion of the Katy Freeway.[5] One analysis found that travel times on the now widest freeway in the world increased by 25 minutes (30 percent) during the morning commute and 23 minutes (55 percent) during the evening commute.[6] Houston is far from alone. Traffic was found to be moving slightly slower after the recent expansion of the I-405 freeway in Los Angeles.[7]
The Danger of Assuming Causal Links: Food Deserts and Nutritional Inequality
Cynical readers may interpret the above example as a case of politicians knowingly making a false promise of congestion reduction to sell a project they want for other reasons. However, efforts to address nutritional inequality and food deserts demonstrate that the attraction of an intuitively compelling solution can leave even the most well-meaning of actors on the wrong side of the evidence.
Nutritional inequality is a significant problem in the United States. One out of every eight American households is food insecure,[8] and a 2010 Journal of the American Medical Association analysis found that the gap between the nutritional quality of the diets of low- and high-income individuals has widened over time.[9] Simultaneously, proximity to supermarkets or other vendors of healthy food options is wildly unequal; for example, 55 percent of zip codes with a median income under $25,000 lack a supermarket.[10] These areas, often referred to as food deserts, have worse nutritional outcomes and higher rates of obesity.
Linking these two factors through a supply-side lens—the lack of healthy food options causes the poor nutritional outcomes—is so intuitive that it hardly seems worth testing. Indeed, the US federal government has felt comfortable making large investments to remedy food deserts in an effort to improve nutritional outcomes, such as the Healthy Food Financing Initiative, which spent over $220 million in grants and technical assistance to support suppliers of healthy food in underserved areas.[11] However, the best available evidence tells a more complicated story.
A recent study by faculty at New York University, Stanford University, and the University of Chicago’s Booth School of Business joins a growing literature that suggests that the lack of nearby healthy food suppliers has only a marginal impact on nutritional inequality.[12] A recent empirical analysis in Applied Economic Perspectives and Policy challenges the assumption that people purchase the majority of food at local retailers.[13] More recent research emphasizes instead how factors such as the price of healthier food and different levels of nutritional knowledge drive differences in nutritional outcomes.
To be fair, many advocates and policy makers are now aware of this evidence and promote interventions that go beyond merely remedying the lack of supply. For example, the same NYU, Stanford, and Chicago Booth researchers found that an annual $830 million subsidy for healthy food targeted at the bottom 25 percent of the income distribution would “increase healthy eating by the same amount as providing bottom-quartile households the same supply conditions as top-quartile households.”[14] They forecast that a similar subsidy of $11 billion per year (15 percent of the current Supplemental Nutrition Assistance Program budget) could effectively eliminate the difference of healthfulness of food purchased between bottom-quartile and top-quartile households.[15] However it is still easy to find support for supply-side solutions that lack any proof of their efficacy. For example, the Centers for Disease Control and Prevention’s webpage continues to encourage policy makers to use zoning changes to incentivize health food retailers to open in underserved areas.[16]
New Risk on the Horizon: Health Care Price Transparency
Readers who are convinced that the evidence-based policy movement has now progressed sufficiently to avoid falling prey to intuitive but ineffective solutions should consider the current policy debate around rising health care costs. President Trump and Democrats alike have advanced various proposals that encourage greater price transparency in the American health care system.[17],[18] Once again, the intuitive logic behind such proposals is appealing: transparent prices, proponents argue, give consumers more information, spur competition among providers, and cause the most expensive providers to reduce their prices for fear of negative press.
The evidence that any of this will actually come to pass is thin at best. In fact, Harvard University’s Leemore Dafny and David Cutler claim that certain types of price-transparency schemes may actually lead to higher health care prices.[19] Specifically, providers would not want to offer price reductions to certain insurers if those deals would subsequently be made public, as other insurers would then push for a similar discount.[20] After noting that there is only limited research on the effects of transparency initiatives, the researchers describe several examples of price-transparency schemes in other sectors that harm rather than help consumers.[21]
Importantly, there are many ways to design a price-transparency system and much more research is needed to understand how such a proposal might best help consumers. However, this issue is one that supporters of evidence-based policy should watch closely. There is a serious risk that policy makers will continue to push intuitively appealing price-transparency schemes as a quick win, despite lacking evidence of their efficacy, while ignoring more impactful policies that are more difficult to explain to voters or pass politically.
A Missed Opportunity We Can’t Afford
The fundamental reason why common-sense ideas can prevail despite evidence to the contrary is that our political system is unable to hold politicians accountable for whether or not their proposals actually work. Without proper feedback mechanisms, politicians are rewarded only to the degree that voters believe their ideas will work.
In each of the above examples, simple theories of change and highly visible outputs are appealing to policy makers and the public. Adding new lanes to a highway or opening a grocery store are undeniably easier to sell than potentially more effective but less straightforward solutions such as congestion pricing or subsidies for healthy food. Empowering consumers and encouraging competition through transparency are similarly easier cases to make than more complicated or contentious attempts to rein in high health care prices.
Rigorous evaluations of the impact of policy changes are still too rare. More importantly, even when they do occur, they are too rarely used to hold decision-makers accountable, either because they are not widely publicized or because they become available years after policy makers who initiated the program have left office. Creating a virtuous cycle that rewards policy makers for highly effective governance and incentivizes further adoption of evidence-based practices is therefore highly complex. The absence of a feedback loop stymies progress toward a higher-performing public sector in a number of areas.
Until such a system emerges, proponents of evidence-based policy need to be especially wary of the power of common-sense ideas to continue being implemented despite lacking any proof of making a difference in individuals’ lives. As the examples above demonstrate, the costs of this occurring are far from trivial. The resources and political capital spent on ineffective ideas cannot be spent on more impactful solutions. It is clear that the fight against common-sense ideas that do not work is not an easy one. However, given the importance of the issues at hand and the increasingly limited funds available to governments, it is one that is most certainly worth fighting.
Jeff Lambart is a master in public policy student at the John F. Kennedy School of Government at Harvard University. With a background in program evaluation, he is passionate about using data and evidence to help governments deliver more impactful social services.
Edited by: Samantha Batel
Photo by: Chris Liverani
[1] Lindsey A. MacFarlane et al., “Influence of Baseline Magnetic Resonance Imaging Features on Outcome of Arthroscopic Meniscectomy and Physical Therapy Treatment of Meniscal Tears in Osteoarthritis,” American Journal of Sports Medicine 47, no. 3 (2019): 612–9.
[2] Raine Sihvonen et al., “Arthroscopic Partial Meniscectomy versus Sham Surgery for a Degenerative Meniscal Tear,” New England Journal of Medicine 369, no. 26 (2013): 2515–24.
[3] Anthony Downs, “The Law of Peak-Hour Expressway Congestion,” Traffic Quarterly 16, no. 3 (1962): 393–409.
[4] Gilles Duranton and Matthew A. Turner, “The Fundamental Law of Road Congestion: Evidence from US Cities,” American Economic Review 101, no. 6 (October 2011): 2616–52.
[5] “Katy Freeway lands $1 billion to fund expansion,” Houston Business Journal, 14 March 2003, https://www.bizjournals.com/houston/stories/2003/03/10/daily54.html.
[6] Joe Cortright, “Reducing congestion: Katy didn’t,” CityCommentary, 16 December 2015, http://cityobservatory.org/reducing-congestion-katy-didnt/.
[7] Joseph Stromberg, “The ‘fundamental rule’ of traffic: building new roads just makes people drive more,” Vox, last updated 18 May 2015, https://www.vox.com/2014/10/23/6994159/traffic-roads-induced-demand.
[8] Alisha Coleman-Jensen et al., Household Food Security in the United States in 2016 (report no. 237, Economic Research Service, US Department of Agriculture, September 2017), 6 [PDF file].
[9] Dong D. Wang et al., “Trends in Dietary Quality Among Adults in the United States, 1999 Through 2010,” JAMA Internal Medicine 174, no. 10 (2014): 1587–95.
[10] “Food deserts may not matter that much,” The Economist, 8 March 2018, https://www.economist.com/united-states/2018/03/08/food-deserts-may-not-matter-that-much.
[11] Hunt Allcott et al., “Food Deserts and the Causes of Nutritional Inequality” (working paper no. 24094, National Bureau of Economic Research, December 2017).
[12] Allcott et al., “Food Deserts and the Causes of Nutritional Inequality.”
[13] Jane Furey, Jacob Alex Klerman, and Todd Grindal, “Retailer Proximity and Nutrition Program Redemptions: Evidence From the Summer Electronic Benefit Transfer For Children Program,” Applied Economic Perspectives and Policy 41, no. 1 (2019): 71–95.
[14] Allcott et al., “Food Deserts and the Causes of Nutritional Inequality.”
[15] Allcott et al., “Food Deserts and the Causes of Nutritional Inequality.”
[16] “Food Desert | Gateway to Health Communication & Social Marketing Practice,” Centers for Disease Control and Prevention, last updated 15 September 2017, https://www.cdc.gov/healthcommunication/toolstemplates/entertainmented/tips/FoodDesert.html.
[17] Kimberly Leonard, “Hospitals to post prices online under new Trump administration rule,” Washington Examiner, 3 August 2018, https://www.washingtonexaminer.com/policy/healthcare/hospitals-to-post-prices-online-under-new-trump-administration-rule.
[18] Robert King, “Senators want more transparency on healthcare prices,” Washington Examiner, 1 March 2018, https://www.washingtonexaminer.com/senators-want-more-transparency-on-healthcare-prices.
[19] David Cutler and Leemore Dafny, “Designing Transparency Systems for Medical Care Prices,” New England Journal of Medicine 364, no. 10 (2011), 894–5.
[20] Cutler and Dafny, “Designing Transparency Systems for Medical Care Prices.”
[21] Cutler and Dafny, “Designing Transparency Systems for Medical Care Prices.”