In October 2022, JAMA Network Open published a study exploring the relationship between the temporary expansion of the child tax credit (CTC) and food insufficiency. The study concluded that the end of the monthly CTC payments in December 2021 led to an increase in food insufficiency among households with children. The Los Angeles Times Editorial Board cited the study in their support of Congress passing a permanent expansion during the lame-duck session. However, instead of revealing something about the CTC and food insufficiency, the study offers another example of bad science misinforming an important policy debate.      

The problems with the study are abundant.

First, the authors used data from the US Census Bureau’s Household Pulse survey, which has regularly surveyed a large sample of US households on a variety of topics since the start of the pandemic. However, the survey suffers from several shortcomings that directly affect the study’s findings.

At each wave, the Census Bureau asks one individual from a sample of households to complete the survey. However, many people refuse the request resulting in a low response rate. According to a report assessing the Pulse survey, US Census Bureau researchers admit “response patterns differ across demographic domains that are correlated with key estimates, which could result in biased estimates.” The Census Bureau further warned: “There is no way of knowing what the HPS estimates would be if all sampled cases responded to the survey.” This alone severely biases any policy analysis using Pulse survey data.

Further, even among those who responded to the survey, some opted not to answer the food insufficiency questions, which also differed by demographics and across survey waves. This further biased the food insufficiency estimates, making the study’s entire approach highly questionable.

But the data source is only one of many flaws. The authors were trying to assess the impact of the CTC monthly payments on food insufficiency rates (defined as often or sometimes not having enough to eat) by comparing trends for households with children to households without children. The authors posited that if food insufficiency for households with children diverged from households without children once the CTC payments ended, they could attribute the change to the CTC payments.

The first problem is that in the absence of a controlled experiment, this sort of analysis requires very strong assumptions—assumptions that in this case were far outside the norm. Specifically, the authors had to assume that food insufficiency trends between the two groups—households with and without children—were identical prior to the CTC payments ending. They justify this assumption by pointing to a few data points (August 2021–December 2021), but even the naked eye can see from their graphs that the food insufficiency patterns were not the same.

Even if the trends were the same, the researchers wrongly attributed the divergence in food insufficiency between the two groups in early 2022 to the end of the CTC payments, neglecting to consider a variety of other factors. Although the authors controlled for a few things that could also be responsible for the divergence (for example, age, gender, race, unemployment insurance receipt), they still missed two of the most important factors: school lunch and inflation.

Unsurprisingly, food insufficiency among households with children increased compared to those without children just as schools were entering summer break in 2022—the same time school lunches end. Moreover, the divergence started before the CTC payments ended, seemingly coinciding with increases in food prices, which plausibly might affect families more than childless adults. The authors failed to control for either factor in their models and therefore, mistakenly concluded that the divergence in food insufficiency was due to the end of the CTC payments.

The figure below recreates the unadjusted food insufficiency rates from the study and extends it through November 2022, the most recent month of Pulse data. The new data further undermines the study’s conclusions. One glaring problem—if the extra money from the CTC payments at the end of 2021 kept food insufficiency low, why did the lump sum payments from the CTC and EITC in March and April of 2022 not have the same effect? This is something the authors do not consider.

Additionally, the most recent months of unadjusted data—September through November 2022—show that the trends reconverge, despite the fact that there were no changes to CTC policy during this time. Unsurprisingly, the reconvergence coincides with the start of the school year, just as low-income children begin receiving free or reduced-price school lunch.

Policy researchers have recently made a habit of conducting analyses using questionable data and methods, and failing to account for the multitude of factors that could influence outcomes. When it comes to important policy debates that effect America’s most disadvantaged groups, we must hold researchers and those who publish research to a higher standard.

The post Bad Science Continues to Plague Debate over CTC Expansion appeared first on American Enterprise Institute – AEI.