The Bizarre Quest to Discredit America’s Most Important Nutrition Survey

Studies that use self-reported data have obvious flaws—but they’re still the most feasible way to gather information.
(Photo: Wladimir Bulgar/Getty Images)
Jun 29, 2015· 4 MIN READ
Dan Nosowitz is a freelance writer based in Brooklyn. He has written for Popular Science, The Awl, BuzzFeed, Modern Farmer, Gawker, Fast Company, and elsewhere.

In 2003, California attorney Stephen Joseph ignited a media firestorm when he sued Kraft Foods and demanded an immediate injunction against the manufacturing of Oreos. Why sue milk’s favorite cookie? Heart-clogging trans fats—and lots of them. Just one day after the suit was filed, Kraft pledged to remove all trans fats from its products.

Twelve years of consistent anti-trans-fat legislation and voluntary corporate removals followed, culminating in an outright ban in June 2015. According to the FDA, the move away from trans fats will prevent 20,000 heart attacks and 7,000 deaths every year.

Even though the idea of trans fats as nutritional demons didn’t enter the national mind-set until the Oreo suit in 2003, the National Health and Nutrition Examination Survey, commonly known as NHANES, had been measuring trans-fatty-acid levels in participants’ blood since 1999. It was only through the numbers attained by this national-level survey—via both self-reporting and physiological testing—that other researchers were able to study the full effects of trans fats and, ultimately, save lives.

Despite the impressive pedigree, NHANES has its critics, and chief among them is Edward Archer of the Nutritional Obesity Research Center at the University of Alabama at Birmingham. Archer wrote a paper, published in the Mayo Clinic Proceedings, further explaining his theory about self-reporting in dietary surveys. In a nutshell: He thinks NHANES is garbage.

“Memory does not work like a video recording,” he said during a 90-minute phone conversation discussing NHANES data, in which he used the word “fraud” or “fraudulent” more than 15 times. “The idea that memory can provide accurate and precise reproductions of past behavior is demonstratably false.”

Archer said the major problem with relying on human memory isn’t that we know it’s wrong but that we don’t know how. Are people underestimating their calorie intake to seem healthier? Overestimating it in an attempt to be realistic? Just getting things wrong? “If someone reports eating an apple, they may be reporting it because they usually have an apple, but in reality that day they had an orange,” he said. “So even the person responding doesn’t necessarily know the accuracy of their memory.”

Not only do NHANES survey methods encourage inaccurate results, Archer said, but he went so far as to call them “pseudoscientific and inadmissible in scientific research.”

Others in the scientific community have problems with NHANES as well. A 2004 study tried to use NHANES data to glean more information about the relationship between low blood levels and neuropsychological development in children. It concluded that the NHANES data was not only of no use in analyzing this relationship but not much good in general, citing “serious shortcomings” including “missing data, odd distributions of scores, and potential inaccuracies in the data collection itself.”

In a phone conversation, Kathryn S. Porter, the director of NHANES since 2013, was much more polite about Archer than Archer was about NHANES. Porter stressed that the 24-hour survey is only one part of NHANES; the data from inarguable tests such as blood pressure tests and urine samples have given the country immense insight into chronic diseases such as diabetes and heart disease and infectious diseases such as hepatitis B and HPV.

So, Why Should You Care? NHANES found that lead levels in the American populace were much too high, leading the EPA to begin phasing out leaded gas in 1975, with an all-out ban in 1996. As leaded gas was being phased out, NHANES data showed a marked improvement in lead levels. That’s the kind of impact this study has: It has effected major change at the pump, America’s national altar if ever there was one. Such national-level surveys are the best way to find out how our bodies react to the environmental and dietary stimuli they come in contact with.

To discredit NHANES is a big deal, so it’s no surprise that criticism of Archer’s work popped up almost immediately. Brenda Davies and Paul Estabrooks wrote an editorial responding to Archer’s paper, pointing out that the 24-hour surveys have been improved through the use of new data collection methods such as the Automated Multiple-Pass Method, which has been shown to be pretty accurate. They also note that Archer has participated in studies that rely on human memory in pretty much the same way as NHANES does. (This is how scientists perpetrate sick owns on each other.)

Davies and Estabrooks also noted that the alternatives are not plausible: You can’t assign someone to monitor a subject 24 hours a day to make sure they’re accurately reporting their habits (at least not until Google makes a nutrition drone that fires Nerf arrows at you for drinking non-diet soda), and forcing people to, say, photograph their meals may alter their food choices.

More damning of Archer’s study is that not only is he not saying anything new, but he’s not even doing it right. That, at least, is the conclusion drawn by the authors of a long response to Archer’s 2013 study that appeared in the journal Advances in Nutrition. “To us it was like nothing new! We have known that energy intake varies from day to day,” said Namanjeet Ahluwalia of NHANES. “Of course self-reported data has bias—everybody knows that.”

Ahluwalia and the authors of the 2013 response noted that, though there is bias, there are decades of statistical work to draw on in adjusting for it, and that the data is extremely useful regardless. The response is sort of harsh for a scientific paper, saying Archer has an “insufficient grasp of the methods” involved in surveying and analyzing the data from the surveys, as well as “large, unsubstantiated jumps to policy implications.” Both Porter and Ahluwalia readily acknowledge that NHANES isn’t perfect; they’re just doing the best they can given their budget and limitations.

What’s more, Archer doesn’t include any alternatives in his paper because he doesn’t want to improve NHANES; his theories about obesity are primarily focused on exercise. Archer has repeatedly stated, against virtually all evidence, that “the American diet is no longer a risk factor for disease.”

In contrast to the vast majority of health and nutrition scientists, Archer thinks that diet is not a major influence on obesity—or at least that exercise is a much bigger one. Not just your own exercise; he has a theory about nongenetic evolution in which inactive mothers produce obese children. (The work produced a minor firestorm thanks to errant quotes stating that women should perform more housework to burn calories so as not to make their children obese.) His theory is not widely accepted.

Archer’s efforts to discredit NHANES are nothing more than a stepping stone for promoting his own theories. “Do I have an agenda? Yes, without a doubt,” he said. “I want to change the world. I want to improve the world, and I think as long as we keep focusing on nutrition, we’re not focusing on everything else.”