Many of you might have woken up yesterday to some pretty surprising news. Local media outlets in the South Bend area reported on a new Harvard study that purports to show that our community is the third unhappiest in the United States. (For those curious, the researchers claim that New York City is the unhappiest place.)
This struck me as odd. While I am not the most well-traveled person — let’s face it, it’s expensive to travel — I have been to both coasts, across the border to Canada, and to several of the country’s largest cities including Atlanta, Chicago, Indianapolis, Kansas City, Los Angeles and San Francisco.
Is South Bend really the third unhappiest city in America?
I don’t think so. And to help prove it, I took a closer look at the study in question.
The Glaeser et al. study uses the CDC’s Behavioral Risk Factor Surveillance System (BRFSS) as their primary data source. The BFRSS is a data collection program that measures “behavioral risk factors” in the adult population. In the CDC’s own words, “The objective of the BRFSS is to collect uniform, state specific data on preventive health practices and risk behaviors that are linked to chronic diseases, injuries, and preventable infectious diseases in the adult population.”
The CDC offloads the task of data collection to individual states. State health departments may conduct the survey themselves, or they may contract it out to a third party. Consistency of reporting across 50 different states can potentially be an issue here but the CDC maintains that they attempt uniform collection methods.
Another potential issue emerges in the way in which the data is collected. All respondents are reached through telephone-based surveys. As the CDC noted in 2005, approximately 95 percent of households have telephone access. This can range from as low as 87 percent to as high as 98 percent in different states, and may be even lower in various cities.
The important point to be made from their collection method is this: Economically vulnerable populations are the most likely to be excluded from the study’s data. This poses a significant problem as this is exactly the population where one would most expect to see higher levels of unhappiness. The CDC does not supplement its data with alternative collection methods, such as door-to-door surveys.
It’s also important to point out that the purpose of the CDC’s data collection is different from the Glaeser study. The government is looking for overall health trends. They do not necessarily need accurate city-level data in small places like South Bend to make broad judgments about regional differences and needs in preventative health practices and risk behaviors.
More to the point, the study’s authors use small amounts of data not intended for their study design to make big judgments about communities. The authors use the sample of respondents in 177 separate Census-designated Metropolitan Statistical Area. For our area, this is the South Bend-Mishawaka MSA, which is just over 318,000 total people in the MSA.
Glaeser et al. note that they only use MSAs “with at least 200 respondents in all years of our sample”. They do not denote how large the sample size was in the communities listed as “ranked by happiness” in their appendix, but we can assume that it is on the lower end, since again, the CDC data is looking for broad trends across states and likely collected less data from smaller places.
If indeed the number of respondents was only 200 people that would mean that the margin of error is roughly plus or minus 7 percentage points. This is a large room for possible error. Using data sets from multiple years can increase the reliability of the data but we still have the problem of how the survey data is collected.
The bottom line is that the Harvard researchers who wrote this study violated a number of pretty basic methodological rules. The first is that they used the data in a way that it was not intended. BRFSS data is collected for government officials to make decisions on health problems across large areas. It’s common for researchers to do this but it requires a certain amount of caution.
Next, the CDC’s BRFSS is collected from 50 different state health departments (plus DC and territories), or a contractor, raising potential issues with consistency.
Then there is the issue of data collection, which is done solely through the telephone with a very small sample size. Low-income and minority households are the most likely to be excluded from a data set based on telephone surveys. How accurate will their data be when high poverty cities such as Gary and Detroit — both of which ranked higher in happiness based on this study — are disproportionately not being surveyed?
Not any single one of these issues would invalidate the study’s findings outright. Together, they certainly call it into question.
So my question to the study’s authors is this: why publish the city list in the appendix in the first place, undoubtedly knowing that these issues exist and that the national media would carelessly pick up on it without posing the same critical questions that I did? Do you care about the negative publicity that your research brings to communities that are trying to rebuild in a positive way after decades of economic stagnation? Or is any publicity good publicity for a Harvard scholar?
South Bend has made a lot of progress as a city since the de-industrialization of the 1960s and 1970s. We are a better, stronger community as a result of the trauma of trying to put the pieces of our once-shattered economy back together. There’s been a lot of momentum in that direction, whether it is a downtown that is in the process of revitalization, new housing developments for young professionals, or high-tech industry making its way into our city at Ignition Park. Yet these negative outside headlines grounded in questionable research methods make our job that much harder.
Instead of looking at numbers in a spreadsheet, I would encourage Harvard’s researchers to come visit the people of South Bend and Mishawaka for themselves. We’re a welcoming, easy-going crowd. Oh, and yes, we’re quite happy. 🙂