Who filled out the National Household Survey? (and why did Statscan cut its census standards in half?)
Oct. 3: Updated with a response from Statistics Canada
Canadians were less likely to fill out Statistics Canada’s National Household Survey if they were very poor. Or very rich. Or lived in less dense neighbourhoods.
Global News analyzed results of the voluntary survey that replaced the country’s most reliable compendium of population data and found it underrepresents people on both income extremes, as well as those most likely to need government services that rely on this data.
The voluntary National Household Survey has come under fire for being less accurate than its predecessor, the mandatory long-form census.
Response rates for the voluntary survey were sharply lower than 2006: over 26% of surveys weren’t returned, compared to 6.5% with the compulsory long-form census five years ago.
Because of that, Statistics Canada sharply lowered its standards: The point at which a neighbourhood’s results are suppressed due to a high non-response rate is 50% this year, compared to a 25% cutoff in 2006.
That means that if this year’s survey were held to the 2006 census’s standards, more than two thirds of the country’s 5,000-odd tracted neighbourhoods – 67 per cent – would have been excluded because the data’s just too unreliable.
“The NHS, being a voluntary survey, had larger non-response rates than would be the case for a mandatory survey,” StatCan spokesperson Edelweiss D’Andrea said in an email. “For this reason, Statistics Canada needed to set a threshold which would reflect this change in methodology.”
Peterborough, Ont. residents were least likely to complete the survey - almost 40% of all surveys mailed to them weren't returned to Statistics Canada. Saint John, N.B. and Abbotsford-Mission, Lethbridge, and Sault Ste. Marie are close behind, all reported a non-response rate around 35%.
In seven communities, more than 90% of census tracts would have failed the 2006 standards: Peterborough, St. John’s, Thunder Bay, St.-Catharines-Niagara, Kelowna, and Saint John, NB. In another 12, between 80% and 90% would have failed.
In Toronto, census tracts with higher numbers of non-English-speakers were more likely to fill out the survey.
The federal Conservatives scrapped the long-form census in 2010, replacing it with a voluntary National Household Survey. Industry Minister James Moore will decide how to proceed for the 2016 census cycle.
Bad data from this survey could have real implications for planning and service allocation. If you use city roads and plumbing, energy infrastructure, schools, immigrant settlement services, public housing or a multitude of other services, you rely on information from Statistics Canada’s census-turned-survey.
“What’s affected the most is our ability to allocate resources well,” Centre for Policy Alternatives senior economist Armine Yalnizyan said in an interview earlier this month. “We’ll be making a lot more mistakes. We’ll be wasting a lot of money.”
“The data from the NHS are very useful,” responds Statistics Canada’s Gabrielle Beaudoin. “In terms of overall population, NHS estimates are published for 97% of the population. At national, provincial and census metropolitan area levels, the accuracy of the data is good. There is more volatility for some smaller areas such as census tracts. In those instances, users are encouraged to use data at higher geographic levels.
“The example cited in this story is one case where a group is underrepresented in the respondents to the survey in some areas. There is a risk of bias in the estimates when respondents differ from non-respondents. While our procedures mitigate the risk of bias, this risk is not eliminated. This applies to all voluntary surveys.”
“As always, we continue to look at ways to improve the next census cycle while protecting Canadians’ privacy,” said Sebastien Gariepy, a spokesperson in Industry Minister James Moore’s office.
Ontario’s still deciding whether to use household survey data (among other things, the Education Ministry wanted data for “several school-aged variables”). Toronto won’t use the data for trendlines at all.
A Vancouver spokesperson told Global News the city has “concerns” the quality of the data “may affect the City’s ability to identify changing socio-economic trends and target services where they are needed most.”
And further Global News analysis of survey respondents shows those least likely to fill it out are the people on either extreme of the income spectrum: The poorest and richest Canadians showed the lowest response rates.
This effect was predicted by critics of the survey, which replaced the compulsory long-form census last used in 2006.
Compliance was also lower at the top of the income scale: Response rates fell for tracts with a median household income of $130,000 and higher.
Almost all Toronto-area census tracts pass muster if you hold them to Statscan’s 2011 standards. But under big chunks of North York, west-end Toronto and areas around Pickering, Durham and Peel had response rates low enough to exclude them under 2006 standards.
Under the old standards, data for much of Edmonton, Lethbridge and Medicine Hat – as well as Airdrie and Bragg Creek – would’ve been too unreliable to use.
Results in Langley, Aldergrove and Mission were deemed good enough for Statscan’s new standards – but they wouldn’t have been five years ago.