Household survey not worth the price: Public health body

Government agencies reliant on census information may not pay for its replacement, whose numbers are less reliable and more expensive.

“I’m not sure that would be a wise use of my budget,” Paul Fleiszer, manager of surveillance and epidemiology at Toronto Public Health, said in an email.

“In the past, our analysis of population health sample surveys would use the census as a gold standard to understand how well the results represented the broader population. We no longer have that gold standard to work with.”

The National Household Survey, whose results Statistics Canada published Wednesday morning, is meant to replace the long-form census, scrapped in 2010. More granular neighbourhood-level data, previously free on request, is now available at an as-yet undisclosed “cost-recovery” price.

But because the survey was voluntary, the only thing the numbers tell us for sure is who fills out surveys.

That in itself could be valuable, notes Carleton University economist Frances Woolley: Disproportionately high response rates among some groups could indicate civic-mindedness.

“What I think is interesting is to connect this to what we know about voluntarism and civic engagement,” she said.

But the people least likely to fill out a voluntary survey are the ones whose information the census needs most – small, specific populations, rural communities, people who tend to be isolated or vulnerable or less engaged with government entities.

Some discrepancies in the data are already apparent: Woolley found a 12.1 per cent drop in the number of people identifying as West Asian, accompanied by a precipitous increase (50.8 per cent) in the country’s Filipino population. Both changes seem unrealistically large for the five years following the 2006 census.

“It’s pretty much impossible to do anything with the published data,” Woolley said.

Statistics Canada itself has warned against relying too heavily on this survey’s results: “Caution must be exercised when NHS estimates are compared with counts produced from the 2006 Census long form, especially when the analysis involves small geographies,” the agency wrote in a chapter on data quality.

The Ontario government hasn’t yet decided whether to trust the household survey’s data, said Finance Ministry spokesman Scott Blodgett.

“Ontario is a major consumer of census data,” he wrote in an email. “We expressed our concerns to the federal government in 2010. … We still have the same concerns today about the quality and comparability of future data collected.”

But it’s “still too early to tell definitively” how the data quality has been affected, he added.

“We will continue to evaluate the data in the National Household Survey to determine how this may impact our programs and services.”

Toronto Public Health’s Fleiszer found Statscan’s caveats “concerning” – enough to question whether it’s worth relying on those numbers at all, or paying extra for the neighbourhood-level data the city would need. That’s a tough call: Those numbers influence decisions about what health services are needed where.

“We would not entertain collecting information on a scale like this,” Fleiszer said. “The cost would be prohibitive. That responsibility lies with Stats Can and the Federal Government. They need to do better.”

Many groups may have no choice: There isn’t an alternative collection of data this comprehensive, Woolley said. “Because it’s so big, it really allows you to drill down on specific groups in a way that no other survey does.”

One option would be to use federal administrative data – from tax returns, employment insurance, immigration information – to connect the dots and create a statistical portrait of Canadians similar to what the long-form census provided. Even then, replicating the long-form census’ cross-correlations could be tricky.

In the meantime, advocacy groups may have to rely on this data to back up their cause, even if they think it’s shaky, because it’s the best evidence they’ve got.

The impacts may not be immediately evident to the average Canadian. And they’ll probably affect people on the margins – those who would need government outreach or assistance – more than those in the middle.

“The thing about policy,” Woolley said, “is when policy is good, you don’t notice it.” If faulty numbers make for flawed decisions, she said, Canadians may notice.

Interactive: Who’s from where? Explore the roots of residents across Canada, according to the National Household Survey

Double click the maps to zoom; click and drag to move around. Hover over a box or circle for more information.