Toronto Life’s neighbourhood rankings: Why show-your-work matters

Toronto Life’s August cover story rated Toronto’s 140 City-defined neighbourhoods by a series of criteria – such as crime, access to transit and education.

“The results are bound to be controversial,” the story said, and it’s true that there was a certain amount of buzz around it in mid-August.

A number of readers, though, found themselves bewildered by how Toronto Life could have arrived at the conclusions it did: Lively comments can be found in TL’s own story, the move smartly real estate blog, the Urban Toronto discussion board, David Fleming’s real estate blog, another real estate blog, The Grid and Twitter.

From my perspective as someone who has been doing data journalism in and about Toronto for the last five years or so, it shows the value of being reasonably transparent with readers about the data you’re using and the choices you made about analyzing and displaying it. Readers can – and do – disagree with the judgment calls you made, but at least they know what they are.

The package’s reception does reinforce, in this case negatively, the show-your-work approach to data journalism. (The alternative is trying to market yourself as a kind of journalistic Delphic oracle, asking readers to take your utterances at face value while refusing to explain them.)

In this case, transparency would have helped readers understand some things in the data that seem, at first glance, counter-intuitive.

“Just eyeballing things,” Ed Keenan said gently, “it’s hard to see what their methodology for some criteria would be.”

A number of people on comment threads and Twitter have been confused by Toronto Life’s approach to transit rankings.

I don’t understand how a ranking of 140 neighbourhoods by transit quality could result in:

11. Greenwood-Coxwell
13. Danforth Villiage-East York
18. Blake-Jones
53. North Riverdale

when all four neighbourhoods are on the Danforth subway line, with North Riverdale closest to downtown, and also served by the King car on Broadview. There may be an underlying logic, but it’s hard to see what it would be.

Schools are also a criterion. Once again, it’s not clear how the rankings are constructed (I would use EQAO scores for public and separate neighbourhood-based elementary schools, and use a point-in-polygon exercise to place them in neighbourhoods. Here’s what a point-based version of that map looks like.)

To take a random example from the magazine’s data, it’s not clear why Toronto Life would rank Kensington-Chinatown #7 of 140 (or Dovercourt-Wallace-Emerson-Junction at #21) – the testing data doesn’t appear to support it. (A school performance map based on catchment areas looks like this.)

Neighbourhood crime mapping is really hard to do well, and there’s a number of approaches to it. The best exercise in local crime mapping I’ve seen in Toronto was done a couple of years ago by a StatsCan researcher who had access to raw police data of 2006 Toronto crime reports under a confidentiality agreement. He mapped each individual crime to a census tract, then tried to establish a ratio of crimes to people potentially vulnerable to crime, where IIRC residents counted as a person and people working in the tract counted as a fraction of a person. This quickly shows that, in relation to the people working there, the downtown core has a fairly low crime rate.

I did a two-part Map of the Week post on it, looking at property crime and violent crime. The maps aren’t online any more (I still have the data, and should really post them somewhere, maybe here) but the screenshots and commentary are still around.

Two observations:

– There is, or was six years ago, a census tract covering part of South Riverdale (5350018, DVP/Eastern/Carlaw/Queen) that made the city’s top 20 for violent crime. But South Riverdale, which covers a larger area, doesn’t pop out in a way that would justify being the worst neighbourhood out of 140, as far as I can tell. (For what it’s worth, my own subjective perception is that there’s less crime, generally, in that area now than there was in the early 2000s.)

– There is an anomaly in the 2006 violent crime data in which the tract northeast of Gerrard and the Don River shows up as #18 of 531 for violent crime. This puzzled me until I realized that all the documented violence in the Don Jail was included. I’d be interested to know how that’s accounted for in the Toronto Life data.

Are Toronto Life’s crime scores based on a ratio, or raw numbers? (Raw numbers would account for Waterfront-Communities-The Island’s score, thanks to small population. But that doesn’t shed useful light on your actual vulnerability to crime if you live in a condo at Bay and Lakeshore.) What is the weighting of violent crime to property crime?

The project also measures diversity. I would very much like to see the data under which Blake-Jones scores lowest in the city, at #140, Regent Park scores #124 and Moss Park is #118. I’m not saying it’s wrong, just that it’s puzzling and that I’d like to know what is being measured and how. (In the 2011 census, tract 5350072.01, a box of Danforth/Pape/Jones/railway, about half the residents were native English speakers; in 2006, Blake-Jones had a higher percentage of immigrants – though not recent immigrants – and a much higher number of people who don’t speak either official language than the city as a whole.)

The whole exercise is more opaque than it should be. It would be a much stronger package if it was more transparent