A new report looking at the inaccuracies of Mainstreet Research’s polling methods during the 2017 Calgary municipal election is pointing fingers at a number of flaws with the poll’s results, what led to them and what happened in the aftermath.
As the mayoral candidates were in the final days of their election campaigns, a Mainstreet Research poll, commissioned by Postmedia — which publishes both the Calgary Herald and the Calgary Sun daily newspapers — changed the narrative, putting candidate Bill Smith in a 13-point lead over incumbent Naheed Nenshi.
“Barring some sort of miracle, he’ll be the mayor on Oct. 16,” Mainstreet president and CEO Quito Maggi said on Oct. 13, 2017.
What happened three days later stunned many Calgarians as Nenshi held tight to the mayor’s seat, winning with a 7.6-point lead.
Days later, Mainstreet Research apologized for what Maggi called a “catastrophic polling failure” that some argued interfered with the democratic process.
Months later, the Marketing Research Intelligence Association launched a review into the polling results. The association said the results were conflicting and underperforming, and the review would focus on the degree of inaccuracy, the reasons for the inaccuracies and whether the results were adequately given to the voting public.
The review was conducted by a panel of three independent academics — Dr. Christopher Adams with the University of Manitoba, professor Paul Adams at Carleton University and Dr. David Zussman with the University of Victoria — and it takes concern with a number of issues that arose as a result of the poll results.
WATCH: A Mainstreet/Postmedia poll finds Bill Smith well positioned to be Calgary’s next mayor with a 13-point lead over Naheed Nenshi in decided and leaning voters. Doug Vaessen has details.
It found the Mainstreet poll “received the greatest media attention during the campaign because of their number, their startling results and their association with the two Calgary dailies significantly affected the course of the campaign.
“They threw Nenshi’s campaign on the defensive, gave impetus to Smith’s campaign, and possibly doomed the prospects of another candidate, André Chabot, who Mainstreet’s poll suggested was not a close contender,” the report said.
In an emailed statement, Maggi said he was “thankful to the members of the panel for their work.”
“I found the report thorough and largely agree with the findings,” he said. “This reaffirms what our review found, numerous factors led to the errors of all the polls.”
The more than 70-page report also takes issue with Mainstreet’s “overconfidence” in its results when critics started voicing concerns.
“Mainstreet executives responded with unshakeable confidence in their results and attacked their critics, often in personal terms, at one point suggesting there would be ‘payback’ after the election results were known,” the panel said.
“The panel found that Mainstreet’s overconfidence and its contentious style of public debate significantly contributed to the embarrassment to the industry when its results were proven to be radically mistaken.”
The report said that confidence contrasted with the firm’s internal concerns over the poll results eventually led it to change its methodology — another point of contention within the polling.
Instead of using random-digit dialing, Mainstreet used phone numbers pulled from a “directory,” which pollster Janet Brown said meant the survey started out with a “flawed sample.”
The experts said the directory was under-representative of young voters who eventually made up a large portion of the unexpectedly high voter turnout. Mainstreet failed to provide more information on what that directory was or where it came from.
One of the biggest critics of the 2017 poll was Mount Royal University political scientist Duane Bratt.
Bratt — who was one of the people interviewed by the panel — called the 2017 poll “a damaging blow to the entire polling industry.
“All can agree that the polling during the 2017 Calgary election was bad,” he told Global News. “Moreover, it had a direct impact on the result — although we cannot properly ascertain what that impact was besides destroying Andre Chabot’s candidacy.”
Polling method change
After the first of the three polls commissioned by Postmedia received criticism, Mainstreet changed its polling method to sample municipal wards, which the panel said restricted its cellphone sample, making the under-representation of younger voters worse.
The panel also found that while two other polls — one by the LRT on the Green Foundation and one by the Canadian Municipal Election Study — “tempered what turned out to be the erroneous narrative created by the Mainstreet polls,” neither of the three met the MRIA’s standards for disclosure.
In addition to Mainstreet itself, the report also placed some of the blame for the media and public confusion on Postmedia, which the panel argued “was not critical enough in its reporting of polls for which it was partially responsible.” Postmedia did not participate in the review.
WATCH: It’s being called the perfect storm. One pollster suggests it’s why they were so wrong in predicting the outcome of the 2017 municipal election. But as Jill Croteau reports, apologies and regret may be too little too late.
Academics told the panel they were interviewed by Postmedia journalists, but said their concerns on the methodological concerns were not then reported to the public.
Brown was one of those interviewed and told Global News she was disappointed the interviewers didn’t have an opportunity to ask the media organization some critical questions, like: “Why didn’t you report on the first calls questioning methodology?”
“It felt like what happened during the Calgary election had serious implications during the democratic process and it was important enough that I think all players should have been able to demonstrate some transparency,” Brown said.
The panel also found Postmedia didn’t share its own concerns with the polls and the way Mainstreet was adjusting its methodology.
As well, “the panel found that many media outlets, including the Postmedia newspapers, did report on the dramatic discrepancies between the Mainstreet polls and others published in the campaign, but that the coverage was not technically sophisticated and would not have left readers fully equipped to evaluate the polls.”
The report also said the MRIA could play a bigger role in developing a framework for higher standards for agencies to follow when publishing polls.
Recommendations, but no accountability
The report outlined several recommendations coming out of the polling debacle, including:
- A continuous updating of industry standards, particularly around disclosure
- A standardized disclosure checklist that polling firms would publish contemporaneously with their polls and which would provide a tool of accountability to the media, public and the MRIA
- An annual report on the state of the polling industry that would evaluate the adherence to industry standards
- The development through these tools of a capacity for the MRIA to react in real time to public or media concerns about polling methodology
- Enhanced services to the media and public, in terms of understanding polls and providing accountability
- The development of codes of conduct by polling firms for their interactions with the media and on social media
It also offered a series of recommendations to media organizations, including that they fully disclose the nature of commercial and financial relationships when getting polling data, applying normal journalistic context and skepticism of polls and developing more sophisticated techniques for reporting on polls.
The panel also recommended Mainstreet and the MRIA find a neutral academic auditor who could evaluate the firm’s practices with the aim of returning to the MRIA.
In what Brown called a “bittersweet” turn of events though, there will be no governing body in Canada to hold Mainstreet and other polling firms accountable for the related recommendations — the MRIA suddenly shut down on July 31 due to financial difficulties.
Bown said she hopes that even if no other organization emerges to act as a pollster watchdog in Calgary, the media sees what happened during the Calgary election as “a cautionary tale.”
Bratt echoed the sentiments.
“There needs to be a new public opinion polling organization to establish standards, codes of conduct and educate media and the public about what constitutes a proper poll,” he said.
Maggi said he was pleased the findings were released despite the MRIA’s closure.
“As we said, it was an honest error, not a conspiracy.”
Global News’ request for comment from Postmedia was not returned.
BELOW: See the complete Mainstreet/Postmedia poll
Mainstreet – Calgary Election 2017 Final by Mainstreet on Scribd
— With files from The Canadian Press