Polls
A few months ago, I ran into a friend of mine in Ottawa, a federal MP who told me that I should put the Calgary Municipal election where it belongs, in the past. I had just finished writing a post about our polling of the Ontario Election and had referenced what lessons we took from the Calgary Mayoral review to improve our results. That was sound advice. Since the Calgary Mayoral polling error, the first in our history, we’ve successfully polled numerous Canadian by-elections, the Alabama state election, the Ontario PC leadership and the Ontario election. Prior to Calgary, we had not had such a miss, although we’d been outside the margin of error on a few occasions.
With the recent demise of the MRIA and the release of the Calgary Mayoral polling report commissioned by MRIA, I feel compelled to reference Calgary once again, though reluctantly against my friend’s advice.
The demise of MRIA
I am not gravely concerned that the absence of the MRIA will have any real impact on the quality of Canadian polling. We are members of WAPOR and there are other North American and International standard organizations. These will remain and the Canadian standards at MRIA were directly adopted from these in 2015. It’s clear from the posts made on the LinkedIn industry discussion group that how MRIA acted in this dissolution was neither viewed as professional or transparent. An organization that is supposed to advocate for professionalism and transparency acting in this way is disappointing.
I am concerned for the individuals who were employed by MRIA and who served in voluntary roles. They deserved better and the former board should attempt to explain. Whatever organization comes from the ashes of MRIA will need to do a better job of making members feel valued and respected.
Other reactions from industry point to the fact that more education is needed about IVR. The panel that reviewed the Calgary polling had full purview to point to the IVR mode as a source of error, but did not. Like our own internal report, the frame design was the culprit in their educated opinion.
The Calgary Polling Review
As I said to a number of media, I was eager to have the independent panel report on the Calgary Mayoral election released and was pleased to meet with the panel members in Toronto this past spring to discuss their review of our work. The panel confirmed the exoneration of Mainstreet from any malicious intent that was the conclusion of two previous studies. I largely agree with the findings, despite some minor inaccuracies that bear pointing out below.
I want to be clear that none of this is a criticism of the panel members. The work they undertook was very complex and had many moving parts, and they relied largely on the witness interviews for the information.
1. “Payback”
On the eve of the election, one of my former staff did an interview that was unacceptable. He suggested to a radio outlet in an interview that he would “single out” critics after the election. This was characterized as “Mainstreet vows payback” by the media outlet that night. I did not see the headline until later that evening and called my staff and the outlet to ask the particulars. The headline was updated to reflect the fact that nobody at any time used the word “payback”, but the “payback” tag remains in the psyche of those who followed the Calgary election since that story was shared thousands of times on social media. Twice in the report, the panel repeats this later corrected headline.
2. CMES
The report states as follows,
“The CMES is a longstanding research program that systematically examines municipal elections in a number of Canadian cities”
The CMES website was registered in August of 2017, the Twitter account for CMES was also started in August 2017, two months before the Calgary election. The Canadian Election Study is a long standing research program started at UBC. To my knowledge they are not related. I later commended the methodology to the panel during our interview, but at the time, they appeared to be a new organization with no previous track record. I am not aware of them publishing any polls before or since the Calgary election. This is not a criticism of CMES or of their researchers. As the report points out, the study was methodologically sound, but the longer fielding made it imperfect for this sort of election.
3. Emphasis on The Mainstreet “Directory”
It appears some conclusions were affected by other interviewees and their allegations.
I have worked with data, specifically directory data, voters lists and other large data sets, for most of my career. In a previous life I was the Manager of Information Technology for the Liberal Party of Canada in Ontario and managed data integration. What I have found over 20 years in using and managing “directories” is that no single source is perfect. To compensate, we buy commercially available phone lists from a “number of sources” and that gets us a more perfect sample frame. This has been supplemented with RDD sample in the past, but was not used in Calgary. Since Calgary, we’ve moved to an exclusive RDD frame and only dial directories for testing purposes, or riding polls that require it.
The mystery that the panel raises with regards to the Mainstreet directory is speculation fueled by the continued drama in Calgary. There is no ambiguity, no mystery, the data we use is available to anyone. Our methodology statement describes the frame as best we can, “a number of sources” which we’ve updated to say “a number of commercially available sources”.
Nenshi
My biggest concern with the report however is not the conclusions or recommendations. These I understand and largely agree with. My biggest surprise was seeing that Mayor Nenshi and a number of his staff continue to insist that this was a deliberate and/or malicious attempt to unseat him according to the report.
The report states:
“Mayor Nenshi and his election team expressed frustration with Mainstreet’s role and continued to suggest, as they did during the campaign, that there might have been a cabal among Mainstreet, Postmedia, and elements of the Calgary business community aimed at unseating him.”
This was presumably from the interviews conducted in Calgary on April 17-18th of 2018. But, when we released our report in December of 2017, and I once again apologized for the polling error, and apologized directly to the Mayor, the media reported he had accepted my apology. At least one member of his campaign team familiar with Mainstreet did retract and apologize for suggesting malicious or deliberate intent.
Our report included the raw data from every poll conducted in Calgary during the fall of 2017, including polls that were not publicly released. This has never been done before, by any pollster. Mayor Nenshi and his advisors have had a chance to review the data, and they have now had a chance to review the independent panel’s report and conclusions. It’s time for the Mayor and his team to formally retract and apologize for those allegations.
Many things are said and done in the heat of a campaign. My mistake was engaging with members of team Nenshi directly on social media and “playing along” in that theatre, as I suggested to the panel. This escalation from criticism of our sample to outright conspiracy was disappointing, but perhaps politically necessary. Our polls have been criticized many times before. At this point, three reports have all concluded that those allegations have no basis in fact. Continuing to restate these allegations, a year after the election, shows a severe lack of judgement, and borders on an abuse of power. Making these allegations to the panel was their mistake. Let’s hope they can admit it and clear the air once and for all.
