Public opinion polls are a necessary part of a functioning democracy. Why have so many been wrong lately, and what effect can it have? Is there a workable solution?
On the night of November 9th, 2016, pollsters and pundits alike were in stunned silence as Donald Trump handily beat Hillary Clinton for the presidency. This wasn’t supposed to be happening, Newsweek had already printed 125,000 copies of its commemorative Madame President edition. Those fortunate enough to have laid their hands on a copy could fetch nearly $10,000 in an online market, while others would later only be able to attract bids of 99 cents.
It was like a clap of thunder, out of nowhere and a complete surprise. None of the polls were indicating that this was about to happen. Afterwards it was determined that many Trump voters either didn’t say who they voted for, or said they voted for Clinton just to avoid the hassle and get back home. It was something nobody had anticipated would happen.
Polls and the democratic process.
Opinion polling plays a very key role in the democratic process, and serves two important functions within it. First, polls give candidates and politicians an idea as to where the voting public stands on the issues. A swift public backlash on a policy can lead to a quick abandonment of said policy, or the need to develop a strategy to make it more palatable to voters.
Second, polling data can be used to influence voter behaviour. If, for example, a poll showed that Hillary Clinton held a substantial lead over Donald Trump (and many did btw) then this might cause some of her voters to stay home, thinking that everybody else has it covered, it’ll only be one less vote for Hillary. It could also incite more people to come out to vote for Donald Trump in an attempt to overcome the deficit and win.
Some serious polling errors.
“Barring some sort of miracle, he’ll be the mayor on Oct. 16.”
Quito Maggi, president and CEO of Mainstreet Research
Early on the morning of Friday, October 13th 2017
Three days before the 2017 Calgary municipal election, a poll was released that showed mayoral candidate Bill Smith had a 13 point lead over the incumbent Naheed Nenshi. Although Smith had appeared to be gaining ground, the result was a surprise, at least to me it was. Don’t get me wrong, I was happy with the result, but it just seemed off somehow.
Two other polls released at about the same time were split, with one favoring Nenshi and the other Smith. Both campaigns would kick into high gear over the final week-end. The candidate polling third, Councillor Andre Chabot, was all but eliminated from the race.
On election night, the incumbent, Nenshi would reclaim the mayor’s chair by seven points over Smith after getting 51% of the vote to Smith’s 44%. Almost immediately, the poll results from Mainstreet would come under scrutiny, and a review was launched by the Marketing Research Intelligence Association (MRIA), the industry’s governing body. A panel of three independent academics would issue a scathing 70 page report in August 2018. Among its findings:
the Mainstreet poll “received the greatest media attention during the campaign because of their number, their startling results and their association with the two Calgary dailies significantly affected the course of the campaign.“They threw Nenshi’s campaign on the defensive, gave impetus to Smith’s campaign, and possibly doomed the prospects of another candidate, André Chabot, who Mainstreet’s poll suggested was not a close contender,”
“Mainstreet executives responded with unshakeable confidence in their results and attacked their critics, often in personal terms, at one point suggesting there would be ‘payback’ after the election results were known,”
The report said that confidence contrasted with the firm’s internal concerns over the poll results eventually led it to change its methodology — another point of contention within the polling
Instead of using random-digit dialing, Mainstreet used phone numbers pulled from a “directory,” which pollster Janet Brown said meant the survey started out with a “flawed sample.”
https://globalnews.ca/news/4375306/report-flawed-calgary-election-polling-mainstreet/
The experts said the directory was under-representative of young voters who eventually made up a large portion of the unexpectedly high voter turnout. Mainstreet failed to provide more information on what that directory was or where it came from.
A change of methodology was also a contributing factor to the final outcome, and the media also received some of the blame as the review found “some of the blame for the media and public confusion on Postmedia, which the panel argued “was not critical enough in its reporting of polls for which it was partially responsible.” Postmedia did not participate in the review.
I should note that it does appear as though Mainstreet has taken the appropriate steps to ensure that issues such as this doesn’t happen again. They were receptive to my inquiries and their polling data from the recent Alberta provincial election was in line with data collected by other firms.
Link to Alberta Provincial Election results report on Excel:
https://1drv.ms/x/s!AjGPeOn-BhVuhnRL-lyLITCHVET1
Academic Explanation
I contacted polling analyst Bryan Breguet to get his take on the recent Alberta provincial election. Polling data for this election is somewhat consistent when taking the margin of error into account and there appear to be no major irregularities as was seen in the Calgary municipal election.
For Bryan, perhaps the most important factor when it comes to polling accuracy is voter turnout. He explains that “..in elections where the turnout changed a lot tend to have more polling errors.” So any time you go from a high turnout election to a low turnout election or vice-versa, the less accurate the polling data will be.
In the 2015 election, turnout was a 22 year high of 58.25% In this election the turnout was just over 70%, much higher than four years ago. But, just as the last election was about the sweeping out of an old dynasty, this one was about a great desire in Alberta to right the wrong of four years ago. And so voters turned out in large numbers, setting a record for the number of votes cast in advance polls at nearly 700,000. There was little doubt who the winner would be.
Polling isn’t as simple as that however as there are a multitude of other factors that can have an effect on polling accuracy. The size of the sample of people polled, the mode of the survey, and perhaps most importantly, the volatility of the electorate, all play into the results of any poll. Human behaviour is something that can never be accurately predicted.
The Accuracy of Public Polls in Provincial Elections
David Coletto, Bryan Breguet
https://ojs.unbc.ca/index.php/cpsr/article/view/1198
Mandatory voting.
Let me lead into this part by making my views clear. I am a proponent of mandatory voting, and not just because I’m a poll geek and I think it would make polling more accurate. I also know the subject came up as part of a Parliamentary committee on electoral reform in 2016.
Submission to the House of Commons Special Committee on Electoral Reform, July 18, 2016
Click to access ThomasPaulG-e.pdf
That being said I also recognize the fact that I am in the minority here. Even all of the pollsters I asked weren’t crazy about mandatory voting either. There are some definite arguments to be made against it, including the the dichotomy of forcing people to do something in a free society. I absolutely agree with that argument and can see the irony in arguing for such a thing.
My argument for mandatory voting is based purely on emotion along with a sense of civic duty and patriotism, though these are things that are not in fashion in society at the moment, much to its detriment.
Yes, we live in a free society where we are able to choose what we think and how we feel, or at least that’s what is supposed to be happening. There are of course limits to these freedoms, but by and large compared to some countries around the world we have a great deal of freedom. For now at least.
What people fail to understand or have just plain forgotten, is that freedom is not free. The freedoms that we enjoy came at a price and that price was paid by the over 66,000 Canadians who gave their lives overseas and the hundreds of thousands whose lives were forever changed fighting for those freedoms. I’m quite certain it wouldn’t kill the majority of people to take a few minutes of their time to cast a ballot.
If you don’t know who to vote for or don’t particularly care for any of the candidates, spoil your ballot by writing none of the above on it. Just get involved in the process. In Australia, where there is mandatory voting, a consistently large percentage of the population turns out to vote, with fines of $80 AUD being levied upon those who don’t show up on voting day.
A source of funding for veterans.
I’m not sure how it works in Australia, but I think a fine of $50 would suffice and could be handled through the income tax system with 100% of the proceeds going towards veterans, on top of budgeted government spending for veterans. Seems reasonable to me though many will disagree. Let them.
A big thanks.
To the research companies who were gracious enough to answer my questions in a timely and eloquent manner, thank you so very much. For as much as polls are generally maligned these days, they play an essential part in the democratic process. Arguments can and have been made that a number of these companies and their polling data is biased. These arguments come from all sides of the political spectrum and the bias of the media reporting the poll results must also be taken into account.
So, for their contributions I would like to personally thank:
- Quito Maggi – President and CEO, Mainstreet Research
- Kyle Braid – Sr. VP Ipsos Public Affairs
- Dr. Lorne Bozinoff – President and CEO, Forum Research
- Ian Large – VP and Partner, Leger 360
- Bryan Breguet – Polling Analyst and guru, tooclosetocall.ca
A parting thought.
I know that what I’m about to say will prove controversial and piss plenty of people off (don’t care) but I’m of the firm belief that the voting age, rather than being lowered, should be raised to 24. I’ll be discussing this in a future post.
The Case for Mandatory Voting in Canada
Mac Harb
D.B.
The Final Word.
Having received some replies after I posted this, I am going to share with you the replies I got from the research companies in this story. I believe in allowing for replies or rebuttals and am not afraid to admit when I’m wrong. What’s important is that everyone has a chance to have their say.
One thing that I was reminded of took me back to my days in front line customer service, and that is the fact that nobody takes notice when you do things right. This is also true of pollsters. Remember that polling is part science and part art, and it’s not as simple as picking a number out of the phone book and calling someone.
As it also turns out I did find one person who agreed with me regarding compulsory voting, and that was Quito Maggi.
Quito Maggi
7:18 AM (10 hours ago)
to me
Hi Derek,
Thanks for taking the time to do this, overall I think it’s a good piece and making the case for mandatory voting in an effective way. I’m a fan of mandatory voting myself, where my family came from it has been the tradition for a long time (Argentina) despite having had challenges with democracy off and on over the 20th Century, it has been largely a stable democracy since the 1980s. They even have mandatory voting in their primary system.
A couple of small points here I would make.
1. Your piece references the unlikely Trump win as an example of significant polling error, but the National numbers were all within the margin of error for virtually all polls. The modellers and aggregators in the US in 2016 were the big failures in fact, they had between a 0 and 3% chance of Trump winning. They did not account for the large margins of victory in California and New York for Clinton being offset by smaller margins of victory in key swing states like Ohio, Pennsylvania, Wisconsin and others for Trump. In the end, the polls were almost perfectly reflective of popular vote, but did not reflect the electoral college. I was one of only a few people to publicly say Donald Trump would win, based on lesser known state polls in those swing states.
2. It’s important to note as well since the MRIA panel report references this and so does your story, that both these polls had Chabot significantly lower than our poll. The suggestion made by observers, and by the panel, that somehow the polls (our polls specifically) were responsible for his poor performance is not supported by the facts. The CMES/Forum poll which was an academic study, showed Chabot in low single digits long before our first poll was published. There are instances where polling affects strategic voting, but there was no evidence to suggest this was the case in Calgary, nor is it possible for a polling firm to know what those effects might be. As you correctly point out, a poll showing a large lead for one candidate might cause complacency among the leading candidates supporters, it might also cause supporters for the other candidate to get motivated, or to some degree both. Part of the reason why we moved most of our polling to paywalled content is to avoid these accusations of “influence”.
3. This last point is more about tone, I think overall your post was fair but makes the same mistake that media has repeated in terms of polling accuracy. I made this point in my last response as well and I was hoping that you might mention it more specifically, polls are correct far more often than incorrect. The fact is, polling errors like Calgary are very rare, prior to 2017, you have to go back to BC 2013 or Alberta 2012 to see similar errors. There is a mathematical certainty that polls will have error from time to time (19 times out of 20). But if you add up the polls done by the firms who polled Alberta in 2019, we are talking about thousands of polls that were correct between 2014 and 2019, with a small number that were incorrect, it’s 99% of the polls that get it right, but only 1% that get the headlines.. All pollsters will tell you, the one headline you never see is “Polls get it right”, it’s only sexy when we get it wrong. Maybe a small reference in your story to that fact would be appropriate given the balance you’ve already shown.
Thanks again and look forward to seeing your work going forward.
RE: [Contact] Media query
Inbox
x
Kyle Braid <Kyle.Braid@ipsos.com>
Wed, Apr 24, 1:28 PM (5 days ago)
to me
Hi Derek,
Happy to try to answer some questions. Obviously it was a miss for the entire industry, underestimating the strength of UCP vote and far overestimating NDP support.
It, however, is going to take some time for us to do an internal analysis of both the phone and online portions of our samples. Not a lot to say yet, except that this does not look like a problem caused by turnout. And I do note that we have seen this before in Alberta in 2008, when many (not all) polls underestimated the strength of Ed Stelmach and the PCs.
Kyle
Kyle Braid| SVP
Ipsos Public Affairs
1075 W Georgia St, 17th floor
Vancouver, BC V6E 3C9
Phone : 778.373.5130 | Mobile : 604.788.24
Results
Inbox
x
Derek Bodner
Bryan Breguet
Tue, Apr 23, 2:18 PM (6 days ago)
to me
Hi,
I’ve never been a fan of mandatory voting. I don’t see the point of forcing uninformed, uninterested people to vote. I have zero issue personally with a lot turnout as far as recognizing the results. I wish we had high turnout but, more importantly, I wish we had more people interested in politics.
As for the polls, I still believe this is an issue of selective turnout. UCP voters were more motivated. I guess many really, really didn’t like that their conservative province had a NDP government for 4 years lol
I wish Election Alberta would release a turnout by age like Election BC does.
Lorne Bozinoff
Tue, Apr 23, 9:27 AM (6 days ago)
to me
It’s true predicting who will vote when turnout is low can be problematic for polling. However I think there needs to be a better justification for mandatory voting.
I think though the turnout was pretty high.
The underestimation of the UCP may have been due to a last minute surge towards them.
Regards,
Lorne Bozinoff
The Advocate-Equalizer 
Not off the top of my head but I will let you know if I think of anything.
LikeLiked by 1 person