Renewed Controversy About Election Polls

by Hubertus Hofkirchner -- Vienna, 30 Sep 2022

Once again, the media and pollsters in Austria are facing harsh criticism for publishing wildly inaccurate forecasts on the occasion of the 2022 state elections in Tyrol. For example, ÖVP (Austria’s conservative “People’s Party”) was predicted to get 28 percent of votes (ORF) but won the elections with a significant margin and 35 percent of votes. The pollsters’ mean absolute error (MAE) averaged an embarrassing 2.3 percentage points. “Some polls were methodical garbage”, explained Austrian media darling and political pundit Peter Filzmaier on the ZiB 2 evening news channel. “Ignore that crock”, recommended Christoph Hofinger in the comment section of the Standard online newspaper, although a “Poll of Polls” of his SORA Institute also barely scraped by the 2.0 percent threshold with an MAE of 1.9 percentage points.

Just an error or election manipulation?

The matter has democratic implications, critics suspect attempts of tactical voter manipulation. It is difficult to refute these accusations which evoke memories of Austria’s recent government crisis after chat protocols triggered a “fake poll” scandal. Moreover, election forecasts are actually the gold standard for assessing the real capabilities of research companies, by comparing their predictions to actual elections outcomes which are highly transparent, objective, and thoroughly audited. Large errors cast doubts on the accuracy of other, less verifiable surveys, such as market studies, innovation, product and marketing research, and public opinion research on controversial political topics.

“The media has the power to enforce better polls”, writes media consultant Peter Plaikner in the Salzburger Zeitung, while criticising the two Austrian market research associations, VdMI and VMÖ, for supposedly failing to self-regulate quality. Therefore, he writes, it is now the media’s responsibility to ensure quality polls or lose even more of its steadily crumbling credibility.

Not just a question of know-how

Obviously, the industry associations are actively dealing with the subject, moreover they possess the necessary methodological know-how for the task. That said, self-regulation is less a question of the “if”, and more of the “how” and - surprise - of money. Also, there are fundamental differences between the two associations. VdMI has 28 members, comprised mostly of traditional research companies, while VMÖ has more than 300 members, including independent and corporate market researchers as well as representatives of young, innovative companies.

VdMI has issued obligatory guidelines which supposedly ensure quality by prescribing procedural standards. According to these, polls need at least n=800 participants, and purely online interviews are not admissible, rather polls must be mixed-mode with telephone or face-to-face interviews. These guidelines are actively enforced, VdMI actually expelled Wolfgang Bachmayer’s OGM institute on grounds of online-only polls, even though their accuracy was superior to guideline-compliant polls.

Quality measurement: By accuracy or by guideline adherence?

This begs the question whether these guidelines need updating. It should be evident that the focus should lie on prediction accuracy, not on a minimum process standard for one single traditional research method. Methods are contingent on the state of technology, they can quickly become obsolete considering the increasing rate of technical progress. For example, 95% of Austrian households had online access in 2021 which was not the case ten years ago. The only true and neutral measure of forecast quality is accuracy, the lowest deviation from the objective result.

The fallacy of defining quality by compulsory mixed-mode fielding is easily demonstrated by the recent election forecasts for Tyrol (Table 1). The mentioned MAE of 2.29 for traditional surveys compared to 1.16 percentage points for the independent prediction market “Wahlfieber”, i.e. double the forecast accuracy, beating all traditional polls. This is not a singular success, as the platform shows similar MAEs for other elections in Germany or Switzerland.

This fact falsifies procedural criteria, because prediction markets are pure-play online, nor do they require n=800 participants, as about two dozen “superforecasters” distilled by their good track record from prior prediction markets are fully sufficient for validity. Prediction markets are based on Bayesian statistics, where recruiting is not aiming for - elusive - representativeness but for diversity of opinions.

Method innovation and progress

For the avoidance of a misunderstanding: frequentist polls certainly have their place but given the many new research methods and different company approaches, an obligatory adherence to one specific method’s process hinders methodic progress.

VMÖ is already discussing new standards to measure and ascertain the quality of election polling which do ensure the needed openness for innovation. Quality can and should be measured by the empirical accuracy of the forecasts, as it is appropriate for Karl Popper’s birth country. with an independent evaluation before, during and after elections. In a second step, these plans will be brought before industry stakeholders and refined. Financing must be considered, as an increase in quality will also incur costs. If all stakeholders can be brought aboard, there is a good chance that Austria will have a modern, objective poll quality control ready just in time for the 2024 parliamentary elections.

Explanation:
Mean absolute error of election forecasts for ÖVP, SPÖ, FPÖ, Grüne, FRITZ, NEOS, MFG, and Other.

• Back to the Management by Predictions Blog
• Follow this blog for more stories, new case studies and how-to's.