Poll survey firms on the dock

Like many others out there, my perception of the ongoing election campaign has been in no small way influenced by the pre-election surveys that are sprouting from everywhere and are now being fed us with increasing regularity. The danger of the campaign being turned into a horse race has happened. Every day, we open the daily papers to check who’s up, who’s down in the latest survey.

So I took more than routine interest in the presentation made by former Senator Francisco S. Tatad at the Club Filipino last Wednesday wherein he called to question the accuracy and integrity of the work of local pollsters, and warned of the manipulation of public opinion by political and commercial interests.

The core of his brief is that: (1) local pollsters have been using methodologies and techniques that are flawed and discredited, and had been discarded sometime ago by social research experts and professional pollsters in the US and advanced countries; (2) local pollsters have ignored the strict standards for professional and ethical practice of public opinion polling that elsewhere are regarded as sacred and inviolate by polling associations and reputable pollsters; and (3) in reporting the survey results, media organizations have been an unwitting and unsuspecting purveyor of dubious findings to the detriment of the election campaign and the public.

To support his indictment, Kit Tatad cited the writings of research experts and professional pollsters that detailed current survey research methodology and its evolution from survey methods and practices that proved “fatally flawed and inaccurate.” High among these discarded methodologies and practices are face-to-face interviewing and quota sampling which produce “contaminated data” according to top US experts. Significantly, these methods and techniques are being used by local survey firms like SWS, Pulse Asia, and the new survey groups to measure Philippine public opinion.

Finally, he underscored the quite remarkable fact that, for a highly sensitive service, local pollsters operate without restraints from either the law or professional standards set by private associations. Unlike market research, which has MORI, political pollsters are pretty much on their own. This, according to Tatad, has resulted in glaring excesses.

At the heart of public opinion polling is essentially an almost incredible proposition: that by getting the opinions and preferences of some 1,500 to 2,500 citizens, pollsters can divine accurately the opinions and preferences of our 94-million citizens or 48-million voters. It takes a leap of faith on our part to believe that there’s a science here somewhere that makes this possible, and that those who practice this in our midst are professional and responsible.

Public opinion polling is mainly a US invention, and it is in America where it is most heavily practiced and most advanced. The story of the craft has been a journey of ups and downs, marked by spectacular failures and then by steady improvement of methodology and professionalism in recent decades. Among the instances of pollsters going wrong are the erroneous forecast of Dewey’s victory over Truman in 1948 by top US polling firms like Gallup, and the more recent 1992 failure to read John Majors’ victory in the United Kingdom.

In grappling with the failures, public opinion polling has sought to rid itself of the methods and practices that tarnished its work and tried to be more scientific and rigorous. According to the literature cited by Tatad, this was the reason for discarding face-to-face interviewing and quota sampling. In the words of pollster Kenneth Warren, these methods were proven to be “fatally flawed and grossly inaccurate.”

Here then is the riddle: If these methods are now passé in the US and advanced countries, why then are they being used for measuring Philippine public opinion by Filipino pollsters? The answer is unsatisfactory. In developing countries and emerging democracies in Eastern Europe, public opinion polling is still in infancy. What is not acceptable for the US and the advanced world is deemed as good enough for less developed countries.

The justification is that quality polling is hard to do in these countries because of poor communication, poor transport systems and prohibitive costs in reaching respondents. So pollsters had to cut corners and come up with “creative solutions.” But this is slammed by critics like David Kennamer, professor of mass communications at Virginia Commonwealth University, who says that these solutions are “indefensible from a methodological perspective.”

Tatad says that there would be no controversy if polling firms, in presenting their survey results, took care to inform the public of their basic limitations and the danger of errors. But instead the local poll surveys were transmogrified into a virtual dictator in the run-up to the May elections. The media, the candidates and the public treated the results as gospel truth. And local pollsters basked in the glow, instead of warning everyone not to be carried away.

Restraint and modesty are the rule for reputable pollsters in the US because theirs is not a science. According to Kenneth Warren, in his book In Defense of Public Opinion Polling: “As hard as professional pollsters may try, it is practically impossible to uphold “by the book” approved polling techniques every step of the way. Contamination will inevitably creep in at each and every polling stage… Even the very best pollsters can only try to limit the extent of the contamination.” 

At a time when we in the private sector are badgering our government and private firms to adopt best practice in their respective realms as a means to make ourselves more competitive in the world, the possibility that our polling firms are cutting corners is most disturbing. It’s one more black eye to add to the many already taken by the country.

The other side of the coin

It’s possible that local survey firms, particularly SWS and Pulse Asia, have something to say in response to Kit Tatad. I welcome their rebuttals. A friend of mine in the US, who is a noted pollster, said the following:

“The reason they do door to door is not to cut corners – it’s to be accurate since many homes don’t have a land line. And they use this method all over the world. Just this year I used it in central Europe and South America – there were phone options but they were inferior so I opted for door to door. That’s also the method I used in two countries in Southeast Asia and it has always been accurate. In the US, Yankelovich has continued to use door to door for some projects because they felt it was superior for some applications like long interviews. I think in the last couple of years they may have stopped, but not because of quality but because of the safety of their interviewers.”

Meanwhile, let’s all treat the poll surveys floating around with a grain of salt. And let’s enjoy the fact that the contest is not over. Teodoro, Gordon, et al, have reason to persevere.