 |
 |
|||
 |
|
|||
| |
|
|||
|
|
|||
Statistically Speaking
On Election Polls: Part II 
by Dr. Romulo A. Virola 1
Secretary General, NSCB
On February 24, Pulse Asia Inc. released the results of its survey showing President Arroyo with 31.9 %, insignificantly ahead of Fernando Poe Jr. with 31.7%. Two days later, on February 26, SWS released its own results with FPJ significantly ahead with 37.5 % while GMA had 28.7 %. The two most-respected pollsters in the country “peddling” results with a difference as statistically significant as heaven and hell! Just as I nostradamused in Part I, two surveys showing conflicting results, so what to do?
And to add Avian flu to injury, the Ibon Foundation chirped with its own results!
So I was about to say, hey, you guys, what is the matter? Killing the goose that has laid many golden eggs? For what? Or maybe for who?
Then the latest SWS survey covering February 17-25 and released 3 March showed results much closer to those of the Pulse Asia survey for February 16-20! The SWS now has GMA with 31.8% in a statistical tie with FPJ’s 30.5%, just like Pulse Asia! Whew!
So what, pray thee, is happening? For sure, not brain gain!
According to Professor Robert M. Worcester, the founder of Modern Opinion Research International, opinions are “ripples on the surface of the public’s consciousness” and therefore shallow and change easily, in contrast to attitudes and values which lie deeper below the surface and are slow to change. Thus, results of opinion polls change over time and shifts in opinion have been recorded many times in many places. But that should not be the end of the SWS-Pulse Asia distraction, should it?
While election surveys are definitely not an infallible oracle; they are really not an obfuscating obtrusion either. So, let me try, statistically speaking, to dispassionately dissect the discomforting discrepancy in the results which understandably, have invited ridicule from agitated analysts. My evaluation shall focus on the two leading candidates, and because of the bird flu, please allow me not to talk about Ibon.
In quality measuring two diverging opinion polls, one should try to answer the following questions, at the minimum.
Were the surveys commissioned, a factor which may affect the integrity of the results?
Do they have different interview periods which may explain the difference?
What are the sample sizes, which have a bearing on their precision?
How are the relevant questions phrased and when during the interview were these questions asked, which can affect the quality of responses?
What is the extent of the undecided and nonresponses, which may cloud the overall picture?
How are the respondents distributed geographically, across socio-economic classes, by age, sex and other relevant classification factors that may bear on the results?
Based on available data, I cite the following facts and observations about the diverging surveys:
The SWS survey was commissioned by the FPJ group. Pulse Asia “undertakes surveys on its own without any party singularly commissioning the research effort”.
While other pollsters may compromise their professional integrity to get business, surely the illegitimate ones, who have none in the first place, given the stature that SWS has legitimately gained over the years, it seems very unlikely that SWS will do that, no matter if blood is thicker than water!
On the other hand, just because the Pulse survey was not commissioned does not make its results beyond suspect. I say though, perish the thought that Pulse Asia will permit itself to be tempted!
The diverging surveys were conducted at different points in time. When the interview periods of SWS and Pulse Asia overlapped in the latest SWS survey, the “statistical perfidy” in the words of a politician, vanished.
Incidentally, a longer survey period can reduce the accuracy of the “snapshot” being captured by a survey - an interviewee may change his/her mind between the first day and the last day of the survey period. Instead of a snapshot, we get a video-clip or a tearjerker of a movie! Extended polling periods should therefore be avoided. Recently, Gallup surveyed American opinion of the personal characteristics and qualities of Bush and Kerry over two days only, February 16-17, 2004 with a national sample of 1006 adults and margin of error of +/- 3 percentage points. And a couple of days ago, a survey released the statistical tie between Bush and Kerry! A case of the American voters aping the Pinoys? Cheers!?
In the SWS survey, the undecided was 6% ; for Pulse Asia it was 8.1%.
The undecided votes and /or nonresponses in a survey sometimes hide a dimension that bears on the final results. If this is the case, proportionately allocating the undecided votes or ignoring them will reflect a distorted picture of voter preference.
Studies show that people belonging to lower socio-economic status are more likely to respond to surveys than their higher strata counterparts. This conclusion has some empirical basis in the Philippines – I am told that respondents from the rural areas even serve merienda to the NSO interviewers while those living in higher-end urban housing centers send their German shepherd to answer the questions! In the 1997 elections in Mexico, proportionately allocating the undecided overestimated the votes for the ruling party – those who were going to vote against the powerful party in power were afraid or not inclined to reveal their opposition to the interviewer.
Thus, if the undecided votes of 8.1% in the Pulse survey were counted in favor of FPJ, the “reversal” of fortunes suffered by FPJ in the Pulse Asia survey is explainable. Further studies on the undecided Pinoys are certainly in order.
Both surveys intriguingly increased their sample sizes!
As I said in Part I, less than 2000 respondents are more than sufficient for election polls in our country of 84.2 million people and 36,350,561 registered voters in 2001. In the United States with its population of 187 million, the Gallup Poll typically uses a sample of only 1000 national adults. If you are not convinced, my favorite sample survey professor, Leslie Kish, will turn in his grave!
The SWS survey increased its sample size from 1200 in previous surveys to 3600 but is now back to 1200. With a wide difference of 10 percentage points between FPJ and GMA in the immediately preceding SWS survey (January 16-22), there did not seem to be enough reason to want to increase the reliability of the results. Unless the gap was expected to have narrowed significantly that it was important for the survey to be able to detect even the slightest difference in voter preferences!
Likewise, the Pulse Asia survey which used to have 1200 adult respondents increased its sample size to 1800 respondents in the January and February surveys.
Why? They were doing so well in the past! I wouldn’t want to say it is strange, but it is! It is common knowledge among survey methodologists that for this kind of survey, the increase in precision ( from +/- 3% to +/- 1.6% for SWS and from +/- 3% to +/- 2% for Pulse Asia) is usually not worth the additional cost. Maybe the elections have become so hot and interesting that both pollsters are ready to spend more?
Is the distribution of the sample respondents “representative” in both surveys.
One very basic principle in survey sampling/opinion polling is that the sample must be “representative” of the population. In other words, when we ask the sample respondents who they will vote for in the May 2004 elections, the estimated proportion in the sample who would vote for a particular candidate should be very close to what we would get if we had asked all the voters, meaning if we had taken a census instead of a survey. A spoonful of dinuguan will taste just as good as the bowlful where you scooped it. What this means is that if certain geographic or socio-economic groups are under- or over-represented in the sample, the results will be biased, unless weighting adjustments are made.
Results of self-selected samples are of course invalid. When you sample mainly voters who you know will vote for you, a self-selected sample ensues, invalidating inferences made out of the sample. And yet, such approaches have been unscrupulously utilized to win support for one’s position. You know by who?
But with the solid experience of both SWS and Pulse Asia, it is very very safe to assume that all this is known by their statisticians. From the releases made by both pollsters, it is also very clear that they make honest efforts to adhere to the World Association for Public Opinion Research (WAPOR) Code of Professional Ethics which requires pollsters to be transparent with their methodologies.
Looking more closely at the two surveys, I noticed that in so far as GMA and FPJ are concerned, the major discrepancy between the SWS and the Pulse Asia surveys emanates from the Mindanao and the D votes where, in both groups, FPJ is way ahead in the SWS survey, BUT with GMA significantly ahead in the Pulse Asia surveys! For all other geographic areas and socio-economic classes, the surveys are consistent as to who is leading who, and their estimates are in fact not far from each other. I prepared my supporting tables on this and believe me, no one commissioned them!
By scrutinizing the allocation of the sample respondents by region and comparing this with the distribution of the registered voters vis-à-vis the relative performance of the two leading candidates, like Hercule Poirot, I stumble into some clues . I am able to do this for Mindanao and the SWS survey but not for the D votes and the Pulse Asia surveys due to data unavailability.
The SWS slightly oversampled in some areas where either FPJ has bigger margins over GMA or GMA has smaller margins over FPJ – Central Mindanao, Western Visayas and the rest of Luzon excluding NCR, especially in Pangasinan. On the other hand, the SWS undersampled in some areas where either GMA is ahead of FPJ or FPJ leads GMA by a smaller margin - Northern and Southern Mindanao, Central Visayas and NCR. Surely, these “add” to the FPJ votes, but they do not fully explain the difference between the surveys. Could it then be those undecided votes? Not necessarily, the SWS figure even went up from 6% to 6.6%!
After all that has been done and said, I jump out of the tub and shout, Eureka Not! No definitive indications are evident! Sorry folks, my verdict is not clear! So I must now suggest that the SWS and the Pulse Asia sit down together and in the spirit of self-regulation and transparency in opinion polling, make their own assessment of their discrepancy and tell us. I do not think they will risk losing their lifelines, but they owe it to the statistical profession and to the public. Their friendship might even blossom!
TABLE 1
COMPARISON OF THE SWS-PULSE ASIA SURVEY RESULTS
BY REGION |
|||||
|
FPJ |
GMA |
ROCO |
LACSON |
|
NCR |
33 |
23 |
24 |
13 |
SWS |
|
34.4 |
22.2 |
19.7 |
16.8 |
PA 1 |
|
31.8 |
22.3 |
18.0 |
14.8 |
PA 2 |
LUZON |
39 |
21 |
19 |
10 |
SWS (my estimate from SWS table) |
|
38 |
27 |
21 |
7 |
PA 1 (my estimate from PA table) |
|
36 |
26 |
15 |
12 |
PA 2 (my estimate from PA table) |
VISAYAS |
26 |
48 |
16 |
3 |
SWS |
|
26 |
49 |
20 |
3.6 |
PA 1 (my estimate from PA table) |
|
22.9 |
45.4 |
20.2 |
6.7 |
PA 2 (my estimate from PA table) |
MINDANAO |
47 |
29 |
12 |
8 |
SWS |
|
35.7 |
38.8 |
13.4 |
10.6 |
PA 1 |
|
31.0 |
37.8 |
15.4 |
8.2 |
PA 2 |
BY SOCIO-ECONOMIC CLASS |
|||||
|
FPJ |
GMA |
ROCO |
LACSON |
|
ABC |
23 |
30 |
24 |
14 |
SWS |
|
24.9 |
27.5 |
25.3 |
13.8 |
PA 1 |
|
22.5 |
33.3 |
16.9 |
13.7 |
PA 2 |
D |
36 |
29 |
18 |
8 |
SWS |
|
32.8 |
35.5 |
19.5 |
7.6 |
PA 1 |
|
29.9 |
32.9 |
17.2 |
10.7 |
PA 2 |
E |
44 |
27 |
14 |
7 |
SWS |
|
43.3 |
29.5 |
15.8 |
8.1 |
PA 1 |
|
39.3 |
28.7 |
13.2 |
9.8 |
PA 2 |
TABLE 2
DISTRIBUTION OF REGISTERED VOTERS AND
SWS SAMPLE RESPONDENTS
|
Registered Voters |
SWS Respondents |
PHILIPPINES |
100 % |
|
LUZON |
56.7 % |
56.5 % |
CAR & Ilocos (exc. Pangasinan) |
4.3 % |
4.2 % |
Cagayan Valley |
3.4 % |
3.5 % |
Pangasinan |
3.0 % |
3.2 % |
Central Luzon |
10.8% |
10.8 % |
Southern Tagalog |
15.1 % |
12.4 % |
Bicol |
5.8 % |
4.8 % |
Islands Off Luzon |
|
3.7 % |
|
|
|
NCR |
14.3 % |
13.9 % |
|
|
|
VISAYAS |
21.1 % |
20.7 % |
Western Visayas |
8.4 % |
9.6 % |
Central Visayas |
7.7 % |
6.1 % |
Eastern Visayas |
5.0 % |
5.0% |
|
|
|
MINDANAO |
22.2 % |
22.9 % |
Western Mindanao |
3.5 % |
4.9 % |
Northern Mindanao |
3.7 % |
5.0 % |
Southern Mindanao |
6.7 % |
6.1 % |
Central Mindanao |
3.2 % |
6.9 % |
|
|
|
Caraga |
2.6 % |
|
ARMM |
2.5 % |
|
Reactions and views are welcome thru email to the author at ra.virola@nscb.gov.ph.
_______________
1 Secretary General of the National Statistical Coordination Board (NSCB) and Chairman of the Statistical Research and Training Center (SRTC). He holds a Ph. D. in Statistics from the University of Michigan in Ann Arbor, USA and has taught mathematics and statistics at the University of the Philippines. He is also a past president of the Philippine Statistical Association.
Posted 08 March 2004.
 |
  |
 |
||||||
  E-mail the webmaster |
Terms of Use |
Home • Top of Page | ||||||
1997-2008,
National Statistical Coordination Board |
||||||||
Read