Nate Silver calls out Pollster on cooking polls

Pro-Republican company Strategic Vision, owners of the polling data firm Pollster, has long been suspected of fraudulently cooking polls to make its clients (the Republican party, primarily) look good. Nate Silver over at Five Thirty-Eight might have just put together sufficient mathematical proof via humans’ “gambler’s fallacy” to take them to task for it.

The specific evidence in question is as follows. I looked at all polling results reported by Strategic Vision LLC since the beginning of 2005; results from 2008 onward are available at their website; other polls were recovered through archive.org. This is a lot of data — well over 100 polls, each of which asked an average of about 15-20 questions.

For each question, I recorded the trailing digit for each candidate or line item. For instance, if Strategic Vision had Barack Obama beating John McCain 48-43 in a particular state, I’d record a tally in the 8 column and another in the 3 column. Or if they had voters opposing a particular policy 50-45, I’d record a tally in the 0 column (for 50) and another in the 5 column (for 45). I did not include “non-response responses” like “other” or “undecided”, nor did I include a tally for third-party candidates in races beteween the two major parties. I also excluded party primaries in which more than two candidates were listed, and approval questions for which more than two choices were provided.

[…]

This data is not random at all. For instance, the trailing digit was ‘8’ on 676 occasions, almost 60 percent more often than the 431 times that it was ‘1’. Over a sample of more than 5,000 data points, such an outcome occurring by chance alone would be an incredible fluke — millions to one against. Bad luck can essentially be ruled out as an explanation.

Pollster has also recently been reprimanded by the AAPOR for not disclosing their methodologies. And no wonder. How can you disclose your methodologies when they are very likely, “make up some numbers that look good for our clients and are close enough to reality that we can say it was just a sampling error?”

Strategic Vision is, of course, feigning ignorance and claiming dirty tricks, looking to crush Silver under the weight of legal recourses. They’re doing the same with AAPOR. Circle the wagons, Repubs.

{advertisement}
Nate Silver calls out Pollster on cooking polls
{advertisement}