Understanding Research – Political Polls and Their Context

Date: October 26, 2012 | IQS Research | News | Comments Off on Understanding Research – Political Polls and Their Context

Yesterday, the President of IQS Research Shawn Herbig spent an hour on the radio discussing some of the intricacies involved in the research and polling process.  Given the current election season, one thing we know for certain is that there is no shortage of polling results being released.

So that begs the question, how do we know which polls are right and which are not?  Is each new poll released on a daily basis reflecting real changes in how we think about the candidates?  Is polling and research indicative of emotions or behaviors, or both?  These are some the things Herbig tackled yesterday.

We posted a discussion late last year about how it may be a good idea to look at what are called polls of  polls, which take into consideration the summation of research done on a particular topic (in this case, political polling).  This will help to “weed out” fluff polls that may not be very accurate, and to place a heavier emphasis on the trend rather than specific points in time.

But beyond this, understanding the the  methodology behind polls is useful when deciding whether or not those results are reliable.  A few things to note:

1. What is the sample size? – Political polls in particular are attempting to gauge what an entire country of over 200 million registered voters think about an election.  A sample size needs to be 385 to be representative of a population of 200 million.  But oftentimes you see polls with around 1,000 respondents.  Oversampling allows researchers to make cuts in the data (say, what women think , or what what African Americans think) and still maintain a comfortable confidence level in the results.

2. How was the sample collected? – Polls on the internet, or ones that are done on media websites, aren’t too trustworthy.  They attract a particular group of respondents, thus skewing the results one way or another.  Scientific research maintains that a sample must be collected randomly in order for those results to be Representative in a population.  In other words, each person selected for a political poll, for instance, must have an equal chance to be selected as any other person in the population.

3.  Understand the context of poll/research – When the poll was taken is crucial in understanding what it is telling us.  For instance, there was a lot of polling done after each one of the presidential debates.  Not only did researchers ask who won the debate, but they also asked who those being polled were going to vote for.  After the first debate (which we could argue went in Romney’s favor), most polls showed the lead Obama had going into the debate had vanished.  Several polls showed Romney with a sizable lead.  But was this a statistical push due to the recent debate and the emotion surrounding it? Or was this increase real?

Recent polls show a leveling between the two candidates now that the debates are over, and a more objective look at the candidates can be achieved.  However, it is nearly impossible to eliminate emotion in responses, especially in a context as controversial a politics.

4. Interpreting Results – Interpretation ties in nicely with understanding the context of the research that you are viewing.  But there is a task for each of us as we interpret, and that is to leave behind our preconceived notions about the results.  This is very hard to do, as it is a natural human instinct to believe what justifies our own reasoning.  This is know as Confirmation Bias, and it can impact the way we accept or discount the research.

Taking all this into account can help us to sift through the commotion and find the value of the research being produced.  This isn’t just for political polling, but can be used for all research that you encounter.  Being good consumers of research can take a lot of effort, but it is the only way to gain a more realistic view of the world around you.

view all

The Storytelling of Political Polls

Date: December 7, 2011 | Shawn Herbig | News | Comments Off on The Storytelling of Political Polls

The Republican Primaries begin next month in January.  This post continues our discussion of how to accurately gauge the political polls out there.  We previously wrote about the dynamics of Straw Polls.  

If you have been keeping up with the Republican primary race, then your head is probably spinning.  I know mine certainly is.  We have already seen the rise, shine, and fall of several political stars – Michelle Bachmann, Rick Perry, Tim Pawlenty, and the latest of whom is Herman Cain.  As quickly as they come into the national spotlight they fade away into those dreaded side positions at the debates.

With so many polls being released, it seems like in the blink of an eye the entire field of candidates can be turned on its head.  Short of a drastic accusation like the one that ensnared Cain, the political climate can change as quickly as an unexpected gust of wind that abruptly lifts the hat off your head.  However, if we sift through the white noise that shrouds this current tumultuous political climate then we can begin to see a much more telling story of what is happening with the public opinion of the field.  And this is important stuff, considering primaries begin next month!

To preface this, I am not taking political positions, but rather am trying to show how understanding polling dynamics can lead to a more accurate reading of what is going on.

So let’s begin by tackling the question of “why.”  Firstly, why are there so many polls?  Secondly, why should we be skeptical of a single one?  To understand why there are so many polls, we must place the question in the context of sensitivity.  Wouldn’t it make sense then, to have so many polls if the political climate did indeed change so frequently?  Because the climate is indeed so sensitive (as public opinion often is) this makes sense.

The chart below shows the high frequency at which these polls are conducted.  Each different color point on the chart represents a separate poll; clearly there have been hundreds since this race started.  Most of the polls here are indeed scientific.  I have filtered out the ones more open to respondent bias, such as pop-up polls and straw polls (to see why such polls are not scientific, read a previous blog post).

There are two primary reasons why the polls are so plentiful.  First is more of a PR thing.  Different polling agencies and news stations will conduct polls to promote their own “expertise” on the current political climate. The second reason is a bit more to the point – that polling is conducted so frequently because the political climate shifts so suddenly.  Here is where that sensitivity I previously mentioned comes into play.  In order to keep up with the political zeitgeist, polls are conducted regularly and often to capture what is going on.

This frequency in polling answers our second question of “why.”  Namely, why should we be skeptical of them?  If a CNN poll is conducted on Monday of this week and a Rasmussen poll is conducted the very next week and we see that results are very different, then we must ask ourselves if this change is real.  Or perhaps we are only seeing a response to a hot political topic that may have made news in between.

For instance, if Candidate A has an immigration policy that strikes a chord with voters based on a recent news story on immigration reform, then we may see a surge in that candidate’s numbers.  But this does not necessarily mean that Candidate A will withstand the test of time.  Herman Cain’s 9-9-9 plan, for example, was appealing because it was so simplistic.  But that simplicity eroded away over time, and we saw his numbers begin to drop (even before the misconduct scandals took hold).  In this sense, we see that time can eventually tear down a candidate if that candidate’s campaign is not built upon a solid foundation.

To put this sway into more detail, let’s take a look at the various front runners and their ratings: Herman Cain (gray), Rick Perry (purple), and current front runner Newt Gingrich (red).  Interestingly enough, looking at the chart below, we can begin to understand where “flavor of the month” comes from in this context.  Rick Perry’s demise was followed by Cain’s rise, and Cain’s demise followed immediately by Gingrich’s rise.  The question now becomes can Gingrich sustain his recent rise in popularity?

I cannot answer that question, nor do I want to.  As a researcher, I am more so interested in these spikes we are seeing in popularity.  But let’s now look at the more middling performing campaigns.  If we look at their poll performance, then we see a much more consistent picture.  The chart below overlays Mitt Romney’s and Ron Paul’s performance in the polls onto what we have already looked at.

In light of this, we see that these two additional candidates have been holding steady over the course of time.  Granted, one could argue that Romney has seen spikes and drops here and there, but these are only blips on the larger radar, and he has always seemed to level out over time.  Perhaps this is evidence of a more attractive message that he is offering, one that has gathered a very loyal cohort of followers.  The same can be said of Paul.  Between these two politicians, we are not seeing evidence of a star studded performance as with the others, but we do see that their campaigns are sustaining.

Reading polls can be an interesting and challenging exercise; their storytelling is both revealing and elusive.  Only when we take them all into consideration, the poll of polls per se, can we really begin to understand what is happening in the field.  I am not trying to make predictions here, but rather attempting to show how research must be placed into context  to fully comprehend the big picture.  A single poll taken in such a dynamic and changing atmosphere is not as powerful as looking at the aggregate results of all of them.  These polls indeed are telling us a story, and we only need to step back and look from afar to capture their narrative.

Polling charts taken from Pollster.com dashboard

view all