Understanding Research – Political Polls and Their Context

Date: October 26, 2012 | IQS Research | News | Comments Off on Understanding Research – Political Polls and Their Context

Yesterday, the President of IQS Research Shawn Herbig spent an hour on the radio discussing some of the intricacies involved in the research and polling process.  Given the current election season, one thing we know for certain is that there is no shortage of polling results being released.

So that begs the question, how do we know which polls are right and which are not?  Is each new poll released on a daily basis reflecting real changes in how we think about the candidates?  Is polling and research indicative of emotions or behaviors, or both?  These are some the things Herbig tackled yesterday.

We posted a discussion late last year about how it may be a good idea to look at what are called polls of  polls, which take into consideration the summation of research done on a particular topic (in this case, political polling).  This will help to “weed out” fluff polls that may not be very accurate, and to place a heavier emphasis on the trend rather than specific points in time.

But beyond this, understanding the the  methodology behind polls is useful when deciding whether or not those results are reliable.  A few things to note:

1. What is the sample size? – Political polls in particular are attempting to gauge what an entire country of over 200 million registered voters think about an election.  A sample size needs to be 385 to be representative of a population of 200 million.  But oftentimes you see polls with around 1,000 respondents.  Oversampling allows researchers to make cuts in the data (say, what women think , or what what African Americans think) and still maintain a comfortable confidence level in the results.

2. How was the sample collected? – Polls on the internet, or ones that are done on media websites, aren’t too trustworthy.  They attract a particular group of respondents, thus skewing the results one way or another.  Scientific research maintains that a sample must be collected randomly in order for those results to be Representative in a population.  In other words, each person selected for a political poll, for instance, must have an equal chance to be selected as any other person in the population.

3.  Understand the context of poll/research – When the poll was taken is crucial in understanding what it is telling us.  For instance, there was a lot of polling done after each one of the presidential debates.  Not only did researchers ask who won the debate, but they also asked who those being polled were going to vote for.  After the first debate (which we could argue went in Romney’s favor), most polls showed the lead Obama had going into the debate had vanished.  Several polls showed Romney with a sizable lead.  But was this a statistical push due to the recent debate and the emotion surrounding it? Or was this increase real?

Recent polls show a leveling between the two candidates now that the debates are over, and a more objective look at the candidates can be achieved.  However, it is nearly impossible to eliminate emotion in responses, especially in a context as controversial a politics.

4. Interpreting Results – Interpretation ties in nicely with understanding the context of the research that you are viewing.  But there is a task for each of us as we interpret, and that is to leave behind our preconceived notions about the results.  This is very hard to do, as it is a natural human instinct to believe what justifies our own reasoning.  This is know as Confirmation Bias, and it can impact the way we accept or discount the research.

Taking all this into account can help us to sift through the commotion and find the value of the research being produced.  This isn’t just for political polling, but can be used for all research that you encounter.  Being good consumers of research can take a lot of effort, but it is the only way to gain a more realistic view of the world around you.

view all

“Significant” Differences May not Always Mean “Practical”

Date: August 9, 2012 | Joshua Holt | News | Comments Off on “Significant” Differences May not Always Mean “Practical”

I know we have talked a bit before about statistical significance in the data.  It’s the kind of thing that keeps us researchers in the business.  Drop a term like “statistical significance” in a party full of researchers, and watch our eyes light up with excitement.  If you thought we were nerds before, just wait!

But because we have not talked about this kind of stuff in a while, I would like to revisit the topic.  Today we take a look at what differences mean in the data.  If data is collected randomly, then differences that we may find between various groups (such as racial groups, gender, client groups, etc.) should be real.  However, aside from the magnitude of the differences, say 30 percentage points between the groups in a response to a particular question or behavior, it can become difficult to know whether or not the difference you are seeing are “real.”

You see, every sample has a “margin of error.”  You know what that is, because you see it all the time in the political polls you are bombarded with as of late.  If you don’t, read about it here.

Unless you have a census, researchers must deal with margins of error, and there are acceptable levels of error we are comfortable with, say +/- 5%.  These exist because there is a possibility that the differences we see between the groups (like the percentage of voters that will choose a particular candidate for president) occur by mere chance.  Refined sampling techniques, like the ones we use, are designed to minimize this possibility.  But again, unless we survey every possible case in a targeted population, this possibility is present.  Census targets are very costly and are trumped by the high precision of random sampling.

Let’s get a little bit more specific.

Say we see differences between males and females in how difficulty they think college will be.  A recent IQS study showed that 55% of African American male adults believe that college will be difficult for high school students.  Only 23% of females believe this.  Now, we can probably tell that this difference is real based on the magnitude in the spread (32%).  We don’t really need a statistical test to tell us this.

But let’s look at another example, one that may not be so clear.  The same study revealed that 41% of white males said that everyone should get a college degree, compared to 48% of white females.  A difference of only 7% is less clear.  Perhaps the difference is real, or perhaps it is occurring because of that possibility of chance due to sampling error that we just discussed.

To ease your anticipation as you sit on the edge of your sets, I can tell you that the difference was indeed “statistically significant.”  In other words, it was real.  There is in fact a difference in opinion between these two groups.  But is the difference  practical?  In other words, is the difference so great as to warrant different marketing campaigns directed toward men and women to raise perceptions of importance for a college education?  Are the additional costs justified?  The answer is probably no.  But sometimes the differences, while real, are too subtle to develop different strategies to address the problem(s).

Thus, there is a difference between “statistical” and “practical.”  Statistical differences are real and meaningful from a data standpoint.  They help guide researches in finding insights in the data. But if you are on the receiving end of the analysis, perhaps these differences are not always practical.  It is often the researcher’s charge to help in delineating between the two.

For more information on this topic, be sure to read our white paper on the matter.  It will give you a deeper understanding of differences that are revealed in data analysis.

Access our White Paper here.


view all

Straw Polls – Should we listen to them?

Date: September 22, 2011 | Shawn Herbig | News | Comments Off on Straw Polls – Should we listen to them?

As a researcher, I absolutely love election season.  While I could say that the reason for this is that I am simply living up to my obligations as a citizen (partly true), the real reason I enjoy it so is because of all the polls that are released.  And because so many polls are released, it can become difficult to decipher which ones are good and which ones are political nonsense.  That is what makes it interesting for a researcher!

There has been a lot of talk in the recent Republican primary race about straw polls.  And each of these polls seem to declare a different victor.  Mitt Romney won the New Hampshire poll, Rep. Ron Paul won both the Washington, D.C. and the California polls, Herman Cain won the Arizona poll, Michele Bachmann was victorious in the Iowa poll.  So many polls, so many different winners.  This begs the question, what exactly are straw polls and should we as potential voters listen to them?

Let’s begin with the first question – what is a straw poll?  There are two broad categories of polling: scientific and unscientific.  Scientific polling uses random sampling controls so that the results from a sample that is drawn is statistically representative of the population.  Previous posts have discussed this greater detail.  Unscientific polling, on the other hand, has no systematic sampling controls in place that would allow for representation of a population.  Historically, a lot of straw polls in the United States have been political in nature, and are usually fielded during election season by a particular political party.  The very name “straw poll” alludes to their nature – it is thought that this idiom alludes to a piece of straw being held in the air to determine which direction the wind is blowing.

Most straw polls are very targeted, very narrow surveys of opinion.  Their main purpose is to take a “snapshot” of a general opinion during a particular point in time.  This seems valid enough, but the difference between scientific and straw polls exists within the methodology.  Most straw polls use a form of convenience sampling that is a bit unorthodox, and the selection bias associated with can be extreme.

It is hard to assign a broad methodology to all straw polls (as each one is different in its own right), but many of them have candidates, such as in the Ames Straw Poll in Iowa, attract voters to cast their vote on who they believe should be the Republican candidate.  If it sounds like political grandstanding, it’s because it is to some degree.  It uses somewhat of an “honor system” whereby anyone can vote (within the parameters), which opens up a whole argument regarding the validity of the polls.

This brings us to our second question – should we pay any heed to the results of these polls?  I previously stated many of the recent straw polls and their victors.  There have been many polls, and there have been many different winners.  But to answer this question, we only need to look at the candidates themselves.  And they certainly place weight on these polls.  Tim Pawlenty dropped out of the Republican primary because of the lack of support the Iowa poll showed for his campaign.  Entire strategies are formulated based on results of straw polls.  That is because these polls show the weaknesses of particular candidates.  And for this reason, candidates are perhaps wise to take caution to what the polls are telling them.

However, are they good predictors of ultimate outcomes?  In answering this question, we are reminded of the 1936 presidential election.  The Literary Digest conducted its own straw poll, which showed Franklin Delano Roosevelt being defeated by a large majority.  We all know this was not the case, and the reason for this catastrophic (as it led to the downfall of the Digest) miscalculation was in the methodology of the poll, which is the main criticism of any straw poll.  The Digest used their mailing list to administer the poll, which consisted of motor vehicle registries and telephone books.  The problem here?  It was the Great Depression – many Americans were too poor to own a car or telephone, and thus a large sector of the population was neglected in this poll (selection bias at its finest), the very sector that was more likely to vote for FDR and his economic reforms.

The point of this post is this: take what you hear from these straw polls with a grain of salt.  They do little to predict outcomes, but can be very valuable to the candidates themselves in adjusting and fine tuning their campaigns.  Although there is a vast expanse of difference that exists between a lot of straw polls and scientific research, it can be surprisingly easy to muddle the reliability of each. However, knowing how to digest the results of research, both good and bad, will help you to avoid unsettling surprises.

view all

Understanding Report Statistics – An upcoming White Paper

Date: August 18, 2011 | Shawn Herbig | News | Comments Off on Understanding Report Statistics – An upcoming White Paper

Pardon the lack of relative dormancy here over the past couple weeks.  Things have been rather busy around here.

Nevertheless, I want to make you aware of a recent white paper we will be releasing on being a smart consumer and reader of statistics.  This white paper is sparked in part by a recent report we released on city services provided by Metro Government here in Louisville.  The study showed that while Police, EMS, and Fire services are generally highly looked upon here, Waste and Transportation services are not to such a high degree.

The report was released and subsequent stories were published on it.  It made headlines on the Courier Journal.  Like usual, however, it sparked a debate over the validity of the results, namely in reference to how the data was gathered.  We found that much of the community does not understand the way random sampling works – how we can collect a sample of 1,092 residents randomly throughout the city and that be representative of the entire community.  In essence, if they weren’t personally asked, then how can we say, for instance, that 91% of Louisville is highly satisfied with Fire services.

Without starting a new discourse on random sampling, the point of the matter here is that such reports can only be useful if they are understood by those who read it.  The audience must be educated to facts of statistics, and that is partly the task of the researchers.

Be on the lookout for our white paper, which we will undoubtedly post a link to on here once published.

view all

Polling: A double-edged sword

Date: May 26, 2011 | Shawn Herbig | News | Comments Off on Polling: A double-edged sword

Let us pretend for a moment that we all understand the foundations of probability theory – because this is a necessity for the purposes of this post.  Even the most seasoned of researchers and statisticians cannot possibly fully grasp something as ethereal as probability.  This is because in a sense probability of occurrences is somewhat akin to gravity – we know it exists because it works.  So long as we don’t go spinning off into space, we know that gravity is indeed doing its job well enough.  Probability is the same way.  We know that if we flip a coin 1 million times, 500,000 of those times will be a heads up occurrence.  (Of course, if gravity were to fail then so would the laws of probability, because once we flip the coin into the air, it would float out into the great unknown reaches of space!)

So, why am I saying this?  Surely it’s not because I have given up on trying to understand why I can do what I do as a researcher without question (though some still question it).  My previous post talked a bit about the power of random sampling.  Similar to gravity and coin flipping, we know that if we randomly choose people out of a particular population, then those people will truly be representative of that population.

Which brings me to this post – a second in a series of the power of sampling, if you will.  Many times, businesses and organizations will throw a short survey up on their website for any “passerby” to take.  These are called polls, and usually consist of a few quick questions aimed at gathering a pulse of a certain group of people.  They have their uses, but they should never be confused with scientific research.  In order for survey research to be scientific, a sample must be collected at random.  Non-random sampling is indeed sampling, but leads to results that cannot be claimed as representative.

Now, we are all familiar with political polling, and some of these polls are indeed scientifically gathered.  However, because of the changing nature of political attitudes, political polling often only is accurate in a particular point in time.  Non-random polling (appropriately referred to as convenience sampling), however, is only accurate of the people who participate in the poll to begin with.  One of the first things you’ll learn (or at least should) in any statistics course is that people who take the time to fill out a poll of convenience (what you typically find in pop up windows when you visit a website) are impassioned to do so.  In other words, they have had either great or terrible experiences with a particular item.  They rarely capture apathetic viewpoints – and let’s face it, most people are indifferent to most things.

But some may argue: “What polls lack in representation, they certainly make up for in convenience.”  And when organizations are concerned about quick answers to their questions, then perhaps that argument makes sense.  But when scrutinized sufficiently, such an argument shatters as quickly as glass house when the ground starts shaking.  Yes, convenience sampling, by its very nature and name, is designed to give quick and cheap estimates.  However, when answers are trying to be forged from intricate questions, decisions should not be made from such unrepresentative findings.  (Hence the double-“edgedness” of polling.)   

Good research demands the appropriate and arduous steps to ensure that what you are basing decisions on, whether they be on how to bolster sales and tackle a new market or printing tomorrow’s news headline on who won the presidency (Dewey ring a bell? Just look to your right ), are accurate and representative.  Again, convenience polling and sampling have their purposes (umm…I guess), but they only tell one side of an infinite sided die.  Such is bliss, but randomness is science!

What about you out there?  Have you stumbled across examples of poorly conducted research (namely from the perspective of sampling issues)?  We would like to hear some of your experiences – and they don’t have to be as mind-blowing and historically signficant as the Dewey-Truman headline.

view all