Understanding Research – Political Polls and Their Context

Date: October 26, 2012 | IQS Research | News | Comments Off on Understanding Research – Political Polls and Their Context



Yesterday, the President of IQS Research Shawn Herbig spent an hour on the radio discussing some of the intricacies involved in the research and polling process.  Given the current election season, one thing we know for certain is that there is no shortage of polling results being released.

So that begs the question, how do we know which polls are right and which are not?  Is each new poll released on a daily basis reflecting real changes in how we think about the candidates?  Is polling and research indicative of emotions or behaviors, or both?  These are some the things Herbig tackled yesterday.

We posted a discussion late last year about how it may be a good idea to look at what are called polls of  polls, which take into consideration the summation of research done on a particular topic (in this case, political polling).  This will help to “weed out” fluff polls that may not be very accurate, and to place a heavier emphasis on the trend rather than specific points in time.

But beyond this, understanding the the  methodology behind polls is useful when deciding whether or not those results are reliable.  A few things to note:

1. What is the sample size? – Political polls in particular are attempting to gauge what an entire country of over 200 million registered voters think about an election.  A sample size needs to be 385 to be representative of a population of 200 million.  But oftentimes you see polls with around 1,000 respondents.  Oversampling allows researchers to make cuts in the data (say, what women think , or what what African Americans think) and still maintain a comfortable confidence level in the results.

2. How was the sample collected? – Polls on the internet, or ones that are done on media websites, aren’t too trustworthy.  They attract a particular group of respondents, thus skewing the results one way or another.  Scientific research maintains that a sample must be collected randomly in order for those results to be Representative in a population.  In other words, each person selected for a political poll, for instance, must have an equal chance to be selected as any other person in the population.

3.  Understand the context of poll/research – When the poll was taken is crucial in understanding what it is telling us.  For instance, there was a lot of polling done after each one of the presidential debates.  Not only did researchers ask who won the debate, but they also asked who those being polled were going to vote for.  After the first debate (which we could argue went in Romney’s favor), most polls showed the lead Obama had going into the debate had vanished.  Several polls showed Romney with a sizable lead.  But was this a statistical push due to the recent debate and the emotion surrounding it? Or was this increase real?

Recent polls show a leveling between the two candidates now that the debates are over, and a more objective look at the candidates can be achieved.  However, it is nearly impossible to eliminate emotion in responses, especially in a context as controversial a politics.

4. Interpreting Results – Interpretation ties in nicely with understanding the context of the research that you are viewing.  But there is a task for each of us as we interpret, and that is to leave behind our preconceived notions about the results.  This is very hard to do, as it is a natural human instinct to believe what justifies our own reasoning.  This is know as Confirmation Bias, and it can impact the way we accept or discount the research.

Taking all this into account can help us to sift through the commotion and find the value of the research being produced.  This isn’t just for political polling, but can be used for all research that you encounter.  Being good consumers of research can take a lot of effort, but it is the only way to gain a more realistic view of the world around you.

view all

Faster Horses Didn’t Help the Automobile Industry and They Won’t Help Your Business Either

Date: October 17, 2011 | Shawn Herbig | News | Comments Off on Faster Horses Didn’t Help the Automobile Industry and They Won’t Help Your Business Either

As a researcher, one of the favorite questions I get asked relates to a comment typically attributed to Henry Ford about faster horses.  The story, as it is supposed to go, says that Henry Ford once proclaimed “If I would have asked my customers what they wanted they would have told me Faster Horses.”    This saying is supposed to offer justification for not doing research.  After all, if a great mind such as Henry Ford didn’t believe in asking his customers what they wanted, why should anyone else?

I have to say, as the former Lead Quality Analyst for the Ford Explorer Vehicle line and as a bit of a Ford history buff I absolutely love this story.  As the founder and president of a market research and data analytics company I love it even more.

So let’s start with the facts and look at the first misstatement. Henry Ford didn’t actually make this statement. There was an interesting article on the Harvard Business Review blog that explores the statement and confirms that it was not made by Ford.  So step one, even though most people give Ford credit for this statement we should probably acknowledge that it was really made by someone else.

Regardless of who said it though, there are some important points about this statement that should be considered by anyone trying to understand their marketplace or the world in general.

Furthermore, there is likely some truth to the literal interpretation of the saying.  In the early days of the automotive industry vehicles (horseless carriages) were competing with horses and horse-drawn carriages for share of wallet and also share of the road.  Had someone asked the typical non-vehicle driving, horse owner what (s)he wanted, a likely response may have been “faster horses”.  So that part we can assume is probably true.

The real question becomes what is meant by “faster horses” ?

Taking the answer literally provides very little information and brings about a unique set of problems.  For instance we would need to consider:

  • What are the safety concerns of faster horses?
  • Will the carriages have to be changed to handle the increased speed?
  • Will wider roads be needed?

Good research would also go on to explore the challenges with the current process.  This could include questions like:

  • Why are horses currently going slow?
  • Have you tried to go faster before? What happened?
  • What things worked when you did try to go faster?
  • What are you feeding the horses?

While interesting, none of those questions or answers really provides a deeper insight into the problem than what was available before asking the question.

As researchers and as consumers of research, we need to go farther.  To really understand the needs of the marketplace we need to focus on the meaning behind the words.  We hear the words the consumer is using but what are they really saying?  What does the consumer mean by faster horses?

In this case, we may infer that faster horses really means that the respondent wants to get from point A to point B more quickly or even more to the point, in less time.  If we focus on the end goal of the consumer (getting from point A to point B in less time) then our whole way of thinking can change. We can begin to explore why the user wants to go faster.  This could include questions like:

  • What are the benefits of going faster?
  • What losses are incurred by the inherent slowness of the current process?
  • What compensation methods does the consumer currently use to cope with slowness in the current system?
  • What opportunities could be realized by going faster?
  • What is the opportunity cost of going slow?
  • Is the desire to go faster or just take less time?

These questions allow us to fully understand the customer’s problem and help put their proposed solution (faster horses) into context.  But without these questions, we are forced to work within a linear framework for problem solving.

It is true that consumers are typically not able to describe the ideal solution to their problem.  However, they are often very capable of telling us about the problem itself.  When research focuses on the underlying problem then the potential solutions can be more fully understood.

By the way, if you really want to read some interesting information about Henry Ford, I recommend looking into the Oscar II.  As a professed pacifist Mr Ford was strongly opposed to the war and the Oscar II was part of his plan to help bring it to a close.  But that is for another post.

view all

How to Really Reduce the Number of Smokers in KY

Date: September 8, 2011 | Shawn Herbig | News | Comments Off on How to Really Reduce the Number of Smokers in KY

smoking lunchThis past Tuesday (9-6-2011) the Center for Disease Control released a new report on smoking among adults in the US.  The results weren’t surprising really.  Smoking overall has declined but not as much as the CDC had hoped.  Overall, about 19.3% of adults (roughly 45 million people) in the US smoke.  That is down from about 20.9% who smoked in 2005.  Furthermore, the people who smoke a lot (i.e. 1.5 packs of cigarettes per day) also declined, going from 13% in 2005 to 8% in 2010.  So not only are fewer people smoking but they are smoking less when they do smoke.  The Courier-Journal also wrote a pretty good story on this study.

While those trends provide some good news we also know that the smoking rate would decrease much faster if teenagers and young adults didn’t start smoking in the first place.  After all, if the supply of new smokers is stopped then the smoking rate would only continue to decline as a result of other attrition elements.  However, this opens up a much larger question about why do kids start smoking in the first place.  There are myriad opinions and thoughts about this subject and unfortunately most of the conversations never progress beyond the opinion stage.

Based on prior research we have been involved with through the Drive Cancer Out program we know that school age children exhibit strong predictive patterns around their likelihood to try smoking.  Those predictive patterns center around their beliefs that:

  • Kids can smoke once in a while without getting addicted or suffering any harmful effects
  • People who smoke are cool

The stronger a child’s association with these two statements, the more likely that child will be to try smoking as they become older…even when they tell you that they know smoking is harmful.  This becomes powerful because when we can identify these children, intervention and deterrent programs can be provided.  However, without the aid of statistics to isolate the key predictive drivers of smoking then all efforts to curtail the problem become subject to opinions and whims.

If the goal is to reduce the number of smokers then the only real path to success involves understanding why people start smoking and deter them prior to the habit taking shape.   Data can help make this a reality whereas opinions often only succeed in expending needless energy and precious resources.

view all

How We Think – The Return Trip Effect

Date: August 31, 2011 | Shawn Herbig | News | Comments Off on How We Think – The Return Trip Effect

A fundamental component of research is exploring how people think and what tools and processes we use when we form decisions.  Over the last few years this field of exploration has increased considerably.  This increase in knowledge has been a benefit for the research profession, but is also a benefit for anyone wanting to have a deeper understanding of what is really going on when a person makes a decision.  Books such as Predictably Irrational and The Invisible Gorilla help to elucidate the hidden processes that take place for all of us.

This week I was again reminded of this field of study when I read an article on the Return Trip Effect.  Most people have experienced this phenomena.  You are taking a trip someplace, maybe you are leaving on vacation.    The trip to get to your destination seems to take forever while the return trip seems to take less time…even though, both trips probably took about the same amount of time.  Many people assume this feeling is because a person is more familiar with the landscape on the return trip and therefore can better anticipate the return trip time.  As it turns out, this is not the real cause.

In reality, the perceived time  difference is a result of people’s anticipation of the destination to which they are traveling.  This anticipation influences their perceptions of the initial outbound trip time making it feel longer.  When people are returning home, their anticipation is much more sedate and as a result they are better able to predict their actual travel time.  As a result, the return trip doesn’t feel as long as the outbound trip.

To contrast this effect, consider your daily commute to and from work.  If this is part of your normal routine then you likely have reasonable expectations about both the outgoing and returning trips.  As a result, both legs of the journey seem to take about the same amount of time.

The article in USA Today does a pretty good job of summarizing the report.  This finding is important not just to help us have better vacations but because of the underlying implications of these findings. We know that people are capable of accurately providing their opinions about a topic.  However, those opinions are subject to change and those opinions are influenced by underlying factors that are not as obvious.  By having a more full understanding of the components that impact a decision we are better able to understand behavior.  And most importantly we know that the only way to change behavior is to fully understand what causes a behavior.

view all