“Significant” Differences May not Always Mean “Practical”

Date: August 9, 2012 | Joshua Holt | News | Comments Off on “Significant” Differences May not Always Mean “Practical”

I know we have talked a bit before about statistical significance in the data.  It’s the kind of thing that keeps us researchers in the business.  Drop a term like “statistical significance” in a party full of researchers, and watch our eyes light up with excitement.  If you thought we were nerds before, just wait!

But because we have not talked about this kind of stuff in a while, I would like to revisit the topic.  Today we take a look at what differences mean in the data.  If data is collected randomly, then differences that we may find between various groups (such as racial groups, gender, client groups, etc.) should be real.  However, aside from the magnitude of the differences, say 30 percentage points between the groups in a response to a particular question or behavior, it can become difficult to know whether or not the difference you are seeing are “real.”

You see, every sample has a “margin of error.”  You know what that is, because you see it all the time in the political polls you are bombarded with as of late.  If you don’t, read about it here.

Unless you have a census, researchers must deal with margins of error, and there are acceptable levels of error we are comfortable with, say +/- 5%.  These exist because there is a possibility that the differences we see between the groups (like the percentage of voters that will choose a particular candidate for president) occur by mere chance.  Refined sampling techniques, like the ones we use, are designed to minimize this possibility.  But again, unless we survey every possible case in a targeted population, this possibility is present.  Census targets are very costly and are trumped by the high precision of random sampling.

Let’s get a little bit more specific.

Say we see differences between males and females in how difficulty they think college will be.  A recent IQS study showed that 55% of African American male adults believe that college will be difficult for high school students.  Only 23% of females believe this.  Now, we can probably tell that this difference is real based on the magnitude in the spread (32%).  We don’t really need a statistical test to tell us this.

But let’s look at another example, one that may not be so clear.  The same study revealed that 41% of white males said that everyone should get a college degree, compared to 48% of white females.  A difference of only 7% is less clear.  Perhaps the difference is real, or perhaps it is occurring because of that possibility of chance due to sampling error that we just discussed.

To ease your anticipation as you sit on the edge of your sets, I can tell you that the difference was indeed “statistically significant.”  In other words, it was real.  There is in fact a difference in opinion between these two groups.  But is the difference  practical?  In other words, is the difference so great as to warrant different marketing campaigns directed toward men and women to raise perceptions of importance for a college education?  Are the additional costs justified?  The answer is probably no.  But sometimes the differences, while real, are too subtle to develop different strategies to address the problem(s).

Thus, there is a difference between “statistical” and “practical.”  Statistical differences are real and meaningful from a data standpoint.  They help guide researches in finding insights in the data. But if you are on the receiving end of the analysis, perhaps these differences are not always practical.  It is often the researcher’s charge to help in delineating between the two.

For more information on this topic, be sure to read our white paper on the matter.  It will give you a deeper understanding of differences that are revealed in data analysis.

Access our White Paper here.


view all

What Makes a Statistically Valid Sample?

Date: February 23, 2012 | Shawn Herbig | News | Comments Off on What Makes a Statistically Valid Sample?

Most people have pretty limited understanding of statistics and research analytics, and they’d probably say they are thankful for that fact, but the reality is that we are bombarded with stats and surveys and analytics every time we watch TV, listen to the radio, log onto the Internet, or go grocery shopping.

So how do companies gather the information they use in their advertising and marketing? How do we know what we’re hearing is legitimate?

When we do a survey and want to determine whether we’re getting an accurate measurement, we need to know if the sample is “statistically valid.” In other words, did we ask enough people to represent the entire population of people we’re studying.

Let’s try an example: Let’s say we want to find out what percentage of the population of Louisville, Kentucky, would say vanilla is their favorite flavor of ice cream.

The best, most accurate way to find out would be to ask every single person in Louisville. But at 1,000,000 people in the region, that’s impossible. So statisticians and researchers have created methods that allow us to ask a sample of the population to get that information we want.

So what is a small enough sample that we can predict how many Louisvillians like vanilla ice cream? Half a million? No. How about 100,000? No, smaller. Maybe 10,000? Nope.

Surprisingly enough, for a population of about one million people, we would only need to survey 384 people to get a reliable answer. There are specific factors we have to consider regarding who to pick, making sure all pockets of the population are included, and the quality of the research basis, but if we can satisfy all those factors, we only have to talk to 384 people total.

And if 80% of those 384 people said vanilla’s the way to go, we would have a margin of error of 5% and we could be confident that 75% to 85% of the entire one million people in Louisville said vanilla was their favorite flavor. (We discussed margin of error in a previous post.)

What’s fascinating is if we wanted to widen our population count to, say 100 million people, you might think we have to ask 38,400 people. Not so. If you wanted to find out how many people out of 100 million would vote for vanilla, you only need to survey 385 people. It is still a statistically valid sample with just 385 people, and still able to gauge the ice cream preferences of one-third of the United States.

After you hit the 10,000 mark in your population, the number needed to survey goes up very slowly.

The world of surveys and statistics is fascinating to say the least, and there’s a lot to understand, but making sure your numbers are on par and statistically valid, is one essential element that makes research the amazing tool for business it can be.

view all

“DEWEY DEFEATS TRUMAN” – A case study in trusting the untrustworthy

Date: June 1, 2011 | Shawn Herbig | News | Comments Off on “DEWEY DEFEATS TRUMAN” – A case study in trusting the untrustworthy

Last week, I posted a commentary on the dangers of trusting polls and research derived from samples of convenience.  In this, the infamous 1948 Dewey-Truman election was referenced, in which the Chicago Tribune headlined that New York Governor and Republican challenger Thomas E. Dewey defeated incumbent President Harry S. Truman.  The headlined letters were simple, and they were big:  “DEWEY DEFEATS TRUMAN.”

Now, we all know that President Dewey went on to accomplish great and terrible things during his reign as commander and chief of the United States of America.  During his term, he helped establish NATO, fought the communist accusations and Red Scare of Senator McCarthy, sent troops to help with the Korean War, fired beloved war hero General MacArthur, and renovated the White House, thus ending his term with a dismal 22 percent approval rating.  Yes, President Dewey was indeed a controversial, both beloved and hated, president.  He is the talk of history classrooms throughout the nation!

Pardon the sarcasm.  In fact, Dewey never won the election.  Despite the Tribune’s headlines, Truman went on to win the electoral vote 303-189, and democrats regained the House and Senate.  After I posted this reference in last week’s post, I slowly began to realize, with the help of a fellow colleague, that perhaps not everyone remembers or knows about this infamous blunder.  And lest we forget, as history is forgotten, so it repeats itself.  So I am doing my part to help such disasters (though comedic as they are) from repeating themselves.

Some have claimed that this blunder was a result of conservative bias within the Tribune, but what underlay this was more so trusting inaccurate exit polling and data sources.  Such controversies have occurred since (namely the Bush-Gore election of 2000), but whereas the media were blame for lax voting in these instances, this was a product of trusting untrustworthy data.

A cautionary tale, to be sure, and one we can still learn from.  I can assure you that newspaper editors joke about this incident to others, but deep inside under locked doors they fear that their own paper may fall victim to such missteps.  Organizations and businesses should take heed as well, as trusting data that is not gathered accurately can lead to decisions that are not in the best interest of your organizations.

And for posterity’s sake, let us once again be reminded of this infamous photograph (Truman taunting the media the day after his victory as he boards a train in St. Louis). 

view all

Polling: A double-edged sword

Date: May 26, 2011 | Shawn Herbig | News | Comments Off on Polling: A double-edged sword

Let us pretend for a moment that we all understand the foundations of probability theory – because this is a necessity for the purposes of this post.  Even the most seasoned of researchers and statisticians cannot possibly fully grasp something as ethereal as probability.  This is because in a sense probability of occurrences is somewhat akin to gravity – we know it exists because it works.  So long as we don’t go spinning off into space, we know that gravity is indeed doing its job well enough.  Probability is the same way.  We know that if we flip a coin 1 million times, 500,000 of those times will be a heads up occurrence.  (Of course, if gravity were to fail then so would the laws of probability, because once we flip the coin into the air, it would float out into the great unknown reaches of space!)

So, why am I saying this?  Surely it’s not because I have given up on trying to understand why I can do what I do as a researcher without question (though some still question it).  My previous post talked a bit about the power of random sampling.  Similar to gravity and coin flipping, we know that if we randomly choose people out of a particular population, then those people will truly be representative of that population.

Which brings me to this post – a second in a series of the power of sampling, if you will.  Many times, businesses and organizations will throw a short survey up on their website for any “passerby” to take.  These are called polls, and usually consist of a few quick questions aimed at gathering a pulse of a certain group of people.  They have their uses, but they should never be confused with scientific research.  In order for survey research to be scientific, a sample must be collected at random.  Non-random sampling is indeed sampling, but leads to results that cannot be claimed as representative.

Now, we are all familiar with political polling, and some of these polls are indeed scientifically gathered.  However, because of the changing nature of political attitudes, political polling often only is accurate in a particular point in time.  Non-random polling (appropriately referred to as convenience sampling), however, is only accurate of the people who participate in the poll to begin with.  One of the first things you’ll learn (or at least should) in any statistics course is that people who take the time to fill out a poll of convenience (what you typically find in pop up windows when you visit a website) are impassioned to do so.  In other words, they have had either great or terrible experiences with a particular item.  They rarely capture apathetic viewpoints – and let’s face it, most people are indifferent to most things.

But some may argue: “What polls lack in representation, they certainly make up for in convenience.”  And when organizations are concerned about quick answers to their questions, then perhaps that argument makes sense.  But when scrutinized sufficiently, such an argument shatters as quickly as glass house when the ground starts shaking.  Yes, convenience sampling, by its very nature and name, is designed to give quick and cheap estimates.  However, when answers are trying to be forged from intricate questions, decisions should not be made from such unrepresentative findings.  (Hence the double-“edgedness” of polling.)   

Good research demands the appropriate and arduous steps to ensure that what you are basing decisions on, whether they be on how to bolster sales and tackle a new market or printing tomorrow’s news headline on who won the presidency (Dewey ring a bell? Just look to your right ), are accurate and representative.  Again, convenience polling and sampling have their purposes (umm…I guess), but they only tell one side of an infinite sided die.  Such is bliss, but randomness is science!

What about you out there?  Have you stumbled across examples of poorly conducted research (namely from the perspective of sampling issues)?  We would like to hear some of your experiences – and they don’t have to be as mind-blowing and historically signficant as the Dewey-Truman headline.

view all

The Power of a Sample – Voodoo or Science?

Date: May 18, 2011 | Shawn Herbig | News | Comments Off on The Power of a Sample – Voodoo or Science?

A recent study carried out by our company and The Civil Rights Project for Jefferson County Public Schools came under fire for a common misconception among those who don’t fully understand the power of random sampling. Without going into a long, drawn-out discussion of what the study entailed, the project aimed to gain an understanding of the Louisville community’s perceptions of the student assignment plan and the diversity goals it seeks to accomplish. Perhaps the methods would not have come under such scrutiny had the findings been less controversial, but regardless, the methods did indeed come under attack.

But if we take a moment to understand the science behind sampling methods, and realize that it is not voodoo magic, then I think the community can begin to focus on the real issues the study uncovered. To put it simply, sampling is indeed science. Without going into the theory of probablity and the numerous mathematical assesssments to test the validity of a sample, we can say that a random sample, so long as the laws of probablity and nature hold true, and some tear in the fabric of the universe has not occured, is certainly representative of any population it attempts to embody.

Let us first begin to understand why this is so. When I taught statistics and probability to undergrads during my days as an instructor, I found I needed to keep this explanation simple – not because my students lacked the intellengence to fully understand this, but more so because probablity theory can get a little sticky, and keeping the examples simple seemed to work best. Imagine we have a coin – a fair sided coin that is not weighted in any way (aside from a screw up from the Treasury, in which case your coin could be worth a bundle of cash). We all know this example. If you flip it, you have a 50-50 chance of getting a particular side of that coin. In essence, that is the law of probability (the simplest of many).

Random sampling is the same way. While there are various methods to go about sampling a population randomly, Simple Random Sampling is the easiest and most commonly used. To put it simply, each member of a population is assigned a unique value, and a random generator picks values within a defined range (say 1 to 1,000,000). Each member of that population has an equal chance of being selected. These chosen members become the lucky ones to be a true representation of a population. They are not “chosen” in the sense that they get to drink the Koolaid and ascend beyond, but they are chosen to speak on behalf of an entire population. Pretty cool, huh?!

These samples are representative because, well, probability tells that it is. I can spend pages and pages of your precious, valuable time discussing why this is the case, but that discussion will undoubtedly put you to sleep. However, this is why not every person in a population needs to be surveyed. And, it is a great cost conserving measure when you only have to sample, say, 500 people to respresent a much larger population. Here I can bore you again with monotonic relationships and exponential sampling benefits, but I will not do that. (You can thank me later).

Now for the real bang! Say you want to measure satisfaction of city services within a small city of 50,000 people. In order to have a representative sample, all you need is a sampling of 382 people (with a 5% margin of error). Now, say that you want to do the same study, only on the entire city of Louisville, with a population of nearly 1.5 million. What size sample do you think you need? Are you ready for this? The number is 385! Wow. Only 3 more randomly selected residents are needed for a population 30 times greater. The beauty of sampling, and wonders of monotonic relationships! More on that later. You can play around with all sorts of sample size calculators (or do it by long hand, if you dare). I suggest this site.

Of course, if you want a smaller margin of error (in essence, if you want to be more confident that your sample is truly accurate of your population), you need to a larger sample. But I’ll post a discussion on margins of error and confidence levels another day. I leave you now to ponder the brillance of statistics!!

view all