Let us pretend for a moment that we all understand the foundations of probability theory – because this is a necessity for the purposes of this post. Even the most seasoned of researchers and statisticians cannot possibly fully grasp something as ethereal as probability. This is because in a sense probability of occurrences is somewhat akin to gravity – we know it exists because it works. So long as we don’t go spinning off into space, we know that gravity is indeed doing its job well enough. Probability is the same way. We know that if we flip a coin 1 million times, 500,000 of those times will be a heads up occurrence. (Of course, if gravity were to fail then so would the laws of probability, because once we flip the coin into the air, it would float out into the great unknown reaches of space!)
So, why am I saying this? Surely it’s not because I have given up on trying to understand why I can do what I do as a researcher without question (though some still question it). My previous post talked a bit about the power of random sampling. Similar to gravity and coin flipping, we know that if we randomly choose people out of a particular population, then those people will truly be representative of that population.
Which brings me to this post – a second in a series of the power of sampling, if you will. Many times, businesses and organizations will throw a short survey up on their website for any “passerby” to take. These are called polls, and usually consist of a few quick questions aimed at gathering a pulse of a certain group of people. They have their uses, but they should never be confused with scientific research. In order for survey research to be scientific, a sample must be collected at random. Non-random sampling is indeed sampling, but leads to results that cannot be claimed as representative.
Now, we are all familiar with political polling, and some of these polls are indeed scientifically gathered. However, because of the changing nature of political attitudes, political polling often only is accurate in a particular point in time. Non-random polling (appropriately referred to as convenience sampling), however, is only accurate of the people who participate in the poll to begin with. One of the first things you’ll learn (or at least should) in any statistics course is that people who take the time to fill out a poll of convenience (what you typically find in pop up windows when you visit a website) are impassioned to do so. In other words, they have had either great or terrible experiences with a particular item. They rarely capture apathetic viewpoints – and let’s face it, most people are indifferent to most things.
But some may argue: “What polls lack in representation, they certainly make up for in convenience.” And when organizations are concerned about quick answers to their questions, then perhaps that argument makes sense. But when scrutinized sufficiently, such an argument shatters as quickly as glass house when the ground starts shaking. Yes, convenience sampling, by its very nature and name, is designed to give quick and cheap estimates. However, when answers are trying to be forged from intricate questions, decisions should not be made from such unrepresentative findings. (Hence the double-“edgedness” of polling.)
Good research demands the appropriate and arduous steps to ensure that what you are basing decisions on, whether they be on how to bolster sales and tackle a new market or printing tomorrow’s news headline on who won the presidency (Dewey ring a bell? Just look to your right →), are accurate and representative. Again, convenience polling and sampling have their purposes (umm…I guess), but they only tell one side of an infinite sided die. Such is bliss, but randomness is science!
What about you out there? Have you stumbled across examples of poorly conducted research (namely from the perspective of sampling issues)? We would like to hear some of your experiences – and they don’t have to be as mind-blowing and historically signficant as the Dewey-Truman headline.
Let us pretend for a moment that we all understand the foundations of probability theory – because this is a necessity for the purposes of this post. Even the most seasoned of researchers and statisticians cannot possibly fully grasp something as ethereal as probability. This is because in a sense probability of occurrences is somewhat akin to gravity – we know it exists because it works. So long as we don’t go spinning off into space, we know that gravity is indeed doing its job well enough. Probability is the same way. We know that if we flip a coin 1 million times, 500,000 of those times will be a heads up occurrence. (Of course, if gravity were to fail then so would the laws of probability, because once we flip the coin into the air, it would float out into the great unknown reaches of space!)
So, why am I saying this? Surely it’s not because I have given up on trying to understand why I can do what I do as a researcher without question (though some still question it). My previous post talked a bit about the power of random sampling. Similar to gravity and coin flipping, we know that if we randomly choose people out of a particular population, then those people will truly be representative of that population.
Which brings me to this post – a second in a series of the power of sampling, if you will. Many times, businesses and organizations will throw a short survey up on their website for any “passerby” to take. These are called polls, and usually consist of a few quick questions aimed at gathering a pulse of a certain group of people. They have their uses, but they should never be confused with scientific research. In order for survey research to be scientific, a sample must be collected at random. Non-random sampling is indeed sampling, but leads to results that cannot be claimed as representative.
Now, we are all familiar with political polling, and some of these polls are indeed scientifically gathered. However, because of the changing nature of political attitudes, political polling often only is accurate in a particular point in time. Non-random polling (appropriately referred to as convenience sampling), however, is only accurate of the people who participate in the poll to begin with. One of the first things you’ll learn (or at least should) in any statistics course is that people who take the time to fill out a poll of convenience (what you typically find in pop up windows when you visit a website) are impassioned to do so. In other words, they have had either great or terrible experiences with a particular item. They rarely capture apathetic viewpoints – and let’s face it, most people are indifferent to most things.
But some may argue: “What polls lack in representation, they certainly make up for in convenience.” And when organizations are concerned about quick answers to their questions, then perhaps that argument makes sense. But when scrutinized sufficiently, such an argument shatters as quickly as glass house when the ground starts shaking. Yes, convenience sampling, by its very nature and name, is designed to give quick and cheap estimates. However, when answers are trying to be forged from intricate questions, decisions should not be made from such unrepresentative findings. (Hence the double-“edgedness” of polling.)
Good research demands the appropriate and arduous steps to ensure that what you are basing decisions on, whether they be on how to bolster sales and tackle a new market or printing tomorrow’s news headline on who won the presidency (Dewey ring a bell? Just look to your right →), are accurate and representative. Again, convenience polling and sampling have their purposes (umm…I guess), but they only tell one side of an infinite sided die. Such is bliss, but randomness is science!
What about you out there? Have you stumbled across examples of poorly conducted research (namely from the perspective of sampling issues)? We would like to hear some of your experiences – and they don’t have to be as mind-blowing and historically signficant as the Dewey-Truman headline.