Steve King at SmallBizLabs asked an interesting question recently: “Are Statistically Significant Research Methods Passe?” He notes that a growing number of surveys that show up these days are not statistically valid because they don’t start with random populations. Steve says they’re “based on informal groups of respondents rather than statistically significant population samples.”
He cites three drivers for this trend:
- Cost and speed–It is much cheaper, easier and quicker to do surveys informally, rather than investing in the rigor of statistically significant research methodology. The appeal of doing 100 quick-and-dirty surveys instead of investing in one statistically rigorous survey is obvious.
- The availability of easy-to-use Internet survey tools–Tools like Survey Monkey make it easy to do quick and simple online surveys. This has resulted in a substantial increase in the number of surveys conducted (and results published), which may or may not measure the population that the survey is trying to understand. Think of it as a litmus test without really knowing whose chemistry you’re testing.
- There is value in the information produced by non-statistical surveys–While informal survey results are not projectable to a broader population, they can be useful. They provide something to react to quickly and cheaply which can help decision-makers and researchers think out of the box. Also, time or resource constraints often eliminate the use of formal surveys, and some information is often better than none.
I agree. And I especially like the fact that it comes from Steve, who is part of Emergent Research, a research and consulting firm that has published some very interesting work. If you’re curious, look at its studies of small business trends.
The key point here is his number three: There is still value, sometimes, even without statistical fundamentals. You just have to stick with common sense. If I remember my business school classes in statistics, if the sample population isn’t random then you can’t project results into the larger population. But if you have a non-random sample that’s directly related to your business needs, then so what?
What I say is, use it well and wisely, recognize that what you’ve got is based on a very narrow subset of the whole world; keep a healthy skepticism, but use the information.
Tim Berry is the founder and chairman of Palo Alto Software and Bplans.com. Follow him on Twitter @Timberry.
