survey, methods, program evaluation

A large sample of the wrong people is just a waste of time: How to tell if your sample is representative

October 03, 2018 | Pieta Blakely

I recently posted a #tiptuesday in which I said that a sample of the wrong people was a waste of time. A friend asked “how do you know it’s the wrong people?” That’s a great question and I’ll address it here. First, what is sampling and why do we do it?

7ef934_3eaa29bd52834bbb9f9c44e9ff1f3876-mv2_d_2800_1867_s_2

We take a sample when we want to know something about a group of people and we don’t have the time or resources to ask all of those people. What is important about a sample is that it reflects the population that want to generalize to. If we use a sophisticated sampling method, we can be fairly sure that our sample will represent our population (though we would still confirm). If we use a convenience sample -- for example, approaching people at a market or festival -- we will want to use demographic data to check that our sample is a good representation of our population.

The most important element is are the people who completed the survey members of the group that we want to talk about. In this case, do they actually live in the neighborhood? Asking for their zip code or address as part of the survey will help as confirm that. Any responses that don’t fit the group that we want to describe, we will discard.

Next, we will want to make sure that the whole neighborhood is represented. We will check that the race/ethnic makeup of the sample is very close to that of the population. For example, if the census data tells us that our neighborhood is 50% African-American, 25% White, and 25% Latinx, our sample should be similar in composition. We can use census data to tell us what the race/ethnic makeup of the neighborhood is. We will also want to ensure that our sample reflects the population of the neighborhood with respect to gender, age, language group, and cultural group. Census data might not be available to describe all the ethnic groups in your community, in those cases, you’ll just want to make sure that at least some of each group is represented in your sample. You will want to do some careful thinking about the ways that your community is diverse and make sure that those elements are included in your sample.

Subscribe to my blog

If you are taking a sample of a school or program, you can probably find the race/ethnic, gender, and age make-up of the group in the program’s enrollment data.

So, what if you find that your sample is not representative? That is, the makeup of the sample is very different from the makeup of the population you want to generalize to. For example, you’ve interviewed all women and no men, or you’ve interviewed parents of young children but not youth or seniors. First, you’ll disclose that in how you talk about your findings. You will say, for example, that parents support building a park with a tot-lot in this neighborhood but we can not speak for other residents of the neighborhood. Then, you’ll take a close look at your sampling method to understand why you missed a group of people, and go collect more data.

Still have questions about your survey or data collection plan? Click the button below to set up a conversation.

Tags: survey, methods, program evaluation