Should I sample on an employee pulse survey?

What is sampling?

Sampling is an approach used in market research and opinion polls, that involves only questioning a subset of people, statistically knowing their views will generally represent the entire group.

As obvious as it sounds, it’s very much a numbers game! As such sampling is only really a viable option for larger organisations.

As a general rule of thumb, we would only consider sampling if you have at least 1000 employees in every cut of data you want to see (i.e. any group or demographic breakdowns that you want to see). Given most companies will want to see at least 5 different cuts of data, this means overall you will need at least 5000 people in your whole company to consider sampling.

How many responses do you need in a sample?

In order to be an accurate reflection of the overall population, you need to make sure you have the right sample size.

To calculate your sample size, you need to know the population size of the group you want to study as well as your:

• Confidence interval – the margin of error, within which the responses for the whole population are likely to sit. STANDARD RECOMMENDATION = ±5 (i.e a margin of error of 5 percentage points, either side of the value)
• Confidence level – the statistical confidence, as a percentage, that your result is accurate. STANDARD RECOMMENDATION = 95% i.e. you can be 95% confident that your result is accurate

The sample size is then worked out using the following equation:

N = population size | e = margin of error | z = z-score

The z-score is the number of standard deviations a proportion is away from the mean and is derived from the confidence level.

Not a statistician? Luckily, we have an online calculator that will allow you to see the number of responses you need in your sample. Calculate your sample size

Things to think about before sampling

Before you begin calculating your sample, bear these things in mind:

1. You should always start by defining the lowest level team and/or demographic cut you want to see results for – this will drive your sampling approach and help you establish how many responses you need. E.g. if you want to understand the responses of individual contributors vs managers, you need to calculate your sample based on the number of managers and individual contributors you have, NOT your overall population size
2. Results have to be able to be generalised – i.e. you know that if you repeated the survey, you would have similar results
3. Ultimately with sampling what you should get is directional information – it will not give you the same depth of results as a survey for the whole population

And when it comes to sampling on an employee pulse survey, you may also need to consider:

• The data also has to feel significant to leaders – even though the data may be statistically significant, will this sample have enough responses for leadership to truly believe in it and make business decisions based on it?
• Employees need to be given equal opportunity to have their say – employee surveys are just as much about giving employees the chance to give feedback as they are about getting representative data, particularly when that feedback is being used to drive decisions or perceptions on employee experiences. So for one employee to be asked to give their views and another to not, (because their views aren’t required for a valid sample) can be a source of frustration.

Sampling adds complexity

A sampling approach will undoubtedly add complexity to the deployment of any survey program. It will require you to think carefully about communication, reporting and participant selection on your program.

Here’s just some of the complexities you could face:

• Communication – you will need to be able to specifically target some employees over others to take your survey each period. Not only this, but you will need to communicate to other employees why they might not be chosen to take the survey
• Reporting – you will need to understand and communicate the confidence intervals of your reports, and explain why some reports may be withheld if the correct threshold isn’t met
• Participant selection – you will need to draw your sample each period, and account for over-surveying or under-surveying in some areas of the business.

Why sample on a pulse survey?

Sampling is often seen as the answer to avoid taking up too much of your employees’ time taking surveys. This is particularly common in manufacturing and retail environments where time ‘on the floor’ is closely measured.

It’s also seen as the main way to avoid survey fatigue – employees getting annoyed at being asked to take surveys.

While sampling can help with the above, it does add complexity and we find there are other ways to avoid survey fatigue however.