Sampling and probability: surveys, bias, margin of error and why size matters

You rarely have data on everyone. Pollsters ask about 2,000 people to estimate how millions will vote; a county surveys a few hundred households. Learning how sampling works helps you trust good surveys and spot bad ones.

Population and sample

• Population: everyone or everything you want to know about (all customers, all voters, all farms in a county).
• Sample: the smaller group you actually measure.
• A statistic describes the sample; we use it to estimate the population.

Good and bad samples

Types of bias

Probability basics

Probability is how likely something is, from 0 (impossible) to 1 (certain). Probability of an event = favourable outcomes ÷ all possible outcomes. A fair coin landing heads: 1 ÷ 2 = 0.5.

The law of large numbers: the more times you repeat a random experiment, the closer the results get to the true probability. Watch it happen:

import random
random.seed(7)

for flips in [10, 100, 1000, 10000]:
    heads = sum(random.random() < 0.5 for _ in range(flips))
    print(f"{flips:>6} flips: {heads / flips:.3f} heads")

With 10 flips the share of heads can be far from 0.5; with 10,000 it's very close.

Why sample size matters

Imagine 40% of a county's farmers use certified seed. Let's draw random samples of different sizes and see how close the estimates get:

import random
random.seed(1)

population = [1] * 40000 + [0] * 60000   # 1 = uses certified seed (40%)

for n in [20, 100, 500, 2000]:
    estimates = []
    for _ in range(200):
        sample = random.sample(population, n)
        estimates.append(sum(sample) / n)
    print(f"sample of {n:>4}: estimates ranged from {min(estimates):.0%} to {max(estimates):.0%}")

Small samples bounce around a lot; bigger samples cluster near the true 40%.

Margin of error (the rough rule)

Surveys report results like "52% ± 3%". For a simple random sample, a quick approximation of the 95% margin of error for a percentage is:

margin of error ≈ 1 ÷ √(sample size)

for n in [100, 400, 1000, 2000, 4000]:
    moe = 1 / n ** 0.5
    print(f"sample of {n:>4}: about ±{moe:.1%}")

Notice that to halve the margin of error you need four times the sample. That's why national polls rarely survey more than a few thousand people.

Summary

• A sample estimates a population; random sampling is the foundation.
• Watch for selection, non-response, leading-question and survivorship bias.
• Bigger random samples give estimates closer to the truth (law of large numbers).
• Margin of error ≈ 1 ÷ √n for percentages; quadruple the sample to halve it.