Sampling Distributions and the CLT

This lesson on Sampling Distributions and the CLT is hands-on and example-driven. You will be able to apply the Central Limit Theorem (CLT) to mathematically justify using parametric statistical tests, such as T-tests, even when your raw population data is not normally distributed. You will interpret the relationship between a parent distribution and the resulting normal sampling distribution of means, allowing you to make informed decisions about sample size and analytical tools.

What You'll Be Able To Do

• Define the Central Limit Theorem and its requirements for application.
• Simulate the process of repeated sampling from non-normal distributions.
• Justify the use of parametric tests based on the normality of sample means.
• Compare and contrast the properties of parent distributions versus sampling distributions.
• Identify the minimum requirements for the CLT to hold true, including exceptions like the Cauchy distribution.
• Interpret histograms representing population and sampling distributions.

Detailed Concept Walkthrough

1. Central Limit Theorem Definition

The CLT states that if you take sufficiently large random samples from any population, the distribution of the sample means will approximate a normal distribution, regardless of the population's original shape. This convergence happens as the sample size (N) increases, centering the distribution around the true population mean (μ).

• Mechanism: Repeatedly drawing samples of size N from a population and calculating the mean (x̄) for each sample generates a new distribution, called the sampling distribution of the mean.
• Under the Hood: As N grows, the standard deviation of the sampling distribution (known as the Standard Error, σₓ̄ = σ / √N) decreases, causing the distribution to become narrower and more tightly clustered around μ.
• Best Practice: Always ensure samples are drawn independently and randomly to satisfy the foundational assumptions of the CLT and avoid introducing systematic sampling bias.

import numpy as np
import matplotlib.pyplot as plt

# 1. Define a non-normal parent population (e.g., Exponential)
population = np.random.exponential(scale=2, size=100000)
sample_size = 30
num_samples = 1000

sample_means = []
for _ in range(num_samples):
    # 2. Draw a random sample of size N
    sample = np.random.choice(population, size=sample_size, replace=True)
    # 3. Calculate and store the mean
    sample_means.append(np.mean(sample))

# Plotting the resulting sampling distribution (should be normal)
# plt.hist(sample_means, bins=50)

Key Takeaway: The CLT transforms the distribution of means into a predictable normal shape, enabling statistical inference.

2. CLT Robustness and Requirements

The CLT is remarkably robust, applying even to highly skewed parent distributions like the Exponential or discrete distributions like the Uniform, provided the sample size is adequate. Its power lies in its independence from the original population shape.

• Mechanism: The averaging process inherent in calculating the sample mean acts as a powerful smoothing function, canceling out the skewness and extreme values present in the parent distribution.
• Under the Hood: The convergence to normality is mathematically guaranteed by the requirement that the random variables (the samples) are independent and identically distributed (i.i.d.) and possess finite variance.
• Best Practice: While N ≥ 30 is a common rule of thumb, for highly skewed distributions, a larger sample size may be necessary to achieve a satisfactory approximation of normality.
• Nuance: The CLT strictly requires that the parent population must possess a defined mean (μ) and finite variance (σ²).

import numpy as np

# Demonstrating the effect of sample size N on standard error
pop_sd = 5 # Assume population standard deviation is 5

N_small = 10  
N_large = 100 

se_n10 = pop_sd / np.sqrt(N_small)
se_n100 = pop_sd / np.sqrt(N_large)

print(f"SE for N={N_small}: {se_n10:.3f}")
print(f"SE for N={N_large}: {se_n100:.3f}")
# SE decreases significantly as N increases, tightening the distribution.

Key Takeaway: The CLT's robustness means you can trust the normality of sample means even when working with non-normal raw data.

3. Justifying Parametric Tests

Because the sampling distribution of the mean is normal (due to the CLT), we can use statistical methods that rely on the assumption of normality, such as T-tests, ANOVA, and constructing confidence intervals. This allows for powerful inference about the population mean (μ).

• Mechanism: Parametric tests require the test statistic (like the t-statistic) to follow a known distribution (like the t-distribution or Z-distribution) under the null hypothesis, which is only true if the underlying sample means are normally distributed.
• Best Practice: If the sample size N is small (N < 30), you must verify that the parent population itself is approximately normal before applying parametric tests; otherwise, use non-parametric alternatives.
• Nuance: The Cauchy Distribution is a critical exception because it lacks a defined mean and variance; therefore, repeated sampling from a Cauchy distribution does not converge to a normal distribution, and the CLT fails.

# Conceptual application of CLT for confidence interval

# Assuming CLT applies (N > 30), we use the Z-score for 95% CI (1.96)

sample_mean = 50
standard_error = 1.5
z_score_95 = 1.96

margin_of_error = z_score_95 * standard_error
lower_bound = sample_mean - margin_of_error
upper_bound = sample_mean + margin_of_error

print(f"95% Confidence Interval: ({lower_bound:.2f}, {upper_bound:.2f})")

Key Takeaway: The CLT is the methodological bridge that allows us to use powerful parametric statistics for inference.

Topics Covered in Sampling Distributions and the CLT

1. Defining the CLT (0:00 - 0:41) — The Central Limit Theorem states that the distribution of sample averages approaches a normal distribution as sample size increases.
2. Uniform Distribution Demo (0:42 - 2:40) — A simulation shows that repeated sampling from a non-normal uniform distribution results in a normally distributed set of sample means.
3. Exponential Distribution Test (2:40 - 3:57) — The CLT's robustness is confirmed by applying the same sampling process to a highly skewed exponential distribution, still yielding a normal distribution of means.
4. Parametric Test Justification (3:58 - 5:12) — The normality of sample means justifies using standard parametric statistical tests like T-tests and ANOVA regardless of the initial population distribution.
5. CLT Caveats and N=30 (5:13 - 5:57) — The common N≥30 rule of thumb is discussed alongside the strict mathematical requirement that the population must possess a defined mean, excluding distributions like the Cauchy.

Statistics for Data Science Cheat Sheet

• CLT (Central Limit Theorem) — Distribution of sample means approaches normal

• Standard Error (σₓ̄) — Standard deviation of the sampling distribution

  σ / np.sqrt(N)
  

• N ≥ 30 — Rule of thumb for CLT approximation of normality

• Parametric Test — Statistical test assuming normality (e.g., T-test)

  scipy.stats.ttest_ind(sample1, sample2)
  

• Cauchy Distribution — Population lacking a defined mean; CLT fails

Comparison Table

Common Pitfalls

• Mistake: Assuming the raw population data must be normal for a T-test. Avoid: Relying on the CLT to justify normality of the sample means.
• Mistake: Believing N=30 guarantees perfect normality for all data. Avoid: Using a larger N if the parent distribution is extremely skewed.
• Mistake: Applying the CLT to distributions without a defined mean. Avoid: Checking if the population possesses a finite mean and variance.

FAQs

• Why does the CLT work even for highly skewed data? The process of averaging multiple independent samples cancels out the skewness and extreme values, forcing the resulting distribution toward symmetry.
• What is the practical difference between population standard deviation and standard error? Population standard deviation measures the spread of individual data points, while standard error measures the spread of the sample means.
• If my sample size is small (N<30), can I still use a T-test? Yes, but only if you can confirm that the underlying parent population is already approximately normally distributed.
• Does the CLT apply to sample medians or variances? No, the CLT specifically applies to the distribution of sample means (or sums), not other statistics like medians or variances.