Multiple Testing and the Family-Wise Error Rate

This lesson on Multiple Testing and the Family-Wise Error Rate is hands-on and example-driven. You will be able to identify and correct the massive inflation of false positives that occurs when running thousands of statistical tests simultaneously. You will master the conceptual logic of the Benjamini-Hochberg procedure, which controls the False Discovery Rate (FDR) in high-throughput data analysis.

What You'll Be Able To Do

• Calculate the expected number of Type I errors given a significance threshold and test count.
• Justify the necessity of multiple testing correction in large-scale experiments.
• Differentiate the P-value distribution generated under the null hypothesis from the alternative hypothesis.
• Apply the Benjamini-Hochberg logic to estimate the proportion of false discoveries in a mixed dataset.
• Describe the data analysis step required between P-value generation and declaring significance.

Detailed Concept Walkthrough

1. Scaling Type I Errors

Running many independent tests (e.g., 10,000 genes) dramatically increases the cumulative probability of observing a false positive (Type I error), even if the individual test error rate (α) is low. This inflation makes standard p < 0.05 cutoffs unusable in high-throughput studies.

• Mechanism: If you run N tests, the expected number of false positives is calculated simply as N × α. For example, 10,000 tests at α=0.05 yields 500 expected false discoveries, which often overwhelms any true signals.
• Under the Hood: The Family-Wise Error Rate (FWER) is the probability of making at least one Type I error across the entire family of tests. FWER approaches 1 rapidly as N increases, necessitating strict control methods like Bonferroni.
• Best Practice: Always calculate the expected number of false positives before interpreting uncorrected P-values in any experiment involving more than 10 simultaneous tests to understand the magnitude of the problem.

N_tests = 10000
alpha = 0.05  # Standard significance threshold

# Expected number of false positives (Type I errors)
expected_false_positives = N_tests * alpha

print(f"Expected False Discoveries: {expected_false_positives}")
# Output: Expected False Discoveries: 500.0

Key Takeaway: The cumulative chance of a false positive approaches 100% as the number of uncorrected tests increases.

2. FDR and Benjamini-Hochberg

The False Discovery Rate (FDR) is a corrective approach designed to control the expected proportion of false positives among only the results declared significant. The Benjamini-Hochberg (BH) procedure is the standard algorithm used to achieve this control.

• Mechanism: FDR is less stringent than FWER, allowing for a small, controlled fraction (e.g., 5%) of false positives among the discoveries. This trade-off significantly increases statistical power compared to methods that aim for zero false positives.
• Under the Hood: The BH procedure sorts P-values and compares them to a dynamically adjusted threshold, p_i ≤ (i/N) × Q, where i is the rank and Q is the desired FDR level. This adaptive thresholding is key to its power.
• Best Practice: Use FDR correction (BH) when maximizing statistical power is critical and a small, controlled proportion of false positives is acceptable, such as in exploratory high-throughput studies like genomics or proteomics.

Key Takeaway: FDR controls the proportion of false positives among discoveries, while FWER controls the probability of making any false positive at all.

3. Uniform P-Value Distribution

When the null hypothesis (H₀) is true for a test (i.e., there is no real effect), the resulting P-values are uniformly distributed across the entire range from 0 to 1. This means P-values are spread evenly, not clustered near 1.

• Mechanism: Uniformity implies that if you run 10,000 true null tests, approximately 5% of the P-values will fall into any 5% bin (e.g., 0.00-0.05, 0.45-0.50, 0.95-1.00).
• Under the Hood: The P-value is mathematically defined such that under H₀, the probability of observing a P-value less than or equal to any threshold α is exactly α. This definition necessitates the uniform spread.
• Best Practice: A flat, uniform P-value histogram confirms that the data contains no true signals, indicating that the null hypothesis holds true for all tests in the family.

import numpy as np

# Simulate 10,000 P-values where H0 is true (uniform distribution)
np.random.seed(42)
p_values_null = np.random.uniform(0, 1, 10000)

# Check how many fall below 0.05 (should be ~500)
significant_by_chance = np.sum(p_values_null < 0.05)
print(f"P-values < 0.05: {significant_by_chance}")

Key Takeaway: P-values derived from true null hypotheses are uniformly spread between 0 and 1.

4. Mixed Distribution and Baseline Estimation

A real experimental P-value histogram is a mixture of the uniform distribution (true nulls/noise) and a distribution skewed toward zero (true alternatives/signal). The uniform baseline allows us to estimate the number of false positives.

• Mechanism: True signals generate P-values concentrated near zero, creating a spike on the left side of the histogram. The remaining tests, which are true nulls, form a flat, uniform baseline across the entire 0-1 range.
• Under the Hood: The height of the uniform baseline (often observed in the P-value range 0.5 to 1.0) estimates π₀, the proportion of true null hypotheses. This baseline height is the expected frequency of false positives in any bin.
• Best Practice: The 'Eyeball Method' uses the height of the uniform baseline to separate true discoveries (the spike above the line) from the expected false discoveries (the portion of the spike sitting on the line).

Key Takeaway: The uniform baseline in a mixed P-value histogram represents the expected rate of false discoveries across all significance levels.

Topics Covered in Multiple Testing and the Family-Wise Error Rate

1. Introduction to FDR (0:00 - 0:30) — The lesson introduces the necessity of adjusting significance thresholds in experiments involving multiple statistical tests.
2. Multiple Testing Problem (0:30 - 4:18) — Repeated testing dramatically increases the cumulative chance of making Type I errors (false positives).
3. Defining FDR and BH (4:19 - 4:44) — False Discovery Rate is a corrective approach controlled by the Benjamini-Hochberg procedure.
4. Null P-Value Distribution (4:46 - 6:13) — P-values generated when the null hypothesis is true are uniformly distributed across the range 0 to 1.
5. Alternative P-Value Distribution (6:13 - 7:50) — P-values generated when the alternative hypothesis is true are heavily concentrated near zero.
6. Mixed Data Histogram (7:51 - 8:30) — A real experimental histogram is the sum of the uniform distribution (noise) and the skewed distribution (signal).
7. The Eyeball Method (8:30 - 9:36) — The height of the uniform baseline is used to estimate and separate true discoveries from false discoveries.

Statistics for Data Science Cheat Sheet

• Type I Error (α) — False positive; rejecting a true null hypothesis

  Expected_FP = N * alpha
  

• False Discovery Rate (FDR) — Controls false positives among significant results

  Q = 0.05
  

• Benjamini-Hochberg (BH) — Procedure to control the False Discovery Rate

  p_adj = multipletests(p_vals, method='fdr_bh')
  

• Uniform Distribution — P-values under H₀ spread evenly 0 to 1

  np.random.uniform(0, 1, 100)
  

• Family-Wise Error Rate (FWER) — Probability of making at least one Type I error

• π₀ — Proportion of true null hypotheses in the data

  Baseline height estimates π₀
  

Comparison Table

Common Pitfalls

• Mistake: Assuming P-values under H₀ cluster near 1 (non-significant). Avoid: Remember that H₀ P-values are uniformly distributed 0 to 1.
• Mistake: Using Bonferroni correction for high-throughput exploratory data. Avoid: Use Benjamini-Hochberg (FDR) to maintain higher statistical power.
• Mistake: Interpreting the spike near zero as 100% true discoveries. Avoid: The spike sits on the uniform baseline, meaning some false positives are included.
• Mistake: Ignoring the need for correction when N is large. Avoid: Calculate N × α to see the expected number of false positives.

FAQs

• Why are P-values under the null hypothesis uniform? The P-value is defined such that the probability of observing a P-value less than or equal to any threshold α is exactly α, which mathematically requires a uniform distribution.
• How does FDR differ from the Family-Wise Error Rate (FWER)? FWER controls the probability of making any false positive error across the family. FDR controls the expected proportion of false positives among the results you declare significant.
• What is the 'eyeball method' used for? It is a conceptual tool to estimate the uniform baseline (π₀) in a mixed P-value histogram, allowing visual separation of true signals from random noise.
• Does Benjamini-Hochberg guarantee that the proportion of false positives is exactly Q? No, BH controls the expected proportion of false positives (FDR) to be less than or equal to Q, not the exact observed proportion.