Variance, Standard Deviation & Percentiles

This lesson on Variance, Standard Deviation & Percentiles is hands-on and example-driven. You will partition sorted numerical distributions into equal subsets using quantiles, map specific observations to their percentile ranks, and assess how small sample sizes and algorithmic variations impact quantile stability.

What You'll Be Able To Do

• Define quantiles as boundary values that divide ordered datasets into equal-sized subgroups.
• Locate the median as the 0.5 quantile (50th percentile) to partition data into two equal halves.
• Calculate the 0.25 and 0.75 quartiles to split continuous distributions into four equal parts.
• Compute the empirical percentile rank of a data point by evaluating the proportion of smaller values.
• Identify discrepancies in quantile calculations across small datasets caused by differing interpolation algorithms.

Detailed Concept Walkthrough

1. Quantiles as Distribution Dividers

Quantiles are cut points that divide a sorted dataset into equally sized, consecutive subsets. They provide rank-based positional summaries of data regardless of underlying scale or distribution shape.

• Mechanism: Data points are first sorted in ascending order from smallest to largest value. Cut points are then placed such that an identical number or proportion of observations falls into each resulting segment.
• Execution Flow: A single cut point creates two equal groups (the median or 0.5 quantile), three cut points create four equal groups (quartiles), and 99 cut points create 100 equal groups (percentiles).
• Nuance: Quantiles refer strictly to the threshold values acting as boundaries, though practitioners informally refer to the segments themselves as quantiles in conversation.

# R implementation: Calculating basic quantiles across a numeric vector
gene_expr <- c(1.2, 1.5, 1.8, 2.1, 2.4, 2.9, 3.1, 3.5, 3.8, 4.2, 4.7, 5.1, 5.6, 6.2, 7.0)

# Calculate arbitrary quantiles (e.g., tertiles: 1/3 and 2/3)
tertiles <- quantile(gene_expr, probs = c(1/3, 2/3))
print(tertiles)

Key Takeaway: Quantiles are boundary lines that partition ordered observations into groups containing equal fractions of the total dataset.

2. Median and Quartiles

The median serves as the central anchor of quantiles, dividing data in half, while quartiles further subdivide data into quarters.

• Mechanism: The 0.5 quantile (50th percentile) is the median, leaving exactly 50% of the data below it and 50% above it. The 0.25 quantile (Q1 / 25th percentile) marks the cutoff for the lowest quarter of data.
• Execution Flow: The 0.75 quantile (Q3 / 75th percentile) marks the boundary below which 75% of observations reside, leaving the top 25% above it.
• Best Practice: Use quartiles alongside the median to summarize central tendency and spread when data is skewed or contains extreme values that would distort the mean.

# Compute the 1st quartile, median, and 3rd quartile
quartiles <- quantile(gene_expr, probs = c(0.25, 0.50, 0.75))

# Output displays 25%, 50%, and 75% cutoff thresholds
print(quartiles)

Key Takeaway: Quartiles split ordered data into four equal groups using three cut points: the 0.25 quantile, the median (0.50), and the 0.75 quantile.

3. Percentiles and Empirical Rank Calculation

Percentiles are quantiles scaled from 1 to 100, representing the percentage of observations that fall below a given value.

• Mechanism: An individual data point's empirical percentile is calculated by counting how many observed values are smaller than that point and dividing by the total number of observations ($n / N$).
• Execution Flow: If 6 out of 15 gene expression values are strictly smaller than a given measurement, that measurement sits at the $6 / 15 = 0.40$ quantile, or the 40th percentile.
• Nuance: While percentiles strictly divide data into 100 buckets, data analysts frequently assign percentile ranks to individual points even in small datasets containing far fewer than 100 rows.

# Calculate empirical percentile rank of a specific value
target_val <- 3.1

# Count values strictly less than target and divide by total
percentile_rank <- sum(gene_expr < target_val) / length(gene_expr)
print(paste("Percentile rank:", percentile_rank * 100, "%"))

Key Takeaway: A data point's percentile rank equals the count of strictly smaller observations divided by the total sample size.

4. Algorithmic Variations and Sample Size Effects

Quantile values are sensitive to sample size and interpolation algorithms when data does not divide cleanly into integers.

• Mechanism: When sample sizes are small (e.g., 15 observations), boundaries rarely land on exact data points, requiring mathematical interpolation between adjacent values.
• Under the Hood: Statistical software packages offer multiple algorithmic formulas (for example, R includes 9 distinct methods in its quantile() function) to interpolate between points.
• Best Practice: In small datasets, recognize that estimated percentiles will fluctuate noticeably based on the selected method, whereas large datasets converge to virtually identical results.

# Comparing different R quantile calculation algorithms (Types 1 through 9)
q_type1 <- quantile(gene_expr, probs = 0.40, type = 1) # Inverse empirical CDF
q_type7 <- quantile(gene_expr, probs = 0.40, type = 7) # R default (linear interpolation)

cat("Type 1:", q_type1, "| Type 7:", q_type7, "\n")

Key Takeaway: Small sample sizes amplify differences between quantile estimation algorithms, while large datasets produce stable, method-independent thresholds.

Topics Covered in Variance, Standard Deviation & Percentiles

1. Introduction to Quantiles (0:00 - 1:14) — Presents the foundational motivation and visual representation of dividing sorted measurements into equal groups.
2. Median as 0.5 Quantile (1:15 - 2:38) — Demonstrates how the median acts as a quantile that cuts a dataset into two equal halves.
3. Quartile Cut Points (2:39 - 3:35) — Explains how the 0.25 and 0.75 quantiles divide ordered data into four equal-sized groups.
4. Percentiles Definition (3:36 - 4:19) — Defines percentiles as quantiles dividing data into 100 parts and compares fraction versus percentage notation.
5. Empirical Rank Calculation (4:20 - 5:05) — Shows how to calculate a specific data point's quantile rank by dividing the count of smaller points by the sample size.
6. Algorithmic Variations (5:06 - 6:30) — Discusses how small sample sizes lead to fluctuating results across R's multiple quantile calculation methods.

Stats & Product Analytics for Analysts Cheat Sheet

• quantile(x, probs = 0.5) — Computes the median / 0.5 quantile cut point

  quantile(c(1.2, 2.4, 3.5), probs = 0.5)
  

• quantile(x, probs = c(0.25, 0.75)) — Calculates the first and third quartiles

  quantile(gene_expr, probs = c(0.25, 0.75))
  

• quantile(x, probs = seq(0, 1, 0.01)) — Generates all percentiles from 0 to 100%

  quantile(gene_expr, probs = seq(0, 1, 0.01))
  

• sum(x < target) / length(x) — Calculates empirical percentile rank of target value

  sum(gene_expr < 3.1) / length(gene_expr)
  

• quantile(x, probs = p, type = n) — Calculates quantile using a specific algorithmic method

  quantile(gene_expr, probs = 0.5, type = 1)
  

Comparison Table

Common Pitfalls

• Mistake: Confusing the quantile cut point value with the percentage fraction representing the rank. Avoid: Treat the percentage as the rank location and the quantile as the corresponding value.
• Mistake: Expecting identical quantile values across different software tools on small datasets. Avoid: Specify the exact quantile calculation algorithm since small samples amplify method discrepancies.
• Mistake: Calculating percentiles without first sorting the dataset in ascending numerical order. Avoid: Order data sequentially before counting observations below a target threshold.

FAQs

• What is the practical difference between a quantile and a percentile? Quantiles express cut points as fractions or decimals (0.0 to 1.0), whereas percentiles express those same cut points on a 0 to 100 scale.
• Why does R provide 9 different methods to calculate quantiles? Different algorithms use different interpolation techniques to estimate cut points that fall between discrete observations in finite sample sets.
• Can you compute percentiles on datasets with fewer than 100 observations? Yes, empirical percentiles are calculated by taking the count of smaller values divided by total values, regardless of sample size.
• Why is the median referred to as the 0.5 quantile? Because it divides the cumulative distribution in half, leaving 50% (0.5) of observations below it and 50% above it.