Think Stats

1. Start with Exploratory Data Analysis (EDA) to Understand Your Data.

To work with data effectively, you have to think on two levels at the same time: the level of statistics and the level of context.

Beyond anecdotes. Moving past casual observations, a statistical approach begins with systematic data collection, like the National Survey of Family Growth (NSFG), designed for valid inferences about a population. This contrasts with anecdotal evidence, which suffers from small sample sizes, selection bias, confirmation bias, and inaccuracy. EDA provides a structured way to uncover patterns and identify limitations.

Data preparation is crucial. Before analysis, data must be cleaned, transformed, and validated. This involves handling special values (e.g., replacing error codes with np.nan), converting units (e.g., centiyears to years), and combining related variables. Validation, by comparing computed statistics with published results, is essential to catch errors and ensure data integrity.

Contextual interpretation. Understanding the origin and meaning of the data is as vital as the numbers themselves. For instance, observing a respondent with multiple miscarriages followed by a live birth reveals a human story behind the statistics. This dual perspective—statistical rigor and empathetic context—is fundamental for effective and ethical data analysis.

2. Visualize Distributions with CDFs for Clearer Insights.

It takes some time to get used to CDFs, but once you do, I think you will find that they show more information, more clearly, than PMFs.

Initial distribution views. Histograms and Probability Mass Functions (PMFs) are basic tools for describing a variable's distribution, showing the frequency or probability of each value. However, for datasets with many unique values, PMFs can become noisy and difficult to interpret, obscuring overall patterns and making comparisons between groups challenging. Binning can help, but choosing the right bin size is often tricky.

CDFs reveal patterns. Cumulative Distribution Functions (CDFs) offer a superior alternative, mapping each value to its percentile rank or cumulative probability. A CDF's smooth, sigmoid shape clearly illustrates the distribution's form, making modes and tails visible without the noise of individual value frequencies. They are particularly effective for comparing two or more distributions, as differences in shape and location become immediately apparent.

Percentile-based summaries. CDFs naturally lead to robust summary statistics. The median (50th percentile) provides a central tendency measure less sensitive to outliers than the mean, while the interquartile range (IQR) quantifies spread. Furthermore, CDFs are instrumental in generating random numbers that follow a specific distribution, by mapping uniformly chosen probabilities to corresponding values via the inverse CDF.

3. Model Empirical Data with Analytic Distributions for Simplification and Understanding.

Like all models, analytic distributions are abstractions, which means they leave out details that are considered irrelevant.

Empirical vs. analytic. Empirical distributions are derived directly from observed data, reflecting all its quirks and measurement errors. In contrast, analytic distributions are mathematical functions characterized by a few parameters, offering a simplified, idealized representation. These models smooth out idiosyncrasies, providing a concise summary of large datasets.

Common analytic models. Several analytic distributions frequently appear in real-world phenomena:

• Exponential distribution: Models interarrival times of events occurring at a constant average rate.
• Normal (Gaussian) distribution: A bell-shaped curve, ubiquitous due to the Central Limit Theorem, characterized by its mean and standard deviation.
• Lognormal distribution: Describes variables whose logarithms are normally distributed, often seen in skewed data like adult weights.
• Pareto distribution: Characterizes phenomena with a "power law" tail, where a small number of items account for a large proportion of the total, such as wealth or city sizes.

Why model? Analytic models serve multiple purposes: they compress data into a few parameters, provide insights into underlying physical processes (e.g., positive feedback leading to Pareto distributions), and facilitate mathematical analysis. While no model perfectly captures reality, a good model effectively highlights relevant aspects while omitting unneeded details, making complex data more manageable and understandable.

4. Quantify Relationships Between Variables Using Correlation and Linear Regression.

Correlation alone does not distinguish between these explanations, so it does not tell you which ones are true.

Visualizing relationships. The first step in exploring relationships between two variables is a scatter plot. However, raw scatter plots can be misleading due to data saturation (overlapping points) or rounding artifacts. Techniques like jittering (adding random noise) or hexbin plots (coloring bins by density) can improve visualization, revealing the true shape and density of the relationship.

Measuring association. Correlation quantifies the strength and direction of a relationship. Pearson's correlation measures linear relationships between variables, but it is sensitive to outliers and skewed distributions. Spearman's rank correlation, which computes Pearson's correlation on the ranks of the data, offers a more robust alternative, less affected by extreme values or non-linear monotonic relationships.

Modeling the trend. Linear least squares regression goes beyond correlation to model the slope of a linear relationship, finding the line that minimizes the sum of squared residuals (deviations from the line). This provides parameters (intercept and slope) for prediction. The goodness of fit can be assessed by the standard deviation of residuals (RMSE of predictions) or the coefficient of determination (R-squared), which indicates the proportion of variance explained by the model.

5. Estimate Population Parameters and Quantify Uncertainty with Sampling Distributions.

People often confuse standard error and standard deviation. Remember that standard deviation describes variability in a measured quantity; in this example, the standard deviation of gorilla weight is 7.5 kg. Standard error describes variability in an estimate.

The estimation challenge. When we measure a sample, we aim to estimate unknown parameters of the larger population (e.g., mean gorilla weight). An estimator, like the sample mean, is a statistic used for this purpose. Estimators have properties such as bias (tendency to over/underestimate) and Mean Squared Error (MSE), which quantifies the average squared difference between estimates and the true parameter.

Sampling distributions reveal variability. Since estimates vary from sample to sample due due to random chance (sampling error), we characterize this variability using a sampling distribution. This is the distribution of an estimator if the experiment were repeated many times. Simulating this process, by repeatedly drawing samples and computing the estimator, allows us to understand its behavior.

Quantifying uncertainty. The sampling distribution provides two key measures of uncertainty:

• Standard Error (SE): The standard deviation of the sampling distribution, indicating how much an estimate is expected to vary from the true parameter on average.
• Confidence Interval (CI): A range (e.g., 90% CI) that includes a given fraction of the sampling distribution, indicating the plausible range for the estimate.
  It's crucial to remember that SE and CI only account for sampling error, not other sources like sampling bias (non-representative samples) or measurement error.

6. Use Hypothesis Testing to Determine if Apparent Effects are Statistically Significant.

The goal of classical hypothesis testing is to answer the question, "Given a sample and an apparent effect, what is the probability of seeing such an effect by chance?"

The classical framework. Hypothesis testing evaluates whether an observed effect in a sample is likely to reflect a real difference in the larger population or merely occurred by chance. This involves four steps:

1. Test Statistic: Quantify the apparent effect (e.g., difference in means, correlation).
2. Null Hypothesis: A model assuming the apparent effect is not real (e.g., no difference between groups).
3. P-value: The probability of observing an effect as extreme as, or more extreme than, the one measured, assuming the null hypothesis is true.
4. Interpretation: If the p-value is low (e.g., < 0.05), the effect is "statistically significant," suggesting it's unlikely due to chance.

Permutation tests. A powerful computational method for calculating p-values is the permutation test. Under the null hypothesis, data from different groups are assumed to come from the same underlying distribution. By shuffling (permuting) the combined data and randomly reassigning it to groups, we simulate many outcomes under the null hypothesis. The p-value is then the fraction of these simulated outcomes where the test statistic is as extreme as, or more extreme than, the observed one.

Errors and power. Hypothesis tests are subject to two types of errors:

• False Positive (Type I Error): Concluding an effect is real when it's due to chance (probability equals the significance threshold, e.g., 5%).
• False Negative (Type II Error): Failing to detect a real effect.
  The "power" of a test is the probability of correctly detecting a real effect of a given size. A test with low power might fail to find a real effect, indicating that "not statistically significant" doesn't mean "no effect," but rather "no detectable effect with this sample size." Replication is vital to confirm findings and guard against false positives.

7. Employ Multiple Regression to Control for Confounding Factors and Improve Predictions.

In this example, mother’s age acts as a control variable; including agepreg in the model 'controls for' the difference in age between first-time mothers and others, making it possible to isolate the effect (if any) of isfirst.

Beyond simple relationships. While simple linear regression models one dependent variable with one explanatory variable, multiple regression extends this to include several explanatory variables. This allows for a more nuanced understanding of complex relationships, especially when explanatory variables are correlated and might confound each other's apparent effects.

Controlling for confounds. Multiple regression is crucial for isolating the true effect of a variable by "controlling for" others. For example, an observed difference in birth weight between first babies and others might be partly explained by differences in mothers' ages. By including mother's age as a control variable in the model, we can determine the independent effect of being a first baby, often revealing that an initial "significant" effect was spurious.

Prediction and data mining. Regression models can be used for prediction, with the R-squared value indicating how much variance in the dependent variable is explained by the model. Data mining, the process of systematically testing many variables, can uncover unexpected predictors. However, this approach carries the risk of finding spurious correlations. Logistic regression extends this framework to predict binary outcomes (e.g., baby's sex) by modeling the log-odds of the outcome, providing probabilities rather than direct numerical predictions.

8. Analyze Time-Dependent Data with Specialized Time Series and Survival Methods.

Most time series analysis is based on the modeling assumption that the observed series is the sum of three components: Trend, Seasonality, Noise.

Understanding time series. A time series is a sequence of measurements ordered by time, often with irregular intervals. For many analyses, it's beneficial to transform these into equally spaced series, for example, by computing daily averages and reindexing to explicitly represent missing data. This allows for a clearer view of how a system evolves over time.

Deconstructing patterns. Time series analysis aims to decompose the observed series into its fundamental components:

• Trend: The long-term, smooth changes in the series.
• Seasonality: Periodic variations, such as daily, weekly, or yearly cycles.
• Noise: Random fluctuations around the trend and seasonal components.
  Techniques like linear regression can model trends, while moving averages (e.g., rolling mean, exponentially-weighted moving average or EWMA) smooth out noise to reveal underlying patterns. Serial correlation and autocorrelation functions help identify periodic behavior by measuring how values correlate with lagged versions of themselves.

Survival analysis for durations. Survival analysis focuses on the duration until an event occurs, often used for lifetimes or time-to-event data. Key concepts include the survival curve, S(t) = 1 - CDF(t), which gives the probability of surviving beyond time t, and the hazard function, which describes the instantaneous risk of an event at time t, given survival up to t. Kaplan-Meier estimation is a crucial method for estimating these functions from incomplete data, such as when some subjects are still "surviving" (e.g., unmarried) at the end of the observation period.

9. Combine Computational and Analytic Methods for Robust and Efficient Analysis.

They are easier to explain and understand.

Computational advantages. Computational methods, such as simulation, resampling, and permutation tests, are central to this book's philosophy. Their primary strengths lie in their intuitive nature, making complex statistical concepts like p-values more accessible. They are also highly robust, requiring fewer assumptions about the underlying data distribution, and versatile, easily adaptable to various problems. Furthermore, their step-by-step nature makes them debuggable, fostering confidence in results.

Analytic efficiency. While computational methods are powerful, analytic methods offer significant advantages in terms of speed and precision, especially for very small p-values. The Central Limit Theorem (CLT) is a cornerstone analytic tool, stating that the distribution of sums or means of independent, identically distributed random variables approaches a normal distribution as the sample size increases, regardless of the original distribution's shape. This allows for rapid calculation of sampling distributions, standard errors, and confidence intervals.

An integrated approach. The most effective strategy often involves combining both methodologies. Computational methods are ideal for initial exploration, understanding, and validating assumptions. If runtime becomes a constraint, analytic methods can be employed for optimization. Crucially, computational results can then serve as a benchmark to cross-validate the accuracy of analytic solutions, ensuring both efficiency and reliability in statistical analysis.

Last updated: December 4, 2025