Statistical Inference
Up to this point, we have studied probability distributions and how to simulate random variables from them. We have generated data from known models and explored their behaviour through repeated sampling.
Statistical inference reverses this perspective.
Instead of starting with a known distribution and generating data, we now start with observed data and attempt to learn about the unknown mechanism that produced it.
In other words:
- Probability asks: If the model is known, what kind of data will we see?
- Inference asks: Given the data we observed, what can we say about the unknown model?
Example: Suppose we flip a coin 100 times and observe 63 heads.
Questions immediately arise:
- Is the coin fair?
- What is the probability of heads?
- How certain are we about our estimate?
- Would we obtain a similar result if we repeated the experiment?
These questions cannot be answered by a single probability calculation. They require reasoning about uncertainty, because:
- The data are random.
- The parameter (e.g., the probability of heads) is unknown.
- Different samples would produce different results.
This variability is the heart of inference.
Simulation plays a crucial role here. It allows us to:
- Approximate sampling distributions,
- Evaluate estimator performance,
- Construct confidence intervals,
- Explore how uncertainty behaves as sample size increases,
- Compare different inferential approaches.
Inference and simulation are therefore deeply connected.
To formalise inference, we distinguish between:
Parameter
A fixed but unknown quantity describing the population, e.g., the true probability of heads, \(p\).
Statistic
A quantity computed from data, e.g., the sample proportion, \(\hat{p}\).
The statistic is random because it depends on random data. The parameter is unknown but fixed (under the frequentist view).
Because statistics vary from sample to sample, we must understand their sampling distribution β the distribution of the estimator across repeated samples.
This is where simulation becomes powerful. Instead of deriving complicated formulas, we can simulate many repeated datasets and observe how the estimator behaves.
Two Frameworks for Inference
There are two major approaches to statistical inference, frequentist inference and bayesian inference.
| Frequentist Inference | Bayesian Inference |
|---|---|
| Treats probability as long-run frequency | Treats probability as a measure of uncertainty or belief |
| β’ Parameters are fixed but unknown. β’ Data are random. β’ Uncertainty is described using sampling distributions. β’ Inference is based on long-run repeated sampling behaviour. |
β’ Parameters are treated as random variables. β’ Prior beliefs are updated using observed data. β’ Uncertainty is described using the posterior distribution. |
| Example: β’ p-value β’ Maximum Likelihood Estimation (MLE) β’ Confidence Intervals β’ Hypothesis testing |
Example: β’ Bayesβ theorem provides the updating mechanism: \(\text{Posterior} \propto \text{Likelihood} \times \text{Prior}\) β’ Credible interval for interval estimation |
| Interpretation: A 95% confidence interval means that, over many repeated samples, 95% of such intervals would contain the true parameter. |
Interpretation: A 95% credible interval means that, given the data and prior, there is a 95% probability that the parameter lies within the interval. |
Example: Suppose we flip a coin 20 times and observe 14 heads. We want to estimate the true probability of heads, \(p\).
In the frequentist framework:
- The true probability \(p\) is fixed but unknown.
- The data (14 heads out of 20) are random.
The natural estimator is the sample proportion:
\[ \hat{p} = \frac{14}{20} = 0.7. \]
We compute a 95% confidence interval:
\[ \hat{p} \pm 1.96 \sqrt{\frac{\hat{p}(1-\hat{p})}{n}} \approx (0.5, 0.9) \]
A 95% confidence interval means:
If we repeated this experiment many times and constructed intervals in the same way, about 95% of those intervals would contain the true value of p.
In the Bayesian framework:
- The parameter \(p\) is treated as a random variable.
- We start with a simple prior belief about \(p\):
\[ p \sim \text{Beta}(1,1), \]
which represents no strong prior preference (mean = 0.5).
After observing 14 heads and 6 tails, the posterior becomes:
\[ p \mid \text{data} \sim \text{Beta}(15,7). \]
Then, we can compute posterior mean and 95% credible interval.
Suppose the 95% credible interval is approximately \((0.48, 0.88)\).
A 95% credible interval means:
Given the data and prior, there is a 95% probability that \(p\) lies in this interval.
Here, \(p\) is treated as random.