13 Workshop Activities

This workshop consists of two parts:

Part 1: Frequentist inference
Part 2: Bayesian inference

13.1 Part 1: Frequentist inference

Exercise 1: Sampling Distribution

Suppose $X_1, X_2, \dots, X_n$ are independent observations from an exponential distribution with rate $\lambda = 1$.

Simulate 1000 samples of size $n = 20$.
For each sample, compute the sample mean $\bar{X}$.
Plot a histogram of the 1000 sample means.
Compute the mean and standard deviation of the simulated sample means.

Questions:

What is the theoretical mean of $X$?
How does the average of the simulated sample means compare with the theoretical mean?
What does this illustrate about the sampling distribution of $\bar{X}$?

Exercise 2: Bias of an Estimator

Let $X_1, X_2, \dots, X_n \sim \text{Uniform}(0,1)$.

Two estimators of the mean $\mu = 0.5$ are:

\[ \hat{\mu}_1 = \bar{X} \]

and

\[ \hat{\mu}_2 = \frac{n+1}{n}\bar{X}. \]

Simulate 1000 samples with $n = 10$.
Compute both estimators for each sample.
Calculate the average value of each estimator.

Questions:

Which estimator appears to be unbiased?
Which estimator tends to overestimate the true mean?
How could simulation help compare estimators?

Exercise 3: MLE for the Bernoulli

Suppose $X_i \sim \text{Bernoulli}(p)$.

Simulate $n = 50$ Bernoulli observations with $p = 0.3$.
Compute the maximum likelihood estimator.
Repeat this simulation 1000 times and record the estimates.

Questions:

Plot a histogram of the 1000 estimates.
What is the mean of the simulated estimates?
Does the estimator appear to be unbiased?

Exercise 4: MLE for Normal Distribution (Exam-style)

Suppose

\[ X_i \sim N(\mu, \sigma^2), \]

where $\sigma^2$ is known.

Write down the likelihood function $L(\mu)$.
Derive the log-likelihood function.
Show that the maximum likelihood estimator of $\mu$ is $\bar{X}.$
Simulate $n=40$ observations from $N(5,1)$
Compute the MLE of $\mu$.
Repeat the simulation 1000 times and plot the distribution of the estimates.

Questions:

What is the mean of the simulated estimates?
What does this suggest about the estimator?
How does the variability change if the sample size increases?

Exercise 5: Confidence Interval for a Mean

Suppose the true distribution is Normal with

\[ \mu = 10, \quad \sigma = 2. \]

Simulate a sample of size $n = 40$.
Compute the sample mean $\bar{X}$.
Construct a 95% confidence interval.

Questions:

Does the interval contain the true mean $\mu = 10$?
Repeat the experiment 1000 times and record whether the interval contains the true mean.

Compute the proportion of intervals that contain the true mean.

Is this proportion close to 0.95?
What does this illustrate about confidence intervals?

13.2 Part 2: Bayesian inference

Exercise 6: Bayes Theorem

Suppose a test is 95% accurate when a disease is present and 97% accurate when the disease is absent. Suppose that 1% of the population has the disease. Let $H$ be the event that a disease is present, $T^+$ be the event of positive test result, and $T^-$ be the event of negative test result.

Draw a tree diagram or probability table to visualise the conditional probabilities
Calculate the true positive probability $P(H \mid T^+)$.
Calculate the false positive probability $P(\neg H \mid T^+)$.
Calculate the true negative probability $P(\neg H \mid T^-)$.
Calculate the false negative probability $P(H \mid T^-)$.

Exercise 7: Bayesian Estimate

(Albert et al., 2009, Chapter 4, Exercise 4.4)

A woman tells you she can predict the sex of a baby by how high it ‘rides’ in the mothers uterus. ‘High’ means a boy and ‘low’ means a girl. You suspect she is only guessing, but you are not completely sure;

Place a prior distribution on the values of her success rate ($p$) in the table below that reflects this belief that she is probably just guessing.

The table is as follows

x=matrix(0,nrow=2, ncol=7)
x[1, ]=c('Success rate p',0,0.25,0.5,0.75,0.9,1)
x[2, ]=c('Pr(p) prior',0,0.1,0.75,0.1,0.04,0.01)
library(knitr)
kable(x)

Success rate p	0	0.25	0.5	0.75	0.9	1
Pr(p) prior	0	0.1	0.75	0.1	0.04	0.01

You conduct an experiment with 10 pregnant women, and the woman is correct 7 times out of 10 in predicting the sex of the unborn baby. Update your probabilities in the table.
Based on the data obtained in part (b) compute the maximum likelihood estimate of $p$ and place a bound on its estimate.
How likely is she to have some skill in gender prediction? How likely is she to be just guessing?
Repeat (b) and (d) with a flat prior on p.

Exercise 8: Conjugate Prior

Suppose the number of customers arriving at a service desk in one hour follows a Poisson distribution with unknown rate $\lambda$:

\[ X_i \sim \text{Poisson}(\lambda), \quad i = 1, \dots, n. \]

A prior distribution is placed on $\lambda$:

\[ \lambda \sim \text{Gamma}(\alpha, \beta), \]

where $\alpha > 0$ and $\beta > 0$.

Assume the prior parameters are

\[ \alpha = 3, \qquad \beta = 1. \]

During five observed hours, the number of arriving customers is recorded as

\[ 8,\; 6,\; 7,\; 9,\; 10. \]

Questions:

Write down the likelihood function for $\lambda$ based on the observed data.
Using the conjugacy of the Gamma prior with the Poisson likelihood, determine the posterior distribution of $\lambda$. Specify the updated parameters of the Gamma distribution.
Compute the posterior mean of $\lambda$.
Find the maximum likelihood estimator (MLE) of $\lambda$ based on the observed data. Compare the MLE with the posterior mean and briefly comment on the difference.
Using statistical software (R or Python), compute a 95% credible interval for $\lambda$ from the posterior distribution.

Hint:

For the Gamma distribution parameterised as $\text{Gamma}(\alpha, \beta)$ with shape $\alpha$ and rate $\beta$: \[ E[\lambda] = \frac{\alpha}{\beta}. \]

For the Gamma–Poisson conjugate model, the posterior parameters are

\[ \alpha_{\text{post}} = \alpha + \sum x_i, \]

\[ \beta_{\text{post}} = \beta + n. \]

Extra Exercises

Let $X_1, X_2, \dots, X_n$ have a joint density function $f(X_1, X_2, \dots, X_n \mid \theta)$. Given $X_1 = x_1, X_2 = x_2, \dots, X_n = x_n$ is observed, the function of $\theta$ defined by

\[ L(\theta) = L(\theta \mid x_1, x_2, \dots, x_n) = f(x_1, x_2, \dots, x_n \mid \theta) \]

is the likelihood function.

In the following questions you are asked to specify the likelihood functions for given scenarios.

A coin is flipped 100 times. Given that there were 55 heads, specify the likelihood function for $\theta$, the probability of heads on a single toss.
Consider the distribution $\text{Multinomial}(n = 6, \theta, \theta, 1 - 2\theta)$. The following two samples are drawn from this distribution: $X = (1,3,2)$ and $X = (2,2,2)$. Specify the likelihood function for each sample and compare them.
An actuary has been advised to use the following claim size distribution as a model for a particular type of claim, with claim sizes measured in units of A$100,

\[ f_X(x) = \frac{x^2}{2\theta^3} e^{-x/\theta}, \qquad 0 < x < \infty, \quad \theta > 0 \]

Let $x_1, x_2, \dots, x_n$ be a random sample of $n$ claim sizes for such claims. Specify the likelihood function and the log-likelihood function.

Consider a sample of 1000 motor insurance policies. We assume that the annual total claim amounts per policy are independent and identically distributed. We denote by $X$ the number of policies with a total amount of over A$5000 claimed in a calendar year, and assume that $X$ has a Binomial distribution with parameters $n = 1000$ and $p$. An analyst wishes to estimate the unknown proportion $p$ of claims with amount greater than A$5000 per year.

Derive the maximum likelihood estimator for $p$.
Derive the Bayesian estimator of $p$ under quadratic loss assuming that the prior distribution of $p$ is beta with parameters $\alpha$ and $\beta$ respectively.
Comment on the relationship between the prior distribution and the posterior distribution of $p$ obtained in (b).
Assume that 50 out of 1000 policies in an actual sample have a total claim amount of over A$5000.
- Estimate $p$ using the MLE.
- Estimate $p$ using the Bayesian estimator under quadratic loss, given that the parameters of the beta prior distribution are $\alpha = 2$ and $\beta = 2$ respectively.
- Comment on the difference between the values estimated in MLE and Bayesian Estimate.

While travelling through the Mushroom Kingdom, Mario and Luigi find some rather unusual coins. They agree on a prior of $f(\theta) \sim \text{Beta}(5,5)$ for the probability of heads, though they disagree on what experiment to run to investigate $\theta$ further.
- Mario decides to flip a coin 5 times. He gets four heads in five flips.
- Luigi decides to flip a coin until the first tails. He gets four heads before the first tail.

Show that Mario and Luigi will arrive at the same posterior on $\theta$, and calculate this posterior.