S2.4 Sampling and hypothesis testing
⤓ Presentation⤓ PowerPointPresentation: opens offline in any browser, simulations live. PowerPoint: editable.
A2 content (Statistics 2, specification sections 4.1–4.6); it uses the binomial and Poisson models from S2.1, the Normal tables from S1 and the approximations from S2.3.
What this page covers
7 points
- Population, census, sample, sampling unit and sampling frame; advantages and disadvantages of a census and of a sample
- Statistics and their sampling distributions, found by listing every possible sample in small cases
- Null and alternative hypotheses, test statistic and significance level
- Critical region, critical value and actual significance level
- One-tailed and two-tailed tests
- Tests for the proportion \(p\) of a binomial distribution and the mean \(\lambda\) of a Poisson distribution, using tables
- Tests using a Normal approximation with a continuity correction, and conclusions in context
In one minute
6 points
- A hypothesis test assumes a value for a population parameter, works out how likely the observed data would be under that assumption, and rejects the assumption only if the data would be unlikely enough.
- Census: every member of the population is observed. Sample: only some are, chosen from a sampling frame (a list of the sampling units).
- A statistic is a quantity calculated from the sample alone (no unknown parameters); its probability distribution over all possible samples is its sampling distribution.
- \(H_0\) states a value of the parameter, e.g. \(p=0.15\) or \(\lambda=4\). \(H_1\) says how it may differ: \(\lt\), \(\gt\) (one-tailed) or \(\ne\) (two-tailed).
- The critical region is the set of values so unlikely under \(H_0\) that we reject it. The actual significance level is the probability of landing in it when \(H_0\) is true; for discrete distributions it is rarely exactly the stated level.
- Conclude in two parts: a decision (reject \(H_0\) or not) and a sentence in context with no certainty: "there is (in)sufficient evidence to suggest …".
Contents
14 sections
- What are a population, a census and a sample?
- What is a statistic, and what is its sampling distribution?
- What is a hypothesis test?
- How do you find a critical region and the actual significance level?
- How do you test a binomial proportion?
- How do you test a Poisson mean?
- When and how do you use a Normal approximation in a test?
- How does a full hypothesis-testing question fit together?
- Exam tips for S2.4
- Common misconceptions
- How this topic connects
- Key formulae
- Check your understanding
- Specification checklist
What are a population, a census and a sample?
Census: observes or measures every member of the population. A sample observes only part of it, and uses that part to draw conclusions about the whole.
Sampling unit: one individual member that could be chosen (one student, one bolt, one household, one 10 m length of cable).
Sampling frame: the list (or map, or register) of the sampling units from which the sample is actually drawn.
Units missing from the frame can never be chosen, so a poor frame biases the sample, however carefully it is randomised.
| Advantages | Disadvantages | |
|---|---|---|
| Census | Every member is included, so the result is exact and unbiased | Slow and expensive for a large population; lots of data to process; impossible when testing destroys the item |
| Sample | Cheaper and quicker; less data; the only option when testing is destructive | Sampling variation; may not be representative if the frame is incomplete or selection is not random |
A sports centre has 3200 registered members. The manager wants to know what members think of proposed new opening hours. She has an up-to-date list of the names of all registered members. (a) Identify the population and a suitable sampling frame. (b) State the sampling unit. (c) Give one advantage and one disadvantage of asking every member (a census) rather than a sample.
(a) Population = everyone the question is about; frame = the list you draw from. Population: the 3200 registered members; sampling frame: the list of names of registered members
(b) One item that could be selected: a registered member
(c) Advantage: every member's opinion is included, so the result is exact for the centre. Disadvantage: contacting 3200 members and processing their replies takes a long time and costs a lot.
The unit is one member (or item); the frame is the list of all units that could be chosen.
What is a statistic, and what is its sampling distribution?
A statistic is a quantity calculated only from the observations in a sample: the sample mean, the median, the total, the largest value, the number of successes. It must not contain unknown parameters, so \(\bar X-\mu\) (with \(\mu\) unknown) is not a statistic.
Different samples give different values, so a statistic is a random variable; its distribution over all possible samples is its sampling distribution. For a small population, list every possible sample, find its probability (multiply, and count the arrangements), calculate the statistic, and collect equal values.
A large box contains discs, each labelled 1, 2 or 5, in the proportions 60%, 30% and 10% respectively. A random sample of three discs is taken (the box is large enough to treat the draws as independent). (a) List all the possible samples. (b) Find the sampling distribution of the median, \(M\), of the sample.
(a) The different samples, ignoring order (10 of them): {1,1,1}, {1,1,2}, {1,1,5}, {1,2,2}, {1,2,5}, {1,5,5}, {2,2,2}, {2,2,5}, {2,5,5}, {5,5,5}.
Probability of each (count the orders: 1, 3 or 6): {1,1,1}: \(0.6^3=0.216\); {1,1,2}: \(3(0.6^2)(0.3)=0.324\); {1,1,5}: \(3(0.6^2)(0.1)=0.108\); {1,2,2}: \(3(0.6)(0.3^2)=0.162\); {1,2,5}: \(6(0.6)(0.3)(0.1)=0.108\); {1,5,5}: \(3(0.6)(0.1^2)=0.018\); {2,2,2}: 0.027; {2,2,5}: 0.027; {2,5,5}: 0.009; {5,5,5}: 0.001.
(b) The median is the middle value; add the probabilities for each median:
| \(m\) | 1 | 2 | 5 |
|---|---|---|---|
| \(P(M=m)\) | 0.216 + 0.324 + 0.108 = 0.648 | 0.162 + 0.108 + 0.027 + 0.027 = 0.324 | 0.018 + 0.009 + 0.001 = 0.028 |
Check: \(0.648+0.324+0.028=1\). \(E(M)=1.436\): a sample median is near the population median, 1, but not always equal to it.
The probabilities of all listed samples must add up to 1. If they do not, an arrangement factor (the 3 in \(3(0.6^2)(0.3)\), or the 6) is usually missing.
What is a hypothesis test?
A hypothesis test asks "could this result just be chance?". Assume nothing has changed, and abandon that assumption only if the data would be surprising enough.
- A hypothesis is a statement about a population parameter: the probability \(p\) in \(B(n,p)\), or the mean \(\lambda\) in \(Po(\lambda)\).
- The null hypothesis \(H_0\) is the claim being tested, assumed true unless the data give enough evidence against it. It always contains "=": \(H_0: p=0.15\).
- The alternative hypothesis \(H_1\) says how the parameter is believed to differ. Read the question: "increased" → \(p\gt0.15\); "decreased / lower" → \(p\lt0.15\) (both one-tailed); "changed / different" → \(p\ne0.15\) (two-tailed).
- The test statistic is the statistic used to decide, e.g. \(X\) = the number of successes in the sample.
- The significance level (e.g. 5%) sets how unlikely a result must be, assuming \(H_0\), before we reject \(H_0\); a two-tailed test splits it between the tails.
Hypotheses are about the population, never the sample. Every test follows the same five-step routine.
A machine fills bags of flour. Over a long period 15% of the bags have been underweight. After the machine is serviced, an engineer takes a random sample of 20 bags to test whether the proportion of underweight bags has changed. (a) Define a suitable test statistic and state its distribution under the null hypothesis. (b) Write down suitable hypotheses and state whether the test is one-tailed or two-tailed.
(a) A count of "successes" in a fixed number of independent trials: \(X\) = the number of underweight bags in the sample of 20; under \(H_0\), \(X\sim B(20,0.15)\)
(b) Hypotheses about the population proportion \(p\) of underweight bags: \(H_0: p=0.15\), \(H_1: p\ne0.15\); two-tailed, because the engineer is looking for a change in either direction.
Writing "\(H_0: X=3\)" or "\(H_0\): 15% are underweight in the sample" is wrong: hypotheses are about the parameter \(p\).
How do you find a critical region and the actual significance level?
The critical region (CR) is the set of values of the test statistic that would lead you to reject \(H_0\). The critical value is the boundary value: the first value inside the critical region.
For a discrete distribution the tail probabilities jump, so the question tells you which rule to use. Read it and follow it:
- "as close as possible to 5% without exceeding 5%" (or "should not exceed 5%"): take the largest region whose probability is at most the level;
- "as close as possible to 5%": take the region whose probability is nearest to the level, even if it is slightly above it — compare the distances of the two probabilities either side of the level.
The two rules often agree, but not always (Worked example 4). In a two-tailed test apply the rule to each tail with half the level. Lower tail: read \(P(X\le c)\) directly; upper tail: \(P(X\ge c)=1-P(X\le c-1)\).
For the test in Worked example 3, using a 10% level of significance: (a) find the critical region, given that the probability in each tail should be as close as possible to 5% without exceeding 5%; (b) state the actual significance level; (c) the engineer finds 6 underweight bags. Comment on this result.
(a) Lower tail, \(X\sim B(20,0.15)\), compare with 0.05: \(P(X=0)=0.0388\le0.05\), \(P(X\le1)=0.1756\gt0.05\), so the lower part is \(X=0\). Upper tail: \(P(X\ge6)=1-0.9327=0.0673\gt0.05\), \(P(X\ge7)=1-0.9781=0.0219\le0.05\), so the upper part is \(X\ge7\). Critical region: \(X=0\) or \(X\ge7\)
(b) Add the two tails: \(0.0388+0.0219=\) 0.0607 (6.07%)
(c) 6 is not in the critical region, so do not reject \(H_0\): there is insufficient evidence to suggest that the proportion of underweight bags has changed.
6 out of 20 is double the old rate, yet with only 20 bags it would occur fairly often by chance. The rule matters: under "as close as possible to 5%" the upper tail would be \(X\ge6\) (0.0673 is 0.0173 from 0.05; 0.0219 is 0.0281 away), and 6 would then be significant.
Using \(P(X\ge7)=1-P(X\le7)\). The event \(X\ge7\) is the complement of \(X\le6\), so the table value needed is \(P(X\le6)\). Off-by-one slips like this move the critical value.
How do you test a binomial proportion?
There are two equivalent ways to carry out step 3 of the routine, and they always give the same decision:
- Probability method: find the probability of the observed value or anything more extreme in the direction of \(H_1\), and compare it with the significance level (with half the level in a two-tailed test).
- Critical region method: find the CR as in the previous section and see whether the observed value lies in it.
If the question does not say which, the probability method is usually quicker.
- "More extreme" means in the direction of \(H_1\): for \(H_1: p\lt0.45\) with 8 observed, find \(P(X\le8)\).
- If the probability is less than or equal to the significance level, the result is significant: reject \(H_0\). Otherwise do not reject \(H_0\).
- \(p\) greater than 0.5: the tables only go up to \(p=0.5\). Count failures instead: if \(X\sim B(n,p)\) then \(Y=n-X\sim B(n,1-p)\), and \(P(X\le k)=P(Y\ge n-k)\).
The owner of a café knows that, in the past, 45% of her customers ordered a hot drink. After introducing a range of iced drinks she believes this proportion has fallen. In a random sample of 30 customers, 8 ordered a hot drink. Test, at the 5% level of significance, whether there is evidence to support her belief. State your hypotheses clearly.
Hypotheses (one-tailed: "fallen"): \(H_0: p=0.45\), \(H_1: p\lt0.45\).
Distribution under \(H_0\): \(X\) = number of hot-drink orders in the sample, \(X\sim B(30,0.45)\).
Probability of 8 or fewer (the direction of \(H_1\)): \(P(X\le8)=0.0312\).
\(0.0312\lt0.05\), so the result is significant: reject \(H_0\). There is sufficient evidence to suggest that the proportion of customers ordering a hot drink has fallen.
Critical region method: \(P(X\le8)=0.0312\le0.05\) but \(P(X\le9)=0.0694\gt0.05\), so the CR is \(X\le8\), and 8 lies in it. Same decision.
The probability you compare must be a tail probability, \(P(X\le8)\), not \(P(X=8)\). A single probability is small for almost any value in a large sample, so comparing it with 5% gives meaningless results.
How do you test a Poisson mean?
The parameter is the mean rate \(\lambda\). Hypotheses look like \(H_0: \lambda=0.8\) (per hour), \(H_1: \lambda\gt0.8\), or \(\lt\), or \(\ne\). Scale the mean to the period observed: events at a mean rate of 0.8 per hour give, over 5 hours, \(X\sim Po(5\times0.8)=Po(4)\), and you read the tables for the scaled mean. The hypotheses may be written for the original rate (\(\lambda=0.8\)) or the scaled one (\(\lambda=4\)), as long as you are consistent and say which period it refers to. Otherwise the routine is identical.
Calls to a garage's breakdown line arrive at random at a mean rate of 0.8 per hour. After a local advertising campaign, the manager records 8 calls in a 5-hour period. (a) Test, at the 5% level of significance, whether the mean rate of calls has increased. (b) Find the critical region for this test.
(a) \(H_0: \lambda=0.8\), \(H_1: \lambda\gt0.8\) (per hour); over 5 hours, under \(H_0\): \(X\sim Po(4)\).
\(P(X\ge8)=1-P(X\le7)=1-0.9489=0.0511\).
\(0.0511\gt0.05\): not significant, do not reject \(H_0\). There is insufficient evidence to suggest that the mean rate of calls has increased.
(b) \(P(X\ge8)=0.0511\gt0.05\) and \(P(X\ge9)=1-0.9786=0.0214\le0.05\), so CR: \(X\ge9\)
Using \(Po(0.8)\) for a 5-hour count. The Poisson mean must match the interval in which the events were counted, so it is \(0.8\times5=4\).
When and how do you use a Normal approximation in a test?
When \(n\) or \(\lambda\) is beyond the tables, approximate the test statistic by a Normal distribution with the same mean and variance (S2.3):
- \(B(n,p)\approx N(np,np(1-p))\) when \(n\) is large and \(p\) is close to 0.5 (in practice, \(np\) and \(n(1-p)\) both comfortably above 5);
- \(Po(\lambda)\approx N(\lambda,\lambda)\) when \(\lambda\) is large (say above 10).
Apply a continuity correction, because the original distribution is discrete: \(P(X\ge72)\approx P(Y\gt71.5)\) and \(P(X\le25)\approx P(Y\lt25.5)\). Then standardise, round \(z\) to 2 d.p. and use \(\Phi(z)\); for a critical region use the percentage points table (1.6449 for 5% in one tail, 1.9600 for 2.5%). Hypotheses are still about \(p\) or \(\lambda\).
In the past, 40% of visitors to a museum bought something from the museum shop. After the shop is redesigned, a random sample of 150 visitors is taken and 72 of them buy something. Using a suitable approximation, test at the 5% level of significance whether the proportion of visitors buying something has increased.
Hypotheses and exact distribution: \(H_0: p=0.4\), \(H_1: p\gt0.4\); under \(H_0\), \(X\sim B(150,0.4)\).
Normal approximation (\(n\) large, \(p\) not far from 0.5): \(np=60\), \(np(1-p)=36\Rightarrow Y\sim N(60,6^2)\).
Continuity correction and standardise: \[P(X\ge72)\approx P(Y\ge71.5)=P\!\left(Z\ge\frac{71.5-60}{6}\right)=P(Z\ge1.92)=1-0.9726=0.0274.\]
\(0.0274\lt0.05\): significant, reject \(H_0\). There is sufficient evidence to suggest that the proportion of visitors buying something from the shop has increased.
Critical region version: \(\frac{x-0.5-60}{6}\ge1.6449\Rightarrow x\ge70.37\), so CR \(X\ge71\). The exact tail is about 0.028.
Write "\(P(X\ge72)\approx P(Y\gt71.5)\)" before you standardise. It is a step of the method in its own right, and it is easy to lose when it is folded silently into the \(z\) calculation.
How does a full hypothesis-testing question fit together?
A long question often combines a critical region from the tables with a later test where the mean is too large for the tables. Only the source of the probabilities changes.
Requests for a technician at an office building arrive at random at a mean rate of 3 per day. After a new computer system is installed, the facilities manager wishes to test whether the rate of requests has changed. (a) State suitable null and alternative hypotheses. (b) Using a 2-day period and a 5% level of significance, find the critical region for the test. The probability in each tail should be as close as possible to 2.5%. (c) Find the actual significance level of the test. (d) In a 2-day period there are 11 requests. State, with a reason, the conclusion of the test. (e) The manager now suspects that the rate has increased. Over the next 20 days there are 75 requests. Using a suitable approximation, test at the 5% level whether the rate has increased.
(a) Two-tailed, because the question says "changed": \(H_0: \lambda=3\), \(H_1: \lambda\ne3\) (requests per day)
(b) Under \(H_0\), over 2 days \(X\sim Po(6)\). Lower tail, nearest to 0.025: \(P(X\le1)=0.0174\) (0.0076 from 0.025) and \(P(X\le2)=0.0620\) (0.0370 from 0.025), so \(X\le1\). Upper tail: \(P(X\ge11)=1-0.9574=0.0426\) (0.0176 from 0.025) and \(P(X\ge12)=1-0.9799=0.0201\) (0.0049 from 0.025), so \(X\ge12\). Critical region: \(X\le1\) or \(X\ge12\)
(c) Add the tails: \(0.0174+0.0201=\) 0.0375
(d) 11 is not in the critical region, so do not reject \(H_0\): there is insufficient evidence that the rate of requests has changed.
(e) One-tailed now; over 20 days the mean is 60, too large for the tables. \(H_0: \lambda=3\), \(H_1: \lambda\gt3\); \(X\sim Po(60)\) under \(H_0\), approximated by \(Y\sim N(60,60)\). \[P(X\ge75)\approx P(Y\gt74.5)=P\!\left(Z\gt\frac{74.5-60}{\sqrt{60}}\right)=P(Z\gt1.87)=1-0.9693=0.0307\] \(0.0307\lt0.05\): significant, reject \(H_0\). There is sufficient evidence to suggest that the rate of requests for a technician has increased.
Equivalently, the CR is \(X\ge74\), since \(60+0.5+1.6449\sqrt{60}=73.24\). 11 in two days was not enough evidence; 75 in twenty days is: larger samples can detect smaller changes.
Exam tips for S2.4
Write \(H_0: p=0.45\) and \(H_1: p\lt0.45\), or \(H_0: \lambda=3\) and \(H_1: \lambda\ne3\). Hypotheses written with \(X\), a sample proportion or words alone are incomplete, and so is an \(H_1\) that does not match the direction in the question.
Write "\(X\sim B(30,0.45)\)" or "\(X\sim Po(4)\)" before any probability; for Poisson it shows the mean is scaled to the right period.
First read the rule the question gives: "as close as possible to 2.5%" or "as close as possible to 2.5% without exceeding 2.5%". Then quote the probability on each side of the level: "\(P(X\ge12)=0.0201\), \(P(X\ge11)=0.0426\)". That pair justifies the boundary.
For a two-tailed test at 5%, compare each tail with 2.5% (and a probability-method tail probability with 0.025). Then add both tails for the actual significance level.
First the decision ("\(0.0307\lt0.05\), reject \(H_0\)"), then the context in the words of the question: "there is sufficient evidence to suggest that the rate has increased". Never "proves", "accept \(H_1\)" or "\(H_0\) is true".
Common misconceptions
| Misconception | Why it is wrong | Correct idea |
|---|---|---|
| Not rejecting \(H_0\) proves \(H_0\) is true. | The data may simply be too few to show a real change (Worked example 4: 6 out of 20). | Say "insufficient evidence to suggest …", not "the proportion is 0.15". |
| The actual significance level equals the stated level. | Discrete tail probabilities jump, so the CR probability is rarely exactly the stated level. | Actual significance level = \(P(\text{test statistic in CR}\mid H_0\text{ true})\), e.g. 0.0607 for a 10% test. |
| \(P(X\ge7)=1-P(X\le7)\). | \(X\ge7\) and \(X\le7\) overlap at 7; the complement of \(X\ge7\) is \(X\le6\). | \(P(X\ge c)=1-P(X\le c-1)\). |
| Compare \(P(X=\text{observed})\) with the significance level. | The test needs the probability of a result at least as extreme, not of one exact value. | Use the tail in the direction of \(H_1\). |
| A census is always better than a sample. | A census may be too costly or slow, and impossible when testing destroys the item. | Choose a sample when cost, time or destructive testing make a census impractical. |
| Any calculated quantity from a sample is a statistic. | A quantity involving an unknown parameter, such as \(\bar X-\mu\), cannot be calculated from the data. | A statistic uses the sample values only (and known constants). |
How this topic connects
S2.1 Binomial and Poisson distributions
Every test here assumes a binomial or Poisson model; the conditions, the cumulative tables and the switch to \(n-X\) for \(p\gt0.5\) come from S2.1.
S1.6 and S2.3 — the Normal distribution and approximations
Standardising, \(\Phi(z)\) and the percentage points table from S1 are exactly what the approximate tests use, together with the continuity correction from S2.3.
P2 binomial expansion
Counting arrangements when listing samples uses the same coefficients \(\binom{n}{r}\) as \((a+b)^n\).
Key formulae
| Formula | Use | In the formulae list? |
|---|---|---|
| \(P(X=x)=\binom{n}{x}p^x(1-p)^{n-x}\); mean \(np\), variance \(np(1-p)\) | binomial test statistic \(X\sim B(n,p)\) under \(H_0\) | Yes |
| \(P(X=x)=\dfrac{e^{-\lambda}\lambda^x}{x!}\); mean = variance = \(\lambda\) | Poisson test statistic \(X\sim Po(\lambda)\) under \(H_0\) | Yes |
| cumulative tables \(P(X\le x)\) | tail probabilities and critical regions | Yes |
| \(P(X\ge c)=1-P(X\le c-1)\) | upper-tail probabilities from cumulative tables | No – learn it |
| \(Y=n-X\sim B(n,1-p)\) | tests with \(p\gt0.5\) (beyond the tables) | No – derive it |
| rate \(\lambda\) per unit \(\Rightarrow Po(k\lambda)\) over \(k\) units | scaling a Poisson mean to the period observed | No – learn it |
| actual significance level = \(P(\text{test statistic in CR}\mid H_0\text{ true})\) | probability of rejecting a true \(H_0\) | No – learn it |
| \(B(n,p)\approx N(np,np(1-p))\); \(Po(\lambda)\approx N(\lambda,\lambda)\) | tests when \(n\) or \(\lambda\) is beyond the tables | No – learn it |
| \(P(X\ge a)\approx P(Y\gt a-\frac12)\), \(P(X\le a)\approx P(Y\lt a+\frac12)\) | continuity correction | No – learn it |
| \(z=\dfrac{y-\mu}{\sigma}\) | standardising for \(\Phi(z)\) and the percentage points | No – learn it |
Check your understanding
1. A factory tests the burning time of its candles by burning them until they go out. Why must it use a sample rather than a census?
The test destroys each candle. A census would leave no candles to sell, so only a sample can be tested.
2. Faults on a cable occur at random at a mean rate of 2.5 per kilometre. What is the distribution of the number of faults on a 4 km length?
\(Po(10)\): the mean scales with the length, \(2.5\times4=10\).
3. \(H_0: p=0.8\), \(H_1: p\lt0.8\), \(n=10\), and 5 successes are observed. Find the probability needed for the test.
Count failures: \(Y=10-X\sim B(10,0.2)\) under \(H_0\). \(P(X\le5)=P(Y\ge5)=1-P(Y\le4)=1-0.9672=0.0328\), which is significant at 5%.
4. A random sample \(X_1,X_2,X_3\) is taken from a population with unknown mean \(\mu\). Is \(X_1+X_2+X_3\) a statistic? Is \(X_1-\mu\)?
The sum is a statistic: it uses only the sample values. \(X_1-\mu\) is not, because \(\mu\) is unknown and so it cannot be calculated from the data.
5. A test at the 5% level has critical region \(X\ge9\) with \(P(X\ge9)=0.0214\). What is the probability of rejecting \(H_0\) when it is true?
0.0214: the actual significance level, the probability of falling in the critical region when \(H_0\) is true. It is less than 5% because the distribution is discrete.
Specification checklist
- (S2 4.1) I can explain the terms population, census, sample, sampling unit and sampling frame, and identify each in a given context.
- (S2 4.1) I can give advantages and disadvantages of taking a census and of taking a sample, including when testing destroys the items.
- (S2 4.2) I can explain what a statistic is and decide whether a given expression is a statistic.
- (S2 4.2) I can find the sampling distribution of a statistic by listing all possible samples from a simple population.
- (S2 4.3) I can write null and alternative hypotheses in terms of a population parameter and define the test statistic and its distribution under the null hypothesis.
- (S2 4.4) I can find a critical region and critical value from tables, and calculate the actual significance level of a test.
- (S2 4.5) I can choose between a one-tailed and a two-tailed test, and split the significance level between the tails when needed.
- (S2 4.6) I can carry out a test for the proportion of a binomial distribution and for the mean of a Poisson distribution, including with a Normal approximation and continuity correction, and state the conclusion in context.