This note covers Bayes' theorem, which reverses a conditional probability: from "how likely is this evidence if the cause is present" to "how likely is the cause now that we have seen the evidence". It is used in medical testing, spam filters, fault diagnosis and many other places where evidence has to be interpreted.
Key ideas
You already know that \(P(A \mid B)\) and \(P(B \mid A)\) are different. Bayes' theorem links them. The starting point is the multiplication rule written two ways:
\[ P(A \cap B) = P(A \mid B)\,P(B) = P(B \mid A)\,P(A). \]Dividing by \(P(A)\) gives \(P(B \mid A) = \dfrac{P(A \mid B)\,P(B)}{P(A)}\). The remaining task is to find \(P(A)\), the overall probability of the evidence.
Law of total probability. The event \(A\) can happen together with \(B\) or together with \(B'\). Adding these two routes gives
\[ P(A) = P(A \mid B)\,P(B) + P(A \mid B')\,P(B'). \]On a tree diagram, this is the sum of the two branches that end in \(A\).
Bayes' theorem for two events is therefore
\[ P(B \mid A) = \frac{P(A \mid B)\,P(B)}{P(A \mid B)\,P(B) + P(A \mid B')\,P(B')}. \]The theorem extends to three events \(B_1, B_2, B_3\) that cannot overlap and together cover every possibility (a partition). Then
\[ P(B_i \mid A) = \frac{P(A \mid B_i)\,P(B_i)}{P(A \mid B_1)P(B_1) + P(A \mid B_2)P(B_2) + P(A \mid B_3)P(B_3)}. \]A practical method that avoids memorising the formula.
- Draw a tree. The first stage is the "cause" (for example, has the disease or not). The second stage is the "evidence" (for example, test positive or negative).
- Multiply along each branch that ends in the evidence you observed.
- Add those products to get \(P(\text{evidence})\). This is the denominator.
- The probability you want is the product for the branch of interest divided by that total.
Why the answer can be surprising. If a cause is rare (a small \(P(B)\)), then even an accurate test produces many false alarms compared with true detections. The reversed probability can then be much smaller than the test's accuracy suggests. The starting probability \(P(B)\) matters as much as the accuracy.
Definitions and formulae
| Term or formula | Meaning / when to use |
|---|---|
| \(P(A) = P(A \mid B)P(B) + P(A \mid B')P(B')\) | Law of total probability: the overall probability of \(A\), summing the routes through \(B\) and \(B'\). |
| \(P(B \mid A) = \dfrac{P(A \mid B)P(B)}{P(A)}\) | Bayes' theorem, with the denominator found by the law of total probability. |
| Partition \(B_1, B_2, B_3\) | Events that do not overlap and cover all possibilities. Their probabilities add to 1. |
| Prior \(P(B)\) | Probability of the cause before the evidence is known. |
| \(P(B \mid A)\) | Probability of the cause after the evidence \(A\) has been observed. |
Worked examples
Example 1 A condition affects 2% of a population. A screening test is positive for 95% of people who have the condition, and positive for 4% of people who do not. A person tests positive. Find the probability they have the condition.
Let \(D\) be "has the condition" and \(T\) be "tests positive". \(P(D) = 0.02\), \(P(D') = 0.98\), \(P(T \mid D) = 0.95\), \(P(T \mid D') = 0.04\).
Branch products: \(P(D \cap T) = 0.02 \times 0.95 = 0.019\) and \(P(D' \cap T) = 0.98 \times 0.04 = 0.0392\).
Total probability of a positive test: \(P(T) = 0.019 + 0.0392 = 0.0582\).
\(P(D \mid T) = \dfrac{0.019}{0.0582} = 0.3264\dots\).
Answer: about 0.326. Most positive results come from people without the condition, because the condition is rare.
Example 2 A factory has three machines. Machine A makes 50% of the items, machine B makes 30% and machine C makes 20%. The proportions of defective items are 2% for A, 3% for B and 5% for C. An item is chosen at random and is defective. Find the probability it came from machine C.
The machines form a partition. Let \(F\) be "defective".
Branch products: \(A\): \(0.5 \times 0.02 = 0.010\). \(B\): \(0.3 \times 0.03 = 0.009\). \(C\): \(0.2 \times 0.05 = 0.010\).
\(P(F) = 0.010 + 0.009 + 0.010 = 0.029\).
\(P(C \mid F) = \dfrac{0.010}{0.029} = \dfrac{10}{29} = 0.3448\dots\).
Answer: \(\dfrac{10}{29} \approx 0.345\).
Example 3 An email filter finds that 30% of incoming messages are spam. The word "free" appears in 80% of spam messages and in 5% of other messages. A message contains the word "free". Find the probability it is spam.
Let \(S\) be "spam" and \(W\) be "contains 'free'". \(P(S) = 0.3\), \(P(W \mid S) = 0.8\), \(P(W \mid S') = 0.05\).
\(P(W) = 0.3 \times 0.8 + 0.7 \times 0.05 = 0.24 + 0.035 = 0.275\).
\(P(S \mid W) = \dfrac{0.24}{0.275} = 0.8727\dots\).
Answer: about 0.873.
Common errors
- Common error: giving \(P(A \mid B)\) as the answer when the question asks for \(P(B \mid A)\). Check which event is "given" in the question.
- Common error: using only the branch of interest as the denominator. The denominator must include every branch that leads to the observed evidence.
- Common error: ignoring the starting probability \(P(B)\) and assuming a 95% accurate test gives a 95% chance of the condition.
- Common error: using a partition whose probabilities do not add to 1, or whose events overlap.
- Common error: placing the evidence at the first stage of the tree and the cause at the second without adjusting the given probabilities to match.
Exam tip: Define each event in words at the start, draw the tree, and put the branch probabilities on it. The tree both organises the working and shows where the denominator comes from.
Check yourself
- \(P(B) = 0.4\), \(P(A \mid B) = 0.5\), \(P(A \mid B') = 0.2\). Find \(P(A)\) and \(P(B \mid A)\).
Answer
\(P(A) = 0.4(0.5) + 0.6(0.2) = 0.2 + 0.12 = 0.32\). \(P(B \mid A) = \dfrac{0.2}{0.32} = 0.625\). - A box is chosen at random with equal probability from box X (3 red, 7 blue) and box Y (6 red, 4 blue). A ball is drawn and is red. Find the probability it came from box X.
Answer
\(P(\text{red}) = 0.5(0.3) + 0.5(0.6) = 0.45\). \(P(X \mid \text{red}) = \dfrac{0.15}{0.45} = \dfrac{1}{3}\). - The probability of rain is 0.25. A forecast says "rain" for 90% of rainy days and for 20% of dry days. Given a forecast of rain, find the probability it rains.
Answer
\(P(\text{forecast}) = 0.25(0.9) + 0.75(0.2) = 0.225 + 0.15 = 0.375\). \(P(\text{rain} \mid \text{forecast}) = \dfrac{0.225}{0.375} = 0.6\). - In Example 1, the condition affects 10% of a different group instead. Find \(P(D \mid T)\) for that group.
Answer
\(P(T) = 0.1(0.95) + 0.9(0.04) = 0.095 + 0.036 = 0.131\). \(P(D \mid T) = \dfrac{0.095}{0.131} = 0.725\). A higher starting probability gives a much higher probability after a positive test.