Bayes' Theorem

While conditional probability helps us find the probability of a future event given a past event (e.g., what is the probability of drawing a second Ace given we drew a first Ace?), Bayes' Theorem allows us to reverse the process.

Named after the Reverend Thomas Bayes, this theorem provides a mathematical way to update our beliefs about a past event or a hypothesis, given new evidence that has just occurred. It is widely used in machine learning, medical testing, and spam filtering.

1. The Core Concept

Imagine a scenario: You wake up, look out the window, and see that the grass is wet. * Hypothesis A: It rained last night. * Hypothesis B: The sprinklers were on.

Both hypotheses can cause the grass to be wet (the evidence). Bayes' Theorem allows you to calculate the probability that it rained, given that the grass is wet, by combining: 1. How likely it is to rain in general. 2. How likely the grass is to be wet if it rains. 3. How likely the grass is to be wet in general (from all possible causes).

2. The Total Probability Rule

Before we can use Bayes' Theorem, we need to understand the Law of Total Probability. This rule helps us find the overall probability of an event ($A$) that can occur under several different, mutually exclusive scenarios ($E_1, E_2, \dots, E_n$).

If events $E_1, E_2, \dots, E_n$ form a partition of the sample space (meaning they are mutually exclusive and exhaustive), then for any event $A$:

$$ P(A) = P(E_1)P(A|E_1) + P(E_2)P(A|E_2) + \dots + P(E_n)P(A|E_n) $$

• In English: The total probability of $A$ happening is the sum of the probabilities of $A$ happening via every possible "path" or scenario.

3. Bayes' Theorem Formula

Bayes' Theorem is essentially the conditional probability formula expanded using the General Multiplication Rule and the Law of Total Probability.

If we want to find the probability of a specific cause $E_i$ given that the effect $A$ has occurred, the formula is:

$$ P(E_i | A) = \frac{P(E_i \cap A)}{P(A)} = \frac{P(E_i) \times P(A|E_i)}{P(A)} $$

If we expand the denominator using the Law of Total Probability for two mutually exclusive causes ($E_1$ and $E_2$), it looks like this:

$$ P(E_1 | A) = \frac{P(E_1) \times P(A|E_1)}{P(E_1)P(A|E_1) + P(E_2)P(A|E_2)} $$

Terminology:

• $P(E_1)$ (Prior Probability): Our initial belief that cause $E_1$ is true, before seeing the new evidence $A$.
• $P(A|E_1)$ (Likelihood): The probability of seeing the evidence $A$, assuming the cause $E_1$ is true.
• $P(A)$ (Marginal Likelihood): The total probability of seeing the evidence $A$ under all possible causes.
• $P(E_1 | A)$ (Posterior Probability): Our updated, revised belief that $E_1$ is true, after observing the evidence $A$.

4. Classic Example: Medical Testing

Medical testing is the most famous application of Bayes' Theorem because tests are never 100% accurate.

The Scenario: A rare disease affects 1% of the population. A test for the disease is 99% accurate (if you have the disease, it tests positive 99% of the time; if you don't have it, it tests negative 99% of the time). You take the test and it comes back positive. What is the actual probability that you have the disease?

Many people intuitively guess 99%. Let's use Bayes' Theorem.

Defining the variables: * $D$: You have the disease. (Prior $P(D) = 0.01$) * $D'$: You don't have the disease. (Prior $P(D') = 0.99$) * $+$: The test is positive. * We want to find $P(D | +)$.

The Likelihoods: * $P(+ | D) = 0.99$ (True positive rate) * $P(+ | D') = 0.01$ (False positive rate: 100% - 99% accuracy)

Applying the Formula: $$ P(D | +) = \frac{P(D) \times P(+ | D)}{P(D)P(+ | D) + P(D')P(+ | D')} $$

$$ P(D | +) = \frac{(0.01)(0.99)}{(0.01)(0.99) + (0.99)(0.01)} $$

$$ P(D | +) = \frac{0.0099}{0.0099 + 0.0099} = \frac{0.0099}{0.0198} = 0.50 $$

The Result: Even though the test is 99% accurate, because the disease is so rare, a positive test only means you have a 50% chance of actually having the disease! Most positive results will actually be false positives from the healthy 99% of the population.

5. Tree Diagrams

When solving Bayes' Theorem problems, memorizing the complex formula is often less effective than drawing a probability tree diagram.

1. First level of branches: Represent the initial, mutually exclusive causes or scenarios (e.g., Disease vs. No Disease). Write their prior probabilities on the branches.
2. Second level of branches: Represent the evidence or outcomes given the first branch (e.g., Positive Test vs. Negative Test). Write the conditional probabilities (likelihoods) here.
3. End nodes: Multiply along the branches to find the joint probabilities ($P(E_1 \cap A)$).
4. Solving: To find $P(E_1 | A)$, take the specific branch ending in $A$ that you care about, and divide it by the sum of ALL branches that end in $A$.