It is good to see an article on Bayesian reasoning with conditional probabilities in the current issue of the Times Literary Supplement: “Thomas Bayes and the crisis in science” by David Papineau (June 28, 2018).
As Professor Papineau points out, Bayesian analysis is used in many fields, including law.
One of the difficulties in discussing Bayesian reasoning, or indeed any complex subject, is that clear and simple points can become obscured by technical terms.
It took me a while to get to grips with Professor Papineau’s coin-tossing illustration. What it is designed to illustrate is an error of reasoning that is, apparently, found in too many published scientific studies. Essentially, the error involves drawing a conclusion from too little information. But Professor Papineau's illustration is of one test, not of an experiment involving a statistically significant number of test results.
If you take a coin – any coin – and toss it five times, and if you get five heads, how likely is it that the coin is biased? Pretend that you do not have special coin-tossing skills that allow you to determine the result of a toss. Also pretend that it doesn't occur to you to just keep tossing the coin to see what proportion of the sequences of five-tosses give results of five-heads.
After only a little reflection you realise that an unbiased coin will, on average, produce five-heads once every 32 times the five-toss sequence is carried out. One in 32 gives a probability of 0.03, approximately. The probability of getting five-heads from an unbiased coin looks very low, and you might be tempted to conclude that, therefore, there is a probability of 0.97 that the coin is biased.
Apparently, a significant number of scientific studies have been published in peer-reviewed journals, reporting conclusions arrived at through that sort of reasoning.
Bayesian analysis, if you are able to do it, will quickly show you that such conclusions are ridiculous, or, as Professor Papineau says, “silly” or “nonsense on stilts”.
If you are a lawyer, you might have to convince a judge or jury that an apparently obvious conclusion, reached by a respected expert, is wrong. It is far from easy to do this, and that may be why Bayesian analysis is taking so long to be routinely applied in courtrooms.
Fundamentally, the probability of getting five-heads if the coin is not biased, is not the same as the probability of the coin not being biased if it produced five-heads. The probability of A, given B, is not the same as the probability of B, given A.
My favourite way of illustrating this is to say: the probability of an animal having four legs, given that it is a sheep, is not the same as the probability of it being a sheep, given that it has four legs. The first tells you something about sheep, the second something about quadrupeds.
We know something about an unbiased coin: about three per cent of the times it is tossed five times it will produce a sequence of five-heads. But what do we know about a coin that has produced a five-head sequence? Is it biased or unbiased? If it is biased, how biased is it? Does it always produce five-heads or only some proportion of the times it is tossed five times? Is a biased coin commonly found or is it rare? Those things need to be known in calculating the probability that the tossed coin which produces a five-head sequence is biased.
At the risk of over-explaining this, let’s ignore - just for a moment - the rarity of biased coins and consider possible results of 100 five-toss sequences for a biased, and an unbiased, coin:
At the risk of over-explaining this, let’s ignore - just for a moment - the rarity of biased coins and consider possible results of 100 five-toss sequences for a biased, and an unbiased, coin:
Biased Unbiased
Five-heads 25 3
Other 75 97
This is not to say that the result is of no use. It does tend to prove the coin is biased. The strength of its tendency to prove bias is the likelihood ratio: the ratio of the probability of five-heads, given the coin is biased (from the above table this is 0.25) to the probability of five-heads, given the coin is unbiased (0.03), a ratio of 8.3 to 1. On the issue of bias, the result should be reported as: whatever the other evidence of bias may be, this result is 8.3 times more likely if the coin is biased than if it is not biased. The other evidence may be from a survey of coins which measured how often we can expect to find biased coins.
Now suppose that such a biased coin is only found once in every ten thousand coins, and that all biased coins have the same bias. The probability of a randomly chosen coin you have tossed and got a five-heads result being biased is, when you do the calculation using Bayes' formula, 0.0008. Eight occurrences in ten thousand. Much lower than the 0.97 probability (97 occurrences in 100) of the coin being biased that might have been reported in a peer-reviewed journal.
Again, this is not as surprising as it may seem at first glance. There may be only one biased coin in 10,000 coins, and one occurrence of five-heads from a biased coin in 40,000 coins (using the one-in-four frequency in the table), but, in round figures, there will also be 1200 occurrences (three per cent) of five-heads from unbiased coins in those 40,000 coins. This is why, on this occurrence of biased coins, a five-head result is much more likely (1200 times more likely) to be from an unbiased coin than from a biased one.
Again, this is not as surprising as it may seem at first glance. There may be only one biased coin in 10,000 coins, and one occurrence of five-heads from a biased coin in 40,000 coins (using the one-in-four frequency in the table), but, in round figures, there will also be 1200 occurrences (three per cent) of five-heads from unbiased coins in those 40,000 coins. This is why, on this occurrence of biased coins, a five-head result is much more likely (1200 times more likely) to be from an unbiased coin than from a biased one.
Only a very brave judge or juror would bet a significant sum that a coin which when tossed produced a five-head sequence was not biased. The bets would go the other way and those significant sums would most probably be lost.
And, as an afterthought: if you feel estimating prior probabilities is a bit haphazard, the Bayesian formula can be turned around to tell you what priors you would need in order to get in the above example P(the coin is biased) = 0.95. You would, before doing the experiment, need to be convinced to a probability of about 0.70 that the coin was biased. This sort of approach is discussed in a paper by David Colquhoun (available courtesy of The Royal Society Publishing). If, as a lawyer, you want an easy introduction to Bayesian reasoning, see my draft paper on propensity evidence.
And, as an afterthought: if you feel estimating prior probabilities is a bit haphazard, the Bayesian formula can be turned around to tell you what priors you would need in order to get in the above example P(the coin is biased) = 0.95. You would, before doing the experiment, need to be convinced to a probability of about 0.70 that the coin was biased. This sort of approach is discussed in a paper by David Colquhoun (available courtesy of The Royal Society Publishing). If, as a lawyer, you want an easy introduction to Bayesian reasoning, see my draft paper on propensity evidence.
