in search of no information

The main purpose of this blog post is to illustrate my ignorance of Bayesian statistics(*), discussing a very simple game with a 2-card deck. By the way, we only consider black and red cards, no jokers etc.



So we begin with the 2-card deck and draw one card - a black card. The question is 'what is the probability that the other card is also black?'.

Fortunately we only need to consider two different hypotheses:

H1 ... both cards are black.

H2 ... a mixed deck (one black, one red).



We update our probabilities using the famous formula:

p(Hi|b) = p(b|Hi) * p(Hi) / p(b)

where b indicates the 'black card event' and p(b) is short hand for the sum p(b|H1)*p(H1) + p(b|H2)*p(H2).

Since we have no further information we use an uninformed prior which does not prefer one hypothesis over the other,
in other words:
p(H1) = p(H2)

and using p(b|H1) = 1 and p(b|H2) = 1/2 we get

p(H1|b) = 2/3 and p(H2|b) = 1/3. (I recommend that you actually plug in the numbers and do the calculation.)



I admit that there is something weird about this result: We have two cards, we pick one and after updating our probabilities have to conclude that the other is more likely being black than red, actually twice as likely.



The issue seems to be our prior, i.e. the choice of p(Hi).

Indeed, if we draw the 2-card deck randomly then the probabilities that it contains bb, br, rb and rr should be the same. We can throw out rr, which leaves us with a 2:1 majority of mixed decks and should therefore use p(H1) = 1/3 and p(H2) = 2/3. As you can check this resolves the weirdness, we get p(H1|b) = p(H2|b) and thus the probability that the 2nd card is also black would be the same as the probability for red.

But can a Bayesian accept such card counting?



And it gets worse: If the 2-card deck was drawn from an initial deck of 2N cards then the probabilities of bb and br are not exactly the same. The probability of the first b is indeed 1/2 but the probability of a 2nd b is lower, (N-1)/(2N-1), and therefore a mixed deck seems actually preferred, depending on the number N, which we don't know. Should we really conclude that red is slightly more likely after seeing black?
Do we have to sum over all possible N with equal weight to get a truly uninformed prior?



But I am sure a true Bayesian would reject all those card counting arguments which smell quite a bit of frequentist reasoning. A truly uninformed prior means that we have no information to prefer one hypothesis over the other. There is a difference between not knowing anything about the 2-card deck and knowing that it was randomly selected. Therefore the symmetric choice p(H1) = p(H2) is the true uninformed prior which properly reflects our indifference and we have to live with the asymmetry of probabilities for the 2nd card.





(*) A true Bayesian would calculate the probability that this post contains sarcasm taking into account the existence of this footnote.