### in search of no information

The main purpose of this blog post is to illustrate my ignorance of Bayesian statistics(*), discussing a very simple game with a 2-card deck. By the way, we only consider black and red cards, no jokers etc.

So we begin with the 2-card deck and draw one card - a black card. The question is 'what is the probability that the other card is also black?'.

Fortunately we only need to consider two different hypotheses:

H1 ... both cards are black.

H2 ... a mixed deck (one black, one red).

We update our probabilities using the famous formula:

p(Hi|b) = p(b|Hi) * p(Hi) / p(b)

where b indicates the 'black card event' and p(b) is short hand for the sum p(b|H1)*p(H1) + p(b|H2)*p(H2).

Since we have no further information we use an uninformed prior which does not prefer one hypothesis over the other,
in other words:
p(H1) = p(H2)

and using p(b|H1) = 1 and p(b|H2) = 1/2 we get

p(H1|b) = 2/3 and p(H2|b) = 1/3. (I recommend that you actually plug in the numbers and do the calculation.)

I admit that there is something weird about this result: We have two cards, we pick one and after updating our probabilities have to conclude that the other is more likely being black than red, actually twice as likely.

The issue seems to be our prior, i.e. the choice of p(Hi).

Indeed, if we draw the 2-card deck randomly then the probabilities that it contains bb, br, rb and rr should be the same. We can throw out rr, which leaves us with a 2:1 majority of mixed decks and should therefore use p(H1) = 1/3 and p(H2) = 2/3. As you can check this resolves the weirdness, we get p(H1|b) = p(H2|b) and thus the probability that the 2nd card is also black would be the same as the probability for red.

But can a Bayesian accept such card counting?

And it gets worse: If the 2-card deck was drawn from an initial deck of 2N cards then the probabilities of bb and br are not exactly the same. The probability of the first b is indeed 1/2 but the probability of a 2nd b is lower, (N-1)/(2N-1), and therefore a mixed deck seems actually preferred, depending on the number N, which we don't know. Should we really conclude that red is slightly more likely after seeing black?
Do we have to sum over all possible N with equal weight to get a truly uninformed prior?

But I am sure a true Bayesian would reject all those card counting arguments which smell quite a bit of frequentist reasoning. A truly uninformed prior means that we have no information to prefer one hypothesis over the other. There is a difference between not knowing anything about the 2-card deck and knowing that it was randomly selected. Therefore the symmetric choice p(H1) = p(H2) is the true uninformed prior which properly reflects our indifference and we have to live with the asymmetry of probabilities for the 2nd card.

(*) A true Bayesian would calculate the probability that this post contains sarcasm taking into account the existence of this footnote.

Jonathan Livengood said...

I'm not quite sure what we're being asked to imagine here. Is it that we know that at least one of two cards is black or is it rather that we know that the first of two cards is black?

In the first case, it seems that we can eliminate from consideration only the pair <r, r> from consideration, and the probability of both black should be 1/3. But in the second case, we get to eliminate both <r, r> and <r, b>, so the probability of <b, b> is 1/2.

In any event, I'm not getting that the probability of both black is greater than the probability of exactly one black, given that at least one is black.

Anyway, it looks like you are raising the following interesting challenge to objective Bayesianism. The objective Bayesian says that we should be indifferent with respect to states of the world -- in this case, ordered pairs of card colors. But you seem to be suggesting that we might be indifferent to processes for generating states of affairs instead. Is that your point or part of it?

wolfgang said...

>> I'm not quite sure what we're being asked to imagine here.

I was a bit more precise in the follow-up post. You have a deck of 2 cards. You take the top card, it turns out to be black. What is the probability that the remaining card is also black?

>> so the probability of is 1/2
this is indeed so if the deck was randomly selected (as I explain). But you don't know that.

>> I'm not getting that the probability of both black is greater than the probability of exactly one black

plug in the numbers.

>> objective Bayesian says that we should be indifferent with respect to states of the world

a Bayesian should be indifferent to all relevant hypotheses when selecting an uninformed prior.

wolfgang said...

Let me explain a bit better why "knowing that randomly selected" is different from the example here.

Assume you have a deck consisting of many cards (not just 2) and you have seen the first 99 and they were all black. What is the probability that the next card is black too?

Well, if you *know* that the deck was randomly selected then the probability for black is still only 1/2

However, if you do not know that then you might conclude that the probability to get yet another black card is substantially more than 1/2.

(But see the follow-up post to this one.)

Jonathan Livengood said...

Wolfgang,

I think you're making a mistake somewhere ... my guess is that you are just swapping around black and red in a late step. (Nothing crucial hangs on it, but it's bugging me.)

If I plug in the numbers, I get P(both black | at least one black) = 1/3, not 2/3.* So, it isn't the case that the probability of the other card being black is twice as large as the probability that it is red. Rather the reverse: the probability that the other card is red (given that we've seen one that is black) is 2/3.

* Here's the calculation:

P(both black) = 1/4
P(~ both black) = 3/4
P(at least one black | both black) = 1
P(at least one black | ~ both black) = 2/3

P(both black | at least one black) = (1/4)*1 / [(1/4)*1 + (3/4)*(2/3)] = (1/4) / (3/4) = 1/3.

wolfgang said...

>> I think you're making a mistake somewhere

yes 8-)

(read the follow-up post, the correct result is completely different 8-)

>> I get P(both black | at least one black) = 1/3

so you think seeing one black makes it less likely that the next is also black ?

how about seeing the sun rising? does this make it more or less likely to see it rise again tomorrow?

wolfgang said...

Jonathan,

you use: P(both black) = 1/4
P(~ both black) = 3/4

so you use a prior which is not uniform in the hypotheses:
the difference is that i used
p(both black) = p(~ both black) = 1/2
which i consider (in this post!) as the uninformed prior.

as i said, frequentist card counting will get you into trouble.

Jonathan Livengood said...

Okay, I see what I was doing wrong now ... well, not wrong exactly. I was assuming that from the start, the Bayesian assigns equal probability to the four possible states of the deck (read from top card to bottom card): black-black, red-black, black-red, and red-red.

That is, I was assigning a uniform prior over the *four* possible states of the universe, not over your two hypotheses. I don't think this is card counting -- it is much worse than that, it's possible world counting!

Anyway, given a uniform prior over the four possible worlds, seeing a black card increases the probability of both being black from 1/4 to 1/3, and it decreases the probability of not-both black from 3/4 to 2/3, which is what I was saying: after seeing a black card, the probability of the other card being black is half the probability of the other card being red.

So I'm not being quite as dumb as it might have looked. ;) Though I will fess up to misreading what you wrote.

Still, given the way you're treating the probabilities later on, considering only H1 and H2 is fishy, right? And here I'll just refer back to my first comment -- it isn't clear whether you mean for us to be considering a problem in which we know that at least one card is black or a problem in which we know that the first card is black. The two conditions are not equivalent.

Maybe you say as much in the follow up ... I'll have to go back and read it again.

wolfgang said...

I certainly don't think your comment was or looked dumb.
Actually, I think it is kind of brave of you to walk into the jungle of truly absolutely really uninformed priors 8-)