One problem with a p-value is that it is hard to interpret if you don’t know the sample size. Andy Gelman talks about this problem on his blog.
Here I’ll contrast the message communicated by a p-value and a Bayesian test by looking at the following three tables:
Table A: [32 18; 18 32]
Table B: [272 228; 228,272]
Table C: [2570 2430; 2430,2570]
Each of the tables has a Pearson chi-square value in the (7.4, 7.8) range and so each has (approximately) the same p-value. But, following Gelman’s reasoning, the small p-value for the large sample size in Table C doesn’t have much significance and and the same p-value for Table A (small sample size) is more impressive.
Here is a reasonable Bayesian test of the equality of two proportions. First, we define a class of priors indexed by the parameter K:
is equivalent to testing H:
(I’ll talk more about this in class.)
Chapter 8 in BCWR gives an expression for the Bayes factor in support of
. In the LearnBayes package, suppose the data is a matrix of two columns where the first column contains the counts of successes of the two samples and the second column contains the sample sizes. Then the Bayes factor (on the log scale) is computed by the single line
Let’s illustrate using this for the three tables:
So the Bayes factor against equality of proportions is exp(.94) = 2.56 for Table A, exp(-0.30) = 0.74 for Table B, and exp(-1.41) = 0.24 for Table C. In contrast to the message communicated by a p-value, the Bayes factor calculation indicates that Table A is the most significant table.