I think using Bayesian methods is a good thing. The reservation I have about this post is that it's motivated as A/B testing, which in classical frequentist testing usually equates to a null hypothesis that some parameter is equal in two populations (in this example, conversion rate under method A is equal to conversion rate under method B). The rest of the post then describes a test of a single population against a known value (is the conversion rate under method B = 5%).
These are not the same problems at all, and it's not clear how the authors propose to extend the tests they're informally describing to the test of equality of a parameter in two populations. It is possible in a Bayesian framework, but it's not this simple.
These are not the same problems at all, and it's not clear how the authors propose to extend the tests they're informally describing to the test of equality of a parameter in two populations. It is possible in a Bayesian framework, but it's not this simple.