Oct 14, 2022

Confidence Does Not Mean What You Think

statistics bayesian

We love intervals because they feel like a safety net, we must be very sure of what we are doing if it’s 95% interval, right?. The issue is that not all 95% intervals are the same. A frequentist confidence interval and a Bayesian credible interval both sound similar but they don’t make the same claims. This distinction is something that directly affects how you communicate uncertainty to stakeholders and how you make decisions based on your data.

A concrete example

Let’s ground this in something real. Suppose we shipped a new signup page and out of $n = 200$ visitors, $x = 26$ signed up. We want an interval for the conversion rate $p$ . Now, depending on whether you take a frequentist or Bayesian approach, that “95% interval” means something quite different.

The frequentist framing gives you a confidence interval: “If I were to repeat this sampling process over and over, 95% of the intervals I compute would contain the true ( $p$ ).” Notice that the 95% refers to the procedure, not the one interval you computed. It’s about what would happen in infinite parallel universes where you keep rerunning the experiment.

The Bayesian framing gives you a credible interval: “Given the data I saw and my prior beliefs, there’s a 95% probability that ( $p$ ) lies in this interval.” Here, the 95% refers directly to your belief about $p$ after seeing the data. It’s a statement about the parameter itself, conditional on the world you actually observed.

With our example of $x = 26$ successes and $n = 200$ trials (giving $\hat p = 0.13$ ), we can compute both types of intervals. The Wilson 95% CI gives us $[0.090, 0.184]$ , the Clopper-Pearson 95% CI gives us $[0.087, 0.185]$ , and a 95% credible interval using a uniform $\beta(1,1)$ prior (which becomes a $\beta(27,175)$ posterior) also gives us $[0.090, 0.184]$ . Visually, they’re nearly identical here. Philosophically, they couldn’t be more different.

Why this matters in practice

The most important issue is interpretation. When you present results to stakeholders, they almost always want the Bayesian sentence. They want to know the probability that the parameter lies in some range, given the data they’re looking at. If you’re using a confidence interval but accidentally describe it using Bayesian language, you’re making a subtle but important error that can lead to poor decisions.

The differences become more pronounced in certain scenarios. With small samples or rare events, the two types of intervals can diverge noticeably, especially when you use informative priors in the Bayesian case. The prior regularizes your estimates, which can be helpful when you don’t have much data. In the frequentist framework, you’re stuck with whatever the procedure gives you, even if it produces nonsensical or extremely wide intervals.

Another critical difference emerges when you’re doing sequential looks at your data (also called “peeking”). Confidence intervals have coverage guarantees only under a fixed sampling plan. If you peek at your data multiple times and recompute confidence intervals, you’re eroding the nominal coverage unless you make adjustments like alpha-spending or use group sequential methods. Bayesian credible intervals, on the other hand, are naturally sequential in that you can update them as new data arrives without breaking their interpretation. Your posterior today becomes your prior tomorrow. That said, the quality of your priors and the fit of your model still matter immensely.

Finally, Bayesian analysis handles decision-centric questions as a first-class citizen. If you need to know the probability that the lift is greater than zero, or that churn is below some target, the posterior gives you that answer directly. With a confidence interval, you’d need to perform additional gymnastics or appeal to p-values, which introduces its own interpretation pitfalls.

A quick conceptual glossary

Before diving deeper, it’s worth pinning down a few terms. The parameter $p$ is the unknown constant we’re trying to estimate, like the true conversion rate. A confidence interval (CI) is a random interval whose coverage is about the method across many repeats. A credible interval (CrI) is a fixed interval conditional on the observed data, where the probability statement is about the parameter $p$ itself given your data and prior beliefs. The prior represents your beliefs about $p$ before seeing the data, and the posterior is your updated beliefs after incorporating the evidence.

The fundamental claims in plain language

Let’s slow down and articulate exactly what each type of interval is claiming.

A 95% confidence interval is the output of a recipe. If we could rewind the world and re-sample under the same conditions, running the same recipe each time, then 95% of those intervals would cover the truth. Today’s interval either covers $p$ or it doesn’t, there is no probability attached to this particular interval containing $p$ . The 95% lives in the long run, across many hypothetical repetitions. It’s a statement about the reliability of your method, not about your current estimate.

A 95% credible interval, on the other hand, takes your prior beliefs and the data you observed and returns an interval such that the posterior probability that $p$ lies inside is 95%. The 95% lives in the current state of knowledge. This is what stakeholders usually think “95% confidence” means, and it’s why the terminology can be so confusing.

When do they match?

In symmetric, well-behaved problems (think Normal likelihoods with weak priors), and with large samples, confidence intervals and credible intervals often end up numerically very close. For proportions specifically, methods like Wilson or Agresti-Coull confidence intervals and a Beta posterior with a weak prior can look remarkably similar when $n$ is large. The mathematics conspire to give you nearly the same numbers, even though the interpretations remain distinct.

Where they diverge

The cracks show up in a few key scenarios. With small samples or rare events, confidence interval methods can be conservative or produce asymmetric intervals that feel unintuitive, while Bayesian priors can stabilize estimates and pull you toward more reasonable values. When you’re peeking at data or doing optional stopping, confidence interval coverage guarantees fall apart because they assume a fixed sampling plan. Reusing the same data stream and recomputing CIs mid-experiment erodes your nominal coverage. Bayesian posteriors remain coherent under sequential looks, your posterior today becomes your prior tomorrow, though of course the quality of your priors and model fit still matter greatly.

Finally, in decision-centric workflows, the Bayesian approach shines. If your question is “What’s the probability the lift is greater than zero?” or “What’s the chance we’re below our churn target?”, the posterior gives you that answer directly.

How to communicate this to stakeholders

When you present results, language matters. If you used a confidence interval, you might say: “Using a method that’s calibrated to cover the true rate 95% of the time across repeated studies, our interval is […]”. Not ideal, but this keeps you honest about what the procedure guarantees.

If you used a credible interval, you can say: “Given the data and our prior assumptions, there’s a 95% probability the true rate lies between […]”. This is usually what people want to hear, and it’s a legitimate claim when you’re doing Bayesian analysis.

When someone inevitably asks, “So are we 95% sure?” and you’ve computed a confidence interval, you need to gently correct: “Not exactly. That 95% describes the long-run reliability of the method, not our probability today. If we want a probability statement about the parameter itself, we can compute a Bayesian credible interval instead.”

Closing thought

Confidence intervals talk about what data would do if we re-ran the world. Credible intervals talk about what we believe given the world we saw. Both are honest, they’re just honest about different things. The trick is remembering which honesty your decision actually needs.