Background

What is a binomial confidence interval?

The binomial distribution represents the likelihood of observing \(k\) successes in \(n\) trials where the probability of success for each trial is \(p\). One often does not know the true value of \(p\) and wishes to estimate this value. After observing \(k\) successes in \(n\) trials with unknown probability of success \(p\), a confidence interval (CI) is constructed in such a way that it contains the true value of \(p\) with some high probability.

In constructing confidence intervals one has to tradeoff between three quantities:

1. Confidence: The probability that the CI contains the true parameter. Often written as \(1-\alpha\) where \(\alpha\) is small.

2. Volume: The length of the CI. If the CI is constructed as a lower bound on \(p\) (i.e. \([\underline{p}, 1]\)), then we care about the length of the CI which is below \(p\). This is known as the shortage.

3. Number of samples: In general, with more samples one can construct CIs with higher confidence and smaller volume.

Why does this package exist?

Existing implementations of binomial CIs do not optimally control the tradeoffs between confidence, volume, and number of samples. In practice, this means that if a user specifies \(k\) and \(n\), existing implementations return CIs with higher/lower confidence than desired and/or with higher volume than necessary. CIs which optimally trade-off between these quantities have the property of being uniformly most accurate (UMA) or uniformly most accurate unbiased (UMAU). Methods for constructing UMA and UMAU binomial confidence intervals has existed since the mid-20th century, but until now have not been implemented in an open-source software package. We implement UMA and UMAU binomial confidence intervals using methods from

• On the treatment of discontinuous random variables by Eudey,

• Testing Statistical Hypotheses by Lehmann and Romano,

• Table of Neyman-shortest unbiased confidence intervals for the binomial parameter by Blyth and Hutchinson.

Existing software implementations for binomial CIs include:

• statsmodels.stats.proportion.proportion_confint (Python)

• scipy.stats._result_classes.BinomTestResult.proportion_ci (Python)

• astropy.stats.binom_conf_interval (Python)

• scipy.stats.binom (Python)

• EBCIC (Python)

• binom.test (R)

• binom.confint (R)

• BinomCI (R)

• HypothesisTests.jl (Julia)

• RobustStats.jl (Julia)

• ClinicalTrialUtilities.jl (Julia)

• binofit (Matlab)

The methods these packages implement include:

• Wald/Normal [1]

• Agresti-Coull [1]

• Clopper-Pearson (*) [1]

• Wilson [1]

• Modified Wilson [1]

• Wilson with continuity correction (*) [1]

• Jeffreys [1]

• Bayesian uniform prior

• Inverting the binomial test (*) [3]

• Arcsine [1]

• Logit [1]

• Probit [8]

• Complementary log [8]

• Likelihood (Profile) [1]

• Witting (*) [4]

• Pratt [5]

• Mid-p [7]

• Blaker (*) [6]

• Second-order corrected [2]

The reference of each method points to a survey paper (if possible) rather than the original derivation of the method.

Only the methods marked with (*) are guaranteed to provide at least as much confidence as desired. However, none of these methods provide exactly the confidence level desired (Witting might, but the listed reference is not freely available online and is only published in German).

It is also worth noting the ump R package. This package implements UMP and UMPU hypothesis tests for the binomial distribution. Such tests could be leveraged to construct UMA and UMAU confidence intervals, but this doesn’t appear to be implemented based on the documented functions. In addition, the authors of this package note that their implemetation has issues with numerical stability.

Existing software implementations for computing binomial CIs with exact confidence levels are then either non-existent or unsatisfactory. Unsurprisingly then, there is also no open-source implementation for computing the expected shortage (or expected excess or expected width) of binomial CIs, and no implementation for computing the worst-case values of these quantities. This package exists to fill this gap.

What exactly does this package do?

This package constructs optimal confidence intervals for the probability of success parameter of a binomial distribution.

Lower Bounds

Given user specified miscoverage rate (\(\alpha\)) and maximum expected shortage (\(\text{MES}\)), return a lower bound on \(p\) that satisfies the following requirements:

1. achieves exact desired coverage: \(\mathbb{P}[\underline{p} \le p] = 1-\alpha\),

2. \([\underline{p}, 1]\) is uniformly most accurate,

3. achieves exact desired maximum expected shortage: \(\max_p \ \mathbb{E}_p[\max (p - \underline{p}, 0)] = \text{MES}\),

4. uses the minimum number of samples \(n\) to achieve requirements 1,2,3.

Upper Bounds

Given user specified miscoverage rate (\(\alpha\)) and maximum expected excess (\(\text{MEE}\)), return an upper bound on \(p\) that satisfies the following requirements:

1. achieves exact desired coverage: \(\mathbb{P}[p \le \overline{p}] = 1-\alpha\),

2. \([0, \overline{p}]\) is uniformly most accurate,

3. achieves exact desired maximum expected excess: \(\max_p \ \mathbb{E}_p[\max (\overline{p} - p, 0)] = \text{MEE}\),

4. uses the minimum number of samples \(n\) to achieve requirements 1,2,3.

Simultaneous Lower and Upper Bounds

Given the user specified miscoverage rate (\(\alpha\)) and maximum expected width (\(\text{MEW}\)), return simultaneous lower and upper bounds on \(p\) that satisfy the following requirements:

1. achieves exact desired coverage: \(\mathbb{P}[\underline{p} \le p \le \overline{p}] = 1-\alpha\),

2. \([\underline{p}, \overline{p}]\) is uniformly most accurate unbiased,

3. achieves exact desired maximum expected width: \(\max_p \ \mathbb{E}_p[\overline{p} - \underline{p}] = \text{MEW}\),

4. uses the minimum number of samples \(n\) to achieve requirements 1,2,3.

Relevant Literature

For more information on the mathematics behind these confidence intervals see our affiliated paper How Generalizable Is My Behavior Cloning Policy? A Statistical Approach to Trustworthy Performance Evaluation

Below are some of the resources that we found most useful for understanding binomial confidence intervals.

• Testing Statistical Hypotheses by Lehmann and Romano

• On the treatment of discontinuous random variables by Eudey

• Table of Neyman-shortest unbiased confidence intervals for the binomial parameter by Blyth and Hutchinson

• Length of Confidence Intervals by Pratt

• More on length of confidence intervals by Madansky

• Binomial confidence intervals by Blyth and Still

• Smallest confidence intervals for one binomial proportion by Wang

• Fuzzy and randomized confidence intervals and p-values by Geyer and Meeden

• Nonoptimality of Randomized Confidence Sets by Casella and Robert

References

1. Interval Estimation for a Binomial Proportion by Brown, Cai, and DasGupta

2. One-sided confidence intervals in discrete distributions by Cai

3. Some Remarks on Confidence or Fiducial Limits by Sterne

4. Mathematische Statistik I. by Witting

5. Binomial Confidence Intervals by Blyth and Still

6. Confidence Curves and Improved Exact Confidence Intervals for Discrete Distributions by Blaker

7. Comment: Randomized Confidence Intervals and the Mid-P Approach by Agresti and Gottard

8. binom by Sundar Dorai-Raj

9. Fuzzy and randomized confidence intervals and p-values by Geyer and Meeden.