refute.PlaceboRefutationResult

Result of a Bayesian placebo-treatment refutation.

Usage

refute.PlaceboRefutationResult(
    outcome,
    treatment,
    observed_ate_draws,
    fold_means,
    fold_sds,
    mu_null_draws,
    tau_het_draws,
    theta_new_draws,
    z_cal,
    p_tail,
    n_permutations,
    significance_level=0.05,
    random_seed=None
)

Produced by refute_placebo() / pathmc.PathModel.refute_placebo(). Holds the real effect’s posterior, the per-permutation placebo summaries, the hierarchical null model’s posterior, and the calibrated comparison of the real effect against the placebo null.

The pipeline passes the placebo test when the systematic placebo bias mu_null credibly includes zero (passes_placebo): a sound estimator should attribute no effect to a treatment whose link to the outcome has been severed by permutation. The real effect survives the null when its calibrated tail probability is small (effect_survives).

Parameters

outcome: str

Outcome variable name.

treatment: str

Treatment variable name (the permuted/placebo variable).

observed_ate_draws: np.ndarray

Posterior draws of the real ATE from the fitted model, shape (n_draws,).

fold_means: np.ndarray

Placebo-fold posterior means m_j, shape (n_permutations,).

fold_sds: np.ndarray

Placebo-fold posterior SDs s_j (floored at 1e-6), shape (n_permutations,).

mu_null_draws: np.ndarray

Posterior draws of the systematic placebo bias mu_null.

tau_het_draws: np.ndarray

Posterior draws of the structural volatility tau_het.

theta_new_draws: np.ndarray

Draws from the null predictive distribution theta_new ~ Normal(mu_null, tau_het).

z_cal: float

Calibrated z statistic of the real ATE against the placebo null, (observed_ate - null_mean) / sqrt(null_sd**2 + mean(s_j**2)), where null_mean and null_sd summarize the null predictive distribution theta_new (see null_mean, null_sd). null_sd (the predictive SD) is used rather than the bare tau_het parameter because it also absorbs posterior uncertainty in mu_null, matching the calibration reference implementation.

p_tail: float

One-sided (directional) calibrated tail probability: the fraction of observable-null predictive draws at least as extreme, in the direction of the observed effect, as the observed ATE. The tail side is selected post hoc from sign(observed_ate - null_mean) (effectively the sign of the observed effect, since the null straddles zero) — it is not a pre-registered side, so do not read it as a fixed one-sided hypothesis test. Floored at 1 / (n_draws + 1) to reflect the Monte Carlo resolution rather than reporting an exact zero.

n_permutations: int

Number of placebo permutations (folds).

significance_level: float = 0.05

Threshold for the effect_survives verdict (default 0.05).

random_seed: int | None = None
Seed used for the permutations and hierarchical fit.

Attributes

Name Description
effect_survives Whether the real effect is distinguishable from the placebo null.
estimated_effect The originally estimated ATE (dowhy’s “estimated effect”).
mu_null Posterior mean of the systematic placebo bias mu_null.
mu_null_hdi 94% highest density interval of the placebo bias mu_null.
new_effect The pooled placebo effect (dowhy’s “new effect”).
null_mean Mean of the null predictive distribution (placebo effect center).
null_sd SD of the null predictive distribution (total placebo volatility).
observed_ate Posterior mean of the real (observed) ATE.
observed_ate_hdi 94% highest density interval of the real ATE.
passes_placebo Whether the placebo null straddles zero.
sigma_pred Predictive-null SD used in the z_cal denominator.
tau_het Posterior mean of the structural volatility tau_het.

effect_survives

Whether the real effect is distinguishable from the placebo null.

effect_survives: bool

True when the calibrated tail probability p_tail is below significance_level: the observed effect is too extreme to be explained by placebo (structural) noise alone.

estimated_effect

The originally estimated ATE (dowhy’s “estimated effect”).

estimated_effect: float

Alias of observed_ate, named to mirror dowhy’s CausalRefutation.estimated_effect.

mu_null

Posterior mean of the systematic placebo bias mu_null.

mu_null: float

mu_null_hdi

94% highest density interval of the placebo bias mu_null.

mu_null_hdi: np.ndarray

new_effect

The pooled placebo effect (dowhy’s “new effect”).

new_effect: float

The systematic placebo bias mu_null pooled across permutations — the effect the pipeline still reports once the treatment-outcome link is severed. A sound pipeline keeps this near zero. This is the Bayesian analogue of dowhy’s new_effect = mean(placebo estimates), but pooled hierarchically and equipped with mu_null_hdi.

null_mean

Mean of the null predictive distribution (placebo effect center).

null_mean: float

null_sd

SD of the null predictive distribution (total placebo volatility).

null_sd: float

observed_ate

Posterior mean of the real (observed) ATE.

observed_ate: float

observed_ate_hdi

94% highest density interval of the real ATE.

observed_ate_hdi: np.ndarray

passes_placebo

Whether the placebo null straddles zero.

passes_placebo: bool

True when the 94% HDI of mu_null contains zero, i.e. the pipeline reports no systematic effect for a permuted treatment. A False verdict flags a pipeline that manufactures effects from noise.

Note

This is a pure interval rule with no region of practical equivalence (ROPE), so a negligibly small but very precisely estimated placebo bias can fail. Read it together with mu_null (the bias magnitude): a mu_null that is tiny relative to the observed effect is practically benign even when its HDI excludes zero.

The opposite failure mode also exists at small n_permutations: with few folds the prior dominates tau_het and inflates the mu_null HDI, so it almost always straddles zero — a genuinely biased pipeline can then be masked as a PASS. Use 8+ folds (see refute_placebo()) before trusting a PASS as evidence the pipeline is unbiased.

sigma_pred

Predictive-null SD used in the z_cal denominator.

sigma_pred: float

sqrt(null_sd**2 + mean(s_j**2)): the null-predictive spread convolved with the mean within-fold estimation variance. The mean of squares (not the square of the mean) is used so that z_cal is coherent with the bootstrap p_tail, which draws Normal(0, s_j) noise and thus has variance mean(s_j**2).

tau_het

Posterior mean of the structural volatility tau_het.

tau_het: float

Methods

Name Description
plot() Plot the refutation result.
summary() Return a dowhy-style textual summary of the refutation.

plot()

Plot the refutation result.

Usage

plot(ax=None, kind="comparison", bins=50)
Parameters
ax: matplotlib.axes.Axes | None = None

Axes to plot on. Creates a new figure if None.

kind: ("comparison", "null") = "comparison"

Which view to draw (default "comparison"):

  • "comparison" — the real (observed) effect and the pooled placebo effect side by side, each as a point with its 94% HDI, against a zero reference. This is the visual analogue of dowhy’s “Estimated effect vs New effect”: a sound pipeline shows the placebo row sitting on zero and the observed row away from it.
  • "null" — the placebo null-predictive distribution (histogram) with the observed ATE marked.
bins: int = 50
Number of histogram bins, used only when kind="null" (default 50).
Returns
matplotlib.figure.Figure
The figure containing the plot.
Raises
ValueError
If kind is unknown or bins is not positive.

summary()

Return a dowhy-style textual summary of the refutation.

Usage

summary()

Mirrors dowhy’s CausalRefutation output (Estimated effect / New effect / p value), with the Bayesian additions of a credible interval on the placebo effect and a pass/fail placebo verdict.

Returns
str
A multi-line summary string.