Testing Your DAG Against Data

Enumerate implied conditional independences and test whether your structural assumptions hold before fitting.
Author

Benjamin Vincent

You’ve drawn a DAG that encodes your causal assumptions — which variables cause which, which paths exist, and which paths don’t exist. But how do you know the data agrees?

Every missing edge in a DAG is a testable prediction. If your DAG says Training affects Performance only through Skill, it predicts that Training and Performance should be conditionally independent given Skill. If the data shows a significant partial correlation between them after controlling for Skill, something is wrong: either the DAG is missing an edge, or the causal structure you assumed doesn’t match reality.

pathmc can enumerate all these predictions and test them automatically — before you run a single MCMC sample.

Setup

import numpy as np
import pandas as pd
import pathmc

n = 500

Every missing edge is a prediction

Consider a chain: X causes M, and M causes Y. There is no direct edge from X to Y — the effect is fully mediated.

X X M M X->M Y Y M->Y
Figure 1: Chain DAG: X affects Y only through M. The missing X → Y edge predicts that X and Y are conditionally independent given M.

This missing edge makes a concrete prediction: once you know M, learning X tells you nothing new about Y. In probability notation, X ⊥⊥ Y | M — X and Y are conditionally independent given M.

.implied_independences() extracts every such prediction from the DAG:

chain_dag = pathmc.model(
    """
    M ~ X
    Y ~ M
    """
)

for ci in chain_dag.implied_independences():
    print(ci)
X ⊥⊥ Y | {M}

One missing edge, one testable prediction. Larger DAGs produce more — each is an opportunity to catch misspecification before it corrupts your causal estimates.

A correct DAG passes the test

To see what “passing” looks like, generate data that genuinely follows the chain structure: X drives M, M drives Y, and there is no shortcut from X to Y.

rng_chain = np.random.default_rng(seed=sum(map(ord, "chain mediation")))

truth_chain = {
    "x_to_m": 0.7,  # X → M coefficient
    "m_to_y": 0.5,  # M → Y coefficient
    "sigma": 0.5,  # residual noise std
}

X = rng_chain.normal(size=n)
M = truth_chain["x_to_m"] * X + rng_chain.normal(scale=truth_chain["sigma"], size=n)
Y = truth_chain["m_to_y"] * M + rng_chain.normal(scale=truth_chain["sigma"], size=n)

df_chain = pd.DataFrame({"X": X, "M": M, "Y": Y})

Fit the model and test the DAG’s prediction:

chain_model = pathmc.model(
    """
    M ~ X
    Y ~ M
    """,
    data=df_chain,
)
chain_model.equations()

\begin{aligned} \beta_{M} &\sim \text{Normal}(mu=0,\, sigma=10) \\ \sigma_{M} &\sim \text{HalfNormal}(sigma=1) \\ \beta_{Y} &\sim \text{Normal}(mu=0,\, sigma=10) \\ \sigma_{Y} &\sim \text{HalfNormal}(sigma=1) \\[6pt] \mu_{M} &= \beta_{0,\,M} + \mathrm{X} \\ \mathrm{M} &\sim \text{Normal}(\mu_{M},\, \sigma_{M}) \\ \mu_{Y} &= \beta_{0,\,Y} + \mathrm{M} \\ \mathrm{Y} &\sim \text{Normal}(\mu_{Y},\, \sigma_{Y}) \end{aligned} 

chain_model.test_implications()

DAG Implication Tests (α = 0.05)

✓ All implied independences are consistent with the data.

IndependencePartial rp-valuePass
X ⊥⊥ Y | {M}0.0460.3086✓

The partial correlation between X and Y — after partialing out M — is close to zero and not significant. The data is consistent with the DAG.

Identification is conditional on the DAG

Identification helpers answer a structural question: if your declared DAG is right, can the causal effect be identified? They do not know whether the DAG omitted a real common cause.

The docstrings for adjustment_sets(), is_identifiable(), collider_warnings(), and frontdoor_identifiable() call this out explicitly. Use test_implications() as the next workflow step when observed data is available.

Suppose the analyst declares a simple mediated effect, T → M → Y, but the real data-generating process also has an omitted confounder U that affects both T and Y:

U U T T U->T Y Y U->Y M M T->M M->Y
Figure 2: The declared DAG omits U, a real common cause of T and Y. The dashed arrows are absent from the model specification. 
rng_omitted = np.random.default_rng(seed=sum(map(ord, "omitted confounder")))

truth_omitted = {
    "u_to_t": 0.8,
    "t_to_m": 0.7,
    "m_to_y": 0.5,
    "u_to_y": 0.6,
    "sigma": 0.5,
}

U = rng_omitted.normal(size=n)
T = truth_omitted["u_to_t"] * U + rng_omitted.normal(
    scale=truth_omitted["sigma"], size=n
)
M3 = truth_omitted["t_to_m"] * T + rng_omitted.normal(
    scale=truth_omitted["sigma"], size=n
)
Y3 = (
    truth_omitted["m_to_y"] * M3
    + truth_omitted["u_to_y"] * U
    + rng_omitted.normal(scale=truth_omitted["sigma"], size=n)
)

df_omitted = pd.DataFrame({"T": T, "M": M3, "Y": Y3})

The declared DAG has no backdoor path from T to Y, so is_identifiable() reports True:

omitted_model = pathmc.model(
    """
    M ~ T
    Y ~ M
    """,
    data=df_omitted,
)
omitted_model.is_identifiable("T", "Y")
True

That result means “identified according to the declared DAG” — not “the declared DAG is true.” Testing the DAG against the observed data exposes the problem:

omitted_model.test_implications()

DAG Implication Tests (α = 0.05)

✗ 1 of 1 implied independences violated.

IndependencePartial rp-valuePass
T ⊥⊥ Y | {M}0.4850.0000✗

The implied independence T ⊥⊥ Y | M fails because the omitted U still links T and Y after conditioning on M. test_implications() cannot name U for you, but it can show that the declared structure is inconsistent with the data.

Catching a missing edge

Now suppose the true data-generating process includes an edge the DAG doesn’t capture. X still drives M, and M still drives Y, but X also affects Y directly.

rng_direct = np.random.default_rng(seed=sum(map(ord, "missing edge")))

truth_direct = {
    "x_to_m": 0.7,  # X → M
    "m_to_y": 0.5,  # M → Y
    "x_to_y_direct": 0.4,  # X → Y (exists in reality, missing from DAG)
    "sigma": 0.5,
}

X2 = rng_direct.normal(size=n)
M2 = truth_direct["x_to_m"] * X2 + rng_direct.normal(
    scale=truth_direct["sigma"], size=n
)
Y2 = (
    truth_direct["m_to_y"] * M2
    + truth_direct["x_to_y_direct"] * X2
    + rng_direct.normal(scale=truth_direct["sigma"], size=n)
)

df_direct = pd.DataFrame({"X": X2, "M": M2, "Y": Y2})

The true structure has three edges — but the analyst proposes a chain with only two:

X X M M X->M Y Y X->Y  missing  from DAG M->Y
Figure 3: True data-generating process. The dashed X → Y edge exists in reality but is absent from the proposed chain DAG (M ~ X; Y ~ M).

The proposed chain DAG predicts X ⊥⊥ Y | M. The data disagrees:

wrong_model = pathmc.model(
    """
    M ~ X
    Y ~ M
    """,
    data=df_direct,
)
wrong_model.equations()

\begin{aligned} \beta_{M} &\sim \text{Normal}(mu=0,\, sigma=10) \\ \sigma_{M} &\sim \text{HalfNormal}(sigma=1) \\ \beta_{Y} &\sim \text{Normal}(mu=0,\, sigma=10) \\ \sigma_{Y} &\sim \text{HalfNormal}(sigma=1) \\[6pt] \mu_{M} &= \beta_{0,\,M} + \mathrm{X} \\ \mathrm{M} &\sim \text{Normal}(\mu_{M},\, \sigma_{M}) \\ \mu_{Y} &= \beta_{0,\,Y} + \mathrm{M} \\ \mathrm{Y} &\sim \text{Normal}(\mu_{Y},\, \sigma_{Y}) \end{aligned} 

wrong_model.test_implications()

DAG Implication Tests (α = 0.05)

✗ 1 of 1 implied independences violated.

IndependencePartial rp-valuePass
X ⊥⊥ Y | {M}0.4370.0000✗

The partial correlation is substantial (around 0.4) and highly significant. The data is telling us: even after accounting for M, X still predicts Y. The DAG is missing the direct X → Y edge.

Fixing the DAG

Adding the direct edge removes the violated prediction and brings the DAG in line with the data:

fixed_model = pathmc.model(
    """
    M ~ X
    Y ~ M + X
    """,
    data=df_direct,
)
fixed_model.equations()

\begin{aligned} \beta_{M} &\sim \text{Normal}(mu=0,\, sigma=10) \\ \sigma_{M} &\sim \text{HalfNormal}(sigma=1) \\ \beta_{Y} &\sim \text{Normal}(mu=0,\, sigma=10) \\ \sigma_{Y} &\sim \text{HalfNormal}(sigma=1) \\[6pt] \mu_{M} &= \beta_{0,\,M} + \mathrm{X} \\ \mathrm{M} &\sim \text{Normal}(\mu_{M},\, \sigma_{M}) \\ \mu_{Y} &= \beta_{0,\,Y} + \mathrm{M} + \mathrm{X} \\ \mathrm{Y} &\sim \text{Normal}(\mu_{Y},\, \sigma_{Y}) \end{aligned} 

print(f"Implied independences: {fixed_model.implied_independences()}")
Implied independences: []

A fully connected graph has no missing edges and therefore no testable predictions — the DAG is compatible with any pattern in the data. This is correct but also less informative: a DAG that predicts nothing can’t be falsified.

ImportantConsistency is necessary, not sufficient

Passing all independence tests means the data doesn’t contradict your DAG. It does not prove the DAG is correct. Multiple DAGs can produce the same set of conditional independences — the data cannot distinguish between them. Domain knowledge remains essential for choosing among compatible structures.

Iterating on a larger DAG

Independence tests become more valuable as DAGs grow, because more missing edges means more testable predictions.

Consider a marketing funnel. The true process has four variables: Budget drives Ads, Ads drive Clicks, Clicks drive Sales — and Budget also has a direct effect on Sales through non-advertising channels (e.g. sales team headcount, trade promotions).

rng_funnel = np.random.default_rng(seed=sum(map(ord, "marketing funnel")))

truth_funnel = {
    "budget_to_ads": 0.8,  # Budget → Ads
    "ads_to_clicks": 0.6,  # Ads → Clicks
    "clicks_to_sales": 0.5,  # Clicks → Sales
    "budget_to_sales": 0.4,  # Budget → Sales (direct, non-ad channel)
    "sigma": 0.5,
}

budget = rng_funnel.normal(loc=10, scale=2, size=n)
ads = truth_funnel["budget_to_ads"] * budget + rng_funnel.normal(
    scale=truth_funnel["sigma"], size=n
)
clicks = truth_funnel["ads_to_clicks"] * ads + rng_funnel.normal(
    scale=truth_funnel["sigma"], size=n
)
sales = (
    truth_funnel["clicks_to_sales"] * clicks
    + truth_funnel["budget_to_sales"] * budget
    + rng_funnel.normal(scale=truth_funnel["sigma"], size=n)
)

df_funnel = pd.DataFrame({
    "Budget": budget,
    "Ads": ads,
    "Clicks": clicks,
    "Sales": sales,
})

An analyst proposes a pure chain — Budget flows through Ads, then Clicks, then Sales — with no direct Budget → Sales edge:

Budget Budget Ads Ads Budget->Ads Clicks Clicks Ads->Clicks Sales Sales Clicks->Sales
Figure 4: Proposed chain funnel. Budget affects Sales only through the Ads → Clicks pipeline. Three missing edges produce three testable predictions.

The DAG has three missing edges, each producing a testable prediction:

funnel_model = pathmc.model(
    """
    Ads ~ Budget
    Clicks ~ Ads
    Sales ~ Clicks
    """,
    data=df_funnel,
)
funnel_model.equations()

\begin{aligned} \beta_{Ads} &\sim \text{Normal}(mu=0,\, sigma=10) \\ \sigma_{Ads} &\sim \text{HalfNormal}(sigma=1) \\ \beta_{Clicks} &\sim \text{Normal}(mu=0,\, sigma=10) \\ \sigma_{Clicks} &\sim \text{HalfNormal}(sigma=1) \\ \beta_{Sales} &\sim \text{Normal}(mu=0,\, sigma=10) \\ \sigma_{Sales} &\sim \text{HalfNormal}(sigma=1) \\[6pt] \mu_{Ads} &= \beta_{0,\,Ads} + \mathrm{Budget} \\ \mathrm{Ads} &\sim \text{Normal}(\mu_{Ads},\, \sigma_{Ads}) \\ \mu_{Clicks} &= \beta_{0,\,Clicks} + \mathrm{Ads} \\ \mathrm{Clicks} &\sim \text{Normal}(\mu_{Clicks},\, \sigma_{Clicks}) \\ \mu_{Sales} &= \beta_{0,\,Sales} + \mathrm{Clicks} \\ \mathrm{Sales} &\sim \text{Normal}(\mu_{Sales},\, \sigma_{Sales}) \end{aligned} 

for ci in funnel_model.implied_independences():
    print(ci)
Ads ⊥⊥ Sales | {Budget, Clicks}
Budget ⊥⊥ Clicks | {Ads}
Budget ⊥⊥ Sales | {Clicks}

Testing against the data:

funnel_model.test_implications()

DAG Implication Tests (α = 0.05)

✗ 1 of 3 implied independences violated.

IndependencePartial rp-valuePass
Ads ⊥⊥ Sales | {Budget, Clicks}0.0130.7773✓
Budget ⊥⊥ Clicks | {Ads}-0.0330.4610✓
Budget ⊥⊥ Sales | {Clicks}0.6320.0000✗

Two of the three predictions hold: Ads is independent of Sales given Budget and Clicks, and Budget is independent of Clicks given Ads. But the third fails: Budget and Sales are not independent given Clicks. The data reveals a direct Budget → Sales pathway that bypasses the advertising funnel.

Refining the DAG

Adding the Budget → Sales edge addresses the violation:

Budget Budget Ads Ads Budget->Ads Sales Sales Budget->Sales Clicks Clicks Ads->Clicks Clicks->Sales
Figure 5: Corrected funnel with direct Budget → Sales edge. The remaining two missing edges still produce testable (and passing) predictions. 
fixed_funnel = pathmc.model(
    """
    Ads ~ Budget
    Clicks ~ Ads
    Sales ~ Clicks + Budget
    """,
    data=df_funnel,
)
fixed_funnel.equations()

\begin{aligned} \beta_{Ads} &\sim \text{Normal}(mu=0,\, sigma=10) \\ \sigma_{Ads} &\sim \text{HalfNormal}(sigma=1) \\ \beta_{Clicks} &\sim \text{Normal}(mu=0,\, sigma=10) \\ \sigma_{Clicks} &\sim \text{HalfNormal}(sigma=1) \\ \beta_{Sales} &\sim \text{Normal}(mu=0,\, sigma=10) \\ \sigma_{Sales} &\sim \text{HalfNormal}(sigma=1) \\[6pt] \mu_{Ads} &= \beta_{0,\,Ads} + \mathrm{Budget} \\ \mathrm{Ads} &\sim \text{Normal}(\mu_{Ads},\, \sigma_{Ads}) \\ \mu_{Clicks} &= \beta_{0,\,Clicks} + \mathrm{Ads} \\ \mathrm{Clicks} &\sim \text{Normal}(\mu_{Clicks},\, \sigma_{Clicks}) \\ \mu_{Sales} &= \beta_{0,\,Sales} + \mathrm{Clicks} + \mathrm{Budget} \\ \mathrm{Sales} &\sim \text{Normal}(\mu_{Sales},\, \sigma_{Sales}) \end{aligned} 

fixed_funnel.test_implications()

DAG Implication Tests (α = 0.05)

✓ All implied independences are consistent with the data.

IndependencePartial rp-valuePass
Ads ⊥⊥ Sales | {Budget, Clicks}0.0130.7773✓
Budget ⊥⊥ Clicks | {Ads}-0.0330.4610✓

All remaining predictions pass. The refined DAG is consistent with the data, and the model is ready for estimation.

TipThe workflow
  1. Specify your DAG from domain knowledge
  2. Run test_implications() to check data consistency
  3. If violations appear, consider whether the data is suggesting a missing edge or unmeasured confounder
  4. Update the DAG and re-test
  5. Once consistent, proceed to sampling and causal inference

This loop happens before fitting. It costs nothing computationally and can save you from drawing causal conclusions from a misspecified model.

Limitations

WarningWhat these tests can and cannot do

Partial correlation assumes linearity. The test computes the correlation between residuals from linear regressions. If two variables have a nonlinear conditional dependence (e.g. X affects Y quadratically), the partial correlation test may miss it.

Small samples reduce power. With few observations or large conditioning sets, the test may fail to detect genuine violations. A passing test does not guarantee the independence holds — it may just mean the sample is too small to detect the association.

Only observed variables are testable. If your DAG includes latent variables, the implied independences involving those variables cannot be tested against data. The test automatically restricts to observable implications.

Summary

  • Every missing edge in a DAG predicts a conditional independence — that two variables should be unrelated after conditioning on certain others.
  • implied_independences() enumerates all these predictions from the DAG structure alone, before any data is involved.
  • test_implications() tests each prediction against observed data using partial correlation. A significant result flags a violation: the data contains an association the DAG says shouldn’t exist.
  • Violations suggest missing edges or incorrect structural assumptions. They guide iterative DAG refinement before fitting.
  • Passing all tests is necessary but not sufficient — multiple DAGs can imply the same independences. Domain knowledge remains essential.
  • The test works before sampling — it uses observed data directly, requires no posterior draws, and is computationally trivial.
NoteReflection

Think about a system in your domain with 4–6 variables. Draw the DAG you believe in, then ask:

  • Which edges are you most uncertain about? Each missing edge produces a testable prediction. If you’re unsure whether A directly affects C or only operates through B, the independence test can weigh in.
  • In a clinical trial, a DAG where treatment affects the biomarker, and the biomarker affects the outcome, predicts treatment ⊥⊥ outcome | biomarker. If that prediction fails, the treatment may have a direct effect that bypasses the biomarker — important for surrogate endpoint validation.
  • In a marketing model, if your DAG says TV ads only affect sales through brand awareness, a violation of TV ⊥⊥ Sales | Awareness suggests a direct response channel you haven’t modeled.