# Causal Inference: Propensity Matching
Author: Nate Shenkute
Created: July 6, 2022 1:57 PM
Link: https://towardsdatascience.com/matching-weighting-or-regression-99bf5cffa0d9
Summary: In a company like the one I work for, it is not always possible/feasible to conduct a proper A/B test due to technicalities. The question is then how do we simulate/approximate an A/B test when we don’t have a randomized controlled trial?
The greatest problem when it comes to comparing statistical differences between two groups, say group A and group B in an AB test, is confounding factors.
> [!note]
> A/B tests or randomized controlled trials are the **gold standard** in causal inference. By randomly exposing units to a treatment we make sure that treated and untreated individuals are comparable, on average, and any outcome difference we observe can be attributed to the treatment effect alone.
>
One way of handling this issue is by controlling for possible confounding factors. For example, by comparing sub-groups with similar attributes such as age, sex, location etc.
One very useful tool for such situations is the `create_table_one` function from Uber's `[causalml](https://causalml.readthedocs.io/)` package. This function produces a covariate balance table containing the average value of the possible confounders across treatment/control groups:
```python
from causalml.match import create_table_one
X = ['male', 'age', 'hours']
cv_table = create_table_one(df,
'dark_mode', X)
display(cv_table)
```
The covariate balance table will give a nice summary of the comparison between the treatment and control groups.
```python
def plot_distributions(df, X, d):
df_long = df.copy()[X + [d]]
df_long[X] =(df_long[X] - df_long[X].mean()) / df_long[X].std()
df_long = pd.melt(df_long, id_vars=d, value_name='value')
sns.violinplot(y="variable", x="value", hue=d, data=df_long, split=True).\
set(xlabel="", ylabel="", title="Normalized Variable Distribution");
plot_distributions(df, X, "dark_mode")
```
<aside>
👨🏾💻 Unless we control for the observed confounding variables, we will not be able to estimate the true statistical effect of the treatment.
</aside>
In graph lingo, the process of taking care of these confounding variables, is called ***blocking the backdoor path***. This is done by **conditioning the analysis** on those intermediate confounding variables (eg. gender).
Conditioning on several confounding factors will allow you to increase the precision of the causality estimates, but does not impact the causal interpretation of the results.
## Matching
Here the idea is to do the analysis by separating the data using confounding the variables eg. gender. But watch out for the arrival of the infamous [Simpson's Paradox](https://en.wikipedia.org/wiki/Simpson's_paradox).
But then, what do you do when you have multiple confounding variables and you still want to match sub-groups? This can be dealt with using some sort of nearest neighbour algorithm that matches users in the treatment group with the most similar users in the control group. The `causalml` package lets you do this:
```python
from causalml.match import NearestNeighborMatch
psm = NearestNeighborMatch(replace=True, ratio=1, random_state=1)
df_matched = psm.match(data=df, treatment_col="dark_mode", score_cols=X)
table1_matched = create_table_one(df_matched, "dark_mode", X)
table1_matched
```
Now take a look at the covariate balance table:
```python
plot_distributions(df_matched, X,
"dark_mode")
```
Note that this algorithm might drop part of the sample if it is not able to find appropriate matches for it.
Here is another way of visualising pre and post covariate balances: using the **balance plot**:
```python
def plot_balance(t1, t2, X):
df_smd = pd.DataFrame({"Variable": X + X,
"Sample": ["Unadjusted" for _ in range(len(X))] + ["Adjusted" for _ in range(len(X))],
"Standardized Mean Difference": t1["SMD"][1:].to_list() +
t2["SMD"][1:].to_list()})
sns.scatterplot(x="Standardized Mean Difference", y="Variable", hue="Sample", data=df_smd).\
set(title="Balance Plot")
plt.axvline(x=0, color='k', ls='--', zorder=-1, alpha=0.3);
plot_balance(table1, table1_matched, X)
```
Now that we have taken care of the confounder differences, we can simply either do a difference of means or run a linear regression of the outcome:
```python
smf.ols("read_time ~ dark_mode",
data=df_matched).fit().summary().tables[1]
```
👨🏾💻 **Note** that we might have matched multiple treated users with the same untreated user, violating the **independence assumption** across observations and, in turn, distorting inference. We have two solutions:
1. **cluster** standard errors at the pre-matching individual level
2. compute standard errors via **bootstrap** (preferred)
We implement the first and cluster standard errors by the original individual identifiers (the dataframe index).
```python
smf.ols("read_time ~ dark_mode", data=df_matched)\
.fit(cov_type='cluster', cov_kwds={'groups': df_matched.index})\
.summary().tables[1]
```
# Sources
[Matching, Weighting, or Regression?](https://towardsdatascience.com/matching-weighting-or-regression-99bf5cffa0d9)
[Lecture 3: Propensity Scores](https://www.youtube.com/watch?v=8gWctYvRzk4)