some rejigging of sections

Author: pat-alt

dev/build/paper/paper.html

@@ -24,8 +24,12 @@ execute:
 
 {{< include sections/methodology.qmd >}}
 
+{{< include sections/methodology_2.qmd >}}
+
 {{< include sections/empirical.qmd >}}
 
+{{< include sections/empirical_2.qmd >}}
+
 {{< include sections/discussion.qmd >}}
 
 {{< include sections/limitations.qmd >}}

dev/build/paper/paper.pdf

@@ -1,5 +1,42 @@
-# Experiments {#sec-empirical}
+# Experiment Setup {#sec-empirical}
 
+## Recourse Generators {#sec-empirical-recourse}
 
+In the case of the baseline counterfactual generator [@wachter2017counterfactual] $f$ is just the idendity function and the number of counterfactuals $K$ is equal to one. This generator, which we shall refer to as **Wachter** in the following, shall serve as the baseline against which all other gradient-based methodologies will be compared. In particular, we include include the following generator in our benchmarking exercises: REVISE [@joshi2019towards], CLUE [@antoran2020getting], DICE [@mothilal2020explaining] and a greedy approach that relies on probabilistic models [@schut2021generating]. 
+
+Both **REVISE** and **CLUE** search counterfactuals in some latent embedding $S \subset \mathcal{S}$ instead of the feature space directly. The latent embedding is learned by a separate generative model that is tasked with learning the data generating process (DGP) of $X$. In this case $f$ in @eq-general corresponds to the decoder part of the generative model, in other words the deterministic function that maps back from the latent embedding to the feature space. Provided the generative model is well-specified, traversing the latent embedding typically results in realistic and plausible counterfactuals, because they are implicitly generated by the (learned) DGP [@joshi2019towards]. CLUE distinguishes itself from REVISE and other counterfactual generators in that it aims to minimize the predictive uncertainty of the model in question $M$. To quantify predictive uncertainty the authors rely on entropy estimates for probabilistic models. The **Greedy** approach proposed by @schut2021generating also works with the subclass of models $\tilde{\mathcal{M}}\subset\mathcal{M}$ that can produce predictive uncertainty estimates. The authors show that in this setting the complexity penalty $h(\cdot)$ in @eq-general is redundant and meaningful counterfactuals can be generated in a fast and efficient manner through a modified Jacobian-based Saliency Map Attack (JSMA). Finally, **DICE** distinguishes itself from all other generators considered here in that it aims to generate a diverse set of $K>1$ counterfactuals. To this end the authors use a complexity penalty $h(\mathbf{s}^\prime)$ that favours diverse outcomes, in the sense that $s_1, ... , s_K$ look as different from each other as possible. 
+
+Our motivation for including these different generators in our analysis, is that they all offer slightly different approaches to generate meaningful counterfactuals for differentiable black-box models. We hypothesize that generating more **meaningful** counterfactuals should mitigate the endogenous dynamics illustrated in @fig-poc in @sec-intro. This intuition stems from the underlying idea that more meaningful counterfactuals are generated by the same or at least a very similar data generating process as the training data. All else equal, counterfactuals that fulfill this basic requirement should be less prone to trigger domain and model shifts. 
+
+## Data {#sec-empirical-data}
+
+We have chosen to work with both synthetic and real-world datasets. Using synthetic data allows us to impose distributional properties that may affect the resulting recourse dynamics. Following @upadhyay2021towards we generate synthetic data in $\mathbb{R}^2$ to also allow for a visual interpretation of the results. Real-world data is used in order to assess if endogenous dynamics also occur in higher-dimensional settings.
+
+### Synthetic data
+
+```{julia}
+using Plots, PlotThemes
+theme(:wong)
+catalogue = AlgorithmicRecourseDynamics.Data.load_synthetic()
+function plot_data(data,title)
+    plt = plot(title=title)
+    scatter!(data)
+    return plt
+end
+plts = [plot_data(data,name) for (name, data) in catalogue]
+plt = plot(plts..., layout=(1,4), size=(850,200))
+savefig(plt, "dev/paper/www/synthetic_data.png")
+```
+
+
+We use four synthetic binary classification datasets consisting of 1000 samples each.^[To see how the data is generated see here: [https://github.com/pat-alt/AlgorithmicRecourseDynamics.jl/blob/main/notebooks/synthetic_datasets.ipynb](https://github.com/pat-alt/AlgorithmicRecourseDynamics.jl/blob/main/notebooks/synthetic_datasets.ipynb)] The datasets are presented in @fig-synthetic-data (see also Appendix A for a formal description). Samples from the negative class are marked in blue while samples of the positive class are marked in orange.
+
+![Synthetic classification datasets used in our experiments.](www/synthetic_data.png){#fig-synthetic-data fig.pos="h" width="8cm" height="2cm"}
+
+Ex-ante we expect to see that Wachter will create a new cluster of counterfactual instances in the proximity of the initial decision boundary. Thus, the choice of a black-box model may have an impact on the paths of the recourse. For generators that use latent space search (@joshi2019towards, @antoran2020getting) or rely on (and have access to) probabilistic models (@antoran2020getting, @schut2021generating) we expect that counterfactuals will end up in regions of the target domain that are densely populated by training samples. Finally, we expect to see the counterfactuals generated by DiCE to be uniformly spread around the feature space inside the target class. 
+
+### Real-world data
+
+Additionally, we use two real-world datasets from the Finance domain. Firstly, we use the Give Me Some Credit dataset which was open-sourced on Kaggle for the task to predict whether a borrower is likely to experience financial difficulties in the next two years [@gmsc_data]. Originally consisting of 250,000 instances with 11 numerical attributes, the dataset was randomly undersampled to result in a balanced subsample made up of 3000 individuals. Secondly, we the German Credit dataset which involves the task of predicting if bank customers are credit-worthy or not [@germancredit1994]. It consists of 700 positive and 300 negative instances charaterized by 7 numerical and 13 categorical attributes. We process the dataset in two ways: (1) the values of the "Personal status and sex" feature are aggregated by the two represented genders; (2) the most common values are calculated for all categorical features such that a feature $x_d$ with the mode $\bar{x}_d$ is transformed into a new binary feature $\tilde{x}_d=\mathbb{I}_{x_{d,i}>=\bar{x}_d}$. Binarization ensures that we can use all counterfactual generators in the benchmark.
 
 

dev/build/paper/paper_files/libs/bootstrap/bootstrap.min.css

@@ -0,0 +1,6 @@
+# Experiments {#sec-empirical-2}
+
+## Endogenous Macrodynamics
+
+## Potential Mitigation Strategies
+