Skip to content

Commit 19c2efc

Browse files
committed
verifying that results check out with updated package
1 parent 7d06842 commit 19c2efc

15 files changed

Lines changed: 72 additions & 87 deletions

File tree

dev/artifacts/generate_artifacts.jl

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ function generate_artifacts(
77
root=".",
88
artifact_toml=joinpath(root, "Artifacts.toml"),
99
deploy=true,
10-
tag="artifacts"
10+
tag="camera-ready"
1111
)
1212

1313
if deploy && !haskey(ENV, "GITHUB_TOKEN")

docs/make.jl

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ makedocs(;
2424
"🎓 Research Paper" => [
2525
"Overview" => "paper/index.md",
2626
"Proof-of-Concept" => "paper/proof_of_concept.md",
27+
"Supplementary Appendix" => "paper/appendix.md",
2728
],
2829
"🧐 Reference" => "_reference.md",
2930
],

docs/src/paper/Project.toml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,9 @@
11
[deps]
22
AlgorithmicRecourseDynamics = "3d1ede72-abb8-4340-bf8e-2ae06849b5ec"
33
CounterfactualExplanations = "2f13d31b-18db-44c1-bc43-ebaf2cff0be0"
4+
DataFrames = "a93c6f00-e57d-5684-b7b6-d8193f3e46c0"
45
Flux = "587475ba-b771-5e3f-ad9e-33799f191a9c"
6+
Images = "916415d5-f1e6-5110-898d-aaa5f9f070e0"
57
LaplaceRedux = "c52c1a26-f7c5-402b-80be-ba1e638ad478"
68
MLJBase = "a7f614a8-145f-11e9-1d2a-a57a1082229d"
79
Plots = "91a5bcdd-55d7-5caf-9e0b-520d859cae80"

docs/src/paper/appendix.qmd

Lines changed: 7 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -1,29 +1,17 @@
11
---
22
title: Supplementary Appendix
3-
format:
4-
html:
5-
self-contained: true
6-
code-fold: true
7-
execute:
8-
echo: true
9-
eval: false
10-
warning: false
11-
toc: true
12-
jupyter: julia-1.7
133
---
144

15-
This is a supplementary appendix to the research paper **Endogenous Macrodynamics in Algorithmic Recourse**. It contains all of the experimental results, including those not highlighted in the actual paper. It also contains additional information about the proposed counterfactual generators.
5+
This is a supplementary appendix to the research paper **Endogenous Macrodynamics in Algorithmic Recourse**. It contains all of the experimental results, including those not highlighted in the actual paper. It also links to additional information about the proposed mitigation strategies.
166

17-
# Experimental Results {#sec-results}
7+
## Experimental Results {#sec-results}
188

19-
{{< include experiments/synthetic.qmd >}}
9+
{{< include experiments/_synthetic.qmd >}}
2010

21-
{{< include experiments/real_world.qmd >}}
11+
{{< include experiments/_real_world.qmd >}}
2212

23-
{{< include experiments/mitigation_strategies.qmd >}}
13+
{{< include experiments/_mitigation_strategies.qmd >}}
2414

25-
# Generators {#sec-generators}
15+
## Generators {#sec-generators}
2616

27-
{{< include generators/gravitational_generator.qmd >}}
28-
29-
{{< include generators/clap_roar_generator.qmd >}}
17+
The generators have been moved into [`CounterfactualExplanations.jl`](https://github.com/pat-alt/CounterfactualExplanations.jl). For more information so the package [documentation](https://pat-alt.github.io/CounterfactualExplanations.jl/dev).
File renamed without changes.
File renamed without changes.
Lines changed: 25 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -1,22 +1,14 @@
11
## Synthetic Data {#sec-app-synthetic}
22

3-
This notebook was used to run the experiments for the synthetic datasets. In the following we first run the experiments and then generate visualizations and tables.
3+
This notebook was used to run the experiments for the synthetic datasets and can be used to reproduce the results in the paper. In the following we first run the experiments and then generate visualizations and tables.
44

55
### Experiments
66

77
```{julia}
8-
#| eval: true
9-
using Pkg; Pkg.activate("dev")
10-
```
8+
#| echo: false
119
12-
```{julia}
13-
#| eval: true
14-
include("dev/utils.jl")
15-
using AlgorithmicRecourseDynamics
16-
using CounterfactualExplanations, Flux, Plots, PlotThemes, Random, LaplaceRedux, LinearAlgebra, Images
17-
theme(:wong)
18-
output_path = output_dir("synthetic")
19-
www_path = www_dir("synthetic");
10+
include("docs/src/paper/setup.jl")
11+
eval(setup)
2012
```
2113

2214
```{julia}
@@ -49,8 +41,13 @@ generators = Dict(
4941
experiments = set_up_experiments(data_sets,models,generators)
5042
```
5143

44+
@fig-test-before shows the test data before running the experiment.
45+
5246
```{julia}
53-
using AlgorithmicRecourseDynamics.Models: model_evaluation
47+
#| label: fig-test-before
48+
#| fig-cap: "Test data before experiment"
49+
#| eval: true
50+
5451
plts = []
5552
for (exp_name, exp_) in experiments
5653
for (M_name, M) in exp_.models
@@ -65,18 +62,16 @@ for (exp_name, exp_) in experiments
6562
end
6663
plt = plot(plts..., layout=(length(choices),length(models)),size=(length(choices)*300,length(models)*300))
6764
savefig(plt, joinpath(www_path,"models_test_before.png"))
65+
plt
6866
```
6967

70-
@fig-test-before shows the test data before running the experiment.
68+
@fig-train-before shows the training data before running the experiment.
7169

7270
```{julia}
73-
#| label: fig-test-before
74-
#| fig-cap: "Test data before experiment"
71+
#| label: fig-train-before
72+
#| fig-cap: "Training data before experiment"
7573
#| eval: true
76-
load(joinpath(www_path,"models_test_before.png"))
77-
```
7874
79-
```{julia}
8075
using AlgorithmicRecourseDynamics.Models: model_evaluation
8176
plts = []
8277
for (exp_name, exp_) in experiments
@@ -92,15 +87,7 @@ for (exp_name, exp_) in experiments
9287
end
9388
plt = plot(plts..., layout=(length(choices),length(models)),size=(length(choices)*300,length(models)*300))
9489
savefig(plt, joinpath(www_path,"models_train_before.png"))
95-
```
96-
97-
@fig-train-before shows the training data before running the experiment.
98-
99-
```{julia}
100-
#| label: fig-train-before
101-
#| fig-cap: "Training data before experiment"
102-
#| eval: true
103-
load(joinpath(www_path,"models_train_before.png"))
90+
plt
10491
```
10592

10693

@@ -111,7 +98,6 @@ evaluate_every = Int(round(n_rounds/n_evals))
11198
n_folds = 5
11299
n_bootstrap = 1
113100
T = 100
114-
using Serialization
115101
results = run_experiments(
116102
experiments;
117103
save_path=output_path,evaluate_every=evaluate_every,n_rounds=n_rounds, n_folds=n_folds, n_bootstrap=n_bootstrap, T=T
@@ -120,7 +106,6 @@ Serialization.serialize(joinpath(output_path,"results.jls"),results)
120106
```
121107

122108
```{julia}
123-
using AlgorithmicRecourseDynamics.Models: model_evaluation
124109
plot_dict = Dict(key => Dict() for (key,val) in results)
125110
fold = 1
126111
for (name, res) in results
@@ -160,10 +145,12 @@ end
160145
#| - "Latent"
161146
#| layout-ncol: 1
162147
#| label: fig-test-after
148+
#| output: true
149+
163150
img_files = readdir(www_path)[contains.(readdir(www_path),"models_test_after")]
164151
img_files = joinpath.(www_path,img_files)
165152
for img in img_files
166-
display(load(img))
153+
display(Images.load(img))
167154
end
168155
```
169156

@@ -212,7 +199,7 @@ end
212199
img_files = readdir(www_path)[contains.(readdir(www_path),"models_train_after")]
213200
img_files = joinpath.(www_path,img_files)
214201
for img in img_files
215-
display(load(img))
202+
display(Images.load(img))
216203
end
217204
```
218205

@@ -243,7 +230,7 @@ end
243230
@fig-line shows the evolution of the evaluation metrics over the course of the experiment.
244231

245232
```{julia}
246-
#| eval: true
233+
#| output: true
247234
#| fig-cap: "Line Charts"
248235
#| fig-subcap:
249236
#| - "Circles"
@@ -255,7 +242,7 @@ end
255242
img_files = readdir(www_path)[contains.(readdir(www_path),"line_chart")]
256243
img_files = joinpath.(www_path,img_files)
257244
for img in img_files
258-
display(load(img))
245+
display(Images.load(img))
259246
end
260247
```
261248

@@ -264,7 +251,7 @@ end
264251
@fig-error shows the evaluation metrics at the end of the experiments.
265252

266253
```{julia}
267-
#| eval: true
254+
#| output: true
268255
#| fig-cap: "Error Bar Charts"
269256
#| fig-subcap:
270257
#| - "Circles"
@@ -276,7 +263,7 @@ end
276263
img_files = readdir(www_path)[contains.(readdir(www_path),"errorbar_chart")]
277264
img_files = joinpath.(www_path,img_files)
278265
for img in img_files
279-
display(load(img))
266+
display(Images.load(img))
280267
end
281268
```
282269

@@ -316,7 +303,7 @@ CSV.write(save_path, df)
316303
```{julia}
317304
using DataFrames, Statistics
318305
df = results[:overlapping].output
319-
df = df[[x ∈ [50] for x in df.n],:]
306+
df = df[[x ∈ maximum(df.n) for x in df.n],:]
320307
gdf = groupby(df, [:generator, :model, :n, :name, :scope])
321308
df_plot = combine(gdf, :value => (x -> [(mean(x),mean(x)+std(x),mean(x)-std(x))]) => [:mean, :ymax, :ymin])
322309
df_plot = df_plot[[name in [:decisiveness, :disagreement, :mmd, :mmd_grid, :model_performance] for name in df_plot.name],:]
@@ -369,7 +356,7 @@ plt <- ggplot(df_plot) +
369356
legend.position="bottom"
370357
)
371358
temp_path <- file.path(tempdir(), "plot.png")
372-
ggsave(temp_path,width=$ncol * $scale_,height=$nrow * $scale_ * 0.75)
359+
ggsave(temp_path, width=$ncol * $scale_,height=$nrow * $scale_ * 0.75)
373360
"""
374361
375362
img = Images.load(rcopy(R"temp_path"))

docs/src/paper/index.qmd

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,6 @@ This part of the documentation is specific to the research paper "Endogenous Mac
1313
using Pkg; Pkg.activate("docs/src/paper")
1414
```
1515

16-
1716
The status of this package environment is shown below:
1817

1918
```{julia}

docs/src/paper/proof_of_concept.qmd

Lines changed: 2 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -5,26 +5,8 @@ title: Proof of concept
55
```{julia}
66
#| echo: false
77
8-
# Environment:
9-
using Pkg; Pkg.activate("docs/src/paper")
10-
11-
# Deps:
12-
using AlgorithmicRecourseDynamics
13-
using AlgorithmicRecourseDynamics.Models
14-
using CounterfactualExplanations
15-
using Flux
16-
using LaplaceRedux
17-
using Markdown
18-
using MLJBase
19-
using Plots
20-
using Random
21-
22-
# Setup
23-
Random.seed!(2023)
24-
theme(:wong)
25-
include("docs/src/utils.jl") # some helper functions
26-
output_path = output_dir("poc") # output directory for artifacts
27-
www_path = www_dir("poc") # output directory for images
8+
include("docs/src/paper/setup.jl")
9+
eval(setup)
2810
```
2911

3012
To start with, we will look at a proof-of-concept that demonstrates the main observation underlying that paper is framed around. In particular, we will use synthetic data to see how endogenous domain shifts and the resulting model shifts can have implications on the validity and cost of algorithmic recourse.
@@ -104,7 +86,6 @@ candidates = findall(ys.==0)
10486
chosen_individuals = rand(candidates, Int(round(μ*length(candidates))))
10587
X′ = copy(X)
10688
y′ = copy(ys)
107-
using CounterfactualExplanations.Counterfactuals: counterfactual, counterfactual_label
10889
factuals = select_factual(counterfactual_data,chosen_individuals)
10990
outcome = generate_counterfactual(factuals, 1, counterfactual_data, mod, gen; initialization=:identity)
11091
X′[:,chosen_individuals] = reduce(hcat, @.(selectdim(counterfactual(outcome), 3, 1)))

docs/src/paper/setup.jl

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
setup = quote
2+
# Environment:
3+
using Pkg; Pkg.activate("docs/src/paper")
4+
5+
# Deps:
6+
using AlgorithmicRecourseDynamics
7+
using AlgorithmicRecourseDynamics.Models
8+
using AlgorithmicRecourseDynamics.Models: model_evaluation
9+
using CounterfactualExplanations
10+
using CounterfactualExplanations: counterfactual, counterfactual_label
11+
using DataFrames
12+
using Flux
13+
using Images
14+
using LaplaceRedux
15+
using Markdown
16+
using MLJBase
17+
using Plots
18+
using Random
19+
using Serialization
20+
21+
# Setup
22+
Random.seed!(2023) # global seed to allow for reproducibility
23+
theme(:wong)
24+
include("docs/src/utils.jl") # some helper functions
25+
output_path = output_dir("poc") # output directory for artifacts
26+
www_path = www_dir("poc") # output directory for images
27+
end

0 commit comments

Comments
 (0)