Skip to content

Commit e8026cf

Browse files
committed
incorporated Cynthia's comments
1 parent 4e59dd5 commit e8026cf

15 files changed

Lines changed: 169 additions & 167 deletions

dev/notebooks/experiments/real_world.qmd

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -151,10 +151,11 @@ df = DataFrame()
151151
for (key, val) in results
152152
n_folds = length(val.experiment.recourse_systems)
153153
for fold in 1:n_folds
154-
for i in length(val.experiment.system_identifiers)
154+
for i in 1:length(val.experiment.system_identifiers)
155155
rec_sys = val.experiment.recourse_systems[fold][i]
156156
model_name, gen_name = collect(val.experiment.system_identifiers)[i]
157157
df_ = evaluate_system(rec_sys, val.experiment; n=n_bootstrap)
158+
df_.data .= key
158159
df_.model .= model_name
159160
df_.generator .= gen_name
160161
df_.fold .= fold
@@ -166,7 +167,7 @@ df = mapcols(x -> typeof(x) == Vector{Symbol} ? string.(x) : x, df)
166167
using RCall
167168
save_path = joinpath(output_path, "bootstrap.csv")
168169
using CSV
169-
CSV.write(save_path)
170+
CSV.write(save_path, df)
170171
```
171172

172173

dev/notebooks/experiments/synthetic.qmd

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -290,10 +290,11 @@ df = DataFrame()
290290
for (key, val) in results
291291
n_folds = length(val.experiment.recourse_systems)
292292
for fold in 1:n_folds
293-
for i in length(val.experiment.system_identifiers)
293+
for i in 1:length(val.experiment.system_identifiers)
294294
rec_sys = val.experiment.recourse_systems[fold][i]
295295
model_name, gen_name = collect(val.experiment.system_identifiers)[i]
296296
df_ = evaluate_system(rec_sys, val.experiment; n=n_bootstrap)
297+
df_.data .= key
297298
df_.model .= model_name
298299
df_.generator .= gen_name
299300
df_.fold .= fold
@@ -305,7 +306,7 @@ df = mapcols(x -> typeof(x) == Vector{Symbol} ? string.(x) : x, df)
305306
using RCall
306307
save_path = joinpath(output_path, "bootstrap.csv")
307308
using CSV
308-
CSV.write(save_path)
309+
CSV.write(save_path, df)
309310
```
310311

311312
### Chart in paper {#sec-app-synthetic-paper}

paper/paper.Rmd

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,7 @@ affiliation:
6363
# email: roy@replicant.offworld
6464
keywords: ["Algorithmic Recourse", "Counterfactual Explanations", "Explainable AI", "Dynamic Systems"]
6565
abstract: |
66-
Existing work on Counterfactual Explanations (CE) and Algorithmic Recourse (AR) has largely been limited to the static setting and focused on single individuals: given some estimated model the goal is to find valid counterfactuals for individual instance that fulfill various desiderata. The ability of such counterfactuals to handle dynamics like data and model drift remains a largely unexplored research challenge at this point. There has also been surprisingly little work on the related question of how the actual implementation of recourse by one individual may affect other individuals. Through this work we aim to close that gap by systematizing and extending existing knowledge. We first show that many of the existing methodologies can be collectively described by a generalized framework. We then argue that the existing framework fails to account for a hidden external cost of recourse, that only reveals itself when studying the endogenous dynamics of recourse at the group level. Through simulation experiments involving various state-of-the-art counterfactual generators and several benchmark datasets, we generate large numbers of counterfactuals and study the resulting domain and model shifts. We find that the induced shifts are substantial enough to likely impede the applicability Algorithmic Recourse in situations that involve competition for scarce resources. Fortunately, we find various potential mitigation strategies that can be used in combination with existing approaches. Our simulation framework for studying recourse dynamics is fast and open-sourced.
66+
Existing work on Counterfactual Explanations (CE) and Algorithmic Recourse (AR) has largely been limited to the static setting and focused on single individuals: given some estimated model, the goal is to find valid counterfactuals for an individual instance that fulfill various desiderata. The ability of such counterfactuals to handle dynamics like data and model drift remains a largely unexplored research challenge at this point. There has also been surprisingly little work on the related question of how the actual implementation of recourse by one individual may affect other individuals. Through this work we aim to close that gap by systematizing and extending existing knowledge. We first show that many of the existing methodologies can be collectively described by a generalized framework. We then argue that the existing framework fails to account for a hidden external cost of recourse, that only reveals itself when studying the endogenous dynamics of recourse at the group level. Through simulation experiments involving various state-of-the-art counterfactual generators and several benchmark datasets, we generate large numbers of counterfactuals and study the resulting domain and model shifts. We find that the induced shifts are substantial enough to likely impede the applicability of Algorithmic Recourse in situations that involve competition for scarce resources. Fortunately, we find various potential mitigation strategies that can be used in combination with existing approaches. Our simulation framework for studying recourse dynamics is fast and open-sourced.
6767
6868
# use some specific Tex packages if needed.
6969
# with_ifpdf: true

paper/paper.pdf

-270 Bytes
Binary file not shown.

0 commit comments

Comments
 (0)