Bayesian Combination of Human and Machine Predictions

This repository contains all the code to reproduce the results of my master thesis on "Bayesian Combination of Human and Machine Predictions" submitted on 24/04/2023.

Table of Contents

• Introduction
• Technologies
• General explanations
• Useful notes

Introduction

In my thesis, I investigated hybrid fusion, i.e. the fusion of human and machine classifier outputs. For this purpose, we developed the ConfidenceApp web app in the thesis to collect our own data. The code of this application can be found at "". After our data was collected, we then compared our fusion method using the Correlated Fusion Model with the State-Of-The-Art method of Steyvers et al. As mentioned at the beginning, you can find the code we used for this in this repository.

Technologies

We had access to a high-performance computer at the Technical University of Darmstadt to carry out our experiments.

We also used the following technologies:

• Python Version: 3.9.16
• JAGS Version: 4.3.0

All other libraries used can be found in the corresponding requirements.txt.

General explanations

The code is divided into three main folders.

• Binary_Experiments
• Jupyter_Notebooks
• Reproduce_Steyvers

The first folder contains experiments we conducted with our new data we collected.

The second folder contains Jupyter notebooks that we used to analyse the experimental data and create plots.

The last folder contains the code we used to reproduce the data from Steyvers et al.

The code is mostly self-explanatory and can be run out-of-the-box with the technical requirements mentioned. In cases where the code requires further explanation, sufficient documentation can be found there.

Useful notes

It is advisable to split the human-machine combinations into pieces beforehand to allow for multiple processing. In our experiments, we did this with the sub-methods "run_1", "run_2", etc. - but these have not been included in the final code for the sake of simplicity.

It is also worth mentioning that we manually included the proportion of true-labels to be masked in the code. If the experiment should be repeated with a different proportion of samples to be masked, the corresponding places in the code have to be adjusted.