Data Cleanup Assistant

A guided, four-step data quality wizard: Import & Review → Automatic Checks → Clean Values → Find Outliers. Nothing is changed to your data until you explicitly apply a fix.

Local run (for testing before deployment)

python -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate
pip install -r requirements.txt
streamlit run app.py

Visit http://localhost:8501.

Self-hosted deployment

Option A — Docker (recommended)

docker build -t data-cleanup-assistant .
docker run -d -p 8501:8501 --name dqa data-cleanup-assistant

Then put a reverse proxy (nginx, Caddy, Traefik) in front of port 8501 for TLS and a proper domain, e.g.:

server {
    listen 443 ssl;
    server_name dqa.yourdomain.org;

    location / {
        proxy_pass http://localhost:8501;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
    }
}

Option B — Bare metal / VM with systemd

python -m venv /opt/dqa/.venv
/opt/dqa/.venv/bin/pip install -r requirements.txt

Create /etc/systemd/system/dqa.service:

[Unit]
Description=Data Cleanup Assistant
After=network.target

[Service]
User=www-data
WorkingDirectory=/opt/dqa
ExecStart=/opt/dqa/.venv/bin/streamlit run app.py --server.port=8501 --server.address=127.0.0.1
Restart=always

[Install]
WantedBy=multi-user.target

sudo systemctl enable --now dqa

Then reverse-proxy the same way as Option A.

Configuration

• .streamlit/config.toml sets the theme (purple #484898 branding) and a 50 MB upload limit. Adjust [server] maxUploadSize if you need to accept larger files.
• No secrets are required for the current feature set. If you later add a KoboToolbox or Google Drive API connection, put credentials in .streamlit/secrets.toml (already git-ignored) and read them via st.secrets["your_key"] — never hardcode them in app.py.

Project structure

app.py                 # main wizard (4 steps)
utils/validators.py     # data quality check functions (missing values,
                         # duplicates, outliers via IQR/MAD, category checks,
                         # multi-select parsing for KoboToolbox exports)
requirements.txt
Dockerfile
.streamlit/config.toml

Known limitations (current version)

• Single-user session state — no shared multi-user accounts or auth yet. Add a reverse-proxy auth layer (e.g. nginx basic auth, OAuth2 Proxy) if this needs to sit behind a login for your team.
• Tested comfortably up to tens of thousands of rows; very large files (millions of rows) will need chunked processing, which isn't implemented.
• The "Find Outliers" step currently supports numeric variables only. KoboToolbox multi-select and categorical consistency checks exist in utils/validators.py but aren't wired into the UI yet.