CommCare Connect

CommCare Connect

Local setup

This assumes you will use the docker compose file in this repo to run services. If that is not the case you may need to edit some settings.

# install uv to manage python versions and dependencies
# https://docs.astral.sh/uv/getting-started/installation/
$ curl -LsSf https://astral.sh/uv/install.sh | sh

# create a venv and install requirements
$ uv sync

# This will also create a virtual environment in .venv if
# one does not already exist. To activate the environment:
source .venv/bin/activate

# For convenience, you can create an alias to activate
# virtual environments in ".venv" and "venv" directories.
# To do that, add the following to your .bashrc or .zshrc file (optional):
alias venv='if [[ -d .venv ]] ; then source .venv/bin/activate ; elif [[ -d venv ]] ; then source venv/bin/activate ; fi'

# Then you can activate virtual environments with
venv

# install git hooks
$ prek install
$ prek run -a

# create env file and edit the settings as needed (or export settings directly)
$ cp .env_template .env

# command for installing new python dependencies
$ uv add <pkg>

GeoDjango / PostGIS Setup

Connect uses GeoDjango and PostGIS. To learn more about the required dependencies, please refer to the official Django documentation: https://docs.djangoproject.com/en/6.0/ref/contrib/gis/install/geolibs/#geolibs-list

For macOS (Intel / Apple Silicon) – using Homebrew:

brew install gdal geos proj

Django may struggle to find the GDAL and GEOS libraries even after they are installed. Follow these steps to set your paths:

1. Get your local library paths

# Get GDAL path
ls $(brew --prefix gdal)/lib/libgdal.dylib

# Get GEOS path
ls $(brew --prefix geos)/lib/libgeos_c.dylib

2. Set it in your .env file

GDAL_LIBRARY_PATH=...
GEOS_LIBRARY_PATH=...

Install WeasyPrint on macOS (https://doc.courtbouillon.org/weasyprint/stable/first_steps.html#macos)

brew install weasyprint

For Ubuntu / Debian Linux:

sudo apt-get install -y binutils libproj-dev gdal-bin

Enabling the PostGIS extension

No migration creates the postgis extension. The inv up setup works because the postgis/postgis image enables it for you when it initialises the database.

If you are using your own PostgreSQL instead, create the extension in the target database before migrating, or the first spatial migration will fail:

psql -d commcare_connect -c "CREATE EXTENSION IF NOT EXISTS postgis;"

# start docker services
$ inv up

# install JS dependencies
$ npm ci

# build JS (optionally watch files for changes and rebuild)
$ inv build-js [-w]

# run Django
$ ./manage.py migrate
$ ./manage.py runserver

Basic Commands

Some useful command are available via the tasks.py file:

$ inv -l

Setting up auth with CommCare HQ

Expose your local service to the internet

• Create an account on ngrok and Install ngrok

  • https://dashboard.ngrok.com/get-started/setup/

• Create a custom domain on ngrok using this link

• Run ngrok http --url=[my-unique-subdomain].ngrok-free.app 8000

• Update your .env file with the host:

  DJANGO_ALLOWED_HOSTS=[my-unique-subdomain].ngrok-free.app
  

Create an OAuth2 application on CommCare HQ

• Navigate to https://staging.commcarehq.org/oauth/applications/
• Create a new application with the following settings:
  • Client Type: Confidential
  • Authorization Grant Type: Authorization Code
  • Redirect URIs: https://[my-unique-subdomain].ngrok-free.app/accounts/commcarehq/login/callback/
  • Pkce required: False
• Copy the Client ID and Client Secret
• Create a new SocialApp locally at http://localhost:8000/admin/socialaccount/socialapp/

Test the OAuth2 flow

• Confirm COMMCARE_HQ_URL points at https://staging.commcarehq.org. That is the default, so you only need to set it in .env if you have overridden it.
• Navigate to http://[my-unique-subdomain].ngrok-free.app/accounts/login/
• Click the "Log in with CommCare HQ" button
• You should be redirected to CommCare HQ to log in
• After logging in, you should be redirected back to the app and logged in

Setting Up Your Users

No social app setup is needed: the "Log in with CommCare HQ" button only appears once a commcarehq social app exists (see Setting up auth with CommCare HQ).

• To create a superuser, run the command below. --email is not accepted, because the custom User model sets REQUIRED_FIELDS = []; set the address afterwards in the admin if you need one.

  $ ./manage.py createsuperuser
  

• To create a normal user account, go to Sign Up and fill out the form. Local settings use ACCOUNT_EMAIL_VERIFICATION = "optional", so you are logged straight in; the confirmation email is still printed to the console if you want to verify the address.

• To promote an existing user to superuser, use this command:

  $ ./manage.py promote_user_to_superuser <email>
  

For convenience, you can keep your normal user logged in on Chrome and your superuser logged in on Firefox (or similar), so that you can see how the site behaves for both kinds of users.

Sample Data

To populate a local database with organizations, opportunities, visits and payments:

# generate_sample_data <num_visits> <org_slug>
$ ./manage.py generate_sample_data 50 demo-org

The organization is created if the slug does not already exist. Each run first deletes the existing opportunities, visits, payments and programs belonging to the two organizations it is about to use. Two optional flags are available:

• --invited_org_slug — reuse a specific organization as the invited org. Without it, a new invited org with a random slug is created on every run, and the previous one is left behind.
• --managed_opportunities — how many managed opportunities to create (default 3)

Adding yourself to the generated organization is a separate step, either through /admin/organization/userorganizationmembership/ or the organization's own member admin pages.

Test coverage

To run the tests, check your test coverage, and generate an HTML coverage report:

$ coverage run -m pytest
$ coverage html
$ open htmlcov/index.html

Running tests with pytest

$ pytest

Live reloading and Sass CSS compilation

$ inv build-js -w

Celery

This app comes with Celery.

To run a celery worker:

celery -A config.celery_app worker -l info

Please note: For Celery's import magic to work, it is important where the celery commands are run. If you are in the same folder with manage.py, you should be right.

To run periodic tasks, you'll need to start the celery beat scheduler service. You can start it as a standalone process:

celery -A config.celery_app beat

or you can embed the beat service inside a worker with the -B option (not recommended for production use):

celery -A config.celery_app worker -B -l info

Deployment

The application runs as Docker containers on EC2, deployed with Kamal. Ansible provisions the instances and manages the container env files. See deploy/README.md for the full picture, including the tooling you need installed to deploy or change Django settings.

Deploying a new version of the app can be done via the "Deploy" workflow on GitHub Actions.

Container logs are shipped to CloudWatch, and can also be read directly with cd deploy && kamal app logs.

For details on how this action is configured see:

• https://aws.amazon.com/blogs/security/use-iam-roles-to-connect-github-actions-to-actions-in-aws/
• https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services

Deploying to the staging environment

The project has a staging environment at https://connect-staging.dimagi.com/, which is connected to the staging environment of CommCare HQ at https://staging.commcarehq.org/.

• Update commcare-connect-staging.yml with the branches you need to include.
• Run deploy/rebuildstaging to build the autostaging branch
• After this, you can deploy to the staging environment by manually running the deploy workflow from here.

Remote Access to machines

Machines can be accessed via SSH using AWS SSM. In order for that to work, you will need to add the following lines to your ssh config (typically located at ~/.ssh/config)

Host i-*
     ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p' --region us-east-1 --profile commcare-connect"

This remote access is required for the ansible commands in the tasks.py file.

Setting up logical replication

• Create another postgres database (in a seperate cluster)
• Add SECONDARY_DATABASE_URL to the .env file
• Enable replication on the default database (requires restart)

        - "wal_level=logical"
        - "max_replication_slots=5"
        - "max_wal_senders=5"
  

• Create table schemas in the secondary database

  ./manage.py migrate_multi
  

• Initialize replication

  ./manage.py setup_logical_replication