Test network often does not start up on first try

[bzp99]

Many have reported that the test environment startup fails on the initial docker compose up (some services don’t start) but a couple more docker compose up invocations fix the issue.

The underlying problem is that all services have healthchecks configured and interdependent services wait for one another to be healthy, not just started. The healthchecks impose timeouts: for example, a check could try pinging the service 3 times, each time waiting 15 seconds for a response (in realitly, there are more parameters). This means that if the service does not reach a ‘healthy’ state (meaning it replies to the ping) in 45 seconds, Docker Compose will give up and consider that service failed and cascade this failure to dependent services. In the meantime, the container is still setting up, so by the time you issue a second docker compose up, the container might be ready to accept connections and respond to the healthcheck ping.

I would keep healthchecks but relax their timeouts.

Rough outline of subtasks:

• Ensure that all services really do have healthchecks configured
• Document the healthchecks (especially that some of them are in the respective Dockerfiles and some are defined in the Compose YAMLs)
• Relax healthcheck timeouts (needs a bit of experimentation and guesswork)
• Document that starting the test environment might fail due to such timeouts and what to do in this case