diff --git a/README.md b/README.md index ab14f131f..2af0285c0 100644 --- a/README.md +++ b/README.md @@ -3,39 +3,48 @@ [![Code Coverage](https://codecov.io/gh/RedHatInsights/patchman-engine/branch/master/graph/badge.svg)](https://codecov.io/gh/RedHatInsights/patchman-engine) # patchman-engine -System Patch Manager is one of the applications for [console.redhat.com](https://console.redhat.com). This application allows users to display and manage available patches for their registered systems. This code repo stores sources for the backend part of the application which provides the REST API to the frontend. +System Patch Manager is one of the applications for [console.redhat.com](https://console.redhat.com). This application allows users to display and manage available patches for their registered systems. This code repo stores sources for the backend part of the application, which provides the REST API for the frontend. -## Table of content +## Table of contents - [Architecture](docs/md/architecture.md) - [Database](docs/md/database.md) - [Development environment](#development-environment) - - [Local running](#local-running) - - [Local app requests](#local-app-requests) - - [Tests running](#tests-running) - - [OpenAPI docs](#openapi-docs) + - [Running locally](#running-locally) + - [Local app requests](#local-app-requests) + - [Testing and debugging](#testing-and-debugging) + - [OpenAPI docs](#openapi-docs) - [Control by private API](#control-by-private-api) - [VMaaS](#vmaas) - [Monitoring](#monitoring) -- [Profiling] (#profiling) +- [Profiling](#profiling) ## Development environment +Ensure that you have Go and Podman installed. ### Running locally -Uses `podman-compose` to deploy the individual project components and supporting containers, which simulate the CMSfR platform and database: +Use `podman-compose` to deploy the individual project components and supporting containers that simulate the CMSfR platform and database: ~~~bash podman-compose up --build # Build images if needed and start containers podman-compose down # Stop and remove containers ~~~ -Use `--profile monitoring` to run local `prometheus` and `grafana`, for example +#### Run with monitoring +Use `--profile monitoring` to run local `prometheus` and `grafana`, for example: ~~~bash podman-compose --profile monitoring up ~~~ -Grafana is accessible at `http://localhost:3000`, Prometheus at `http://localhost:9090` +Grafana is available at and Prometheus at . +#### Run a component in the host OS +Run a single component in the host OS while running the rest in podman-compose: +~~~bash +podman-compose stop evaluator_upload # stop a single component using podman-compose +export $(xargs < conf/local.env) +./scripts/entrypoint.sh evaluator # (or listener, or manager) run the component in the host OS +~~~ ### Local app requests -When podman compose is running, you can test the app using dev shell scripts: +When podman-compose is running, use dev shell scripts to test the app: ~~~bash cd dev/scripts ./systems_list.sh # show systems @@ -44,94 +53,89 @@ cd dev/scripts ./platform_upload.sh # simulate archive upload to trigger listener and evaluator_upload ~~~ -### Running in host OS -Run single component in host OS, rest in podman-compose: -~~~bash -podman-compose stop evaluator_upload # stop single component running using podman-compose -export $(xargs < conf/local.env) -./scripts/entrypoint.sh evaluator # (or listener, or manager) run component in host OS -~~~ +### Testing and debugging +A large part of the application functionality is covered with tests; this requires running a test database and mocked services. All of this is encapsulated in the test configuration, which is run using podman-compose. It also includes static code analysis, database migration tests, and a Dockerfile check. It is used to check pull requests too. -### Running tests -We cover a large part of the application functionality with tests; this requires also running a test database and mocked services. This is all encapsulated into the configuration runable using podman-compose command. It also includes static code analysis, database migration tests and dockerfiles checking. It's also used when checking pull requests for the repo. +Use this command to run the whole test suite: ~~~bash podman-compose -f docker-compose.test.yml up --build --abort-on-container-exit ~~~ -### Run single test or set of tests in containers -- Open `./scripts/go_test.sh` file -- Comment line that runs all tests -- Uncomment and modify the last line to run one or a set of tests -- Run the same command as for running all tests (from above) +#### Run one or more tests instead +1. Open `./scripts/go_test.sh` file. +2. Comment out the line that runs all tests. +3. Uncomment and modify the last line to specify one or a set of tests. +4. Run the same command as for the whole suite (from above). -### Run single test locally -After running all test suite, testing platform components are still running (kafka, platform, db). This is especially useful when fixing some test or adding a new one. You need to have golang installed. +#### Run a single test locally +After running the entire test suite (without `--abort-on-container-exit` flag), testing platform components (`kafka`, `platform`, `db`) are still up. This is especially useful when fixing a test or adding a new one. ~~~bash podman-compose -f docker-compose.test.yml up --build --no-start # build images podman-compose -f docker-compose.test.yml run test ./scripts/go_test.sh './evaluator -run TestEvaluate' # run "TestEvaluate" test from "evaluator" component ~~~ -### Run tests in VS Code - -Prerequisite is to have [Go Extension](https://marketplace.visualstudio.com/items?itemName=golang.Go) +#### Run tests in VS Code +A prerequisite is to have the [Go Extension](https://marketplace.visualstudio.com/items?itemName=golang.Go) installed. -To set it up copy the example settings from `.vscode/settings.example.json`: -``` +To set things up, copy the example settings from `.vscode/settings.example.json`: +~~~bash cp .vscode/settings.example.json .vscode/settings.json -``` - -### Access to dev/test database +~~~ -When a podman compose (either dev or test one) is running, then the database can -be access directly by executing -``` +#### Access to the dev/test database +While podman-compose (either dev or test) is running, execute the following to access the database directly: +~~~bash podman exec -it db psql -d patchman -U admin -``` +~~~ -or locally using `psql` with: -``` +or locally using `psql.sh`, as follows: +~~~bash export $(cat conf/local.env conf/database_admin.env | xargs ) 2>/dev/null; ./dev/scripts/psql.sh -``` +~~~ + +#### Kafka control +Control and inspect the Kafka instance using: +~~~bash +podman-compose exec kafka bash # enter kafka component and run inside: +/usr/bin/kafka-topics --list --bootstrap-server=kafka:9092 # show created topics + +# list all messages sent to a topic +/usr/bin/kafka-console-consumer --bootstrap-server=kafka:9092 --topic platform.inventory.events --from-beginning + +# send debugging message to a topic +echo '{"id":"00000000-0000-0000-0000-000000000002"}' | /usr/bin/kafka-console-producer --broker-list kafka:9092 --topic patchman.evaluator.upload +~~~ ### OpenAPI docs -Our REST API is documented using OpenAPI v3. On a local instance it can be accessed on . +The REST API is documented using OpenAPI v3. On a local instance, it can be accessed at . -To update/regenerate OpenAPI sources run: +For the first time, ensure that you have `swaggo/swag` binary installed: +~~~bash +go get -u github.com/swaggo/swag/cmd/swag +~~~ + +Run this command to update/regenerate OpenAPI source files: ~~~bash -go get -u github.com/swaggo/swag/cmd/swag # download binary to generate, do it first time only ./scripts/generate_docs.sh ~~~ ## Control by private API -There is a private API accessible only from inside of `vmaas_sync` container. It allows running component routines manually. In local environment it can be tested like this: +There is a private API accessible only from the `vmaas_sync` container, which allows triggering component routines manually. In a local environment, test it like this: ~~~bash -podman exec -it patchman-engine_vmaas_sync_1 ./sync.sh # trigger advisories syncing event. -podman exec -it patchman-engine_vmaas_sync_1 ./re-calc.sh # trigger systems recalculation event. +podman exec -it patchman-engine_vmaas_sync_1 ./sync.sh # trigger advisories syncing event. +podman exec -it patchman-engine_vmaas_sync_1 ./re-calc.sh # trigger systems recalculation event. podman exec -it patchman-engine_vmaas_sync_1 ./caches-check.sh # trigger account caches checking. ~~~ ## VMaaS -This project uses [VMaaS](https://github.com/RedHatInsights/vmaas) for retrieving information about advisories, and resolving which advisories can be applied to which systems. For local development this repo contains VMaaS service mock as a part of platform mock allowing independent running of the service using podman-compose. +This project uses [VMaaS](https://github.com/RedHatInsights/vmaas) for retrieving information about advisories and resolving which advisories can be applied to which systems. For local development, the platform mock contains a VMaaS service mock. ## Monitoring Each application component (except for the database) exposes metrics for [Prometheus](https://prometheus.io/) on `/metrics` endpoint (see [docker-compose.yml](docker-compose.yml) for ports). Runtime logs can be sent to Amazon CloudWatch if configuration environment variables are set (see [awscloudwatch.go](base/utils/awscloudwatch.go)). -## Kafka control -Your can control and inspect def Kafka instance using: -~~~bash -docker-compose exec kafka bash # enter kafka component and run inside: -/usr/bin/kafka-topics --list --bootstrap-server=kafka:9092 # show created topics - -# list all messages send to a topic -/usr/bin/kafka-console-consumer --bootstrap-server=kafka:9092 --topic platform.inventory.events --from-beginning - -# send debugging message to a topic -echo '{"id":"00000000-0000-0000-0000-000000000002"}' | /usr/bin/kafka-console-producer --broker-list kafka:9092 --topic patchman.evaluator.upload -~~~ - ## Run SonarQube code analysis ~~~bash export SONAR_HOST_URL=https://sonar-server @@ -141,23 +145,24 @@ podman-compose -f dev/sonar/docker-compose.yml up --build ~~~ ## Update Grafana config map -Copy Grafana board json config to the temporary file, e.g. `grafana.json` and run: +Copy Grafana board JSON config to a temporary file (e.g. `grafana.json`) and run: ~~~bash ./scripts/grafana-json-to-yaml.sh grafana.json > ./dashboards/app-sre/grafana-dashboard-insights-patchman-engine-general.configmap.yaml ~~~ ## Profiling -App can be profiled using [/net/http/pprof](https://pkg.go.dev/net/http/pprof). Profiler is exposed on app's private port. +The app can be profiled using [/net/http/pprof](https://pkg.go.dev/net/http/pprof). Profiler is exposed on app's private port. + ### Local development -- set `ENABLE_PROFILE=true` in the `cont/common.env` -- `docker-compose up --build` -- `go tool pprof http://localhost:{port}/debug/pprof/{heap|profile|block|mutex}` -available ports: -- 9000 - manager -- 9002 - listener -- 9003 - evaluator-upload -- 9004 - evaluator-recalc -### Admin API -- set `ENABLE_PROFILE_{container_name}=true` in the ClowdApp -- download the profile file using internal api `/api/patch/admin/pprof/{manager|listener|evaluator_upload|evaluator_recalc}/{heap|profile|block|mutex|trace}` -- `go tool pprof ` +1. Set `ENABLE_PROFILE=true` in the `conf/common.env`. +2. Run `podman-compose up --build`. +3. Run `go tool pprof http://localhost:{port}/debug/pprof/{heap|profile|block|mutex}` with: + - `9000` - manager, + - `9002` - listener, + - `9003` - evaluator-upload, or + - `9004` - evaluator-recalc. + +### Using Admin API +1. Set `ENABLE_PROFILE_{container_name}=true` in the ClowdApp. +2. Download the profile file using internal API `/api/patch/admin/pprof/{manager|listener|evaluator_upload|evaluator_recalc}/{heap|profile|block|mutex|trace}`. +3. Run `go tool pprof `.