fix: record no event when deleting a missing software

CHANGELOG.md

@@ -13,23 +13,44 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   privately and which versions get fixes.
 - CI checks for the Helm chart with `helm lint` and `kubeconform`, and for
   GitHub Actions workflows with `actionlint`.
+- `entityType`, `entityId`, `type` and `actor` filters on `GET /v1/events`.
+- An `extraEnv` value on the Helm chart, for environment variables
+  without a dedicated chart value.
 
 ### Changed
 
 - JSON Patch operations on `id`, `createdAt`, `updatedAt` and
   `catalogId` are refused with `422`, as they already were in a JSON
   merge patch. `test` operations on those fields are still allowed.
+- Every `500` carries the same generic detail. A few answered `db error`.
+- The events model no longer has a `deleted_at` column. An existing
+  database keeps the column, unused.
 
 ### Fixed
 
 - `?search` on a list other than the logs answered `500`. The parameter
   is now ignored there.
 - The Helm chart now renders a valid `autoscaling/v2` HPA and quotes the
   numeric `MAX_REQUESTS` environment value.
+- Concurrent `PATCH` requests to the `analysis` endpoint of the same software
+  or catalog no longer discard namespaces written by other scanners.
 - A JSON Patch replacing the `id` of a software answered `200` but did
   not save the software.
 - A JSON Patch could store a malformed URL or email. The patched entity
   now goes through the same validation as a `POST`.
+- A `DELETE` of a missing software answered `404` but still recorded a
+  delete event and notified the webhooks.
+- The event recorded by an analysis `PATCH` had no actor.
+- Catalogs, bundles and webhooks wrote no event to the audit trail.
+- A webhook `POST` with the url of an existing one answered `500` instead
+  of `409`.
+- The `409` for a duplicate catalog `alternativeId` now names the field.
+- An error the API did not classify answered `404` with the raw error
+  text. It is now a `500` with a generic detail.
+- Deleting a publisher or software did not delete its logs.
+- The `filter` parameter of `GET /v1/logs` was missing from the OpenAPI
+  spec, and the README named a Go version older than the one required.
+- The sources of a patched catalog were returned in a random order.
 
 ### Security
 

README.md

@@ -9,8 +9,8 @@
 <p align="center">
   <a href="https://www.bestpractices.dev/projects/14158">
     <img src="https://www.bestpractices.dev/projects/14158/badge">
-  </a> 
-  
+  </a>
+
   <img alt="License" src="https://img.shields.io/github/license/publiccodeyml/open-catalog-api?color=brightgreen">
 </p>
 
@@ -23,7 +23,8 @@
 </div>
 
 <p align="center">
-  <strong>Open Catalog API</strong> is a RESTful API for public software catalogs.
+  <strong>Open Catalog API</strong> is a RESTful API for public software
+  catalogs.
   It powers national open source catalogs for public administrations:
 </p>
 
@@ -32,7 +33,7 @@
 
 ## Requirements
 
-* Golang 1.25+
+* Golang 1.27+
 * [PostgreSQL](https://www.postgresql.org/)
 
 ## Development
@@ -48,8 +49,8 @@ To start developing:
 
 Docker Compose will bring up the app and PostgreSQL containers.
 
-Wait until the Docker logs explicitly say the API is up and you can use its
-endpoints at `http://localhost:3000/v1/`.
+Wait until the Docker logs explicitly say the API is up and you can use
+its endpoints at `http://localhost:3000/v1/`.
 
 ## Configuration
 
@@ -73,11 +74,12 @@ You can configure the API with environment variables:
 
   If not set, the API will run in read only mode.
 
-* `ENVIRONMENT` (optional): possible values `test`, `development`, `production`.
+* `ENVIRONMENT` (optional): `test` turns the rate limiter and the event
+  purge off, for the test suite. Any other value is a normal run.
   Default `production`.
 
-* `MAX_REQUESTS` (optional): number of requests per minute after which responses
-  will be ratelimited.
+* `MAX_REQUESTS` (optional): number of requests per minute after which
+  responses will be ratelimited.
   Default: no limit.
 
 * `TRUSTED_PROXIES` (optional): comma separated list of addresses or
@@ -110,8 +112,8 @@ You can configure the API with environment variables:
 
 ## Contributing
 
-This project exists also thanks to your contributions! Here is a list of people
-who already contributed to this repository:
+This project exists also thanks to your contributions! Here is a list of
+people who already contributed to this repository:
 
 <a href="https://github.com/publiccodeyml/open-catalog-api/graphs/contributors">
   <img

catalogs_test.go

@@ -173,7 +173,7 @@ func TestCatalogEndpoints(t *testing.T) {
 			},
 			expectedCode:        409,
 			expectedContentType: "application/problem+json",
-			expectedBody:        `{"title":"can't create Catalog","detail":"already exists","status":409}`,
+			expectedBody:        `{"title":"can't create Catalog","detail":"alternativeId already exists","status":409}`,
 		},
 		{
 			description: "POST catalog missing name",
@@ -841,6 +841,10 @@ func TestCatalogDeleteDBChecks(t *testing.T) {
 
 		assert.Equal(t, 0, dbCount(t, "catalogs", "id", catalogID))
 		assert.Equal(t, 0, dbCount(t, "catalog_sources", "catalog_id", catalogID))
+
+		// The fixtures carry two delete events, the catalog adds a third.
+		assert.Equal(t, 2, dbCount(t, "events", "entity_id", catalogID))
+		assert.Equal(t, 3, dbCount(t, "events", "type", "delete"))
 	})
 }
 
@@ -905,6 +909,40 @@ func TestCatalogSourcesDBChecks(t *testing.T) {
 		assert.Equal(t, "gitlab", dbValue(t, "catalog_sources", "driver", "catalog_id", italiaID))
 	})
 
+	t.Run("PATCH returns the sources sorted by url", func(t *testing.T) {
+		loadFixtures(t)
+
+		body := `{"sources":[{"url":"https://c.example.org/one"},{"url":"https://a.example.org/two"},{"url":"https://b.example.org/three"}]}`
+		req, err := newTestRequest("PATCH", "/v1/catalogs/"+italiaID, strings.NewReader(body))
+		require.NoError(t, err)
+		req.Header = map[string][]string{
+			"Authorization": {goodToken},
+			"Content-Type":  {"application/json"},
+		}
+
+		res, err := app.Test(req, -1)
+		require.NoError(t, err)
+		require.Equal(t, 200, res.StatusCode)
+
+		var response struct {
+			Sources []struct {
+				URL string `json:"url"`
+			} `json:"sources"`
+		}
+		require.NoError(t, json.NewDecoder(res.Body).Decode(&response))
+
+		urls := make([]string, 0, len(response.Sources))
+		for _, source := range response.Sources {
+			urls = append(urls, source.URL)
+		}
+
+		assert.Equal(t, []string{
+			"https://a.example.org/two",
+			"https://b.example.org/three",
+			"https://c.example.org/one",
+		}, urls)
+	})
+
 	t.Run("PATCH persists args as JSON", func(t *testing.T) {
 		loadFixtures(t)
 

charts/open-catalog-api/Chart.yaml

@@ -18,7 +18,7 @@ kubeVersion: ">=1.23.0-0"
 # This is the chart version. This version number should be incremented each time you make changes
 # to the chart and its templates, including the app version.
 # Versions are expected to follow Semantic Versioning (https://semver.org/)
-version: 0.7.0
+version: 0.8.0
 
 # This is the version number of the application being deployed. This version number should be
 # incremented each time you make changes to the application. Versions are not expected to

charts/open-catalog-api/README.md

@@ -1,6 +1,6 @@
 # open-catalog-api
 
-![Version: 0.7.0](https://img.shields.io/badge/Version-0.7.0-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: v1.4.0](https://img.shields.io/badge/AppVersion-v1.4.0-informational?style=flat-square)
+![Version: 0.8.0](https://img.shields.io/badge/Version-0.8.0-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: v1.4.0](https://img.shields.io/badge/AppVersion-v1.4.0-informational?style=flat-square)
 
 RESTful API for public administration Open Source catalogs
 
@@ -15,6 +15,7 @@ RESTful API for public administration Open Source catalogs
 | autoscaling.targetCPUUtilizationPercentage | int | `80` |  |
 | databaseDSN | string | `""` | Database connection string, e.g. "host= port=5432 dbname= user= password= sslmode=require". |
 | deploymentAnnotations | object | `{}` |  |
+| extraEnv | list | `[]` | Additional environment variables for the API container, as Kubernetes EnvVar objects, e.g. TRUSTED_PROXIES or EVENT_RETENTION_DAYS. |
 | extraVolumeMounts | list | `[]` |  |
 | extraVolumes | list | `[]` |  |
 | fullnameOverride | string | `""` |  |

charts/open-catalog-api/templates/deployment.yaml

@@ -58,6 +58,9 @@ spec:
                 secretKeyRef:
                   name: {{ default (include "open-catalog-api.fullname" .) .Values.useExistingSecret }}
                   key: pasetoKey
+            {{- with .Values.extraEnv }}
+            {{- toYaml . | nindent 12 }}
+            {{- end }}
           livenessProbe:
             httpGet:
               path: /livez

charts/open-catalog-api/values.yaml

@@ -116,3 +116,6 @@ serviceMonitor:
 extraVolumes: []
 
 extraVolumeMounts: []
+
+# -- (list) Additional environment variables for the API container, as Kubernetes EnvVar objects, e.g. TRUSTED_PROXIES or EVENT_RETENTION_DAYS.
+extraEnv: []

events_actor_test.go

@@ -67,6 +67,81 @@ func TestEventActor(t *testing.T) {
 			expectedCode:  200,
 			expectedActor: "editor",
 		},
+		{
+			description:   "POST catalog",
+			method:        http.MethodPost,
+			path:          "/v1/catalogs",
+			body:          `{"name":"actor test catalog","sources":[{"url":"https://actor-test.example.org/catalog"}]}`,
+			subject:       "curator",
+			expectedCode:  200,
+			expectedActor: "curator",
+		},
+		{
+			description:   "PATCH catalog",
+			method:        http.MethodPatch,
+			path:          catalogPath + italiaID,
+			body:          `{"name":"actor test patched catalog"}`,
+			subject:       "curator",
+			entityID:      italiaID,
+			expectedCode:  200,
+			expectedActor: "curator",
+		},
+		{
+			description:   "POST bundle",
+			method:        http.MethodPost,
+			path:          "/v1/bundles",
+			body:          `{"name":"actor test bundle","softwareIds":["` + swissSoftwareID + `"]}`,
+			subject:       "curator",
+			expectedCode:  200,
+			expectedActor: "curator",
+		},
+		{
+			description:   "PATCH bundle",
+			method:        http.MethodPatch,
+			path:          "/v1/bundles/" + fixtureBundleID,
+			body:          `{"name":"actor test patched bundle"}`,
+			subject:       "curator",
+			entityID:      fixtureBundleID,
+			expectedCode:  200,
+			expectedActor: "curator",
+		},
+		{
+			description:   "DELETE bundle",
+			method:        http.MethodDelete,
+			path:          "/v1/bundles/" + fixtureBundleID,
+			subject:       "curator",
+			entityID:      fixtureBundleID,
+			expectedCode:  204,
+			expectedActor: "curator",
+		},
+		{
+			description:   "POST webhook",
+			method:        http.MethodPost,
+			path:          "/v1/software/webhooks",
+			body:          `{"url":"https://actor-test.example.org/hook"}`,
+			subject:       "operator",
+			expectedCode:  200,
+			expectedActor: "operator",
+		},
+		{
+			description:   "DELETE webhook",
+			method:        http.MethodDelete,
+			path:          "/v1/webhooks/24bc1b5d-fe81-47be-9d55-910f820bdd04",
+			subject:       "operator",
+			entityID:      "24bc1b5d-fe81-47be-9d55-910f820bdd04",
+			expectedCode:  204,
+			expectedActor: "operator",
+		},
+		{
+			description:   "PATCH software analysis",
+			method:        http.MethodPatch,
+			path:          softwarePath + swissSoftwareID + "/analysis",
+			body:          `{"actor-test":{"v":1}}`,
+			subject:       "scanner",
+			entityID:      swissSoftwareID,
+			expectedCode:  200,
+			expectedActor: "scanner",
+		},
 		{
 			description:   "PATCH software",
 			method:        http.MethodPatch,

events_test.go

@@ -170,6 +170,88 @@ func TestEventsEndpoints(t *testing.T) {
 		},
 
 		// GET /events/:id
+		{
+			description:         "GET events of one entity type",
+			query:               "GET /v1/events?entityType=publishers",
+			headers:             authHeaders,
+			expectedCode:        200,
+			expectedContentType: "application/json",
+			validateFunc: func(t *testing.T, response map[string]any) {
+				data := assertListResponse(t, response)
+				assert.Len(t, data, 3)
+
+				for _, event := range data {
+					assert.Equal(t, "publishers", event["entityType"])
+				}
+			},
+		},
+		{
+			description:         "GET events of one entity",
+			query:               "GET /v1/events?entityId=c353756e-8597-4e46-a99b-7da2e141603b",
+			headers:             authHeaders,
+			expectedCode:        200,
+			expectedContentType: "application/json",
+			validateFunc: func(t *testing.T, response map[string]any) {
+				data := assertListResponse(t, response)
+				assert.Len(t, data, 3)
+
+				for _, event := range data {
+					assert.Equal(t, "c353756e-8597-4e46-a99b-7da2e141603b", event["entityId"])
+				}
+			},
+		},
+		{
+			description:         "GET events of one type",
+			query:               "GET /v1/events?type=update",
+			headers:             authHeaders,
+			expectedCode:        200,
+			expectedContentType: "application/json",
+			validateFunc: func(t *testing.T, response map[string]any) {
+				data := assertListResponse(t, response)
+				assert.Len(t, data, 3)
+
+				for _, event := range data {
+					assert.Equal(t, "update", event["type"])
+				}
+			},
+		},
+		{
+			description:         "GET events of one actor",
+			query:               "GET /v1/events?actor=crawler",
+			headers:             authHeaders,
+			expectedCode:        200,
+			expectedContentType: "application/json",
+			validateFunc: func(t *testing.T, response map[string]any) {
+				data := assertListResponse(t, response)
+				assert.Len(t, data, 4)
+
+				for _, event := range data {
+					assert.Equal(t, "crawler", event["actor"])
+				}
+			},
+		},
+		{
+			description:         "GET events with two filters",
+			query:               "GET /v1/events?actor=editor&entityType=publishers",
+			headers:             authHeaders,
+			expectedCode:        200,
+			expectedContentType: "application/json",
+			validateFunc: func(t *testing.T, response map[string]any) {
+				data := assertListResponse(t, response)
+				assert.Len(t, data, 2)
+
+				// The filters survive in the pagination links.
+				assertPaginationLinks(t, response, nil, nil)
+			},
+		},
+		{
+			description:         "GET events with an unknown type",
+			query:               "GET /v1/events?type=rename",
+			headers:             authHeaders,
+			expectedCode:        422,
+			expectedBody:        `{"title":"can't get Events","detail":"type must be one of create, update, delete","status":422}`,
+			expectedContentType: "application/problem+json",
+		},
 		{
 			description:         "GET event without a token",
 			query:               "GET /v1/events/" + eventWithActorID,