[Bug] Unable to use TLS with the Operator

[galvesribeiro]

What happened:

The HC operator do not work if healthChecksScheme is https as advertised in the documentation here.

What you expected to happen:

When setting that flag in the HealthCheck CR to https, that it can talk with the monitored services using TLS correctly, assuming that the certificate chain is trusted.

How to reproduce it (as minimally and precisely as possible):

Have a CA established and trusted, for example, use cert-manager as the CA and issuer of your application and enable TLS on the app. Then use trust-manager or whatever other mean to inject the CA cert as trusted.

Then, deploy the operator as normal with the HealthCheck CR having spec.healthChecksPath set to https.

You will have something like this on the UI deployment:

Configuring image to work with InMemory provider
warn: Microsoft.AspNetCore.DataProtection.Repositories.FileSystemXmlRepository[60]
      Storing keys in a directory '/root/.aspnet/DataProtection-Keys' that may not be persisted outside of the container. Protected data will be unavailable when container is destroyed.
info: HealthChecks.UI.Core.HostedService.UIInitializationHostedService[0]
      Initializing UI Database
warn: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[35]
      No XML encryptor configured. Key {8490d4d2-864b-42f7-8bc1-ab181ed36e47} may be persisted to storage in unencrypted form.
info: HealthChecks.UI.Image.Program[0]
      HealthChecks Push Endpoint Enabled

And something like this on the operator:

[23:13:25 INF] Starting Kubernetes client using host: https://172.21.0.1:443/
[23:13:25 INF] The operator is starting
[23:13:28 INF] Creating secret for hc resource - namespace healthchecks
[23:13:29 INF] Secret healthcheck-ui-secret has been created
[23:13:29 INF] Creating deployment for hc resource - namespace healthchecks
[23:13:29 INF] The UI Path UiPath has been configured to /
[23:13:30 INF] Deployment healthcheck-ui-deploy has been created
[23:13:30 INF] Creating service for hc resource - namespace healthchecks
[23:13:30 INF] Adding annotation app.kubernetes.io/name to ui service with value healthchecks-ui
[23:13:30 INF] Adding annotation environment to ui service with value system
[23:13:30 INF] Service healthcheck-ui-svc has been created
[23:13:30 INF] The UI replica healthcheck-ui-deploy in healthchecks is not available yet, retrying...1/10
[23:13:35 INF] Service watcher started for namespace All
[23:13:36 INF] [PushService] Namespace healthchecks - Sending Type: Added - Service configuration with uri : https://172.21.237.79:443/healthz to ui endpoint: https://172.21.86.126:80
[23:13:36 INF] Start processing HTTP request POST https://172.21.86.126:80/healthchecks/push?key=d966288b-7980-4ae7-8bbb-655b54812e7e
[23:13:36 INF] Sending HTTP request POST https://172.21.86.126:80/healthchecks/push?key=d966288b-7980-4ae7-8bbb-655b54812e7e
[23:13:36 ERR] Error notifying healthcheck service: The SSL connection could not be established, see inner exception.

If you remove the TLS, you remove the flag or set to http, then you have this in the operator:

[23:38:45 INF] Starting Kubernetes client using host: https://172.21.0.1:443/
[23:38:46 INF] The operator is starting
[23:38:48 INF] Creating secret for hc resource - namespace healthchecks
[23:38:49 INF] Creating deployment for hc resource - namespace healthchecks
[23:38:49 INF] Creating service for hc resource - namespace healthchecks
[23:38:50 INF] Service watcher started for namespace All
[23:38:51 INF] [PushService] Namespace healthchecks - Sending Type: Added - Service configuration with uri : http://172.21.237.79:443/healthz to ui endpoint: http://172.21.86.126:80
[23:38:52 INF] Start processing HTTP request POST http://172.21.86.126/healthchecks/push?key=d966288b-7980-4ae7-8bbb-655b54812e7e
[23:38:52 INF] Sending HTTP request POST http://172.21.86.126/healthchecks/push?key=d966288b-7980-4ae7-8bbb-655b54812e7e
[23:38:52 INF] Received HTTP response headers after 291.1615ms - 200
[23:38:52 INF] End processing HTTP request after 394.3837ms - 200
[23:38:52 INF] [PushService] Notification result for configuration - status code: OK

And this on the UI deployment:

      Storing keys in a directory '/root/.aspnet/DataProtection-Keys' that may not be persisted outside of the container. Protected data will be unavailable when container is destroyed.
info: HealthChecks.UI.Core.HostedService.UIInitializationHostedService[0]
      Initializing UI Database
warn: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[35]
      No XML encryptor configured. Key {8b9b60c7-cb74-4666-a929-c3511dbf8518} may be persisted to storage in unencrypted form.
info: HealthChecks.UI.Image.Program[0]
      HealthChecks Push Endpoint Enabled
info: HealthChecks.UI.Image.PushService.HealthChecksPushService[0]
      [Push] New service added: configuration with uri: http://172.21.237.79:443/healthz
fail: HealthChecks.UI.Core.HostedService.HealthCheckReportCollector[0]
      GetHealthReport threw an exception when trying to get report from http://172.21.237.79:443/healthz configured with name configuration.
      System.Net.Http.HttpRequestException: An error occurred while sending the request.
       ---> System.IO.IOException: The response ended prematurely.
         at System.Net.Http.HttpConnection.FillAsync(Boolean async)
         at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean async, Boolean foldedHeadersAllowed)
         at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         --- End of inner exception stack trace ---
         at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
         at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
         at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
         at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)
         at HealthChecks.UI.Core.HostedService.HealthCheckReportCollector.GetHealthReport(HealthCheckConfiguration configuration) in /src/src/HealthChecks.UI/Core/HostedService/HealthCheckReportCollector.cs:line 101

In other words, the request is attempted to be made as HTTP to the HTTPS endpoint

Anything else we need to know?:

The flag for HTTPS should theoretically be used when requesting the /healthz endpoint on the services. However, when setting it to https the communication between the operator and the UI pod is also being forced to be HTTPS which will obviously fail since it doesn't have a valid certificate.

I could issue a TLS certificate with cert-manager and mount into the UI Pod as long as inject the CA on the operator IF the PodSpec was made available into the HealthCheck CRD or some way to mount volumes, so we can map TLS certificates inside the pod for the UI. However, even if that is possible, we sill need to be able to pass configuration to the application to use those certs.

In other words, unless I am missing something based on the little documentation we have available, the HTTPS flag is useless.

Environment:

• .NET Core version: .Net 10
• Healthchecks version: 9.0.0 (Nuget on the services being monitored) / 5.0.0 (operator)
• Operative system: Linux

[galvesribeiro]

I'll leave it open to make awareness but I was able to workaround differently. I ditched the operator since it is useless right now, and deployed the UI alone in kubernetes and wrapped it using the appsettings.json as a configMap. The root CA is injected thru cert-manager/trust-manager. Maybe it will help someone having the same issue.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: healthchecks-ui
  namespace: healthchecks
  labels:
    app.kubernetes.io/name: healthchecks-ui
    environment: system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: healthchecks-ui
  template:
    metadata:
      labels:
        app: healthchecks-ui
        app.kubernetes.io/name: healthchecks-ui
        environment: system
    spec:
      volumes:
        - name: config
          configMap:
            name: healthchecks-ui-config
            defaultMode: 420
      containers:
        - name: healthchecks-ui
          image: xabarilcoding/healthchecksui:5.0.0
          ports:
            - name: http
              containerPort: 80
              protocol: TCP
          env:
            - name: ui_path
              value: /
            - name: storage_provider
              value: InMemory
          resources:
            limits:
              cpu: 500m
              memory: 512Mi
            requests:
              cpu: 50m
              memory: 256Mi
          volumeMounts:
            - name: config
              readOnly: true
              mountPath: /app/appsettings.json
              subPath: appsettings.json
          livenessProbe:
            httpGet:
              path: /
              port: 80
              scheme: HTTP
            initialDelaySeconds: 10
            timeoutSeconds: 1
            periodSeconds: 30
            successThreshold: 1
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /
              port: 80
              scheme: HTTP
            initialDelaySeconds: 5
            timeoutSeconds: 1
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 3
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      serviceAccountName: healthchecks-ui
      serviceAccount: healthchecks-ui
      securityContext: {}
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%
      maxSurge: 25%
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: healthchecks-ui-config
  namespace: healthchecks
spec: {}
data:
  appsettings.json: |-
    {
      "Logging": {
        "LogLevel": {
          "Default": "Information",
          "Microsoft": "Warning",
          "Microsoft.Hosting.Lifetime": "Information",
          "HealthChecks": "Information"
        }
      },
      "AllowedHosts": "*",
      "HealthChecksUI": {
        "HealthChecks": [],
        "Webhooks": [],
        "EvaluationTimeInSeconds": 30,
        "MinimumSecondsBetweenFailureNotifications": 300,
        "KubernetesDiscoveryService": {
          "Enabled": true,
          "ServicesLabel": "my/healthcheck",
          "ServicesSchemeAnnotation": "my/healthcheck-scheme",
          "ServicesPathAnnotation": "my/healthcheck-path",
          "ServicesPortAnnotation": "my/healthcheck-port",
          "HealthPath": "/healthz",
          "RefreshTimeInSeconds": 120,
          "Namespaces": []
        }
      }
    }

I hope it help.