nginx: dial an app service over TLS when it publishes an https URL

[mkitti]

Dials an app service over TLS when the service publishes an https:// URL. Stacked on #15; the application half is JaneliaSciComp/fileglancer#443. Both must land, and this one is inert on its own.

Why

#15 hardcodes proxy_pass http://$upstream. Two shipped Fileglancer apps front themselves with Caddy and already publish an https:// URL — JaneliaSciComp/marimo_ai_sandbox (self-signed cert generated on the compute node) and another shipped app (Caddy tls internal on a fixed 8443). A plaintext request at a TLS listener is not quietly unencrypted; it is refused. So the vhost as it stands breaks those apps the moment apps.service_proxy_domain is set.

Fileglancer now reports the scheme in a second header and this dials it:

auth_request_set $upstream $upstream_http_x_fg_upstream;
auth_request_set $upscheme $upstream_http_x_fg_upstream_scheme;
proxy_pass $upscheme://$upstream;

Two directives worth a reviewer's attention

• proxy_ssl_verify off is deliberate, and is not a shortcut. Those certificates are generated on the compute node at launch and signed by nobody, so there is no trust anchor to verify against; and the node and port change every launch, so there is nothing stable to pin either. This is opportunistic encryption — it keeps the service token off the wire in cleartext and authenticates nothing. That is no worse than the plaintext hop it replaces, and no better. The options for actually authenticating the upstream are recorded in docs/superpowers/specs/2026-09-02-app-service-upstream-tls-options.md in the application PR, including why two of them are dead ends (proxy_ssl_trusted_certificate accepts no variables, and nginx has no upstream-fingerprint directive).
• proxy_ssl_name is left at its default on purpose. The default is the proxy_pass host, which is the name those certificates actually carry. Setting it to $host — the job-<id> subdomain — would send an SNI value no app's certificate has.

Verified

nginx -t passes with this file included in a wrapper config using throwaway certs and unprivileged ports, same as #15.

Beyond that, live: a variable scheme in proxy_pass was the one load-bearing assumption, so I stood up a harness where a single nginx 1.24 instance plays every role — a plaintext upstream, a self-signed TLS upstream, a stand-in for /api/apps/resolve answering the two headers, and this vhost:

job-1 (published http):  code=200 body=PLAINTEXT-UPSTREAM
job-2 (published https): code=200 body=TLS-UPSTREAM
job-9 (no upstream):     code=403

With the scheme hardcoded back to http://, the same harness reproduces the breakage: job-2 returns 400, The plain HTTP request was sent to HTTPS port. So both the bug and the fix are demonstrated rather than argued.

Still not verifiable until DNS and the certificate exist: whether each app behaves end to end behind the proxy. That is #15's checklist.

For #15's checklist

Both Caddy apps should be on the per-app verification list — they are the only apps that exercise the TLS path at all:

• marimo_ai_sandbox — Marimo notebook loads and stays connected over the proxied HTTPS upstream
• the other Caddy-fronted app — expected to 404 today, see below

That app's Caddyfile site address is Host-matched ({$CADDY_HOSTNAME}:8443, localhost:8443, 127.0.0.1:8443), and this vhost passes Host $host through unchanged — load-bearing for JupyterLab's WebSocket origin check. Caddy therefore sees job-<id>.services.int.janelia.org, matches no site, and refuses. The fix is a catch-all :8443 site address in that repo, not a change here; and its GitHub OAuth callback is pinned to a fixed host and port besides. Do not weaken this configuration to accommodate it.

@krokicki @JaneliaSciComp/fileglancer

[neomorphic]

I can see one thing that looks like an issue.

Blocking: an empty $upscheme is a 500, not a fallback to http

If the resolve response has no X-Fg-Upstream-Scheme, $upscheme is empty and proxy_pass evaluates to ://host:port. nginx refuses that URL outright:

invalid URL prefix in "://127.0.0.1:8081" ... host: "job-2.svc"

and answers 500 for every proxied request. You can reproduce it in the same style as your harness (nginx 1.31.1, stub resolver that sends the header for job-1 and omits it for job-2):

job-1 (scheme header present): code=200 body=PLAINTEXT-UPSTREAM
job-2 (scheme header missing): code=500
job-9 (no upstream):           code=403

That matters because the header is missing in a build we've already shipped. Fileglancer's _resolve_ok only started sending X-Fg-Upstream-Scheme at e0b80fe1; the resolve endpoint predates it. The 3.2.0a1 tag — the alpha cut for dev — points at 8eb85a2c, which sends X-Fg-Upstream alone. So deploying this against the fileglancer currently on dev turns a working http proxy into a wall of 500s.

That's the opposite of "inert on its own": this PR is only inert if the app side (#443) deploys first or simultaneously. #443's _resolve_ok docstring argues compatibility in one direction (old nginx ignores the new header) and is silent on this one.

How about a map with a default, alongside the existing $connection_upgrade map at the top of the file:

map $upscheme $dial_scheme {
  default http;
  https   https;
}

proxy_pass $dial_scheme://$upstream;

map is evaluated lazily, so $dial_scheme is computed at proxy_pass time, after auth_request_set has populated $upscheme. Verified in the same harness:

job-1 (scheme=http):    code=200 body=PLAINTEXT-UPSTREAM
job-2 (header missing): code=200 body=PLAINTEXT-UPSTREAM   ← falls back to http
job-3 (scheme=HTTPS):   code=502                           ← correctly dialed TLS at a plaintext listener
job-4 (scheme=gopher):  code=200 body=PLAINTEXT-UPSTREAM   ← garbage falls back to http
job-9 (no upstream):    code=403

It removes the deployment-ordering constraint between the two repos entirely. And it stops the hub trusting the header's value: the scheme originates from a file the user's job wrote, and while #443 gates it to http/https today, the proxy shouldn't depend on that.

Nit

The comment above location = /_fg_resolve still says the endpoint "answers 204 with X-Fg-Upstream: <host>:<port>" — worth mentioning the scheme header there now that this block reads it.

Everything else

proxy_ssl_verify off and leaving proxy_ssl_name at its default are the right calls for certs generated on the node at launch, and the in-file comment explains it well enough that nobody will "fix" it later.

[mkitti]

Pushed a fix in 9c19627 for the "empty `$upscheme` is a 500" issue from the review: added a `map $upscheme $dial_scheme { default http; https https; }` alongside the existing `$connection_upgrade` map, and `proxy_pass` now dials `$dial_scheme` instead of `$upscheme` directly. Also updated the `/_fg_resolve` comment to mention `X-Fg-Upstream-Scheme`. This removes the deployment-ordering dependency on #443 and stops trusting the header's raw value in the URL scheme.

[neomorphic]

@mkitti Comment has been edited

[mkitti]

This was determined to be not be necessary in the data environment.