[nexus] add an ExternalHttpClient for sending requests outside the rack

[hawkw]

Depends on #10872.

There are various circumstances in which Nexus must make HTTP requests to endpoints external to the rack, at URLs which are often user-configurable. The primary example of this is for delivering alerts via webhooks, where the alert payload is sent in an HTTP POST request to the user-configured webhook receiver endpoint.

When making external requests, it is important to ensure that they are never sent to IP addresses on the underlay network, which are also reachable from a Nexus zone. This is necessarily primarily to defend against a class of attacks called server-side request forgery, or SSRF, in which an attacker manipulates the capability of a server that exposes a way to make user-controlled requests to send malicious requests to destinations that the attacker could not otherwise access. To be specific, an attacker who wished to influence the behavior of an underlay network service that they would otherwise not be able to access could exploit the ability of Nexus to communicate both on the underlay and externally by configuring a webhook receiver endpoint that points at an API exposed by an underlay network service, and then trigger a webhook liveness probe, sending a request to that underlay service. In the case of webhooks specifically, there are limited avenues to exploit such a vulnerability,¹ but it must be protected against nonetheless.

Furthermore, it is also possible for a user-provided URL to point to an underlay address due to a non-malicious misconfiguration. The IPv6 ULA address space used for the underlay network is quite large, but because the address ranges to use are provided by the operator at RSS-time, it is entirely possible for someone to accidentally run a service (such as a webhook receiver) within their network but external to the rack on an address which collides with the Oxide underlay network. In such a case, the operator would be surprised and dismayed when their webhook is sent successfully...to an underlay network service that doesn't know what to do with it. Thus, we want to fail loudly rather than quietly doing the wrong thing, for the benefit of debugging legitimate configuration mistakes.

Currently, webhook requests protect against SSRF and accidental underlay network address collisions at the network level, by binding connections exclusively to the Nexus zone's OPTE interface, using the IP_BOUND_IF and IPV6_BOUND_IF sockopts. Binding only on this interface ensures that the connection cannot route to the underlay. The backstory for this branch is that @rcgoodfellow and I recently discovered that this doesn't always work due to a race condition in how control plane zone routes are configured (see #10829). Initially, I thought that once the race was fixed, the approach of binding directly to the OPTE interface would never work, so it was necessary to reimplement the SSRF-prevention mechanism at the application layer. It turns out that this was not actually the case, and Ry was able to make a fairly simple fix (#10837) that would allow this technique to continue working.

However, there are some advantages to rejecting underlay IPs at the application level in addition to relying on network isolation. One big one is defense in depth: while binding on an interface that can never route to underlay IPs is in many ways a more reliable defense than maintaining a list of denied subnets in the application,² this also relies on the assumption that the routing rules for that interface won't change in a way that makes it fail open. Therefore, doing both seems better than doing either one. Furthermore, the application-level rejection has the advantage of improved visibility: trying to connect to an underlay address can be rejected with an explicit error message that makes it clear why it didn't work, rather than with a generic "address unreachable" error when we can't open a connection to that address. In future changes, we can expose this to the operator more directly, so they can tell if they have accidentally pointed a webhook or something at a colliding address. Therefore, I decided to keep working on the code I had started to implement application-layer filtering even though we have a fix for the network-level isolation.

This branch, therefore, introduces a new ExternalHttpClient type, which is intended as The Canonical Way for Nexus to construct clients to external endpoints. It configures a reqwest::Client which is bound to the OPTE interface only, and wraps it in a layer that rejects underlay addresses. It is necessary to check for such addresses in several places: first, we must check if the user-provided URL's host is an IP address literal which is on an underlay network or other disallowed subnet. If the host is a domain name, we must also check if the IP addresses resolved for that domain are underlay addresses (which could occur maliciously as part of a subdomain takeover or other form of DNS hijacking attack). Finally, we must also apply this policy when following redirects,³ checking whether the redirect URL is an underlay address, or if it is to a domain name that resolves to such an address. This ended up being rather a lot of code.

I've also changed the webhook delivery client to use the new ExternalHttpClient. Follow-up work will include:

• Changing the client used for SAML IdP requests to also use ExternalHttpClient.
• Recording the error message returned by a webhook delivery request in the database so that it may be reported to operators.
• Checking the webhook receiver endpoint URL against this policy when a receiver is created or updated, as well as when actually sending requests to it.⁴

Footnotes

1. Because webhook requests are POSTs, and any data a receiver endpoint sends in response to such a request is not exposed to the caller that triggers a probe, the capability to exploit this for data exfiltration is somewhat limited. And, the attacker cannot cause Nexus to send arbitrary request payloads through webhook requests, so their ability to request operations on underlay services depends on those services having endpoints which will accept a POST with an arbitrary JSON body. ↩

2. Since, well, it would be easy to forget to add a new underlay subnet to that list... ↩

3. This doesn't apply in the webhook delivery case, since webhook requests never follow redirects. But, since the intent is that all external requests should eventually use this code, I went ahead and implemented it. ↩

4. Note that this alone would not be sufficient to ensure that we defend against SSRF attacks, since, as I mentioned above, an attacker could compromise a DNS server to point a previously-okay domain name at a disallowed IP. Therefore, we must always check when we actually attempt to connect to a specific thing. But, eager validation would help in the case of legitimate accidental misconfigurations by returning an error to the user immediately, rather than waiting until we actually try to talk to that thing. ↩