From 8c1772d83f7129c3bc43903edda3303cbf0a771e Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Tue, 27 Oct 2026 16:05:58 -0700 Subject: [PATCH 1/3] disk_blob: bound the time that an HTTP blob request may take HttpBlob sent each request through a hyper client with no connect timeout, no deadline for the response or its body, and no TCP keepalive, so a request that stopped making progress without failing never completed. That happens when the network silently stops delivering a connection's packets: the client waits for data that never comes, with no FIN or reset to end the wait. A guest waits for its disk reads, typically without a timeout of its own, since Linux storvsc trusts the host to complete every command, so one such request hangs the guest for good. microsoft/nvx#170 traced intermittent Linux/KVM VMM-test hangs to exactly this: a guest SCSI READ on a lazily fetched blob disk stayed in storvsp's in-flight table, never completed, until petri's ten-minute watchdog killed the test. Bound every request in time, with the limits of a new HttpBlobOptions, which HttpBlob::with_options takes and HttpBlob::new defaults: - connect_timeout, 10 s, bounds establishing the TCP connection. - response_timeout, 20 s, bounds the wait for the response headers, connecting and sending the request included. - body_idle_timeout, 20 s, bounds each wait for more of the body. It limits silence rather than duration, so that a slow response that is still arriving is never cut off. - operation_timeout, 120 s, bounds the request as a whole, so that even a response that trickles in too slowly ever to finish fails. Enable TCP keepalive, after 30 s idle with 3 probes 10 s apart, so that an idle pooled connection whose path has died is closed instead of being discovered by the next request, and middleboxes that drop idle flows see traffic. The HEAD request with which HttpBlob::new probes the blob's length, and any redirects that it follows, are bounded the same way, under one operation deadline. Each request now runs, body included, on the crate's tokio runtime, whose timers need tokio's time feature, and is aborted if the caller drops the read. A request that runs out of time fails with an io::ErrorKind::TimedOut error that names the limit, which the guest sees as a failed read that it can handle rather than one that never completes. A connect timeout, which the connector reports as the cause of its error, fails with such an error too. The defaults are a thousand times the tens of milliseconds that a range request to Azure Blob Storage normally takes, and fail a hopeless request well before a test watchdog such as petri's would. Refuse, without a request, a read whose last byte would lie past the largest offset that a u64 can hold, which the range header could not name; HttpBlob::read computed that offset unchecked. No guest read gets there, since the disk checks a guest's range against its own size, but Blob::read is public. The test HTTP server can now stall before the response or part way through the body, holding the connection open and silent as a dead network path does, send a slow body, follow a script for its first requests, and count what it sees, so that tests wait for its state rather than sleep. Each new test reads on a thread of its own and fails, rather than hanging the test runner, if the read takes more than ten seconds; with HttpBlob as it was, every stall test fails on that limit. The tests cover a stall before the response and one mid-body, which now fail the read, and close the connection, in bounded time; a slow body, which is not a stall; an endless body, which the operation timeout bounds; a stalled HEAD, which fails the open; a range past the largest offset, which is refused; and the search for a timeout among an error's causes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- Cargo.lock | 2 + Guide/src/reference/backends/storage.md | 2 +- vm/devices/storage/disk_blob/Cargo.toml | 4 +- vm/devices/storage/disk_blob/src/blob/http.rs | 490 +++++++++++++++--- .../disk_blob/src/tests/http_server.rs | 419 +++++++++++---- vm/devices/storage/disk_blob/src/tests/mod.rs | 224 ++++++++ 6 files changed, 957 insertions(+), 184 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 8820ea7548e..53d95267d5a 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1561,9 +1561,11 @@ dependencies = [ "inspect", "once_cell", "pal_async", + "parking_lot", "scsi_buffers", "storage_tests", "tempfile", + "test_with_tracing", "thiserror 2.0.16", "tokio", "vhd1_defs", diff --git a/Guide/src/reference/backends/storage.md b/Guide/src/reference/backends/storage.md index 389795dfbbf..a1d6578bd6f 100644 --- a/Guide/src/reference/backends/storage.md +++ b/Guide/src/reference/backends/storage.md @@ -17,7 +17,7 @@ blob, or a layered composition of multiple backends. | Vhd1Disk | [`disk_vhd1`](https://openvmm.dev/rustdoc/linux/disk_vhd1/index.html) | VHD1 fixed file | Cross-platform | Parses VHD footer for geometry. | | VhdmpDisk | `disk_vhdmp` | Windows vhdmp driver | Windows | Dynamic and differencing VHD/VHDX. | | VhdxDisk | [`vhdx`](vhdx.md) | VHDX file | Cross-platform | Pure-Rust VHDX parser. Dynamic, fixed, and differencing. | -| BlobDisk | [`disk_blob`](https://openvmm.dev/rustdoc/linux/disk_blob/index.html) | HTTP / Azure Blob | Cross-platform | Read-only. HTTP range requests. | +| BlobDisk | [`disk_blob`](https://openvmm.dev/rustdoc/linux/disk_blob/index.html) | HTTP / Azure Blob | Cross-platform | Read-only. HTTP range requests, with timeouts. | | BlockDeviceDisk | [`disk_blockdevice`](https://openvmm.dev/rustdoc/linux/disk_blockdevice/index.html) | Linux block device or file | Linux | io_uring, resize via uevent, PR passthrough. Default for raw files on Linux in both OpenHCL and OpenVMM. | | NvmeDisk | [`disk_nvme`](https://openvmm.dev/rustdoc/linux/disk_nvme/index.html) | Physical NVMe (VFIO) | Linux/Windows | User-mode NVMe driver. Resize via AEN. | | StripedDisk | [`disk_striped`](https://openvmm.dev/rustdoc/linux/disk_striped/index.html) | Multiple Disks | Cross-platform | Stripes data across underlying disks. | diff --git a/vm/devices/storage/disk_blob/Cargo.toml b/vm/devices/storage/disk_blob/Cargo.toml index a207a2365f2..40eea051f03 100644 --- a/vm/devices/storage/disk_blob/Cargo.toml +++ b/vm/devices/storage/disk_blob/Cargo.toml @@ -30,13 +30,15 @@ thiserror.workspace = true zerocopy.workspace = true # tokio use is allowed in this crate only. # FUTURE: replace this with our own executor -tokio = { version = "1", features = ["rt-multi-thread"] } +tokio = { version = "1", features = ["rt-multi-thread", "time"] } [dev-dependencies] disk_vhd1.workspace = true pal_async.workspace = true +parking_lot.workspace = true storage_tests.workspace = true tempfile.workspace = true +test_with_tracing.workspace = true [lints] workspace = true diff --git a/vm/devices/storage/disk_blob/src/blob/http.rs b/vm/devices/storage/disk_blob/src/blob/http.rs index 7d90be22503..d1848ee8d47 100644 --- a/vm/devices/storage/disk_blob/src/blob/http.rs +++ b/vm/devices/storage/disk_blob/src/blob/http.rs @@ -14,42 +14,117 @@ use async_trait::async_trait; use http::uri::Scheme; use http_body_util::BodyExt; use http_body_util::Empty; +use hyper::Method; use hyper::Request; +use hyper::Response; use hyper::StatusCode; use hyper::Uri; +use hyper::body::Incoming; use hyper_tls::HttpsConnector; use hyper_util::client::legacy::Client; use hyper_util::client::legacy::connect::HttpConnector; use hyper_util::rt::TokioExecutor; use inspect::Inspect; use once_cell::sync::OnceCell; +use std::fmt; use std::fmt::Debug; use std::io; +use std::sync::Arc; +use std::time::Duration; +use thiserror::Error; +use tokio::time::Instant; + +/// Timeouts for the requests that an [`HttpBlob`] makes. +/// +/// A guest waits for its disk reads to complete, typically without a timeout +/// of its own, so a request that never completes hangs the guest. And a request +/// can stop making progress without failing: when the network silently stops +/// delivering a connection's packets, the client is left waiting for data that +/// will never come, with no reset or FIN to end the wait. So every request is +/// bounded in time: +/// +/// - It must receive the response headers within +/// [`response_timeout`](Self::response_timeout), and then more of the body at +/// least every [`body_idle_timeout`](Self::body_idle_timeout). These limit +/// silence rather than duration, so that a connection that has stopped is +/// detected without cutting off a slow response that is still arriving. +/// - [`operation_timeout`](Self::operation_timeout) bounds the request as a +/// whole, so that even a response that trickles in too slowly ever to finish +/// fails in bounded time. +/// +/// A request that fails returns an error, which the guest sees as a failed +/// read that it can handle, rather than as a read that never completes. +/// +/// The defaults suit a blob store such as Azure Blob Storage, where a range +/// request normally completes in tens of milliseconds. A connection that is +/// silent for 20 seconds, a thousand times that, is taken to be dead, and no +/// request takes more than two minutes, so a hopeless request fails well before +/// a typical test watchdog, such as petri's ten minutes, would intervene. +#[derive(Debug, Clone)] +pub struct HttpBlobOptions { + /// How long to wait for a TCP connection to be established. + pub connect_timeout: Duration, + /// How long a request waits for the response headers, including the time + /// to connect and to send the request. + pub response_timeout: Duration, + /// How long a request waits for more of the response body. + pub body_idle_timeout: Duration, + /// How long a request may take in all. + pub operation_timeout: Duration, +} + +impl Default for HttpBlobOptions { + fn default() -> Self { + Self { + connect_timeout: Duration::from_secs(10), + response_timeout: Duration::from_secs(20), + body_idle_timeout: Duration::from_secs(20), + operation_timeout: Duration::from_secs(120), + } + } +} + +/// How long a connection may be idle before TCP keepalive probes check that +/// the server can still be reached. Without them, an idle pooled connection +/// whose network path has died is discovered only when a request is sent on it, +/// at the cost of a response timeout; with them, it is closed after +/// [`TCP_KEEPALIVE_RETRIES`] unanswered probes. The probes also keep the +/// connection alive in middleboxes that drop idle flows. +const TCP_KEEPALIVE_TIME: Duration = Duration::from_secs(30); +/// The interval between TCP keepalive probes that go unanswered. +const TCP_KEEPALIVE_INTERVAL: Duration = Duration::from_secs(10); +/// How many TCP keepalive probes go unanswered before the connection is closed. +const TCP_KEEPALIVE_RETRIES: u32 = 3; + +type HttpClient = Client, Empty<&'static [u8]>>; /// A blob backed by an HTTP/HTTPS connection. +/// +/// Its requests are bounded in time, as [`HttpBlobOptions`] describes. #[derive(Debug, Inspect)] pub struct HttpBlob { #[inspect(skip)] - client: Client, Empty<&'static [u8]>>, + requester: Arc, #[inspect(debug)] version: http::Version, #[inspect(display)] uri: Uri, len: u64, - #[inspect(skip)] - tokio_handle: tokio::runtime::Handle, } static TOKIO_RUNTIME: OnceCell = OnceCell::new(); impl HttpBlob { - /// Connects to `url` and returns an object to access it as a blob. + /// Connects to `url` and returns an object to access it as a blob, with the + /// default [`HttpBlobOptions`]. pub async fn new(url: &str) -> anyhow::Result { - let mut uri: Uri = url.parse()?; + Self::with_options(url, HttpBlobOptions::default()).await + } - let connector = HttpsConnector::new(); - let builder = Client::builder(TokioExecutor::new()); - let client = builder.build(connector); + /// Connects to `url` and returns an object to access it as a blob, with the + /// timeouts that `options` specifies. + pub async fn with_options(url: &str, options: HttpBlobOptions) -> anyhow::Result { + let uri: Uri = url.parse()?; let handle = TOKIO_RUNTIME .get_or_try_init(tokio::runtime::Runtime::new) @@ -57,28 +132,120 @@ impl HttpBlob { .handle() .clone(); + let requester = Arc::new(Requester::new(options, handle)); + let (uri, response) = requester.run(requester.clone().probe(uri)).await?; + + let len = response + .headers() + .get("Content-Length") + .context("missing blob length")? + .to_str() + .context("couldn't parse blob length")? + .parse() + .context("couldn't parse blob length")?; + + let version = response.version(); + + Ok(Self { + requester, + version, + uri, + len, + }) + } +} + +#[async_trait] +impl Blob for HttpBlob { + async fn read(&self, buf: &mut [u8], offset: u64) -> io::Result<()> { + // HTTP byte ranges are inclusive at both ends, so an empty one cannot + // be expressed, and computing its end offset would underflow. There is + // nothing to fetch in any case. + if buf.is_empty() { + return Ok(()); + } + // A request names the range by its first and last offsets, so the + // last must be representable. + if offset.checked_add(buf.len() as u64 - 1).is_none() { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + "read range ends past the largest representable offset", + )); + } + let requester = self.requester.clone(); + let uri = self.uri.clone(); + let len = buf.len(); + let data = self + .requester + .run(async move { requester.get(uri, offset, len).await }) + .await?; + buf.copy_from_slice(&data); + Ok(()) + } + + fn len(&self) -> u64 { + self.len + } +} + +/// Makes the requests for an [`HttpBlob`], with timeouts. +#[derive(Debug)] +struct Requester { + options: HttpBlobOptions, + client: HttpClient, + tokio_handle: tokio::runtime::Handle, +} + +impl Requester { + fn new(options: HttpBlobOptions, tokio_handle: tokio::runtime::Handle) -> Self { + let mut http = HttpConnector::new(); + // The HTTPS connector hands HTTPS URIs to this one to connect. + http.enforce_http(false); + http.set_connect_timeout(Some(options.connect_timeout)); + http.set_keepalive(Some(TCP_KEEPALIVE_TIME)); + http.set_keepalive_interval(Some(TCP_KEEPALIVE_INTERVAL)); + http.set_keepalive_retries(Some(TCP_KEEPALIVE_RETRIES)); + Self { + client: Client::builder(TokioExecutor::new()) + .build(HttpsConnector::new_with_connector(http)), + options, + tokio_handle, + } + } + + /// Runs `fut` on the tokio runtime, whose timers and connections it needs, + /// and returns its output. Dropping the returned future aborts `fut`. + async fn run(&self, fut: impl 'static + Send + Future) -> T { + let mut task = AbortOnDrop(self.tokio_handle.spawn(fut)); + match (&mut task.0).await { + Ok(output) => output, + // The task is only aborted when this future is dropped, so it must + // have panicked. + Err(err) => std::panic::resume_unwind(err.into_panic()), + } + } + + /// Follows `uri` through any redirects to the blob, and returns the blob's + /// URI and the response to a HEAD request for it. + async fn probe(self: Arc, mut uri: Uri) -> anyhow::Result<(Uri, Response)> { + // One deadline covers all the redirects. + let deadline = deadline_after(self.options.operation_timeout); + let response_timeout = self.options.response_timeout; let mut redirect_count = 0; - let response = loop { + loop { if redirect_count > 5 { anyhow::bail!("too many redirects"); } - let response = handle - .spawn( - client.request( - Request::builder() - .uri(&uri) - .method("HEAD") - .body(Empty::new()) - .unwrap(), - ), - ) + let response = self + .request(RequestKind::Head, deadline, |client| { + head_once(client, uri.clone(), response_timeout) + }) .await - .unwrap() .context("failed to query blob size")?; let next_uri: Uri = match response.status() { - StatusCode::OK => break response, + StatusCode::OK => return Ok((uri, response)), StatusCode::MOVED_PERMANENTLY | StatusCode::FOUND | StatusCode::TEMPORARY_REDIRECT @@ -101,81 +268,240 @@ impl HttpBlob { uri = next_uri; redirect_count += 1; + } + } + + /// Reads `len` bytes at `offset` from the blob at `uri`. + async fn get(self: Arc, uri: Uri, offset: u64, len: usize) -> io::Result> { + let deadline = deadline_after(self.options.operation_timeout); + let response_timeout = self.options.response_timeout; + let body_idle_timeout = self.options.body_idle_timeout; + self.request(RequestKind::Get { offset, len }, deadline, |client| { + get_once( + client, + uri.clone(), + offset, + len, + response_timeout, + body_idle_timeout, + ) + }) + .await + } + + /// Makes `request`, by calling `attempt` with the client to use, and fails + /// it if it has not completed by `deadline`. + async fn request( + &self, + request: RequestKind, + deadline: Instant, + attempt: F, + ) -> io::Result + where + F: FnOnce(HttpClient) -> Fut, + Fut: Future>, + { + match tokio::time::timeout_at(deadline, attempt(self.client.clone())).await { + Ok(result) => result, + Err(_) => Err(io::Error::new( + io::ErrorKind::TimedOut, + RequestError::OperationTimeout { + request, + timeout: self.options.operation_timeout, + }, + )), + } + } +} + +/// Returns the instant `timeout` from now, or a distant one if that cannot be +/// represented. +fn deadline_after(timeout: Duration) -> Instant { + let now = Instant::now(); + now.checked_add(timeout) + .unwrap_or_else(|| now + Duration::from_secs(60 * 60 * 24 * 365)) +} + +/// Makes one attempt at a HEAD request for `uri`. +async fn head_once( + client: HttpClient, + uri: Uri, + response_timeout: Duration, +) -> io::Result> { + let request = Request::builder() + .uri(uri) + .method(Method::HEAD) + .body(Empty::new()) + .unwrap(); + send(client, request, response_timeout).await +} + +/// Makes one attempt at reading `len` bytes at `offset` from the blob at `uri`. +async fn get_once( + client: HttpClient, + uri: Uri, + offset: u64, + len: usize, + response_timeout: Duration, + body_idle_timeout: Duration, +) -> io::Result> { + let request = Request::builder() + .uri(uri) + .header( + hyper::header::RANGE, + format!("bytes={}-{}", offset, offset + len as u64 - 1), + ) + .body(Empty::new()) + .unwrap(); + let response = send(client, request, response_timeout).await?; + + if !response.status().is_success() { + return Err(io::Error::other(RequestError::Status(response.status()))); + } + + let mut body = response.into_body(); + let mut data = Vec::with_capacity(len); + loop { + let frame = match tokio::time::timeout(body_idle_timeout, body.frame()).await { + Ok(Some(frame)) => frame.map_err(io::Error::other)?, + Ok(None) => break, + Err(_) => { + return Err(io::Error::new( + io::ErrorKind::TimedOut, + RequestError::BodyTimeout { + timeout: body_idle_timeout, + received: data.len(), + expected: len, + }, + )); + } }; + if let Some(chunk) = frame.data_ref() { + if chunk.len() > len - data.len() { + return Err(io::Error::other("server did not respect range query")); + } + data.extend_from_slice(chunk); + } + } - let len = response - .headers() - .get("Content-Length") - .context("missing blob length")? - .to_str() - .context("couldn't parse blob length")? - .parse() - .context("couldn't parse blob length")?; + if data.len() < len { + return Err(io::ErrorKind::UnexpectedEof.into()); + } - let version = response.version(); + Ok(data) +} - Ok(Self { - client, - version, - uri, - len, - tokio_handle: handle, - }) +/// Sends `request`, and waits up to `response_timeout` for the response +/// headers. +async fn send( + client: HttpClient, + request: Request>, + response_timeout: Duration, +) -> io::Result> { + match tokio::time::timeout(response_timeout, client.request(request)).await { + Ok(Ok(response)) => Ok(response), + // The connector gave up on connecting. + Ok(Err(err)) if is_timed_out(&err) => Err(io::Error::new( + io::ErrorKind::TimedOut, + RequestError::ConnectTimeout(err), + )), + Ok(Err(err)) => Err(io::Error::other(err)), + Err(_) => Err(io::Error::new( + io::ErrorKind::TimedOut, + RequestError::ResponseTimeout(response_timeout), + )), } } -#[async_trait] -impl Blob for HttpBlob { - async fn read(&self, mut buf: &mut [u8], offset: u64) -> io::Result<()> { - // HTTP byte ranges are inclusive at both ends, so an empty one cannot - // be expressed, and computing the end offset below would underflow. - // There is nothing to fetch in any case. - if buf.is_empty() { - return Ok(()); +/// Returns whether `err`, or any error in its chain of sources, is a timeout, +/// as when the connector gives up on connecting. +fn is_timed_out(err: &(dyn std::error::Error + 'static)) -> bool { + let mut next = Some(err); + while let Some(err) = next { + if err + .downcast_ref::() + .is_some_and(|err| err.kind() == io::ErrorKind::TimedOut) + { + return true; } - let mut response = self - .tokio_handle - .spawn( - self.client.request( - Request::builder() - .uri(&self.uri) - .header( - hyper::header::RANGE, - format!("bytes={}-{}", offset, offset + buf.len() as u64 - 1,), - ) - .body(Empty::new()) - .unwrap(), - ), - ) - .await - .unwrap() - .map_err(io::Error::other)?; + next = err.source(); + } + false +} - if !response.status().is_success() { - return Err(io::Error::other(response.status().to_string())); - } +/// Aborts a tokio task when dropped. +struct AbortOnDrop(tokio::task::JoinHandle); - while let Some(frame) = response.body_mut().frame().await { - let frame = frame.map_err(io::Error::other)?; - if let Some(data) = frame.data_ref() { - let len = data.len(); - if len > buf.len() { - return Err(io::Error::other("server did not respect range query")); - } - let (this, rest) = buf.split_at_mut(len); - this.copy_from_slice(data); - buf = rest; +impl Drop for AbortOnDrop { + fn drop(&mut self) { + self.0.abort(); + } +} + +#[derive(Debug, Error)] +enum RequestError { + #[error("connection timed out")] + ConnectTimeout(#[source] hyper_util::client::legacy::Error), + #[error("no response within {0:?}")] + ResponseTimeout(Duration), + #[error("no data for {timeout:?}, after {received} of {expected} bytes")] + BodyTimeout { + timeout: Duration, + received: usize, + expected: usize, + }, + #[error("{request} did not complete within {timeout:?}")] + OperationTimeout { + request: RequestKind, + timeout: Duration, + }, + #[error("server returned {0}")] + Status(StatusCode), +} + +/// A request that an [`HttpBlob`] makes. +#[derive(Debug, Copy, Clone)] +enum RequestKind { + /// A HEAD request, to probe the blob's length. + Head, + /// A range request for `len` bytes at `offset`. `len` is not zero, and the + /// offset of the range's last byte, `offset + len - 1`, does not overflow. + Get { offset: u64, len: usize }, +} + +impl fmt::Display for RequestKind { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match *self { + Self::Head => f.write_str("HEAD"), + Self::Get { offset, len } => { + write!(f, "GET bytes={}-{}", offset, offset + len as u64 - 1) } } + } +} - if !buf.is_empty() { - return Err(io::ErrorKind::UnexpectedEof.into()); - } +#[cfg(test)] +mod tests { + use super::is_timed_out; + use std::io; + use test_with_tracing::test; - Ok(()) - } + /// An error caused by another, as the connector's errors are. + #[derive(Debug, thiserror::Error)] + #[error("wrapper")] + struct Wrapper(#[source] Box); - fn len(&self) -> u64 { - self.len + #[test] + fn a_timeout_is_found_anywhere_in_the_chain() { + let timed_out = || io::Error::new(io::ErrorKind::TimedOut, "connect"); + assert!(is_timed_out(&timed_out())); + assert!(is_timed_out(&Wrapper(Box::new(Wrapper(Box::new( + timed_out() + )))))); + assert!(!is_timed_out(&io::Error::other("refused"))); + assert!(!is_timed_out(&Wrapper(Box::new(io::Error::new( + io::ErrorKind::ConnectionRefused, + "refused", + ))))); } } diff --git a/vm/devices/storage/disk_blob/src/tests/http_server.rs b/vm/devices/storage/disk_blob/src/tests/http_server.rs index 7491b19dc31..aa1ccfa4d0c 100644 --- a/vm/devices/storage/disk_blob/src/tests/http_server.rs +++ b/vm/devices/storage/disk_blob/src/tests/http_server.rs @@ -9,17 +9,28 @@ //! importantly, the interesting cases are all responses a well-behaved server //! would never produce. See [`Behavior`]. //! +//! A [`Script`] can make the server misbehave for the first few requests only. +//! The server counts what it sees in [`Stats`], so that tests can wait for it +//! to reach a given point rather than sleep, and can check how many requests a +//! read made. +//! //! [`HttpBlob`]: crate::blob::http::HttpBlob +use parking_lot::Condvar; +use parking_lot::Mutex; use std::io::Read; use std::io::Write; use std::net::TcpListener; use std::net::TcpStream; +use std::sync::Arc; +use std::time::Duration; +use std::time::Instant; -/// How the server should respond to range requests. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] +/// How the server should respond to a request. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] pub enum Behavior { /// Answer with exactly the requested range. + #[default] Correct, /// Answer with one byte less than was asked for. `Blob::read` is required /// to report this as `UnexpectedEof` rather than as a partial success, @@ -32,140 +43,348 @@ pub enum Behavior { RangeNotSatisfiable, /// Send the headers, then close the connection part way through the body. CloseMidBody, + /// Read the request, then send nothing at all, holding the connection open + /// until the client closes it or the server is released. This is what a + /// client sees when the network silently stops delivering a connection's + /// packets: no data, but no FIN or reset to end the wait either. + StallBeforeResponse, + /// Send the headers and half the body, then stall as for + /// [`Behavior::StallBeforeResponse`]. + StallMidBody, + /// Answer correctly, but send the body in `pieces` pieces, pausing for + /// `interval` before each: slow, but never silent for much longer than + /// `interval`. + SlowBody { pieces: usize, interval: Duration }, +} + +/// How the server responds to each request. +#[derive(Debug, Clone, Default)] +pub struct Script { + /// How to respond to the first HEAD requests, in order. Later ones are + /// answered correctly. A HEAD response has no body, so a behavior that + /// concerns the body acts as its counterpart that does not: a stall + /// stalls before the response, and the rest answer correctly. + pub head: Vec, + /// How to respond to the first GET requests, in order. + pub get: Vec, + /// How to respond to the GET requests after those in `get`. + pub then: Behavior, +} + +/// What the server has seen so far. +#[derive(Debug, Clone, Copy, Default)] +pub struct Stats { + /// HEAD requests received. + pub heads: usize, + /// GET requests received. + pub gets: usize, + /// Connections that the server has stalled. + pub stalled: usize, + /// Stalled connections that the client has since closed. + pub abandoned: usize, } /// A server bound to an ephemeral port on the loopback interface. pub struct TestHttpServer { port: u16, + shared: Arc, +} + +struct Shared { + content: Vec, + script: Script, + state: Mutex, + changed: Condvar, +} + +#[derive(Default)] +struct State { + stats: Stats, + released: bool, } impl TestHttpServer { - /// Starts a server publishing `content`. + /// Starts a server publishing `content`, which answers every range request + /// as `behavior` says. + pub fn new(content: Vec, behavior: Behavior) -> Self { + Self::with_script( + content, + Script { + then: behavior, + ..Default::default() + }, + ) + } + + /// Starts a server publishing `content`, which answers requests as `script` + /// says. /// - /// The serving thread runs until the process exits. Tests are short-lived + /// The listening thread runs until the process exits. Tests are short-lived /// and nextest gives each one its own process, so there is nothing to clean - /// up. - pub fn new(content: Vec, behavior: Behavior) -> Self { + /// up. Stalled connections are closed when the server is dropped. + pub fn with_script(content: Vec, script: Script) -> Self { let listener = TcpListener::bind("127.0.0.1:0").unwrap(); let port = listener.local_addr().unwrap().port(); + let shared = Arc::new(Shared { + content, + script, + state: Mutex::default(), + changed: Condvar::new(), + }); + let listener_shared = shared.clone(); std::thread::spawn(move || { for stream in listener.incoming() { let Ok(stream) = stream else { break }; - // A misbehaving response deliberately drops the connection, so - // errors here are expected and not worth reporting. - let _ = serve(stream, &content, behavior); + let shared = listener_shared.clone(); + // A thread per connection, so that a stalled connection does + // not hold up the others. + std::thread::spawn(move || { + // A misbehaving response deliberately drops the connection, + // so errors here are expected and not worth reporting. + let _ = shared.serve(stream); + }); } }); - Self { port } + Self { port, shared } } /// The URL of the blob. pub fn url(&self) -> String { format!("http://127.0.0.1:{}/blob", self.port) } + + /// Returns what the server has seen so far. + pub fn stats(&self) -> Stats { + self.shared.state.lock().stats + } + + /// Waits up to `timeout` for `done` to return true for the server's + /// [`Stats`], and returns them. Panics, naming `what` it was waiting for, + /// if `done` is still false at the end. + pub fn wait_for( + &self, + what: &str, + timeout: Duration, + mut done: impl FnMut(&Stats) -> bool, + ) -> Stats { + let deadline = Instant::now() + timeout; + let mut state = self.shared.state.lock(); + while !done(&state.stats) { + if self + .shared + .changed + .wait_until(&mut state, deadline) + .timed_out() + && !done(&state.stats) + { + panic!("timed out waiting for {what}: {:?}", state.stats); + } + } + state.stats + } + + /// Ends every stall, now and in future, by closing the connection. + pub fn release(&self) { + self.shared.state.lock().released = true; + self.shared.changed.notify_all(); + } } -/// Reads one request and writes one response, then closes the connection. -fn serve(mut stream: TcpStream, content: &[u8], behavior: Behavior) -> std::io::Result<()> { - let request = read_headers(&mut stream)?; - let mut lines = request.lines(); - let method = lines - .next() - .and_then(|l| l.split(' ').next()) - .unwrap_or_default() - .to_owned(); - - // `HttpBlob::new` probes the length with a HEAD request. - if method.eq_ignore_ascii_case("HEAD") { - write!( - stream, - "HTTP/1.1 200 OK\r\n\ - Content-Length: {}\r\n\ - Accept-Ranges: bytes\r\n\ - Connection: close\r\n\r\n", - content.len() - )?; - return Ok(()); +impl Drop for TestHttpServer { + fn drop(&mut self) { + // A failing test may leave a client waiting on a stalled connection. + // Closing the connection ends the wait. + self.release(); } +} - // Header names are case-insensitive, and hyper does not necessarily send - // them in the case they were built with. - let range = lines.find_map(|line| { - let (name, value) = line.split_once(':')?; - name.eq_ignore_ascii_case("range") - .then(|| value.trim().strip_prefix("bytes=").map(str::to_owned)) - .flatten() - }); - let Some(range) = range else { - write!( - stream, - "HTTP/1.1 400 Bad Request\r\nConnection: close\r\n\r\n" - )?; - return Ok(()); - }; - let (start, end) = range.split_once('-').unwrap(); - let start: usize = start.parse().unwrap(); - // The range is inclusive at both ends. - let end: usize = end.parse().unwrap(); - - if behavior == Behavior::RangeNotSatisfiable || start >= content.len() { - write!( - stream, - "HTTP/1.1 416 Range Not Satisfiable\r\n\ - Content-Range: bytes */{}\r\n\ - Connection: close\r\n\r\n", - content.len() - )?; - return Ok(()); +impl Shared { + /// Serves one request on `stream`, then closes it. + fn serve(&self, mut stream: TcpStream) -> std::io::Result<()> { + // Send each write at once, so that the client sees the pieces and + // pauses of a response as the behavior describes them. + stream.set_nodelay(true)?; + let request = read_headers(&mut stream)?; + // An empty request means the client has closed the connection. + if request.is_empty() { + return Ok(()); + } + self.respond(&mut stream, &request) + } + + /// Counts a request, and returns how to respond to it: as the next of the + /// `scripted` behaviors, if any remain, and otherwise as `then`. + fn count( + &self, + counter: impl FnOnce(&mut Stats) -> &mut usize, + scripted: &[Behavior], + then: Behavior, + ) -> Behavior { + let mut state = self.state.lock(); + let count = counter(&mut state.stats); + let behavior = scripted.get(*count).copied().unwrap_or(then); + *count += 1; + drop(state); + self.changed.notify_all(); + behavior + } + + /// Holds the connection open without sending anything, until the client + /// closes it or the server is released. + fn stall(&self, stream: &mut TcpStream) { + self.state.lock().stats.stalled += 1; + self.changed.notify_all(); + // Poll, rather than block, so that a release is noticed promptly. + let _ = stream.set_read_timeout(Some(Duration::from_millis(10))); + let mut buf = [0; 512]; + while !self.state.lock().released { + match stream.read(&mut buf) { + // A stalled server ignores anything more that the client sends. + Ok(n) if n > 0 => {} + Err(err) + if matches!( + err.kind(), + std::io::ErrorKind::WouldBlock | std::io::ErrorKind::TimedOut + ) => {} + // The client has closed or reset the connection. + _ => { + self.state.lock().stats.abandoned += 1; + self.changed.notify_all(); + break; + } + } + } } - if behavior == Behavior::IgnoreRange { + /// Answers one request. + fn respond(&self, stream: &mut TcpStream, request: &str) -> std::io::Result<()> { + let content = &self.content; + let mut lines = request.lines(); + let method = lines + .next() + .and_then(|l| l.split(' ').next()) + .unwrap_or_default() + .to_owned(); + + // `HttpBlob::new` probes the length with a HEAD request. + if method.eq_ignore_ascii_case("HEAD") { + if let Behavior::StallBeforeResponse | Behavior::StallMidBody = self.count( + |stats| &mut stats.heads, + &self.script.head, + Behavior::Correct, + ) { + self.stall(stream); + return Ok(()); + } + write!( + stream, + "HTTP/1.1 200 OK\r\n\ + Content-Length: {}\r\n\ + Accept-Ranges: bytes\r\n\ + Connection: close\r\n\r\n", + content.len() + )?; + return Ok(()); + } + + let behavior = self.count(|stats| &mut stats.gets, &self.script.get, self.script.then); + + // Header names are case-insensitive, and hyper does not necessarily send + // them in the case they were built with. + let range = lines.find_map(|line| { + let (name, value) = line.split_once(':')?; + name.eq_ignore_ascii_case("range") + .then(|| value.trim().strip_prefix("bytes=").map(str::to_owned)) + .flatten() + }); + let Some(range) = range else { + write!( + stream, + "HTTP/1.1 400 Bad Request\r\nConnection: close\r\n\r\n" + )?; + return Ok(()); + }; + let (start, end) = range.split_once('-').unwrap(); + let start: usize = start.parse().unwrap(); + // The range is inclusive at both ends. + let end: usize = end.parse().unwrap(); + + if behavior == Behavior::StallBeforeResponse { + self.stall(stream); + return Ok(()); + } + + if behavior == Behavior::RangeNotSatisfiable || start >= content.len() { + write!( + stream, + "HTTP/1.1 416 Range Not Satisfiable\r\n\ + Content-Range: bytes */{}\r\n\ + Connection: close\r\n\r\n", + content.len() + )?; + return Ok(()); + } + + if behavior == Behavior::IgnoreRange { + write!( + stream, + "HTTP/1.1 200 OK\r\n\ + Content-Length: {}\r\n\ + Connection: close\r\n\r\n", + content.len() + )?; + return stream.write_all(content); + } + + let end = end.min(content.len() - 1); + let full = &content[start..=end]; + + // `ShortBody` sends a well-formed response that is simply shorter than + // was asked for -- the Content-Length matches what is actually sent. + // That is precisely the case the `Blob` contract is about: the client + // sees a successful response, and must not mistake a partly filled + // buffer for success. A one-byte range cannot be shortened this way, + // since an empty body has no `Content-Range` that describes it, so send + // it in full. + let body = if behavior == Behavior::ShortBody && full.len() > 1 { + &full[..full.len() - 1] + } else { + full + }; + write!( stream, - "HTTP/1.1 200 OK\r\n\ + "HTTP/1.1 206 Partial Content\r\n\ + Content-Range: bytes {}-{}/{}\r\n\ Content-Length: {}\r\n\ Connection: close\r\n\r\n", - content.len() + start, + start + body.len() - 1, + content.len(), + body.len(), )?; - return stream.write_all(content); - } - let end = end.min(content.len() - 1); - let full = &content[start..=end]; - - // `ShortBody` sends a well-formed response that is simply shorter than was - // asked for -- the Content-Length matches what is actually sent. That is - // precisely the case the `Blob` contract is about: the client sees a - // successful response, and must not mistake a partly filled buffer for - // success. A one-byte range cannot be shortened this way, since an empty - // body has no `Content-Range` that describes it, so send it in full. - let body = if behavior == Behavior::ShortBody && full.len() > 1 { - &full[..full.len() - 1] - } else { - full - }; - - write!( - stream, - "HTTP/1.1 206 Partial Content\r\n\ - Content-Range: bytes {}-{}/{}\r\n\ - Content-Length: {}\r\n\ - Connection: close\r\n\r\n", - start, - start + body.len() - 1, - content.len(), - body.len(), - )?; - - if behavior == Behavior::CloseMidBody { - // Unlike `ShortBody`, the Content-Length above promises the full body. - // Sending less and hanging up leaves the client with a truncated - // message rather than a well-formed short one. - stream.write_all(&body[..body.len() / 2])?; - return Ok(()); + // Unlike `ShortBody`, the Content-Length above promises the full body, + // so sending less and then hanging up, or stalling, leaves the client + // with a truncated message rather than a well-formed short one. + let half = &body[..body.len() / 2]; + match behavior { + Behavior::CloseMidBody => stream.write_all(half), + Behavior::StallMidBody => { + stream.write_all(half)?; + self.stall(stream); + Ok(()) + } + Behavior::SlowBody { pieces, interval } => { + for piece in body.chunks(body.len().div_ceil(pieces.max(1))) { + std::thread::sleep(interval); + stream.write_all(piece)?; + } + Ok(()) + } + _ => stream.write_all(body), + } } - - stream.write_all(body) } /// Reads up to and including the blank line that terminates the headers. diff --git a/vm/devices/storage/disk_blob/src/tests/mod.rs b/vm/devices/storage/disk_blob/src/tests/mod.rs index 5794b4353c8..504baf0464a 100644 --- a/vm/devices/storage/disk_blob/src/tests/mod.rs +++ b/vm/devices/storage/disk_blob/src/tests/mod.rs @@ -6,18 +6,26 @@ mod http_server; use crate::BlobDisk; +use crate::blob::Blob; use crate::blob::file::FileBlob; use crate::blob::http::HttpBlob; +use crate::blob::http::HttpBlobOptions; use disk_backend::Disk; use disk_backend::DiskError; use guestmem::GuestMemory; use http_server::Behavior; +use http_server::Script; use http_server::TestHttpServer; use pal_async::async_test; use scsi_buffers::OwnedRequestBuffers; use std::io::Read; use std::io::Seek; use std::io::SeekFrom; +use std::sync::mpsc; +use std::thread::JoinHandle; +use std::time::Duration; +use std::time::Instant; +use test_with_tracing::test; const SECTOR_SIZE: usize = 512; const DISK_SIZE: u64 = 1024 * 1024; @@ -150,3 +158,219 @@ async fn misbehaving_server_never_reports_success() { } } } + +/// How long a fault test waits for a request. One still pending by then is +/// taken to be hung for good, which is the failure these tests exist to catch. +const LIMIT: Duration = Duration::from_secs(10); + +/// Content in which every offset is distinguishable from its neighbors, so that +/// data from the wrong place does not pass for the right data. +fn content() -> Vec { + (0..DISK_SIZE).map(|i| (i % 251) as u8).collect() +} + +/// Timeouts short enough that a stall is detected in half a second, so that the +/// fault tests are quick, but still far longer than a loopback request takes. +fn fast_options() -> HttpBlobOptions { + HttpBlobOptions { + response_timeout: Duration::from_millis(500), + body_idle_timeout: Duration::from_millis(500), + operation_timeout: Duration::from_secs(5), + ..Default::default() + } +} + +/// A future running on a thread of its own. See [`background`]. +struct Background { + recv: mpsc::Receiver, + thread: JoinHandle<()>, +} + +/// Runs `fut` on a thread of its own. +/// +/// The test's thread then only waits for the output, for at most [`LIMIT`], so +/// that a request that never completes fails the test rather than hanging the +/// test runner. +fn background(fut: impl 'static + Send + Future) -> Background { + let (send, recv) = mpsc::channel(); + let thread = std::thread::spawn(move || { + let output = pal_async::DefaultPool::run_with(async move |_| fut.await); + // The test may have stopped waiting. + let _ = send.send(output); + }); + Background { recv, thread } +} + +impl Background { + /// Waits up to [`LIMIT`] for the output, panicking if there is none. + fn wait(self, what: &str) -> T { + match self.recv.recv_timeout(LIMIT) { + Ok(output) => output, + Err(mpsc::RecvTimeoutError::Timeout) => { + panic!("{what} did not complete within {LIMIT:?}") + } + // The thread panicked. Report that panic rather than this one. + Err(mpsc::RecvTimeoutError::Disconnected) => { + std::panic::resume_unwind(self.thread.join().unwrap_err()) + } + } + } +} + +/// Runs `fut` to completion on a thread of its own, failing the test if that +/// takes longer than [`LIMIT`]. +fn bounded(what: &str, fut: impl 'static + Send + Future) -> T { + background(fut).wait(what) +} + +/// Opens the blob that `server` publishes and reads `len` bytes at `offset` +/// from it, within [`LIMIT`]. +fn read_blob( + server: &TestHttpServer, + options: HttpBlobOptions, + offset: u64, + len: usize, +) -> (HttpBlob, std::io::Result>) { + let url = server.url(); + bounded("read", async move { + let blob = HttpBlob::with_options(&url, options).await.unwrap(); + let mut buf = vec![0; len]; + let r = blob.read(&mut buf, offset).await.map(|()| buf); + (blob, r) + }) +} + +/// A server, or the network path to it, that goes silent before the response +/// fails the read in bounded time, and the stalled connection is closed. +#[test] +fn stall_before_response_fails_read() { + let server = TestHttpServer::new(content(), Behavior::StallBeforeResponse); + let (_blob, r) = read_blob(&server, fast_options(), 0, 4096); + match r { + Err(err) if err.kind() == std::io::ErrorKind::TimedOut => {} + r => panic!("expected a timeout, got {:?}", r.map(|_| ())), + } + assert_eq!(server.stats().gets, 1); + server.wait_for("the stalled connection to be closed", LIMIT, |stats| { + stats.abandoned == 1 + }); +} + +/// A guest waits for its disk reads, typically without a timeout of its own, +/// so a read that never completes hangs the guest. When the server never +/// recovers, the read must fail instead, in bounded time. +#[test] +fn persistent_stall_fails_disk_read() { + let server = TestHttpServer::new(content(), Behavior::StallMidBody); + let url = server.url(); + let options = fast_options(); + let r = bounded("disk read", async move { + let blob = HttpBlob::with_options(&url, options).await.unwrap(); + let disk = Disk::new(BlobDisk::new(blob)).unwrap(); + let mem = GuestMemory::allocate(SECTOR_SIZE * 8); + disk.read_vectored( + &OwnedRequestBuffers::linear(0, SECTOR_SIZE * 8, true).buffer(&mem), + 0, + ) + .await + }); + match r { + Err(DiskError::Io(err)) if err.kind() == std::io::ErrorKind::TimedOut => {} + r => panic!("expected a timeout, got {r:?}"), + } + assert_eq!(server.stats().gets, 1); + server.wait_for("the stalled connection to be closed", LIMIT, |stats| { + stats.abandoned == 1 + }); +} + +/// A response that keeps arriving is not a stall, however long it takes. +#[test] +fn slow_body_is_not_a_stall() { + let content = content(); + let options = fast_options(); + // Each pause is well within the idle timeout, but the body as a whole takes + // longer than either timeout. + let interval = options.body_idle_timeout / 4; + let server = TestHttpServer::new( + content.clone(), + Behavior::SlowBody { + pieces: 8, + interval, + }, + ); + let start = Instant::now(); + let (_blob, r) = read_blob(&server, options.clone(), 0, 64 * 1024); + let elapsed = start.elapsed(); + assert!(r.unwrap() == content[..64 * 1024], "wrong data"); + assert_eq!(server.stats().gets, 1, "the slow response was retried"); + assert!( + elapsed > options.body_idle_timeout.max(options.response_timeout), + "the response was not slow: {elapsed:?}" + ); +} + +/// A response that keeps arriving, but too slowly ever to finish, is bounded by +/// the operation timeout instead. +#[test] +fn endless_body_is_bounded_by_operation_timeout() { + let options = HttpBlobOptions { + operation_timeout: Duration::from_millis(1500), + ..fast_options() + }; + // One byte at a time: the body would take minutes. + let server = TestHttpServer::new( + content(), + Behavior::SlowBody { + pieces: 4096, + interval: options.body_idle_timeout / 10, + }, + ); + let start = Instant::now(); + let (_blob, r) = read_blob(&server, options.clone(), 0, 4096); + let elapsed = start.elapsed(); + match r { + Err(err) if err.kind() == std::io::ErrorKind::TimedOut => {} + r => panic!("expected a timeout, got {:?}", r.map(|_| ())), + } + assert!( + elapsed >= options.operation_timeout, + "timed out early: {elapsed:?}" + ); + assert_eq!( + server.stats().gets, + 1, + "retried after the operation timeout" + ); +} + +/// Opening the blob probes it with a HEAD request, which must not hang either. +#[test] +fn stalled_head_fails_open() { + let server = TestHttpServer::with_script( + content(), + Script { + head: vec![Behavior::StallBeforeResponse], + ..Default::default() + }, + ); + let url = server.url(); + let r = bounded("open", async move { + HttpBlob::with_options(&url, fast_options()).await.map(drop) + }); + assert!(r.is_err(), "opened a blob whose server never answered"); + assert_eq!(server.stats().heads, 1); +} + +/// A range whose last byte would lie past the largest offset that a request +/// can name is refused, without a request, rather than overflowing. +#[test] +fn range_past_the_largest_offset_is_refused() { + let server = TestHttpServer::new(content(), Behavior::Correct); + let (_blob, r) = read_blob(&server, fast_options(), u64::MAX - 100, 4096); + match r { + Err(err) if err.kind() == std::io::ErrorKind::InvalidInput => {} + r => panic!("expected InvalidInput, got {:?}", r.map(|_| ())), + } + assert_eq!(server.stats().gets, 0); +} From b47786f2a6c78aaf7183adfa23d224de1eb942e9 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Tue, 27 Oct 2026 16:08:06 -0700 Subject: [PATCH 2/3] disk_blob: retry HTTP blob requests that fail transiently With its requests bounded in time, HttpBlob fails a read whose request stalls instead of hanging the guest. But the network failures that stall a request are usually transient, and so are the resets and throttling responses of a busy blob store, so failing the guest's read is a poor outcome when another attempt would succeed. A guest handles a failed read badly, too: firmware and boot loaders may give up on the disk, and Linux may issue the same read again and again. microsoft/nvx#170's CI logs show both kinds of failure on the same runners: reads that stalled for good, and reads that failed with "Connection reset by peer". Retry a request whose attempt failed in a way that the next attempt might not: one that timed out, lost its connection before or during the response, or received status 408, 429, 500, 502, 503, or 504. Wait before each retry, from initial_backoff, 250 ms, doubling up to max_backoff, 4 s, and make up to max_attempts, 4, attempts in all, all within the request's operation_timeout. Return any other failure at once, since another attempt would only repeat it: another 4xx status, a server that ignores the range, or a well-formed body shorter than the range, which the Blob contract reports as UnexpectedEof. A request whose every attempt stalls fails after about 82 s, with an error that names the request and its attempts and keeps the last attempt's error as its source. Make each retry on a new connection, through a client that keeps no idle connections: whatever broke the previous attempt's connection may have broken the idle ones too. A timed-out attempt's connection is dropped with the attempt and never returned to the pool. First attempts still reuse idle connections, which TCP keepalive prunes. Warn, rate limited, of each retry and of each request that fails for good, naming the request by its range rather than by its URL, which may carry a credential. The test HTTP server can now reset the connection before the response or part way through the body, answer 503, and serve several requests on one connection. The tests now check that each kind of transient failure, alone or one after another, is retried to success with the right data and with every stalled connection closed; that a stalled HEAD probe is retried; that a persistent stall fails the read and the open after the attempt limit; that a stalled connection is not reused; and that permanent failures are not retried. A body that the server cuts off part way is now retried, which adds about 1.75 s of backoff to the misbehaving-server test. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- Cargo.lock | 2 + Guide/src/reference/backends/storage.md | 2 +- vm/devices/storage/disk_blob/Cargo.toml | 2 + vm/devices/storage/disk_blob/src/blob/http.rs | 228 +++++++++++++----- .../disk_blob/src/tests/http_server.rs | 129 +++++++--- vm/devices/storage/disk_blob/src/tests/mod.rs | 165 +++++++++++-- 6 files changed, 420 insertions(+), 108 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 53d95267d5a..2da98457e67 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1563,11 +1563,13 @@ dependencies = [ "pal_async", "parking_lot", "scsi_buffers", + "socket2", "storage_tests", "tempfile", "test_with_tracing", "thiserror 2.0.16", "tokio", + "tracelimit", "vhd1_defs", "vm_resource", "zerocopy", diff --git a/Guide/src/reference/backends/storage.md b/Guide/src/reference/backends/storage.md index a1d6578bd6f..c325e0a5090 100644 --- a/Guide/src/reference/backends/storage.md +++ b/Guide/src/reference/backends/storage.md @@ -17,7 +17,7 @@ blob, or a layered composition of multiple backends. | Vhd1Disk | [`disk_vhd1`](https://openvmm.dev/rustdoc/linux/disk_vhd1/index.html) | VHD1 fixed file | Cross-platform | Parses VHD footer for geometry. | | VhdmpDisk | `disk_vhdmp` | Windows vhdmp driver | Windows | Dynamic and differencing VHD/VHDX. | | VhdxDisk | [`vhdx`](vhdx.md) | VHDX file | Cross-platform | Pure-Rust VHDX parser. Dynamic, fixed, and differencing. | -| BlobDisk | [`disk_blob`](https://openvmm.dev/rustdoc/linux/disk_blob/index.html) | HTTP / Azure Blob | Cross-platform | Read-only. HTTP range requests, with timeouts. | +| BlobDisk | [`disk_blob`](https://openvmm.dev/rustdoc/linux/disk_blob/index.html) | HTTP / Azure Blob | Cross-platform | Read-only. HTTP range requests, with timeouts and retries. | | BlockDeviceDisk | [`disk_blockdevice`](https://openvmm.dev/rustdoc/linux/disk_blockdevice/index.html) | Linux block device or file | Linux | io_uring, resize via uevent, PR passthrough. Default for raw files on Linux in both OpenHCL and OpenVMM. | | NvmeDisk | [`disk_nvme`](https://openvmm.dev/rustdoc/linux/disk_nvme/index.html) | Physical NVMe (VFIO) | Linux/Windows | User-mode NVMe driver. Resize via AEN. | | StripedDisk | [`disk_striped`](https://openvmm.dev/rustdoc/linux/disk_striped/index.html) | Multiple Disks | Cross-platform | Stripes data across underlying disks. | diff --git a/vm/devices/storage/disk_blob/Cargo.toml b/vm/devices/storage/disk_blob/Cargo.toml index 40eea051f03..bda34ff3b9c 100644 --- a/vm/devices/storage/disk_blob/Cargo.toml +++ b/vm/devices/storage/disk_blob/Cargo.toml @@ -16,6 +16,7 @@ guestmem.workspace = true vm_resource.workspace = true inspect = { workspace = true, features = ["filepath"] } +tracelimit.workspace = true anyhow.workspace = true async-trait.workspace = true @@ -36,6 +37,7 @@ tokio = { version = "1", features = ["rt-multi-thread", "time"] } disk_vhd1.workspace = true pal_async.workspace = true parking_lot.workspace = true +socket2.workspace = true storage_tests.workspace = true tempfile.workspace = true test_with_tracing.workspace = true diff --git a/vm/devices/storage/disk_blob/src/blob/http.rs b/vm/devices/storage/disk_blob/src/blob/http.rs index d1848ee8d47..0a3bd487ddb 100644 --- a/vm/devices/storage/disk_blob/src/blob/http.rs +++ b/vm/devices/storage/disk_blob/src/blob/http.rs @@ -34,7 +34,7 @@ use std::time::Duration; use thiserror::Error; use tokio::time::Instant; -/// Timeouts for the requests that an [`HttpBlob`] makes. +/// Timeouts and retries for the requests that an [`HttpBlob`] makes. /// /// A guest waits for its disk reads to complete, typically without a timeout /// of its own, so a request that never completes hangs the guest. And a request @@ -43,11 +43,18 @@ use tokio::time::Instant; /// will never come, with no reset or FIN to end the wait. So every request is /// bounded in time: /// -/// - It must receive the response headers within +/// - Each attempt must receive the response headers within /// [`response_timeout`](Self::response_timeout), and then more of the body at /// least every [`body_idle_timeout`](Self::body_idle_timeout). These limit /// silence rather than duration, so that a connection that has stopped is /// detected without cutting off a slow response that is still arriving. +/// - An attempt that fails in a way that the next one might not -- by timing +/// out, by losing its connection, or with status 408, 429, 500, 502, 503, or +/// 504 -- is retried on a new connection, after a delay that starts at +/// [`initial_backoff`](Self::initial_backoff) and doubles with each retry, up +/// to [`max_backoff`](Self::max_backoff), for up to +/// [`max_attempts`](Self::max_attempts) attempts in all. Any other failure is +/// returned at once. /// - [`operation_timeout`](Self::operation_timeout) bounds the request as a /// whole, so that even a response that trickles in too slowly ever to finish /// fails in bounded time. @@ -57,20 +64,29 @@ use tokio::time::Instant; /// /// The defaults suit a blob store such as Azure Blob Storage, where a range /// request normally completes in tens of milliseconds. A connection that is -/// silent for 20 seconds, a thousand times that, is taken to be dead, and no -/// request takes more than two minutes, so a hopeless request fails well before -/// a typical test watchdog, such as petri's ten minutes, would intervene. +/// silent for 20 seconds, a thousand times that, is taken to be dead; a request +/// whose every attempt stalls fails after about 80 seconds; and no request +/// takes more than two minutes. That rides out a transient network failure, +/// but still fails a hopeless request well before a typical test watchdog, such +/// as petri's ten minutes, would intervene. #[derive(Debug, Clone)] pub struct HttpBlobOptions { /// How long to wait for a TCP connection to be established. pub connect_timeout: Duration, - /// How long a request waits for the response headers, including the time + /// How long an attempt waits for the response headers, including the time /// to connect and to send the request. pub response_timeout: Duration, - /// How long a request waits for more of the response body. + /// How long an attempt waits for more of the response body. pub body_idle_timeout: Duration, - /// How long a request may take in all. + /// How long a request may take in all, across all its attempts. pub operation_timeout: Duration, + /// How many attempts to make at a request, including the first. At least + /// one attempt is always made. + pub max_attempts: u32, + /// The delay before the first retry. + pub initial_backoff: Duration, + /// The longest delay before a retry. + pub max_backoff: Duration, } impl Default for HttpBlobOptions { @@ -80,6 +96,9 @@ impl Default for HttpBlobOptions { response_timeout: Duration::from_secs(20), body_idle_timeout: Duration::from_secs(20), operation_timeout: Duration::from_secs(120), + max_attempts: 4, + initial_backoff: Duration::from_millis(250), + max_backoff: Duration::from_secs(4), } } } @@ -100,7 +119,8 @@ type HttpClient = Client, Empty<&'static [u8]>>; /// A blob backed by an HTTP/HTTPS connection. /// -/// Its requests are bounded in time, as [`HttpBlobOptions`] describes. +/// Its requests are bounded in time, and retried, as [`HttpBlobOptions`] +/// describes. #[derive(Debug, Inspect)] pub struct HttpBlob { #[inspect(skip)] @@ -122,7 +142,7 @@ impl HttpBlob { } /// Connects to `url` and returns an object to access it as a blob, with the - /// timeouts that `options` specifies. + /// timeouts and retries that `options` specifies. pub async fn with_options(url: &str, options: HttpBlobOptions) -> anyhow::Result { let uri: Uri = url.parse()?; @@ -188,11 +208,15 @@ impl Blob for HttpBlob { } } -/// Makes the requests for an [`HttpBlob`], with timeouts. +/// Makes the requests for an [`HttpBlob`], with timeouts and retries. #[derive(Debug)] struct Requester { options: HttpBlobOptions, + /// Makes first attempts, which may reuse idle connections. client: HttpClient, + /// Makes retries, each on a new connection: whatever broke the previous + /// attempt's connection may have broken the idle ones too. + retry_client: HttpClient, tokio_handle: tokio::runtime::Handle, } @@ -205,9 +229,12 @@ impl Requester { http.set_keepalive(Some(TCP_KEEPALIVE_TIME)); http.set_keepalive_interval(Some(TCP_KEEPALIVE_INTERVAL)); http.set_keepalive_retries(Some(TCP_KEEPALIVE_RETRIES)); + let connector = HttpsConnector::new_with_connector(http); Self { - client: Client::builder(TokioExecutor::new()) - .build(HttpsConnector::new_with_connector(http)), + client: Client::builder(TokioExecutor::new()).build(connector.clone()), + retry_client: Client::builder(TokioExecutor::new()) + .pool_max_idle_per_host(0) + .build(connector), options, tokio_handle, } @@ -289,31 +316,83 @@ impl Requester { .await } - /// Makes `request`, by calling `attempt` with the client to use, and fails - /// it if it has not completed by `deadline`. + /// Makes `request`, by calling `attempt` with the client to use, and + /// retrying it as the options say, until it succeeds, fails for good, or + /// `deadline` passes. async fn request( &self, request: RequestKind, deadline: Instant, - attempt: F, + mut attempt: F, ) -> io::Result where - F: FnOnce(HttpClient) -> Fut, - Fut: Future>, + F: FnMut(HttpClient) -> Fut, + Fut: Future>, { - match tokio::time::timeout_at(deadline, attempt(self.client.clone())).await { - Ok(result) => result, - Err(_) => Err(io::Error::new( - io::ErrorKind::TimedOut, - RequestError::OperationTimeout { - request, - timeout: self.options.operation_timeout, - }, - )), + let mut backoff = self.options.initial_backoff.min(self.options.max_backoff); + let mut attempts = 0; + loop { + attempts += 1; + let client = if attempts == 1 { + &self.client + } else { + &self.retry_client + }; + let (err, retry) = + match tokio::time::timeout_at(deadline, attempt(client.clone())).await { + Ok(Ok(output)) => return Ok(output), + Ok(Err(Failure::Transient(err))) => (err, true), + Ok(Err(Failure::Timeout(err))) => { + (io::Error::new(io::ErrorKind::TimedOut, err), true) + } + Ok(Err(Failure::Permanent(err))) => return Err(err), + Err(_) => { + let err = RequestError::OperationTimeout { + request, + timeout: self.options.operation_timeout, + }; + (io::Error::new(io::ErrorKind::TimedOut, err), false) + } + }; + + let next_attempt = Instant::now().checked_add(backoff); + if retry + && attempts < self.options.max_attempts + && next_attempt.is_some_and(|next| next < deadline) + { + tracelimit::warn_ratelimited!( + error = &err as &dyn std::error::Error, + %request, + attempt = attempts, + ?backoff, + "http blob request failed, retrying" + ); + tokio::time::sleep(backoff).await; + backoff = backoff.saturating_mul(2).min(self.options.max_backoff); + continue; + } + + tracelimit::warn_ratelimited!( + error = &err as &dyn std::error::Error, + %request, + attempts, + "http blob request failed" + ); + return Err(if retry && attempts > 1 { + io::Error::new( + err.kind(), + RequestError::Exhausted { + request, + attempts, + source: err, + }, + ) + } else { + err + }); } } } - /// Returns the instant `timeout` from now, or a distant one if that cannot be /// represented. fn deadline_after(timeout: Duration) -> Instant { @@ -327,7 +406,7 @@ async fn head_once( client: HttpClient, uri: Uri, response_timeout: Duration, -) -> io::Result> { +) -> Result, Failure> { let request = Request::builder() .uri(uri) .method(Method::HEAD) @@ -344,7 +423,7 @@ async fn get_once( len: usize, response_timeout: Duration, body_idle_timeout: Duration, -) -> io::Result> { +) -> Result, Failure> { let request = Request::builder() .uri(uri) .header( @@ -356,36 +435,41 @@ async fn get_once( let response = send(client, request, response_timeout).await?; if !response.status().is_success() { - return Err(io::Error::other(RequestError::Status(response.status()))); + return Err(Failure::Permanent(io::Error::other(RequestError::Status( + response.status(), + )))); } let mut body = response.into_body(); let mut data = Vec::with_capacity(len); loop { let frame = match tokio::time::timeout(body_idle_timeout, body.frame()).await { - Ok(Some(frame)) => frame.map_err(io::Error::other)?, + Ok(Some(Ok(frame))) => frame, Ok(None) => break, + // The connection failed part way through the body. + Ok(Some(Err(err))) => return Err(Failure::Transient(io::Error::other(err))), Err(_) => { - return Err(io::Error::new( - io::ErrorKind::TimedOut, - RequestError::BodyTimeout { - timeout: body_idle_timeout, - received: data.len(), - expected: len, - }, - )); + return Err(Failure::Timeout(RequestError::BodyTimeout { + timeout: body_idle_timeout, + received: data.len(), + expected: len, + })); } }; if let Some(chunk) = frame.data_ref() { if chunk.len() > len - data.len() { - return Err(io::Error::other("server did not respect range query")); + return Err(Failure::Permanent(io::Error::other( + "server did not respect range query", + ))); } data.extend_from_slice(chunk); } } if data.len() < len { - return Err(io::ErrorKind::UnexpectedEof.into()); + // A well-formed response that is shorter than was asked for. Another + // attempt would get the same. + return Err(Failure::Permanent(io::ErrorKind::UnexpectedEof.into())); } Ok(data) @@ -397,20 +481,39 @@ async fn send( client: HttpClient, request: Request>, response_timeout: Duration, -) -> io::Result> { - match tokio::time::timeout(response_timeout, client.request(request)).await { - Ok(Ok(response)) => Ok(response), +) -> Result, Failure> { + let response = match tokio::time::timeout(response_timeout, client.request(request)).await { + Ok(Ok(response)) => response, // The connector gave up on connecting. - Ok(Err(err)) if is_timed_out(&err) => Err(io::Error::new( - io::ErrorKind::TimedOut, - RequestError::ConnectTimeout(err), - )), - Ok(Err(err)) => Err(io::Error::other(err)), - Err(_) => Err(io::Error::new( - io::ErrorKind::TimedOut, - RequestError::ResponseTimeout(response_timeout), - )), + Ok(Err(err)) if is_timed_out(&err) => { + return Err(Failure::Timeout(RequestError::ConnectTimeout(err))); + } + // The request is well formed, so this is a failure to connect, or to + // keep the connection. + Ok(Err(err)) => return Err(Failure::Transient(io::Error::other(err))), + Err(_) => { + return Err(Failure::Timeout(RequestError::ResponseTimeout( + response_timeout, + ))); + } + }; + + let status = response.status(); + if matches!( + status, + StatusCode::REQUEST_TIMEOUT + | StatusCode::TOO_MANY_REQUESTS + | StatusCode::INTERNAL_SERVER_ERROR + | StatusCode::BAD_GATEWAY + | StatusCode::SERVICE_UNAVAILABLE + | StatusCode::GATEWAY_TIMEOUT + ) { + return Err(Failure::Transient(io::Error::other(RequestError::Status( + status, + )))); } + + Ok(response) } /// Returns whether `err`, or any error in its chain of sources, is a timeout, @@ -438,6 +541,16 @@ impl Drop for AbortOnDrop { } } +/// Why an attempt failed, which decides whether to retry it. +enum Failure { + /// The attempt failed in a way that the next attempt might not. + Transient(io::Error), + /// The attempt timed out, which the next attempt might not. + Timeout(RequestError), + /// The attempt failed in a way that the next attempt would repeat. + Permanent(io::Error), +} + #[derive(Debug, Error)] enum RequestError { #[error("connection timed out")] @@ -455,6 +568,13 @@ enum RequestError { request: RequestKind, timeout: Duration, }, + #[error("{request} failed after {attempts} attempts")] + Exhausted { + request: RequestKind, + attempts: u32, + #[source] + source: io::Error, + }, #[error("server returned {0}")] Status(StatusCode), } diff --git a/vm/devices/storage/disk_blob/src/tests/http_server.rs b/vm/devices/storage/disk_blob/src/tests/http_server.rs index aa1ccfa4d0c..6d756fab95d 100644 --- a/vm/devices/storage/disk_blob/src/tests/http_server.rs +++ b/vm/devices/storage/disk_blob/src/tests/http_server.rs @@ -9,10 +9,11 @@ //! importantly, the interesting cases are all responses a well-behaved server //! would never produce. See [`Behavior`]. //! -//! A [`Script`] can make the server misbehave for the first few requests only. -//! The server counts what it sees in [`Stats`], so that tests can wait for it -//! to reach a given point rather than sleep, and can check how many requests a -//! read made. +//! A [`Script`] can make the server misbehave for the first few requests only, +//! which models the transient failures that [`HttpBlob`] must recover from. The +//! server counts what it sees in [`Stats`], so that tests can wait for it to +//! reach a given point rather than sleep, and can check how many attempts a +//! read took. //! //! [`HttpBlob`]: crate::blob::http::HttpBlob @@ -51,6 +52,13 @@ pub enum Behavior { /// Send the headers and half the body, then stall as for /// [`Behavior::StallBeforeResponse`]. StallMidBody, + /// Reset the connection instead of answering. + ResetBeforeResponse, + /// Send the headers and half the body, then reset the connection. + ResetMidBody, + /// Answer `503 Service Unavailable`, as an overloaded or throttling server + /// does. + ServiceUnavailable, /// Answer correctly, but send the body in `pieces` pieces, pausing for /// `interval` before each: slow, but never silent for much longer than /// `interval`. @@ -63,12 +71,16 @@ pub struct Script { /// How to respond to the first HEAD requests, in order. Later ones are /// answered correctly. A HEAD response has no body, so a behavior that /// concerns the body acts as its counterpart that does not: a stall - /// stalls before the response, and the rest answer correctly. + /// stalls before the response, a reset resets before it, and the rest + /// answer correctly. pub head: Vec, /// How to respond to the first GET requests, in order. pub get: Vec, /// How to respond to the GET requests after those in `get`. pub then: Behavior, + /// Serve further requests on a connection after a response, as real + /// servers do, rather than closing it after each one. + pub keep_alive: bool, } /// What the server has seen so far. @@ -199,17 +211,21 @@ impl Drop for TestHttpServer { } impl Shared { - /// Serves one request on `stream`, then closes it. + /// Serves requests on `stream` until it is closed. fn serve(&self, mut stream: TcpStream) -> std::io::Result<()> { // Send each write at once, so that the client sees the pieces and // pauses of a response as the behavior describes them. stream.set_nodelay(true)?; - let request = read_headers(&mut stream)?; - // An empty request means the client has closed the connection. - if request.is_empty() { - return Ok(()); + loop { + let request = read_headers(&mut stream)?; + // An empty request means the client has closed the connection. + if request.is_empty() + || !self.respond(&mut stream, &request)? + || !self.script.keep_alive + { + return Ok(()); + } } - self.respond(&mut stream, &request) } /// Counts a request, and returns how to respond to it: as the next of the @@ -231,7 +247,9 @@ impl Shared { /// Holds the connection open without sending anything, until the client /// closes it or the server is released. - fn stall(&self, stream: &mut TcpStream) { + /// + /// Returns false, since the connection cannot carry another request. + fn stall(&self, stream: &mut TcpStream) -> bool { self.state.lock().stats.stalled += 1; self.changed.notify_all(); // Poll, rather than block, so that a release is noticed promptly. @@ -254,10 +272,11 @@ impl Shared { } } } + false } - /// Answers one request. - fn respond(&self, stream: &mut TcpStream, request: &str) -> std::io::Result<()> { + /// Answers one request. Returns whether the connection can carry another. + fn respond(&self, stream: &mut TcpStream, request: &str) -> std::io::Result { let content = &self.content; let mut lines = request.lines(); let method = lines @@ -265,26 +284,35 @@ impl Shared { .and_then(|l| l.split(' ').next()) .unwrap_or_default() .to_owned(); + let close = if self.script.keep_alive { + "" + } else { + "Connection: close\r\n" + }; // `HttpBlob::new` probes the length with a HEAD request. if method.eq_ignore_ascii_case("HEAD") { - if let Behavior::StallBeforeResponse | Behavior::StallMidBody = self.count( + match self.count( |stats| &mut stats.heads, &self.script.head, Behavior::Correct, ) { - self.stall(stream); - return Ok(()); + Behavior::StallBeforeResponse | Behavior::StallMidBody => { + return Ok(self.stall(stream)); + } + Behavior::ResetBeforeResponse | Behavior::ResetMidBody => return Ok(reset(stream)), + Behavior::ServiceUnavailable => return service_unavailable(stream, close), + _ => {} } write!( stream, "HTTP/1.1 200 OK\r\n\ Content-Length: {}\r\n\ Accept-Ranges: bytes\r\n\ - Connection: close\r\n\r\n", + {close}\r\n", content.len() )?; - return Ok(()); + return Ok(true); } let behavior = self.count(|stats| &mut stats.gets, &self.script.get, self.script.then); @@ -302,16 +330,18 @@ impl Shared { stream, "HTTP/1.1 400 Bad Request\r\nConnection: close\r\n\r\n" )?; - return Ok(()); + return Ok(false); }; let (start, end) = range.split_once('-').unwrap(); let start: usize = start.parse().unwrap(); // The range is inclusive at both ends. let end: usize = end.parse().unwrap(); - if behavior == Behavior::StallBeforeResponse { - self.stall(stream); - return Ok(()); + match behavior { + Behavior::StallBeforeResponse => return Ok(self.stall(stream)), + Behavior::ResetBeforeResponse => return Ok(reset(stream)), + Behavior::ServiceUnavailable => return service_unavailable(stream, close), + _ => {} } if behavior == Behavior::RangeNotSatisfiable || start >= content.len() { @@ -319,10 +349,11 @@ impl Shared { stream, "HTTP/1.1 416 Range Not Satisfiable\r\n\ Content-Range: bytes */{}\r\n\ - Connection: close\r\n\r\n", + Content-Length: 0\r\n\ + {close}\r\n", content.len() )?; - return Ok(()); + return Ok(true); } if behavior == Behavior::IgnoreRange { @@ -330,10 +361,11 @@ impl Shared { stream, "HTTP/1.1 200 OK\r\n\ Content-Length: {}\r\n\ - Connection: close\r\n\r\n", + {close}\r\n", content.len() )?; - return stream.write_all(content); + stream.write_all(content)?; + return Ok(true); } let end = end.min(content.len() - 1); @@ -357,7 +389,7 @@ impl Shared { "HTTP/1.1 206 Partial Content\r\n\ Content-Range: bytes {}-{}/{}\r\n\ Content-Length: {}\r\n\ - Connection: close\r\n\r\n", + {close}\r\n", start, start + body.len() - 1, content.len(), @@ -369,24 +401,55 @@ impl Shared { // with a truncated message rather than a well-formed short one. let half = &body[..body.len() / 2]; match behavior { - Behavior::CloseMidBody => stream.write_all(half), + Behavior::CloseMidBody => { + stream.write_all(half)?; + Ok(false) + } Behavior::StallMidBody => { stream.write_all(half)?; - self.stall(stream); - Ok(()) + Ok(self.stall(stream)) + } + Behavior::ResetMidBody => { + stream.write_all(half)?; + Ok(reset(stream)) } Behavior::SlowBody { pieces, interval } => { for piece in body.chunks(body.len().div_ceil(pieces.max(1))) { std::thread::sleep(interval); stream.write_all(piece)?; } - Ok(()) + Ok(true) + } + _ => { + stream.write_all(body)?; + Ok(true) } - _ => stream.write_all(body), } } } +/// Makes closing the connection send a reset rather than a FIN: with a zero +/// linger timeout, closing a socket discards any unsent data and aborts the +/// connection. +/// +/// Returns false, since the connection cannot carry another request. +fn reset(stream: &mut TcpStream) -> bool { + let _ = socket2::SockRef::from(&*stream).set_linger(Some(Duration::ZERO)); + false +} + +/// Answers `503 Service Unavailable`. Returns whether the connection can carry +/// another request. +fn service_unavailable(stream: &mut TcpStream, close: &str) -> std::io::Result { + write!( + stream, + "HTTP/1.1 503 Service Unavailable\r\n\ + Content-Length: 0\r\n\ + {close}\r\n" + )?; + Ok(true) +} + /// Reads up to and including the blank line that terminates the headers. fn read_headers(stream: &mut TcpStream) -> std::io::Result { let mut request = Vec::new(); diff --git a/vm/devices/storage/disk_blob/src/tests/mod.rs b/vm/devices/storage/disk_blob/src/tests/mod.rs index 504baf0464a..db5d6e6544f 100644 --- a/vm/devices/storage/disk_blob/src/tests/mod.rs +++ b/vm/devices/storage/disk_blob/src/tests/mod.rs @@ -176,6 +176,9 @@ fn fast_options() -> HttpBlobOptions { response_timeout: Duration::from_millis(500), body_idle_timeout: Duration::from_millis(500), operation_timeout: Duration::from_secs(5), + max_attempts: 3, + initial_backoff: Duration::from_millis(10), + max_backoff: Duration::from_millis(40), ..Default::default() } } @@ -240,22 +243,74 @@ fn read_blob( }) } -/// A server, or the network path to it, that goes silent before the response -/// fails the read in bounded time, and the stalled connection is closed. -#[test] -fn stall_before_response_fails_read() { - let server = TestHttpServer::new(content(), Behavior::StallBeforeResponse); - let (_blob, r) = read_blob(&server, fast_options(), 0, 4096); - match r { - Err(err) if err.kind() == std::io::ErrorKind::TimedOut => {} - r => panic!("expected a timeout, got {:?}", r.map(|_| ())), - } - assert_eq!(server.stats().gets, 1); - server.wait_for("the stalled connection to be closed", LIMIT, |stats| { - stats.abandoned == 1 +/// Reads through a server that fails the first GET requests as `failures` says, +/// and checks that the read recovers: one retry per failure, the right data, +/// and every stalled connection closed rather than leaked or reused. +fn check_recovers(failures: &[Behavior]) { + let content = content(); + let server = TestHttpServer::with_script( + content.clone(), + Script { + get: failures.to_vec(), + ..Default::default() + }, + ); + let (offset, len) = (12345, 64 * 1024); + let (_blob, r) = read_blob(&server, fast_options(), offset, len); + let data = r.unwrap_or_else(|err| panic!("{failures:?}: read failed: {err:?}")); + assert!( + data == content[offset as usize..][..len], + "{failures:?}: read returned the wrong data" + ); + assert_eq!( + server.stats().gets, + failures.len() + 1, + "{failures:?}: wrong number of attempts" + ); + server.wait_for("the stalled connections to be closed", LIMIT, |stats| { + stats.abandoned == stats.stalled }); } +/// A server, or the network path to it, that goes silent before the response. +#[test] +fn stall_before_response_is_retried() { + check_recovers(&[Behavior::StallBeforeResponse]); +} + +/// A server, or the network path to it, that goes silent part way through the +/// body. +#[test] +fn stall_mid_body_is_retried() { + check_recovers(&[Behavior::StallMidBody]); +} + +#[test] +fn reset_before_response_is_retried() { + check_recovers(&[Behavior::ResetBeforeResponse]); +} + +#[test] +fn reset_mid_body_is_retried() { + check_recovers(&[Behavior::ResetMidBody]); +} + +#[test] +fn close_mid_body_is_retried() { + check_recovers(&[Behavior::CloseMidBody]); +} + +#[test] +fn service_unavailable_is_retried() { + check_recovers(&[Behavior::ServiceUnavailable]); +} + +/// Every attempt is bounded and retried alike, up to the attempt limit. +#[test] +fn consecutive_failures_are_retried() { + check_recovers(&[Behavior::ResetBeforeResponse, Behavior::StallMidBody]); +} + /// A guest waits for its disk reads, typically without a timeout of its own, /// so a read that never completes hangs the guest. When the server never /// recovers, the read must fail instead, in bounded time. @@ -264,6 +319,7 @@ fn persistent_stall_fails_disk_read() { let server = TestHttpServer::new(content(), Behavior::StallMidBody); let url = server.url(); let options = fast_options(); + let attempts = options.max_attempts as usize; let r = bounded("disk read", async move { let blob = HttpBlob::with_options(&url, options).await.unwrap(); let disk = Disk::new(BlobDisk::new(blob)).unwrap(); @@ -278,9 +334,9 @@ fn persistent_stall_fails_disk_read() { Err(DiskError::Io(err)) if err.kind() == std::io::ErrorKind::TimedOut => {} r => panic!("expected a timeout, got {r:?}"), } - assert_eq!(server.stats().gets, 1); - server.wait_for("the stalled connection to be closed", LIMIT, |stats| { - stats.abandoned == 1 + assert_eq!(server.stats().gets, attempts); + server.wait_for("the stalled connections to be closed", LIMIT, |stats| { + stats.abandoned == attempts }); } @@ -344,9 +400,10 @@ fn endless_body_is_bounded_by_operation_timeout() { ); } -/// Opening the blob probes it with a HEAD request, which must not hang either. +/// Opening the blob probes it with a HEAD request, which must recover from a +/// stall too. #[test] -fn stalled_head_fails_open() { +fn stalled_head_is_retried() { let server = TestHttpServer::with_script( content(), Script { @@ -355,11 +412,79 @@ fn stalled_head_fails_open() { }, ); let url = server.url(); + let blob = bounded("open", async move { + HttpBlob::with_options(&url, fast_options()).await.unwrap() + }); + assert_eq!(blob.len(), DISK_SIZE); + assert_eq!(server.stats().heads, 2); +} + +#[test] +fn persistent_head_stall_fails_open() { + let options = fast_options(); + let attempts = options.max_attempts as usize; + let server = TestHttpServer::with_script( + content(), + Script { + head: vec![Behavior::StallBeforeResponse; attempts + 1], + ..Default::default() + }, + ); + let url = server.url(); let r = bounded("open", async move { - HttpBlob::with_options(&url, fast_options()).await.map(drop) + HttpBlob::with_options(&url, options).await.map(drop) }); assert!(r.is_err(), "opened a blob whose server never answered"); - assert_eq!(server.stats().heads, 1); + assert_eq!(server.stats().heads, attempts); +} + +/// A connection whose request timed out must be closed, not returned to the +/// pool, where a later request would find it and stall in turn. +#[test] +fn stalled_connection_is_not_reused() { + let content = content(); + let server = TestHttpServer::with_script( + content.clone(), + Script { + get: vec![Behavior::Correct, Behavior::StallBeforeResponse], + keep_alive: true, + ..Default::default() + }, + ); + let url = server.url(); + let (_blob, results) = bounded("reads", async move { + let blob = HttpBlob::with_options(&url, fast_options()).await.unwrap(); + let mut results = Vec::new(); + for i in 0..3 { + let mut buf = vec![0; 4096]; + results.push(blob.read(&mut buf, i * 4096).await.map(|()| buf)); + } + (blob, results) + }); + for (i, r) in results.into_iter().enumerate() { + let data = r.unwrap_or_else(|err| panic!("read {i} failed: {err:?}")); + assert!(data == content[i * 4096..][..4096], "read {i}: wrong data"); + } + // Only the stalled request was retried. + assert_eq!(server.stats().gets, 4); + server.wait_for("the stalled connection to be closed", LIMIT, |stats| { + stats.abandoned == 1 + }); +} + +/// A failure that a new attempt would only repeat is returned at once. +#[test] +fn permanent_errors_are_not_retried() { + for behavior in [ + Behavior::ShortBody, + Behavior::IgnoreRange, + Behavior::RangeNotSatisfiable, + ] { + let server = TestHttpServer::new(content(), behavior); + let (_blob, r) = read_blob(&server, fast_options(), 0, 4096); + assert!(r.is_err(), "{behavior:?}: reported success"); + assert_eq!(server.stats().gets, 1, "{behavior:?}: retried"); + } } /// A range whose last byte would lie past the largest offset that a request From b92daa350c93914b2eb556acf47886cc313d396b Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Tue, 27 Oct 2026 16:11:20 -0700 Subject: [PATCH 3/3] disk_blob: show HTTP blob requests in inspect A guest that hangs on its disk leaves little to go on. The watchdog dumps in microsoft/nvx#170 showed a guest READ stuck in storvsp's in-flight table, but not which layer of the lazily fetched disk held it: the blob's inspect node showed only its URI, length, and HTTP version. Telling a stalled HTTP request apart from a slow one, or from a stall elsewhere, took a reproduction. List each request in progress in the blob's inspect node, under in_flight: the request and its range, the current attempt, what it is waiting for, which is the response, more of the body, or the delay before a retry, how long ago it started, and how much of the body has arrived. Count the attempts made, the retries, the attempts cut short by a timeout, and the requests that failed for good, so that a dump also shows how the blob's requests have fared. A request leaves the table when it completes, fails, or is dropped with its read. The tests check that inspection shows a stalled request while it waits, and an empty table and the retry once the request completes, and that a stalled connection that is not reused costs exactly one retry and one timeout. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- Cargo.lock | 1 + vm/devices/storage/disk_blob/Cargo.toml | 4 +- vm/devices/storage/disk_blob/src/blob/http.rs | 179 ++++++++++++++++-- vm/devices/storage/disk_blob/src/tests/mod.rs | 62 +++++- 4 files changed, 224 insertions(+), 22 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 2da98457e67..07f89ec0da6 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1559,6 +1559,7 @@ dependencies = [ "hyper-tls", "hyper-util", "inspect", + "inspect_counters", "once_cell", "pal_async", "parking_lot", diff --git a/vm/devices/storage/disk_blob/Cargo.toml b/vm/devices/storage/disk_blob/Cargo.toml index bda34ff3b9c..3a23c9b0733 100644 --- a/vm/devices/storage/disk_blob/Cargo.toml +++ b/vm/devices/storage/disk_blob/Cargo.toml @@ -16,6 +16,7 @@ guestmem.workspace = true vm_resource.workspace = true inspect = { workspace = true, features = ["filepath"] } +inspect_counters.workspace = true tracelimit.workspace = true anyhow.workspace = true @@ -27,6 +28,7 @@ hyper = { workspace = true, features = ["client", "http1", "http2"] } hyper-tls.workspace = true hyper-util = { workspace = true, features = ["client", "client-legacy", "http1", "http2"] } once_cell.workspace = true +parking_lot.workspace = true thiserror.workspace = true zerocopy.workspace = true # tokio use is allowed in this crate only. @@ -35,8 +37,8 @@ tokio = { version = "1", features = ["rt-multi-thread", "time"] } [dev-dependencies] disk_vhd1.workspace = true +inspect = { workspace = true, features = ["initiate"] } pal_async.workspace = true -parking_lot.workspace = true socket2.workspace = true storage_tests.workspace = true tempfile.workspace = true diff --git a/vm/devices/storage/disk_blob/src/blob/http.rs b/vm/devices/storage/disk_blob/src/blob/http.rs index 0a3bd487ddb..ee209a616c5 100644 --- a/vm/devices/storage/disk_blob/src/blob/http.rs +++ b/vm/devices/storage/disk_blob/src/blob/http.rs @@ -25,11 +25,16 @@ use hyper_util::client::legacy::Client; use hyper_util::client::legacy::connect::HttpConnector; use hyper_util::rt::TokioExecutor; use inspect::Inspect; +use inspect_counters::SharedCounter; use once_cell::sync::OnceCell; +use parking_lot::Mutex; +use std::collections::BTreeMap; use std::fmt; use std::fmt::Debug; use std::io; use std::sync::Arc; +use std::sync::atomic::AtomicU64; +use std::sync::atomic::Ordering; use std::time::Duration; use thiserror::Error; use tokio::time::Instant; @@ -123,7 +128,7 @@ type HttpClient = Client, Empty<&'static [u8]>>; /// describes. #[derive(Debug, Inspect)] pub struct HttpBlob { - #[inspect(skip)] + #[inspect(flatten)] requester: Arc, #[inspect(debug)] version: http::Version, @@ -209,15 +214,31 @@ impl Blob for HttpBlob { } /// Makes the requests for an [`HttpBlob`], with timeouts and retries. -#[derive(Debug)] +#[derive(Debug, Inspect)] struct Requester { + #[inspect(skip)] options: HttpBlobOptions, /// Makes first attempts, which may reuse idle connections. + #[inspect(skip)] client: HttpClient, /// Makes retries, each on a new connection: whatever broke the previous /// attempt's connection may have broken the idle ones too. + #[inspect(skip)] retry_client: HttpClient, + #[inspect(skip)] tokio_handle: tokio::runtime::Handle, + /// Attempts made, retries included. + requests: SharedCounter, + /// Attempts made after a failed attempt. + retries: SharedCounter, + /// Attempts cut short by a timeout. + timeouts: SharedCounter, + /// Requests that failed, after any retries. + failures: SharedCounter, + /// The requests in progress. + in_flight: InFlightTable, + #[inspect(skip)] + next_id: AtomicU64, } impl Requester { @@ -237,6 +258,12 @@ impl Requester { .build(connector), options, tokio_handle, + requests: SharedCounter::new(), + retries: SharedCounter::new(), + timeouts: SharedCounter::new(), + failures: SharedCounter::new(), + in_flight: InFlightTable::default(), + next_id: AtomicU64::new(0), } } @@ -265,7 +292,7 @@ impl Requester { } let response = self - .request(RequestKind::Head, deadline, |client| { + .request(RequestKind::Head, deadline, |client, _| { head_once(client, uri.clone(), response_timeout) }) .await @@ -303,50 +330,65 @@ impl Requester { let deadline = deadline_after(self.options.operation_timeout); let response_timeout = self.options.response_timeout; let body_idle_timeout = self.options.body_idle_timeout; - self.request(RequestKind::Get { offset, len }, deadline, |client| { - get_once( - client, - uri.clone(), - offset, - len, - response_timeout, - body_idle_timeout, - ) - }) + self.request( + RequestKind::Get { offset, len }, + deadline, + |client, progress| { + get_once( + client, + uri.clone(), + offset, + len, + response_timeout, + body_idle_timeout, + progress, + ) + }, + ) .await } - /// Makes `request`, by calling `attempt` with the client to use, and - /// retrying it as the options say, until it succeeds, fails for good, or - /// `deadline` passes. + /// Makes `request`, by calling `attempt` with the client to use and a + /// handle to report progress through, and retrying it as the options + /// say, until it succeeds, fails for good, or `deadline` passes. async fn request( - &self, + self: &Arc, request: RequestKind, deadline: Instant, mut attempt: F, ) -> io::Result where - F: FnMut(HttpClient) -> Fut, + F: FnMut(HttpClient, Progress) -> Fut, Fut: Future>, { + let tracker = self.track(request); let mut backoff = self.options.initial_backoff.min(self.options.max_backoff); let mut attempts = 0; loop { attempts += 1; + tracker.0.update(|entry| entry.start_attempt(attempts)); + self.requests.increment(); let client = if attempts == 1 { &self.client } else { &self.retry_client }; let (err, retry) = - match tokio::time::timeout_at(deadline, attempt(client.clone())).await { + match tokio::time::timeout_at(deadline, attempt(client.clone(), tracker.0.clone())) + .await + { Ok(Ok(output)) => return Ok(output), Ok(Err(Failure::Transient(err))) => (err, true), Ok(Err(Failure::Timeout(err))) => { + self.timeouts.increment(); (io::Error::new(io::ErrorKind::TimedOut, err), true) } - Ok(Err(Failure::Permanent(err))) => return Err(err), + Ok(Err(Failure::Permanent(err))) => { + self.failures.increment(); + return Err(err); + } Err(_) => { + self.timeouts.increment(); let err = RequestError::OperationTimeout { request, timeout: self.options.operation_timeout, @@ -367,11 +409,14 @@ impl Requester { ?backoff, "http blob request failed, retrying" ); + self.retries.increment(); + tracker.0.update(|entry| entry.state = State::BackingOff); tokio::time::sleep(backoff).await; backoff = backoff.saturating_mul(2).min(self.options.max_backoff); continue; } + self.failures.increment(); tracelimit::warn_ratelimited!( error = &err as &dyn std::error::Error, %request, @@ -392,7 +437,28 @@ impl Requester { }); } } + + /// Lists `request` in [`Requester::in_flight`] until the returned tracker + /// is dropped. + fn track(self: &Arc, request: RequestKind) -> Tracker { + let id = self.next_id.fetch_add(1, Ordering::Relaxed); + self.in_flight.0.lock().insert( + id, + InFlight { + request, + attempt: 0, + state: State::AwaitingResponse, + started: Instant::now(), + received: 0, + }, + ); + Tracker(Progress { + requester: self.clone(), + id, + }) + } } + /// Returns the instant `timeout` from now, or a distant one if that cannot be /// represented. fn deadline_after(timeout: Duration) -> Instant { @@ -423,6 +489,7 @@ async fn get_once( len: usize, response_timeout: Duration, body_idle_timeout: Duration, + progress: Progress, ) -> Result, Failure> { let request = Request::builder() .uri(uri) @@ -440,6 +507,7 @@ async fn get_once( )))); } + progress.update(|entry| entry.state = State::ReceivingBody); let mut body = response.into_body(); let mut data = Vec::with_capacity(len); loop { @@ -463,6 +531,7 @@ async fn get_once( ))); } data.extend_from_slice(chunk); + progress.update(|entry| entry.received = data.len()); } } @@ -600,6 +669,76 @@ impl fmt::Display for RequestKind { } } +/// The requests in progress, by ID. +#[derive(Debug, Default)] +struct InFlightTable(Mutex>); + +impl Inspect for InFlightTable { + fn inspect(&self, req: inspect::Request<'_>) { + inspect::iter_by_key(self.0.lock().iter()).inspect(req); + } +} + +/// A request in progress, as inspection shows it. +#[derive(Debug, Inspect)] +struct InFlight { + /// The request, such as `GET bytes=0-4095`. + #[inspect(display)] + request: RequestKind, + /// The current attempt, counting from 1. + attempt: u32, + /// What the request is waiting for. + state: State, + /// How long ago the request started. + #[inspect(rename = "elapsed", with = "Instant::elapsed")] + started: Instant, + /// How much of the response body the current attempt has received. + received: usize, +} + +impl InFlight { + fn start_attempt(&mut self, attempt: u32) { + self.attempt = attempt; + self.state = State::AwaitingResponse; + self.received = 0; + } +} + +/// What a request in progress is waiting for. +#[derive(Debug, Copy, Clone, Inspect)] +enum State { + /// The response headers, connecting first if need be. + AwaitingResponse, + /// More of the response body. + ReceivingBody, + /// The delay before the next attempt. + BackingOff, +} + +/// A handle through which an attempt reports its progress to inspection. +#[derive(Clone)] +struct Progress { + requester: Arc, + id: u64, +} + +impl Progress { + fn update(&self, f: impl FnOnce(&mut InFlight)) { + if let Some(entry) = self.requester.in_flight.0.lock().get_mut(&self.id) { + f(entry); + } + } +} + +/// Removes a request from [`Requester::in_flight`] when dropped. +struct Tracker(Progress); + +impl Drop for Tracker { + fn drop(&mut self) { + self.0.requester.in_flight.0.lock().remove(&self.0.id); + } +} + #[cfg(test)] mod tests { use super::is_timed_out; diff --git a/vm/devices/storage/disk_blob/src/tests/mod.rs b/vm/devices/storage/disk_blob/src/tests/mod.rs index db5d6e6544f..f6646db5454 100644 --- a/vm/devices/storage/disk_blob/src/tests/mod.rs +++ b/vm/devices/storage/disk_blob/src/tests/mod.rs @@ -21,6 +21,7 @@ use scsi_buffers::OwnedRequestBuffers; use std::io::Read; use std::io::Seek; use std::io::SeekFrom; +use std::sync::Arc; use std::sync::mpsc; use std::thread::JoinHandle; use std::time::Duration; @@ -243,6 +244,11 @@ fn read_blob( }) } +/// Returns the JSON for `path` in the blob's inspection output. +fn inspect_json(blob: &HttpBlob, path: &str) -> String { + inspect::inspect(path, blob).results().json().to_string() +} + /// Reads through a server that fails the first GET requests as `failures` says, /// and checks that the read recovers: one retry per failure, the right data, /// and every stalled connection closed rather than leaked or reused. @@ -452,7 +458,7 @@ fn stalled_connection_is_not_reused() { }, ); let url = server.url(); - let (_blob, results) = bounded("reads", async move { + let (blob, results) = bounded("reads", async move { let blob = HttpBlob::with_options(&url, fast_options()).await.unwrap(); let mut results = Vec::new(); for i in 0..3 { @@ -466,12 +472,66 @@ fn stalled_connection_is_not_reused() { assert!(data == content[i * 4096..][..4096], "read {i}: wrong data"); } // Only the stalled request was retried. + assert_eq!(inspect_json(&blob, "retries"), "1"); + assert_eq!(inspect_json(&blob, "timeouts"), "1"); assert_eq!(server.stats().gets, 4); server.wait_for("the stalled connection to be closed", LIMIT, |stats| { stats.abandoned == 1 }); } +/// Inspection shows what a request that is taking long is waiting for, so that +/// a hung guest's disk can be told apart from a slow one. +#[test] +fn inspect_shows_stalled_request() { + let content = content(); + let server = TestHttpServer::with_script( + content.clone(), + Script { + get: vec![Behavior::StallBeforeResponse], + ..Default::default() + }, + ); + let url = server.url(); + // Long enough that the stall does not time out while it is inspected. + // Releasing the server ends it instead. + let options = HttpBlobOptions { + response_timeout: LIMIT * 3, + operation_timeout: LIMIT * 3, + ..fast_options() + }; + let blob = Arc::new(bounded("open", async move { + HttpBlob::with_options(&url, options).await.unwrap() + })); + let read = background({ + let blob = blob.clone(); + async move { + let mut buf = vec![0; 4096]; + blob.read(&mut buf, 8192).await.map(|()| buf) + } + }); + server.wait_for("the request to stall", LIMIT, |stats| stats.stalled == 1); + let in_flight = inspect_json(&blob, "in_flight"); + for expected in [ + r#""request":"GET bytes=8192-12287""#, + r#""attempt":1"#, + r#""state":"awaiting_response""#, + ] { + assert!( + in_flight.contains(expected), + "{expected} not in {in_flight}" + ); + } + + // Ending the stall fails the attempt, and the retry succeeds. + server.release(); + let data = read.wait("read").unwrap(); + assert!(data == content[8192..][..4096], "wrong data"); + assert_eq!(inspect_json(&blob, "in_flight"), "{}"); + assert_eq!(inspect_json(&blob, "retries"), "1"); + assert_eq!(inspect_json(&blob, "timeouts"), "0"); +} + /// A failure that a new attempt would only repeat is returned at once. #[test] fn permanent_errors_are_not_retried() {