Flake in o11y integration test

[github-actions]

Failure in Google Cloud Build on main:

• Build: gcb-pm-integration (rust-sdk-testing)
• Build Log: https://console.cloud.google.com/cloud-build/builds;region=us-central1/cd62cf1e-e9c0-4ac9-80a7-a07153e0f881?project=1044885715437
• Commit: d6cf097

Associated Pull Requests:

• test(discovery): disable flaky test #5981

[github-actions]

@googleapis/cloud-sdk-rust-team A critical issue has been created, please respond immediately.

[coryan]

The log says:

---- signals::run stdout ----
--
  | Trace not found yet (or internal error), retrying... Error: Error { kind: Service(ServiceDetails { status_code: Some(404), headers: Some({"vary": "X-Origin", "vary": "Referer", "vary": "Origin,Accept-Encoding", "content-type": "application/json; charset=UTF-8", "date": "Wed, 01 Jul 2026 15:03:26 GMT", "server": "ESF", "x-xss-protection": "0", "x-frame-options": "SAMEORIGIN", "x-content-type-options": "nosniff", "server-timing": "gfet4t7; dur=1812", "accept-ranges": "none", "transfer-encoding": "chunked"}), status: Status { code: NotFound, message: "Trace not found: 8cdfe74938ee110581a21dd4359e07fb", details: [] } }), source: None }
  | Trace not found yet (or internal error), retrying... Error: Error { kind: Service(ServiceDetails { status_code: Some(404), headers: Some({"vary": "X-Origin", "vary": "Referer", "vary": "Origin,Accept-Encoding", "content-type": "application/json; charset=UTF-8", "date": "Wed, 01 Jul 2026 15:04:29 GMT", "server": "ESF", "x-xss-protection": "0", "x-frame-options": "SAMEORIGIN", "x-content-type-options": "nosniff", "server-timing": "gfet4t7; dur=2295", "accept-ranges": "none", "transfer-encoding": "chunked"}), status: Status { code: NotFound, message: "Trace not found: 8cdfe74938ee110581a21dd4359e07fb", details: [] } }), source: None }
  | Trace not found yet (or internal error), retrying... Error: Error { kind: Service(ServiceDetails { status_code: Some(404), headers: Some({"vary": "X-Origin", "vary": "Referer", "vary": "Origin,Accept-Encoding", "content-type": "application/json; charset=UTF-8", "date": "Wed, 01 Jul 2026 15:06:31 GMT", "server": "ESF", "x-xss-protection": "0", "x-frame-options": "SAMEORIGIN", "x-content-type-options": "nosniff", "server-timing": "gfet4t7; dur=2036", "accept-ranges": "none", "transfer-encoding": "chunked"}), status: Status { code: NotFound, message: "Trace not found: 8cdfe74938ee110581a21dd4359e07fb", details: [] } }), source: None }
  | Trace not found yet (or internal error), retrying... Error: Error { kind: Service(ServiceDetails { status_code: Some(404), headers: Some({"vary": "X-Origin", "vary": "Referer", "vary": "Origin,Accept-Encoding", "content-type": "application/json; charset=UTF-8", "date": "Wed, 01 Jul 2026 15:08:33 GMT", "server": "ESF", "x-xss-protection": "0", "x-frame-options": "SAMEORIGIN", "x-content-type-options": "nosniff", "server-timing": "gfet4t7; dur=2045", "accept-ranges": "none", "transfer-encoding": "chunked"}), status: Status { code: NotFound, message: "Trace not found: 8cdfe74938ee110581a21dd4359e07fb", details: [] } }), source: None }
  | Trace not found yet (or internal error), retrying... Error: Error { kind: Service(ServiceDetails { status_code: Some(404), headers: Some({"vary": "X-Origin", "vary": "Referer", "vary": "Origin,Accept-Encoding", "content-type": "application/json; charset=UTF-8", "date": "Wed, 01 Jul 2026 15:10:35 GMT", "server": "ESF", "x-xss-protection": "0", "x-frame-options": "SAMEORIGIN", "x-content-type-options": "nosniff", "server-timing": "gfet4t7; dur=1719", "accept-ranges": "none", "transfer-encoding": "chunked"}), status: Status { code: NotFound, message: "Trace not found: 8cdfe74938ee110581a21dd4359e07fb", details: [] } }), source: None }
  | Error: Timed out waiting for trace

The test runs for 5 minutes, which seems fairly generous.

First things first, I will disable the test. I think we may want to use a different strategy to run these end-to-end tests... At this point, we are no longer testing our code, we are testing the response time of the service.

[dbolduc]

FWIW, we did send the trace.

So yes, the build failed because the service was slower than we expected.

[coryan]

I think we should refactor the test to use showcase and capture the traces / metrics / logs using mock collectors, instead of using the service.

The only code that we would not be testing with that approach is the code to send traces to Google's OTLP endpoint, which we don't publish to any package, it is just a proof of concept, so why test it?