Skip to content

fix: OTLP export silently skips when endpoints are configured but no auth token is present.#199

Merged
jingxiang-z merged 1 commit into
NVIDIA:mainfrom
miconeilaws:feat/custom-otlp-export
Jun 4, 2026
Merged

fix: OTLP export silently skips when endpoints are configured but no auth token is present.#199
jingxiang-z merged 1 commit into
NVIDIA:mainfrom
miconeilaws:feat/custom-otlp-export

Conversation

@miconeilaws

@miconeilaws miconeilaws commented May 20, 2026

Copy link
Copy Markdown
Contributor

Description

Bug: OTLP export silently skips when endpoints are configured but no auth token is present.

The README lists OTLP remote export as a supported format and the metadata store allows configuring metrics_endpoint/logs_endpoint independently. However, exportToHTTP has an early return when AuthToken == "" that silently drops all data even though the HTTP writer already handles missing tokens correctly (just omits the header).

This removes the redundant gate so the export path works as documented when endpoints are configured.

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

Summary by CodeRabbit

  • New Features

    • HTTP export proceeds when no authentication token is configured; a warning is logged if endpoints are present but the token is empty.
    • If the server returns an updated token, the client adopts and attempts to persist it; export still completes even if persistence fails.
  • Tests

    • Added tests validating export without an auth token and verifying token-refresh and persistence behavior via a test server.

@coderabbitai

coderabbitai Bot commented May 20, 2026

Copy link
Copy Markdown

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

exportToHTTP now skips only when both endpoints are unset; if endpoints exist but AuthToken is empty it logs a warning and still calls httpWriter.Send, handling any returned jwt_assertion by updating in-memory config and persisting it to metadata; tests validate refresh persistence and header behavior.

Changes

HTTP export authentication guard removal

Layer / File(s) Summary
Exporter HTTP flow and JWT persistence
internal/exporter/exporter.go
Call httpWriter.Send when MetricsEndpoint or LogsEndpoint are set even if AuthToken is empty; log a warning for empty token; on Send success, if a non-empty jwt_assertion differs from the current token, persist it via updateTokenInMetadata and update e.options.config.AuthToken.
Tests: JWT refresh and no-auth export
internal/exporter/exporter_test.go
Replace and add subtests: one starts an httptest server returning a jwt_assertion header and verifies in-memory and persisted token refresh using a temporary SQLite metadata DB; another configures endpoints with an empty AuthToken, asserts POSTs occur without an Authorization header using an atomic counter.

Sequence Diagram

sequenceDiagram
  participant Exporter
  participant HTTPWriter
  participant MetadataDB
  Exporter->>HTTPWriter: Send(payload, maybe-empty AuthToken)
  HTTPWriter-->>Exporter: response + jwt_assertion (newToken)
  Exporter->>MetadataDB: updateTokenInMetadata(newToken)
  MetadataDB-->>Exporter: success / failure
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Poem

🐰 I hopped to send a call with pockets bare of key,
The server hummed a secret back and dropped it next to me.
I tucked the shiny token in a warm and tiny nest,
Sent headers plain when empty, then sat down to run the test.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly and specifically describes the main bug being fixed: OTLP export being silently skipped when endpoints are configured but auth token is missing.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@miconeilaws miconeilaws force-pushed the feat/custom-otlp-export branch from 5682feb to 33f9ba0 Compare May 20, 2026 16:34

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
internal/exporter/exporter_test.go (1)

563-590: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Remove obsolete test that conflicts with the new behavior.

This test expects the old behavior where exportToHTTP would skip when AuthToken is empty. However, the PR removes that early-return guard, allowing exports to proceed without an auth token when endpoints are configured. This test now conflicts with the new test at lines 647-689, which validates the same scenario (endpoints configured, no auth token) but expects the HTTP request to actually be made.

The test name "skips when no auth token configured" and the comment on line 585 ("Should not error, just skip gracefully") confirm this is testing obsolete behavior that the PR explicitly changes.

🗑️ Suggested action

Remove this entire test case since it validates behavior that no longer exists after this PR's changes.

-	t.Run("skips when no auth token configured", func(t *testing.T) {
-		cfg := &config.HealthExporterConfig{
-			Interval:        metav1.Duration{Duration: 1 * time.Minute},
-			Timeout:         metav1.Duration{Duration: 30 * time.Second},
-			MetricsEndpoint: "http://example.com",
-			LogsEndpoint:    "http://example.com",
-			AuthToken:       "",
-		}
-
-		ctx := context.Background()
-		exporter, err := New(ctx, WithConfig(cfg), WithMachineID("test-machine-id"))
-		require.NoError(t, err)
-		require.NotNil(t, exporter)
-
-		he := exporter.(*healthExporter)
-
-		healthData := &collector.HealthData{
-			MachineID: "test-machine",
-			Timestamp: time.Now(),
-		}
-
-		err = he.exportToHTTP(ctx, healthData)
-		require.NoError(t, err) // Should not error, just skip gracefully
-
-		// Cleanup
-		err = exporter.Stop()
-		require.NoError(t, err)
-	})
-
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@internal/exporter/exporter_test.go` around lines 563 - 590, The test "skips
when no auth token configured" is obsolete because exportToHTTP no longer
early-returns when AuthToken is empty; remove the entire t.Run block that
defines that test (including the cfg setup, New(...), the exportToHTTP call and
cleanup) to avoid conflicting with the updated behavior validated later; search
for the test by its name string and by references to exportToHTTP and
healthExporter in this file to locate and delete it.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@internal/exporter/exporter_test.go`:
- Around line 563-590: The test "skips when no auth token configured" is
obsolete because exportToHTTP no longer early-returns when AuthToken is empty;
remove the entire t.Run block that defines that test (including the cfg setup,
New(...), the exportToHTTP call and cleanup) to avoid conflicting with the
updated behavior validated later; search for the test by its name string and by
references to exportToHTTP and healthExporter in this file to locate and delete
it.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 7f272271-3a48-4b3c-b8e3-e8e867678b78

📥 Commits

Reviewing files that changed from the base of the PR and between 9dda598 and 33f9ba0.

📒 Files selected for processing (2)
  • internal/exporter/exporter.go
  • internal/exporter/exporter_test.go
💤 Files with no reviewable changes (1)
  • internal/exporter/exporter.go

@miconeilaws miconeilaws force-pushed the feat/custom-otlp-export branch from 33f9ba0 to 016cd2f Compare May 20, 2026 16:52

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)
internal/exporter/exporter_test.go (2)

618-659: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Verify at least one request was sent in the no-auth path.

At Line 620/654, header assertions run only if the handler is hit. If export is skipped, this test still passes. Add an explicit “request observed” assertion.

Suggested request-observed guard
 	t.Run("exports without auth token when endpoints configured", func(t *testing.T) {
+		requestObserved := make(chan struct{}, 1)
 		// Create mock server that accepts requests without Authorization header
 		server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			select {
+			case requestObserved <- struct{}{}:
+			default:
+			}
 			assert.Equal(t, http.MethodPost, r.Method)
 			assert.Empty(t, r.Header.Get("Authorization"), "should not send Authorization header when token is empty")
 			w.WriteHeader(http.StatusOK)
 		}))
 		defer server.Close()
@@
 		err = he.exportToHTTP(ctx, healthData)
 		require.NoError(t, err)
+		select {
+		case <-requestObserved:
+		default:
+			t.Fatal("expected at least one HTTP export request")
+		}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@internal/exporter/exporter_test.go` around lines 618 - 659, The test
currently only asserts headers inside the httptest handler which will silently
pass if no request is made; modify the test for the "exports without auth token
when endpoints configured" case to record that the handler was hit (e.g.,
increment an atomic counter or send a value on a channel from the httptest
handler) and after calling he.exportToHTTP(ctx, healthData) assert that at least
one request was observed; locate the handler closure created for server :=
httptest.NewServer(...) and the export call to healthExporter.exportToHTTP to
add the observation mechanism and final assertion, then run exporter.Stop() as
before.

563-616: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Assert JWT refresh outcomes, not just success.

At Line 610 the test only verifies exportToHTTP returns no error; it never checks that the new JWT was persisted or reflected in config, so a broken refresh path can still pass.

Suggested test assertions
 		err = he.exportToHTTP(ctx, healthData)
 		require.NoError(t, err)
+
+		storedToken, err := pkgmetadata.ReadMetadata(ctx, tmpDB, pkgmetadata.MetadataKeyToken)
+		require.NoError(t, err)
+		assert.Equal(t, newToken, storedToken)
+		assert.Equal(t, newToken, he.options.config.AuthToken)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@internal/exporter/exporter_test.go` around lines 563 - 616, The test
currently only checks exportToHTTP returned no error; update it to also assert
the JWT was refreshed: after calling he.exportToHTTP(ctx, healthData) assert
that the active token was replaced with newToken (e.g. check cfg.AuthToken or
he.config.AuthToken) and, if the implementation persists tokens to the DB, query
the temporary DB (tmpDB) to verify the stored token equals newToken; include
these assertions before exporter.Stop() so the test fails if the refresh path
did not persist the new JWT.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@internal/exporter/exporter_test.go`:
- Around line 618-659: The test currently only asserts headers inside the
httptest handler which will silently pass if no request is made; modify the test
for the "exports without auth token when endpoints configured" case to record
that the handler was hit (e.g., increment an atomic counter or send a value on a
channel from the httptest handler) and after calling he.exportToHTTP(ctx,
healthData) assert that at least one request was observed; locate the handler
closure created for server := httptest.NewServer(...) and the export call to
healthExporter.exportToHTTP to add the observation mechanism and final
assertion, then run exporter.Stop() as before.
- Around line 563-616: The test currently only checks exportToHTTP returned no
error; update it to also assert the JWT was refreshed: after calling
he.exportToHTTP(ctx, healthData) assert that the active token was replaced with
newToken (e.g. check cfg.AuthToken or he.config.AuthToken) and, if the
implementation persists tokens to the DB, query the temporary DB (tmpDB) to
verify the stored token equals newToken; include these assertions before
exporter.Stop() so the test fails if the refresh path did not persist the new
JWT.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 9037926c-15d8-48af-b0de-d8494706b0ae

📥 Commits

Reviewing files that changed from the base of the PR and between 33f9ba0 and 016cd2f.

📒 Files selected for processing (2)
  • internal/exporter/exporter.go
  • internal/exporter/exporter_test.go
💤 Files with no reviewable changes (1)
  • internal/exporter/exporter.go

@miconeilaws miconeilaws force-pushed the feat/custom-otlp-export branch from 016cd2f to fdad7ae Compare May 25, 2026 10:01

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
internal/exporter/exporter_test.go (1)

568-620: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Use the actual JWT refresh response header key expected by the HTTP writer.

At Line 569, the test sets X-New-JWT-Token, but the writer reads jwt_assertion (see internal/exporter/writer/http.go:273-283). This prevents the refresh path from being exercised as intended and can break the assertions at Lines 615 and 620.

Suggested fix
-		server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
-			w.Header().Set("X-New-JWT-Token", newToken)
+		server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			w.Header().Set("jwt_assertion", newToken)
 			w.WriteHeader(http.StatusOK)
 			_, _ = w.Write([]byte("SUCCESS"))
 		}))
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@internal/exporter/exporter_test.go` around lines 568 - 620, The test fails to
exercise the JWT refresh path because the fake server sets header
"X-New-JWT-Token" while the HTTP writer expects "jwt_assertion"; update the test
server response in exporter_test.go (the handler used by New/WithConfig and
exportToHTTP via healthExporter) to set the header key "jwt_assertion" (or the
shared constant if one exists) with newToken so exportToHTTP and the underlying
HTTP writer will pick up and persist the refreshed token.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@internal/exporter/exporter_test.go`:
- Around line 568-620: The test fails to exercise the JWT refresh path because
the fake server sets header "X-New-JWT-Token" while the HTTP writer expects
"jwt_assertion"; update the test server response in exporter_test.go (the
handler used by New/WithConfig and exportToHTTP via healthExporter) to set the
header key "jwt_assertion" (or the shared constant if one exists) with newToken
so exportToHTTP and the underlying HTTP writer will pick up and persist the
refreshed token.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 300bdae6-9d2b-42ad-816e-88afff46a09c

📥 Commits

Reviewing files that changed from the base of the PR and between 016cd2f and fdad7ae.

📒 Files selected for processing (2)
  • internal/exporter/exporter.go
  • internal/exporter/exporter_test.go
💤 Files with no reviewable changes (1)
  • internal/exporter/exporter.go

@miconeilaws miconeilaws force-pushed the feat/custom-otlp-export branch from fdad7ae to 62856ed Compare May 25, 2026 10:37
@jingxiang-z

Copy link
Copy Markdown
Collaborator

@miconeilaws The change looks good to me. Could you please rebase this branch on the latest main?
Also, could you share your use case? Are you trying to connect the agent to a local OTEL Collector, or to a remote OTLP endpoint that does not require auth?

Comment thread internal/exporter/exporter.go
@miconeilaws miconeilaws force-pushed the feat/custom-otlp-export branch from 62856ed to aa014dc Compare June 2, 2026 10:00
@miconeilaws

miconeilaws commented Jun 2, 2026

Copy link
Copy Markdown
Contributor Author

Thanks guys, rebased and kept the warning log.

I'm trying to forward the telemetry to my existing OTel node-local collector so I can ingest it in my existing telemetry pipeline.

Ideally I would also want a TLS config for auth as thats what I do for other things like scraping prometheus endpoints. I did evaluate the prometheus option here but it doesn't give me the health classifications as those seem to get emitted as logs.

No auth is fine for some prototyping for now.

@miconeilaws

Copy link
Copy Markdown
Contributor Author

@jingxiang-z @rsampaio let me know of any other blockers

@jingxiang-z

Copy link
Copy Markdown
Collaborator

@miconeilaws Thanks. Could you rebase the latest main to include the security fix? Also could you change the PR title to be fix: ...

@miconeilaws miconeilaws changed the title bug: OTLP export silently skips when endpoints are configured but no auth token is present. fix: OTLP export silently skips when endpoints are configured but no auth token is present. Jun 4, 2026
… auth token

Replace the hard AuthToken == "" early return with a warning log. The
HTTP writer already handles empty tokens correctly by omitting the
Authorization header, so non-enrolled users can now push metrics and
logs to any OTLP-compatible endpoint when endpoints are configured.

Signed-off-by: Michael O'Neill <miconeil@amazon.com>
@miconeilaws miconeilaws force-pushed the feat/custom-otlp-export branch from aa014dc to d5fd340 Compare June 4, 2026 21:03
@miconeilaws

Copy link
Copy Markdown
Contributor Author

@jingxiang-z done, thanks a lot!

@jingxiang-z jingxiang-z merged commit c1bab9c into NVIDIA:main Jun 4, 2026
9 checks passed
@jingxiang-z

Copy link
Copy Markdown
Collaborator

@miconeilaws Thanks for your contribution! Approved and merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants