Decision: replace the hand-rolled OFREP poller with a push-based flag channel

[aepfli]

Decision: replace the hand-rolled OFREP poller with a push-based flag channel

Updated after review. Two options added (G, H), the recommendation moved from
D to H, and a factual correction applied to G — see Changelog at the end.

Context

FlagdClient is ~180 lines of custom code that POSTs to flagd's OFREP endpoint
(http://$FLAGD_HOST:8016/ofrep/v1/evaluate/flags/telemetryLevel) every 5s from a
daemon thread, regex-parses "value" out of the JSON, and stores the result in an
AtomicReference<Boolean> that FilteringSpanProcessor reads once per span.

What that costs us today:

• One HTTP request per JVM every 5s, forever, whether or not anything changed.
  At 1000 instrumented JVMs that is 200 req/s against flagd purely as a keepalive.
• Up to 5s of staleness on a flag flip (tunable, but the interval trades directly
  against request volume — the classic polling bind).
• A JSON "parser" that is a regex. It matches the first "value": "..." in the
  body, so it silently depends on flagd's field ordering and only works for string
  flags. Any structural change to the OFREP response, or an error envelope that
  happens to contain a value key, mis-parses rather than fails.
• Split error semantics. A non-200 hard-resets to "no suppression" while a thrown
  exception keeps the last known value — two different policies for what is arguably
  the same condition. There is no notion of provider readiness, staleness, or
  "flag not found" vs "flagd unreachable".
• We carry it. Retry/backoff, TLS, auth, evaluation context, targeting rules, and
  any future OFREP revision are ours to implement and keep current.

The per-span hot path is already cheap (one volatile read + one enum compare) and is
not what this issue is about. This is about the flag-delivery channel.

Constraints

Three constraints shape every option below, and all three are load-bearing:

1. Footprint is a first-class requirement, not a nice-to-have. This ships as an
   agent extension loaded into the extension classloader of every customer
   application. Today the JAR is ~20 KB with zero runtime dependencies. Anything we
   add must be shaded, and a multi-MB gRPC/netty stack inside someone else's JVM is a
   materially different product than what we ship now.
2. HttpURLConnection was a deliberate choice, not laziness — see the class
   javadoc. java.net.http.HttpClient hit classloader/instrumentation conflicts with
   the agent. Any new transport re-opens that question.
3. We will not out-maintain the official client. Reimplementing evaluation,
   targeting, or protocol semantics by hand is off the table. This constrains how
   much we may hand-roll, not whether we may — see H.

Two clarifications that shaped the options

OFREP and in-process are mutually exclusive. OFREP is remote evaluation: send a
context, flagd evaluates, get a value. In-process means receiving the ruleset and
evaluating locally. OFREP does not serve rulesets — that is the sync API
(/v1/flags, port 8017, PR #2037). "OFREP, but in-process" is not a combination that
exists; the options below pick one or the other.

In-process buys this extension almost nothing. We evaluate a single string flag
with a hardcoded empty context, once per flag change. In-process pays off when
evaluation is frequent and context-dependent (per-request targeting, user attributes).
Here it would mean shipping a JsonLogic engine, a JSON-schema validator, and both
jackson and gson in order to compare a string. Noted because "in-process" reads as the
sophisticated choice and, for this specific workload, is not.

Options

A. Keep the custom client, make the polling cheaper

Switch to the OFREP bulk endpoint (POST /ofrep/v1/evaluate/flags) with
If-None-Match, so unchanged polls return a bodyless 304; add jittered intervals
and backoff on failure.

• Pro: no new dependencies, small diff, keeps the current failure model.
• Con: still polling — the request count is unchanged, only the bytes shrink.
• Con: staleness unchanged.
• Con: blocked on verification. The OFREP spec (v0.3.0) defines
  ETag/If-None-Match → 304, but flagd's own OFREP docs do not mention
  implementing it. If flagd ignores If-None-Match, this option buys nothing.
• Con: still our code to maintain.

B. Custom SSE client against OFREP's eventStreams

OFREP v0.3.0 added an eventStreams array to the bulk-evaluation response: the
server advertises an SSE URL, the client connects and receives refetchEvaluation
events (with Last-Event-ID resume and a server-suggested retry), then re-fetches.
This is the "OFREP stream" the original question asked about — it does exist in the
protocol.

• Pro: true push, no dependencies (SSE over HttpURLConnection is a readable loop).
• Con: flagd does not appear to implement it. flagd's OFREP service is documented
  as experimental and lists only the two evaluate endpoints. If flagd ships it later,
  this becomes the cleanest option in the list and supersedes H.
• Con: still our code, now with reconnect/resume state to get right.

C. Official OpenFeature Java SDK + flagd provider, RPC mode

dev.openfeature.contrib.providers:flagd (0.14.0), default resolver, gRPC to flagd on
8013. The provider holds flagd's EventStream open, caches STATIC results in an
LRU, and invalidates on configuration_change. We subscribe to
PROVIDER_CONFIGURATION_CHANGED and re-resolve into an AtomicBoolean.

• Pro: zero polling; sub-second propagation.
• Pro: reconnect, backoff, TLS, auth, caching, readiness/error events all maintained
  upstream. We stop owning the transport.
• Pro: real evaluation semantics (reason, variant, error codes) instead of a regex.
• Con: pulls the gRPC stack into an agent extension — see the dependency note below.
• Con: evaluation still crosses the network on cache miss.

D. Official flagd provider, in-process mode

Same artifact, Config.Resolver.IN_PROCESS, gRPC FlagSyncService/SyncFlags stream on
8015. flagd pushes the whole ruleset; the provider evaluates locally against the
in-memory config. This is what the Python otelfeature-instrument already does — the
README's comparison table lists Java's "Poll (every 5 seconds)" against Python's
"Push (gRPC sync stream)" as the one asymmetry between the two implementations.

• Pro: zero polling and zero per-evaluation network.
• Pro: sub-second propagation; the stream is idle when nothing changes.
• Pro: brings Java to parity with Python; one story to document.
• Pro: survives flagd restarts/outages on the last known ruleset.
• Con: same gRPC dependency as C, for a benefit ("in-process") this workload does not
  actually use — see the clarification above.
• Con: requires flagd's sync service reachable on 8015.

E. In-process resolver in FILE mode / OFO sidecar file-sync

Provider watches a local flag-config file that a flagd sidecar (or the OpenFeature
Operator) keeps in sync.

• Pro: no network from the JVM at all; fastest and most failure-isolated.
• Con: the provider's file mode is itself documented as polling the file every 5
  seconds — it relocates the poll rather than removing it.
• Con: requires a sidecar/operator; not viable for a plain -javaagent on a VM.

F. Hand-rolled Connect-protocol sync client (full ruleset) — superseded by H

Streaming SyncFlags ourselves over Connect/HTTP and evaluating locally.
Reimplements the official sync client wholesale, which constraint 3 forbids.
Kept only to distinguish it from H, which streams the same transport for
notification only and is a much smaller thing to own.

G. Official flagd-http-connector + flagd's new HTTP sync endpoint

flagd PR #2037 adds
--sync-http-port (default 8017) serving GET /v1/flags, the unary HTTP
equivalent of gRPC FetchAllFlags: the flag configuration document, with selector
support and both ETag and Last-Modified. If-None-Match and If-Modified-Since
are honoured, so steady-state polling costs a bodyless 304. On the Java side,
dev.openfeature.contrib.tools:flagd-http-connector is a supported custom Connector
for the provider's in-process resolver, with useHttpCache for exactly this
revalidation.

• Pro: officially maintained on both ends; we own no protocol code.
• Pro: 304 revalidation makes steady-state polling nearly free server-side.
• Con: it does not avoid gRPC. tools/flagd-http-connector compile-depends on
  providers/flagd, which compile-depends on grpc-netty-shaded, grpc-protobuf,
  grpc-stub and protobuf-java. Choosing an HTTP transport does not shed the
  stack — verified against the poms on main.
• Con: it uses java.net.http.HttpClient, the exact class constraint 2 warns about.
• Con: still polling (60s default). It makes polling cheap and upstream's problem;
  it does not remove it.
• Con: gated on #2037 merging and shipping in a flagd release. Open as of writing.

H. OFREP for values + flagd EventStream over plain HTTP for change notification ⭐

Keep OFREP on 8016 exactly as today for the flag value. Replace the 5s timer with a
long-lived connection to flagd.evaluation.v1.Service/EventStream on 8013, and
re-fetch only when flagd says configuration_change.

flagd is built on connect-go, and 8013 serves plain HTTP as well as gRPC — from
flagd's own README:

curl -X POST "http://localhost:8013/flagd.evaluation.v1.Service/ResolveString" \
  -d '{"flagKey":"myStringFlag","context":{}}' -H "Content-Type: application/json"

Connect serves server streams over HTTP/1.1 as length-prefixed frames, which is an
ordinary InputStream read loop on the HttpURLConnection we already use.

• Pro: zero new dependencies. JAR stays ~20 KB. Constraint 1 fully satisfied.
• Pro: no steady-state polling — one idle connection replaces a request every 5s.
• Pro: sub-second propagation.
• Pro: transport unchanged, so constraint 2's classloader/instrumentation wound is not
  re-opened.
• Pro: degrades to exactly today's behaviour — keep a slow (60s+) poll as a reconnect
  safety net, and a stream failure costs latency, not correctness.
• Con: ~100–150 lines of framing and reconnect logic that we own. This is the
  concession against constraint 3, and it is deliberate: scoped to notification
  only, the payload just has to mean "something changed" — the value still comes
  from OFREP, and no evaluation, targeting, or ruleset semantics are reimplemented.
• Con: couples us to a flagd wire detail (the event stream) rather than to OFREP alone.
• Con: blocked on one verification — see open question 1.

Note: flagd-core without the provider

dev.openfeature.contrib.tools:flagd-core:2.0.1 is a genuinely gRPC-free artifact —
the official in-process evaluation engine on its own. It is the honest answer to
"in-process without gRPC", and worth remembering if this extension ever needs real
targeting rules. It still pulls flagd-api, jackson-databind, gson, json-logic-java,
json-schema-validator, commons-lang3, semver4j and commons-codec — around eight jars
requiring shading, to evaluate one string flag. Not proposed now; recorded so the
option is not rediscovered from scratch later.

Recommendation

H, with D as the fallback if the streaming verification fails.

Rationale, in priority order:

1. Footprint is the binding constraint. C, D and G all pull grpc-netty-shaded +
   protobuf into an extension that is 20 KB today and gets loaded into every customer
   JVM. H changes the dependency count by zero.
2. It removes polling rather than optimizing it. A and G leave the request rate
   where it is. H replaces N-JVMs-times-every-5s with one idle connection each.
3. The hand-rolled surface is small and bounded. We keep OFREP — an official,
   versioned protocol — for everything semantic. What we own is a read loop over a
   documented event stream, with a poll as the fallback path. That is a materially
   smaller commitment than the sync client F proposed.
4. In-process is not worth paying for here. One string flag, empty context, read
   once per change. D's headline advantage does not apply to this workload.
5. It fails safe. Stream down → slow poll → today's behaviour. No new failure mode.

Suggested shape (for scoping only, not a design doc):

• Keep FLAGD_HOST; add FLAGD_EVENT_PORT (default 8013) alongside FLAGD_PORT.
• On startup: one OFREP fetch for the initial value, then open the event stream on a
  daemon thread. Both async — extension customize() runs on the critical startup
  path and must not block.
• On configuration_change: one OFREP re-fetch. On stream close: reconnect with
  jittered exponential backoff.
• Retain a slow poll (FLAGD_POLL_INTERVAL_SECONDS, default raised to 60s) purely as
  a safety net, not as the primary channel.
• Replace AtomicReference<Boolean> with a plain volatile boolean — it currently
  boxes on every span read. Free win, independent of this decision.
• Default to no suppression until the first successful fetch, matching today's
  fail-open behaviour.
• Replace the regex with a minimal targeted parse, or accept it as-is given the
  response shape is now the only thing we parse.

Open questions

1. Blocking: does EventStream stream over HTTP/1.1 Connect? Verify with a curl
   against a local flagd (Content-Type: application/connect+json) that frames arrive
   incrementally and do not require HTTP/2. flagd's cheat sheet shows only grpcurl
   for 8013, so this is documented-by-omission. If this fails, H collapses and the
   recommendation reverts to D. Roughly a ten-minute experiment; do it first.
2. Does HttpURLConnection hold a never-ending chunked response cleanly? Read
   timeouts must be disabled or handled on the stream, unlike the current 5s poll.
3. Self-instrumentation. Confirm the agent does not instrument our own long-lived
   connection and generate spans that our own processor then filters. Lower risk than
   under C/D/G since the transport is unchanged, but it is a new long-lived
   connection rather than short polls.
4. Deployment. Does every target environment expose 8013 alongside 8016? If some
   expose only OFREP, the slow-poll fallback is the whole story for those and we
   should say so explicitly in the README.
5. If we fall back to D: shading (io.grpc, com.google.protobuf,
   dev.openfeature → io.otelfeature.shaded.*, which also isolates the
   OpenFeatureAPI singleton from applications that use OpenFeature themselves), JAR
   size, and gRPC/netty self-instrumentation all return as blocking questions.

Not recommended

• A alone. Even if flagd honours If-None-Match, request rate and staleness are
  unchanged.
• F. Superseded by H, which uses the same transport for a fraction of the code.
• G. Does not deliver the footprint saving that motivated it, and is gated on an
  unmerged upstream PR. Revisit if #2037 lands and the gRPC dependency can be
  excluded when only a custom connector is used.
• E. Requires a sidecar and still polls.

Watch list

• B becomes the best option in this list the moment flagd implements OFREP's
  eventStreams. It is standard, push-based, dependency-free, and would let us delete
  H's hand-rolled stream entirely. Worth an upstream issue asking for it.
• #2037 landing changes G's availability but not its dependency problem.

Changelog

• Correction: an earlier reading suggested flagd-http-connector avoided the gRPC
  stack. It does not — it depends on providers/flagd, which depends on
  grpc-netty-shaded, grpc-protobuf, grpc-stub and protobuf-java. G is written
  above with the corrected assessment.
• Options G and H added; recommendation moved from D to H; F marked superseded.

References

• OFREP spec / OpenAPI (bulk endpoint, ETag/304, eventStreams SSE):
  https://openfeature.dev/docs/reference/other-technologies/ofrep/openapi/
• flagd OFREP service (experimental, port 8016):
  https://flagd.dev/reference/flagd-ofrep/
• flagd README — HTTP on port 8013:
  https://github.com/open-feature/flagd/blob/main/README.md
• flagd Java provider (resolvers, ports, caching): https://flagd.dev/providers/java/
• flagd protobuf services (evaluation.v1.Service/EventStream,
  sync.v1.FlagSyncService/SyncFlags): https://buf.build/open-feature/flagd
• flagd PR #2037 — HTTP fetch equivalent of gRPC FetchAllFlags:
  feat: HTTP fetch equivalent of the gRPC FetchAllFlags open-feature/flagd#2037
• flagd-http-connector:
  https://github.com/open-feature/java-sdk-contrib/tree/main/tools/flagd-http-connector
• flagd-core:
  https://github.com/open-feature/java-sdk-contrib/tree/main/tools/flagd-core