Tasks/lpm v2 - #23
Open
mikezupper wants to merge 274 commits into
Open
Conversation
…d LOC teams Three point-in-time outbound records completing the downstream set: - paid-job packet (openai + transcode): each surveyed workaround mapped to what replaces it — mode-mismatch retry loops, per-mode offering duplication, guessInteractionMode, compensating settle(0)/close(0), client body mutation for usage frames. Flags the transcode live path as paid-session work with an open gateway-owned-ingest descriptor question, and is explicit that recipient rotation is NOT fixed here. - paid-session packet (vtuber + daydream): frames vtuber's broker bypass as the design signal it was, maps bearer vaults, reattach tri-states, route rehydration, bespoke usage callbacks, and params.extras smuggling to their replacements, and raises the no-broker dev case as a real requirement we deliberately did not cover. - LOC packet: JOB_MODES/SESSION_OPEN_MODES collapse to two protocol names that stop growing; refill policy and SDK top-up channel come from declared axes; documents the normative balance object; reports two findings in their code (sender=b"" in get_session_debits, the debit_seq naming collision); proposes recipient rotation as the joint design item. Refs beads lnm-n4d.2 lnm-n4d.3 lnm-n4d.4.
…ol+axes The schema change left its own examples invalid. minimal.json now uses protocol + job/session axes across four capabilities (two job shapes, one multipart, one session with rtmp-hls/v1) and validates against schema.json — including fixing a pre-existing flaw where the placeholder signature never matched the schema's hex pattern, so the example is now actually verifiable. Canonical JCS reference regenerated. Changelog gains the breaking 1.0.0 entry. Refs bead lnm-2ji.4.
The manifest schema change left the repo internally inconsistent: the coordinator published manifests the schema rejects, pool-controller rendered broker configs the new broker refuses to load, and the registry daemon's strict decoder would have failed on any v1 manifest. This completes it. capability-broker: /registry/offerings now emits the declared axes (job.transports, or the session object in the manifest's published vocabulary — flat host-config lease/heartbeat mapped to the schema's nested objects, required axes defaulted explicitly via Advertised* helpers). Without this the coordinator had protocol-only offerings and could not build a valid manifest. SessionCap gains the advertised axes it was missing (attachment, metering, refill, tolerance_band_pct, runway_increment_units) with validation, including the schema's cross-axis rule that broker-observed metering requires inband-ws attachment. orch-coordinator: protocol + opaque axes pass-through carried into the signed bytes; publishes spec_version 1.0.0; hard-fails a broker whose offerings lack their axes rather than signing an invalid manifest; diff/roster surfaces renamed (?mode= -> ?protocol=, drift kind mode_changed -> protocol_changed) and axes drift is now detected. pool-controller: renders protocol + job.transports (defaulting to [unary] with a surfaced warning when pool data carries none) and explicitly refuses paid-session offers it cannot render completely rather than emitting a config that would take every pool capability down; a contract test decodes rendered YAML against a transcription of the broker's grammar. service-registry-daemon: axes pass through as raw JSON and are mirrored to gateways, which select routes on session.descriptor_schema; the strict decoder would otherwise have rejected every v1 manifest. secure-orch-console: protocol changes stay critical for the sign cycle and axes changes are now classified critical too — session.attachment alone can move the entire data plane without touching the protocol tag. payment-daemon + infra: operator-facing host-config snippets and the onboarding example emit the new grammar. Refs bead lnm-2ji.4.
Turning the two conformance SKIPs into real scenarios immediately found two defects and one unimplemented spec option. 1. Rebind never re-asserted the payee session. The session record survived a restart but the first debit failed with 'session not found' and the runner retried forever. Recovery now calls the idempotent OpenSession on rebind — and treats AlreadyOpen=false as proof the payment layer lost the session, taking the explicit terminal outcome instead of serving work it cannot bill. This also makes the broker survive a payment daemon that restarted independently, which production does and the old code did not. 2. Winddown precedence was undefined and lease won. A session past both its lease and its heartbeat threshold reported lease_expired, sending operators to funding when the runner was actually dead. heartbeat_lost now takes precedence; the spec states the rule. 3. lease.policy: fixed was declared in the manifest schema and never implemented, so every lease was funding-derived. Implemented in config, validation, the engine, and the advertised axes. Conformance: restart-rebind now asserts what §9.2 actually requires — either branch of rebind-or-terminal, with the forbidden outcomes (second work_id, skipped usage, runner left serving) checked in both; heartbeat-enforcement uses a fixed-lease offering so funding cannot preempt liveness. Both run for real in auto mode. Result: 20 passed, 0 failed, 0 skipped. Refs beads lnm-cch.5 lnm-cch.2.
…nfig Deferred when nothing deployed the session engine; the engine ships now, so the deployment surface has to carry its state. compose gains a named broker-state volume at /var/lib/livepeer and a read-only mount for the sealing key, with the boot expectations documented inline — including the snapshot-ordering rule, since this file holds debit sequence numbers the payment daemon already recorded. The example host-config documents external_base_url and session_store as required once any paid-session capability is served. Refs bead lnm-cch.1.
Their review found one defect and three real gaps; all fixed. DEFECT: runtime-descriptor §2's worked example was tagged sfu-room/v1 but contradicted that schema three ways (max_uses:1 where sfu-room is unbounded, missing the required mint_url, and a private part sfu-room does not define) — an implementer copying it would ship a room that fails on the second participant. The example is now a deliberately fictional example-runtime/v1: a framework page defines the envelope, not any workload's fields, so it can never drift from a real schema again. sfu-room/v1 (1.0.1-draft): - Metering section rewritten. It said participant-minutes are computed from the gateway's mint/TTL records; the offering axis says runner-reported, and mint/TTL over-bills a token minted for someone who never connects or a participant who leaves mid-TTL. The runner is the usage authority; mint records are the dual-meter cross-check, and what the gateway bills its own customers from is the gateway's choice. - room-status added to the grant's operations. status_url was marked gateway-only and named as the verifiability hook, but the only grant operation was participant-token-mint, so no gateway could authenticate the probe. A hook the gateway cannot authenticate against is not a hook. - Work unit stated as an offering property, not a schema property — participant_seconds is as valid as participant_minutes. - Participant-token TTL ceiling made explicit at 300s. paid-session/v1: any accepted event refreshes liveness (a usage tick satisfies heartbeat enforcement — no separate emitter needed), and unknown event-envelope fields are explicitly tolerated so runners can carry their own correlation fields. runtime-descriptor: v1 immutability now states its consequence outright — a runtime host migration is a customer-visible session end. conformance: --job-unit / --session-unit flags so implementations run the suite at their own units instead of patching scenarios. Verified: 20 passed / 0 failed at --session-unit participant_seconds.
Point-by-point record: the descriptor-example defect and all four sfu-room points applied, units made an offering property with the suite verified at participant_seconds, their two observations promoted into the spec, and the conformance-skip note answered — both scenarios now run for real, so they can cite the wire suite rather than our engine tests for B4/B5. The tag request is flagged as the repo owner's call, pending. Refs bead lnm-n4d.1.
No tag is being cut. The reply now points them at tasks/refactor-interaction-modes-and-billing directly and, since their reason for wanting a tag was immutability, tells them to pin the SHA they build against on their side — same property, no dependency on our release cadence. Refs bead lnm-n4d.1.
… backend paths Meeting-team follow-up asked us to make this unambiguous before two implementers assume opposite things. A descriptor's status_url and the broker's configured session.runner.status_path are different endpoints with different callers and different credentials: the broker calls its configured paths with operator credentials, the gateway calls descriptor coordinates with a grant secret. A runner may serve both from one endpoint but nothing requires it, and one whose broker-facing endpoint authenticates only the broker should expose a separate gateway-facing endpoint and publish that. Stated in the framework as the general principle and in sfu-room/v1 where they will look for it. Refs bead lnm-n4d.1.
… ledger The meeting team found that restart-rebind never reached the rebind branch: auto mode's payment layer is the broker's in-process mock, which restarts WITH the broker and loses state, so OpenSession reports not-already-open and every restarted session took the terminal branch. The rebind assertions were written but never ran — and for a meeting product, 'the broker restarted and every live meeting ended' versus 'meetings continued' are entirely different operational profiles that the suite could not tell apart. payment.Mock gains optional persistence (payment_daemon.mock_state_path). This models the real daemon, which persists to BoltDB; an amnesiac mock models a daemon that does not exist. Leaving the path unset keeps the amnesiac behavior, which is now used deliberately. Conformance runs both branches deterministically: restart-rebind against a surviving ledger now REQUIRES rebind (terminal there is a failure), and a new restart-terminal-when-unbillable wipes only the payment state — broker session store intact — to require the fail-closed branch with the forbidden outcomes checked on it too. 21 passed, 0 failed, 0 skipped. Refs bead lnm-khp.7.
Two items. Their restart-rebind finding was correct and is fixed by harness work rather than by downgrading the claim — the rebind branch now executes and the terminal branch is forced deterministically, so citing the wire suite for B5 is finally accurate. Their status_url question is confirmed and now stated in the spec rather than left to be inferred. Also notes that their pinned SHA predates both, and that neither changes a reviewed contract. Refs bead lnm-n4d.1.
The suite was green at 20/20 but did not execute every bullet in the specs' own Conformance sections. Now 32/32, with the remaining edges documented rather than implied. Added (paid-job): in-flight retry refused with job_in_flight (concurrent request against a slow backend route), and severed-stream mid-body settling to a terminal outcome that a retry replays. Added (paid-session): lease-expiry winddown isolated from heartbeat via a fixed one-second lease; bounded-refill proving the will_refuse_next_refill advertisement PRECEDES the refusal; and wire coverage for the optional control-WS binding (push of usage/balance/ ended, acks for gateway frames, uniform-401 on upgrade, HTTP agreeing with the push) which skips cleanly when an implementation advertises no events_ws. Added (runtime-descriptor): malformed-grant fail-closed, grants still absent after a restart, and per-schema public-by-contract fixtures for all four shipped schemas. Previously only sfu-room was exercised, so the framework's promise that a schema change moving a sensitive field into the public part fails conformance rather than review did not actually hold for rtmp-hls, scope-passthrough, or trickle-egress. Broker: refill bounded was declared in the schema, advertised by the registry, and never enforced -- the same class of gap as the fixed lease policy. TopUp now refuses bounded offerings, and balance advertises will_refuse_next_refill from open so the warning always precedes the refusal. README gains a "what this suite does not cover" section: exactly-once debit under injected payment failure and payment-closed on fail-closed opens are not black-box observable and name the implementation-side evidence required; restart branch coverage depends on the deployer's payment layer; payment validity is mocked. Refs beads lnm-khp.1 lnm-khp.2 lnm-khp.3 lnm-khp.4 lnm-khp.5 lnm-khp.6.
They observe correctly that the suite certifies brokers, not runners, and offer to build a runner-conformance mode. Draft reply accepts it upstream and recommends they build the first cut, for a reason specific to this repo: we have no runner implementation, only test doubles, so we could only validate our own fake against a suite designed around it. Our side is the harness seam (Ctx.Runner is a concrete fake today), the agreed shape, and review/merge. Also corrects their sizing — 18 call sites drive the fake runner directly and do not survive substitution; the harness carries, the scenario bodies do not — and names the two runner obligations that are not black-box assertable. Refs bead lnm-9kj.
Architectural gap raised while reviewing the runner-conformance question. host-config.yaml makes operators hand-transcribe facts only the runner knows — descriptor_schema, work_unit.name, the runner's own API paths, transports, metering, capability id, health semantics, model identity — and the broker then cross-checks the duplicates at runtime. The usage_unit_mismatch check exists only because the unit is stated twice; that is the class of drift the meeting team hit. States the split (the runner declares what it IS, the operator declares what it COSTS and where it RUNS; a runner must never set its own price), proposes runner self-description read at startup and reload, and names the safety constraint that shapes it: those facts flow into the cold-key-signed manifest, so the design is read-diff-acknowledge rather than auto-adopt, surfacing as drift the operator must accept. Also records three further gaps it exposes — no version negotiation, no declared session_params shape, readiness approximated by probe recipes — and one live bug: validation requires work_unit.extractor on every capability though the session engine never uses one, so operators invent extractor types that are never called. Refs epic lnm-of1.
…sion §11) Runner obligations were scattered across §7.1/§7.2/§7.3 and the descriptor framework, so a runner author had to reverse-engineer their contract from a protocol written mostly from the broker's point of view — the meeting team assembled the list by hand, which is the evidence this was missing. §11 collects them: session creation, grants, usage events, termination, and the three things a runner never does (talk to the payment layer, set price, treat its claims as the buyer's billing truth). Each row carries the failure signature, which is the new information — a unit mismatch rejecting every event for the session's lifetime is diagnosable in advance or it is an afternoon. The section is deliberately derivative: numbered sections govern on conflict, so the checklist cannot drift into a competing contract. This is what this repo owes runner authors — a stated contract — as distinct from hosting a harness that tests their code. Refs bead lnm-of1.1.
… the contract The earlier draft conceded too much: it committed us to a harness seam and to reviewing a runner-certification suite for components we do not build and cannot run. This repo ships no runner, so such a harness could only validate our own fake against a suite designed around it, and would rot into laundering non-conformance as evidence. The reply now declines hosting with that reasoning, points them at paid-session §11 as the deliverable we actually owe runner authors, keeps the sizing correction (18 fake-runner call sites do not survive substitution) and the two obligations that are not black-box assertable, offers to link their harness so the next author does not repeat it, and flags the runner-self-description design since it addresses the mechanism behind the unit-drift hazard they raised. Refs beads lnm-9kj lnm-of1.1.
Two steps toward host-config carrying only facts the operator owns. Extractors (lnm-of1.6): validation required work_unit.extractor on every capability, but the session engine never runs one — paid-session usage is runner-reported, and grep showed zero references. Operators had to invent a type that is never called; the conformance suite declared seconds-elapsed purely to pass validation. Extractors are now required for paid-job and rejected for paid-session, because silently ignoring config is how config drifts into a lie. Self-description (lnm-of1.2, spec §7.1.1): an optional session.runner.describe_path lets the broker read what the runner says it implements and compare it against the configured tuple. Contradictions in work_unit, descriptor_schema, or capability_id are fatal at startup — that configuration cannot work, and a work_unit mismatch would otherwise reject every usage event for a session's lifetime. Unreachability is only a warning; metering, path, and heartbeat-cadence mismatches warn. Nothing is adopted: published offerings are cold-key signed, so a runner must never be able to change what an orchestrator advertises and sells. Broker green; conformance 32/32. Refs beads lnm-of1.2 lnm-of1.6.
…roker The self-description check failed broker startup on any contradiction, which contradicts the spec it implements: §7.1.1 says a contradiction is fatal to THAT capability. One bad tuple was taking down every other capability an operator serves. A contradicted tuple is now quarantined: withheld from session opens (answering capability_not_served) and filtered out of /registry/offerings, so the network stops routing paid work to something that cannot serve it, while the broker starts and its siblings keep working. The reason is logged and readable per tuple. Test covers the whole shape: broker starts, contradicted tuple quarantined with the field named, healthy sibling untouched and still advertised. Refs bead lnm-of1.3.
Health probes were operator-authored HTTP recipes approximating a fact the runner holds exactly (model loaded, GPU free, queue depth), and the default probe just hits the backend root and calls any 200 healthy. A runner may now declare a readiness endpoint in its self-description; the broker points the probe there when it would otherwise use that default. An operator-written probe always wins. This does not breach the never-adopt rule: readiness is live data that changes how liveness is measured, never what the orchestrator advertises and sells, which is the cold-key-signed part. Implementing it surfaced an ordering bug in the previous commit — the describe pass ran after the health manager was constructed, so a derived probe would have been ignored. Reconciliation is now a config-only pass that runs before the server is built, which also gives reload a single path to take. Refs bead lnm-of1.5.
A capability's parameter requirements were undocumented: a gateway sent params blind and learned what was required from a create-time failure, after payment had already been validated. A runner may now declare the shape in its self-description; the broker relays it verbatim through the advertised offering so gateways can validate before opening. Description, not validator: the broker keeps passing session_params through untouched and never rejects an open on the strength of it. The runner remains the authority on its own inputs, and a broker enforcing a stale copy of that contract would recreate the two-sources-of-truth problem this whole line of work exists to remove. Broker green; conformance 32/32. Refs bead lnm-of1.4.
The original purge scoped the protocol repo, the broker, and design docs; payment-daemon and infra were never in the blast radius, so mode-era references survived there. The dangerous one: payment-daemon's operator runbook §11 documented the session-control-plus-media mode driver, including CLI flags the binary no longer has (--container-runtime, --webrtc-udp-port-min/max, --session-control-max-concurrent-sessions). An operator following it would fail to start the broker. Replaced with a removal note pointing at the paid-session guidance. Its RTMP work-units paragraph described the FFmpeg progress extractor and LiveCounter — machinery deleted with internal/media — and cross-referenced broker runbook sections that no longer exist; rewritten around runner-reported usage. §6.5 now speaks paid-job/paid-session and states the exactly-once invariant that constrains what the daemon sees. Also: payer_daemon.proto's GetSessionDebits comment named two dead modes (hand-synced into the generated bindings — comment only, no wire impact; there is no in-repo buf config for proto-go to regenerate from). payoutsim scenario inputs still used mode tags while payoutsim itself emits protocol tags. The infra onboarding example and README described per-mode offering duplication as a requirement; it is now a pricing choice, since one offering declaring [unary, stream] serves both callers. Repo-wide sweep is clean apart from deliberate historical context: the 'replaces these modes' lines in the v1 specs and the post-mortem in design-docs/interaction-modes.md. Refs bead lnm-2ji.5.
Decision: no unpaid path in paid-session/v1. A mode where sessions run without payment authority is what fail-closed exists to prevent, and anything that exists for dev gets misconfigured into production eventually; it would also be a second code path through the engine, which is the thing the redesign removes. The packet no longer invites a debate. It states the decision, and replaces synthesizeStaticRoute with the concrete alternative: the broker runs standalone with payment_daemon.mock (optionally persistent), no registry, no wallet, no chain — their dev stack goes from two processes to three. That is evidenced rather than asserted: the conformance suite runs exactly that configuration and executes 32 scenarios against it, including broker restart with session rebind. If it turns out to be a real problem rather than an inconvenience, that is a conversation — the door is open, the default is no.
…ar removal Version 1.4.1 -> 2.0.0 (major: the manifest lost interaction_mode, and PROCESS.md makes manifest changes a spec-wide bump). Applied to VERSION, image-versions.env, build-images.sh, the manifest example and changelog, orch-coordinator's SpecVersion, and every stale compose/.env/Makefile pin. Fixes a bug in the process: spec_version was being emitted as "1.0.0" as though the manifest schema had its own counter, but the schema says that field carries the spec-wide version. Doc-drift sweep across the whole repo. The dangerous class was documentation describing things that do not exist: - capability-broker/scripts/smoke.sh was 100% dead — POSTed to /v1/cap with Livepeer-Mode headers, and its probe URLs pointed at unreachable hosts so every request would have 503'd anyway. Rewritten against POST /v1/job; verified 19/19 against a live v2.0.0 container. - service-registry-daemon docs told operators to reload the static overlay with SIGHUP; the daemon traps only SIGINT/SIGTERM, so following the doc kills the process. - payment-daemon's runbook documented seven flags/metrics that do not exist (--receiver-ev, MaxEV, --store-path, --log-level, pending_redemptions_total, ...), and a commit had silently swallowed its "Common failure modes" heading while two sections still pointed at it. - pool-payout-executor documented a mark-failed-fatal subcommand and pool-controller a livepeer-broker-apply binary; neither exists. - protocol README claimed the conformance suite "is being rebuilt" when it had shipped; headers README listed two deleted headers as required and Request-Id as optional when it is now the idempotency key. - extractors/openai-usage.md told brokers they MAY rewrite the request body, contradicting paid-job §3.1's MUST NOT. Code fixes the sweep surfaced: - backend.transport ffmpeg-subprocess and session-runner still passed validation with nothing implementing them, so a config would load, validate, get advertised, then fail at request time. Removed, along with the encoder-profile and session-runner grammar they required. - middleware.Metrics was defined but wired into neither paid chain, so three Grafana panels were permanently empty. Wired into both, and the metrics renamed off the dead taxonomy (livepeer_mode_* -> livepeer_paid_*) with the dashboard updated to match. - manifest schema was missing session_params_schema, so a manifest carrying the axis this branch added would fail validation. - R8 and two core beliefs still described the seven-mode typology. The requirement is unchanged and better served, so they were restated rather than rewritten. Build path repaired: build-images.sh referenced two Dockerfiles deleted in the purge. The session-runner-stub entry is gone; the conformance suite has a real Dockerfile again (URL mode, with the loopback caveat documented). All 11 images build at v2.0.0. Verified: six Go modules build/vet/test clean, manifest examples validate, conformance 32/32, broker smoke 19/19.
The runbook told operators to alert on heartbeat_lost and recovery_failed, but alerts.yaml shipped no such rule — advice with nothing behind it. Six rules added across the paid-session and paid-job surfaces. The one worth understanding is BrokerSessionsDyingByHeartbeat, which pages on heartbeat_lost *dominating* winddowns rather than merely occurring. Isolated losses mean a runner crashed; losses dominating winddowns are the signature of a wrong external_base_url, where runners cannot reach the callback URL so every session opens, reports nothing, and dies on schedule. The annotation says so and lists what to check. The rest: elevated heartbeat loss as fleet health, any recovery_failed (rare by design — a burst after restart means the payment layer did not survive), opens failing above 10%, unauthorized runner events, and backend errors on job exchanges. Thresholds are ratios with volume floors rather than bare counts, so they behave at any traffic level, and the file header states plainly that the numbers are starting points to tune — a rule that cries wolf gets muted, and a muted rule is worse than no rule. Also fixes the file header's pointer to a runbook section that no longer exists (§2.8 -> §5) and documents the livepeer_paid_* metrics in the runbook, which carry the capability/offering labels livepeer_protocol_* deliberately does not.
The suite's fake job backend and fake session runner used
httptest.NewServer, which always binds 127.0.0.1. That was invisible
while the fakes only ever ran next to a broker the suite started
itself, but it made the published image useless without --network
host: on an ordinary docker network the broker could not reach the
fakes and every session scenario failed at create.
Replace httptest with a listener the operator addresses:
--fakes-listen interface to bind (default 127.0.0.1)
--fakes-advertise host the broker reaches them on (default: the
bind host, or this machine's name when binding
0.0.0.0 — the name docker DNS resolves)
--fakes-backend-port pin the ports, because the broker's offerings
--fakes-runner-port name the fakes before the suite starts
Validated end to end on a shared docker network, no host networking:
29 passed, 0 failed, 3 skipped (the three URL-mode restart skips).
Auto mode is unchanged and still 32/32.
Two health-probe traps surfaced once a broker ran long enough to probe,
both now documented and covered by the example:
- a broker that starts before the suite has been probing dead
addresses, so its backends are unselectable and the first scenarios
get 503; --warmup waits for probes to recover.
- the always-error/slow/longstream offerings point at routes that 500
or answer slowly on purpose, so their health probe must aim at the
backend base URL or the offering is marked unreachable.
examples/docker-network/ carries the compose file and reference-broker
config the run above used.
Refs bead lnm-dwa
LOC came back with seven contract questions. Checking each against the code turned six of them into defects on our side, so the reply concedes rather than explains, and every claim in it is anchored to file:line. The worst one is not what they described. Jobs derive the payee work_id from the payment (hex recipient_rand_hash); sessions mint a UUID. The payee daemon mints a fresh recipient rand for an unseen work_id, so a session-open payment's tickets are validated against a rand the sender never saw — and the open path drops TicketsRejected, so the session opens unfunded instead of failing closed. Conformance stayed green because the mock credits without validating. In chain mode no paid session can validate a real ticket. The rest, all confirmed: per_units is parsed by the coordinator and then dropped (no manifest field, resolver pins units_per_price to 1) while broker settlement divides by the payment's denominator; operator `extra` keys shadow the signed protocol/axes declaration in the resolver projection; job fingerprints hash content length, not content; session opens replay on the request id with no fingerprint; top-ups ignore the request id entirely; and the payer daemon does not implement GetSessionDebits at all, so LOC's empty sender was never the binding constraint. Filed as epic lnm-sqe with 12 children (one P0). The reply also asks four questions back — per_units carry-vs-pin, whether Livepeer-Settlement needs signing, what an open replay returns after a lost response, and rotation working-session scheduling — and corrects the packet's claim that work_id was untouched, which held for jobs only. Not sent: awaiting review. Refs bead lnm-n4d.4, lnm-sqe
LOC answered all three: carry per_units, sign Livepeer-Settlement, re-deliver credentials on an identical replay. All accepted. This reply writes each one down at the level they asked for. Specifying the denominator turned up a bigger problem than "dropped in transit". The payee ledger has no denominator at all — store.go debits price x work_units — and the broker never passes per_units to OpenSession, nor uses it in the job total or the runway estimate. Only the settlement record honours it. An offering at per_units: N debits N times the intended rate while advertising a flattened quote and attesting a third number. Latent only because every live offering runs per_units: 1; LOC's migration is what makes it real. lnm-sqe.3 raised to P0. Adopted from LOC, normatively: bill(U) = ceil(U x amount_wei / per_units) over cumulative units, refills funding the difference of cumulative targets so no sequence of refills accumulates rounding error. amount_wei and per_units pin at session open, like face value. Settlement signing (lnm-sqe.13): a broker hot key delegated by the orch cold key through the manifest, so discovery and rotation ride the registry trust path LOC already verifies — no second PKI. JCS/secp256k1, payload binds identity, quote, cumulative units and value, with settlement_seq monotonic per work_id as the replay binding. Plus a server-to-server read so financially material data need not cross the customer's SDK at all. Replay re-delivery (lnm-sqe.12) requires all four of: same request id, identical fingerprint, re-proved payer identity, exact recorded outcome. This reverses paid-session/v1's exactly-once secret delivery — a funded undrivable session is the worse failure. Not sent: awaiting review, and the rotation participant is still ours to name. Refs bead lnm-n4d.4, lnm-sqe
LOC's last three details, all accepted, two of which improve what we sent. Our key-discovery answer was written for the wrong consumer. We said delegation discovery rides the manifest LOC already verifies; LOC does not verify manifests, it consumes the registry daemon's verified projection. So the delegations project onto SelectedRoute — all currently-valid keys with their windows, newest first — and LOC persists them with its immutable route snapshot. The daemon stays the verifier and LOC does not grow a manifest verifier for one field. Their rotation point closes a real hole. We proposed settlement_seq monotonic per work_id; rotation mints a new work_id, so that counter would have restarted at each rotation, and the cumulative billing curve would have restarted with it — under-billing through ceil() on a reset total. The payload now binds session_id, work_id, rotation_generation and predecessor_work_id, settlement_seq is monotonic per session_id, and the record carries both cumulative (logical session) and per-generation subtotals. Cumulative billed value is one ceil over cumulative units, never a sum of per-generation ceilings. Authoritative billing quantity is cumulative debited_units. In the engine claimed and debited cannot diverge — the debit precedes the commit and both totals move in one atomic update — so the record carries both and a mismatch is a defect to assert on. Direct query specified: same signed envelope byte for byte so LOC has one verifier, plus state and a fresh issued_at, and no shared secret — TLS plus work_id possession, because integrity comes from the signature, not the channel, and a bearer would buy an N x M secret distribution problem for nothing. Not sent: awaiting review, participant still to name. Refs bead lnm-n4d.4, lnm-sqe
…r own paid-session/v1 minted its own work_id (a UUID) and handed it to the payee daemon. The daemon binds a session — and the recipient rand every ticket was minted against — to the payment's recipient_rand_hash, so an invented id bound the session to a rand the sender never saw and no ticket in the payment could validate against it. paid-job/v1 already derived the id from the payment; the two protocols disagreed about identity and only one of them was right. It stayed invisible because the open path then dropped the daemon's verdict: ProcessPayment reports an all-rejected batch in its result, not as an error, and the engine read only the sender off it. So a session whose every ticket was rejected opened with no funded runway and died at its first lease check, presenting a payment fault as a broker fault. The mock credits without validating, so conformance stayed green over it. Both halves fixed together — the identity fix is unobservable without the verdict: - DerivePayeeWorkID moves to internal/payment, so the derivation has one home instead of one per protocol. That duplication was the bug. - Open derives work_id from the payment, falling back to the request id for stub bytes exactly as the job path does. - An all-rejected batch fails closed with payment_invalid on open (payee session closed, capacity released, runner never bound) and refuses a top-up rather than extending a lease nobody funded. Partial rejections still credit; the balance they produce is honest and §5 handles it. - INVALID_RECIPIENT_RAND is named in the error: it is the payee rotating its rand under a live payer, which is the rotation design's problem, and saying so beats a generic rejection. paid-session.md 1.0.3-draft states the two-identifier rule normatively — work_id is the payee-issued hash, session_id is an opaque broker handle and never a payment key — plus the fail-closed rule for a rejected batch. The spec's silence on both is what let two implementations diverge. No conformance scenario: the suite treats the payment envelope as opaque base64 by design and has no wire verb to make a payee reject tickets. Covered by sessionengine unit tests instead. 32/32 conformance still green; broker suite green. Refs bead lnm-sqe.1, lnm-sqe.2
Digests read back from the registry after the push; the console pulled by digest reports v2.0.0-f9b68fbc5456. The tag is split for now: broker, payment-daemon, coordinator and conformance remain the 1241d76 build.
Two names for the same thing on the secure-orch host: protocol-daemon took --eth-urls (a failover list) while service-registry-daemon took --chain-rpc (one URL, dialed twice: a raw ethclient for the ServiceRegistry reads and a one-element chain-commons multi-RPC for everything else). The broker host's payment-daemon and the pool payout executor each had their own single-URL spelling on top. Every daemon now takes the same comma-separated list, primary first, under --chain-rpc-urls (executor.rpc_urls in the payout executor's YAML). Hard cut; no aliases. - service-registry-daemon: one multi-RPC client opened from the full list, shared by the Controller refresher, round poller, pool discovery and the ServiceRegistry reads, so getServiceURI fails over like everything else. chain.EthConfig takes a ContractCaller interface instead of a concrete ethclient. The built-in arb1.arbitrum.io default is gone: resolver mode outside --dev refuses to start without the list. The operator names the endpoints, never the binary. - protocol-daemon: flag rename only; it already consumed a list. - payment-daemon, pool-payout-executor: both hold a concrete *ethclient.Client in every chain-backed provider, so they select at startup — dial in order, keep the first that connects and reports the expected chain id, warn per rejected candidate with the host only (API keys live in the path; the dial error is redacted too). Runtime failover for these two is tracked separately. Empty still means dev mode in payment-daemon; the integration-stack harness relies on it. Also removes four helpers in the registry daemon's publisher package that commit 2b0ed89 orphaned; they were the only lint findings. Refs bead lnm-eoq
…t is required Three operator-facing cleanups across every scenario and component compose stack. CHAIN_RPC_URLS replaces ETH_URLS and CHAIN_RPC. One variable per host; every service that reads the chain gets the same comma-separated list (--chain-rpc-urls, from commit eda0a56). It is required: compose fails with a message when it is missing, and no default exists anywhere but the example value in .env.example. The payout executor still reads its list from the mounted YAML (executor.rpc_urls), so on the pool stack it is one list per host for payment-daemon only; noted in that .env. KEYSTORE_FILE / KEYSTORE_PASSWORD_FILE replace the two pairs that both named the cold key on the secure-orch host (PROTOCOL_KEYSTORE* and SECURE_ORCH_KEYSTORE / SECURE_ORCH_PASSWORD_FILE); protocol-daemon and the console mount from the one pair. Hot wallets keep their own names with the same suffix: PAYMENT_KEYSTORE_FILE and POOL_PAYOUT_EXECUTOR_KEYSTORE_FILE. Every .env.example now has two sections: Required, with no defaults written in, and Overrides, every line commented out with its default and what it controls. For that to work the compose files carry the defaults themselves: every ${VAR} is now ${VAR:-default} or ${VAR:?message}. All 30 stack/overlay combinations render from a .env holding only the Required section. One behaviour change: the component-local protocol-daemon and registry-daemon composes used an empty METRICS_PORT to mean listener off; a commented override needs a default, so the listener is on unless the flag is deleted. Hard cut on all names; docs/references and completed plans untouched. Refs beads lnm-eoq, lnm-hah, lnm-c4g
…ecutor Two of the five chain-talking modules carry their own copies of the glue chain-commons exists to share: RPC dial, Controller resolve, gas, send-then-confirm, round polling. Plan 0016 §11.Q1 chose that when chain-commons was the prior suite's deprecated module and said to revisit if a second chain-talking component appeared. Three have. Five stages, one bead each under epic lnm-tou, plus a stage 0 for the simulated-chain test layer and a 75% per-package coverage floor. Metric names stay; startup reconciliation rules for in-flight redemptions and payouts are decided here.
config.ParseRPCURLs trims entries and rejects a blank one (a stray comma is almost always a typo in an operator's env file, and dropping it silently would change which endpoint is primary). protocol-daemon and service-registry-daemon use it in place of their own splitCSV and csvList; payment-daemon and the payout executor follow in plan 0048 stage 1. Refs bead lnm-tou.3
…d is not a fault Three pieces plan 0048 needs before payment-daemon and the payout executor move their transaction lifecycle onto chain-commons. testing/simchain exposes go-ethereum's in-process simulated backend as providers/rpc.RPC, with account seeding, deploy, mine, and a Wrap that fails the first N calls, so real signed transactions, nonces, receipts, replacement and restart-resume can be exercised without a network. The txintent processor now has four tests on it: submit lands with exactly one send, adopt confirms with zero sends, submit-after-adopt spends once, a refused first send is retried. txintent.Manager.Adopt takes over a transaction sent by a previous implementation: same idempotency key as Submit, one recorded attempt, status submitted, picked up by Resume. It is how the executor's in-flight payouts survive the upgrade in stage 4. Replacement now bumps from the gas oracle when an adopted attempt's caps are unknown instead of dereferencing nil. errors.Classify gained ClassNotFound for ethereum.NotFound. A receipt poll on a not-yet-mined transaction was classified transient, so the multi-RPC client retried it six times with backoff, about 31 s per poll, and struck the endpoint each time. Not mined yet is a state, not a fault: one attempt, no failover, no strike. The reorg receipts provider already keys on the sentinel and keeps polling. Refs bead lnm-tou.1
…it from the environment internal/ethclient holds providers/rpc.RPC instead of a concrete go-ethereum client, opened once by rpc/multi from executor.rpc_urls, so every chain call fails over across the list with the circuit breaker instead of pinning whichever endpoint answered first at startup. dialFirstHealthy is gone. A slog adapter and a lazily registering Prometheus recorder put chain-commons's livepeer_chain_* series on the existing /metrics listener. Send and confirm logic is unchanged; stage 4 replaces it. CHAIN_RPC_URLS in the environment, when set, overrides executor.rpc_urls (parsed by the shared chain-commons parser), and the pool-node, pool-orchestrator and component compose stacks pass it through, so a host has one RPC list. The Dockerfile copies chain-commons into the build context for the relative replace. Coverage gate installed at 75% per package (plan 0048 §2.6); the one pre-existing package below it is exempted against bead lnm-tou.7. Refs beads lnm-tou.2, lnm-tou.3
…oracle Every chain-backed provider (chain-id check, round clock, gas price, ticket broker) takes providers/rpc.RPC, opened once by rpc/multi from --chain-rpc-urls with the shared parser, so redemption and round reads fail over across the list with the circuit breaker. DialFirstHealthy and the local CSV parser are gone. Empty list still means dev mode. The in-tree Controller resolver is replaced by chain-commons controller/eth with the same per-contract override flags. The gas price provider is rebuilt on chain-commons's gas oracle behind the existing providers.GasPrice interface: one eth_gasPrice per refresh, multiplier applied, last good value served from memory, exactly what the runbook documents. Flag surface is unchanged. Metric names are unchanged: every livepeer_payment_* series the alert rules and the Grafana board reference is still registered and incremented from the same place. chain-commons's own livepeer_chain_* series land on the same /metrics through a name-addressed recorder adapter that registers lazily. Coverage gate installed at 75% per package (plan 0048 §2.6). Touched packages: chain 100, chaincommons 97, gasprice/oracle 94, ticketbroker 88, clock/onchain 84. Packages the plan does not touch and that were already below the floor are exempted against bead lnm-tou.7; stage 4 lifts service/settlement. Refs beads lnm-tou.2, lnm-tou.4
SendNativeTransfer and ConfirmTransaction, the executor's hand-rolled nonce/gas/receipt loop, are replaced by chain-commons services/txintent behind a small engine (internal/payouts): one manager and processor per process over their own BoltDB file, Dispatch submits an intent keyed by the controller intent id so a repeat of the same payout produces one chain transaction across restarts, Outcome waits with a bound, Track adopts a transaction the previous implementation already broadcast. Reconciliation on every confirm pass: a controller intent in submitted is adopted from the nonce the executor has always written into external_ref, or from the transaction itself by hash, and driven to confirmation with zero re-sends; one no endpoint knows is reported pending with the reason and left to the existing stale handling. Resume runs on open, so a process restart re-drives anything non-terminal. Upgrade is stop, upgrade, start. Failure reasons are now chain-commons codes; the auto-requeue matcher learns which of them clear on their own (a stalled fee market after the replacement budget, a throttled or lagging endpoint) and which need an operator (signing failure, a nonce already consumed). Behaviour changes: confirmation_blocks maps to the processor's depth as max(1, N-1); a receipt not found yet is reported pending instead of being marked failed and relying on requeue. Tests run on the simulated chain: one send per payout with the balance delta asserted, idempotency within a run and across a restart over the same store, revert to failed, stall to a same-nonce replacement, restart-resume over a real BoltDB file, adopt from external_ref, adopt by reading the chain, unknown transaction. payouts 82%, ethclient 96%, config 82%; coverage gate green. Refs bead lnm-tou.5
RedeemWinningTicket no longer runs its own nonce counter, send and receipt poll. It reads usedTickets first (a ticket the pre-upgrade loop already landed drains without a transaction), then submits a chain-commons txintent keyed by the ticket hash and waits. The key is the idempotency guarantee: a redemption in flight across a restart, or confirmed during downtime, is returned rather than re-sent; a failed intent is re-driven at a fresh nonce unless it reverted. Resume runs at boot. Upgrade is stop, upgrade, start. Wiring: --txintent-db (default txintents.db beside --db) on chain-commons store/bolt, so internal/store keeps sole ownership of the sessions file; the daemon's keystore behind a chain-commons keystore adapter; the stage-3 gas oracle shared with the processor so the fee cap is exactly eth_gasPrice x multiplier; reorg receipts at --redemption-confirmations, where 0 now means the processor default of four rather than first receipt. Settlement classifies from chain-commons errors as well as its own sentinels. One behaviour change: a reverted redemption drains as reason=reverted instead of staying queued. Before, the ticket was re-sent every tick; under the idempotency key it would have sat in the queue forever, so draining is the consistent outcome. Tests run a real broker and processor on the simulated chain against a hand-assembled stub TicketBroker (no solc): one send per redemption with usedTickets flipped, already-used pre-check with zero sends, revert maps to ErrTxFailed and is not re-sent, three concurrent redeems of one ticket produce one transaction, a refused first send is retried, restart-resume over a real BoltDB file confirms with zero extra sends. Rotation and go-livepeer wire-compat suites unchanged. settlement 89% (was 63%, exemption removed), ticketbroker 89%; coverage gate green. Refs bead lnm-tou.5
Submit and Adopt did read-then-write with no lock between them, so two goroutines submitting the same key at once could both miss the existence check, both persist, and both dispatch a processor: the key spent twice, at two nonces, the second reverting. The payment-daemon's three-concurrent-redeems test found it. A manager-wide mutex now covers the check and the write in both paths; dispatch happens after release. Tests submit the same key from 32 goroutines and race Adopt against Submit; both fail without the lock. Refs bead lnm-tou.5
The relative replace added in 334522c needs the sibling module visible to the builder; the executor, protocol and registry Dockerfiles already do this.
The in-tree clock/onchain poller, with its manual ABI calls against RoundsManager and BondingManager, is replaced by clock/chain: a chain-commons timesource poller at --clock-refresh-interval sharing the daemon's Controller, subscribed through roundclock. A round or last-initialized transition sets the round and reads blockHashForRound once per round; every tick records the L1 head and re-reads the transcoder pool size. Flag surface unchanged; every livepeer_payment_* series the alerts and dashboard name is still emitted. chain-commons gains the three reads payment-daemon needed on its bindings — LastInitializedRound, BlockHashForRound, GetTranscoderPoolSize — so no manual ABI call remains in the daemon. Behaviour differences: a failing pool read no longer blocks the head update in the same tick; Start reports a timesource open failure instead of being fire-and-forget. The ticket broker's submit mutex is gone now that txintent serialises first submits itself (494533f); the three-concurrent-redeems test still produces one transaction. A Resubmit refused because a concurrent caller already re-drove the intent now waits on that re-drive. Tests: fakes for every clock path, and a simulated-chain run against hand-assembled RoundsManager and BondingManager stubs covering init, initializeRound landing mid-round, head advance, pool change, and a new round not yet initialized. clock/chain 93%; coverage gate green. Refs bead lnm-tou.6
The Dockerfile copied only cmd/ into the builder, so the image has not compiled since the agent grew internal/attach and internal/desiredstate (949d669). Copy internal/ too, download modules before the source copy so the layer caches, build static and trimmed like the other images, and stamp main.version from the VERSION build arg the build script already passes.
…er set lnm-9x7 feedback from the openai-runners team, and my packet's error: it said the agent accepted an array of contract entries and it did not — Fetch unmarshalled one object. It now accepts one entry or an array, one per capability a container serves (the audio runner's transcriptions and translations; a vendor-backed runner with several models), validating each, refusing a repeated capability id, and attaching each under a derived local id (<id>.<n>) that routeFor maps back to the container. runner-contract.md 1.1.0 says so, and says to prefer CAPABILITY_NAME-style selection on a pool host, where the pool places one template per service. Found on the way: the tunnel built its route table once per session from the configured runners, while on a pool-managed host the set comes from desired state and changes while the tunnel is up — a service placed after connect was attached but unroutable. Routes are now built from the live set per request. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDAcRvgbjYnRHJgacvKgyX
Answer to the openai-runners team (lnm-9x7): their CUDA images install cu128 PyTorch wheels whose kernels start at sm_75, so a GTX 1080 fails at the first op; they will ship cu126 Pascal variants under a tag suffix and asked how a template selects one. A key of the form <vendor>/<class> in runner_compose.image is a build for that one class and wins over the vendor default. It is the template author's knowledge of the image, so it lives in the image map: a second template per class would duplicate the price, the certification and the ladder for one tag, and could not share the offering id anyway. Validated at load (a known class of that vendor; every admitted class covered by its key or the default), gated at placement, picked at render. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDAcRvgbjYnRHJgacvKgyX
…mage keys Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDAcRvgbjYnRHJgacvKgyX
…tries; packet corrections The array-form dedupe keyed on capability_id, which refused the openai-runners chat proxy's one-entry-per-served-model array. The broker's rule (runner-attach §4.1) is (capability_id, identity); the agent now applies the same. A dated corrections note for all six packets records two errors the team found — response-header takes 'header' not 'name', and --serve-runner takes no image, so real verification is a real attach that this side runs — and one omission, request-formula counting a string field's code points, now in its doc. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDAcRvgbjYnRHJgacvKgyX
…TMPS edge, any-vendor image The transcode team's review of the 2026-09-02 packet, validated against their live repository. Three of its corrections were mine: ABR is already a synchronous SSE stream (only VOD still returns 202), its request carries typed per-artifact destinations rather than one output_url, and the live runner is one vendor-less image. One was a wrong assumption: the HTTP edge cannot carry RTMP. One was the operator's to decide, and the operator chose it: work is metered as aggregate frame-megapixels — this repository's own plan 0011 said so before the rig's flat per-job rate displaced it — claimed by the runner in the stream's Livepeer-Work-Units trailer and read by the existing response-trailer extractor, not by a broker extractor bound to one runner's event schema. In this repository: the certification sink accepts any path under a run's scope, so a ladder's manifest, playlists and media land; the image map gains an 'any' key for a build that runs on every card; runner_compose.rtmp_port declares an ingest, the member bundle publishes 1936, the agent terminates RTMPS on it with the HTTP edge's certificate and pipes to the runner, and the pool sets LIVEPEER_PUBLIC_RTMP_URL on that service (plan 0046 §2.7). The four transcode templates are repriced per 1000 frame-megapixels against a dated reference, the ABR probe carries the runner's v2 body, and the VOD probes carry a proposed v2 body the team confirms. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDAcRvgbjYnRHJgacvKgyX
…gations are met; no total timeout on streams Plan 0048, from the transcode team's three 'earlier blockers', each verified: (1) the URL-backend client's five-minute http.Client.Timeout counted the whole body and cut any encode longer than that before its trailer — replaced by connect, response-header and per-read idle bounds, so a stream that keeps producing is never cut and one that goes silent is; (2) Open looked the request id up and then funded, paid and created a runner session before persisting it, so two opens with one id could both do all three and a crash between payment and persist left a funded payee session with no record — the id is now reserved in a bolt bucket before any side effect, each stage records what it opened, the second open gets open_in_flight, the reservation dies in the transaction that persists the session, and Recover undoes what a crash abandoned; (3) winddown marked a session terminal whether or not terminate and payment close succeeded while Sweep skipped terminal records, so 'will retry on sweep' was false — a winddown with an obligation outstanding now stays winding_down, keeps its capacity, refuses events, fires no outcome, and is retried by Sweep and Recover until both are met, then reports once. Also from that review: certification's reach completes the RTMP handshake for rtmp(s) fields and the live template dials rtmp_url; placement allows one ingest template per host (ingest_port_taken); the pricing reference's tier label is corrected in a superseding note. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDAcRvgbjYnRHJgacvKgyX
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDAcRvgbjYnRHJgacvKgyX
The openai-runners team caught it against the code: fields is numeric only and a string under it is missing; code-point counting is the separate text_fields map, which their TTS contract already declares. The corrections note's §3 said otherwise and is superseded in place by a later-the-same-day paragraph; the extractor is unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDAcRvgbjYnRHJgacvKgyX
…laim, ingest direction) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDAcRvgbjYnRHJgacvKgyX
Their review of 83802ba, all four correct: (1) an open under an in-flight id with different content returned open_in_flight instead of request_id_reuse — the reservation's fingerprint is now compared; (2) the stage writes were unchecked, so a failed write could leave a reservation Recover could not undo — each is checked and fails the open closed; (3) the secrets were minted after the reservation, so a generation error stranded the id — they are minted before it; (4) winddown released capacity and reported the outcome before the terminal write was confirmed — the write comes first and is checked, and a failed write leaves the record pending with its obligations recorded as met for the next sweep. Plus the focused multi-card ingest_port_taken placement test. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDAcRvgbjYnRHJgacvKgyX
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDAcRvgbjYnRHJgacvKgyX
… and a stale mock broker Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDAcRvgbjYnRHJgacvKgyX
…cks the LOC sidecars Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDAcRvgbjYnRHJgacvKgyX
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.