[Nexthop][fboss2-dev] add fboss2-dev config/delete qos default-queue-config commands - #1408
Open
vybhav-nexthop wants to merge 3 commits into
Open
[Nexthop][fboss2-dev] add fboss2-dev config/delete qos default-queue-config commands#1408vybhav-nexthop wants to merge 3 commits into
vybhav-nexthop wants to merge 3 commits into
Conversation
vybhav-nexthop
force-pushed
the
nos-7161-qos-default-queue-config
branch
from
July 27, 2026 05:28
d589036 to
03826bd
Compare
This PR adds two commands for the switch-wide default port queue
settings:
```
fboss2-dev config qos default-queue-config <queue-id> <attr> <value> [<attr> <value> ...]
fboss2-dev delete qos default-queue-config <queue-id>
```
**What is defaultPortQueues?** Each front-panel port has a set of
hardware egress queues, and each queue carries settings for scheduling
(strict-priority vs. weighted round robin), buffer allocation (reserved
and shared bytes, MMU scaling factor) and congestion management (ECN
marking / early drop thresholds). `SwitchConfig.defaultPortQueues` (a
`list<PortQueue>` in `switch_config.thrift`) is the fallback queue
config the agent applies to every port that has no explicit
`portQueueConfigName` override — one place to define the queue behavior
for the whole switch. The config is sparse: only the queues you define
are changed; queues absent from the list are reset to the ASIC defaults
(WRR, weight 1).
**How the commands behave:**
- **Set/merge semantics (config)**: if the queue-id already exists in
`defaultPortQueues` its entry is updated in place; otherwise a new entry
is created (with `scheduling = WEIGHTED_ROUND_ROBIN`, since the thrift
field is required). Multiple `<attr> <value>` pairs can be set in one
invocation.
- **Supported attributes**: `scheduling` (full enum names or short `WRR`
/ `SP` / `DRR`), `weight`, `reserved-bytes`, `shared-bytes`,
`scaling-factor`, `stream-type`, `buffer-pool-name`, and
`active-queue-management` (see below).
- **Delete removes the whole entry**: `delete qos default-queue-config
<queue-id>` erases that queue's entry from `defaultPortQueues`,
reverting the queue to ASIC defaults at the next coldboot. Per-attribute
deletion is intentionally not supported — re-run the config command to
change individual attributes. Deleting a queue-id that has no entry is
an error (fails loudly on typos instead of staging a no-op coldboot
commit), consistent with the copp delete commands.
- **Validation**: enum attributes are checked against the thrift enums
(`QueueScheduling`, `MMUScalingFactor`, `StreamType`,
`QueueCongestionBehavior`) and the error message lists every valid
value; byte/weight/threshold values must be non-negative; unknown
attributes, missing values and malformed queue-ids are rejected before
anything is staged. Queue-ids are bounded to `0..128` (shared with the
queuing-policy parser).
- **Commit level: AGENT_COLDBOOT** for both commands — queue buffer
carving cannot be applied hitlessly, so committing restarts the agents.
A queue can act on packets *before* it fills, to signal congestion early
instead of tail-dropping when full. Each AQM policy pairs a
**detection** ramp (when to act) with a **congestion-behavior** (what to
do):
- **detection** — `detection linear minimum-length <bytes>
maximum-length <bytes> probability <pct>`: a WRED ramp over queue depth.
Below `minimum-length` do nothing; between min and max act with a
probability rising to `probability%`; above `maximum-length` act always.
- **congestion-behavior** — `congestion-behavior <ECN|EARLY_DROP>`:
- `EARLY_DROP` — drop packets early (WRED). Works for any traffic.
- `ECN` — mark the Congestion-Experienced bit instead of dropping; only
for ECN-capable flows.
`PortQueue.aqms` is a **list**, so a single queue can carry **both** an
ECN and an EARLY_DROP policy at once (mark ECN-capable traffic, drop the
rest) — the ASIC applies the matching one per packet.
`congestion-behavior` is the **key** that identifies which AQM entry an
edit targets, so it is required whenever `active-queue-management` is
given:
```
fboss2-dev config qos default-queue-config 6 active-queue-management \
congestion-behavior ECN detection linear minimum-length 40000 maximum-length 60000 probability 100
fboss2-dev config qos default-queue-config 6 active-queue-management \
congestion-behavior EARLY_DROP detection linear minimum-length 30000 maximum-length 50000 probability 10
```
**Usage:**
```
$ fboss2-dev config qos default-queue-config 0 scheduling WRR weight 10
Successfully configured default-queue-config queue-id 0
$ fboss2-dev config session commit # AGENT_COLDBOOT — restarts agents
$ fboss2-dev delete qos default-queue-config 0
Successfully deleted default-queue-config queue-id 0
$ fboss2-dev config session commit # AGENT_COLDBOOT
```
- **AQM edits require `congestion-behavior`.** It is the key that
selects which policy on the queue to edit. Without it the target was
ambiguous, and creating an entry silently committed `behavior =
EARLY_DROP` (thrift enum default 0). The command now appends a new entry
when the named behavior is absent (so ECN and EARLY_DROP coexist)
instead of clobbering `aqms[0]`, and rejects a missing or duplicated
`congestion-behavior`.
- **Transactional edit.** The queue is built on a local copy and spliced
into `defaultPortQueues` only after every argument validates, so a bad
argument can't leave a half-built queue/AQM entry in the config session.
- **Shared PortQueue attribute handling.** `config qos
default-queue-config` and `config qos queuing-policy` configure the same
`cfg::PortQueue` attributes into different parts of the switch config;
the shared logic now lives in `PortQueueConfigUtils::applyPortQueueConfig`
used by both, and the `0..128` queue-id bound is one shared constant.
- **fake-SAI shutdown fix.** The agent's SAI graceful-exit sets
`SwitchPreShutdown`; the fake SAI didn't handle it, returned
`SAI_STATUS_INVALID_PARAMETER` and aborted the agent (a thrown
`SaiApiError` inside the noexcept shutdown handler `std::terminate()`s)
on every commit-triggered restart under the simulator. The fake SAI now
accepts it as a no-op, matching real adapters. Real hardware is
unaffected.
- `commands/config/qos/PortQueueConfigUtils.{h,cpp}` — shared
`applyPortQueueConfig`: PortQueue scalar + AQM attribute handling.
- `commands/config/qos/default_queue_config/CmdConfigQosDefaultQueueConfig.{h,cpp}`
— new config command (transactional apply, behavior-keyed AQM).
- `commands/delete/qos/CmdDeleteQos.{h,cpp}`,
`commands/delete/qos/default_queue_config/CmdDeleteQosDefaultQueueConfig.{h,cpp}`
— new `delete qos` subtree + handler.
- `commands/config/qos/queuing_policy/CmdConfigQosQueuingPolicyQueueId.cpp`
— migrated onto the shared helper.
- `utils/CmdUtilsCommon.h` — shared `kMaxQueueId` queue-id bound.
- `agent/hw/sai/fake/FakeSaiSwitch.cpp` — accept
`SAI_SWITCH_ATTR_PRE_SHUTDOWN` as a no-op.
- `CmdListConfig.cpp` — register the new commands.
- unit + integration tests; cmake/BUCK wiring.
- **Unit**: `cmd_config_test` passes. AQM coverage pins the new behavior
(each fails on a revert): ECN+EARLY_DROP coexistence (`aqms.size()==2`,
both entries preserved), edit-by-behavior preserves the other entry,
bare `congestion-behavior` preserves existing detection,
missing/duplicate behavior rejected, no-attributes rejected; delete
queue-id bound (`>128` rejected, `128` accepted).
- **Integration (real hardware, NH-4010-F, real-SAI)**:
`ConfigQosDefaultQueueConfigTest` all pass, sw/hw agents `NRestarts=0`
(no agent crash), device restored to its pre-test state.
```
[ OK ] SetWeightOnDefaultQueue (48.0 s)
[ OK ] SetSchedulingOnDefaultQueue (48.7 s)
[ OK ] CreateThenDeleteQueueEntry (49.6 s)
[ OK ] AqmEcnAndEarlyDropCoexist (50.1 s) # ECN + EARLY_DROP both in running config
[ PASSED ] 4 tests.
```
- **Simulator (fake-SAI)**: the build-and-test check was crashing
(SIGSEGV) because the fake SAI aborted on `SwitchPreShutdown` during the
commit-triggered agent restart; with the fake-SAI no-op fix the check
is green.
Three of the four tests in the suite asserted behavior that already has
coverage a layer down and does not need a device:
- SetWeightOnDefaultQueue and SetSchedulingOnDefaultQueue duplicated
setWeightExistingQueue, setSchedulingShortName and
setSchedulingFullEnumName in
fboss/cli/fboss2/test/config/CmdConfigQosDefaultQueueConfigTest.cpp,
which runs the same handler against a seeded ConfigSession.
- AqmEcnAndEarlyDropCoexist duplicated aqmCoexistEcnAndEarlyDrop in that
same file, and on the agent side PortQueueTests.cpp already pushes a
queue carrying both an ECN and an EARLY_DROP policy through the real
ThriftConfigApplier (aqmState, aqmBadState).
What is left, CreateThenDeleteQueueEntry, covers the only thing the other
layers cannot reach: that a committed config session survives the agent
restart and lands in the running config, and that delete takes it back out.
Both commands commit at ConfigActionLevel::AGENT_COLDBOOT, so each removed
test was costing two agent restarts. Restarts per run drop from eight to two.
Picking an unused queue id also removes the hadEntry conditional-restore
branch the old tests needed for ASICs with an empty defaultPortQueues.
rabbit-nexthop
force-pushed
the
nos-7161-qos-default-queue-config
branch
from
July 29, 2026 12:29
ae27344 to
a640c18
Compare
The rationale lives in the PR description instead.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds two commands for the switch-wide default port queue
settings:
What is defaultPortQueues? Each front-panel port has a set of
hardware egress queues, and each queue carries settings for scheduling
(strict-priority vs. weighted round robin), buffer allocation (reserved
and shared bytes, MMU scaling factor) and congestion management (ECN
marking / early drop thresholds).
SwitchConfig.defaultPortQueues(alist<PortQueue>inswitch_config.thrift) is the fallback queueconfig the agent applies to every port that has no explicit
portQueueConfigNameoverride — one place to define the queue behaviorfor the whole switch. The config is sparse: only the queues you define
are changed; queues absent from the list are reset to the ASIC defaults
(WRR, weight 1).
How the commands behave:
defaultPortQueuesits entry is updated in place; otherwise a new entryis created (with
scheduling = WEIGHTED_ROUND_ROBIN, since the thriftfield is required). Multiple
<attr> <value>pairs can be set in oneinvocation.
scheduling(full enum names or shortWRR/
SP/DRR),weight,reserved-bytes,shared-bytes,scaling-factor,stream-type,buffer-pool-name, andactive-queue-management(see below).delete qos default-queue-config <queue-id>erases that queue's entry fromdefaultPortQueues,reverting the queue to ASIC defaults at the next coldboot. Per-attribute
deletion is intentionally not supported — re-run the config command to
change individual attributes. Deleting a queue-id that has no entry is
an error (fails loudly on typos instead of staging a no-op coldboot
commit), consistent with the copp delete commands.
(
QueueScheduling,MMUScalingFactor,StreamType,QueueCongestionBehavior) and the error message lists every validvalue; byte/weight/threshold values must be non-negative; unknown
attributes, missing values and malformed queue-ids are rejected before
anything is staged. Queue-ids are bounded to
0..128(shared with thequeuing-policy parser).
carving cannot be applied hitlessly, so committing restarts the agents.
Active Queue Management (AQM)
A queue can act on packets before it fills, to signal congestion early
instead of tail-dropping when full. Each AQM policy pairs a
detection ramp (when to act) with a congestion-behavior (what to
do):
detection linear minimum-length <bytes> maximum-length <bytes> probability <pct>: a WRED ramp over queue depth.Below
minimum-lengthdo nothing; between min and max act with aprobability rising to
probability%; abovemaximum-lengthact always.congestion-behavior <ECN|EARLY_DROP>:EARLY_DROP— drop packets early (WRED). Works for any traffic.ECN— mark the Congestion-Experienced bit instead of dropping; onlyfor ECN-capable flows.
PortQueue.aqmsis a list, so a single queue can carry both anECN and an EARLY_DROP policy at once (mark ECN-capable traffic, drop the
rest) — the ASIC applies the matching one per packet.
congestion-behavioris the key that identifies which AQM entry anedit targets, so it is required whenever
active-queue-managementisgiven:
Usage:
Extra things done in this PR
config qos default-queue-configandconfig qos queuing-policyconfigure the samecfg::PortQueueattributes into different parts of the switch config;the shared logic now lives in
PortQueueConfigUtils::applyPortQueueConfigused by both, and the
0..128queue-id bound is one shared constant.SwitchPreShutdown; the fake SAI didn't handle it, returnedSAI_STATUS_INVALID_PARAMETERand aborted the agent (a thrownSaiApiErrorinside the noexcept shutdown handlerstd::terminate()s)on every commit-triggered restart under the simulator. The fake SAI now
accepts it as a no-op, matching real adapters. Real hardware is
unaffected.
Changed files
commands/config/qos/PortQueueConfigUtils.{h,cpp}— sharedapplyPortQueueConfig: PortQueue scalar + AQM attribute handling.commands/config/qos/default_queue_config/CmdConfigQosDefaultQueueConfig.{h,cpp}— new config command (transactional apply, behavior-keyed AQM).
commands/delete/qos/CmdDeleteQos.{h,cpp},commands/delete/qos/default_queue_config/CmdDeleteQosDefaultQueueConfig.{h,cpp}— new
delete qossubtree + handler.commands/config/qos/queuing_policy/CmdConfigQosQueuingPolicyQueueId.cpp— migrated onto the shared helper.
utils/CmdUtilsCommon.h— sharedkMaxQueueIdqueue-id bound.agent/hw/sai/fake/FakeSaiSwitch.cpp— acceptSAI_SWITCH_ATTR_PRE_SHUTDOWNas a no-op.CmdListConfig.cpp— register the new commands.Test Plan
cmd_config_testpasses. AQM coverage pins the new behavior(each fails on a revert): ECN+EARLY_DROP coexistence (
aqms.size()==2,both entries preserved), edit-by-behavior preserves the other entry,
bare
congestion-behaviorpreserves existing detection,missing/duplicate behavior rejected, no-attributes rejected; delete
queue-id bound (
>128rejected,128accepted).test.
NRestarts=0on the sw agent before and after — both agentrestarts in the run were the deliberate coldboots, not crashes — and
the device is left at its pre-test config.
Review Findings
Reviewed with fboss-review (11-reviewer pass) prior to submission;
findings addressed. No open issues above the confidence threshold.