Skip to content

[Nexthop][fboss2-dev] add fboss2-dev config/delete qos default-queue-config commands - #1408

Open
vybhav-nexthop wants to merge 3 commits into
facebook:mainfrom
nexthop-ai:nos-7161-qos-default-queue-config
Open

[Nexthop][fboss2-dev] add fboss2-dev config/delete qos default-queue-config commands#1408
vybhav-nexthop wants to merge 3 commits into
facebook:mainfrom
nexthop-ai:nos-7161-qos-default-queue-config

Conversation

@vybhav-nexthop

@vybhav-nexthop vybhav-nexthop commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR adds two commands for the switch-wide default port queue
settings:

fboss2-dev config qos default-queue-config <queue-id> <attr> <value> [<attr> <value> ...]
fboss2-dev delete qos default-queue-config <queue-id>

What is defaultPortQueues? Each front-panel port has a set of
hardware egress queues, and each queue carries settings for scheduling
(strict-priority vs. weighted round robin), buffer allocation (reserved
and shared bytes, MMU scaling factor) and congestion management (ECN
marking / early drop thresholds). SwitchConfig.defaultPortQueues (a
list<PortQueue> in switch_config.thrift) is the fallback queue
config the agent applies to every port that has no explicit
portQueueConfigName override — one place to define the queue behavior
for the whole switch. The config is sparse: only the queues you define
are changed; queues absent from the list are reset to the ASIC defaults
(WRR, weight 1).

How the commands behave:

  • Set/merge semantics (config): if the queue-id already exists in
    defaultPortQueues its entry is updated in place; otherwise a new entry
    is created (with scheduling = WEIGHTED_ROUND_ROBIN, since the thrift
    field is required). Multiple <attr> <value> pairs can be set in one
    invocation.
  • Supported attributes: scheduling (full enum names or short WRR
    / SP / DRR), weight, reserved-bytes, shared-bytes,
    scaling-factor, stream-type, buffer-pool-name, and
    active-queue-management (see below).
  • Delete removes the whole entry: delete qos default-queue-config <queue-id> erases that queue's entry from defaultPortQueues,
    reverting the queue to ASIC defaults at the next coldboot. Per-attribute
    deletion is intentionally not supported — re-run the config command to
    change individual attributes. Deleting a queue-id that has no entry is
    an error (fails loudly on typos instead of staging a no-op coldboot
    commit), consistent with the copp delete commands.
  • Validation: enum attributes are checked against the thrift enums
    (QueueScheduling, MMUScalingFactor, StreamType,
    QueueCongestionBehavior) and the error message lists every valid
    value; byte/weight/threshold values must be non-negative; unknown
    attributes, missing values and malformed queue-ids are rejected before
    anything is staged. Queue-ids are bounded to 0..128 (shared with the
    queuing-policy parser).
  • Commit level: AGENT_COLDBOOT for both commands — queue buffer
    carving cannot be applied hitlessly, so committing restarts the agents.

Active Queue Management (AQM)

A queue can act on packets before it fills, to signal congestion early
instead of tail-dropping when full. Each AQM policy pairs a
detection ramp (when to act) with a congestion-behavior (what to
do):

  • detectiondetection linear minimum-length <bytes> maximum-length <bytes> probability <pct>: a WRED ramp over queue depth.
    Below minimum-length do nothing; between min and max act with a
    probability rising to probability%; above maximum-length act always.
  • congestion-behaviorcongestion-behavior <ECN|EARLY_DROP>:
    • EARLY_DROP — drop packets early (WRED). Works for any traffic.
  • ECN — mark the Congestion-Experienced bit instead of dropping; only
    for ECN-capable flows.

PortQueue.aqms is a list, so a single queue can carry both an
ECN and an EARLY_DROP policy at once (mark ECN-capable traffic, drop the
rest) — the ASIC applies the matching one per packet.
congestion-behavior is the key that identifies which AQM entry an
edit targets, so it is required whenever active-queue-management is
given:

# ECN on queue 6
fboss2-dev config qos default-queue-config 6 active-queue-management \
    congestion-behavior ECN detection linear minimum-length 40000 maximum-length 60000 probability 100

# add EARLY_DROP alongside it (a second entry — does not clobber the ECN one)
fboss2-dev config qos default-queue-config 6 active-queue-management \
    congestion-behavior EARLY_DROP detection linear minimum-length 30000 maximum-length 50000 probability 10

Usage:

$ fboss2-dev config qos default-queue-config 0 scheduling WRR weight 10
Successfully configured default-queue-config queue-id 0
$ fboss2-dev config session commit          # AGENT_COLDBOOT — restarts agents

$ fboss2-dev delete qos default-queue-config 0
Successfully deleted default-queue-config queue-id 0
$ fboss2-dev config session commit          # AGENT_COLDBOOT

Extra things done in this PR

  • Shared PortQueue attribute handling. config qos default-queue-config and config qos queuing-policy configure the same
    cfg::PortQueue attributes into different parts of the switch config;
    the shared logic now lives in PortQueueConfigUtils::applyPortQueueConfig
    used by both, and the 0..128 queue-id bound is one shared constant.
  • fake-SAI shutdown fix. The agent's SAI graceful-exit sets
    SwitchPreShutdown; the fake SAI didn't handle it, returned
    SAI_STATUS_INVALID_PARAMETER and aborted the agent (a thrown
    SaiApiError inside the noexcept shutdown handler std::terminate()s)
    on every commit-triggered restart under the simulator. The fake SAI now
    accepts it as a no-op, matching real adapters. Real hardware is
    unaffected.

Changed files

  • commands/config/qos/PortQueueConfigUtils.{h,cpp} — shared
    applyPortQueueConfig: PortQueue scalar + AQM attribute handling.
  • commands/config/qos/default_queue_config/CmdConfigQosDefaultQueueConfig.{h,cpp}
    — new config command (transactional apply, behavior-keyed AQM).
  • commands/delete/qos/CmdDeleteQos.{h,cpp},
    commands/delete/qos/default_queue_config/CmdDeleteQosDefaultQueueConfig.{h,cpp}
    — new delete qos subtree + handler.
  • commands/config/qos/queuing_policy/CmdConfigQosQueuingPolicyQueueId.cpp
    — migrated onto the shared helper.
  • utils/CmdUtilsCommon.h — shared kMaxQueueId queue-id bound.
  • agent/hw/sai/fake/FakeSaiSwitch.cpp — accept
    SAI_SWITCH_ATTR_PRE_SHUTDOWN as a no-op.
  • CmdListConfig.cpp — register the new commands.
  • unit + integration tests; cmake/BUCK wiring.

Test Plan

  • Unit: cmd_config_test passes. AQM coverage pins the new behavior
    (each fails on a revert): ECN+EARLY_DROP coexistence (aqms.size()==2,
    both entries preserved), edit-by-behavior preserves the other entry,
    bare congestion-behavior preserves existing detection,
    missing/duplicate behavior rejected, no-attributes rejected; delete
    queue-id bound (>128 rejected, 128 accepted).
  • Integration (real hardware, Nexthop device, real-SAI): one end-to-end
    test. NRestarts=0 on the sw agent before and after — both agent
    restarts in the run were the deliberate coldboots, not crashes — and
    the device is left at its pre-test config.
[       OK ] ConfigQosDefaultQueueConfigTest.CreateThenDeleteQueueEntry (45564 ms)
[  PASSED  ] 1 test.

Review Findings

Reviewed with fboss-review (11-reviewer pass) prior to submission;
findings addressed. No open issues above the confidence threshold.

@vybhav-nexthop
vybhav-nexthop requested review from a team as code owners July 23, 2026 04:53
@meta-cla meta-cla Bot added the CLA Signed label Jul 23, 2026
@vybhav-nexthop
vybhav-nexthop force-pushed the nos-7161-qos-default-queue-config branch from d589036 to 03826bd Compare July 27, 2026 05:28
This PR adds two commands for the switch-wide default port queue
settings:

```
fboss2-dev config qos default-queue-config <queue-id> <attr> <value> [<attr> <value> ...]
fboss2-dev delete qos default-queue-config <queue-id>
```

**What is defaultPortQueues?** Each front-panel port has a set of
hardware egress queues, and each queue carries settings for scheduling
(strict-priority vs. weighted round robin), buffer allocation (reserved
and shared bytes, MMU scaling factor) and congestion management (ECN
marking / early drop thresholds). `SwitchConfig.defaultPortQueues` (a
`list<PortQueue>` in `switch_config.thrift`) is the fallback queue
config the agent applies to every port that has no explicit
`portQueueConfigName` override — one place to define the queue behavior
for the whole switch. The config is sparse: only the queues you define
are changed; queues absent from the list are reset to the ASIC defaults
(WRR, weight 1).

**How the commands behave:**

- **Set/merge semantics (config)**: if the queue-id already exists in
`defaultPortQueues` its entry is updated in place; otherwise a new entry
is created (with `scheduling = WEIGHTED_ROUND_ROBIN`, since the thrift
field is required). Multiple `<attr> <value>` pairs can be set in one
invocation.
- **Supported attributes**: `scheduling` (full enum names or short `WRR`
/ `SP` / `DRR`), `weight`, `reserved-bytes`, `shared-bytes`,
`scaling-factor`, `stream-type`, `buffer-pool-name`, and
`active-queue-management` (see below).
- **Delete removes the whole entry**: `delete qos default-queue-config
<queue-id>` erases that queue's entry from `defaultPortQueues`,
reverting the queue to ASIC defaults at the next coldboot. Per-attribute
deletion is intentionally not supported — re-run the config command to
change individual attributes. Deleting a queue-id that has no entry is
an error (fails loudly on typos instead of staging a no-op coldboot
commit), consistent with the copp delete commands.
- **Validation**: enum attributes are checked against the thrift enums
(`QueueScheduling`, `MMUScalingFactor`, `StreamType`,
`QueueCongestionBehavior`) and the error message lists every valid
value; byte/weight/threshold values must be non-negative; unknown
attributes, missing values and malformed queue-ids are rejected before
anything is staged. Queue-ids are bounded to `0..128` (shared with the
queuing-policy parser).
- **Commit level: AGENT_COLDBOOT** for both commands — queue buffer
carving cannot be applied hitlessly, so committing restarts the agents.

A queue can act on packets *before* it fills, to signal congestion early
instead of tail-dropping when full. Each AQM policy pairs a
**detection** ramp (when to act) with a **congestion-behavior** (what to
do):

- **detection** — `detection linear minimum-length <bytes>
maximum-length <bytes> probability <pct>`: a WRED ramp over queue depth.
Below `minimum-length` do nothing; between min and max act with a
probability rising to `probability%`; above `maximum-length` act always.
- **congestion-behavior** — `congestion-behavior <ECN|EARLY_DROP>`:
  - `EARLY_DROP` — drop packets early (WRED). Works for any traffic.
- `ECN` — mark the Congestion-Experienced bit instead of dropping; only
for ECN-capable flows.

`PortQueue.aqms` is a **list**, so a single queue can carry **both** an
ECN and an EARLY_DROP policy at once (mark ECN-capable traffic, drop the
rest) — the ASIC applies the matching one per packet.
`congestion-behavior` is the **key** that identifies which AQM entry an
edit targets, so it is required whenever `active-queue-management` is
given:

```
fboss2-dev config qos default-queue-config 6 active-queue-management \
    congestion-behavior ECN detection linear minimum-length 40000 maximum-length 60000 probability 100

fboss2-dev config qos default-queue-config 6 active-queue-management \
    congestion-behavior EARLY_DROP detection linear minimum-length 30000 maximum-length 50000 probability 10
```

**Usage:**

```
$ fboss2-dev config qos default-queue-config 0 scheduling WRR weight 10
Successfully configured default-queue-config queue-id 0
$ fboss2-dev config session commit          # AGENT_COLDBOOT — restarts agents

$ fboss2-dev delete qos default-queue-config 0
Successfully deleted default-queue-config queue-id 0
$ fboss2-dev config session commit          # AGENT_COLDBOOT
```

- **AQM edits require `congestion-behavior`.** It is the key that
selects which policy on the queue to edit. Without it the target was
ambiguous, and creating an entry silently committed `behavior =
EARLY_DROP` (thrift enum default 0). The command now appends a new entry
when the named behavior is absent (so ECN and EARLY_DROP coexist)
instead of clobbering `aqms[0]`, and rejects a missing or duplicated
`congestion-behavior`.
- **Transactional edit.** The queue is built on a local copy and spliced
into `defaultPortQueues` only after every argument validates, so a bad
argument can't leave a half-built queue/AQM entry in the config session.
- **Shared PortQueue attribute handling.** `config qos
default-queue-config` and `config qos queuing-policy` configure the same
`cfg::PortQueue` attributes into different parts of the switch config;
the shared logic now lives in `PortQueueConfigUtils::applyPortQueueConfig`
used by both, and the `0..128` queue-id bound is one shared constant.
- **fake-SAI shutdown fix.** The agent's SAI graceful-exit sets
`SwitchPreShutdown`; the fake SAI didn't handle it, returned
`SAI_STATUS_INVALID_PARAMETER` and aborted the agent (a thrown
`SaiApiError` inside the noexcept shutdown handler `std::terminate()`s)
on every commit-triggered restart under the simulator. The fake SAI now
accepts it as a no-op, matching real adapters. Real hardware is
unaffected.

- `commands/config/qos/PortQueueConfigUtils.{h,cpp}` — shared
  `applyPortQueueConfig`: PortQueue scalar + AQM attribute handling.
- `commands/config/qos/default_queue_config/CmdConfigQosDefaultQueueConfig.{h,cpp}`
  — new config command (transactional apply, behavior-keyed AQM).
- `commands/delete/qos/CmdDeleteQos.{h,cpp}`,
  `commands/delete/qos/default_queue_config/CmdDeleteQosDefaultQueueConfig.{h,cpp}`
  — new `delete qos` subtree + handler.
- `commands/config/qos/queuing_policy/CmdConfigQosQueuingPolicyQueueId.cpp`
  — migrated onto the shared helper.
- `utils/CmdUtilsCommon.h` — shared `kMaxQueueId` queue-id bound.
- `agent/hw/sai/fake/FakeSaiSwitch.cpp` — accept
  `SAI_SWITCH_ATTR_PRE_SHUTDOWN` as a no-op.
- `CmdListConfig.cpp` — register the new commands.
- unit + integration tests; cmake/BUCK wiring.

- **Unit**: `cmd_config_test` passes. AQM coverage pins the new behavior
  (each fails on a revert): ECN+EARLY_DROP coexistence (`aqms.size()==2`,
  both entries preserved), edit-by-behavior preserves the other entry,
  bare `congestion-behavior` preserves existing detection,
  missing/duplicate behavior rejected, no-attributes rejected; delete
  queue-id bound (`>128` rejected, `128` accepted).
- **Integration (real hardware, NH-4010-F, real-SAI)**:
  `ConfigQosDefaultQueueConfigTest` all pass, sw/hw agents `NRestarts=0`
  (no agent crash), device restored to its pre-test state.

```
[ OK ] SetWeightOnDefaultQueue      (48.0 s)
[ OK ] SetSchedulingOnDefaultQueue  (48.7 s)
[ OK ] CreateThenDeleteQueueEntry   (49.6 s)
[ OK ] AqmEcnAndEarlyDropCoexist    (50.1 s)   # ECN + EARLY_DROP both in running config
[ PASSED ] 4 tests.
```

- **Simulator (fake-SAI)**: the build-and-test check was crashing
  (SIGSEGV) because the fake SAI aborted on `SwitchPreShutdown` during the
  commit-triggered agent restart; with the fake-SAI no-op fix the check
  is green.
Three of the four tests in the suite asserted behavior that already has
coverage a layer down and does not need a device:

  - SetWeightOnDefaultQueue and SetSchedulingOnDefaultQueue duplicated
    setWeightExistingQueue, setSchedulingShortName and
    setSchedulingFullEnumName in
    fboss/cli/fboss2/test/config/CmdConfigQosDefaultQueueConfigTest.cpp,
    which runs the same handler against a seeded ConfigSession.
  - AqmEcnAndEarlyDropCoexist duplicated aqmCoexistEcnAndEarlyDrop in that
    same file, and on the agent side PortQueueTests.cpp already pushes a
    queue carrying both an ECN and an EARLY_DROP policy through the real
    ThriftConfigApplier (aqmState, aqmBadState).

What is left, CreateThenDeleteQueueEntry, covers the only thing the other
layers cannot reach: that a committed config session survives the agent
restart and lands in the running config, and that delete takes it back out.

Both commands commit at ConfigActionLevel::AGENT_COLDBOOT, so each removed
test was costing two agent restarts. Restarts per run drop from eight to two.

Picking an unused queue id also removes the hadEntry conditional-restore
branch the old tests needed for ASICs with an empty defaultPortQueues.
@rabbit-nexthop
rabbit-nexthop force-pushed the nos-7161-qos-default-queue-config branch from ae27344 to a640c18 Compare July 29, 2026 12:29
The rationale lives in the PR description instead.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant