Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 53 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ A Netflow Parser library for Cisco V5, V7, V9, and IPFIX written in Rust. Suppor
- [Pending Flow Caching](#pending-flow-caching)
- [Filtering Versions](#filtering-versions)
- [Error Handling Configuration](#error-handling-configuration)
- [Chained messages and error recovery](#chained-messages-and-error-recovery)
- [Custom Enterprise Fields (IPFIX)](#custom-enterprise-fields-ipfix)
- [Netflow Common](#netflow-common)
- [Re-Exporting Flows](#re-exporting-flows)
Expand Down Expand Up @@ -353,6 +354,20 @@ let parser = NetflowParser::builder()

When using template-based protocols (V9/IPFIX), data packets sometimes arrive before their corresponding template. By default these flows are reported as `NoTemplate` and lost. With pending flow caching enabled, these flows are automatically cached and replayed when the template arrives.

**Not every `NoTemplate` flowset is replayable.** `NoTemplateInfo::reason`
distinguishes the two cases, and `NoTemplateInfo::is_replayable()` answers the
question directly:

| `NoTemplateReason` | Meaning | Cached for replay |
| --- | --- | --- |
| `MissingTemplate` | The template ID is not in the cache. The records may decode once it arrives. | yes |
| `UndecodableBody` | The Set is well framed but its body does not match the octets present. | no |

An `UndecodableBody` flowset is never cached, because no template arriving later
changes its octets — caching it would retry it forever. It is still reported in
`flowsets` so the loss stays visible. Such a Set is skipped rather than ending
the walk, so the Sets behind it still decode.

**Disabled by default.** Enable via the builder pattern:

```rust
Expand Down Expand Up @@ -469,10 +484,18 @@ for packet in result.packets {

// Check for errors
if let Some(e) = result.error {
eprintln!("Error after {} packets: {}", packet_count, e);
eprintln!("{} packets parsed, first error: {}", packet_count, e);
}
```

**`error` is the first error, and packets may follow it.** A malformed IPFIX
message is discarded on its own rather than ending the parse: because the IPFIX
Message Header carries its own length, the parser resumes at the next message in
the datagram (see [Chained messages](#chained-messages-and-error-recovery)). So
`result.packets` can contain packets from both before and after the reported
error, and when several messages fail, `error` holds the first — the one closest
to the root cause.

**iter_packets()** yields `Result<NetflowPacket, NetflowError>` for per-packet error handling:

```rust,ignore
Expand All @@ -485,7 +508,35 @@ for result in parser.iter_packets(&buffer) {
}
```

**Error types**: `Incomplete`, `UnsupportedVersion`, `FilteredVersion`, `ParseError`, `Partial`. All implement `Display` and `std::error::Error`.
**Iteration continues past a recoverable error.** When a discarded IPFIX message
locates the next one, the iterator yields the error and resumes, so a single bad
message does not end the stream. It stops only on an error that leaves no
trustworthy boundary to resume from — a declared length below the 16-octet
Message Header, one that leaves nothing behind it, or a filtered version. v5, v7
and v9 cannot locate the next message after a failure and always stop.

### Chained messages and error recovery

A single datagram may carry several chained IPFIX messages.
[RFC 7011 section 9.1](https://www.rfc-editor.org/rfc/rfc7011.html#section-9.1)
requires discarding a malformed *message*, not the rest of the datagram, so the
parser resumes at the next message whenever the header locates one. One bad
message therefore costs only its own records.

`AutoScopedParser::parse_from_source` resumes the same way, and re-derives the
scope from each surviving message's header, so a message is never decoded with
another observation domain's templates.

**Error types**: `Incomplete`, `UnsupportedVersion`, `FilteredVersion`, `ParseError`, `Partial`, `MalformedSetFraming`, `DecodedOutputLimitExceeded`. All implement `Display` and `std::error::Error`.

`MalformedSetFraming` is reported when an IPFIX Set Header declares a length
that cannot be walked past — below the 4-octet Set Header, or beyond the end of
the Set region. It carries the offending Set's `offset`, its `declared` length
and the octets `available`, plus the discarded message's `export_time`,
`sequence_number` and `observation_domain_id`, so a collector can identify the
exporter at fault and account for the records it did not receive. It is
deliberately distinct from `Partial`, which continues to mean a datagram
truncated in flight.

#### Error Sample Size Configuration

Expand Down
26 changes: 26 additions & 0 deletions RELEASES.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,32 @@
elsewhere, `pcaps/IPFIX.pcap` included, so retiring both captures loses no
coverage.

* **A Set this parser cannot decode no longer discards the Sets behind it.**
Set iteration stopped at the first Set whose body failed to decode and
dropped the rest of the message without reporting anything — the remaining
half of #310, which the Set-framing check could not reach, because such a Set
is perfectly well framed and only its *body* is undecodable. A Template Set
declaring more fields than its octets carry, for instance, silently destroyed
every valid Data Set behind it while the message reported success.

Because Set framing is now validated across the whole chain before any Set is
parsed, the next Set's offset is known even when the current one fails, so the
walk steps over it and keeps decoding. The skipped Set is reported as
`FlowSetBody::NoTemplate` carrying the new
`NoTemplateInfo::reason` of `NoTemplateReason::UndecodableBody`, so the loss
stays visible without consuming the single `ParseResult::error` slot.

`reason` also distinguishes the pre-existing case: a flowset whose template is
merely absent reports `NoTemplateReason::MissingTemplate` and remains eligible
for pending-flow replay. An undecodable Set never is — no template arriving
later changes its octets, so caching it would retry it forever.
`NoTemplateInfo::is_replayable()` exposes the distinction.

Three bundled test snapshots recorded `flowsets: []` before this change and
now show the Sets that were being dropped, so they were not testing what their
names claimed. `NoTemplateInfo` is `#[non_exhaustive]`, so adding `reason` is
not a breaking change.

* **A malformed IPFIX message no longer discards the rest of the datagram.**
RFC 7011 section 9.1 discards the *Message*, but parsing stopped at the first
message-level error and abandoned every message chained behind it — so one
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -147,3 +147,4 @@ expression: parser.parse_bytes(&packet).packets
- 0
- 0
truncated: false
reason: MissingTemplate
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,46 @@ expression: "NetflowParser::default().parse_bytes(&packet).packets"
export_time: 16909060
sequence_number: 0
observation_domain_id: 16909060
flowsets: []
flowsets:
- header:
header_id: 2
length: 20
body:
NoTemplate:
template_id: 2
raw_data:
- 1
- 0
- 0
- 3
- 0
- 8
- 0
- 4
- 0
- 12
- 0
- 4
- 0
- 65
- 0
- 0
truncated: false
reason: UndecodableBody
- header:
header_id: 256
length: 12
body:
NoTemplate:
template_id: 256
raw_data:
- 1
- 2
- 3
- 4
- 1
- 2
- 3
- 4
truncated: false
reason: MissingTemplate
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,35 @@ expression: "NetflowParser::default().parse_bytes(&packet).packets"
export_time: 1670052913
sequence_number: 0
observation_domain_id: 0
flowsets: []
flowsets:
- header:
header_id: 2
length: 26
body:
NoTemplate:
template_id: 2
raw_data:
- 1
- 4
- 0
- 2
- 128
- 103
- 255
- 255
- 24
- 77
- 128
- 103
- 255
- 255
- 0
- 0
- 24
- 77
- 129
- 64
- 0
- 4
truncated: false
reason: UndecodableBody
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,110 @@ expression: "NetflowParser::default().parse_bytes(&packet).packets"
export_time: 1744871721
sequence_number: 100000
observation_domain_id: 0
flowsets: []
flowsets:
- header:
header_id: 1
length: 14
body:
NoTemplate:
template_id: 1
raw_data:
- 4
- 232
- 0
- 1
- 0
- 6
- 0
- 34
- 0
- 4
truncated: false
reason: UndecodableBody
- header:
header_id: 1256
length: 8
body:
NoTemplate:
template_id: 1256
raw_data:
- 0
- 0
- 0
- 1
truncated: false
reason: MissingTemplate
- header:
header_id: 2
length: 44
body:
Template:
template_id: 258
field_count: 9
fields:
- field_type_number: 8
field_length: 4
field_type:
IANA: SourceIpv4address
- field_type_number: 12
field_length: 4
field_type:
IANA: DestinationIpv4address
- field_type_number: 7
field_length: 2
field_type:
IANA: SourceTransportPort
- field_type_number: 11
field_length: 2
field_type:
IANA: DestinationTransportPort
- field_type_number: 4
field_length: 1
field_type:
IANA: ProtocolIdentifier
- field_type_number: 2
field_length: 4
field_type:
IANA: PacketDeltaCount
- field_type_number: 1
field_length: 4
field_type:
IANA: OctetDeltaCount
- field_type_number: 22
field_length: 4
field_type:
IANA: FlowStartSysUpTime
- field_type_number: 21
field_length: 4
field_type:
IANA: FlowEndSysUpTime
- header:
header_id: 258
length: 33
body:
Data:
fields:
- - - IANA: SourceIpv4address
- Ip4Addr: 70.70.1.1
- - IANA: DestinationIpv4address
- Ip4Addr: 120.120.1.1
- - IANA: SourceTransportPort
- DataNumber:
U16: 12345
- - IANA: DestinationTransportPort
- DataNumber:
U16: 32777
- - IANA: ProtocolIdentifier
- ProtocolType: Tcp
- - IANA: PacketDeltaCount
- DataNumber:
U32: 100
- - IANA: OctetDeltaCount
- DataNumber:
U32: 125000000
- - IANA: FlowStartSysUpTime
- DataNumber:
U32: 1744871721
- - IANA: FlowEndSysUpTime
- DataNumber:
U32: 1744871721
8 changes: 8 additions & 0 deletions src/variable_versions/ipfix/parser.rs
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ use crate::template_store::{
encode_ipfix_options_template, encode_ipfix_template, encode_v9_options_template,
encode_v9_template,
};
use crate::variable_versions::NoTemplateReason;
use crate::variable_versions::config::DEFAULT_MAX_RECORDS_PER_FLOWSET;
use crate::variable_versions::enterprise_registry::EnterpriseFieldRegistry;
use crate::variable_versions::field_value::FieldValue;
Expand Down Expand Up @@ -645,6 +646,12 @@ impl IPFixParser {
for (i, flowset) in ipfix.flowsets.iter_mut().enumerate() {
match &mut flowset.body {
FlowSetBody::NoTemplate(info) => {
// A Set whose body could not be decoded is not waiting on a
// template: no template that arrives later changes its
// octets, so caching it for replay would retry it forever.
if !info.is_replayable() {
continue;
}
// Reject flowsets with impossibly small headers (RFC minimum is 4).
// Also reject truncated raw_data (oversized entry at parse time).
// The flowset is kept in output as diagnostic data.
Expand Down Expand Up @@ -1747,6 +1754,7 @@ impl FlowSetBody {
template_id: id,
raw_data,
truncated,
reason: NoTemplateReason::MissingTemplate,
};
Ok((&[] as &[u8], FlowSetBody::NoTemplate(info)))
} else {
Expand Down
Loading
Loading