Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
stage of an outer graph (with `finish()` and the context reaching into it), a
`Void`-terminated sink graph, and the in-band tick message. EXAMPLE.md walks
through both in sections 8 and 9.
- **Named merge slots** — a fan-in group can name the slots of a merge,
`(raw: msg, checked: valid) -> Merge`, and is then matched by name, in any
order: the merge receives its slots in the order it declares them. A merge
declares names with `MergeStage::mergeInputNames()` (override it in a
`TypedMergeFilter` / `UniformMergeFilter` subclass), or with the new
overloads `registerTypedMergeFilter<Out, Ins...>(name, slotNames, merger)`
and `registerMergeFilter<Out>(name, slotNames, combiner)`. Unknown or missing
names, a group that names only some slots, and names on a merge that declares
none are build-time diagnostics; type mismatches name the slot
(`slot 'raw' of 'Merge' expects ...`). Positional groups keep working and are
matched by position. `dsl::StageNode::slotNames` records the names, and
`dsl::toMermaid` labels the links with them.
- **`TODO.md`** — the planned work, with what the library does today, the gap,
an API sketch and the workaround for each item; the README roadmap summarizes
it.
Expand All @@ -74,6 +86,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
the `std::shared_ptr` for composites that forward it.

### Changed
- `MergeStage::mergeInputTypes()` is no longer pure; it defaults to no slot
types, so a merge can declare only names. `MergeFilter` now implements
`MergeStage`, and `AnyMessageFilter` has a new `mergeInputNames()`, which
defaults to none.
- **Breaking:** `AnyMessageFilter` has new pure virtuals
`setContext(std::shared_ptr<GraphContext>)` and `finish()`; custom
implementations must forward both to the stages they wrap. They are pure
Expand Down
44 changes: 44 additions & 0 deletions EXAMPLE.md
Original file line number Diff line number Diff line change
Expand Up @@ -271,6 +271,49 @@ compiler; they are shortened here.)
lambda, and `UniformMergeFilter<In, Out>` covers the other shape: any number of
slots, all of the same type (N variants of one computation, combined).

### Named slots

Slot types cannot catch everything: when two slots have the *same* type,
swapping them still passes validation. A merge can name its slots so that the
group says which edge goes where:

```cpp
class CompareFilter : public TypedMergeFilter<std::string, std::string, std::string>
{
public:
std::optional<std::string> merge(std::optional<std::string>&& before,
std::optional<std::string>&& after) override
{
return std::format("{} => {}", before.value_or("-"), after.value_or("-"));
}

std::vector<std::string> mergeInputNames() const override
{
return {"before", "after"};
}
};
static FilterRegistrar<CompareFilter> registerCompare("Compare");
```

```cpp
DslFilterGraph<std::string, int> pipeline(R"dsl(
in -> Uppercase -> upper
upper -> Reverse -> reversed
(after: reversed, before: upper) -> Compare -> compared -> Print(prefix="[named] ") -> out
)dsl");

pipeline.filter(std::string{"Hello, filterGraph!"});
```

```text
[named] HELLO, FILTERGRAPH! => !HPARGRETLIF ,OLLEH
```

The group lists `after` first, but the slots are matched by name, so `merge()`
still receives `before` first. A misspelled name, or a slot the group leaves
out, is reported when the graph is built. A group without names still feeds
`Compare` by position.

## 5. Several named outputs

A graph can have more than one output. `out.<key>` names each one, and the
Expand Down Expand Up @@ -779,6 +822,7 @@ The complete output of `textPipeline` (sections 1–7):
[typed] HELLO, FILTERGRAPH! (19 chars)
[typed check] 3:20: slot 1 of 'Report' expects 'class std::basic_string<char,...>' but edge 'length' carries 'unsigned __int64'
[typed check] 3:20: slot 2 of 'Report' expects 'unsigned __int64' but edge 'upper' carries 'class std::basic_string<char,...>'
[named] HELLO, FILTERGRAPH! => !HPARGRETLIF ,OLLEH
[outputs] upper=HELLO, FILTERGRAPH! length=19 long=dropped
[check] 1:7: unknown stage type 'Uppercas' — did you mean 'Uppercase'?
[check] 2:7: could not construct 'MinLength': [json.exception.out_of_range.403] key 'minLength' not found
Expand Down
41 changes: 39 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -254,7 +254,9 @@ edge -> Stage -> edge -> Stage(key=value) -> edge
stage, one slot per edge, in the order listed. A merge either declares its
slot types (see [Typed merges](#typed-merges)) or takes them untyped as
`MergeInputs` (`std::vector<std::any>`), as
`registerMergeFilter<OutputType>(name, combiner)` does.
`registerMergeFilter<OutputType>(name, combiner)` does. A group may name the
slots of a merge that declares names, `(raw: msg, checked: valid) -> Merge`,
and is then matched by name, in any order (see [Named slots](#named-slots)).
- `#` starts a comment that runs to the end of the line.

### How a graph runs
Expand Down Expand Up @@ -313,6 +315,42 @@ Mis-wiring a group is then a build-time diagnostic, not a wrong result:
`MergeFilter` / `registerMergeFilter` declare no slot types; their edges stay
unchecked, and the combiner reads the slots with `std::any_cast`.

### Named slots

Slot types cannot tell two slots of the *same* type apart: written in the wrong
order, they pass every check and run silently wrong. A merge can therefore name
its slots, and a group can refer to them by name:

```cpp
class Compare : public TypedMergeFilter<Verdict, Stats, Stats>
{
public:
std::optional<Verdict> merge(std::optional<Stats>&& before, std::optional<Stats>&& after) override { ... }

std::vector<std::string> mergeInputNames() const override { return {"before", "after"}; }
};

// The same for merges registered from a function:
registerTypedMergeFilter<Verdict, Stats, Stats>("Compare", {"before", "after"}, merger);
registerMergeFilter<Verdict>("CompareAny", {"before", "after"}, combiner);
```

```text
(after: current, before: baseline) -> Compare -> verdict
```

- A **named group** is matched by name, so its order does not matter: the merge
receives its slots in the order it declares them. An unknown name
(`'Compare' has no slot named 'befor' — did you mean 'before'?`) and a slot
the group does not feed are build-time diagnostics.
- A **positional group** still works with a merge that names its slots, matched
by position, so existing graphs stay valid. A merge that names its slots has
exactly that many.
- A group may name all of its slots or none. Naming the slots of a merge that
declares no names is an error.

`dsl::toMermaid` labels the links of a named group with the slot names.

### Choosing the output type

`DslFilterGraph<InputType, OutputType>`'s `OutputType` states what the graph's
Expand Down Expand Up @@ -586,7 +624,6 @@ Planned improvements, not yet implemented, live in [TODO.md](TODO.md) — each
with what the library does today, the gap, an API sketch and the workaround
available in the meantime. The current list:

- **Named merge slots**, so a fan-in group matches by name, not position.
- **Several named graph inputs** (`in.orders`, `in.quotes`), mirroring
`GraphOutputs`.
- **Stage labels and typed access** (`Summarize@stats`, `graph.stage<T>("stats")`).
Expand Down
77 changes: 18 additions & 59 deletions TODO.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,67 +18,26 @@ Examples use the generic stages of the [README](README.md) (`Parse`,

| # | Item | Impact | Compatibility | Workaround today |
|---|------|--------|---------------|------------------|
| 1 | [Named merge slots](#1-named-merge-slots) | medium | additive if done carefully | rely on group order |
| 2 | [Several named graph inputs](#2-several-named-graph-inputs) | medium | additive (new graph shape) | `std::variant` input + select stages |
| 3 | [Stage labels and typed access](#3-stage-labels-and-typed-access) | medium | additive (DSL syntax) | side registry filled by creator lambdas |
| 4 | [Track which stage short-circuited](#4-track-which-stage-short-circuited) | medium | additive | tap the edge, or log in the stage |
| 5 | [Config-aware `registerMergeFilter`](#5-config-aware-registermergefilter) | low | additive | subclass + `FilterRegistrar` creator |
| 6 | [Injectable registry, duplicate detection](#6-injectable-registry-duplicate-detection) | low | mostly additive | unique names |
| 7 | [Documentation: 0..n outputs, large messages](#7-documentation-0n-outputs-large-messages) | doc only | — | — |
| 8 | [Known limitations to lift](#8-known-limitations-to-lift) | low–medium | additive | JSON config; read `GraphOutputs` carefully |
| 1 | [Several named graph inputs](#1-several-named-graph-inputs) | medium | additive (new graph shape) | `std::variant` input + select stages |
| 2 | [Stage labels and typed access](#2-stage-labels-and-typed-access) | medium | additive (DSL syntax) | side registry filled by creator lambdas |
| 3 | [Track which stage short-circuited](#3-track-which-stage-short-circuited) | medium | additive | tap the edge, or log in the stage |
| 4 | [Config-aware `registerMergeFilter`](#4-config-aware-registermergefilter) | low | additive | subclass + `FilterRegistrar` creator |
| 5 | [Injectable registry, duplicate detection](#5-injectable-registry-duplicate-detection) | low | mostly additive | unique names |
| 6 | [Documentation: 0..n outputs, large messages](#6-documentation-0n-outputs-large-messages) | doc only | — | — |
| 7 | [Known limitations to lift](#7-known-limitations-to-lift) | low–medium | additive | JSON config; read `GraphOutputs` carefully |

Items of the same list that are done, and therefore not repeated here:
**type-checked merge inputs** (`TypedMergeFilter` / `UniformMergeFilter` /
`registerTypedMergeFilter`), the **end-of-stream hook**
(`MessageFilter::finish()`), both described in the README, and **examples for
`registerTypedMergeFilter`), **named merge slots**
(`(raw: msg, checked: valid) -> Merge`, `MergeStage::mergeInputNames()`), the
**end-of-stream hook** (`MessageFilter::finish()`), all described in the
README, and **examples for
the newer features** — `apps/statefulPipeline` and `apps/compositePipeline`,
walked through in [EXAMPLE.md](EXAMPLE.md) sections 8 and 9.

---

## 1. Named merge slots

**Today.** Slots are positional, in the order the DSL group lists them. A merge
stage can only know which upstream a slot came from by convention. Since
type-checked merges shipped, a mis-*typed* slot is caught when the graph is
built — but two slots of the *same* type in the wrong order still pass
validation and then run silently wrong.

**Proposal.** Optional slot names in the group, checked against the names the
stage declares:

```text
(raw: msg, checked: valid) -> Summarize -> out.stats
```

```cpp
// empty = positional, today's behaviour
virtual std::vector<std::string> mergeInputNames() const { return {}; }
```

Behaviour:

- **Stage declares names, group uses names:** slots are matched by name, so the
order in the group no longer matters. Unknown or missing names produce
diagnostics.
- **Stage declares names, group is positional:** allowed, matched by position
(keeps today's graphs valid).
- **Stage declares nothing:** names in the group are an error (*'Summarize' has
no named slots*).

Together with the slot types a typed merge already declares, this gives every
slot a name *and* a type, which also makes `dsl::toMermaid` output
self-explanatory.

**Compatibility.** Keep `MergeInputs` a `std::vector<std::any>` and reorder the
values into declared order before calling the stage, so stage code does not
change. Making `MergeInputs` a struct with names would be a breaking change and
is not needed.

**Workaround today.** Rely on group order, and give same-typed slots distinct
wrapper types so that the existing slot-type check can tell them apart.

## 2. Several named graph inputs
## 1. Several named graph inputs

**Today.** A graph has exactly one input edge `in`, with one type. A graph that
consumes several kinds of message needs a single `std::variant` input, and then
Expand Down Expand Up @@ -113,7 +72,7 @@ keeps the single `in` edge.
**Workaround today.** A `std::variant` input plus one select stage per
alternative, each dropping the alternatives it does not handle.

## 3. Stage labels and typed access
## 2. Stage labels and typed access

**Today.** The owner of a `DslFilterGraph` cannot reach a stage instance. The
DSL has no labels (`#` starts a comment), and the compiled plan is private.
Expand Down Expand Up @@ -143,7 +102,7 @@ auto& stats = graph.stage<SummarizeFilter>("stats"); // throws if unknown or the
every instance it builds in a side registry, or have the stage publish what the
owner needs through the `GraphContext`.

## 4. Track which stage short-circuited
## 3. Track which stage short-circuited

**Today.** When a graph drops a message, neither `DslFilterGraph` nor
`AnyFilterChain` tells the caller *which* stage returned `std::nullopt`.
Expand All @@ -164,7 +123,7 @@ so that variant needs an overload or an opt-in.
**Workaround today.** Tap the suspect edge with a logging stage ending in `end`,
or log inside the stage that decides to drop.

## 5. Config-aware `registerMergeFilter`
## 4. Config-aware `registerMergeFilter`

**Today.** `registerMergeFilter<Out>(name, combiner)` ignores the DSL config and
**copies one combiner into every instance**. A combiner lambda that captures a
Expand Down Expand Up @@ -194,7 +153,7 @@ one.

**Workaround today.** Subclass and register with a creator lambda.

## 6. Injectable registry, duplicate detection
## 5. Injectable registry, duplicate detection

**Today.** `FilterRegistry::instance()` is a process-wide singleton, and
`registerFilter` **silently overwrites** an existing name.
Expand Down Expand Up @@ -223,7 +182,7 @@ Ship it behind a transition: warn first, or add
**Workaround today.** Keep names unique, and register test fakes under their own
names.

## 7. Documentation: 0..n outputs, large messages
## 6. Documentation: 0..n outputs, large messages

No API change. These points belong in the README / EXAMPLE.md, because they come
up as soon as stages carry state:
Expand All @@ -240,7 +199,7 @@ up as soon as stages carry state:
its members persists across messages. Say explicitly that this is supported
and intended, and that each graph construction creates fresh instances.

## 8. Known limitations to lift
## 7. Known limitations to lift

The [current limitations](README.md#current-limitations) the README lists, as
work items:
Expand Down
37 changes: 36 additions & 1 deletion apps/textPipeline/main.cpp
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
// Generic, domain-unrelated example: a small text-processing pipeline that
// demonstrates the core building blocks of filterGraph:
// - MessageFilter: the base stage interface
// - FilterGraph: compile-time chaining of stages
// - FilterRegistry + DslFilterGraph: runtime graphs described in the text DSL,
// with fan-out (a "tap"), fan-in (a merge) and several named outputs
// with fan-out (a "tap"), fan-in (merges, with typed and named slots) and
// several named outputs
// - validateDslGraph: every problem in a broken graph, located by line:column
// - JsonFilterGraph: the JSON format, which remains supported
#include <filterGraph/core/filterGraph/DslFilterGraph.hpp>
Expand All @@ -24,6 +25,7 @@
#include <memory>
#include <optional>
#include <string>
#include <vector>

using filterGraph::DslFilterGraph;
using filterGraph::FilterGraph;
Expand Down Expand Up @@ -173,6 +175,26 @@

static FilterRegistrar<ReportFilter> registerReport("Report");

// A merge with two slots of the same type, where the order matters. The slot
// types cannot tell them apart, so it names them: a group can then match them
// by name, `(after: x, before: y) -> Compare`, in any order.
class CompareFilter : public TypedMergeFilter<std::string, std::string, std::string>
{
public:
std::optional<std::string> merge(std::optional<std::string>&& before,
std::optional<std::string>&& after) override
{
return std::format("{} => {}", before.value_or("-"), after.value_or("-"));
}

std::vector<std::string> mergeInputNames() const override
{
return {"before", "after"};
}
};

static FilterRegistrar<CompareFilter> registerCompare("Compare");

// Only needed by the JSON example: in the DSL, fan-out is built in.
static const bool sRegisterFanout = [] {
registerFanoutFilter<std::string>("Fanout");
Expand Down Expand Up @@ -240,6 +262,19 @@
}
}

// 3c) Named slots: Compare's two slots have the same type, so only their
// names say which is which. The group lists them in the other order;
// the merge still receives `before` first.
{
DslFilterGraph<std::string, int> pipeline(R"dsl(
in -> Uppercase -> upper
upper -> Reverse -> reversed
(after: reversed, before: upper) -> Compare -> compared -> Print(prefix="[named] ") -> out
)dsl");

pipeline.filter(std::string{"Hello, filterGraph!"});
}

// 4) Several named outputs of different types, returned as GraphOutputs.
{
DslFilterGraph<std::string> analysis(R"dsl(
Expand Down
18 changes: 18 additions & 0 deletions libs/filterGraph/core/filterGraph/AnyMessageFilter.hpp
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
#pragma once

#include <filterGraph/core/filterGraph/MergeStage.hpp>
Expand All @@ -6,8 +6,10 @@
#include <any>
#include <memory>
#include <optional>
#include <string>
#include <typeindex>
#include <utility>
#include <vector>

namespace filterGraph {

Expand Down Expand Up @@ -42,6 +44,13 @@
{
return {};
}

// The slot names of a merge stage (see MergeStage); empty for a merge with
// positional slots and for every stage that is not a merge.
virtual std::vector<std::string> mergeInputNames() const
{
return {};
}
};

// Adapts a concrete MessageFilter<Filter::InType, Filter::OutType> to the
Expand Down Expand Up @@ -95,6 +104,15 @@
return {};
}

std::vector<std::string> mergeInputNames() const override
{
if (const auto* merge = dynamic_cast<const MergeStage*>(mFilter.get()))
{
return merge->mergeInputNames();
}
return {};
}

private:
std::shared_ptr<Filter> mFilter;
};
Expand Down
Loading
Loading