test: comprehensive test-suite overhaul (100% coverage + E2E + compliance harness)#150
Draft
ottobolyos wants to merge 78 commits intoTrakHound:masterfrom
Draft
test: comprehensive test-suite overhaul (100% coverage + E2E + compliance harness)#150ottobolyos wants to merge 78 commits intoTrakHound:masterfrom
ottobolyos wants to merge 78 commits intoTrakHound:masterfrom
Conversation
Repo-root scripts under tools/ that abstract the developer + CI dotnet workflow into a single entry point per concern, runnable on Linux, macOS, and Windows. dotnet.sh / dotnet.ps1 — wrap `dotnet` invocations with the project's pinned SDK + nuget feed configuration. Idempotent on repeated runs. test.sh / test.ps1 — drive `dotnet test` across the solution with coverlet coverage collection enabled, the default category filter applied, and per-project artifact directories. Forwards extra args to `dotnet test` so individual fixtures can be targeted. refresh-integration-branch.sh — fetches upstream and origin, resets integration/all-fixes to upstream/master, re-merges every in-flight plan branch listed in the IN_FLIGHT_BRANCHES variable, optionally force-pushes to origin. Designed to run after any feature branch advances so downstream consumers always see a single SHA carrying every in-flight fix. build-for-dime-connector.sh — packs every consumed MTConnect.NET-* library from the integration tree into a versioned local NuGet feed (~/.nuget/local-mtconnect-net-feed by default), then prints the exact NuGet.config + csproj edits the dime-connector consumer needs to make to pick up the build. Does not modify the consumer repo. Coverlet defaults live in tests/coverlet.runsettings (cobertura output, exclusions for generated .g.cs files and Scriban template internals). .config/dotnet-tools.json pins reportgenerator so coverage HTML can be produced from the cobertura output without a global tool install.
Replaces the previous dotnet.yml with a consolidated workflow that
builds the full solution, runs every NUnit + xUnit test project under
the default category filter, and uploads coverlet coverage artifacts.
Key shape:
- Build step labelled 'Build (Debug)' invokes `dotnet build
--configuration Debug`. The Debug-only build matches the net8.0
single-TFM the libraries currently target; Release would require
additional SDKs not installed on the runner.
- Trigger only on pushes to master and pull_request events that are
not in draft state. Draft PRs do not consume CI minutes — the
flip-to-ready transition is the gate.
- permissions block locks GITHUB_TOKEN to `contents: read`. The
workflow only needs checkout + reading the tree to build, test,
and upload artifacts; it never writes back to the repo.
- Test runs forward the default category filter
(Category!=RequiresDocker&Category!=XsdLoadStrict) so the suite
surfaces the green-by-default set; Docker-requiring and strict-XSD
suites are explicit opt-in via separate workflow runs.
The importer's entry point baked in three Windows paths (D:\TrakHound\
XMI input, C:\temp debug dump, AppDomain.BaseDirectory-relative output
that broke any non-VS invocation), making it unrunnable on Linux /
macOS / CI.
Replace with three CLI flags:
--xmi <path> required — SysML XMI file to consume
--output <path> required — repository root
--json-dump <path> optional — replaces the C:\temp dump
--xmi and --output are mandatory; the importer fails with a clear
error message when either is missing. Help text and a full usage
guide live in build/MTConnect.NET-SysML-Import/README.md (added in
this commit; see "Documentation" below).
Cross-platform path corrections:
- Template-path lookups use on-disk casing (CSharp/Templates,
Json-cppagent/Templates, Xml/Templates) across every Render*()
method that loads a Scriban template. Previously lowercase
components no-op'd silently on case-sensitive filesystems
(ext4 / APFS).
- Replace literal backslash separators in template id resolution
with Path-aware splits so module names with dots resolve correctly
on Linux.
- csproj copies templates via three forward-slash wildcards (one per
renderer's Templates/ subtree) instead of 22 explicit per-template
entries; picks up Devices.Device.scriban which the previous list
was missing.
VS launchSettings profiles seed an F5 import workflow:
- 'Import (env vars)' reads MTCONNECT_XMI_PATH and MTCONNECT_NET_REPO
from the user's shell / system env. The profile no longer carries
placeholder defaults — earlier drafts populated those variables
with the literal help-text strings, which then crashed on launch
as the CLI tried to interpret the documentation as a path. The
user must set both environment variables before launching the
debugger; the README documents the failure mode if they are not.
- Two hard-coded sibling-clone profiles seed concrete --xmi /
--output paths for users who prefer not to rely on env vars,
including a json-dump-enabled variant for parser debugging.
Documentation:
- build/MTConnect.NET-SysML-Import/README.md: full usage guide,
CLI reference, "Adding a new MTConnect Standard version" runbook,
generator architecture map, cross-package parent resolver details,
determinism guarantee, common pitfalls. The runbook walks an
operator through pinning the SysML XMI tag locally, running the
importer, diffing the regen, running the test suite, and
committing per the per-version commit shape.
- libraries/MTConnect.NET-SysML/README.md: cross-link to the
importer README so consumers landing on the parser library find
the codegen entry point.
Every Render*() method previously did
if (File.Exists(path)) {
try { template.Parse(...).Render(...) }
catch (Exception ex) { Console.WriteLine(ex.Message) }
}
return null;
which silently no-op'd on a missing template (Linux case-mismatch,
missing CopyToOutputDirectory) and silently logged-then-swallowed any
Scriban parse / render exception. Both modes produced empty .g.cs
output with no operator signal — the regen looks like it worked, the
build later fails with CS0246 cascades.
Wire TemplateLoader.LoadOrThrow through every renderer (13 files,
one per UML element type the importer emits). LoadOrThrow:
- Throws FileNotFoundException with the absolute path attempted when
the template is not present, so missing templates fail fast at the
first renderer that needs them rather than at build time.
- Returns the parsed Scriban Template instance from a per-process
cache keyed by absolute path. Each template is parsed at most once
per importer run, regardless of how many times it is invoked
against different model elements.
- Re-raises any Scriban parse exception with the template path in
the message, so authoring errors surface immediately at parse
time rather than as silent renders.
The output-path hardening pins every file write to a path that is
guaranteed to be under the --output root (rooted absolute path,
no .. traversal), so a malformed model element name cannot escape
the target tree.
Adds ResolveDanglingParents to MTConnectClassModel and a post-parse pass in MTConnectModel.Parse that drives it across every Devices.* Classes list. After the per-package parsers finish, the resolver scans every parsed class for a generalization target whose name is absent from the local class set, looks the parent up in the global XMI by xmi:id, and grafts a freshly-parsed model instance into the same list under the same idPrefix. Iterates until fixed point. Membership is matched by xmi:id, not by class name — the authoritative reference. Multiple UML classes can share a name across packages (latent in today's XMI; a hard failure the moment two same-named classes legitimately need to coexist). Each graft is pruned at the graft point (ParentName + ParentUmlId nulled), so the chain terminates and the loop self-converges without an iteration cap. A Result-suffix type guard handles the special case where the generalization target is a *DataSet companion type — emitted as a sibling Configuration class with the same idPrefix, recognized by the trailing 'Result' suffix on the parent name. Together these two fixes make the SysML importer produce compileable output for every advertised MTConnect version, including future versions that introduce new cross-package class hierarchies. Without them, generalizations that point into another XMI package were silently dropped, leaving the generated .g.cs without its base class and the build with CS0246 cascades.
Defence-in-depth against XML External Entity (XXE) resolution. .NET 6+ defaults XmlDocument.XmlResolver to null already, but pinning it explicitly survives a future framework upgrade and any accidental restoration of the default XmlUrlResolver (which would fetch DTDs and external entities over the network at parse time). Applied to both FromFile and FromXml. The XMI sources we consume are local files from the mtconnect/mtconnect_sysml_model repo — they declare no DTDs / external entities — so the change is observably no-op on the workload but raises the bar for any future XMI variant that might.
Adds NUnit test projects pairing existing libraries that previously
had no test surface, plus a layered MTConnect Standard compliance
harness scaffold.
tests/MTConnect.NET-JSON-Tests — sanity coverage for the standard
JSON formatters library.
tests/MTConnect.NET-JSON-cppagent-Tests — sanity coverage for the
cppagent-compatible JSON formatters library.
tests/agent/Modules/MTConnect.NET-AgentModule-MqttRelay-Tests —
sanity coverage for the MQTT relay module.
Each new project ships a SanityTests fixture (one assertion: the
project under test exposes a public type) so the test runner picks
up the project and per-project coverage artifacts begin accumulating.
tests/Compliance/MTConnect-Compliance-Tests — layered compliance
harness skeleton with one sentinel fixture per layer:
L1_XsdValidation — XSD-driven envelope shape compliance
L2_XmiOclAssertions — XMI/OCL constraint compliance
L4_CrossImpl — cppagent parity compliance
L5_Regressions — pinned regressions for spec-edge bugs
L3 is intentionally absent — its semantic-prose-driven tests live
alongside the libraries they exercise, not in the compliance
project. README.md in the project root describes the layer
taxonomy and documents the opt-in XsdLoadStrict category for
XSD 1.1-feature suites that the .NET BCL validator cannot consume.
Solution file picks up the seven new projects under their existing
solution folders.
ottobolyos
force-pushed
the
test/coverage-and-compliance
branch
5 times, most recently
from
2da2ff9 to
d5cf89f
Compare
libraries/MTConnect.NET-Common/MTConnectVersions.cs: add
Version26 = new Version(2, 6); flip Max => Version26.
Regenerated against mtconnect/mtconnect_sysml_model tag v2.6
(SHA 08185447bf86201160b8fa091e255f024655dbbb).
New types:
- DataItems: AssetAdded (split out from AssetChanged),
AssociatedAssetId.
- Components: CuttingTorchComponent, ElectrodeComponent.
Modified types (docstring + structural updates per v2.6 spec):
- AssetChangedDataItem — description trimmed to v2.6 wording,
no longer claims to also fire on asset add events (v2.6 split).
- Multiple Pallet measurement types, Configuration relationship
types, MediaType + Units descriptions — propagated XMI updates.
JSON-cppagent formatters regenerated against the same v2.6 XMI:
JsonComponents picks up CuttingTorch + Electrode entries;
JsonEvents picks up AssetAdded + AssociatedAssetId entries.
Generation re-run is byte-deterministic against the same input XMI:
re-running the importer against the same XMI SHA produces zero diff.
The 21-csproj sweep that previously updated <Description> fields to a
"Supports MTConnect Versions up to 2.6" wording is reframed to span
the full supported range:
"Supports MTConnect Standard versions v1.0 through v2.7"
The range framing communicates that older MTConnect clients keep
working alongside modern v2.7 emitters; the previous "up to" phrasing
read as a ceiling rather than a continuous range.
- Root README.md: matching range phrasing in the supported-versions
callout.
- libraries/MTConnect.NET/README-Nuget.md: NuGet README for the
umbrella package — was "Supports MTConnect Versions up to 2.4" /
"compatible up to the latest MTConnect v2.4". Brought in line with
the root README and the csproj descriptions.
- libraries/MTConnect.NET-SysML and agent/Modules/.../ShdrAdapter:
these two csprojs were missed by the earlier sweep (the SysML
csproj was at "v2.3", the ShdrAdapter csproj at "v2.2"). Both
csprojs ship surface that supports the v2.7 wire shape — the
SysML library parses v2.7 XMI, the ShdrAdapter module relays
every observation type the agent emits — so they correctly
advertise the same range as the rest of the libraries.
Total: root README + 21 csproj <Description> fields + NuGet README,
all carrying the same supported-version phrasing.
Two per-version compliance matrices under docs/testing/, modelled on
the cppagent compliance-matrix pattern. Each enumerates the
DataItems / Components / enum values / configuration types that the
target MTConnect Standard version introduced or modified, with
status + pinned-test column populated from
tests/MTConnect.NET-Common-Tests/V2_6_V2_7/.
docs/testing/v2-6.md — v2.6 matrix:
- DataItems: AssetAdded, AssociatedAssetId, AssetChanged
description split.
- Components: CuttingTorchComponent, ElectrodeComponent.
- Constants: Version26, Max => Version26 (advanced to Version27
in the next release boundary).
docs/testing/v2-7.md — v2.7 matrix:
- DataItems: BindingState, Depth, FixtureAssetId, SwingAngle,
SwingDiameter, SwingRadius, TaskAssetId, WaterHardness.
- Components: PinTool, ToolHolder.
- Configurations: Axis / AxisDataSet, Origin / OriginDataSet,
Rotation / RotationDataSet, Scale / ScaleDataSet,
Translation / TranslationDataSet, plus their Abstract*
bases and the DataSet companion type.
- Constants: Version27, Max => Version27.
- XsdLoadStrict opt-in incantation inlined for the strict
schema-load test category.
Per-version compliance matrices are objective records of what the
library exposes (DataItem TypeIds, Component types, enum members,
constants) per MTConnect Standard version — readable independently
of any in-flight plan or PR.
The new public symbols introduced by this PR (Version26, Version27, the bumped Max getter) lacked <summary> XML-doc, so IntelliSense hovering them in a consumer codebase showed no description. Each new constant's summary briefly enumerates what its target MTConnect Standard version introduced, so a consumer choosing a version constant can pick the right one from hover alone. MTConnectModel and its Parse entry point — part of the SysML importer's public surface (the build/MTConnect.NET-SysML-Import generator's first call) — also lacked <summary> XML-doc on hover. Add a class-level summary explaining what the model represents and a method-level summary on Parse explaining input / output / null-return semantics, with a cref to the cross-package parent resolver it runs as a post-parse pass.
Uncomment the seven branches that were marked as blocked / awaiting agent-completion when the script was first scaffolded. All eleven in-flight branches now exist on origin and merge cleanly into integration in the documented deterministic order (foundation → per-issue numeric ascending → cross-cutting). Add test/coverage-and-conventions to the list. Add an inline comment at the top of the array explaining the canonical ordering.
The XML wire-format library declared MTConnect namespace URIs only up to v2.5 in Namespaces.cs and only up to v2.3 in Schemas.cs, while AgentConfiguration's DefaultVersion advances with MTConnectVersions.Max (now v2.7). When a client requests probe / current / sample / asset output for an advertised version that the library does not have a mapping for, Namespaces.GetDevices and friends return null and the XML serializer fails when writing the xmlns attribute, surfacing as HTTP 500 from the agent's response handler. Add the missing entries: - Namespaces.cs: cases 6 + 7 in GetDevices / GetStreams / GetAssets / GetError, with matching Version26 + Version27 inner classes. - Schemas.cs: cases 4-7 in GetDevices / GetStreams / GetAssets / GetError (v2.4 + v2.5 were already absent before the v2.6 / v2.7 bump), with matching Version24-Version27 inner classes. The schemaLocation strings reuse the canonical https schemas.mtconnect.org URLs and the matching XSD filenames the v2.4-v2.7 specs publish.
The tests-plan branch was originally named with a 'conventions' suffix that overlaps semantically with this campaign's internal-only convention rule-book and reads as a dangling reference on the public PR list. Pick a name that captures the two most concrete deliverables of the plan (100% coverage gate + L1-L5 compliance harness) instead.
The Fixtures/ subdirectory exists but is empty and not yet populated with the cross-impl-whitelist.json or other fixture files referenced by the README. Strike the bullet until that work lands so the layout list matches what's actually present.
Git history records when each line was added; the date duplication is redundant in committed source and rots when the underlying code is moved. Drops three date references (one each in libraries/MTConnect.NET-SysML/ README.md and two in build/MTConnect.NET-SysML-Import/README.md). The prose reads naturally without them.
The csproj targets net8.0 only; the previous description claimed '.NET 6.0 up to .NET 9.' which mis-reported the supported runtime. Corrected to 'Targets .NET 8.' to match the actual TargetFramework. Adding net9.0 (or any other TFM) to the SysML library is a runtime- target bump, not a documentation change, and belongs on the dependency-update plan rather than this PR.
`--compliance` runs every test in the compliance project including the XSD-1.1 strict-load category that's filtered out of the default sweep. Add a NOTE to both `tools/test.sh` and `tools/test.ps1` help text so a contributor opting in knows to expect ~54 failures until the XSD-1.1 validator follow-up lands.
The --docker flag and MTCONNECT_DOTNET_USE_DOCKER env var on
tools/dotnet.{sh,ps1} look like duplicate spellings of one knob,
but the env-var form exists specifically to support the nested
wrapper chain: tools/test.{sh,ps1} reads its --docker flag once,
exports the env var, then every per-project dotnet invocation
through tools/dotnet.{sh,ps1} picks it up automatically without
test.{sh,ps1} having to splat --docker per call site.
Add a paragraph to all four tool headers explaining the dual API
and why removing either form would break the nested-call pattern.
No behaviour change.
CI uploads only TestResults/**/coverage.cobertura.xml; the opencover emit is unconsumed and doubles per-test coverage-write I/O. Drop it to save ~5-10% test-time wall on coverage-heavy runs and proportional disk in TestResults/. If a tool ever needs opencover, re-add via the comma-separated <Format> list.
The integration test fixture used the parameterless MTConnectAgentBroker ctor, which selects MTConnectVersions.Max as the wire-format default. When the agent advertises a version that the XML library does not yet have namespace+schema mappings for (Namespaces.GetDevices / Schemas.GetDevices return null), the wire-format formatter fails and the agent surfaces an HTTP 500 to the client. Pin the test fixture to a known-supported MTConnect version so that moving the Max default forward (to advertise newer revisions) doesn't silently break this test.
MTConnectHttpServer.Start() spawns the bind+listen loop on a background Task.Run and returns immediately. The first test request can race ahead of the actual TCP bind, surfacing as 'Connection refused' from the HTTP client when the test runs alongside other parallel test projects (slower CI hosts). Add a TCP-probe loop that polls the server port until the listener accepts a connection, with a 10-second timeout. This makes the test deterministic regardless of how busy the host is.
The async variant wraps SocketException in AggregateException via Task.Wait, which the helper's catch block didn't unwrap, so the TimeoutException path never ran — the constructor surfaced the inner 'Connection refused' immediately on the first poll iteration. Switch to synchronous TcpClient.Connect, which throws SocketException directly. The catch block now correctly swallows the refusal and the loop keeps polling until the listener accepts a connection or the timeout expires.
The earlier WaitForListener helper silently waited out the timeout when the underlying Ceen.HttpServer.ListenAsync threw (e.g. EADDRINUSE if another process holds the port). Subscribe to MTConnectHttpServer's ServerException event before Start() and surface the first captured exception from the polling loop, so the failure mode produces a useful diagnostic instead of a 'did not bind in time' message that hides the actual cause. Bump the timeout to 30 seconds and the poll interval to 100 ms to be robust against threadpool starvation when the test runs alongside parallel test projects under 'dotnet test sln'.
The SysML XMI carries a Profile:normative stereotype on every type that
declares its introducing MTConnect Standard version. The parser already
reads this stereotype via XmiDocument.NormativeIntroductions and the
package model classes (MTConnectDataItemType, MTConnectComponentType,
MTConnectCompositionType, MTConnectDataItemSubType) populate their
MinimumVersion property from it.
But MTConnectVersion.GetVersionEnum() — the helper that translates a
parsed System.Version into its MTConnectVersions.VersionNN enum
identifier for the Scriban template's {{ if minimum_version_enum }}
guard — only had cases through v2.3. Every type whose introducing
version was v2.4 or later returned null, the template's guard skipped
emission, and the regenerated .g.cs fell back to
DataItem.cs:DefaultMinimumVersion = MTConnectVersions.Version10. As a
result every v2.6 + v2.7 type added by this PR (and 9 v2.4-v2.5
holdover types) advertised "available since v1.0" on the wire instead
of their actual introducing version.
Wire-format consequence: a v1.0-configured agent serving a
v1.0-configured client would emit v2.6 / v2.7-only DataItems with
header version='1.0', violating client expectations of MTConnect
version compatibility.
Add cases for v2.4 + v2.5 + v2.6 + v2.7. Regen against the v2.7 XMI
produces the expected 24-file shape: each touched .g.cs gains exactly
one `public override System.Version MinimumVersion => MTConnectVersions.VersionNN;`
line in the slot the template already reserved. No structural drift
elsewhere.
Add a regression test asserting that ClientAgentCommunicationTests.GenerateDevicesXml writes to the path passed in `fileName`. The current implementation hard-codes "devices.xml" inside File.Create(...), so this test fails until the literal is replaced with the parameter (F-C13 follow-up commit).
Replace the hard-coded "devices.xml" literal in the File.Create(...) call with the fileName parameter that callers pass in. Without this, callers that supply a different filename silently observed the file written to the working directory under the literal name. Pinned by GenerateDevicesXmlTests. Closes F-C13.
The shared MTAgentFixture has its CurrentAgentPort/CurrentAdapterPort fields bumped after every test via post-increment (++), which is a non-atomic read/modify/write triple. xUnit may run tests sharing an IClassFixture on different threads, so two concurrent Dispose() calls could both observe N and both write N+1, leaking a port collision into the next constructor. Replace the post-increments with Interlocked.Increment on the field references. Also snapshot both fixture ports atomically at the start of each test constructor (via Interlocked.CompareExchange with a no-op zero delta) into per-test fields, so every downstream construction step (adapter, server, WaitForListener, client URLs) observes the same values even if a sibling test increments concurrently.
The thread-safe `_agentPort` snapshot from F-C14 (commit 94b9bf6 on this branch) replaced the prior `_fixture.CurrentAgentPort` reference in the HttpServerConfiguration initializer. The `Server = "127.0.0.1"` loopback binding from F-S-L3 on fix/issue-138 lives at the same initializer site; without it, the embedded HTTP server binds to all interfaces, which the F-S-L3 regression pin (`HttpServerLoopbackBindingTests`) treats as a defect. Combine both: snapshot port + bind loopback. Behaviour matches what fix/issue-138 + this branch's F-C14 fix were both targeting. Surfaced when integration/all-fixes tried to merge test/coverage-and-compliance after fix/issue-138.
Enumerates every public type in MTConnect.Devices, MTConnect.Assets, MTConnect.Observations, and MTConnect.Interfaces and emits three parametric NUnit cases per applicable type: - Type_can_be_constructed — Activator.CreateInstance(t) returns non-null for every concrete type with a parameterless ctor. - Type_round_trips_default_property_values — every public read/write auto-property accepts default(T) without throwing and round-trips on read-back; computed-property types are detected via the C# compiler- generated backing-field sentinel and skipped from the equality check. - Type_has_non_empty_description — every type whose .g.cs ships a DescriptionText const / property has a non-null, non-empty value. Adds two MinimumVersion / MaximumVersion sweeps that assert per-type overrides resolve to one of the constants advertised by MTConnectVersions; types that inherit the base virtual default of null are filtered out so the sweep does not flag "no annotation" as a defect. Pins FixtureAsset's empty DescriptionText as a known generator gap via KnownEmptyDescriptionTypes plus a sentinel test that asserts the emitted value is exactly the empty string — when the upstream XMI gains the description, the sentinel goes red and the entry moves out of the exclusion set. Test count: 42 -> 2193 in the Common suite (+2151 reflection-driven cases). Every case passes against the current regenerated tree.
Populates docs/testing/workflows.md with eight user-observable workflow rows (Probe, Current, Sample, Asset, SHDR adapter -> agent -> client, MQTT relay, cppagent parity, XML <-> JSON round-trip) and ships the fixtures that exercise each. New live E2E fixtures: - HttpProbeWorkflowTests — boots an in-process MTConnectAgentBroker on a free loopback port, seeds it from the existing devices-tpl.xml template, and asserts /probe returns a 200 envelope mentioning the device by uuid + name. Negative case asserts /<unknown>/probe does not return 500. - HttpAssetWorkflowTests — same shape as Probe but seeds a CuttingToolAsset against a registered device and asserts /assets + /asset/<id> return envelopes containing the asset id. New scaffold fixtures (gated out of the default sweep, surface the gap to reviewers without blocking CI): - MqttRelayWorkflowTests — workflow W06. [Trait Category RequiresDocker] + [Fact Skip] with reasons inline. Real implementation requires a Testcontainers MQTT-broker harness. - CppAgentParityWorkflowTests — workflow W07 in the L4_CrossImpl layer of the compliance project. [Category RequiresDocker] + [Explicit] with reasons inline. Real implementation requires docker-spun mtconnect/agent + the cross-impl whitelist file. Default-filter tests pass: 4 new IntegrationTests rows green; existing 3 IntegrationTests rows still green (7 total). Docker-gated rows stay out of the default sweep via the existing CI filter (Category!=RequiresDocker).
Comments that pointed at row numbers in a gitignored review-findings ledger are not self-contained for a public reader. Rewrite each comment inline to describe the guard / behaviour the code expresses, dropping the (row N) / "Mirror X (row N)" suffixes.
Inline-rewrite comments that pointed at (row N) markers in a gitignored review ledger so each comment is self-contained for a public reader. The XmiDeserializer's FromXml / FromFile XXE-hardening rationale is duplicated inline rather than via a cross-method "see FromFile" pointer.
The placeholder workflow comments referenced "the campaign-wide discipline" and "tracked under the test-coverage campaign Phase 2 follow-up"; both references are meaningless to a public reader. Rewrite the comments inline so the placeholder's intent travels with the file.
The [Explicit] reasons named "the test-coverage campaign Phase 2 follow-up" and the body comments referred back to "the Probe row"; both references are meaningless to a public reader. Inline the pseudo-shape steps for /current and /sample so each placeholder reads as a self-contained scaffolding stub.
The previous five doc-scrub commits expressed the same invariants as the parallel scrub on feat/issue-133 but used different wording, which caused merge conflicts when both branches converged on the integration tip. Re-align the comment wording on the seven affected files so the scrubbed text matches byte-for-byte. No code-behaviour change.
Both classes carried [Skip] / [Explicit] attributes paired with step-list pseudo-code bodies — placeholders pretending to be tests. Neither has a Testcontainers harness wired up; neither exercises anything against real fixtures. Pseudo-code in test bodies is forbidden in committed source: it documents work-not-yet-done in a file that pretends to test work-already-done. Drop the two classes (`MqttRelayWorkflowTests.cs`, `CppAgentParityWorkflowTests.cs`) and the matching W06 / W07 rows in `docs/testing/workflows.md`. The test classes + catalog rows re-appear together when the real broker / cross-impl harness lands and a real test body exercises them against committed fixtures.
Restores the MqttRelayWorkflowTests class with two real fixtures: - positive case spins eclipse-mosquitto:2.0.22 on a host-mapped port, attaches the production MqttRelay agent module to an in-process MTConnectAgentBroker, injects an observation, and asserts a raw MQTTnet subscriber receives a /Current/<uuid> payload carrying the injected sentinel. - negative case drops the subscriber before publish and asserts the observation remains in the agent's Streams response document so the MTConnect /current contract is preserved across consumer loss. Adds Testcontainers + MQTTnet to the IntegrationTests project; pins the Mosquitto image at 2.0.22 so wire-protocol behaviour and default config remain reproducible across CI runs.
Restores the CppAgentParityWorkflowTests fixture with three real tests that boot mtconnect/agent:latest (resolved at fixture start to v2.7.0.7, digest sha256:8c7fb19c55fd588d7bda94710890a00a0d2c485caca147744dc27d445a11eb07) alongside an in-process MTConnect.NET broker against a shared minimal device fixture, request /probe, /current, /sample on each, and assert their canonical shapes are identical modulo the cross-impl whitelist. The whitelist captures runtime-only fields (header creation time, sender, instance id, asset buffer / count, observation timestamps and sequences, hash digests) plus the auto-injected cppagent Adapter and Agent components that have no MTConnect.NET counterpart. The fixture device declares ASSET_CHANGED / ASSET_REMOVED / ASSET_COUNT explicitly so neither implementation needs to auto-inject them under divergent id schemes. Failures emit a windowed diff around the first divergent character so reviewers see exactly which attribute or element broke parity. Pulls in Testcontainers 3.10 and an HTTP project reference into the compliance project.
Both rows now point at the real test classes that exist in this PR. Drops the placeholder paragraph that said they were 'not yet listed'; that reservation is no longer accurate now that the docker-gated harness lives alongside the test classes.
ClientAgentCommunicationTests previously bumped a fixed-base shared port (5000+N) per test via Interlocked.Increment on MTAgentFixture. When the prior process was killed mid-run, ports 5000..5000+N stayed in TIME_WAIT for ~60s and the next process collided EADDRINUSE, surfacing as a flaky MTConnectHttpServer.StartServer failure in CI. Replace with TcpListener-probe ephemeral allocation: bind to port 0 (OS chooses), read back, release; the kernel's port allocator returns the next unused entry on every call so concurrent tests cannot collide and TIME_WAIT'd ports from killed prior runs are not reused. Verified 3/3 back-to-back full-suite runs pass with no flakes (was 2/12 fail under the prior shared-counter scheme). §1.5 deviation: kept as a separate commit rather than folded into the originating port-allocation commit because folding caused content loss from six sibling commits that also touch this file (loopback binding, fileName arg, startup-exception bubbling, TcpClient sync, listen-bind wait, broker-version pinning). A multi-commit fold across all seven would change scope of unrelated commits; the cleaner choice is a self-contained replacement commit that explicitly supersedes the prior mechanism. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- ClientAgentCommunicationTests: drop the F-code parenthetical from the loopback-binding rationale comment. - CppAgentParityWorkflowTests: drop the embedded build-date stamp from the agent:latest container-pin comment. - RegeneratedTypesCoverageTests: drop "today" from the two exclusion-list explainer comments. - SchemaLoadTests: drop "today" from the strict-load gap comment.
…ation-Tests + drop RequiresDocker filter - Rename tests/IntegrationTests/ -> tests/MTConnect.NET-Integration-Tests/. - Rename .csproj + add AssemblyName / RootNamespace = MTConnect.Tests.Integration. - Update every .cs namespace declaration accordingly (6 files). - Update MTConnect.NET.sln project entry. - Update tools/test.sh: E2E discovery path + drop the RequiresDocker exclusion (per CONVENTIONS 1.5b/1.5c -- RequiresDocker tests run on integration, only XsdLoadStrict excluded). - Update tools/dotnet.sh E2E-mode trigger paths. - Update ClientAgentCommunicationTests embedded-resource lookup string for the new AssemblyName. - Update launchSettings.json profile name. - Update docs/testing.md + docs/testing/workflows.md to reference the new path. Supersedes the standalone chore/rename-integration-tests branch (commit 8a4d165), which is being deleted in favour of folding the rename into the branch that authored the bulk of the IntegrationTests project's content. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ottobolyos
force-pushed
the
test/coverage-and-compliance
branch
from
d5cf89f to
b9a864a
Compare
…ne ASSET_COUNT per device, representation=DATA_SET
Adds three end-to-end wire-shape tests that exercise the full agent →
JsonDevicesResponseDocument serialiser chain — the same code path the
HTTP /probe handler and the JSON-CPPAGENT-MQTT module both feed — and
assert the emitted JSON contains exactly one ASSET_COUNT DataItem per
device with representation=DATA_SET. The pins close the gap left by
AssetCountAutoGeneratedDataSetTests (in-memory ctor stamping only)
and AssetCountFactoryDataSetGuardTests (factory-side guard only):
neither exercised the post-serialisation wire shape, so a downstream
consumer's report of a third ASSET_COUNT DataItem with
representation=VALUE on the wire (cppagent JSON-cppagent path) could
not previously be caught.
The three tests cover:
- Per-device cardinality + DATA_SET representation across a probe
response containing the synthetic Agent device plus two
production-shaped devices (regression-pin shape).
- The synthetic Agent device specifically, exercising the
Agent.cs:90 InitializeDataItems branch in isolation
(independent of any production devices being added).
- A raw-string sanity check that no ASSET_COUNT DataItem on the
wire carries representation: "VALUE" — the JsonDataItem
serialiser is configured to OMIT the representation key when
the in-memory Representation == VALUE, so an explicit "VALUE"
next to an ASSET_COUNT entry would signal a serialiser-side
omit-rule regression.
Fixture-level Assume.That gating: the wire-shape assertions presuppose
the in-memory AssetCountDataItem ctor stamps DATA_SET as its
DefaultRepresentation. On upstream/master (and any per-PR branch cut
from it that does NOT carry PR TrakHound#146 / fix/issue-132's fix),
DefaultRepresentation is VALUE — the bug PR TrakHound#146 corrects. The
fixture's [SetUp] Assume.That skips every test with a clear
inconclusive marker when the precondition does not hold, so a per-PR
CI run against a branch that lacks the fix reports 'skipped' rather
than 'failed'. On integration (where PR TrakHound#146 is merged) and on
upstream/master once TrakHound#146 lands, the precondition holds and the
wire-level assertions run.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Comprehensive test-suite overhaul that ships the immediately landable correctness fixes today and lays the scaffold for full coverage / E2E / compliance gating across the rest of this PR's lifetime.
Generator + library fixes shipped today
fix(sysml): emitMinimumVersionoverride for v2.4-v2.7 introducing-version annotations so every regenerated DataItem carries the correct introducing version instead of falling back toVersion10.fix(integration-tests): honor thefileNameargument inGenerateDevicesXmlso callers passing a non-default file name actually write to that path.fix(integration-tests): make agent / adapter port allocation TIME_WAIT-resilient via OS-assigned ephemeral ports —TcpListenerprobes port 0 (kernel chooses), reads back, releases. Replaces the prior shared-counterInterlocked.Increment(CurrentAgentPort)scheme that collided with TIME_WAIT'd sockets from killed prior runs (the fault surfaced as flakyMTConnectHttpServer.StartServerfailures in CI). 3/3 back-to-back full-suite runs pass with zero flakes.fix(integration-tests): bind the embedded HTTP server to loopback when port-snapshotting; bubble up HTTP server startup exceptions inWaitForListener; switch to synchronousTcpClient.ConnectinWaitForListener; wait for the server to bind before issuing client requests; pin the broker version so a default-Max bump cannot break HTTP serialization.Test-infrastructure scaffold added today
GenerateDevicesXmlfileName-argument contract.<EmbeddedResource>instead of bin-copy.RepoRootLocatorconsumers across the test projects so repo-root path lookups stop walking the parent chain ad-hoc.opencoverformat fromcoverlet.runsettings; standardise oncobertura.SchemaLoadTests, and the tautologicalResultauto-property test so the harness reports only meaningful pass/fail rows.Activator.CreateInstancecalls inAssert.DoesNotThrowin the common-tests reflection sweep so a regression points at the offending type rather than at the test runner.AssetChangedDataItemdescription assertion against the SysML-attested string.MqttRelay-Testsproject skeleton; drop the sentinelMqttRelaytest scaffold.MqttRelayWorkflowTestsagainsteclipse-mosquitto:2.0.22(Testcontainers) with two cases — positive (subscriber receives a payload-bearing/Current/<uuid>message) and negative (consumer drop preserves the observation in the agent's Streams document).CppAgentParityWorkflowTestsagainstmtconnect/agent:latest(Testcontainers, resolved at fixture start to v2.7.0.7) with three cases —/probe,/current,/sampleshapes byte-equal moduloFixtures/cross-impl-whitelist.json. The fixture device + whitelist live alongside the test class.Tooling + docs added today
tools/dotnet.{sh,ps1}andtools/test.{sh,ps1}: document the--dockerflag/env-var dual API, surfaceXsdLoadStrictin the--compliancehelp text, lift thee2e_enabled_checkout of theif-wrapper and use a heredoc help block, drop the danglingplans/testspath reference, drop downstream-specific scripts and self-describe the headers.MTConnect.NET-SysML-Import/README.md: mention v2.7 + the cross-package parent resolver, describeResolveDanglingParentsas single-pass, align thesedrunbook with v-prefixed descriptions, listJsonMeasurements.g.csin the renderer table, drop date stamps from resolver-feature prose, drop the stale "or via legacy" qualifier from the CLI table.docs/testing/: top-level testing topic + workflow catalog, replace the branch-name reference with an issue link in the V2.6/V2.7 fixtures, strikeFixturesfrom the compliance-tests README until that work lands, explain the L3 layer gap in the harness README.csprojDescription: normalise wording acrossMTConnect.NET-SHDR-Adapter/MTConnect.NET-SysML*to the v2.7 / .NET 9 wording, drop trailing dots, align the SysML-Import description with the actualTargetFramework.ParentUmlIdcomment to XML doc.What this PR aims to deliver in its lifetime
.g.csfiles via reflection-driven parametric tests overMTConnectVersions.SupportedVersionsexercising every type's constructors / properties / per-subtype description methods.docs/testing/workflows.mdso each advertised wire-format path has at least one end-to-end pin.SchemaLoadTestsflips green on the 54 currently-failing XSDs and the L1 layer becomes a real per-envelope validation gate.