spec: entity registration MQTT API revamp

[albinsuresh]

Proposed changes

A design doc detailing the existing issues with the entity registration MQTT APIs and a proposal to address them, with minimal backward compatibility breakage.

Types of changes

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Improvement (general improvements like code refactoring that doesn't explicitly fix a bug or add any new functionality)
• Documentation Update (if none of the other choices apply)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)

Paste Link to the issue

Checklist

• I have read the CONTRIBUTING doc
• I have signed the CLA (in all commits with git commit -s. You can activate automatic signing by running just prepare-dev once)
• I ran just format as mentioned in CODING_GUIDELINES
• I used just check as mentioned in CODING_GUIDELINES
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)

Further comments

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[rina23q]

Thank you for your accurate analysis and problem breakdown; it is greatly appreciated.

The proposal is debatable - hope as you expected. I left my opinions on the proposal.

[rina23q]

From the client's perspective (I'm imaging a custom cloud mapper development), it is a hassle because client always has to do validation on @source from the payload to distinguish the marked entity.
If a registered channel is separated from a request channel, it would be easy for clients as it just to subscribe the registered entity topic. However, I saw this idea is rejected in Proposal C.

[albinsuresh]

I wouldn't say rejected. The duplication of the same data across 2 topic channels was my concern. We can't change the existing request channels and their "retained" requirement. So, the the clients will continue publishing retained registration messages and if the agent duplicates the same (or a normalised version) on a different channel, the original retained request is now redundant and stays with the broker with no value. That was my concern with proposal C and the only reason why I put this as the preferred one. If we weren't limited by backward compatibility, I also would have picked proposal C.

We can explore that proposal further if the duplication isn't a concern OR, if the agent is allowed to clear the registration message published by the users once the normalised version is published to its own channel. Even with the proposal A, the agent is updating the user's request messages. So, clearing them can also be treated as another mutation. These are all options and nothing is accepted/rejected until we all agree. So, let's discuss, debate and finalise.

[rina23q]

Note: What I said Proposal C (new output channel: registration stays on te/+/+/+/+ exactly as today.) is renamed to Proposal B in this commit. 7eb4597

[albinsuresh]

Yes. The former Proposal B with a new registration topic was moved into the "Ideal Solution" section earlier in the doc, which triggered this rename.

[rina23q]

I have a suggestion: add a @last-updated timestamp to validated entity messages

Given that a custom mapper has its own entity store to manage their device IDs. When the mapper restarts and receives a burst of retained messages, it can compare the timestamp against its own cache to decide whether to update or skip.
Without timestamp, it may need to validate all keys and values in the cache if they are up-to-date.

[albinsuresh]

The mappers currently don't maintain a persistent entity store. What they have is a transient entity cache and it is rebuilt on every restart from the retained messages on the broker, as that's the source of truth. But yeah, once the agent becomes the owner of those retained entity metadata messages, a @last-updated field could be useful to persistent clients. Especially if we make the mappers persistent again, the way they used to be in the past. Earlier they were relying on the equality of the entity metadata between the entity store state and the message received.

[reubenmiller]

What they have is a transient entity cache and it is rebuilt on every restart from the retained messages on the broker, as that's the source of truth.

We can change this so that the mapper's request the entity store information via the http interface provided by the tedge-agent though?

[Bravo555]

Once we're at it, also would be nice to provide both @created-at and @last-updated-at to know if the entity was created previously and updated, or just has been created (when both timestamps are equal)

[albinsuresh]

Yes. That would make the startup logic of the mapper more straightforward, where it doesn't have to deal with any of the ordering problems between parent-child entities during that initial surge. The only issue here would be the added strong dependency on the agent process, in addition to mosquitto, during mapper startups, as it won't be able to do anything until the agent is also available. TBH, for starup I don't think that's a big deal.

[rina23q]

Actually I like the Proposal C. Is it really a problem to have two channels, one for registration and the other for truth of source? Their data might be same, but might be different if the registration message is malformed. I'm pretty sure that third-party services wants to subscribe only the truth of source channel.

[albinsuresh]

I don't think leaving an already processed request retained on the broker would be acceptable. That's just wasted resources. I'd say proposal C is feasible if the agent is allowed to clear the registration messages published by the clients once the normalised version is retained on its own channel.

[reubenmiller]

I would of thought we also have some issues with processing the order of messages when sent to the cloud.

For instance, with the service registration and status updates can result in the status sometimes being ignored.

[reubenmiller]

There is another order of messages issue, but more related to the difference to the Cumulocity MQTT and HTTP , where a child device software list can easily fail if the REST API updating the software list is sent before the child device has been registered via MQTT. This race condition has been seen within the "Software list should be populated during startup" test on the "software-update-child.robot" suite.

[Bravo555]

After thinking on it a bit, I'm convinced by the Proposal A. Adding a marker property that only later clients expect and verify is an elegant way to solve the problem with as few breaking changes as possible.
Though I do expect most of the value to come from resolving the inconsistencies between MQTT and HTTP API and updating the tests to cover these missing cases.
If the benefits were ever large enough for clients to update, we could also provide additional mechanisms under a meta topic like te/device/d_id/service/s_id/meta, but "providing additional registration/patch methods that eliminate some of the footguns" is probably not big enough of a benefit to justify the complexity of having to support and document both these methods.
For the same reason, I considered an A+B solution, where, instead of removing the registration at entity metadata topic, we could still process them while also allowing registration at a different topic like te/device/d_id/service/s_id/meta/register. One advantage of that approach is that new clients could opt into this new topic and their registration wouldn't trigger as duplicate "registration" message for older clients who observe this topic but aren't aware of the @source property, but that also probably be a doubtful proposition of bringing in more complexity than it removes.

[Bravo555]

question: would the agent overwriting these topics on bad input by itself, without adding @source: agent, be sufficient to solve these issues?

Indeed without @source: agent mappers wouldn't know if the entity is valid so they could send invalid entities to the cloud, but if these entities were then overwritten by the agent to clear the topic, they could be immediately removed. This is suboptimal to be sure, but the the problem described here is only about leftover broken retained messages, which I think could be solved that way?

In the proposal A, there's a note:

Restoring or clearing a retained message that the agent did not author is a fundamental change. It must be documented and comunicated heavily on roll-out.

Could this potentially be a breaking change?

Asking just for clarity of which portions of the proposal address which issues and how the change could be decomposed into a number of smaller action items.

[albinsuresh]

question: would the agent overwriting these topics on bad input by itself, without adding @source: agent, be sufficient to solve these issues?

Indeed without @source: agent mappers wouldn't know if the entity is valid so they could send invalid entities to the cloud, but if these entities were then overwritten by the agent to clear the topic, they could be immediately removed. This is suboptimal to be sure, but the the problem described here is only about leftover broken retained messages, which I think could be solved that way?

If we're re-using the same te/+/+/+/+ topic both as the request channel as well as the agent's authoritative channel, then the @source field is critical to avoid the mappers or clients from reacting to an invalid request. Yes, the agent would eventually replace the invalid message with a valid one as per proposal 1. But, what if the agent is down when the request was sent? The mapper would have already reacted to the original request and it wouldn't get that "correction message" until the agent comes back up. By making the mappers and other clients react only to messages with the @source field, we are preventing such issues.

In the proposal A, there's a note:

Restoring or clearing a retained message that the agent did not author is a fundamental change. It must be documented and comunicated heavily on roll-out.

Could this potentially be a breaking change?

This is one part that I'm not fully sure about. I don't expect any normal "publishing clients" that registers themselves and pushes data or receiving commands (which is likely the majority of clients) to be affected at all by this change, as they are less likely to subscribe to the te/+/+/+/+ topics that they are publishing to.

But, if there are other clients like mappers that were tracking the registration messages, they would definitely get notified. But, here also I'm not sure if it can be considered a "breaking behaviour" as those clients should have been prepared to received entity update messages on the same channel, as we've supported that (though in a broken manner). So, I'd expect those clients to treat the addition of the @source field as yet another update.

Asking just for clarity of which portions of the proposal address which issues and how the change could be decomposed into a number of smaller action items.

Sure, I'll add a section or a table after each proposal capturing how it addresses each and every listed issue.

[Bravo555]

As far as I understand, we can't do anything about this issue for the entity metadata topic that wouldn't be a breaking change, so the solution is to just use HTTP API.
Or, for clients who for some reason can't/won't use HTTP, we could consider providing additional methods for registering/updating entity metadata under some new te/device/d_id/service/s_id/meta topic.

[albinsuresh]

As far as I understand, we can't do anything about this issue for the entity metadata topic that wouldn't be a breaking change, so the solution is to just use HTTP API.

The fix considered for this, over the MQTT API, was for the agent to restore the original entity registration message if the cleared (deleted) entity had any children that are not deleted yet. So, the clear message would be considered as a delete request only for "leaf" entities. But yeah, this clearly is a breaking change and it would introduce yet another point of divergence from the HTTP API.

Or, for clients who for some reason can't/won't use HTTP, we could consider providing additional methods for registering/updating entity metadata under some new te/device/d_id/service/s_id/meta topic.

This was proposal B. But, introducing a new channel for registration is a strict no-go as we will have to support registrations/deregistrations over the existing te/+/+/+/+ topic anyway for backward compatibility.

[reubenmiller]

I'm having my doubts here about just switching to using the Cumulocity REST API to do device creation. Mostly because we don't have any good buffering mechanism when the device is offline.

[reubenmiller]

I'm wondering if we could take a slightly different approach to the ordering issue on the cloud side, e.g. what do we need to do to ensure the cloud device is created properly regardless of sending a registration message, telemetry, or digital twin info?

[albinsuresh]

Had considered that option, where a placeholder entity is created first, followed by updating it when the actual registration message arrives. The main hindrance for this approach is the external id. Since there aren't any MQTT APIs available to update the external ID, that was a dead end. The only other option was HTTP API. But there isn't an atomic PUT API available to update the external ID either. It will be a multi-step process with a POST followed by a DELETE. Since both solutions require HTTP, there isn't really much difference as some buffering solution for offline handling will be required either way.

[reubenmiller]

In newer Cumulocity versions there should be an atomic creation of a managed object along with an external id, see Cumulocity API. I'm not sure if we could use it there since we still need to support older Cumulocity versions, however it will give us a future simplification.

Below shows an example how to use the new api to create a managed object with a given external id in the same call, where if the external id already exists, then the managed object will not be created.

$ c8y inventory create --template '{externalIds:[{externalId:"rmi_test_exid_001",type:"c8y_Serial"}]}' --name rmi_test_exid_001

Then when trying it create it again, it will return a HTTP status code 409 (conflict/duplicate):

$ c8y inventory create --template '{externalIds:[{externalId:"rmi_test_exid_001",type:"c8y_Serial"}]}' --name rmi_test_exid_001

{
  "error": "identity/Duplicate",
  "info": "https://cumulocity.com/api/core/",
  "message": "ID [type=c8y_Serial, value=rmi_test_exid_001] was already bound to different Global ID."
}

[albinsuresh]

That's a good addition on the C8Y side, which we can use to create the device with the initial external id when an entity is registered.

But just to be clear, the problem that I was highlighting in my previous commit wasn't about providing external id during creation. I was talking about the case where an entity registration message is sent with an explicit @id to be used as the external id followed by data messages, but the mapper receives its data messages first. If we go with the approach of creating a provisional entity first for that data message, only to correct its metadata when the actual registration message arrives, we'll be forced to generate an external id for that provisional entity first (derived from the topic scheme) and then later change it to the one provided in the registration message. This can't be done over MQTT but only using REST APIs (multiple API calls).

[rina23q]

I'm definitely in favor of Proposal B (former Proposal C) — my comments argue it deserves as deep an exploration as Proposal A, and I've sketched a refined version in the Proposal B section. Take this as a friendly counter-proposal: everything else in the document (issue catalogue, A.2–A.7) has my full support.

[rina23q]

Isn't it rather a problem of our c8y-mapper implementation? We can first send unknown status for initial creation. Then, update the status either up or down if it's known. I think it is more reasonable to decouple "registration" and "status update".

status can be up, down, unknown, or any custom status
https://cumulocity.com/docs/device-integration/fragment-library/#announcing-a-service-to-the-platform

[albinsuresh]

Yup. That's the default approach proposed in A.7. The rest of A.7 is about other enhancements like support for initial status in the registration message itself, if some users don't want to see the unknown to up transition at all.

[rina23q]

Agreed — ordering is the same in both proposals, since it comes from MQTT's per-topic ordering, not from the topic design. But then it should not count as a reason to reject Proposal B. Note also that per the A.4 caveats, Proposal A alone even makes the window a bit worse, because the marked republish always comes after the raw publish.

[rina23q]

Another big advantage: a separate source-of-truth channel can be protected by a broker ACL, only tedge-agent can publish to it. In Proposal A this is impossible, because the same topic must stay writable by all clients for registration. Constraint 6 says bad writers should be handled by broker ACLs, but actually only Proposal B makes that possible for the truth topic.

[rina23q]

I agree clearing is not acceptable. But then I don't understand why the remaining duplication is a serious drawback. If the old topic is treated as input only, it does not matter whether a message stays there or not:

• Migrated consumers read only the new truth channel, so the leftover copy on the old topic does not affect them.
• Legacy consumers get exactly today's behaviour: same view, same known problems. No regression.
• The retained input even keeps one useful property: a registration published while tedge-agent is down is still delivered when the agent comes back.

So the real cost of "B without clearing" is documentation ("publish here, read truth there"), not a technical problem. I think this bullet should say concretely what harm the duplication causes. Probably the storage cost, but is it really that bad? It is only one extra retained copy per entity (Proposal A also publishes each registration twice anyway).

[albinsuresh]

Legacy consumers get exactly today's behaviour: same view, same known problems. No regression.

Agree. That's the true backward compatibility where existing users are fully unaffected, though they'd continue suffering from today's limitations.

The retained input even keeps one useful property: a registration published while tedge-agent is down is still delivered when the agent comes back.

We don't necessarily need retained for that. Just the persistent session of the agent would be sufficient.

I think this bullet should say concretely what harm the duplication causes. Probably the storage cost, but is it really that bad? It is only one extra retained copy per entity (Proposal A also publishes each registration twice anyway).

Having two copies itself is the disadvantage IMO. Yes, it'd be just a few 100 messages or a few 1000s in larger fleets. The storage impact might be minimal as these are small JSON records. But still, it's unnecessary wastage of resources, which is why I was insisting on the cleanup. But that comes at the cost of breaking backward compatibility as well. So, that duplication might be the cost that we must pay to maintain backward compatibility.

[rina23q]

We don't necessarily need retained for that. Just the persistent session of the agent would be sufficient.

True. And that's actually nice. It means the duplication is not permanent in principle. Once backward compatibility is dropped in some future major version, requests can become non-retained (and the leftover retained copies safely cleaned up).

So, that duplication might be the cost that we must pay to maintain backward compatibility.

Agreed, and the constraint comes from today's design.

[rina23q]

Consumers must migrate in Proposal A too: every consumer has to parse each payload and check the @source marker just to know if the message is trustworthy. With a separate channel, the topic itself is the filter — the broker does the work, no payload inspection needed. So the migration cost is the same or higher in A.

[albinsuresh]

Agree. Though the topic doesn't change, that alone doesn't provide backward compatibility to the user as they'll still have to update their logic to accommodate the @source marker.

[rina23q]

Note: What I said Proposal C (new output channel: registration stays on te/+/+/+/+ exactly as today.) is renamed to Proposal B in this commit. 7eb4597

[rina23q]

I prefer this proposal (the former Proposal C), and I'd like to propose a refined version:

1. Registration stays on the existing te/+/+/+/+, unchanged. Retained is not required for requests, but still accepted (and useful: a registration published while tedge-agent is down is delivered when it comes back). The retained state of this topic just carries no authority anymore — it is input, not truth.
2. tedge-agent subscribes, validates, and on success publishes the normalized entity retained to a result channel (rejections go to te/errors). The agent's entity store stays the real source of truth; the result channel is its projection for MQTT consumers.
3. The result channel is ACL-protected: only tedge-agent may publish, everyone else subscribes. No client can overwrite validated state — enforced by the broker, not by convention.

With this, the subscription itself becomes the trust filter — no payload inspection, no marker, no restore/clear machinery, and it works even while the agent is down.

To be clear, this is not the earlier idea of a new channel for registration (former Proposal B): registration stays where it is; only the validated output gets the new channel. So constraint 1 holds fully, and legacy consumers of te/+/+/+/+ see today's behaviour unchanged.

[albinsuresh]

You've raised some really convincing arguments to make me consider Proposal B as the preferred solution.

The argument that Proposal A is more backward compatible than Proposal B doesn't hold true for the reasons you highlighted: re-using the same topic alone doesn't guarantee backward compatibility in this case, as the new contract also forces existing clients to update their logic handling updated messages published back to that topic. I made a wrong assumption that most users would only be publishing to those topics and not subscribed to the same. But I myself have contradicted on that point where publishing back to the same topic was also done to keep existing subscribers happy.

Maintaining a separate tedge-agent owned retained topic definitely feels like the cleaner approach now, with zero impact to existing clients that might already be subscribed to the te/+/+/+/+ topics. So, the only 2 outstanding points are:
1. Duplication of data: do we care? I'd say yes, we care, but I guess this is the cost we must pay for backward compatibility. There's no other way around it.
2. Can we clear the duplicate data on the request channel? Unfortunately no, as it'd affect all existing clients subscribed to those topic and might interpret that clear message as an entity deletion request.

[rina23q]

Thanks for reconsidering, much appreciated!

Point 1: As discussed in this thread, the duplication is not permanent. It goes away once the old contract is eventually dropped.

Point 2: Good point. An empty retained message on the input channel is a delete request in today's API, so we cannot use it just to remove the duplicated data. Legacy subscribers could even propagate it as an entity deletion. It's worth capturing in the doc when revising.

[didier-wenzek]

These specs are difficult to grasp. Not only because they are lengthy, but mainly because they are focused on mechanisms without a clear data model nor user points of view. What is a device registration? Why a registration can be rejected by the agent? Shouldn't a device be in a better position than the agent to provide its registration data? What is auto-registration? How auto-registration interacts with registration? Why @id cannot be changed, while @parent can? What is the relationship between registration metadata and twin data?

• If I understand correctly, the main difference between proposal A and B is where the sanitized registration messages are published to. On the same topic with a source field added by the agent for A, on a second topic owned by the agent in the case of proposal B.

[didier-wenzek]

Not sure that the following is listed some where:

• name and type are published on two independent topics as entity registration or twin data

[albinsuresh]

Not captured separately as name and type are just twin data and these 2 were the primary ones for which we introduced the "inline twin data in registration message" for, so that the name of the device is set properly right from the start. I agree that it's worth capturing that point separately.

[didier-wenzek]

I don't see how a mix of protocol (HTTP + MQTT) can remove ordering problem entirely.

Without a version attached to registration messages and snapshots, how a late joiner can detect that a registration message has to be applied or not to its current snapshot?

[didier-wenzek]

Sorry, but to me, that sounds like gibberish.

[albinsuresh]

Yeah, I get that from an external reader's pov. The sophisticated sounding gibberish has piled up over time, after multiple back-and-forth conversations with the AI, to a point where I had to explicitly prompt it to "simplify the language" right before this final commit. A few like this still slipped through. I'll do another pass.

[didier-wenzek]

Is this one-shot registration-with-twin really convenient?

As a user I would prefer an API that let me see what I SET with a subsequent GET, over an API that let me save a request by conflating two concepts.

[albinsuresh]

The name and type fragments were the primary drivers behind this requirement, where this would let the C8Y mapper create the cloud entity with the "right name and type" right from the start rather than using provisional values first, which gets corrected later. But, since we have considered the notion of provisional entities as a solution for ordering problems, may be it isn't such a big deal to "correct things eventually".

[albinsuresh]

These specs are difficult to grasp. Not only because they are lengthy, but mainly because they are focused on mechanisms without a clear data model nor user points of view. What is a device registration? Why a registration can be rejected by the agent? Shouldn't a device be in a better position than the agent to provide its registration data? What is auto-registration? How auto-registration interacts with registration? Why @id cannot be changed, while @parent can? What is the relationship between registration metadata and twin data?

Fair criticism. I didn't include a detailed section on the data model because neither proposals were fundamentally changing the existing data model concepts like registration, auto registration, twin metadata etc. The focus was on reshaping the API to addressing the catalogue of issues that have been identified, without fundamentally changing the core concepts maintaining backward compatibility.

However, you've highlighted a major gap: though concepts like registration and auto-registration were documented in the original specs, several questions that you've raised above like the "difference between registration metadata vs twin data" or the "relevance of @id in the entity store" aren't clearly documented anywhere. Over time, these concepts got defined purely through an 'API-driven' lens, which has muddled our core data model. So, yes, I agree that we should address those gaps at least in this round.

• If I understand correctly, the main difference between proposal A and B is where the sanitized registration messages are published to. On the same topic with a source field added by the agent for A, on a second topic owned by the agent in the case of proposal B.

Yes.