spec: support Cumulocity's service commands

[rina23q]

Proposed changes

Initial proposal to support Cumulocity's Service Commands feature.

Types of changes

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Improvement (general improvements like code refactoring that doesn't explicitly fix a bug or add any new functionality)
• Documentation Update (if none of the other choices apply)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)

Paste Link to the issue

#4242
Roadmap: thin-edge/thin-edge.io-roadmap#98

Checklist

• I have read the CONTRIBUTING doc
• I have signed the CLA (in all commits with git commit -s. You can activate automatic signing by running just prepare-dev once)
• I ran just format as mentioned in CODING_GUIDELINES
• I used just check as mentioned in CODING_GUIDELINES
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)

Further comments

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[albinsuresh]

A good first draft. Though some of the design aspects didn't feel natural on an initial read, the latter parts of the doc explains why they are designed like that. For example, things like the relevance of the serviceType in the payload or the reason for limiting the service command support to a single service_command type. I feel, moving those explanations closer to where those contracts are introduced for the first time would have made the review easier.

The parts that I didn't particularly like is this reliance on the serviceType in the payload. When tedge-agent, the owner of the entity store, is the execution engine as well, it should be able to fetch and provide the service type for the target entity to the workflow engine. That being said, AFAIK, the notion of type was never mandatory for a service entity. I believe we'll have to make it mandatory for every service that's interested in this feature.

On a side note, it'd be nice if we use semantic line breaks in markdown docs like this, so that review comments can be added to specific lines. I had added an instruction for the same in the AGENTS.md file in this PR: https://github.com/thin-edge/thin-edge.io/pull/4233/changes#diff-a54ff182c7e8acf56acfd6e4b9c3ff41e2c41a31c9b211b2deb9df75d9a478f9. I guess I should have merged that change indepedenntly.

[albinsuresh]

Why did we pick a different topic model compared to the device's command support where the topic itself captures the command type as cmd/<cmd-type>? How is a service_command different from a standard cmd? The target being a service is implied in the entity type that it is attached to, right?

[rina23q]

The proposed service_command is a standard command, following the usual te/<entity>/cmd/<operation>/<cmd-id> scheme.

I didn't propose to carry an action inside <operation> like ../cmd/restart or ../cmd/service_start because:

• Every service needs to declarec8y_SupportedServiceCommands. It is similar to supported log/config types. For config/log, we use the "types": ["..."]" pattern.
• For a single workflow to react as you answered in spec: support Cumulocity's service commands #4243 (comment).

[reubenmiller]

And I wouldn't say it necessarily differs from the software_update command which can both install and remove software.

[albinsuresh]

I was purely looking at it from an thin-edge entity perspective where both devices and services are just entities. So, when a device entity's restart command support is expressed as te/<device>/cmd/restart, why doesn't the service, which is also yet another kind of entity, express it as te/<service>/cmd/restart. That would have been the natural expectation, but I understand that the primary motivation for this shape came from the desire to have a single workflow manage all service commands.

[rina23q]

te/device/<device>/service/<service>/cmd/restart is still valid and mapped to c8y_Restart by the c8y mapper. Now we need support for c8y_ServiceCommand where the command is included in the payload "command": "RESTART".

[albinsuresh]

te/device//service//cmd/restart is still valid and mapped to c8y_Restart by the c8y mapper.

Sorry for pressing on this, but this exactly was the source of my original confusion: what's the difference between te/device/<device>/service/<service>/cmd/restart and te/device/<device>/service/<service>/cmd/service_command {"command": "restart"}? Purely from a thin-edge perspective (not from the mapper's).

I understand that the c8y service concept differentiates the way commands are defined for devices vs services. I don't know why they chose that approach. Just trying to see if the same approach has a justification from the thin-edge perspective as well. Of course we have the freedom to choose whatever we want from a wide spectrum of choices (cmd/restart or service_cmd/restart or cmd/service_command). In my view, the first one was the most natural representation (consistent with the device entities). When we deviate from that, there must be a clear reason for that, right? Beyond how c8y or any other cloud has designed the same. I'm trying to get that cloud-agnostic thin-edge perspective out.

While reading the rest of the doc, the only reason that I could find (beyond inheriting the same shape as C8Y), was the convenience of having a single workflow file for service_command type. But for that reason alone, the command API need not take this form. We could still have separate commands like te/device/<device>/service/<service>/cmd/restart, te/device/<device>/service/<service>/cmd/reload etc and have an agent level contract that all commands aimed at services would be delegated to a dedicated service_command workflow or multiple service-specific workflows captured under /etc/tedge/operations/services.

[reubenmiller]

I would read cmd/restart as cmd/reboot, then we don't have an issue. The c8y_Restart is more of a reboot command, and sending a reboot to a service doesn't make as much sense as a classic restart action. So I wouldn't view the two commands as being the same.

Since all service related commands like enable/disable/start/stop/restart, having them under one command type is useful as it would help us limit the commands being executed in parallel to have more predictable behaviour.

[didier-wenzek]

I'm aligned with Albin here. Would the plan to only support start/stop/restart commands for services, a generic service_command would make it. However, I understand that the idea is to let users extend the set of command supported by a service. Having then an independent workflow would make more sense from a user perspective and would also introduce less specific cases in the engine.

[rina23q]

@albinsuresh's

Just trying to see if the same approach has a justification from the thin-edge perspective as well.

I admit that this proposal is heavily influenced by the Cumulocity specification, especially since our initial motivation was to support Cumulocity's service commands. From the thin-edge perspective, it is critical that we keep our core concept of being cloud-agnostic.

So, in terms of topic schema, indeed, te/device/<device>/service/<service>/cmd/<start|restart|stop|custom> sounds more natural as a thin-edge API.

I want to highlight that a workflow definition should be totally different depending on the target type, device or service. The command naming collision must be avoided (especially restart). If we go with the service_command model, this naming collision is guaranteed not to happen.

Note: if we go with the per-command model, the c8y mapper somehow has to aggregate all the service command capabilities and send them as c8y_SupportedServiceCommands in one shot. This will be significantly more complex compared to the service_command model.

[albinsuresh]

I would read cmd/restart as cmd/reboot, then we don't have an issue. The c8y_Restart is more of a reboot command, and sending a reboot to a service doesn't make as much sense as a classic restart action. So I wouldn't view the two commands as being the same.

I was of the opinion that the meaning of restart must be left to the entity. So, for devices, it's a reboot and for a child device or service, it could be something else.

Since all service related commands like enable/disable/start/stop/restart, having them under one command type is useful as it would help us limit the commands being executed in parallel to have more predictable behaviour.

Yeah, agree with that grouping aspect. But the part that I was not quite comfortable with was the idea of having a single workflow for everything: every service and all of its custom commands. That makes this "all-in-one" workflow very difficult to extend/override. Granular workflows would have been easier to manage.

I want to highlight that a workflow definition should be totally different depending on the target type, device or service. The command naming collision must be avoided (especially restart). If we go with the service_command model, this naming collision is guaranteed not to happen.

I think it's worth exploring the path of extending workflows to services based on their types as it gives a natural shape to this feature and also enables us to extend the same for child devices as well in future. I think I've come across this requirement in the past where some user was exploring the option to define workflows for child devices in the main device itself (when the child device is not capable of running its own tedge-agent). So, a device/service "type-based" workflows solution might get us both capabilities.

Note: if we go with the per-command model, the c8y mapper somehow has to aggregate all the service command capabilities and send them as c8y_SupportedServiceCommands in one shot. This will be significantly more complex compared to the service_command model.

We can take the same aggregation approach that we already have for different operations of the main device, right?

[albinsuresh]

Feels redundant. The topic already conveys that information and I'm wondering that the point of that serviceType field is, which is already part of the entity metadata.

[rina23q]

serviceName is redundant as it's duplicated as topic ID.
serviceType is very important. In this example, the value is service, but for example, the value for tedge-container-plugin is either container or container-group. It is a tick to distinguish init system managed service or other custom type.

[reubenmiller]

Whilst the "serviceName" might be redundant, it will however make it easier for commands to reference the information via the payload body rather than having to parse the topic.

[albinsuresh]

The serviceName must match the name twin data of the service or the name in the topic id (te/device/main/service/<service-name>)? I'm assuming it's the latter as the default topic scheme was a concrete requirement of this initial cut.

[rina23q]

Here I simply copied the serviceName value from the c8y operation payload, which reflects name of the service's MO.
To be honest, I'm not sure which value makes the most sense. I feel this field is not that important, as the topic already tells which entity is targeted. Subject to change during the implementation phase.

[albinsuresh]

That scheme seems to rely on the default topic scheme to precisely subscribe only to the services associated to the given device. But since the topic based relation between the device and its services only exists for the default topic scheme, it wouldn't work for devices following custom topic schemes, right?

[rina23q]

Correct. So, the goal here should be "how to filter services of my device" should be done based on the records of entity store.
However, for PoC or very first iteration, maybe it is not too bad to use topic scheme to filter out as it is easy for sure.

[albinsuresh]

For the PoC, yes, I wouldn't mind. But before the public release, we'll have to find a better way.

[albinsuresh]

Now I understand why you went with a fixed cmd/service_command type scheme rather than cmd/<any-command-type>: To have a single workflow to handle all service commands.

[didier-wenzek]

By contrast, having a single workflow to handle all service commands is precisely what bothers me.

Grouping start/stop/restart in a single command can make sense. But, if we want to add other service specific command we will have to implement a switch within the generic workflow while that switch is already implemented at the engine level - but blocked in the case of services.

[albinsuresh]

What's the type here? The serviceType value? Wouldn't that be too broad that covers multiple services with a value like systemd or container? We'd need a more granular mapping between the service and its corresponding plugin, right? I mean each systemd service might handle a similarly named operation differently.

[rina23q]

Yes, <type> is the serviceType value.
But note that the default value service is not a new, broad category introduced by this design. It's the value thin-edge already uses as the default serviceType for its service components in Cumulocity today (tedge-agent, tedge-mapper-c8y, ...). For exactly these services, the execution mechanism is already fully abstracted by system.toml: one wrapper covers every init system and every service.

The plugin dispatch just extends the same idea to other types: the type selects the control mechanism (docker, ...), not the per-service behavior — the plugin receives the service name (<plugin> <command> <name>) and the per-service semantics live in the service itself, exactly as with systemd units.

And if a service ever needs truly dedicated execution logic, serviceType is an open set: register it under its own type and ship a plugin for it.

[albinsuresh]

Suggested change

	| any other type | **service plugin**: `/etc/tedge/service-plugins/<type> <command> <name>`; the command (standard or custom) is passed through, the plugin decides what it supports |
	| any other type | **service plugin**: `/etc/share/tedge/service-plugins/<type> <command> <name>`; the command (standard or custom) is passed through, the plugin decides what it supports |

As per the convention used by other plugins.

[rina23q]

Correctly: /usr/share/tedge/... ;)

By the way, probably we should consider moving /etc/tedge/sm-plugin to /usr/share/tedge/sm-plugin at some point.

[albinsuresh]

How would tedge know what all custom commands this plugin supports? All the supported commands must be published to the cloud as well, right? It feels like we're missing a list command, unless there's some other mechanism that replaces that.

[rina23q]

No, list is not needed.
All other features (software management, config management, and log upload) indeed have a discovery model for their supported types. The key is who has to declare it. All the others are device-scoped, while here a service must declare its supported commands. That's why I chose the push-based design for the supported types. (= a service publishes to te/<entity>/cmd/service_command). Already a service registration requires a publishing an entity registration message.

[albinsuresh]

That's not a limitation of this operation alone. Even config management and log management operations suffers from the same, where a single config type can't not choose to bypass the workflow. It's all or nothing.

[albinsuresh]

Is the tedge service command integrated with the plugins as well? Or it's limited to the standard commands (start, stop, restart etc) only?

[rina23q]

Fully integrated — tedge service is the dispatcher, not just the system.toml wrapper.
--service-type service (the default) routes to system.toml, any other type delegates to /usr/share/tedge/service-plugins/<type>, and custom commands pass through in both cases. So, for example, user can run them by hand:

• tedge service stop tedge-mapper-c8y => using system.toml definition
• tedge service pause my-container --service-type container => calling /usr/share/tedge/service-plugins/container pause my-container

[albinsuresh]

So, if we want to add a reload command support to a systemd service, in addition to the standard commands defined in the system.toml, the expectation is to create a standalone plugin for that type and redefine the start, stop and restart commands as well? The system.toml won't be used anymore for that service, even for the standard commands, right?

[rina23q]

For init system commands, I expect adding the definition to system.toml.

However, the current system.toml implementation rejects unknown keys (InitConfigToml has deny_unknown_fields), so a small change is required to accept additional command definitions.

If a custom plugin is provided for the default type, that plugin takes over all its commands, and system.toml is no longer used for that service type.

[albinsuresh]

Now I understand why that serviceType is passed in the payload.

[albinsuresh]

On a second read, the relevance of serviceType in the payload is even more clear to me. As far as I understood, its primary purpose is the plugin identification. I'm just wondering if it'd make sense to keep the : suffixed format (<config-type>:<plugin-type>) as used by the software, config and log plugins here as well for consistency in the plugin contracts. The mapper can then extract that suffix as the serviceType when it is needed for cloud interactions.

A few suggestions on the doc:

1. An executive summary of the design at the beginning of the Design section itself would make it easier to follow the detailed design notes that follows. From this executive summary, what I'd expect is a hint of which components (mapper vs agent vs plugin) is responsible for which aspects of the message exchanges and which parts of the message payload are relevant for them.
2. One end-to-end example of the message flow between all the components (agent vs mapper vs plugin vs the service itself). If you can include the following variants in that example, it'd be even better:
   1. systemd services (managed via the system.toml) with the following variants to better understand how the command delegation is split between the system.toml (for the standard commands) and the plugin (for custom commands).
      1. One that supports only the standard commands (start, stop and restart) defined in the system.toml
      2. One tht supports the standard commands as well as a custom command.
      3. One that only supports custom commands and not the standard ones.
   2. A non-systemd service that supports the standard commands as well as custom commands. The tedge-container-bundle can be used for this example.

[Bravo555]

Very compelling reasons why central executor approach is superior, but also I think made a little bit harder by the MQTT scheme.

This mirrors somewhat the problems associated with the registration topics. The problems stems from the fact that the same command topics are written to and read from by multiple parties:

• tedge-mapper-c8y (which writes the operation to the topic with init status and then reads until its status is successful or failed)
• tedge-agent (which drives the command to completion under central executor approach, not a problem in practice because it plays well with the mapper and is always online)
• the service the command is addressed to (in the distributed executors approach, a bigger problem here because we don't control it, it could publish invalid payloads or be offline)

[Bravo555]

While a good way to handle permissions now, I'm afraid at some point we'll have to revisit the idea of using sudoers rules to give root permissions to tedge processes. It makes it harder to configure only necessary permissions, puts the entire burden on tedge to not do anything bad, and sudo is an SUID binary, which have problems, so much so that people are trying to replace sudo with something that isn't a SUID binary, like run0.

[rina23q]

Agreed it needs revisiting eventually. However, this proposal doesn't introduce anything new around how we manage sudo. Even no new sudoers entry is needed since /usr/bin/tedge is supposed to be shipped in sudoers file already. That's one advantage of introducing a new tedge sub-command.

[reubenmiller]

Since this is limited to the tedge service command, I would include enable and disable commands. That makes it easier to interact with the services via the command line.

[rina23q]

Agreed. enable, disable (and is_active) are already part of InitConfig, so exposing them through tedge service costs nothing on the config layer. I'll include them.

Going further, as I wrote in #4243 (comment), my ambition is to support any custom command for init-managed services: once system.toml accepts additional command definitions, whatever a user defines there (e.g. reload) becomes available through the same tedge service <command> <name> interface.

[rina23q]

@albinsuresh

I'm just wondering if it'd make sense to keep the : suffixed format (<config-type>:<plugin-type>) as used by the software, config and log plugins here as well for consistency in the plugin contracts. The mapper can then extract that suffix as the serviceType when it is needed for cloud interactions.

The :: suffix in config/log management exists because one entity (the device) has to declare a flat list of types from many plugins. (e.g. tedge-agent::journald)
Do we need a flat list here? I don't think so, because a service is already an entity. Its registration carries the type that routing needs and c8y has serviceType property natively, not only serviceName.

[rina23q]

Review feedback is addressed in three fixup commits:

1ce3e2e is format only: it reflows the doc with semantic line breaks as suggested, without changing any wording.
d0c4f85 and 6cc5642 the actual content updates: plugin path, enable/disable, system.toml schema note, service type sourcing, topic-scheme limitation, a design overview, and end-to-end examples.

[albinsuresh]

The main remaining concern is about who's responsible for registering all the services running on the device as tedge service entities. Do we expect all the services to register themselves? Or we expect the device admin to write wrappers for each and every service that'll do the registration? Or can we push this responsibility also to the plugin with a "discovery mode"?

[albinsuresh]

Does the tedge-container-plugin provide a mechanism for individual plugins to declare the custom commands that they support, so that container_1 can declare that it supports cmd_a and cmd_b while container_2 declares cmd_x and cmd_y?

[rina23q]

Currently no. But it should be technically possible, for example, using container labels.
Note that the plugin registers service entities via the HTTP API and publishes health status
per service via MQTT. So it can also publish one more retained capability message per container,
based on its labels.

[albinsuresh]

I was purely looking at it from an thin-edge entity perspective where both devices and services are just entities. So, when a device entity's restart command support is expressed as te/<device>/cmd/restart, why doesn't the service, which is also yet another kind of entity, express it as te/<service>/cmd/restart. That would have been the natural expectation, but I understand that the primary motivation for this shape came from the desire to have a single workflow manage all service commands.

[albinsuresh]

I was curious to see an example of a non-tedge non-systemd service that needs support for custom commands like backup.

Assume that we have that third-party service that accepts the backup signal via some interface (socket or HTTP or even MQTT). Who is responsible for the entity registration? I'm assuming that some wrapper process is responsible for the entity registration and reporting the supported service command types (unless we can modify the third-party code itself to do this, which is highly unlikely). Then there is the dedicated plugin cli as well, that gets invoked when the command is triggered. This plugin would be responsible for triggering that command via the service's dedicated interface (socket/HTTP/MQTT).

I'm not sure if I like this split process design required for each and every service type that supports some custom commands. When the service is a third-party service, i should have been easier for the device admin to externally define the thin-edge integration for that service, rather than expecting him to modify that service code or write wrappers over it. It would have been nicer if the plugin itself could do the registration and command type declarations as well and this is why I was feeling that a "plugin discovery phase" would be needed.

[rina23q]

Who is responsible for the entity registration? I

I feel this is unrelated to the service command design itself; it is more about our service entity registration model.

In today's registration design, a user or a component has to register a service (= push model). If a third-party component should exist in the thin-edge/c8y world, there is always someone who registers it. This proposal's capability declaration (service_command and its types) follows the same model. The same someone publishes one more retained message.

If a discovery model (= pull model) is needed for service registration, that should be addressed in your registration spec PR ;)

[albinsuresh]

I feel this is unrelated to the service command design itself; it is more about our service entity registration model.

It was not just about the service registration but sending the supported command types as well, which is a core part of this command support design. What I was trying to say was that, we should have an end-to-end picture on how a third-party service developer can integrate their service into the thin-edge ecosystem, with the proposed design.

For example, when we defined command support for different operations like software_update, config_management etc, we didn't just define the topic scheme to capture the supported software/config/log types. We also made the agent responsible for the "supported types metadata declaration" so that the third-part software developer could just focus on his software alone: just write the plugin that's all about his software/config/log type and does not care about the MQTT topic scheme.

So, I was trying to see if we can absorb more of that registration/supported service commands metadata publish responsibilities into the agent by extending the plugin surface or anything else. If not the registration, then at least the latter.

If a discovery model (= pull model) is needed for service registration, that should be addressed in your registration spec PR ;)

The registration spec PR can definitely define the service registration aspects, but going beyond that, like discovery of supported commands via plugins would be stretching it too far. I feel at least that aspect definitely belongs here.

Once we have that clear picture of what the user has to do with a given design, then we can easily try to pull as much responsibilities as well can into the agent. It is quite possible that we might end up saying that the registration and publishing the supported types is the user's responsibility. We just need to be sure that that's only way around, and agent couldn't do much more on their behalf.

[rina23q]

Ok, now I fully understand what you request here.

Because services are always registered by someone, I was standing on the side that whoever registers a service also knows which commands it supports, so it can push one additional capability message.

Now I was thinking how discovery (pull model) can work, and honestly it seems better. Let's use tedge-container-plugin and a nodered container as an example.

1. tedge-container-plugin ships /usr/share/tedge/service-plugins/container.
2. tedge-container-plugin finds nodered container and registers it (as it does today).
3. tedge-agent sees the registration message with the "type":"container". It checks: does /usr/share/tedge/service-plugins/container exists? Yes.
4. tedge-agent queries /usr/share/tedge/service-plugins/container list nodered (note: the subcommand name is just an example to explain the flow)
   and the plugin prints

start
restart
stop
pause

5. The agent publishes the service's capability on behalf of the plugin.
   patten A: one-command model

[te/device/main/service/nodered/cmd/service_command]
{ "types": ["start", "stop", "restart", "pause"] }

(By the way, I started feeling types here might be confusing with serviceType. Probably better to rename commands.)

pattern B: per-command model

[te/device/main/service/nodered/cmd/start] {}
[te/device/main/service/nodered/cmd/stop] {}
[te/device/main/service/nodered/cmd/restart] {}
[te/device/main/service/nodered/cmd/pause] {}

For init-system services, tedge-agent declares supported commands based on what defined in system.toml by default, however, we need to make exceptions for tedge-agent, mappers, and mosquitto. Otherwise the command can stop the connection to cloud completely.

I will update the doc later.

[albinsuresh]

The serviceName must match the name twin data of the service or the name in the topic id (te/device/main/service/<service-name>)? I'm assuming it's the latter as the default topic scheme was a concrete requirement of this initial cut.

[albinsuresh]

For the PoC, yes, I wouldn't mind. But before the public release, we'll have to find a better way.

[didier-wenzek]

Correct. Currently, the agent only processes operations addressed to its parent device.

However, this can be extended so the agent executes operation workflows on behalf of any service running on the same device (i.e. having the same parent).

[rina23q]

Yes, this proposal includes extending the agent that reacts to its own services.
Actually, I tried it before writing this proposal and the code change was very small.

[didier-wenzek]

Is the plan to let users define specific workflows for service operations?

Is the aim to trigger operations on daemon such as mosquitto?

[rina23q]

Is the plan to let users define specific workflows for service operations?

No, the plan is to ship a default workflow, but users can modify it. The principle stays the same as our existing config_update, log_upload and so on.

Is the aim to trigger operations on daemon such as mosquitto?

Yes, as long as the daemon is managed by init system (mosquitto is in this case), the workflow triggers the operation with the new tedge service command.

[didier-wenzek]

I don't understand the point with that section. Nothing seems to be wrong, but nothing seems useful either.

[rina23q]

To be honest, it's more like instructions to AI. Subject to remove.

[didier-wenzek]

I disagree with that argument. Matching c8y API and making the task easier for the c8y mapper is not a goal. Having a consistent approach for devices and services matters most.

[rina23q]

You are right. This is a dilemma of this feature. The requirement came from c8y. However, we need to be cloud-agnostic.

[didier-wenzek]

Yeah. But note that the agent has access to the entity store and that should not be an issue.

Actually this shortcut has been taken so often, that I'm afraid that custom topic schemes are not working at all.

[didier-wenzek]

This new command is a great addition.

But then, I don't understand why the plan is to have a single service_command operation. Using tedge service is would be easy to write specific workflows for each service operations.

[didier-wenzek]

This is even more serious: if stopped the agent cannot be restarted from the cloud. Same for the c8y mapper.

[didier-wenzek]

Stopping or disabling the c8y mapper from c8y would let the device disconnected for ever.
Starting the c8y mapper from c8y has no chance to work.

Suggested change

	{ "types": ["start", "stop", "restart", "enable", "disable"] }
	{ "types": ["restart""] }

However, both could make sense in a context where there are two cloud connections, one being used to control the other.

[rina23q]

Right. However, here I just wanted to put one example. So I will choose another service that doesn't kill cloud connectivities.

[didier-wenzek]

The current behavior of the agent is to ignore any unknown operation request. Specifically for that reason: that operation might be processed by another component.

[rina23q]

Yes. Actually I found this is an interesting point.

• one-command model (/cmd/service_command): we can fail an unknown service's command, because the workflow operation is targeted to service_command, which is "known".
• per-command model (e.g. /cmd/stop): have to ignore an unknown command, because the service's command == workflow operation.