[Bug]: Conflicting or ambiguous enforcement actions for a policy-entity combination lead to invisible processing failure

[wagnerand-moz]

Is there an existing issue for this?

• I have searched the existing issues

What happened?

AMO, and therefore Cinder, have a requirement that there must not be more than one enforcement action for a policy-entity combination. If there is, processing would silently fail (quoting @eviljeff: "there is a log in Cinder of all webhook activity, and it'd be marked as 4xx there, but no-one is routinely monitoring that"). The invisible failure is problematic.

At the same time, this requirement itself is implicit and invisible as well. Cinder will allow such conflicts to be created when changing a policy. There also is no policy list view (or otherwise) that would show or hint at that issue. There isn't even a way to see policy+entity -> enforcement action, not even a view to see the policy-entity mappings. Therefore we lack a method of preventing, detecting or auditing this issue.

The impact of that to happen is significant: Enforcement would be recorded in Cinder (I presume), but not recorded on AMO and also not actually carried out, resulting in not just de-synced information but actual moderation decisions not being applied, without anybody noticing.

What did you expect to happen?

I am not sure what exactly is feasible here.

Ideally we'd prevent a user from being able to make changes that would result in ambiguous enforcement action mappings. At least we need a way to detect it immediately when a user does it and raise very visible alerts and notifications. We also need to figure out what we'd do with decisions in that state. Can we hold them in some way, or potentially deny them...or something else?

┆Issue is synchronized with this Jira Task

[eviljeff]

Ideally we'd prevent a user from being able to make changes that would result in ambiguous enforcement action mappings.

Not possible - Cinder doesn't know the enforcement action mappings are ambiguous (I'm not clear on what is supposed to happen by design in their system if there are multiple enforcement actions - maybe they intend the actions to never overlap?)

At least we need a way to detect it immediately when a user does it and raise very visible alerts and notifications.

The earliest AMO would know that a policy had "invalid" enforcement actions would be when it was synced. (Unfortunately that command is asynchronous so if we did detect an error it couldn't be shown in-line on django admin)

We also need to figure out what we'd do with decisions in that state. Can we hold them in some way, or potentially deny them...or something else?

tbd. At the point we get a decision with multiple conflicting enforcement actions if AMO denies the decision, then the feedback to Cinder is limited to a 4xx, which as noted isn't routinely monitored anyway. (And that decision payload is asynchronous after the job has been closed in any case - AMO can't veto the decision)

[diox]

Another thing to consider: possibly having an alert in Redash or whatever ?

[diox]

Slightly related: #16258 is caused by the fact that we're rejecting a decision action that we don't allow to come from Cinder. The decision lands in 2nd level approval as a result. Maybe we could do something similar here (this should be a rare occurrence). The decision could be fixed by an override in Cinder (possibly in AMO as well if we fixed that).

[diox]

Discussed in Operations office hours: we want to add better documentation/runbooks to avoid that situation, and in parallel add safeguards around that, probably using the method highlighted above to send such decisions to 2nd level approval queue.