What we observed
A monitor created by a single check-in whose body carried
monitor_config.failure_issue_threshold: 2 alerted on one error check-in. The
notification read "An error check-in was detected" — singular.
Minutes earlier, in the same project and through the same alert rule, an
already-existing monitor with the same threshold sent two errors and produced
"2 error check-ins detected".
The only difference between the two is that one monitor already existed, and the
other was created by the check-in that carried the config.
What we ruled out
We first assumed the stored config was the default (1) at creation and the payload's
value applied later. That is not what happens. Two probes, each with a fresh slug
confirmed 404 immediately before creation, each created by a single check-in:
- the monitor did not exist for 2.58 s after the ingest endpoint returned
202
- the very first successful read already showed
failure_issue_threshold: 2
So there is no observable interval in which the monitor exists carrying the default.
The stored configuration is correct from the first moment it is readable.
What we think is happening
The creating check-in appears to be evaluated for issue creation before, or
independently of, the monitor it creates — so the threshold that is correctly
stored is not the one applied to that first check-in.
This looks adjacent to prior reports rather than new:
One more observation, unexplained
A third probe received 202 from the ingest endpoint and was never created —
40 reads over ~10 s all returned 404, and it did not exist minutes later. An
identical request seconds afterwards succeeded. Observed once; we did not
investigate further.
What we have not done
We have not sent a deliberate error check-in to a newly upserted monitor to
confirm the alerting path, because doing so pages an on-call channel. Everything
above is either a config read or an incidental observation.
Question
Is the first check-in that upserts a monitor expected to be evaluated against that
payload's failure_issue_threshold? If not, is that ordering documented anywhere —
we could not find it in the crons docs.
What we observed
A monitor created by a single check-in whose body carried
monitor_config.failure_issue_threshold: 2alerted on one error check-in. Thenotification read "An error check-in was detected" — singular.
Minutes earlier, in the same project and through the same alert rule, an
already-existing monitor with the same threshold sent two errors and produced
"2 error check-ins detected".
The only difference between the two is that one monitor already existed, and the
other was created by the check-in that carried the config.
What we ruled out
We first assumed the stored config was the default (1) at creation and the payload's
value applied later. That is not what happens. Two probes, each with a fresh slug
confirmed 404 immediately before creation, each created by a single check-in:
202failure_issue_threshold: 2So there is no observable interval in which the monitor exists carrying the default.
The stored configuration is correct from the first moment it is readable.
What we think is happening
The creating check-in appears to be evaluated for issue creation before, or
independently of, the monitor it creates — so the threshold that is correctly
stored is not the one applied to that first check-in.
This looks adjacent to prior reports rather than new:
groups
mark_failedracing so the status guard is bypassed.Cited as the closest prior description of the shape, not as an open defect.
One more observation, unexplained
A third probe received
202from the ingest endpoint and was never created —40 reads over ~10 s all returned 404, and it did not exist minutes later. An
identical request seconds afterwards succeeded. Observed once; we did not
investigate further.
What we have not done
We have not sent a deliberate
errorcheck-in to a newly upserted monitor toconfirm the alerting path, because doing so pages an on-call channel. Everything
above is either a config read or an incidental observation.
Question
Is the first check-in that upserts a monitor expected to be evaluated against that
payload's
failure_issue_threshold? If not, is that ordering documented anywhere —we could not find it in the crons docs.