AggregatorRegistry Metrics Error - Operation Timed out

[austonpramodh]

I am trying to integrate prom client into a cluster-enabled project. Unfortunately, I don't get any response from the Aggregator endpoint. I do get responses from the "/metrics" from the worker nodes, but when I hit the "/cluster_metrics" I get an error saying "Operation Timed out".

I would like to understand how the aggregator gets the metrics from the works, This might help me understand if I am doing and mistakes.

Master Server

Worker Server - Already runs an HTTP server, therefore created a handler.

Error

[austonpramodh]

I found the solution for this, So looking at the ClusterMtrics file, It uses even emitter to collect the metrics from the workers. Therefore the AggregatorRegistry class needs to be instantiated on workers as well which I wasn't doing, I was just instantiating it in the master process.

Thanks. Closing this issue.

[zbjornson]

This was actually a mistake in a recent release, see #464. I haven't had any time to work on prom-client to address it, but you found the workaround.

[SpComb]

We ran into the same issue as a regression, where cluster metrics stopped working after an upgrade from 13.1.0 -> 14.0.1.

Perhaps this issue should be re-opened, as #464 has not been merged, and there isn't yet any released version that fixes this issue?

[zbjornson]

Since so much time has passed and since a few releases have been made since the regression, I'm going to try to find a non-breaking fix that allows either usage pattern.

[morgaan]

I hoped that the latest major version (15) update would address that. It's not. Feel like I'm stuck with 13.1.0 for another while.
I thought using cluster was kind of the norm. If this is, that means that the breaking change introduced in 13.2.0 affects a lot of people and may deserve a bit of attention 🤷. I believe the fix lives in this PR: #464

[shirady]

Hello,
we're using version "prom-client": "15.1.3" and facing the error Operation timed out..
In our case, we instantiated the AggregatorRegistry in the master and all it's a workers (it happens at the beginning of a module loaded in each one of them).

We saw a case where we've got this error and we suspect that it happens when one of the workers is loaded and could not respond on time, we would like to suggest configuring the timeout:
https://github.com/siimon/prom-client/blob/c1d76c5d497ef803f6bd90c56c713c3fa811c3e0/lib/cluster.js#L54-L62
We can use the 5000 milliseconds as a default value for the argument.

Another thing that we asked ourselves while trying to read the code in lib/cluster was:
What happens when a fork exits and is recreated during that time?
https://github.com/siimon/prom-client/blob/c1d76c5d497ef803f6bd90c56c713c3fa811c3e0/lib/cluster.js#L70-L76
In this part we can see that the master is sending the messages to its connected forks - what happens if a fork was exited and recreated between the time the message was sent and was waiting to listen to it?

What happens if one of the workers disconnects? (closes the IPC channel)
https://nodejs.org/api/process.html#event-disconnect

It can be helpful if you could share your experience on how to investigate the error more.