keys show up, but no data

[numonedad]

Trying to load any metrics into metrics explorer shows a "no data for this time range" in the graph. The metrics are being pushed as I see metrics ingress bytes count is >0. Is there some way to debug the metrics actually being processed and viewable in metrics explorer? One special thing to note, I do not use GKE. I have to untoggle 'show active' to find these metrics keys.

Here's my prometheus config

global:
  scrape_interval: 15s
  evaluation_interval: 30s
  # scrape_timeout is set to the global default (10s).

  external_labels:
    monitor: codelab
    foo: bar

scrape_configs:
  - job_name: thegraph
    metrics_path: /metrics
    static_configs:
      - targets:
          - thegraph:8040
        labels:
          _generic_location: us-central1-a
          _generic_namespace: thegraph
  - job_name: stackdriver_sidecar
    metrics_path: /metrics
    static_configs:
      - targets:
          - stackdriver:9091
        labels:
          _generic_location: us-central1-a
          _generic_namespace: stackdriver

[mzwennes]

I am seeing the same issue on a GKE cluster. I see a list of inactive metrics in Cloud Monitoring, but neither of them contain any data.

Relevant Prometheus config (metrics show up in Prometheus):

scrape_configs:
- job_name: serviceMonitor/ingress-nginx/ingress-nginx-controller/0
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s

Versions used:

• Stackdriver Prometheus Sidecar - gcr.io/stackdriver-prometheus/stackdriver-prometheus-sidecar:0.10.0
• Prometheus - Tested with 2.30 and 2.19.0 (to match with compatibility matrix in the README)

Args passed to Stackdriver Prometheus Sidecar:

"--stackdriver.project-id=${var.gcp_project_id}",
"--prometheus.wal-directory=${var.data_dir}/wal",
"--stackdriver.kubernetes.location=${var.gke_region}",
"--stackdriver.kubernetes.cluster-name=${var.gke_cluster_name}",

Stackdriver Prometheus Sidcar logs output:

│ level=warn ts=2021-11-12T10:10:28.769Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = Field timeSeries[35].points[0].distributionValue had an invalid value: Distribution |explicit_buckets.bounds| does not have at least one entry."                        │
│ level=warn ts=2021-11-12T10:10:58.664Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = Field timeSeries[35].points[0].distributionValue had an invalid value: Distribution |explicit_buckets.bounds| does not have at least one entry."                        │
│ level=warn ts=2021-11-12T10:11:28.688Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = Field timeSeries[35].points[0].distributionValue had an invalid value: Distribution |explicit_buckets.bounds| does not have at least one entry."                        │
│ level=warn ts=2021-11-12T10:11:58.785Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = Field timeSeries[35].points[0].distributionValue had an invalid value: Distribution |explicit_buckets.bounds| does not have at least one entry."                        │
│ level=warn ts=2021-11-12T10:12:28.593Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = Field timeSeries[35].points[0].distributionValue had an invalid value: Distribution |explicit_buckets.bounds| does not have at least one entry."                        │
│ level=warn ts=2021-11-12T10:12:58.701Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = Field timeSeries[35].points[0].distributionValue had an invalid value: Distribution |explicit_buckets.bounds| does not have at least one entry."                        │
│ level=warn ts=2021-11-12T10:13:28.748Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = Field timeSeries[35].points[0].distributionValue had an invalid value: Distribution |explicit_buckets.bounds| does not have at least one entry."                        │
│ level=warn ts=2021-11-12T10:13:58.582Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = Field timeSeries[35].points[0].distributionValue had an invalid value: Distribution |explicit_buckets.bounds| does not have at least one entry."                        │
│ level=warn ts=2021-11-12T10:14:28.577Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = Field timeSeries[35].points[0].distributionValue had an invalid value: Distribution |explicit_buckets.bounds| does not have at least one entry."                        │
│ level=warn ts=2021-11-12T10:14:58.684Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = Field timeSeries[35].points[0].distributionValue had an invalid value: Distribution |explicit_buckets.bounds| does not have at least one entry."                        │
│ level=warn ts=2021-11-12T10:15:28.754Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = Field timeSeries[35].points[0].distributionValue had an invalid value: Distribution |explicit_buckets.bounds| does not have at least one entry."