> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sambanova.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshoot monitoring

Work through this page when an install completes but produces no data, or when a device stops reporting.

## Where to start

| Symptom | Section |
| :- | :- |
| A correct install returns no metrics at all | [No data in Prometheus or Grafana](#no-data-in-prometheus-or-grafana) |
| `pdu_up 0` or `xrdu_up 0` for a device | [Device reachability](#device-reachability) |
| Some XRDU sensors report and others do not | [XRDU probe failures](#xrdu-probe-failures) |
| No RDU metrics, but PDU and XRDU work | [The SambaNova exporter is not reporting](#the-sambanova-exporter-is-not-reporting) |
| Dashboards are missing or empty | [An empty dashboard](#an-empty-dashboard) |
| Rack power reads about three times too high | Filter on `level="total"`. See [PDU exporter labels](exporter-pdu#labels). |
| Pods will not start | [Pods and images](#pods-and-images) |

## No data in Prometheus or Grafana

Check these in order. The first two cause almost every case.

### 1. The probe was never created

```bash theme={}
kubectl -n monitoring get probe,servicemonitor
```

`<exporter>.enabled` deploys the exporter. `<exporter>.probe.enabled` makes Prometheus scrape the devices. With only the first, the exporter runs and produces nothing. Confirm what Helm actually applied:

```bash theme={}
helm -n monitoring get values hwmon
```

### 2. Prometheus is not selecting the probe

This is the most common reason a correct install produces no data. A `Probe` or `ServiceMonitor` without the label your Prometheus selects on is created successfully and then never scraped — with no error anywhere.

```bash theme={}
kubectl get prometheus -A -o jsonpath='{.items[*].spec.probeSelector}'; echo
kubectl get prometheus -A -o jsonpath='{.items[*].spec.serviceMonitorSelector}'; echo

kubectl -n monitoring get probe -o jsonpath='{.items[*].metadata.labels}'; echo
```

The labels must match. `kube-prometheus-stack` selects on `release: <its-release-name>`. An empty selector (`{}`) selects everything. Fix by setting `probe.labels` and `serviceMonitor.labels` in your values file and upgrading.

### 3. The target is down

Open the Prometheus UI and go to **Status**, then **Targets**. Each enabled `Probe` appears as a job with one target per device. A target listed as down carries the scrape error.

A target that does not appear at all means Prometheus is not selecting the probe. Return to step 2.

## Device reachability

An exporter pod reaches a PDU or BMC only if the node it runs on has a route to the management network. Nodes usually carry a front-end and a management address both, but not always, and where they do not, no chart setting produces a metric.

Run these checks **from a cluster node**, not from your workstation. List your own device addresses in each loop. The second loop prompts for the XRDU BMC password so it stays out of your shell history.

```bash theme={}
for ip in <pdu-1-ip> <pdu-2-ip>; do
  code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 10 \
    "http://$ip/api/sys" 2>/dev/null)
  echo "PDU  $ip -> http=${code:-none} curl_exit=$?"
done
```

```bash theme={}
read -rs -p "XRDU BMC password: " XRDU_PASSWORD; echo

for ip in <xrdu-0-ip> <xrdu-1-ip>; do
  code=$(curl -sk -o /dev/null -w '%{http_code}' --max-time 10 \
    -u "root:$XRDU_PASSWORD" \
    "https://$ip/xyz/openbmc_project/sensors/enumerate" 2>/dev/null)
  echo "XRDU $ip -> http=${code:-none} curl_exit=$?"
done

unset XRDU_PASSWORD
```

| Result | Meaning | Fix |
| :- | :- | :- |
| `http=200` | The node can reach the device. Two or more seconds is normal for these devices. | — |
| `http=401` | Wrong XRDU password | See [Update the XRDU password](#update-the-xrdu-password) |
| `http=000` with a non-zero `curl_exit` | No route. Wrong cluster, or the node has no interface on the management network. | Pin the exporter pods to nodes that do, with `nodeSelector` |

Both `-k` and `-u` are required for an XRDU. The BMC uses a self-signed certificate and requires authentication; a plain `curl https://...` fails on the certificate in a way that looks like the device being down.

If some nodes have a route and others do not, pin the exporters:

```yaml theme={}
pdu-exporter:
  nodeSelector:
    <label-that-marks-management-network-nodes>: "true"
xrdu-exporter:
  nodeSelector:
    <label-that-marks-management-network-nodes>: "true"
```

<Note>
  An unreachable device returns HTTP 200 with `pdu_up 0` or `xrdu_up 0`, not an error status. This is deliberate: a non-200 response would make Prometheus discard the whole scrape, including the `up` sample your outage alert depends on.
</Note>

## XRDU probe failures

For an XRDU, `xrdu_up 0` covers a rejected credential, a certificate problem, and an unreachable device alike. The log distinguishes them.

```bash theme={}
kubectl -n monitoring logs deploy/xrdu-exporter | grep 'probe failed'
```

| Log or symptom | Cause | Fix |
| :- | :- | :- |
| `x509: negative serial number` | Exporter older than 0.1.1 | Use image 0.1.1 or later |
| `401 Unauthorized` | Wrong password | See [Update the XRDU password](#update-the-xrdu-password) |
| `dial tcp ... i/o timeout` | The node cannot route to the BMC network | See [Device reachability](#device-reachability) |
| `response was not the JSON envelope` | Not an XRDU BMC at that address | Check the address against `snctl rack show` |
| Pod `CreateContainerConfigError` | The Secret has no `XRDU_PASSWORD` key | Recreate the Secret with the correct key names. See [Update the XRDU password](#update-the-xrdu-password). |

### Update the XRDU password

The exporter reads the password from the Secret only when its pod starts. After you change the Secret, restart the exporter.

Read the current password where SambaRack Manager is installed:

```bash theme={}
snctl secret get --device-type xrdu --show-value
```

Then, on the host with `kubectl` access, replace the Secret and restart the exporter:

```bash theme={}
read -rs -p "XRDU BMC password: " XRDU_PASSWORD; echo

kubectl -n monitoring create secret generic xrdu-creds \
  --from-literal=XRDU_USERNAME=root \
  --from-literal=XRDU_PASSWORD="$XRDU_PASSWORD" \
  --dry-run=client -o yaml | kubectl apply -f -

unset XRDU_PASSWORD

kubectl -n monitoring rollout restart deploy/xrdu-exporter
```

## The SambaNova exporter is not reporting

This exporter is a host process. The chart does not deploy it and cannot start it.

```bash theme={}
kubectl -n monitoring get pods -l app=sambanova-exporter -o wide
kubectl -n monitoring get endpoints sambanova-exporter -o yaml | grep -E 'addresses|notReady'
```

A pod that is Running but never Ready, or an endpoint under `notReadyAddresses`, means the host service is down on that node. The readiness probe follows the host process deliberately, so a dead exporter drops out of the Service instead of reporting zeros.

On that host module:

```bash theme={}
systemctl is-active sambanova_exporter
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:9101/metrics
```

| Result | Cause | Fix |
| :- | :- | :- |
| `inactive` and `000` | The service is disabled, which is the state a fresh system ships in | `systemctl enable --now sambanova_exporter` |
| Package not found | SambaNova Runtime is not installed on this host | Install SambaNova Runtime 1.26.3-1 or later, then `yum install -y sambanova-runtime-exporter` |
| `active`, but no exporter pod runs on that node | The chart schedules the pod only on nodes labeled `snRduArch` | Check that the node has the `snRduArch` label |
| The service fails to start | The configuration file exists but cannot be read or parsed | Check `systemctl status sambanova_exporter`. A missing file is ignored; an unparseable one is fatal. |

## An empty dashboard

Check in order:

1. **The exporter is not enabled, or `probe.enabled` is off.** See [No data in Prometheus or Grafana](#no-data-in-prometheus-or-grafana).
2. **Prometheus is not scraping the targets.** Open **Status**, then **Targets**.
3. **A panel needs a label you did not set.** Per-rack power needs a `rack` label on each PDU target, and the RDU dashboard needs `sambanova-exporter.serviceMonitor.rackMap`. See [Customize the dashboards](monitoring-reference-dashboards#customize-the-dashboards).
4. **The dashboard does not appear at all.** The sidecar label did not match. See [Install with the Helm chart](monitoring-reference-dashboards#install-with-the-helm-chart), then search Grafana for `sambarack` — on a default `kube-prometheus-stack` the dashboards land in the General folder.

## Pods and images

| Symptom | Cause | Fix |
| :- | :- | :- |
| `ImagePullBackOff` | No pull secret, or one built from a short-lived access token | Recreate it with the `_json_key` form and the key file. See [Create the pull secret](monitoring-install#create-the-pull-secret). |
| `exec format error`, pod crash-loops | An arm64 image was pulled on an Apple-silicon workstation and pushed to an amd64 cluster | Repull with `crane pull --platform linux/amd64`, as the install guide's `crane pull` commands do |
| `helm install` fails on an unrecognized kind | The prometheus-operator CRDs are missing while `probe.enabled` is on | Install `kube-prometheus-stack`, or leave the probe switches off |
| Prometheus pods hang in `ImagePullBackOff` after everything else looks fine | `prometheus-config-reloader` was not carried across the air gap. The operator injects it as a sidecar at runtime, so it never appears as an `image:` in the rendered manifests. | Carry the `prometheus-config-reloader` image that matches your prometheus-operator version |
| The operator pod sits in `ContainerCreating` with `MountVolume.SetUp failed for volume "tls-secret"` | The admission webhook is disabled but `prometheusOperator.tls.enabled` is still true | Set `prometheusOperator.tls.enabled: false` |
| `401 unauthorized` from the registry after a successful login | `gcloud auth login` was run afterwards, making your user account active | Rerun `gcloud auth activate-service-account --key-file=<path-to-service-account-key>` |

## Collect diagnostics for SambaNova

```bash theme={}
kubectl -n monitoring get pods,svc,probe,servicemonitor -o wide
helm -n monitoring get values hwmon
kubectl -n monitoring logs deploy/pdu-exporter  --tail=200
kubectl -n monitoring logs deploy/xrdu-exporter --tail=200
kubectl get prometheus -A -o jsonpath='{.items[*].spec.probeSelector}'
helm -n monitoring list
```

On each host module that is not reporting:

```bash theme={}
systemctl status sambanova_exporter
journalctl -u sambanova_exporter --since "1 hour ago"
```

Include the chart version, the exporter image tags, and whether the install is connected or air-gapped. Redact device credentials before sharing.

## Next steps

* [Install monitoring](monitoring-install)
* [Monitoring overview](monitoring-overview)