Troubleshoot SambaRack Monitoring - SambaNova Documentation

Work through this page when an install completes but produces no data, or when a device stops reporting.

Where to start

Symptom	Section
A correct install returns no metrics at all	No data in Prometheus or Grafana
`pdu_up 0` or `xrdu_up 0` for a device	Device reachability
Some XRDU sensors report and others do not	XRDU probe failures
No RDU metrics, but PDU and XRDU work	The SambaNova exporter is not reporting
Dashboards are missing or empty	An empty dashboard
Rack power reads about three times too high	Filter on `level="total"`. See PDU exporter labels.
Pods will not start	Pods and images

No data in Prometheus or Grafana

Check these in order. The first two cause almost every case.

1. The probe was never created

kubectl -n monitoring get probe,servicemonitor

<exporter>.enabled deploys the exporter. <exporter>.probe.enabled makes Prometheus scrape the devices. With only the first, the exporter runs and produces nothing. Confirm what Helm actually applied:

helm -n monitoring get values hwmon

2. Prometheus is not selecting the probe

This is the most common reason a correct install produces no data. A Probe or ServiceMonitor without the label your Prometheus selects on is created successfully and then never scraped — with no error anywhere.

kubectl get prometheus -A -o jsonpath='{.items[*].spec.probeSelector}'; echo
kubectl get prometheus -A -o jsonpath='{.items[*].spec.serviceMonitorSelector}'; echo

kubectl -n monitoring get probe -o jsonpath='{.items[*].metadata.labels}'; echo

The labels must match. kube-prometheus-stack selects on release: <its-release-name>. An empty selector ({}) selects everything. Fix by setting probe.labels and serviceMonitor.labels in your values file and upgrading.

3. The target is down

Open the Prometheus UI and go to Status, then Targets. Each enabled Probe appears as a job with one target per device. A target listed as down carries the scrape error. A target that does not appear at all means Prometheus is not selecting the probe. Return to step 2.

Device reachability

An exporter pod reaches a PDU or BMC only if the node it runs on has a route to the management network. Nodes usually carry a front-end and a management address both, but not always, and where they do not, no chart setting produces a metric. Run these checks from a cluster node, not from your workstation. List your own device addresses in each loop. The second loop prompts for the XRDU BMC password so it stays out of your shell history.

for ip in <pdu-1-ip> <pdu-2-ip>; do
  code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 10 \
    "http://$ip/api/sys" 2>/dev/null)
  echo "PDU  $ip -> http=${code:-none} curl_exit=$?"
done

read -rs -p "XRDU BMC password: " XRDU_PASSWORD; echo

for ip in <xrdu-0-ip> <xrdu-1-ip>; do
  code=$(curl -sk -o /dev/null -w '%{http_code}' --max-time 10 \
    -u "root:$XRDU_PASSWORD" \
    "https://$ip/xyz/openbmc_project/sensors/enumerate" 2>/dev/null)
  echo "XRDU $ip -> http=${code:-none} curl_exit=$?"
done

unset XRDU_PASSWORD

Result	Meaning	Fix
`http=200`	The node can reach the device. Two or more seconds is normal for these devices.	—
`http=401`	Wrong XRDU password	See Update the XRDU password
`http=000` with a non-zero `curl_exit`	No route. Wrong cluster, or the node has no interface on the management network.	Pin the exporter pods to nodes that do, with `nodeSelector`

Both -k and -u are required for an XRDU. The BMC uses a self-signed certificate and requires authentication; a plain curl https://... fails on the certificate in a way that looks like the device being down. If some nodes have a route and others do not, pin the exporters:

pdu-exporter:
  nodeSelector:
    <label-that-marks-management-network-nodes>: "true"
xrdu-exporter:
  nodeSelector:
    <label-that-marks-management-network-nodes>: "true"

An unreachable device returns HTTP 200 with pdu_up 0 or xrdu_up 0, not an error status. This is deliberate: a non-200 response would make Prometheus discard the whole scrape, including the up sample your outage alert depends on.

XRDU probe failures

For an XRDU, xrdu_up 0 covers a rejected credential, a certificate problem, and an unreachable device alike. The log distinguishes them.

kubectl -n monitoring logs deploy/xrdu-exporter | grep 'probe failed'

Log or symptom	Cause	Fix
`x509: negative serial number`	Exporter older than 0.1.1	Use image 0.1.1 or later
`401 Unauthorized`	Wrong password	See Update the XRDU password
`dial tcp ... i/o timeout`	The node cannot route to the BMC network	See Device reachability
`response was not the JSON envelope`	Not an XRDU BMC at that address	Check the address against `snctl rack show`
Pod `CreateContainerConfigError`	The Secret has no `XRDU_PASSWORD` key	Recreate the Secret with the correct key names. See Update the XRDU password.

Update the XRDU password

The exporter reads the password from the Secret only when its pod starts. After you change the Secret, restart the exporter. Read the current password where SambaRack Manager is installed:

snctl secret get --device-type xrdu --show-value

Then, on the host with kubectl access, replace the Secret and restart the exporter:

read -rs -p "XRDU BMC password: " XRDU_PASSWORD; echo

kubectl -n monitoring create secret generic xrdu-creds \
  --from-literal=XRDU_USERNAME=root \
  --from-literal=XRDU_PASSWORD="$XRDU_PASSWORD" \
  --dry-run=client -o yaml | kubectl apply -f -

unset XRDU_PASSWORD

kubectl -n monitoring rollout restart deploy/xrdu-exporter

The SambaNova exporter is not reporting

This exporter is a host process. The chart does not deploy it and cannot start it.

kubectl -n monitoring get pods -l app=sambanova-exporter -o wide
kubectl -n monitoring get endpoints sambanova-exporter -o yaml | grep -E 'addresses|notReady'

A pod that is Running but never Ready, or an endpoint under notReadyAddresses, means the host service is down on that node. The readiness probe follows the host process deliberately, so a dead exporter drops out of the Service instead of reporting zeros. On that host module:

systemctl is-active sambanova_exporter
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:9101/metrics

Result	Cause	Fix
`inactive` and `000`	The service is disabled, which is the state a fresh system ships in	`systemctl enable --now sambanova_exporter`
Package not found	SambaNova Runtime is not installed on this host	Install SambaNova Runtime 1.26.3-1 or later, then `yum install -y sambanova-runtime-exporter`
`active`, but no exporter pod runs on that node	The chart schedules the pod only on nodes labeled `snRduArch`	Check that the node has the `snRduArch` label
The service fails to start	The configuration file exists but cannot be read or parsed	Check `systemctl status sambanova_exporter`. A missing file is ignored; an unparseable one is fatal.

An empty dashboard

Check in order:

The exporter is not enabled, or probe.enabled is off. See No data in Prometheus or Grafana.
Prometheus is not scraping the targets. Open Status, then Targets.
A panel needs a label you did not set. Per-rack power needs a rack label on each PDU target, and the RDU dashboard needs sambanova-exporter.serviceMonitor.rackMap. See Customize the dashboards.
The dashboard does not appear at all. The sidecar label did not match. See Install with the Helm chart, then search Grafana for sambarack — on a default kube-prometheus-stack the dashboards land in the General folder.

Pods and images

Symptom	Cause	Fix
`ImagePullBackOff`	No pull secret, or one built from a short-lived access token	Recreate it with the `_json_key` form and the key file. See Create the pull secret.
`exec format error`, pod crash-loops	An arm64 image was pulled on an Apple-silicon workstation and pushed to an amd64 cluster	Repull with `crane pull --platform linux/amd64`, as the install guide’s `crane pull` commands do
`helm install` fails on an unrecognized kind	The prometheus-operator CRDs are missing while `probe.enabled` is on	Install `kube-prometheus-stack`, or leave the probe switches off
Prometheus pods hang in `ImagePullBackOff` after everything else looks fine	`prometheus-config-reloader` was not carried across the air gap. The operator injects it as a sidecar at runtime, so it never appears as an `image:` in the rendered manifests.	Carry the `prometheus-config-reloader` image that matches your prometheus-operator version
The operator pod sits in `ContainerCreating` with `MountVolume.SetUp failed for volume "tls-secret"`	The admission webhook is disabled but `prometheusOperator.tls.enabled` is still true	Set `prometheusOperator.tls.enabled: false`
`401 unauthorized` from the registry after a successful login	`gcloud auth login` was run afterwards, making your user account active	Rerun `gcloud auth activate-service-account --key-file=<path-to-service-account-key>`

Collect diagnostics for SambaNova

kubectl -n monitoring get pods,svc,probe,servicemonitor -o wide
helm -n monitoring get values hwmon
kubectl -n monitoring logs deploy/pdu-exporter  --tail=200
kubectl -n monitoring logs deploy/xrdu-exporter --tail=200
kubectl get prometheus -A -o jsonpath='{.items[*].spec.probeSelector}'
helm -n monitoring list

On each host module that is not reporting:

systemctl status sambanova_exporter
journalctl -u sambanova_exporter --since "1 hour ago"

Include the chart version, the exporter image tags, and whether the install is connected or air-gapped. Redact device credentials before sharing.

Get started

Models

Features

Build

Resources

Troubleshoot monitoring

Where to start

No data in Prometheus or Grafana

1. The probe was never created

2. Prometheus is not selecting the probe

3. The target is down

Device reachability

XRDU probe failures

Update the XRDU password

The SambaNova exporter is not reporting

An empty dashboard

Pods and images

Collect diagnostics for SambaNova

Next steps

​Where to start

​No data in Prometheus or Grafana

​1. The probe was never created

​2. Prometheus is not selecting the probe

​3. The target is down

​Device reachability

​XRDU probe failures

​Update the XRDU password

​The SambaNova exporter is not reporting

​An empty dashboard

​Pods and images

​Collect diagnostics for SambaNova

​Next steps

Where to start

No data in Prometheus or Grafana

1. The probe was never created

2. Prometheus is not selecting the probe

3. The target is down

Device reachability

XRDU probe failures

Update the XRDU password

The SambaNova exporter is not reporting

An empty dashboard

Pods and images

Collect diagnostics for SambaNova

Next steps