Install SambaRack Monitoring - SambaNova Documentation

This guide installs SambaRack hardware monitoring on a Kubernetes cluster in either of two modes.

Mode	Use it when
Connected	The cluster can pull container images and Helm charts from the SambaNova registry.
Air-gapped	The cluster has no internet access. You download the artifacts on a connected workstation, carry them in, and push them to a local registry.

Enable the SambaNova exporter on every host module
Collect your device addresses
Make the artifacts available to your cluster
Write a values file
Install the chart
Verify the metrics
Install the dashboards

For device reachability problems, empty dashboards, and probe failures, see Troubleshooting.

Before you start

Cluster requirements

Requirement	Check
Kubernetes 1.35 or later	`kubectl version`
cluster-admin access	`kubectl auth can-i '' '' --all-namespaces`
Helm 3	`helm version --short`
prometheus-operator CRDs	`kubectl get crd probes.monitoring.coreos.com servicemonitors.monitoring.coreos.com`
A Prometheus that selects objects by label	Step 4 finds the label
Grafana with a dashboard sidecar	Optional. Step 7

If your cluster doesn’t run Prometheus and Grafana yet, set them up first using the Prometheus and Grafana reference architecture.

Install namespace

Create the monitoring namespace that the chart and its secrets install into:

kubectl create namespace monitoring --dry-run=client -o yaml | kubectl apply -f -

SambaRack Manager

Make sure SambaRack Manager is set up, with your rack imported and the XRDU secret set. Step 2 uses it to look up device addresses and the BMC password.

SambaNova Runtime

SambaNova Runtime 1.26.3-1 or later is installed on each host module. It ships the sambanova-runtime-exporter package, which the chart cannot install.

Connected
Air-gapped

On the machine you install from: gcloud and helm 3. Both that machine and the cluster nodes need network access to the SambaNova registry.

On a connected Linux (amd64) workstation: crane, helm 3, gcloud, and jq.In the air-gapped environment:

Any OCI registry your cluster nodes can pull from — Harbor, Artifactory or a similar private registry. The chart only needs the images reachable at global.imageRegistry.
A host with the crane binary that can push to that registry. Carry the binary in with the artifacts; it is statically linked and has no dependencies.
A host with kubectl and helm access to the cluster.

These roles are often on different hosts. The host with kubectl may not reach the registry, and the host that reaches the registry may have no kubectl. Confirm which host can do what before you start.

Step 1. Enable the SambaNova exporter on every host

Do this first, in both modes. The chart does not deploy this exporter and cannot start it — it only makes the host port scrapeable. Until the service is running on every RDU node, the rest of the install produces no RDU metrics. On each SambaRack host module, as a privileged user:

yum install -y sambanova-runtime-exporter
systemctl enable sambanova_exporter
systemctl start sambanova_exporter

Verify. The service must report active and the metrics endpoint must return 200:

systemctl is-active sambanova_exporter
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:9101/metrics

Step 2. Collect your device addresses

Read the addresses from the rack inventory rather than retyping them. Run this where SambaRack Manager is installed:

snctl rack show <rack-id>

Rack 001 (Part Number: 1001934, Type: SN40L-16)
Component                   Hostname     IP           Port  Username
--------------------------  -----------  -----------  ----  --------
Rack/001/pdu/1              rack1-pdu1   192.0.2.11   ...   ...
Rack/001/pdu/2              rack1-pdu2   192.0.2.12   ...   ...
Rack/001/Node/0/host        rack1-h1     192.0.2.20   ...   ...
Rack/001/Node/0/bmc         rack1-h1-sp  192.0.2.21   ...   ...
Rack/001/Node/0/xrdu/0      rack1-h1xr0  192.0.2.30   ...   ...
Rack/001/Node/0/xrdu/1      rack1-h1xr1  192.0.2.31   ...   ...

Hostnames and addresses above are placeholders. Add -o wide for Kubernetes node names, or -o json for the same inventory as JSON. Everything you need goes into one file, the values file in Step 4. Nothing else in this guide needs you to record these separately.

From the inventory	Goes into
IP of each `pdu` row	`pdu-exporter.targets[].address`
IP of each `xrdu` row	`xrdu-exporter.targets[].address`
Hostname of each `xrdu` row	`xrdu-exporter.targets[].name`
The rack ID	The `rack` label on every target
Kubernetes node name of each `Node/<n>/host` row	`sambanova-exporter.serviceMonitor.rackMap`

The XRDU BMC password is read separately in Step 4, straight into a Kubernetes Secret. It never belongs in the values file.

Step 3. Make the artifacts available to your cluster

SambaNova gives you the registry coordinates along with your service-account key. Throughout this guide, <registry-host> is the registry hostname and <sambanova-registry> is the full repository path the chart and images are published under.

Connected
Air-gapped

Authenticate to the SambaNova registry

gcloud auth activate-service-account --key-file=<path-to-service-account-key>
gcloud auth print-access-token | helm registry login <registry-host> -u oauth2accesstoken --password-stdin

Create the pull secret

The kubelet needs its own registry credential. Your helm login does not provide one.

kubectl -n monitoring create secret docker-registry sambarack-pull \
  --docker-server=<registry-host> \
  --docker-username=_json_key \
  --docker-password="$(cat <path-to-service-account-key>)"

Nothing else is needed. The kubelet pulls the images and Helm pulls the chart directly in Step 5.

Download the artifacts

On the connected workstation, log in:

gcloud auth activate-service-account --key-file=<path-to-service-account-key>
TOKEN="$(gcloud auth print-access-token)"
printf %s "$TOKEN" | crane auth login <registry-host> -u oauth2accesstoken --password-stdin
printf %s "$TOKEN" | helm registry login <registry-host> -u oauth2accesstoken --password-stdin

Then pull the chart and images:

mkdir -p out
REPO=<sambanova-registry>

helm pull oci://$REPO/sambanova-hardware-monitoring --version 0.2.4 -d out
crane pull --platform linux/amd64 $REPO/pdu-exporter:0.1.3  out/pdu-exporter-0.1.3.tar
crane pull --platform linux/amd64 $REPO/xrdu-exporter:0.1.5 out/xrdu-exporter-0.1.5.tar
crane pull --platform linux/amd64 registry.access.redhat.com/ubi9/ubi-minimal:9.5 out/ubi-minimal-9.5.tar

Transfer the artifacts

Copy the four files, plus the crane binary, into the air-gapped environment using whatever method your site approves. Put them on a host that can push to your local registry.

Do not carry the service account key inside. It grants access to the SambaNova registry and has no use in the air gap. It is the only artifact here that grants registry access.

Push the images into your local registry

crane auth login <local-registry-host> -u <registry-user> --password-stdin

crane push pdu-exporter-0.1.3.tar  <local-registry-host>/<project-path>/pdu-exporter:0.1.3
crane push xrdu-exporter-0.1.5.tar <local-registry-host>/<project-path>/xrdu-exporter:0.1.5
crane push ubi-minimal-9.5.tar     <local-registry-host>/<project-path>/ubi-minimal:9.5

crane push writes the archive to the destination reference you name, so the image keeps the name and tag you give it here. Keep the image names (pdu-exporter, xrdu-exporter, ubi-minimal) and tags unchanged: the chart builds each reference from global.imageRegistry, the repository name, and the tag.

If crane fails on the registry certificate, install your registry’s CA on the host rather than passing --insecure. That flag disables verification for the whole transfer and has to be repeated on every command.

Confirm the push:

crane ls <local-registry-host>/<project-path>/pdu-exporter
crane ls <local-registry-host>/<project-path>/xrdu-exporter
crane ls <local-registry-host>/<project-path>/ubi-minimal

Create the local registry pull secret

Run this on the host with kubectl access. The kubelet needs its own credential; a registry login on a node does nothing for it.

kubectl -n monitoring create secret docker-registry local-registry-pull \
  --docker-server=<local-registry-host> \
  --docker-username=<registry-user> \
  --docker-password=<registry-password>

Step 4. Write a values file

Create the device credential secret

An XRDU BMC has no guest read, so the XRDU exporter needs its password. Read it where SambaRack Manager is installed:

snctl secret get --device-type xrdu --show-value

Then, on the host with kubectl access, create the Secret. The prompt keeps the password out of your shell history:

read -rs -p "XRDU BMC password: " XRDU_PASSWORD; echo

kubectl -n monitoring create secret generic xrdu-creds \
  --from-literal=XRDU_USERNAME=root \
  --from-literal=XRDU_PASSWORD="$XRDU_PASSWORD"

unset XRDU_PASSWORD

Find your Prometheus label

Your Probe and ServiceMonitor objects need the label your Prometheus selects on:

kubectl get prometheus -A -o jsonpath='{.items[*].spec.probeSelector}'; echo
kubectl get prometheus -A -o jsonpath='{.items[*].spec.serviceMonitorSelector}'; echo

kube-prometheus-stack selects on release: <its-release-name>. An empty selector ({}) selects everything and needs no label.

A Probe or ServiceMonitor without the label your Prometheus selects on is created successfully and then never scraped. This is the most common reason a correct install produces no data.

The values file

Create monitoring-values.yaml. This is the one file that holds your configuration — every address you looked up in Step 2 goes here, and helm get values reads it back at upgrade time. For global.imageRegistry, use <sambanova-registry> for a connected install, or <local-registry-host>/<project-path> for an air-gapped one. For <pull-secret-name>, use the pull secret you created in Step 3: sambarack-pull for a connected install, or local-registry-pull for an air-gapped one.

global:
  imageRegistry: <sambanova-registry>
  imagePullSecrets:
    - name: <pull-secret-name>      # sambarack-pull or local-registry-pull

# Runs on the host; the chart only makes it scrapeable.
sambanova-exporter:
  enabled: true
  serviceMonitor:
    enabled: true
    interval: 15s
    labels:
      release: <prometheus-release-label>
    rackMap:                        # node name -> rack ID
      <node-name>: "<rack-id>"

pdu-exporter:
  enabled: true
  probe:
    enabled: true                   # without this, nothing scrapes the PDUs
    labels:
      release: <prometheus-release-label>
  targets:
    - address: <pdu-1-ip>
      name: <rack-id>-pdu1          # becomes the `pdu` label
      labels:
        rack: "<rack-id>"           # required for per-rack power panels
    - address: <pdu-2-ip>
      name: <rack-id>-pdu2
      labels:
        rack: "<rack-id>"

xrdu-exporter:
  enabled: true
  auth:
    existingSecret: xrdu-creds
  probe:
    enabled: true
    labels:
      release: <prometheus-release-label>
  targets:
    - address: <xrdu-0-ip>
      name: <xrdu-0-hostname>       # becomes the `xrdu` label
      labels:
        rack: "<rack-id>"
        node: <node-name>
        xrdu_index: XRDU_0          # lets you join with sambanova_* series
    # one entry per XRDU BMC

Setting	Effect
`<exporter>.enabled`	Deploys the exporter
`<exporter>.probe.enabled`	Makes Prometheus scrape the devices. With only `enabled`, the exporter runs and produces nothing.
`<exporter>.targets`	Required when a device exporter is enabled. The chart refuses to render without it.

Each target needs only address. name replaces the address as the device label, and every key under labels is added to every series from that device. See PDU exporter and XRDU exporter for all target options. Leave the image tags out unless SambaNova tells you otherwise. The chart defaults are known-good versions, and both have a floor: the PDU exporter needs 0.1.2 or later and the XRDU exporter 0.1.1 or later against real hardware.

Step 5. Install the chart

Connected
Air-gapped

helm upgrade --install hwmon \
  oci://<sambanova-registry>/sambanova-hardware-monitoring \
  --version 0.2.4 \
  --namespace monitoring \
  --values monitoring-values.yaml \
  --wait --timeout 5m

Run this on the host with kubectl and helm access, from the directory holding the chart archive:

helm upgrade --install hwmon ./sambanova-hardware-monitoring-0.2.4.tgz \
  --namespace monitoring \
  --values monitoring-values.yaml \
  --wait --timeout 5m

Every pod should be Running, and each enabled device exporter should have a Probe:

kubectl -n monitoring get pods,svc
kubectl -n monitoring get probe,servicemonitor
helm -n monitoring get values hwmon

The last command prints the values Helm applied, which is the quickest way to confirm your target list and labels arrived as intended. Then confirm the SambaNova exporter Service has an endpoint for each RDU node:

kubectl -n monitoring get pods -l app=sambanova-exporter -o wide
kubectl -n monitoring get endpoints sambanova-exporter -o yaml | grep -E 'addresses|notReady'

A pod that is Running but never Ready, or an endpoint under notReadyAddresses, means the host service is down on that node. Return to Step 1. The readiness probe follows the host process deliberately, so a dead exporter drops out of the Service instead of reporting zeros.

The chart schedules that pod only on nodes labeled snRduArch. To tune the exporter, see SambaNova exporter configuration.

Step 6. Verify the metrics

Query an exporter from inside the cluster

This proves the exporter, the credentials, and the network route in one step, and works before Prometheus is scraping. The exporter images have no shell, so run a short-lived pod alongside them, using the ubi-minimal image the chart already requires. In the commands below, <image-registry> is the same value you set for global.imageRegistry, and <pull-secret-name> is the pull secret from Step 3. The check pod needs the pull secret explicitly, because kubectl run does not attach it.

kubectl -n monitoring run hwmon-check --rm -i --restart=Never \
  --image=<image-registry>/ubi-minimal:9.5 \
  --overrides='{"spec":{"imagePullSecrets":[{"name":"<pull-secret-name>"}]}}' \
  --command -- \
  curl -s "http://pdu-exporter:9820/probe?target=<pdu-1-ip>"

A three-phase PDU reports one sample per phase plus a total:

pdu_up 1
pdu_info{device="<device-id>",firmware="5.10.8",label="Geist Upgradable rPDU",model="I-03",part="UU30010L",serial="<serial>",state="normal"} 1
pdu_power_watts{device="<device-id>",entity="phase0",label="Phase A",level="phase"} 729
pdu_power_watts{device="<device-id>",entity="phase1",label="Phase B",level="phase"} 245
pdu_power_watts{device="<device-id>",entity="phase2",label="Phase C",level="phase"} 706
pdu_power_watts{device="<device-id>",entity="total0",label="Total",level="total"} 1680
pdu_scrape_duration_seconds 0.690300349

The same command against the XRDU exporter:

kubectl -n monitoring run hwmon-check --rm -i --restart=Never \
  --image=<image-registry>/ubi-minimal:9.5 \
  --overrides='{"spec":{"imagePullSecrets":[{"name":"<pull-secret-name>"}]}}' \
  --command -- \
  sh -c 'curl -s "http://xrdu-exporter:9821/probe?target=<xrdu-0-ip>" | grep -E "xrdu_up|fan_rpm|celsius" | head'

xrdu_up 1
xrdu_fan_rpm{sensor="fm0_fan0"} 7389
xrdu_fan_rpm{sensor="fm0_fan1"} 7140
xrdu_temperature_celsius{sensor="inlet"} 25.25
xrdu_temperature_celsius{sensor="outlet"} 46.062

Check every device at once. List your own PDU addresses in the loop:

kubectl -n monitoring run hwmon-check --rm -i --restart=Never \
  --image=<image-registry>/ubi-minimal:9.5 \
  --overrides='{"spec":{"imagePullSecrets":[{"name":"<pull-secret-name>"}]}}' \
  --command -- \
  sh -c 'for ip in <pdu-1-ip> <pdu-2-ip>; do
           echo "$ip $(curl -s "http://pdu-exporter:9820/probe?target=$ip" | grep "^pdu_up")"
         done'

Result	Meaning
`up 1`	The device answered
`up 0` with HTTP 200	The device did not answer. For an XRDU this also covers a rejected credential. See Troubleshooting.

A full XRDU probe returns about 97 series: six fan modules of two fans each plus two PSU fans, and a temperature series per sensor.

Query through Prometheus

Open Grafana at the URL your organization uses, go to Explore, select the Prometheus data source, and run:

sum by (rack) (pdu_power_watts{level="total"})
pdu_up
xrdu_up
count by (xrdu) (xrdu_fan_rpm)
count(sambanova_platform_info)

sum by (rack) returns one series per rack. pdu_up and xrdu_up return one series per device. count by (xrdu) returns the number of fan sensors per XRDU. count(sambanova_platform_info) returns the number of hosts reporting. An empty result means Prometheus is not scraping the targets; see Troubleshooting. To confirm the scrape itself, open the Prometheus UI and go to Status, then Targets. Each enabled Probe appears as a job with one target per device.

Step 7. Install the dashboards

Step 5 installed the dashboards, because dashboards.enabled defaults to true. If you set it to false, turn it back on:

Connected
Air-gapped

helm upgrade --install hwmon \
  oci://<sambanova-registry>/sambanova-hardware-monitoring \
  --version 0.2.4 -n monitoring --reuse-values \
  --set dashboards.enabled=true

helm upgrade --install hwmon ./sambanova-hardware-monitoring-0.2.4.tgz \
  -n monitoring --reuse-values \
  --set dashboards.enabled=true

--reuse-values keeps the exporter and target configuration you already applied. See Reference dashboards for the other install methods and what each dashboard shows.

Upgrade

Hardware monitoring chart and exporter versions ship in the SambaRack Manager release notes. Upgrade as part of each release cycle. To upgrade a connected install, rerun the helm upgrade --install command from Step 5 with the new --version. For an air-gapped install, repeat Step 3 with the new versions and rerun the command with the new .tgz. Your values file is the input both times. If you no longer have it, recover it from the cluster:

helm -n monitoring get values hwmon -o yaml > monitoring-values.yaml

Uninstall

helm uninstall hwmon --namespace monitoring

This removes everything the chart created. It does not stop the host sambanova_exporter service, and it does not delete secrets you created yourself.

Next steps

SambaNova exporter, PDU exporter, and XRDU exporter — the metrics and labels available for your own dashboards and alerts
Reference dashboards
Troubleshooting

Get started

Models

Features

Build

Resources

Install monitoring

Before you start

Cluster requirements

Install namespace

SambaRack Manager

SambaNova Runtime

Step 1. Enable the SambaNova exporter on every host

Step 2. Collect your device addresses

Step 3. Make the artifacts available to your cluster

Authenticate to the SambaNova registry

Create the pull secret

Download the artifacts

Transfer the artifacts

Push the images into your local registry

Create the local registry pull secret

Step 4. Write a values file

Create the device credential secret

Find your Prometheus label

The values file

Step 5. Install the chart

Step 6. Verify the metrics

Query an exporter from inside the cluster

Query through Prometheus

Step 7. Install the dashboards

Upgrade

Uninstall

Next steps

​Before you start

​Cluster requirements

​Install namespace

​SambaRack Manager

​SambaNova Runtime

​Step 1. Enable the SambaNova exporter on every host

​Step 2. Collect your device addresses

​Step 3. Make the artifacts available to your cluster

​Authenticate to the SambaNova registry

​Create the pull secret

​Download the artifacts

​Transfer the artifacts

​Push the images into your local registry

​Create the local registry pull secret

​Step 4. Write a values file

​Create the device credential secret

​Find your Prometheus label

​The values file

​Step 5. Install the chart

​Step 6. Verify the metrics

​Query an exporter from inside the cluster

​Query through Prometheus

​Step 7. Install the dashboards

​Upgrade

​Uninstall

​Next steps

Before you start

Cluster requirements

Install namespace

SambaRack Manager

SambaNova Runtime

Step 1. Enable the SambaNova exporter on every host

Step 2. Collect your device addresses

Step 3. Make the artifacts available to your cluster

Authenticate to the SambaNova registry

Create the pull secret

Download the artifacts

Transfer the artifacts

Push the images into your local registry

Create the local registry pull secret

Step 4. Write a values file

Create the device credential secret

Find your Prometheus label

The values file

Step 5. Install the chart

Step 6. Verify the metrics

Query an exporter from inside the cluster

Query through Prometheus

Step 7. Install the dashboards

Upgrade

Uninstall

Next steps