> ## Documentation Index > Fetch the complete documentation index at: https://docs.sambanova.ai/docs/llms.txt > Use this file to discover all available pages before exploring further. # Install monitoring This guide installs SambaRack hardware monitoring on a Kubernetes cluster in either of two modes. | Mode | Use it when | | :- | :- | | **Connected** | The cluster can pull container images and Helm charts from the SambaNova registry. | | **Air-gapped** | The cluster has no internet access. You download the artifacts on a connected workstation, carry them in, and push them to a local registry. | 1. [Enable the SambaNova exporter on every host module](#step-1-enable-the-sambanova-exporter-on-every-host) 2. [Collect your device addresses](#step-2-collect-your-device-addresses) 3. [Make the artifacts available to your cluster](#step-3-make-the-artifacts-available-to-your-cluster) 4. [Write a values file](#step-4-write-a-values-file) 5. [Install the chart](#step-5-install-the-chart) 6. [Verify the metrics](#step-6-verify-the-metrics) 7. [Install the dashboards](#step-7-install-the-dashboards) For device reachability problems, empty dashboards, and probe failures, see [Troubleshooting](monitoring-troubleshooting). ## Before you start ### Cluster requirements | Requirement | Check | | :- | :- | | Kubernetes 1.35 or later | `kubectl version` | | cluster-admin access | `kubectl auth can-i '*' '*' --all-namespaces` | | Helm 3 | `helm version --short` | | prometheus-operator CRDs | `kubectl get crd probes.monitoring.coreos.com servicemonitors.monitoring.coreos.com` | | A Prometheus that selects objects by label | [Step 4](#step-4-write-a-values-file) finds the label | | Grafana with a dashboard sidecar | Optional. [Step 7](#step-7-install-the-dashboards) | If your cluster doesn't run Prometheus and Grafana yet, set them up first using the [Prometheus and Grafana reference architecture](../../../reference-architecture/observability/monitoring-prom). ### Install namespace Create the `monitoring` namespace that the chart and its secrets install into: ```bash theme={} kubectl create namespace monitoring --dry-run=client -o yaml | kubectl apply -f - ``` ### SambaRack Manager Make sure [SambaRack Manager](../srm-overview) is set up, with your rack imported and the XRDU secret set. [Step 2](#step-2-collect-your-device-addresses) uses it to look up device addresses and the BMC password. ### SambaNova Runtime * SambaNova Runtime 1.26.3-1 or later is installed on each host module. It ships the `sambanova-runtime-exporter` package, which the chart cannot install. On the machine you install from: `gcloud` and `helm` 3. Both that machine and the cluster nodes need network access to the SambaNova registry. On a connected Linux (amd64) workstation: [`crane`](https://github.com/google/go-containerregistry/blob/main/cmd/crane/README.md), `helm` 3, `gcloud`, and `jq`. In the air-gapped environment: * Any OCI registry your cluster nodes can pull from — Harbor, Artifactory or a similar private registry. The chart only needs the images reachable at `global.imageRegistry`. * A host with the `crane` binary that can push to that registry. Carry the binary in with the artifacts; it is statically linked and has no dependencies. * A host with `kubectl` and `helm` access to the cluster. These roles are often on different hosts. The host with `kubectl` may not reach the registry, and the host that reaches the registry may have no `kubectl`. Confirm which host can do what before you start. ## Step 1. Enable the SambaNova exporter on every host Do this first, in both modes. The chart does not deploy this exporter and cannot start it — it only makes the host port scrapeable. Until the service is running on every RDU node, the rest of the install produces no RDU metrics. On each SambaRack host module, as a privileged user: ```bash theme={} yum install -y sambanova-runtime-exporter systemctl enable sambanova_exporter systemctl start sambanova_exporter ``` Verify. The service must report `active` and the metrics endpoint must return `200`: ```bash theme={} systemctl is-active sambanova_exporter curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:9101/metrics ``` ## Step 2. Collect your device addresses Read the addresses from the rack inventory rather than retyping them. Run this where SambaRack Manager is installed: ```bash theme={} snctl rack show ``` ```text theme={} Rack 001 (Part Number: 1001934, Type: SN40L-16) Component Hostname IP Port Username -------------------------- ----------- ----------- ---- -------- Rack/001/pdu/1 rack1-pdu1 192.0.2.11 ... ... Rack/001/pdu/2 rack1-pdu2 192.0.2.12 ... ... Rack/001/Node/0/host rack1-h1 192.0.2.20 ... ... Rack/001/Node/0/bmc rack1-h1-sp 192.0.2.21 ... ... Rack/001/Node/0/xrdu/0 rack1-h1xr0 192.0.2.30 ... ... Rack/001/Node/0/xrdu/1 rack1-h1xr1 192.0.2.31 ... ... ``` Hostnames and addresses above are placeholders. Add `-o wide` for Kubernetes node names, or `-o json` for the same inventory as JSON. Everything you need goes into one file, the values file in [Step 4](#step-4-write-a-values-file). Nothing else in this guide needs you to record these separately. | From the inventory | Goes into | | :- | :- | | IP of each `pdu` row | `pdu-exporter.targets[].address` | | IP of each `xrdu` row | `xrdu-exporter.targets[].address` | | Hostname of each `xrdu` row | `xrdu-exporter.targets[].name` | | The rack ID | The `rack` label on every target | | Kubernetes node name of each `Node//host` row | `sambanova-exporter.serviceMonitor.rackMap` | The XRDU BMC password is read separately in [Step 4](#step-4-write-a-values-file), straight into a Kubernetes Secret. It never belongs in the values file. ## Step 3. Make the artifacts available to your cluster SambaNova gives you the registry coordinates along with your service-account key. Throughout this guide, `` is the registry hostname and `` is the full repository path the chart and images are published under. ### Authenticate to the SambaNova registry ```bash theme={} gcloud auth activate-service-account --key-file= gcloud auth print-access-token | helm registry login -u oauth2accesstoken --password-stdin ``` ### Create the pull secret The kubelet needs its own registry credential. Your `helm` login does not provide one. ```bash theme={} kubectl -n monitoring create secret docker-registry sambarack-pull \ --docker-server= \ --docker-username=_json_key \ --docker-password="$(cat )" ``` Nothing else is needed. The kubelet pulls the images and Helm pulls the chart directly in [Step 5](#step-5-install-the-chart). ### Download the artifacts On the connected workstation, log in: ```bash theme={} gcloud auth activate-service-account --key-file= TOKEN="$(gcloud auth print-access-token)" printf %s "$TOKEN" | crane auth login -u oauth2accesstoken --password-stdin printf %s "$TOKEN" | helm registry login -u oauth2accesstoken --password-stdin ``` Then pull the chart and images: ```bash theme={} mkdir -p out REPO= helm pull oci://$REPO/sambanova-hardware-monitoring --version 0.2.4 -d out crane pull --platform linux/amd64 $REPO/pdu-exporter:0.1.3 out/pdu-exporter-0.1.3.tar crane pull --platform linux/amd64 $REPO/xrdu-exporter:0.1.5 out/xrdu-exporter-0.1.5.tar crane pull --platform linux/amd64 registry.access.redhat.com/ubi9/ubi-minimal:9.5 out/ubi-minimal-9.5.tar ``` ### Transfer the artifacts Copy the four files, plus the `crane` binary, into the air-gapped environment using whatever method your site approves. Put them on a host that can push to your local registry. Do not carry the service account key inside. It grants access to the SambaNova registry and has no use in the air gap. It is the only artifact here that grants registry access. ### Push the images into your local registry ```bash theme={} crane auth login -u --password-stdin crane push pdu-exporter-0.1.3.tar //pdu-exporter:0.1.3 crane push xrdu-exporter-0.1.5.tar //xrdu-exporter:0.1.5 crane push ubi-minimal-9.5.tar //ubi-minimal:9.5 ``` `crane push` writes the archive to the destination reference you name, so the image keeps the name and tag you give it here. Keep the image names (`pdu-exporter`, `xrdu-exporter`, `ubi-minimal`) and tags unchanged: the chart builds each reference from `global.imageRegistry`, the repository name, and the tag. If `crane` fails on the registry certificate, install your registry's CA on the host rather than passing `--insecure`. That flag disables verification for the whole transfer and has to be repeated on every command. Confirm the push: ```bash theme={} crane ls //pdu-exporter crane ls //xrdu-exporter crane ls //ubi-minimal ``` ### Create the local registry pull secret Run this on the host with `kubectl` access. The kubelet needs its own credential; a registry login on a node does nothing for it. ```bash theme={} kubectl -n monitoring create secret docker-registry local-registry-pull \ --docker-server= \ --docker-username= \ --docker-password= ``` ## Step 4. Write a values file ### Create the device credential secret An XRDU BMC has no guest read, so the XRDU exporter needs its password. Read it where SambaRack Manager is installed: ```bash theme={} snctl secret get --device-type xrdu --show-value ``` Then, on the host with `kubectl` access, create the Secret. The prompt keeps the password out of your shell history: ```bash theme={} read -rs -p "XRDU BMC password: " XRDU_PASSWORD; echo kubectl -n monitoring create secret generic xrdu-creds \ --from-literal=XRDU_USERNAME=root \ --from-literal=XRDU_PASSWORD="$XRDU_PASSWORD" unset XRDU_PASSWORD ``` ### Find your Prometheus label Your `Probe` and `ServiceMonitor` objects need the label your Prometheus selects on: ```bash theme={} kubectl get prometheus -A -o jsonpath='{.items[*].spec.probeSelector}'; echo kubectl get prometheus -A -o jsonpath='{.items[*].spec.serviceMonitorSelector}'; echo ``` `kube-prometheus-stack` selects on `release: `. An empty selector (`{}`) selects everything and needs no label. A `Probe` or `ServiceMonitor` without the label your Prometheus selects on is created successfully and then never scraped. This is the most common reason a correct install produces no data. ### The values file Create `monitoring-values.yaml`. This is the one file that holds your configuration — every address you looked up in Step 2 goes here, and `helm get values` reads it back at upgrade time. For `global.imageRegistry`, use `` for a connected install, or `/` for an air-gapped one. For ``, use the pull secret you created in Step 3: `sambarack-pull` for a connected install, or `local-registry-pull` for an air-gapped one. ```yaml theme={} global: imageRegistry: imagePullSecrets: - name: # sambarack-pull or local-registry-pull # Runs on the host; the chart only makes it scrapeable. sambanova-exporter: enabled: true serviceMonitor: enabled: true interval: 15s labels: release: rackMap: # node name -> rack ID : "" pdu-exporter: enabled: true probe: enabled: true # without this, nothing scrapes the PDUs labels: release: targets: - address: name: -pdu1 # becomes the `pdu` label labels: rack: "" # required for per-rack power panels - address: name: -pdu2 labels: rack: "" xrdu-exporter: enabled: true auth: existingSecret: xrdu-creds probe: enabled: true labels: release: targets: - address: name: # becomes the `xrdu` label labels: rack: "" node: xrdu_index: XRDU_0 # lets you join with sambanova_* series # one entry per XRDU BMC ``` | Setting | Effect | | :- | :- | | `.enabled` | Deploys the exporter | | `.probe.enabled` | Makes Prometheus scrape the devices. With only `enabled`, the exporter runs and produces nothing. | | `.targets` | Required when a device exporter is enabled. The chart refuses to render without it. | Each target needs only `address`. `name` replaces the address as the device label, and every key under `labels` is added to every series from that device. See [PDU exporter](exporter-pdu#targets) and [XRDU exporter](exporter-xrdu#targets) for all target options. Leave the image tags out unless SambaNova tells you otherwise. The chart defaults are known-good versions, and both have a floor: the PDU exporter needs 0.1.2 or later and the XRDU exporter 0.1.1 or later against real hardware. ## Step 5. Install the chart ```bash theme={} helm upgrade --install hwmon \ oci:///sambanova-hardware-monitoring \ --version 0.2.4 \ --namespace monitoring \ --values monitoring-values.yaml \ --wait --timeout 5m ``` Run this on the host with `kubectl` and `helm` access, from the directory holding the chart archive: ```bash theme={} helm upgrade --install hwmon ./sambanova-hardware-monitoring-0.2.4.tgz \ --namespace monitoring \ --values monitoring-values.yaml \ --wait --timeout 5m ``` Every pod should be Running, and each enabled device exporter should have a `Probe`: ```bash theme={} kubectl -n monitoring get pods,svc kubectl -n monitoring get probe,servicemonitor helm -n monitoring get values hwmon ``` The last command prints the values Helm applied, which is the quickest way to confirm your target list and labels arrived as intended. Then confirm the SambaNova exporter Service has an endpoint for each RDU node: ```bash theme={} kubectl -n monitoring get pods -l app=sambanova-exporter -o wide kubectl -n monitoring get endpoints sambanova-exporter -o yaml | grep -E 'addresses|notReady' ``` A pod that is Running but never Ready, or an endpoint under `notReadyAddresses`, means the host service is down on that node. Return to [Step 1](#step-1-enable-the-sambanova-exporter-on-every-host). The readiness probe follows the host process deliberately, so a dead exporter drops out of the Service instead of reporting zeros. The chart schedules that pod only on nodes labeled `snRduArch`. To tune the exporter, see [SambaNova exporter configuration](exporter-sambanova#configuration). ## Step 6. Verify the metrics ### Query an exporter from inside the cluster This proves the exporter, the credentials, and the network route in one step, and works before Prometheus is scraping. The exporter images have no shell, so run a short-lived pod alongside them, using the `ubi-minimal` image the chart already requires. In the commands below, `` is the same value you set for `global.imageRegistry`, and `` is the pull secret from Step 3. The check pod needs the pull secret explicitly, because `kubectl run` does not attach it. ```bash theme={} kubectl -n monitoring run hwmon-check --rm -i --restart=Never \ --image=/ubi-minimal:9.5 \ --overrides='{"spec":{"imagePullSecrets":[{"name":""}]}}' \ --command -- \ curl -s "http://pdu-exporter:9820/probe?target=" ``` A three-phase PDU reports one sample per phase plus a total: ```text theme={} pdu_up 1 pdu_info{device="",firmware="5.10.8",label="Geist Upgradable rPDU",model="I-03",part="UU30010L",serial="",state="normal"} 1 pdu_power_watts{device="",entity="phase0",label="Phase A",level="phase"} 729 pdu_power_watts{device="",entity="phase1",label="Phase B",level="phase"} 245 pdu_power_watts{device="",entity="phase2",label="Phase C",level="phase"} 706 pdu_power_watts{device="",entity="total0",label="Total",level="total"} 1680 pdu_scrape_duration_seconds 0.690300349 ``` The same command against the XRDU exporter: ```bash theme={} kubectl -n monitoring run hwmon-check --rm -i --restart=Never \ --image=/ubi-minimal:9.5 \ --overrides='{"spec":{"imagePullSecrets":[{"name":""}]}}' \ --command -- \ sh -c 'curl -s "http://xrdu-exporter:9821/probe?target=" | grep -E "xrdu_up|fan_rpm|celsius" | head' ``` ```text theme={} xrdu_up 1 xrdu_fan_rpm{sensor="fm0_fan0"} 7389 xrdu_fan_rpm{sensor="fm0_fan1"} 7140 xrdu_temperature_celsius{sensor="inlet"} 25.25 xrdu_temperature_celsius{sensor="outlet"} 46.062 ``` Check every device at once. List your own PDU addresses in the loop: ```bash theme={} kubectl -n monitoring run hwmon-check --rm -i --restart=Never \ --image=/ubi-minimal:9.5 \ --overrides='{"spec":{"imagePullSecrets":[{"name":""}]}}' \ --command -- \ sh -c 'for ip in ; do echo "$ip $(curl -s "http://pdu-exporter:9820/probe?target=$ip" | grep "^pdu_up")" done' ``` | Result | Meaning | | :- | :- | | `up 1` | The device answered | | `up 0` with HTTP 200 | The device did not answer. For an XRDU this also covers a rejected credential. See [Troubleshooting](monitoring-troubleshooting). | A full XRDU probe returns about 97 series: six fan modules of two fans each plus two PSU fans, and a temperature series per sensor. ### Query through Prometheus Open Grafana at the URL your organization uses, go to **Explore**, select the Prometheus data source, and run: ```promql theme={} sum by (rack) (pdu_power_watts{level="total"}) pdu_up xrdu_up count by (xrdu) (xrdu_fan_rpm) count(sambanova_platform_info) ``` `sum by (rack)` returns one series per rack. `pdu_up` and `xrdu_up` return one series per device. `count by (xrdu)` returns the number of fan sensors per XRDU. `count(sambanova_platform_info)` returns the number of hosts reporting. An empty result means Prometheus is not scraping the targets; see [Troubleshooting](monitoring-troubleshooting#no-data-in-prometheus-or-grafana). To confirm the scrape itself, open the Prometheus UI and go to **Status**, then **Targets**. Each enabled `Probe` appears as a job with one target per device. ## Step 7. Install the dashboards Step 5 installed the dashboards, because `dashboards.enabled` defaults to `true`. If you set it to `false`, turn it back on: ```bash theme={} helm upgrade --install hwmon \ oci:///sambanova-hardware-monitoring \ --version 0.2.4 -n monitoring --reuse-values \ --set dashboards.enabled=true ``` ```bash theme={} helm upgrade --install hwmon ./sambanova-hardware-monitoring-0.2.4.tgz \ -n monitoring --reuse-values \ --set dashboards.enabled=true ``` `--reuse-values` keeps the exporter and target configuration you already applied. See [Reference dashboards](monitoring-reference-dashboards) for the other install methods and what each dashboard shows. ## Upgrade Hardware monitoring chart and exporter versions ship in the [SambaRack Manager release notes](/docs/en/release-notes/sambarack-manager). Upgrade as part of each release cycle. To upgrade a connected install, rerun the `helm upgrade --install` command from [Step 5](#step-5-install-the-chart) with the new `--version`. For an air-gapped install, repeat [Step 3](#step-3-make-the-artifacts-available-to-your-cluster) with the new versions and rerun the command with the new `.tgz`. Your values file is the input both times. If you no longer have it, recover it from the cluster: ```bash theme={} helm -n monitoring get values hwmon -o yaml > monitoring-values.yaml ``` ## Uninstall ```bash theme={} helm uninstall hwmon --namespace monitoring ``` This removes everything the chart created. It does not stop the host `sambanova_exporter` service, and it does not delete secrets you created yourself. ## Next steps * [SambaNova exporter](exporter-sambanova), [PDU exporter](exporter-pdu), and [XRDU exporter](exporter-xrdu) — the metrics and labels available for your own dashboards and alerts * [Reference dashboards](monitoring-reference-dashboards) * [Troubleshooting](monitoring-troubleshooting)