> ## Documentation Index
> Fetch the complete documentation index at: https://sambanova-systems.mintlify.site/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Install and verification

## Step 1 – Set up your environment

Prepare these environment variables before proceeding with the installation. Create a `.env` file to hold your environment-specific values and source it before running any commands in this guide.

```bash .env theme={null}
NAMESPACE=monitoring
REGISTRY=<registry-url>                  # provided by SambaNova
REGISTRY_SERVER=us-west2-docker.pkg.dev  # SambaNova artifact registry; confirm with your rep if different
GCP_KEY_PATH=/path/to/gcp-key.json       # path to your SambaNova-provided key file
CHART_VERSION=<chart-version>            # provided by SambaNova
XRDU_USERNAME=<xrdu-bmc-username>        # read-only BMC user (see Prerequisites)
XRDU_PASSWORD=<xrdu-bmc-password>        # read-only BMC password
```

```bash theme={null}
source .env
```

<Warning>
  The `.env` file and GCP key file you create during installation contain credentials. Restrict permissions after creating them:

  ```bash theme={null}
  chmod 600 .env
  chmod 600 /path/to/gcp-key.json
  echo ".env" >> .gitignore
  ```
</Warning>

***

## Step 2 – Create the monitoring namespace

```bash theme={null}
kubectl create namespace $NAMESPACE
```

If the namespace already exists, this command will return an error – that is fine, continue to Step 3.

***

## Step 3 – Create the image pull secret

This secret allows Kubernetes to pull the exporter images from the SambaNova registry using the service account key file SambaNova provided.

```bash theme={null}
kubectl create secret docker-registry sn-registry-creds \
  --docker-server=$REGISTRY_SERVER \
  --docker-username=_json_key \
  --docker-password="$(cat "$GCP_KEY_PATH")" \
  --namespace=$NAMESPACE
```

***

## Step 4 – Create the XRDU credentials secret

This secret holds the read-only BMC credentials the XRDU exporter uses to authenticate when scraping each XRDU BMC.

```bash theme={null}
kubectl create secret generic sn-xrdu-creds \
  --from-literal=XRDU_USERNAME=$XRDU_USERNAME \
  --from-literal=XRDU_PASSWORD=$XRDU_PASSWORD \
  --namespace=$NAMESPACE
```

***

## Step 5 – Authenticate to the OCI registry

```bash theme={null}
gcloud auth print-access-token | helm registry login $REGISTRY_SERVER \
  --username=oauth2accesstoken \
  --password-stdin
```

***

## Step 6 – Prepare your values override file

The chart needs to know how to discover your XRDU BMCs. Choose the mode that matches your environment. Example files are included in the Helm chart package at `examples/discovery.yaml` (Auto Discovery tab) and `examples/manual.yaml` (Manual tab).

<Tabs>
  <Tab title="Auto Discovery">
    Auto discover based on node hostname and XRDU index. The chart generates candidate hostnames from the pattern and pings each one to discover active XRDUs.

    Use this mode if your cluster uses DNS-based XRDU hostname resolution – where XRDU BMC hostnames follow a predictable DNS pattern.

    ```yaml values.yaml theme={null}
    sn-xrdu-exporter:
      existingSecret: sn-xrdu-creds     # the secret created in Step 4
      autoXrduDiscovery: true
      serviceMonitor:
        enabled: true
      discovery:
        pattern: "$HOSTNAME-xrdu$I-sp"  # chart template variables, not shell variables: $HOSTNAME = node hostname, $I = XRDU index
        indexRange:
          start: 0
          end: 7                        # adjust to match the number of XRDUs per node (standard rack has 8)
        pingTimeoutSeconds: 1

    sambanova-exporter:
      serviceMonitor:
        enabled: true

    ipmi-exporter:
      serviceMonitor:
        enabled: true
    ```
  </Tab>

  <Tab title="Manual">
    Use this mode to manually specify every XRDU BMC IP and the node it belongs to.

    Use this mode when your environment does not use the DNS naming convention above – for example, most standard on-premises customer deployments.

    ```yaml values.yaml theme={null}
    sn-xrdu-exporter:
      existingSecret: sn-xrdu-creds     # the secret created in Step 4
      autoXrduDiscovery: false
      mountHostEtcHosts: false          # set true only if targets use hostnames (not IPs) that are in /etc/hosts
      serviceMonitor:
        enabled: true
      xrdu:
        targets:
          # One entry per XRDU BMC. Use `snctl node show <node>` to get these values.
          - node: <kubernetes-node-name>   # exact match of `kubectl get nodes` NAME column – must align with the node name in snctl
            xrdu_ip: <xrdu-bmc-ip>        # use xrdu.hostname from `snctl node show <node-name>` for this specific XRDU
            xrdu: <xrdu-label>            # human-readable label, e.g. rack1-host1-xrdu0
            rack: <rack-id>               # must align with the rack name in snctl, e.g. rack-01
            group: <group-id>             # must align with the group name in snctl, e.g. g1
          # Add one entry for every XRDU in your deployment

    sambanova-exporter:
      serviceMonitor:
        enabled: true

    ipmi-exporter:
      serviceMonitor:
        enabled: true
    ```

    **Filling in the target fields:**

    | Field     | Where it comes from                                                                  | Example             |
    | :-------- | :----------------------------------------------------------------------------------- | :------------------ |
    | `node`    | `kubectl get nodes` – use the exact value in the `NAME` column                       | `rack1-host1`       |
    | `xrdu_ip` | `snctl node show <node-name>` – use the `xrdu.hostname` field for this specific XRDU | `192.168.6.1`       |
    | `xrdu`    | Your choice – becomes a Prometheus label                                             | `rack1-host1-xrdu0` |
    | `rack`    | Align with the rack name in snctl – becomes a Prometheus label                       | `rack-01`           |
    | `group`   | Align with the group name in snctl – becomes a Prometheus label                      | `g1`                |
  </Tab>
</Tabs>

Configure the Prometheus scrape interval in your ServiceMonitor settings. **A recommended default is 30–60 seconds**:

```yaml values.yaml theme={null}
    serviceMonitor:
      enabled: true
      interval: 60s
```

***

## Step 7 – Install / Upgrade the Helm chart

```bash theme={null}
helm upgrade --install sambanova-hardware-monitoring \
  oci://$REGISTRY/sambanova-hardware-monitoring \
  --version $CHART_VERSION \
  --namespace $NAMESPACE --create-namespace \
  -f values.yaml
```

The chart deploys three DaemonSets, three Services, and three ServiceMonitor resources. Prometheus will begin scraping.

***

## Verification

Work through each check in order. Each one confirms a layer on top of the previous.

### Check 1 – DaemonSet pods are running

```bash theme={null}
kubectl get pods -n $NAMESPACE -l app=ipmi-exporter -o wide
kubectl get pods -n $NAMESPACE -l app=sambanova-exporter -o wide
kubectl get pods -n $NAMESPACE -l app=sn-xrdu-exporter -o wide
```

All pods should show `Running`. The `NODE` column must show your SambaRack node names.

If the label selector returns nothing, the chart may use `app.kubernetes.io/name` instead of `app`. Fall back to:

```bash theme={null}
kubectl get pods -n $NAMESPACE | grep -E 'ipmi|sambanova|xrdu'
```

<Note>
  If a pod is `Pending` or in `CrashLoopBackOff`, see the [Troubleshooting](./troubleshooting) section.
</Note>

### Check 2 – Prometheus targets are up

Open the Prometheus UI and navigate to **Status → Targets**. Confirm all three exporter targets show `UP (n/n)` – all instances scraped successfully.

If Prometheus is not available, skip to Check 3.

### Check 3 – Exporters are responding

**With Prometheus:** Port-forward each exporter and confirm it returns metrics.

```bash theme={null}
# IPMI exporter – expect lines starting with ipmi_
kubectl port-forward -n $NAMESPACE daemonset/ipmi-exporter 9289:9289 &
sleep 2 && curl -s http://localhost:9289/metrics | grep "^ipmi_" | head -5
pkill -f "port-forward.*9289"

# SambaNova exporter – expect lines starting with sambanova_
kubectl port-forward -n $NAMESPACE daemonset/sambanova-exporter 9101:9101 &
sleep 2 && curl -s http://localhost:9101/metrics | grep "^sambanova_" | head -5
pkill -f "port-forward.*9101"

# XRDU exporter – expect lines starting with xrdu_
kubectl port-forward -n $NAMESPACE daemonset/sn-xrdu-exporter 8076:8076 &
sleep 2 && curl -s http://localhost:8076/metrics | grep "^xrdu_" | head -5
pkill -f "port-forward.*8076"
```

<Frame caption="Prometheus UI - All Exporters">
  <img src="https://mintcdn.com/sambanova-systems/aFwO25BcPwOFnlsb/images/hardware-monitoring-prometheus-ui-all-exporters-1.png?fit=max&auto=format&n=aFwO25BcPwOFnlsb&q=85&s=1ead0abbbcce50c45aef513991f1149e" alt="Hardware Monitoring Prometheus Ui All Exporters 1" width="2538" height="752" data-path="images/hardware-monitoring-prometheus-ui-all-exporters-1.png" />
</Frame>

<Frame caption="Prometheus UI - Node Exporter">
  <img src="https://mintcdn.com/sambanova-systems/aFwO25BcPwOFnlsb/images/hardware-monitoring-prometheus-ui.png?fit=max&auto=format&n=aFwO25BcPwOFnlsb&q=85&s=66b8828560dddcbc8f3221ac32a48393" alt="Hardware Monitoring Prometheus Ui" width="2514" height="1582" data-path="images/hardware-monitoring-prometheus-ui.png" />
</Frame>

<Frame caption="Endpoint Metric Details">
  <img src="https://mintcdn.com/sambanova-systems/aFwO25BcPwOFnlsb/images/hardware-monitoring-metrics-details.png?fit=max&auto=format&n=aFwO25BcPwOFnlsb&q=85&s=a7abcc8ca0f222ee7bf93746642608a6" alt="Hardware Monitoring Metrics Details" width="1466" height="1646" data-path="images/hardware-monitoring-metrics-details.png" />
</Frame>

**Without Prometheus:** Iterate over the exporter pods by label and hit each metrics endpoint directly.

```bash theme={null}
# List all exporter pods with their IPs
kubectl get pods -n $NAMESPACE -l 'app in (ipmi-exporter,sambanova-exporter,sn-xrdu-exporter)' \
  -o custom-columns='NAME:.metadata.name,IP:.status.podIP,PORT:.spec.containers[0].ports[0].containerPort'

# Then exec into any pod in the namespace and curl each IP:port
kubectl exec -n $NAMESPACE <any-running-pod> -- \
  curl -s http://<pod-ip>:<port>/metrics | head -10
```

<Note>
  Each `port-forward` selects one pod from the DaemonSet at random. This verifies one node's exporter. To test a specific node's pod, use `kubectl port-forward -n $NAMESPACE pod/<pod-name> <port>:<port>` after identifying the pod name from Check 1.
</Note>

<Note>
  If metrics are not flowing or pods are not coming up, check for any misconfiguration in your `values.yaml`. Common issues are covered in the [Troubleshooting](./troubleshooting) section.
</Note>

***

## Air-gap installation

If your cluster cannot reach the SambaNova artifact registry, package the chart and images on a machine that can, then transfer them to the air-gapped environment.

### 1. On a machine with registry access

Authenticate and pull the chart and images:

```bash theme={null}
gcloud auth print-access-token | helm registry login $REGISTRY_SERVER \
  --username=oauth2accesstoken \
  --password-stdin

helm pull oci://$REGISTRY/sambanova-hardware-monitoring --version $CHART_VERSION
# Produces: sambanova-hardware-monitoring-<version>.tgz

docker pull $REGISTRY/sn-ipmi-exporter:<ipmi-version>
docker pull $REGISTRY/sn-xrdu-exporter:<xrdu-version>
docker save \
  $REGISTRY/sn-ipmi-exporter:<ipmi-version> \
  $REGISTRY/sn-xrdu-exporter:<xrdu-version> \
  -o sambanova-hw-monitoring-images.tar
```

Contact your SambaNova representative for the image tag versions that match your chart version.

### 2. Transfer to the air-gapped environment

Copy `sambanova-hardware-monitoring-<version>.tgz` and `sambanova-hw-monitoring-images.tar` to the air-gapped cluster.

### 3. Load images into the internal registry

```bash theme={null}
docker load -i sambanova-hw-monitoring-images.tar
docker tag $REGISTRY/sn-ipmi-exporter:<ipmi-version> $INTERNAL_REGISTRY/sn-ipmi-exporter:<ipmi-version>
docker tag $REGISTRY/sn-xrdu-exporter:<xrdu-version> $INTERNAL_REGISTRY/sn-xrdu-exporter:<xrdu-version>
docker push $INTERNAL_REGISTRY/sn-ipmi-exporter:<ipmi-version>
docker push $INTERNAL_REGISTRY/sn-xrdu-exporter:<xrdu-version>
```

### 4. Install / Upgrade from the local chart package

```bash theme={null}
helm upgrade --install sambanova-hardware-monitoring \
  sambanova-hardware-monitoring-$CHART_VERSION.tgz \
  --namespace $NAMESPACE --create-namespace \
  --set global.imageRegistry=$INTERNAL_REGISTRY \
  -f values.yaml
```

***

After the upgrade, re-run the Verification checks to confirm all pods are still `Running` and metrics are flowing.

If the upgrade fails or pods become unhealthy, roll back:

```bash theme={null}
helm history sambanova-hardware-monitoring -n $NAMESPACE
helm rollback sambanova-hardware-monitoring -n $NAMESPACE   # restores the previous release
```

***

## What's next

<CardGroup cols={2}>
  <Card title="Monitoring metrics reference" icon="table" href="./monitoring-metrics">
    Full catalog of all Prometheus metrics exposed by each exporter
  </Card>

  <Card title="Building dashboards in Grafana" icon="chart-line" href="./building-dashboards">
    Connect Grafana and build a complete hardware monitoring dashboard
  </Card>
</CardGroup>