> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sambanova.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# SambaNova exporter

The SambaNova exporter (`sambanova_exporter`) publishes RDU telemetry from each SambaRack host module: platform health, and tile, DDR, HBM, and PCIe performance. Unlike the PDU and XRDU exporters, it runs on the host it monitors, and Prometheus scrapes each host directly.

| Property | Value |
| :- | :- |
| Port | 9101 by default. Set `http_port` in the configuration file. |
| Runs as | A systemd service, `sambanova_exporter`, on each host module |
| Installed by | The `sambanova-runtime-exporter` package, with SambaNova Runtime. The Helm chart does not install it. |
| Default state | Disabled. Enable and start it after installing or upgrading the package. |
| Requires | The host module up, the package installed, and the service active. The chart only makes it scrapeable; it cannot start it. |
| Configuration file | `/opt/sambaflow/etc/sambanova_exporter.conf`, written by the package. |

## Install and run the service

For the full procedure in context, see [Install monitoring, Step 1](monitoring-install#step-1-enable-the-sambanova-exporter-on-every-host). As a privileged user, on each host module:

```bash theme={}
yum install -y sambanova-runtime-exporter
systemctl enable sambanova_exporter
systemctl start sambanova_exporter
```

Verify. The service must report `active` and the metrics endpoint must return `200`:

```bash theme={}
systemctl is-active sambanova_exporter
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:9101/metrics
```

To stop and disable it:

```bash theme={}
systemctl stop sambanova_exporter
systemctl disable sambanova_exporter
```

To use a different configuration file, run `systemctl edit sambanova_exporter.service` and set the `SAMBANOVA_EXPORTER_CONFIG_FILE` environment variable. The command line option `-c <file>` overrides the variable. If the file is not found, the exporter ignores it. If the file exists but cannot be read or parsed, the exporter fails.

## How the chart exposes it

The chart does not deploy the exporter. It makes the host port scrapeable:

* A placeholder DaemonSet on the host network gives a Kubernetes Service one endpoint per RDU node. Its readiness probe follows the host service, so a dead exporter leaves the endpoint list.
* A `ServiceMonitor` scrapes the Service and adds two labels the exporter does not publish: `node`, from the node name, and `rack`, from the `rackMap` value you declare. Rack is not discoverable, so without `rackMap` the series have `node` but no `rack`.

```yaml theme={}
sambanova-exporter:
  enabled: true
  serviceMonitor:
    enabled: true
    interval: 15s
    rackMap:
      <node-name>: "<rack-id>"
```

The chart deploys the placeholder DaemonSet by default, but `serviceMonitor.enabled` defaults to `false`. Set it to `true`, or Prometheus does not scrape RDU metrics.

## Collectors

Each collector can be turned on or off independently. All are on by default, and the HBM collector starts only on platforms that include HBM.

| Collector | Publishes | Enable setting |
| :- | :- | :- |
| Platform | Platform info, timestamps, faults, errors | `enable_platform_collector` |
| Tile | Tile frequency, state percentages, owning process | `enable_tile_collector` |
| DDR | DDR channel request and byte rates | `enable_ddr_collector` |
| HBM | HBM channel-group request and byte rates | `enable_hbm_collector` |
| PCIe | PCIe request and byte rates | `enable_pcie_collector` |

## Metrics

### Platform collector

Series with no labels below have one entry per platform.

| Metric | Labels | Description |
| :- | :- | :- |
| `sambanova_platform_info` | `platform_name`, `runtime_version`, `samba_runtime_version`, `bmc_version`, `rduc_version`, `pef_version`, `snml_version` | Platform identity and component versions. Always `1`. |
| `sambanova_platform_boot_timestamp_seconds` | none | When the system booted, in seconds since the Unix epoch |
| `sambanova_platform_snd_active_timestamp_seconds` | none | When `snd.service` last became active. `0.0` if it has not yet, `-1.0` if it cannot be determined. |
| `sambanova_platform_snd_inactive_timestamp_seconds` | none | When `snd.service` last became inactive. `0.0` if it has not yet, `-1.0` if it cannot be determined. |
| `sambanova_platform_fault_count` | none | Component faults reported by SambaNova system management |
| `sambanova_platform_error_count` | none | Component errors reported by SambaNova system management |
| `sambanova_platform_data_timestamp_seconds` | none | Time of the last successful monitoring sample |
| `sambanova_platform_collect_seconds` | none | Elapsed seconds of the latest collection |

<Note>
  A non-zero `sambanova_platform_error_count` is normal. Watch how fast it grows, not its value.
</Note>

### Tile collector

Labels: `xrdu`, `rdu`, `tile`.

| Metric | Description |
| :- | :- |
| `sambanova_tile_frequency_mhz` | Clock frequency of the tile |
| `sambanova_tile_idle_percent` | Percent of time the tile was idle |
| `sambanova_tile_pload_percent` | Percent of time in Program Load state |
| `sambanova_tile_exec_percent` | Percent of time in Execute state |
| `sambanova_tile_aload_percent` | Percent of time in Argument Load state |
| `sambanova_tile_pid` | Process ID of the controlling process, if the tile is allocated |

Series with one entry per platform:

| Metric | Description |
| :- | :- |
| `sambanova_tile_offline_total` | Monitored tiles that were not present and online, or did not respond, in the last sample |
| `sambanova_tile_data_timestamp_seconds` | Time of the last successful sample |
| `sambanova_tile_collect_seconds` | Elapsed seconds of the latest collection |

<Tip>
  To get RDU utilization across a rack, subtract the mean idle percentage from 100, and filter out the sentinel value: `100 - avg by (rack) (sambanova_tile_idle_percent != -1)`.
</Tip>

### DDR collector

Labels: `xrdu`, `rdu`, `ddrch`.

| Metric | Description |
| :- | :- |
| `sambanova_ddrch_rd_requests` | DDR channel read requests per second |
| `sambanova_ddrch_wr_requests` | DDR channel write requests per second |
| `sambanova_ddrch_rdB_bytes` | DDR channel read bytes per second |
| `sambanova_ddrch_wrB_bytes` | DDR channel write bytes per second |

Series with one entry per platform:

| Metric | Description |
| :- | :- |
| `sambanova_ddrch_offline_total` | Monitored components that were not present and online, or did not respond, in the last sample |
| `sambanova_ddr_data_timestamp_seconds` | Time of the last successful sample |
| `sambanova_ddr_collect_seconds` | Elapsed seconds of the latest collection |

Throughput metrics report `-1.0` for a component that was offline. See [Sentinel value](#sentinel-value).

### HBM collector

Available on platforms that include HBM. Labels: `xrdu`, `rdu`, `hbm`, `hbmch_grp`.

| Metric | Description |
| :- | :- |
| `sambanova_hbmch_grp_rd_requests` | HBM channel-group read requests per second |
| `sambanova_hbmch_grp_wr_requests` | HBM channel-group write requests per second |
| `sambanova_hbmch_grp_rdB_bytes` | HBM channel-group read bytes per second |
| `sambanova_hbmch_grp_wrB_bytes` | HBM channel-group write bytes per second |

Series with one entry per platform:

| Metric | Description |
| :- | :- |
| `sambanova_hbmch_grp_offline_total` | Monitored components that were not present and online, or did not respond, in the last sample |
| `sambanova_hbm_data_timestamp_seconds` | Time of the last successful sample |
| `sambanova_hbm_collect_seconds` | Elapsed seconds of the latest collection |

Throughput metrics report `-1.0` for a component that was offline. See [Sentinel value](#sentinel-value).

### PCIe collector

Labels: `xrdu`, `rdu`, `pcie`. `D2R` and `R2D` name who starts the operation. `D2R` is a request from the local RDU (the device) to a remote RDU or host. `R2D` is a request from a remote host or RDU to the local RDU.

| Direction of data | Metric | Description |
| :- | :- | :- |
| Remote to local RDU | `sambanova_pcie_d2r_rd_requests` | D2R read requests per second |
| | `sambanova_pcie_d2r_rdB_bytes` | D2R read bytes per second |
| | `sambanova_pcie_r2d_wr_requests` | R2D write requests per second |
| | `sambanova_pcie_r2d_wrB_bytes` | R2D write bytes per second |
| Local RDU to remote | `sambanova_pcie_d2r_wr_requests` | D2R write requests per second |
| | `sambanova_pcie_d2r_wrB_bytes` | D2R write bytes per second |
| | `sambanova_pcie_r2d_rd_requests` | R2D read requests per second |
| | `sambanova_pcie_r2d_rdB_bytes` | R2D read bytes per second |

Series with one entry per platform:

| Metric | Description |
| :- | :- |
| `sambanova_pcie_offline_total` | Monitored components that were not present and online, or did not respond, in the last sample |
| `sambanova_pcie_data_timestamp_seconds` | Time of the last successful sample |
| `sambanova_pcie_collect_seconds` | Elapsed seconds of the latest collection |

Throughput metrics report `-1.0` for a component that was offline. See [Sentinel value](#sentinel-value).

## Labels

| Label | Values | Description |
| :- | :- | :- |
| `xrdu` | `XRDU_0` to `XRDU_7` | The host's own index of the XRDU, in PCIe enumeration order. Not the same as the `xrdu` label on `xrdu_*` series. See [XRDU exporter](exporter-xrdu#labels). |
| `rdu` | An index | RDU number within the XRDU |
| `tile` | An index | Tile within the RDU |
| `ddrch` | An index | DDR channel |
| `hbm`, `hbmch_grp` | An index | HBM stack and channel group |
| `pcie` | An index | PCIe interface |
| `platform_name`, `*_version` | Identity strings | On `sambanova_platform_info` only |
| `node` | A Kubernetes node name | Added by the chart's ServiceMonitor |
| `rack` | Your rack ID | Added by the chart's ServiceMonitor, from `rackMap`. Absent if you do not set `rackMap`. |
| `instance` | `IP:9101` | Added by Prometheus. There is no hostname label, so use `sambanova_platform_info` to map an IP to a machine. |

## Sentinel value

All throughput and tile metrics (except `sambanova_tile_pid`) report `-1.0` for any XRDU, RDU, or component that was not present and online, or did not respond, at the last sample. `avg` includes `-1.0` and `min` returns it, so filter it out in every aggregation:

```promql theme={}
avg by (node) (sambanova_tile_exec_percent != -1)
```

`sambanova_tile_pid` lists a tile only if it was allocated to a process at the last sample.

## Configuration

The file is in INI format: `[section]` headers followed by `option = value` lines. Section names are case sensitive. Option names and values are not. Leave a value empty to take the default.

### `[sambanova_exporter]`

| Option | Default | Description |
| :- | :- | :- |
| `log_level` | `info` | `debug`, `info`, `warning`, or `error`. A global default that collectors can override. Messages logged while the file is being read always use `info`. |
| `http_port` | `9101` | Port of the exporter's HTTP server |
| `update_interval` | `10` | Seconds between collector updates. Minimum `1`. A scrape always returns the latest update. |
| `include_rdus` | all | XRDU and RDU components to monitor. Mutually exclusive with `exclude_rdus`. |
| `exclude_rdus` | none | XRDU and RDU components not to monitor. Mutually exclusive with `include_rdus`. |
| `enable_<name>_collector` | `true` | For `platform`, `tile`, `ddr`, `pcie`, and `hbm` |

`include_rdus` and `exclude_rdus` take a comma-separated list of `xrdu/rdu` pairs. Ranges and `*` are allowed. For example, `include_rdus = 0-2/*,3/0` monitors every RDU on XRDU 0 to 2, and RDU 0 on XRDU 3. `exclude_rdus = 2/1,3/*` skips RDU 1 on XRDU 2, and every RDU on XRDU 3. The offline counts cover only monitored RDUs.

### Collector sections

Each of `[platform_collector]`, `[tile_collector]`, `[ddr_collector]`, `[hbm_collector]`, and `[pcie_collector]` accepts `log_level`. Every one except the platform collector also accepts `update_interval`, `include_rdus`, and `exclude_rdus`, which override the global values for that collector.

`[tile_collector]` also accepts:

| Option | Default | Description |
| :- | :- | :- |
| `state_frequency` | `19` | Times per second the tile monitoring thread samples tile state. Minimum `1`. Percentages are based on `state_frequency * update_interval` samples. |

### Example

```ini theme={}
[sambanova_exporter]
log_level = info
http_port = 9101
update_interval = 10
exclude_rdus = 3/*
enable_pcie_collector = false

[tile_collector]
state_frequency = 19
```

## Next steps

* [Install monitoring](monitoring-install)
* [Reference dashboards](monitoring-reference-dashboards)
* [Troubleshooting](monitoring-troubleshooting)