> ## Documentation Index > Fetch the complete documentation index at: https://docs.sambanova.ai/docs/llms.txt > Use this file to discover all available pages before exploring further. # Monitoring overview SambaRack hardware monitoring reports the physical condition of the rack: power draw at the PDU, fan speed and temperature at each XRDU, and RDU utilization and health on each host module. Metrics are labeled by node and rack, so you can see the whole fleet or drill into a single RDU. * **Track how your RDUs are being used**. Monitor tile activity per RDU to find idle capacity, balance workloads across nodes and racks, and plan capacity. * **Catch faults and errors early**. Track platform fault and error counts per RDU and alert when they increase, so you act on early warning signs rather than after a failure. * **Keep your hardware within safe thermal limits**. Follow temperatures and fan speeds across the rack to detect cooling problems before they cause throttling or hardware stress. * **Track rack power consumption**. Follow power draw across your racks to understand energy use and plan power and cooling capacity. ## What hardware monitoring is Hardware monitoring collects telemetry from the physical infrastructure a SambaRack is built from — power distribution units, baseboard management controllers, and the RDUs themselves. It answers questions about the machine. | Layer | Question it answers | Where it comes from | | :- | :- | :- | | Hardware | Is this rack drawing more power than its circuit allows? Is a fan failing? | The hardware exporters described on these pages | | Workload | Is inference slow? Is the queue growing? Which model endpoint is erroring? | [SambaStack observability](../../../reference-architecture/observability/overview) | The two are separate signal sets with separate exporters, but they share one Prometheus and one Grafana. That is deliberate: most real incidents are diagnosed by looking at both at once. ## Components | Component | Tool | Description | | :- | :- | :- | | RDU metrics | [SambaNova exporter](exporter-sambanova) | Reports RDU health and throughput from each host module. | | Power metrics | [PDU exporter](exporter-pdu) | Reports power draw from the rack PDUs. | | Thermal metrics | [XRDU exporter](exporter-xrdu) | Reports fan speed and temperature from the XRDU BMCs. | | Deployment | [`sambanova-hardware-monitoring` Helm chart](monitoring-install) | Deploys the exporters and makes them scrapeable by Prometheus. Installs the dashboards by default. | | Visualization | [Reference dashboards](monitoring-reference-dashboards) | Three example Grafana dashboards, one per exporter. | ## Architecture ```text theme={} VISUALIZATION +---------------------------------------------------------------------+ | Grafana | | (Dashboards) | +---------------------------------------------------------------------+ | v +---------------------------------------------------------------------+ | Prometheus | | (kube-prometheus-stack, in cluster) | +---------------------------------------------------------------------+ | ServiceMonitor | Probe | Probe v v v +-----------------+ +---------------------+ +---------------------+ | SambaNova | | PDU exporter | | XRDU exporter | | exporter :9101 | | :9820 | | :9821 | | (host, systemd) | | (Deployment) | | (Deployment) | +-----------------+ +---------------------+ +---------------------+ | local | management net | management net v v v +-----------------+ +---------------------+ +---------------------+ | RDUs on the | | Rack PDUs | | XRDU BMCs | | host module | | | | | +-----------------+ +---------------------+ +---------------------+ | SAMBARACK HARDWARE | +---------------------------------------------------------------------+ ``` The SambaNova exporter is a host process on the machine it monitors. Prometheus scrapes each host directly, as it would a node exporter. The PDU and XRDU exporters are **multi-target** exporters. A PDU and a BMC have no operating system to run an agent on, so one exporter process answers for every device. Prometheus names the device in the `target` query parameter of each scrape (`/probe?target=
`). Adding a rack means adding its device addresses to your values file and running `helm upgrade`. The exporters themselves do not change. ## Operating model * **Prometheus-native** — The exporters publish standard Prometheus exposition format. Any scraper that reads that format can collect them. * **Kubernetes-native deployment** — The chart uses prometheus-operator CRDs (`Probe`, `ServiceMonitor`) and follows the `kube-prometheus-stack` values convention. * **Customer-operated** — You install, upgrade, and operate the stack. SambaNova provides the exporters, the chart, the reference architecture, and the dashboards. * **Dashboards are a starting point** — The three bundled dashboards demonstrate the metrics. Copy the panels you need into the dashboards your team already runs. ## Integration with other monitoring platforms | Platform | Support | | :- | :- | | Prometheus and Grafana | Supported. This is the reference implementation. | | OpenTelemetry | The exporters are OTEL-compatible: an OTEL collector can scrape them natively. The Helm chart has no OTEL CRDs, so write your own collector scrape configuration against the exporter Services. | | Zabbix, Datadog, Splunk, and others | No packaged integration today. Point your collector at the exporter endpoints, or at Prometheus `remote_write`. | Nothing in the exporters is proprietary to Prometheus. The chart's Prometheus-specific parts are the `Probe` and `ServiceMonitor` objects, which you can leave disabled. ## Prerequisites If you have not installed a monitoring stack already, set up Prometheus and Grafana first using the [Prometheus and Grafana reference architecture](../../../reference-architecture/observability/monitoring-prom). | Requirement | Description | | :- | :- | | Kubernetes cluster | 1.35 or later | | Helm 3 | Helm 2 cannot install an OCI chart | | prometheus-operator CRDs | `Probe` and `ServiceMonitor`. Provided by `kube-prometheus-stack`. | | A running Prometheus | Must select the chart's objects by label. See [Install monitoring](monitoring-install). | | Grafana | Only for the reference dashboards | | Network route | From cluster nodes to the management network, for the PDU and XRDU exporters | | SambaRack Manager (`snctl`) | Supplies device addresses and the XRDU BMC password. See [SambaRack Manager overview](../srm-overview). | | SambaNova Runtime | 1.26.3-1 or later, on each host module. Ships the `sambanova-runtime-exporter` package. | | Registry credentials | Contact SambaNova for a service account key. | ## Next steps Hardware monitoring supports connected and air-gapped installs. [Install monitoring](monitoring-install) walks through both in order, starting with enabling the SambaNova exporter on every host module. Before you install, confirm the cluster nodes can reach the PDUs and XRDU BMCs on the management network. See [Device reachability](monitoring-troubleshooting#device-reachability). * [Install monitoring](monitoring-install) * [SambaNova exporter](exporter-sambanova), [PDU exporter](exporter-pdu), and [XRDU exporter](exporter-xrdu) * [Reference dashboards](monitoring-reference-dashboards)