> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sambanova.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Monitoring with Prometheus and Grafana

SambaStack's monitoring and observability reference architecture uses open-source components to provide metrics, logs, and dashboards for on-premises deployments. This guide describes how to deploy Prometheus, Grafana, the Prometheus Operator, and Node Exporter into a monitoring namespace using the official kube-prometheus-stack Helm chart.

> [!NOTE]
>
> **Reference Architecture Note:** This setup uses third-party components (Prometheus, Grafana, etc.). Versions, defaults, and command syntax may change over time. Address any issues not specific to SambaStack to the vendor or project that owns that component.

## Prerequisites

Before you begin, ensure the following requirements are met:

* **kubectl** – Configured with access to your target Kubernetes cluster
* **Helm** (latest version) – Verify with `helm version`
* **jq** – For parsing JSON output during verification
* **Monitoring namespace** – If it does not exist, it will be created in Step 1
* **Storage class** – A valid storage class for Prometheus persistent storage

### Optional prerequisites

* **OpenSearch** – Required only if you want Grafana to visualize logs. If deploying with OpenSearch integration, complete the [OpenSearch deployment](https://docs.sambanova.ai/docs/en/v2.3.2/sambastack/reference-architecture/observability/log-storage.md) first. The `opensearch-initial-admin-password` secret must exist.

> [!NOTE]
>
> **Deployment Order:** If using the full monitoring stack, deploy in this order: OpenSearch → Fluent Bit → Prometheus/Grafana. If you only need metrics (no log visualization), you can deploy Prometheus/Grafana independently.

## Resource requirements

The following are recommended minimum resources for the monitoring stack:

| Component | CPU Request | Memory Request | Storage |
| - | - | - | - |
| Prometheus (per replica) | 500m | 2Gi | 30Gi |
| Grafana | 250m | 512Mi | – |
| Node Exporter (per node) | 100m | 128Mi | – |
| Prometheus Operator | 200m | 256Mi | – |

> [!NOTE]
>
> For larger deployments (100+ nodes or high-cardinality metrics), increase Prometheus memory to 4–8Gi and storage to 50–100Gi. Adjust `retentionSize` accordingly in the values file.

## Architecture overview

| Component | Purpose |
| - | - |
| **Prometheus** | Collects and stores time-series metrics from SambaStack services and cluster nodes |
| **Prometheus Operator** | Manages Prometheus configuration via Kubernetes CRDs (ServiceMonitor, etc.) |
| **Node Exporter** | Exposes node/rack-level host metrics for Prometheus to scrape |
| **Grafana** | Visualizes Prometheus metrics and OpenSearch logs in dashboards |

### Data flows

**Metrics path:**

> SambaStack services / Node Exporter → /metrics → Prometheus → Grafana dashboards

**Logs path (requires OpenSearch):**

> Pods / system logs → Fluent Bit → OpenSearch → Grafana (log panels)

## Deployment steps

### Step 1: Create the monitoring namespace

Skip this step if the namespace already exists from an OpenSearch deployment.

Create the namespace configuration file at `~/.sambastack-observability/monitoring-namespace.yaml`:

```yaml theme={}
# monitoring-namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: monitoring
```

Apply the configuration:

```bash theme={}
kubectl apply -f monitoring-namespace.yaml
```

### Step 2: Add Helm repository

```bash theme={}
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
```

### Step 3: Create Grafana admin credentials secret

Create a secret with base64-encoded username and password at `~/.sambastack-observability/monitoring/grafana-initial-admin-credentials-secret.yaml`:

```yaml theme={}
# grafana-initial-admin-credentials-secret.yaml
apiVersion: v1
data:
  admin-user: <base64-encoded-username>
  admin-password: <base64-encoded-password>
kind: Secret
metadata:
  name: grafana-initial-admin-credentials
  namespace: monitoring
type: Opaque
```

> [!TIP]
>
> To base64 encode a value: `echo -n 'your-value' | base64`

Apply the secret:

```bash theme={}
kubectl -n monitoring apply -f grafana-initial-admin-credentials-secret.yaml
```

### Step 4: Create values file

Create the following file `~/.sambastack-observability/monitoring/prometheus-grafana-values.yaml`. Choose the appropriate configuration based on whether you're integrating with OpenSearch.

> [tab] With OpenSearch Integration
>
> Use this configuration if you have deployed OpenSearch and want log visualization in Grafana.
>
> > [!WARNING]
> >
> > This configuration requires the `opensearch-initial-admin-password` secret to exist in the `monitoring` namespace. See [Log Storage - OpenSearch](https://docs.sambanova.ai/docs/en/v2.3.2/sambastack/reference-architecture/observability/log-storage.md).
>
> > [accordion] View complete values file (with OpenSearch)
> >
> > ```yaml theme={}
> > # prometheus-grafana-values.yaml (with OpenSearch)
> > alertmanager:
> >   enabled: false
> > kubeStateMetrics:
> >   enabled: false
> > kubernetesServiceMonitors:
> >   enabled: false
> > defaultRules:
> >   create: false
> > nodeExporter:
> >   enabled: true
> > prometheusOperator:
> >   serviceMonitor:
> >     selfMonitor: false
> >   nodeSelector: {}
> >   tolerations: []
> >   tls:
> >     enabled: false
> >   admissionWebhooks:
> >     enabled: false
> > prometheus:
> >   enabled: true
> >   serviceMonitor:
> >     selfMonitor: false
> >   prometheusSpec:
> >     replicas: 2
> >     nodeSelector: {}
> >     tolerations: []
> >     serviceMonitorSelector:
> >       matchLabels: {}
> >     ruleSelector:
> >       matchLabels: {}
> >     scrapeConfigSelector:
> >       matchLabels: {}
> >     retentionSize: 25GB
> >     storageSpec:
> >       volumeClaimTemplate:
> >         spec:
> >           storageClassName: <your-storage-class>
> >           resources:
> >             requests:
> >               storage: 30Gi
> > grafana:
> >   enabled: true
> >   defaultDashboardsEnabled: false
> >   serviceMonitor:
> >     enabled: false
> >   admin:
> >     existingSecret: grafana-initial-admin-credentials
> >     userKey: admin-user
> >     passwordKey: admin-password
> >   extraEnvs:
> >   - name: OPENSEARCH_INITIAL_ADMIN_PASSWORD
> >     valueFrom:
> >       secretKeyRef:
> >         name: opensearch-initial-admin-password
> >         key: OPENSEARCH_INITIAL_ADMIN_PASSWORD
> >   plugins:
> >   - grafana-opensearch-datasource
> >   datasources:
> >     datasources.yaml:
> >       apiVersion: 1
> >       datasources:
> >         - name: Prometheus
> >           type: prometheus
> >           access: proxy
> >           url: http://kube-prometheus-stack-prometheus:9090
> >           isDefault: false
> >         - name: OpenSearch-Logs
> >           type: grafana-opensearch-datasource
> >           access: proxy
> >           url: https://opensearch-cluster-master.{{ .Release.Namespace }}.svc.cluster.local:9200
> >           withCredentials: true
> >           basicAuth: true
> >           basicAuthUser: admin
> >           basicAuthPassword: $__env{OPENSEARCH_INITIAL_ADMIN_PASSWORD}
> >           editable: true
> >           readOnly: false
> >           jsonData:
> >             timeField: "@timestamp"
> >             database: logs-7d
> >             tlsSkipVerify: true
> >             logLevelField: log_level
> >             logMessageField: message
> >             version: 3.3.2
> >             versionLabel: OpenSearch 3.3.2
> >             flavor: opensearch
> >             maxConcurrentShardRequests: 5
> >             pplEnabled: true
> >             serverless: false
> > ```

> [tab] Without OpenSearch (Metrics Only)
>
> Use this configuration if you only need Prometheus metrics without log visualization.
>
> > [accordion] View complete values file (metrics only)
> >
> > ```yaml theme={}
> > # prometheus-grafana-values.yaml (metrics only)
> > alertmanager:
> >   enabled: false
> > kubeStateMetrics:
> >   enabled: false
> > kubernetesServiceMonitors:
> >   enabled: false
> > defaultRules:
> >   create: false
> > nodeExporter:
> >   enabled: true
> > prometheusOperator:
> >   serviceMonitor:
> >     selfMonitor: false
> >   nodeSelector: {}
> >   tolerations: []
> >   tls:
> >     enabled: false
> >   admissionWebhooks:
> >     enabled: false
> > prometheus:
> >   enabled: true
> >   serviceMonitor:
> >     selfMonitor: false
> >   prometheusSpec:
> >     replicas: 2
> >     nodeSelector: {}
> >     tolerations: []
> >     serviceMonitorSelector:
> >       matchLabels: {}
> >     ruleSelector:
> >       matchLabels: {}
> >     scrapeConfigSelector:
> >       matchLabels: {}
> >     retentionSize: 25GB
> >     storageSpec:
> >       volumeClaimTemplate:
> >         spec:
> >           storageClassName: <your-storage-class>
> >           resources:
> >             requests:
> >               storage: 30Gi
> > grafana:
> >   enabled: true
> >   defaultDashboardsEnabled: false
> >   serviceMonitor:
> >     enabled: false
> >   admin:
> >     existingSecret: grafana-initial-admin-credentials
> >     userKey: admin-user
> >     passwordKey: admin-password
> >   datasources:
> >     datasources.yaml:
> >       apiVersion: 1
> >       datasources:
> >         - name: Prometheus
> >           type: prometheus
> >           access: proxy
> >           url: http://kube-prometheus-stack-prometheus:9090
> >           isDefault: true
> > ```

> [!NOTE]
>
> Replace `<your-storage-class>` with your cluster's storage class. To find available storage classes: `kubectl get storageclass`

### Component configuration summary

| Component | Status | Purpose |
| - | - | - |
| Prometheus | ✓ Enabled | Metrics storage and querying |
| Prometheus Operator | ✓ Enabled | Manages Prometheus config via CRDs |
| Grafana | ✓ Enabled | Visualization for metrics and logs |
| Node Exporter | ✓ Enabled | Node/rack-level host metrics |
| Alertmanager | ✗ Disabled | Not enabled in this minimal reference |

### Step 5: Install kube-prometheus-stack

Run the Helm install command:

```bash theme={}
helm upgrade --install kube-prometheus-stack \
  prometheus-community/kube-prometheus-stack \
  --version 60.2.0 \
  -n monitoring \
  -f prometheus-grafana-values.yaml
```

This deploys Prometheus, Prometheus Operator, Node Exporter, and Grafana.

### Step 6: Create ServiceMonitor for inference router

Prometheus uses a ServiceMonitor to discover and scrape metrics from the SambaStack inference router. Create the following file `~/.sambastack-observability/monitoring/inference-router-sm.yaml`.

```yaml theme={}
# inference-router-sm.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: inference-router-sm
  namespace: monitoring
spec:
  endpoints:
    - interval: 30s
      port: inference-router
      path: /v1/metrics
  selector:
    matchLabels:
      sambanova.ai/app: inference-router
```

Apply the ServiceMonitor:

```bash theme={}
kubectl apply -f inference-router-sm.yaml
```

## Verification

### Check pod status

Check Prometheus pods:

```bash theme={}
kubectl -n monitoring get pods -l app.kubernetes.io/name=prometheus
```

Expected output:

```text theme={}
NAME                                    READY   STATUS    RESTARTS   AGE
prometheus-kube-prometheus-stack-prometheus-0   2/2     Running   0          5m
prometheus-kube-prometheus-stack-prometheus-1   2/2     Running   0          5m
```

Check Grafana pod:

```bash theme={}
kubectl -n monitoring get pods -l app.kubernetes.io/name=grafana
```

Expected output:

```text theme={}
NAME                                          READY   STATUS    RESTARTS   AGE
kube-prometheus-stack-grafana-XXXXX-XXXXX     1/1     Running   0          5m
```

Check Node Exporter pods:

```bash theme={}
kubectl -n monitoring get pods -l app.kubernetes.io/name=prometheus-node-exporter
```

### Access the UIs

> [tab] Prometheus
>
> Port-forward Prometheus:
>
> ```bash theme={}
> kubectl -n monitoring port-forward svc/kube-prometheus-stack-prometheus 9090:9090 &
> ```
>
> Access at: [http://localhost:9090](http://localhost:9090)
>
> To stop: `pkill -f "port-forward.*9090"`

> [tab] Grafana
>
> Port-forward Grafana:
>
> ```bash theme={}
> kubectl -n monitoring port-forward svc/kube-prometheus-stack-grafana 8080:80 &
> ```
>
> Access at: [http://localhost:8080](http://localhost:8080)
>
> To stop: `pkill -f "port-forward.*8080"`

### Retrieve Grafana credentials

```bash theme={}
kubectl -n monitoring get secret grafana-initial-admin-credentials -o json \
  | jq -r '.data | to_entries[] | "\(.key): \(.value | @base64d)"'
```

### Verify Prometheus targets

1. Access Prometheus UI at [http://localhost:9090](http://localhost:9090)
2. Navigate to **Status → Targets**
3. Verify `node-exporter` targets show as UP
4. Verify `inference-router-sm` target shows as UP (if SambaStack is running)

### Verify Grafana datasources

1. Log in to Grafana at [http://localhost:8080](http://localhost:8080)
2. Navigate to **Connections → Data sources**
3. Verify **Prometheus** datasource shows "Data source is working"
4. If using OpenSearch integration, verify **OpenSearch-Logs** datasource shows "Data source is working"

***

## Success criteria

The installation is complete when:

* Prometheus pods (2 replicas) are running in the `monitoring` namespace
* Grafana pod is running and accessible
* Node Exporter pods are running on all nodes
* Prometheus shows Node Exporter targets as UP
* Grafana accepts the configured admin credentials
* Prometheus datasource in Grafana shows "Data source is working"
* (If configured) OpenSearch datasource in Grafana shows "Data source is working"

## Import Node Exporter dashboard

To visualize node/rack-level metrics from Node Exporter:

**1. Log in to Grafana**

Use the credentials retrieved in the verification section.

**2. Navigate to Import**

Go to **Dashboards → New → Import**.

**3. Import Dashboard**

Enter dashboard ID: `1860` and click **Load**.

**4. Select Datasource**

Select **Prometheus** as the datasource and click **Import**.

This dashboard includes: CPU usage, memory usage, disk I/O, network metrics, and node health.

> [!NOTE]
>
> Dashboard ID 1860 is the community "Node Exporter Full" dashboard. This is ideal for per-rack visibility in SambaStack deployments.

## Configuration reference

| Parameter | Default | Description |
| - | - | - |
| `prometheus.prometheusSpec.replicas` | 2 | Number of Prometheus replicas for HA |
| `prometheus.prometheusSpec.retentionSize` | 25GB | Maximum storage before oldest data is deleted |
| `prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage` | 30Gi | PVC size for Prometheus data |
| `grafana.admin.existingSecret` | grafana-initial-admin-credentials | Secret containing admin credentials |
| `nodeExporter.enabled` | true | Deploy Node Exporter DaemonSet |

## Troubleshooting

### Prometheus pods stuck in Pending

**Symptom:** Prometheus pods remain in `Pending` status.

**Cause:** PersistentVolumeClaim cannot be fulfilled.

**Solution:**

```bash theme={}
# Check PVC status
kubectl -n monitoring get pvc

# Verify storage class exists
kubectl get storageclass
```

### Grafana shows "Data source is not working" for OpenSearch

**Symptom:** OpenSearch datasource test fails in Grafana.

**Possible causes:**

1. **OpenSearch not deployed:** Deploy OpenSearch first. See [Log Storage - OpenSearch](https://docs.sambanova.ai/docs/en/v2.3.2/sambastack/reference-architecture/observability/log-storage.md).

2. **Secret missing:** Verify the secret exists:
   ```bash theme={}
   kubectl -n monitoring get secret opensearch-initial-admin-password
   ```

3. **OpenSearch not ready:** Check OpenSearch pod status:
   ```bash theme={}
   kubectl -n monitoring get pod opensearch-cluster-master-0
   ```

### ServiceMonitor not scraping targets

**Symptom:** Custom ServiceMonitor targets don't appear in Prometheus.

**Solution:** Verify the ServiceMonitor is in the `monitoring` namespace and labels match:

```bash theme={}
kubectl -n monitoring get servicemonitor
kubectl -n monitoring describe servicemonitor inference-router-sm
```

### Node Exporter pods not running on all nodes

**Symptom:** Fewer Node Exporter pods than cluster nodes.

**Cause:** Nodes may have taints that prevent scheduling.

**Solution:** Add tolerations to the Node Exporter configuration or remove node taints.

## Next steps

After Prometheus and Grafana are running:

1. **Create custom dashboards** – Build dashboards for SambaStack-specific metrics like inference latency, QPS, and accelerator utilization.

2. **Add more ServiceMonitors** – Create ServiceMonitors for other SambaStack components that expose Prometheus metrics.

3. **Enable alerting** – Configure Alertmanager for production monitoring. Update the values file to set `alertmanager.enabled: true`.

4. **Explore logs** – If OpenSearch and Fluent Bit are deployed, use Grafana's Explore feature to query the `logs-7d` index.

## Cleanup

To remove the monitoring stack from your cluster:

Uninstall the Helm release:

```bash theme={}
helm uninstall kube-prometheus-stack -n monitoring
```

Delete the Grafana credentials secret:

```bash theme={}
kubectl delete secret grafana-initial-admin-credentials -n monitoring
```

Delete Prometheus PersistentVolumeClaims to free storage:

```bash theme={}
kubectl -n monitoring delete pvc -l app.kubernetes.io/name=prometheus
```

Delete the ServiceMonitor:

```bash theme={}
kubectl delete servicemonitor inference-router-sm -n monitoring
```

> [!WARNING]
>
> Deleting the PVCs permanently removes all stored metrics data. Ensure you have exported any important metrics before proceeding.

> [!NOTE]
>
> If you're removing the entire monitoring stack including OpenSearch and Fluent Bit, you can delete the entire namespace: `kubectl delete namespace monitoring`. This removes all resources but is irreversible.