Configure SambaNova Runtime components

Administrators with superuser privileges can stop and restart the SambaNova Daemon (SND) or change runtime log levels. Administrators might perform other tasks, such as resetting RDUs, as described in Runtime Troubleshooting.

Several tasks on this doc page require administrator privileges.

Manage SND

The SambaNova Daemon (SND) starts automatically when the host boots. When SND starts, it initializes the hardware resources and loads the driver. You can then run applications on the hardware.

All users can see SND status, as follows:

Command Usage Effect

status

$ systemctl status snd

Check the SND service

Administrators can use these systemctl(1) commands to manage SND:

Command Usage Effect

start

$ sudo systemctl start snd

Start the SND service

stop

$ sudo systemctl stop snd

Stop the SND service

disable

$ sudo systemctl disable snd

Disable SND service and de-register SND auto-start on boot

enable

$ sudo systemctl enable snd

Enable SND service and register SND auto-start on boot

edit

$ sudo systemctl edit snd

Edit environment configurations. Requires a service restart to take effect

Change Runtime Log Levels

The Runtime package supports several logs. This section discusses how to change log levels.

Change application log levels

Application logs are sent to /var/log/sambaflow/runtime/sn.log. ERR level application messages are also printed to the console.

We support the following application log levels:

  • NOTICE

  • WARNING

  • ERR

  • CRIT

  • ALERT

  • EMERG

To change the log level, set the SF_RNT_LOG_LEVEL environment variable before calling your SambaFlow or SambaRuntime script or binary. For example:

SF_RNT_LOG_LEVEL=INFO python3 my_app.py
SF_RNT_LOG_LEVEL=INFO ./my_app.bin
If the application is run with sudo, then the levels can only be changed with the environment variable prefixing the command, as shown in the example above. Exported environment variables are not propagated to applications that are run with sudo.

Change SND log levels

You need superuser privileges to view SND logs or make changes to them.

SND logs are sent /var/log/sambaflow/runtime/snd.log. To set snd log levels, define SF_RNT_LOG_LEVEL in the override.conf file. The override.conf` file persists even after reinstalling SND or rebooting the system, so you need to do this step only when you want to change the SND log level.

  1. Open the file for edit.

    systemctl edit snd

  2. Add an environment variable, as follows:

    [Service]
Environment="SF_RNT_LOG_LEVEL=<log_level>"

    Replace <log_level> with your desired log level. You can use the same levels as for application logs.

  3. Restart SND

    systemctl restart snd

Change kernel log levels

Kernel logs appear in both dmesg and /var/log/kern.log. Administrators can change the kernel log level.

To get the default log level:

modprobe rdu && modprobe rdu_mem_map && modprobe rdu_peer_mem_client && sudo systemctl start snd

To change the log level:

modprobe rdu rdu_log_level=my_level && modprobe rdu_mem_map && modprobe rdu_peer_mem_client && sudo systemctl start snd

For my_level, specify the integer associated with the log level that you want:

  • 127 — INFO (default)

  • 63 — WARNING

  • 31 — ERR

snconfig tool

The snconfig (SambaNova Configuration) tool displays, queries, configures, and manages system resources on a DataScale system.

snconfig is part of the sambaflow package and can be found at /opt/sambaflow/bin/snconfig.

To see available options, run snconfig --help.

sntilestat tool

The sntilestat tool displays the status and utilization of each tile within each Reconfigurable Dataflow Unit (RDU) in each XRDU chassis in the system.

sntilestat is part of the sambaflow package and can be found at /opt/sambaflow/bin/sntilestat.

For an introduction and some examples for sntilestat, run man sntilestat.