Configure SambaNova Runtime components

Administrators with superuser privileges can stop and restart the SambaNova Daemon (SND) or change runtime log levels. Administrators might perform other tasks, such as resetting RDUs, as described in Runtime Troubleshooting.

Manage SND

The SambaNova Daemon (SND) starts automatically when the host boots. When SND starts, it initializes the hardware resources and loads the driver. You can then run applications on the hardware.

All users can see SND status, as follows:

Command Usage Effect


$ systemctl status snd

Check the SND service

Administrators can use these systemctl(1) commands to manage SND:

Command Usage Effect


$ sudo systemctl start snd

Start the SND service


$ sudo systemctl stop snd

Stop the SND service


$ sudo systemctl disable snd

Disable SND service and de-register SND auto-start on boot


$ sudo systemctl enable snd

Enable SND service and register SND auto-start on boot


$ sudo systemctl edit snd

Edit environment configurations. Requires a service restart to take effect

Change Runtime Log Levels

The Runtime package supports several logs. This section discusses how to change log levels.

Change application log levels

Application logs are sent to /var/log/sambaflow/runtime/sn.log. ERR level application messages are also printed to the console.

We support the following application log levels:



  • ERR

  • CRIT



To change the log level, set the SF_RNT_LOG_LEVEL environment variable before calling your SambaFlow or SambaRuntime script or binary. For example:

SF_RNT_LOG_LEVEL=INFO ./my_app.bin
If the application is run with sudo, then the levels can only be changed with the environment variable prefixing the command, as shown in the example above. Exported environment variables are not propagated to applications that are run with sudo.

Change SND log levels

You need superuser privileges to view SND logs or make changes to them.

SND logs are sent /var/log/sambaflow/runtime/snd.log. To set snd log levels, define SF_RNT_LOG_LEVEL in the override.conf file. The override.conf` file persists even after reinstalling SND or rebooting the system, so you need to do this step only when you want to change the SND log level.

  1. Open the file for edit.

    systemctl edit snd
  2. Add an environment variable, as follows:


    Replace <log_level> with your desired log level. You can use the same levels as for application logs.

  3. Restart SND

    systemctl restart snd

Change kernel log levels

Kernel logs appear in both dmesg and /var/log/kern.log. Administrators can change the kernel log level.

To get the default log level:

modprobe rdu && modprobe rdu_mem_map && modprobe rdu_peer_mem_client && sudo systemctl start snd

To change the log level:

modprobe rdu rdu_log_level=my_level && modprobe rdu_mem_map && modprobe rdu_peer_mem_client && sudo systemctl start snd

For my_level, specify the integer associated with the log level that you want:

  • 127 — INFO (default)

  • 63 — WARNING

  • 31 — ERR

Use GraphCLI to examine graphs

GraphCLI allows users to stop/break at a specified graph state and get information from a running graph on the RDU. The list of debug commands below should explain what each command does

  1. Run an application on a DataScale host.

    • Ensure that the SNCLI_SERVER environment variable is set when the app is launched

      $ SNCLI_SERVER="localhost:50052" python /path/to/app <args>
  2. Launch the GraphCLI client in another window:

    $ /opt/sambaflow/diag/
  3. Connect the GraphCLI client to a graph. The graph is generated by the compiler when you compile the model.

    $ connect graph localhost:50052
  4. Get graph information or disconnect, for example:

    $ show
    $ dump symbol <symbol name> vsnode <ID>
    $ disconnect

Graph CLI debug commands

Command Description


Connect to a remote target


Disconnect from a remote target


Exit from Graph CLI

dump symbol <symbol name> vsnode <ID>

Dump graph symbol contents on a particular vsnode


Show graph information, including symbols, arguments, host functions, resources, profile

break section <section_id> state <state_id>

Add a breakpoint at the specified section:state

delete section <section_id> state <state_id>

Delete a breakpoint at the specified section:state


List all breakpoints


Go to the next graph FSM state


Continue graph FSM execution


List all valid section:state combinations in the current run


Context-sensitive help

enter / space


up / down

Move through commands in history


Delete and abort the current command