SambaNova messages and logs

When you compile or run a model, the output to screen is fairly detailed. This doc page explains:

What to look out for in compile and run output.
Where to find logs that are generated for different parts of the product, and what each log does.

Standard output

When you compile or run a Python application, the application itself and the different layers in the SambaFlow stack each send messages to standard output, by default to screen. Each layer also saves messages to a separate log file.

Standard output messages during compile

The output you see during compile is currently fairly detailed to help SambaNova customer support understand the processes and potential errors if there are problems.

You can ignore many of the stdout messages.

Message types:

[info ] messages usually provide details that might help SambaNova Customer Support. You can ignore them.
[warning Samba] messages show potential problems that are not serious enough to stop compilation. Warnings are often shown in color and might have information about potential improvements to your application.
Other [warning ] messages are meant for SambaNova customer support. You can ignore them.
[error ] messages usually means that compilation does not succeed. However, the compiler retries compilation after certain kinds of errors.

Check if a PEF file is generated to determine if compilation was successful.

Information you can use:

Information about the application log, which is stored in sn.log is shown close to the beginning of the compile. Here’s an example:
```
Log ID initialized to: [snuser1][python][3363382] at /var/log/sambaflow/runtime/sn.log
```
Information about compilation success and the location of the PEF file is show at the end of the compile. Here’s an example:
```
[info    ]: Plasma compile succeeded.
[info    ] [prism]: PEF file /home/snuser1/app-test/out/logreg//logreg.pef created
```

Standard output messages during run (training or inference)

You can control stdout message detail during run using the --log-level argument. In most cases, setting the log level to error makes sense, as in the following example.

python logreg.py run --num-epochs 2 --pef=out/logreg-pef/logreg-pef.pef --log-level error

The output shows:

The location of the application log (sn.log)
The result of the run.

Here’s an example screenshot.

Logs

When you compile a model or run training or inference, SambaNova creates several log files. Some files are primarily for use by customer support, others can help you find reasons for problems with your model.

Logs overview

This overview table includes locations of different logs and links to additional information.

Log Location Description

Log	Location	Description
Application log	`/var/log/sambaflow/runtime/sn.log`	Userspace logs generated by the models that are running on the machine (typically Python apps). If a model fails to run, check here first. Information about the log ID is in the output to screen, for example, `Log ID initialized to: [snuser1][python][3363382] at /var/log/sambaflow/runtime/sn.log`
SambaNova daemon log	/var/log/sambaflow/runtime/snd.log	Information about hardware initialization, event handling, and error handling. See SND logs.
dmesg(1) logs		Kernel logs that are generated by the RDU driver. Check here first if you suspect a Linux or hardware issue. See dmesg(1) logs.
compiler logs	Output directory of the compile	The messages that are output to screen are also saved to several separate log files in same directory as your PEF file. See Compiler logs.

Application log

/var/log/sambaflow/runtime/sn.log

Userspace logs generated by the models that are running on the machine (typically Python apps). If a model fails to run, check here first.

Information about the log ID is in the output to screen, for example, Log ID initialized to: [snuser1][python][3363382] at /var/log/sambaflow/runtime/sn.log

SambaNova daemon log

/var/log/sambaflow/runtime/snd.log

Information about hardware initialization, event handling, and error handling. See SND logs.

dmesg(1) logs

Kernel logs that are generated by the RDU driver. Check here first if you suspect a Linux or hardware issue. See dmesg(1) logs.

compiler logs

Output directory of the compile

The messages that are output to screen are also saved to several separate log files in same directory as your PEF file. See Compiler logs.

Compiler logs

The messages that the compiler sends to stdout are also saved to several different log files.

The compiler logs are primarily for use by SambaNova customer support.

Logs include <app-name>.samba.log for application and PyTorch messages and <app-name>.mac.log for lower-level compiler information. The information primarily useful for SambaNova customer support.

Logs for training or inference run

The messages that SambaFlow sends to stdout when you perform a training or inference run are stored in /out/<app-name>.samba.log.

Common errors

This section of common errors will be updated as to include errors that you can resolve yourself. For all errors nor listed here, contact Customer Support.

Samba errors

A Samba error occurs at the highest level of the stack, either in the application or in the level immediately below.

Action: If you see an error that includes Samba, check your application code. If you can’t resolve the problem by making changes to the application code, contact SambaNova Customer Support.

[Prism too many PMUs] error

If you see a [Prism too many PMUs] error, it means that the code doesn’t fit on a single chip.

Action: Make these changes, then recompile:

Decrease the size of the tensors in your application.
If you’re working with large images, use convolutional tiling to resolve the issue.