Run SambaTune and examine reports

After installation, you can run SambaTune from the command line.

You have to run SambaTune with the application yaml file as input before you can see results of the performance analysis.

Overview

Running Sambatune includes running sambatune and running sambatune_ui.

  1. First you run sambatune and pass in a YAML file for your model. See Run the sample application.

  2. Then you can run the sambatune_ui command. See Run the SambaTune GUI.

  3. Finally, you can explore with the SambaTune GUI and examine SambaTune reports.

Run the sample application

A sample application, linear_net.py is included with your installation at /opt/sambaflow/apps/micros/linear_net.py. The application requires that the sambaflow-apps-micros package is installed.

To run the linear_net.py sample application:

  1. Log in to the Linux console of a host that is attached to the DataScale hardware.

  2. Run the application. You have several options:

    • Run the application in benchmarking mode (the default):

      $ sambatune linear_net.yaml

      where linear.yaml is a user-specified configuration file that is included in the sambaflow-apps-micros package. Here’s an example:

      app: /path/to/linear.py
model-args: -b 128 -mb 64 --in-features 512 --out-features 128
compile-args: compile --plot
run-args: -n 10000

    • Run the application in instrument-only mode. The space after -- is required.

      $ sambatune --modes instrument -- /opt/sambaflow/sambatune/configs/linear_net.yaml

    • Run in all modes. The space after -- is required.

      $ sambatune --modes benchmark instrument run -- /opt/sambaflow/sambatune/configs/linear_net.yaml

Run sambatune --help for a list of all options. See SambaTune input arguments for details on configuration options.

Understand how SambaTune collects data

When you run the sample application:

  1. SambaTune compiles the application with the user-specified model-args , compile-args and SambaFlow-supported instrumentation flags.

  2. After successful compile, SambaTune:

    1. Runs the application on the RDU and collects performance data.

    2. Runs the application in benchmark mode with user-specified run-args to collect latency, throughput, and hardware utilization statistics.

  3. At the end of a successful run, SambaTune:

    1. Collates compile-time and run-time statistics.

    2. Generates performance reports. See Explore SambaTune Reports.

    3. Displays the reports in the SambaTune GUI to help you identify potential hotspots. See Explore with the SambaTune GUI.

SambaTune input arguments

You can customize your SambaTune run with the following input arguments:

Table 1. SambaTune input arguments
Option Description Dependencies Type

app

Name of the application.

string

compile-args

Arguments to pass to the SambaFlow compiler. compile-args + model-args are used for compilation (generating the PEF file).

app

string

model-args

Arguments to pass for running a specific model, like batch size. compile-args + model-args are used for compilation (generating the PEF file).

app

string

run-args

Arguments to pass when running the app that are used in addition to model-args, for example, learning rate. The run-args and model-args are both used when you run the model (represented by the PEF file).

app

string

env

Runtime environment variables (optional). See Table 2

app

dict

For subprocesses that are created by SambaTune, you can configure the following environment variables:

Table 2. Environment variables for SambaTune subprocesses
Option Description Type

SF_RNT_FSM_POLL_BUSY_WAIT

1 to enable Graph completion busy wait

int

SF_RNT_DMA_POLL_BUSY_WAIT

1 to enable DMA completion busy wait

int

Run the SambaTune GUI

The SambaTune GUI allows you to read the reports that are generated by one or more SambaTune runs in a web browser.

You install the SambaTune GUI on the client system where the web browser runs. Unlike the host system, the client does not have direct access to RDU.

For release 1.15 of SambaTune, contact SambaNova customer support through the SambaNova support portal at https://support.sambanova.ai for client install instructions.

  1. On the machine where you installed the SambaTune GUI package, call sambatune_ui.

    You can specify some arguments to this command. Run sambatune_ui --help to see the list of arguments.

  2. When the sambatune_ui command completes, you see a URL, username, and password for accessing the GUI. Note down the password, which changes each time you call the sambatune_ui command.

You can now examine the results of the SambaTune run in the SambaTune GUI. See Explore with the SambaTune GUI.

Troubleshooting

This section has troubleshooting information.

Symptom

SambaTune encountered an error during the run.

Explanation

A SambaTune run may encounter errors due to any number of reasons, ranging from incorrect input configuration to compile error to run or post-processing error.

Solution

All run related information is saved to the output directory ($DUMP_ROOT/artifact_root/sambatune_gen/<input_config_name>_<timestamp>). The status of the run can be checked in run.log or status_summary.log. The details of a failed step can be checked in status_debug.log. For assistance, contact Customer Support and provide the compressed output directory for further diagnosis.