Run SambaTune and examine reports

After installation, you can run SambaTune from the command line.

You have to run SambaTune with the application yaml file as input before you can see results of the performance analysis.

Overview

Running Sambatune includes running sambatune and running sambatune_ui.

First you run sambatune and pass in a YAML file for your model. See Run the sample application.
Then you can run the sambatune_ui command. See Run the SambaTune GUI.
Finally, you can explore with the SambaTune GUI and examine SambaTune reports.

Run the sample application

A sample application, linear_net.py is included with your installation at /opt/sambaflow/apps/micros/linear_net.py. The application requires that the sambaflow-apps-micros package is installed.

To run the linear_net.py sample application:

Log in to the Linux console of a host that is attached to the DataScale hardware.
Run the application. You have several options:
- Run the application in benchmarking mode (the default):
  $ sambatune linear_net.yaml
  where linear.yaml is a user-specified configuration file that is included in the sambaflow-apps-micros package. Here’s an example:
  app: /path/to/linear.py model-args: -b 128 -mb 64 --in-features 512 --out-features 128 compile-args: compile --plot run-args: -n 10000
- Run the application in instrument-only mode. The space after -- is required.
  $ sambatune --modes instrument -- /opt/sambaflow/sambatune/configs/linear_net.yaml
- Run in all modes. The space after -- is required.
  $ sambatune --modes benchmark instrument run -- /opt/sambaflow/sambatune/configs/linear_net.yaml

Run sambatune --help for a list of all options. See SambaTune input arguments for details on configuration options.

Understand how SambaTune collects data

When you run the sample application:

SambaTune compiles the application with the user-specified model-args , compile-args and SambaFlow-supported instrumentation flags.
After successful compile, SambaTune:
1. Runs the application on the RDU and collects performance data.
2. Runs the application in benchmark mode with user-specified run-args to collect latency, throughput, and hardware utilization statistics.
At the end of a successful run, SambaTune:
1. Collates compile-time and run-time statistics.
2. Generates performance reports. See Explore SambaTune Reports.
3. Displays the reports in the SambaTune GUI to help you identify potential hotspots. See Explore with the SambaTune GUI.

SambaTune input arguments

You can customize your SambaTune run with the following input arguments:

Table 1. SambaTune input arguments
Option	Description	Dependencies	Type
app	Name of the application.		string
compile-args	Arguments to pass to the SambaFlow compiler. `compile-args` + `model-args` are used for compilation (generating the PEF file).	app	string
model-args	Arguments to pass for running a specific model, like batch size. `compile-args` + `model-args` are used for compilation (generating the PEF file).	app	string
run-args	Arguments to pass when running the app that are used in addition to model-args, for example, learning rate. The `run-args` and `model-args` are both used when you run the model (represented by the PEF file).	app	string
env	Runtime environment variables (optional). See Table 2	app	dict

For subprocesses that are created by SambaTune, you can configure the following environment variables:

Table 2. Environment variables for SambaTune subprocesses
Option	Description	Type
SF_RNT_FSM_POLL_BUSY_WAIT	1 to enable Graph completion busy wait	int
SF_RNT_DMA_POLL_BUSY_WAIT	1 to enable DMA completion busy wait	int

Run the SambaTune GUI

The SambaTune GUI allows you to read the reports that are generated by one or more SambaTune runs in a web browser.

You install the SambaTune GUI on the client system where the web browser runs. Unlike the host system, the client does not have direct access to RDU.

For release 1.16 of SambaTune, contact SambaNova customer support through the SambaNova support portal at https://support.sambanova.ai for client install instructions.

On the machine where you installed the SambaTune GUI package, call sambatune_ui.

You can specify some arguments to this command. Run sambatune_ui --help to see the list of arguments.
When the sambatune_ui command completes, you see a URL, username, and password for accessing the GUI. Note down the password, which changes each time you call the sambatune_ui command.

You can now examine the results of the SambaTune run in the SambaTune GUI. See Explore with the SambaTune GUI.

Troubleshooting

This section has troubleshooting information.

Error during run

Symptom

SambaTune encountered an error during the run.

Explanation

A SambaTune run may encounter errors due to any number of reasons, ranging from incorrect input configuration to compile error to run or post-processing error.

Solution

All run related information is saved to the output directory ($DUMP_ROOT/artifact_root/sambatune_gen/<input_config_name>_<timestamp>). The status of the run can be checked in run.log or status_summary.log. The details of a failed step can be checked in status_debug.log. For assistance, contact Customer Support and provide the compressed output directory for further diagnosis.

psysnprof error with model not enabled for SambaTune

Symptom

Error pysnprof.prof_lib.PerfException: App returned a non-zero exit code on models not enabled for SambaTune.

Explanation

Many models that SambaNova creates include a code snippet that enables the SambaNova profiler. For those models that get the error above, that code is missing and you can’t run SambaTune.

Solution

Add code like the following example, which makes the profiler collect information when you train the model.

def train

        if self.args.samba_runtime_profile:
            samba.session.start_runtime_profile()
        for epoch in tqdm(range(self.epochs)):
            train_metrics = self.train_epoch(train_loader, epoch, train_sampler)

            val_metrics = self._evaluation(val_loader, epoch)
            self.save_checkpoint(epoch)
            if self.args.no_moving_average:
                self.epoch_step = 0

            # Save the best validation checkpoint.
            if not self.run_benchmark and val_metrics["acc"] > self.best_val:
                self.best_val = val_metrics["acc"]
                self.save_checkpoint(epoch, is_best_ckpt=True)
                val_metrics["is_best_ckpt"] = True

        if self.args.samba_runtime_profile:
            samba.session.end_runtime_profile()