Compiler argument reference

Name of the PEF file that the compiler generates and of the subdirectory for other compilation artifacts. By default the compiler uses the compilation timestamp and the process ID to name the subdirectory.

Use this argument to give your PEF file a meaningful name. For example, use today’s date in the name or use an inf extension for compilation for inference. If you experiment with different sets of hyperparameters, consider using them in the name. For example, if you are changing batch sizes in different training runs, add -b32 or -b16 to the PEF name to note the batch. For example:

Specify --inference to compile and generate a PEF for inference. If you specify this argument, the compiler performs only a forward pass and doesn’t perform certain optimizations. See How model compilation works.

Just as with compilation for a training run, you can specify a PEF file and other compiler arguments. For example:

Optional output folder. The compiler places the PEF file and log files in the output folder, which defaults to ./out/<pef-name>. To set the folder explicitly, run a command like this:

Informs the compiler which batch size will later be used during training. Set batch-size to 4, 8, 16, 32, or even higher to support more efficient training. The highest value you can use depends on the model and on available hardware resources. With different batch sizes your training might go faster or slower (to achieve the target accuracy). It usually takes some experimentation to find the right batch size for a model.

Shows verbose log output, similar to the --debug output.

When you run the compiler with --verbose, specify an output directory. The information about the location of the generated PEF file is no longer at the end of the output. For example:

When you work on a model with SambaNova Support, they might ask you to start the model in debug mode. In debug mode, the compiler sends more messages to stdout (and the logs). In addition, the --help output shows some arguments that are customarily used only with customer support.

Specifies a non-default directory where all logs with warnings are sent.

When you specify -o0, each PyTorch operator is compiled independently.

This is the safest option, but because the compiler doesn’t perform optimizations, training and inference take longer than with o3. See Compiler optimization modes for details.

Here’s a very simple example. We’re working on additional examples.

With -o1, PyTorch operators are fused together into subgraphs. Each subgraph is compiled independently.

See Compiler optimization modes for details.

Optimization rules .yaml file to use with o1.

Enable compiler optimizations with o1. This argument is currently required when you compile with o1. We expect to remove this argument soon.

With -o3, the compiler has a global view of the entire graph. With this release (1.17), -o3 is the default.

This option usually has the longest compile time but fast runtime performance when used with model-specific HD files. Because the compiler attempts to optimize the whole graph, compilation might fail in some cases.

With -o3, you can optionally annotate subgraphs with `--enable-hypersection. In that case, each annotated subgraph is compiled independently. If there are duplicate subgraphs, only one is compiled and reused.

Specifies the compiler mode. Using this flag with the right model type improves performance.

If you’re running in o3 mode (the default), then you can annotate your model’s Python code to tell the compiler about duplicate subgraphs and get the performance improvements. Use this option to enable this optimization.

Sometimes the compiler underestimates or overestimates the RDU resources that are needed for some decisions. Overestimation can results in compilation failures and underestimation can result in bad performance. If compilation fails, you can use this flag to force the compiler to assume it has fewer resources available than it has.

Specify 3 or 4 floats. A float of 1.0 means that the compiler can see all avaiiable resources.

For example:

The compiler assumes that it can use all available resources for forward graphs and 80% for backward and optimizer graphs.

Use this option if you have a legacy model that relies on an earlier version of the compiler and you see suboptimal compilation.

Allows you to compile for a different target architecture. For example, if you’re compiling on an SN30 system but expect to run the model on an SN20 system, you can use this flag.

Default is native, that is, the compiler targets the architecture of the hardware that you’re running on.

Performs compilation so the PEF runs on an SN20 system even if you’re compiling on an SN30 system or on a CPU-only node.

Instructs the compiler to run in tensor parallel mode.

Specifies the number of chips, that is, RDUs. Tensor parallel works only on 1 node, that is, up to 8 RDUs on SN10 2PDU (aka SN20) and SN30 systems. Defaults to 1 RDU.

Causes the compiler to add the gather and reduce sections and buffers to the dataflow graph to support data parallel operation. See How to use data parallel mode for some prerequisites and best practices.

Defines the minimum number of application replicas to be launched when the model is trained in data parallel mode. For compilation, set the value to 2. The actual number of replicas to be launched is defined at runtime.