Hyperparameter reference

SambaFlow supports command-line arguments and hyperparameters.

  • Command-line arguments can be predefined or application-specific.

    • SambaFlow defines a set of common arguments for compiling or running a model. See, for example, SambaSession.run() API Reference.

    • In addition, the application code can add model-specific arguments for compile and for run.

  • The list of hyperparameters is fixed. Each model determines which hyperparameter is used. For example, a model without dropout does not use the dropout hyperparameter.

You can change the value of a hyperparameter while the application is running. For example, you might adjust the lr hyperparameter over time as your model learns more and more.

This doc page briefly explains the use of hyperparameters, and lists hyperparameters that are supported by SambaFlow.

Supported hyperparameters

SambaFlow supports the following hyperparameters. For each hyperparameter, the default value is the value that was used to compile the PEF. SambaNova recommends strongly that you explicitly set the desired value when you run your model, at least for the first time.

Each model determines its hyperparameters and stores them in a hyperparameter dictionary. The model can change the value programmatically as the model is running.
Hyperparameter Type Description

lr

float

Learning rate. Controls the size of the optimizer step during model weight updates.

momentum

float

The momentum factor for the SGD optimizer.

weight_decay

float

Scales model weights before the optimizer step, used to help prevent overfitting by reducing the complexity of the model.

p

float

Dropout rate value for all dropout nodes in the model.

<node_name>_p

float

Dropout rate value for a specified node.

Example

A hyperparameter dictionary is a set of key:value pairs. Here’s an example that defines both a hyperparameter dictionary and a dropout rate:

hyper_dict = {'lr': learning_rate}
dropout_dict = {'p': dropout_rate}

hyperparam_dict = {**hyper_dict, **dropout_dict}

Later in the code, we can then specify those values.

hyperparam_dict = {}
hyperparam_dict['lr'] = 0.0
hyperparam_dict['p'] = 0.0

Example: Change to a non-default scheduler

The lr