SN40L-16

SN40L-16 provides improved performance by increasing the number of Reconfigurable Dataflow Units™ (RDUs) available per node from 8 to 16.

Nodes and RDUs

A node describes the collection of SambaNova Systems' Reconfigurable Dataflow Units (RDUs) available. For example, with SambaNova’s SN10, SN30, and SN40L-8, a total of eight RDUs are available per node. SN40L-16 increases the available RDUs to 16 per node.

If a job or endpoint requires more than one RDU to start, the RDUs must be available on the same node. With 16 RDUs per node, SN40L-16 increases the availability of RDUs to start heavy RDU resource jobs or endpoints.

Naming convention

  • Our existing SN40L, which includes eight RDUs per node, will be renamed to SN40L-8.

  • Our 16 RDUs per node is named SN40L-16.

The SambaStudio interface will include SN40L-16 in the following locations:

Supported models

The table below describes the SambaNova provide CoEs and their expert models that are supported on SN40L-16. Search the CoE name or expert model name in the Model Hub to view SN40L-16 information in the corresponding model cards.

Batch inference is not supported for the expert models listed in the table.
CoE Name Expert Model Name Dynamic Batch Size Supported Fine-tuned Checkpoints Inference Supported Add Checkpoint From Storage Inference Supported

Llama 3.1 405B with E5-mistral

Meta-Llama-3.1-405B-Instruct

e5-mistral-7b-instruct

1, 4

1, 4, 8, 16, 32

No

Yes

No

Yes

Llama 3.1 70B with E5-mistral

Meta-Llama-3.1-70B-Instruct

e5-mistral-7b-instruct

1, 4, 8, 16, 32

1, 4, 8, 16, 32

No

Yes

No

Yes

Llama 3.1 8B with E5-mistral

Meta-Llama-3.1-8B-Instruct

e5-mistral-7b-instruct

1, 4, 8, 16, 32

1, 4, 8, 16, 32

Yes

Yes

Yes

Yes