SambaRack Manager Firmware Updates - SambaNova Documentation

This guide covers auditing and upgrading firmware on SambaRack hardware components using the snctl firmware command group.

Key concepts

Read this section once. Everything after it depends on these six things.

Prerequisites

Firmware commands authenticate to the target devices, so device credentials must be registered first. See Device credentials.

snctl secret set --device-type bmc
snctl secret set --device-type card
snctl secret set --device-type host
snctl secret set --device-type xrdu

Which device types support what

Not every component that can be audited can be upgraded through SambaRack Manager.

Device type	`firmware version`	`firmware upgrade`
`bios`	Yes	Yes
`bmc`	Yes	Yes
`xrdu`	Yes	Yes
`card` (Thor NIC)	Yes	Yes
`pdu`	Yes	No
`serial_terminal`	Yes	No
`access_switch`	Yes	No
`data_switch`	Yes	No
`all`	Yes	Yes — upgradeable types only

Choosing a target

Either --rack-id or --group must be specified. They are mutually exclusive. Both subcommands accept either one.

snctl firmware version --rack-id 001 --device-type all
snctl firmware version --group production --device-type all
snctl firmware upgrade --group production --device-type bmc

(Optional) Slotted device types and `--index`

xrdu and card are slotted — they have multiple instances per node, so --index can narrow the operation to specific slots. Pass a comma-separated list, for example 0,2,5.

Behaviour	Detail
Omitting `--index`	Targets all instances on the target racks
Valid for	`xrdu` and `card` only
Rejected for	`bios`, `bmc`, and `all` — these have no slots
`card` numbering	Selects network interfaces by their position in the rack type’s interface list — the same numbering shown in the `Slot` column of `snctl firmware version --device-type card`
Index matches nothing	Error. The command reports which index is missing and which are available, then exits non-zero without flashing anything

XRDUs flash two components

Each XRDU contains two independently flashed components — OBMC and RDU-C. They are reported separately by firmware version, and both must be supplied separately when passing images manually.

An XRDU upgrade powers the chassis off before flashing, because the flashing tool refuses to write to a powered-on chassis. This power-off applies to every XRDU behind the endpoint, not only the slots named by --index. Plan XRDU upgrades as whole-node maintenance regardless of how many slots you are targeting.

Where firmware images come from

Environment	Behaviour
Connected (default)	Omit `--image`. SambaRack Manager downloads the plan-of-record firmware, verifies its checksum, and caches it locally. Repeat upgrades of the same version reuse the cache.
Airgapped	Supply the file yourself with `--image`. See Airgapped environments.

The cache directory is shown as Cache Location in snctl config show. Change it with snctl config update --cache-location <dir>. See Initial setup.

Check firmware versions

Audit installed firmware against the plan of record (POR).

snctl firmware version \
  (--rack-id <rack_id> | --group <group_name>) \
  [--device-type <device_type>] \
  [--index <indices>]

Flag	Description
`-r, --rack-id <id>`	Rack to target (required unless `--group` is used)
`--group <name>`	Target every rack in a named group
`--device-type <type>`	`all` (default), or any type from the support table
`--index <indices>`	Slot indices for `xrdu` and `card` — see Slotted device types
`--protocol <protocol>`	Override the default protocol: `ssh`, `rest`, or `redfish` (endpoint-dependent)
`-o, --output <format>`	`text` (default), `wide`, or `json`

snctl firmware version --rack-id 001 --device-type all

Rack  Node             Component  Status  CURR   POR
----  ---------------  ---------  ------  -----  -----
001   sn40l-16-001-n1  BIOS       PASS    2.4.1  2.4.1
001   sn40l-16-001-n1  BMC        FAIL    1.07   1.08

  1 racks: 1 ok, 0 failed  (12.4s)

Reading the output

Column	Meaning
`CURR`	The version currently installed
`POR`	The plan-of-record target version
`Status`	`PASS` when they match, `FAIL` when they differ, `ERROR` when the component could not be read

FAIL means the firmware is out of date — not that the command failed. The check ran successfully and correctly reported a version mismatch.

XRDUs report their two components on separate rows:

Rack  Node             Component    Status  CURR   POR
----  ---------------  -----------  ------  -----  -----
001   sn40l-16-001-n1  XRDU/0/OBMC  PASS    5.1.4  5.1.4
001   sn40l-16-001-n1  XRDU/0/RDUC  FAIL    5.1.2  5.1.4

Wide output

-o wide adds Protocol, Model, and Source columns. Source is the artifact URL of the POR firmware, which you need for airgapped upgrades:

snctl firmware version --rack-id 001 --device-type bios -o wide

Rack  Node             Component  Status  CURR   POR    Protocol  Model     Source
----  ---------------  ---------  ------  -----  -----  --------  --------  ------------------------------------------
001   sn40l-16-001-n1  BIOS       PASS    2.4.1  2.4.1  redfish   SN40L-16  gs://<bucket>/firmware/bios/2.4.1/bios.bin

Upgrade firmware

Push firmware to a rack’s components.

snctl firmware upgrade \
  (--rack-id <rack_id> | --group <group_name>) \
  --device-type <device_type> \
  [--image <path>] \
  [--index <indices>]

Flag	Description
`-r, --rack-id <id>`	Rack to target (required unless `--group` is used)
`--group <name>`	Target every rack in a named group
`--device-type <type>`	`all`, `bios`, `bmc`, `xrdu`, or `card` — required
`--image <path>`	Local firmware image. Optional — downloaded automatically if omitted
`--index <indices>`	Slot indices for `xrdu` and `card` — see Slotted device types
`-p, --parallelism-racks <n>`	Max racks in flight (default `1`)
`--parallelism-devices <n>`	Global max nodes in flight across all racks (`0` = uncapped)
`--protocol <protocol>`	Override the default protocol (endpoint-dependent)

Recommended order and pre-conditions

There is no strict dependency across the supported components. However, when upgrading more than one, use this order: CARD → BIOS → BMC → XRDU

Device	Pre-condition	Post-upgrade operations	Expected duration
CARD	`snd` driver daemon active on the host (waited for automatically)	No power cycle. Firmware version check	~10 min — ~5 min upgrade, ~5 min verification
BIOS	Host powered on	Host power off/on, SR-IOV verify, node power cycle, version check	~1 hour — ~5 min upgrade, ~30 min SR-IOV disable and verify, ~20 min power cycle
BMC	None	Power cycle, version check	~25 min — ~5 min upgrade, ~20 min power cycle
XRDU	Host powered off	Power on, version check	~30 min — ~20 min upgrade, ~10 min power on

BIOS and XRDU have opposite power pre-conditions — BIOS needs the host on, XRDU needs it off.

Verifying an upgrade. Run snctl firmware version --rack-id <rack_id> --device-type all about 5 minutes after a successful run to confirm the new version took effect.

Examples

# Upgrade BMC firmware, image downloaded automatically
snctl firmware upgrade --rack-id 001 --device-type bmc

# Upgrade one XRDU
snctl firmware upgrade --rack-id 001 --device-type xrdu --index 3

# Upgrade several XRDUs
snctl firmware upgrade --rack-id 001 --device-type xrdu --index 3,4,5,6

# Upgrade only network interfaces 0 and 3 on every node
snctl firmware upgrade --rack-id 001 --device-type card --index 0,3

Progress output

Each component reports staged progress as it flashes, followed by a per-component result list and a summary. Components already at the target version are skipped.

#Example output : THOR Card Upgrade

snctl firmware upgrade --rack-id 001 --device-type card

firmware upgrade  rack=001
[card slot=1] 30% flashing
[card slot=2] 30% flashing
[card slot=6] 100% already at target version — skipping
[card slot=8] 100% already at target version — skipping
[card slot=1] 80% verifying
[card slot=2] 80% verifying
[card slot=1] 100% done
[card slot=2] 100% done
...
OK    card node=node1 slot=1
OK    card node=node1 slot=2
OK    card node=node1 slot=6
OK    card node=node1 slot=8
...

  9 components: 9 ok, 0 failed  (328.7s)

Airgapped environments

If your environment has no GCS access, supply the firmware file yourself with --image.

Look up the download URL

Run firmware version -o wide and read the Source column for the component you are upgrading:

snctl firmware version --rack-id 001 --device-type bios -o wide

Download the file out of band

From a host that does have GCS access, download it and transfer it into the airgapped environment:

gsutil cp gs://<bucket>/<path-from-source-column> ./bios.bin

Pass the local path

snctl firmware upgrade --rack-id 001 --device-type bios --image /path/to/bios.bin

Image path format

Device type	`--image` format
`bios`, `bmc`, `card`	A bare path with no prefix
`xrdu`	Passed twice, each with an explicit component prefix

snctl firmware upgrade --rack-id 001 --device-type xrdu \
  --image obmc=/path/to/obmc-5.1.4.tgz \
  --image rduc=/path/to/rduc-5.1.4.tgz

XRDU upgrades report both components per slot, and step through staging, chassis power-off, flashing of the primary and recovery partitions for each component, a reboot, and verification:

[xrdu_obmc slot=2] 3% checking staging space
[xrdu_obmc slot=2] 5% powering off chassis
[xrdu_obmc slot=2] 15% staging image
[xrdu_obmc slot=2] 30% flashing bmc primary
[xrdu_obmc slot=2] 45% rebooting
[xrdu_obmc slot=2] 60% staging bmc
[xrdu_obmc slot=2] 64% flashing bmc recovery
[xrdu_obmc slot=2] 68% staging rduc
[xrdu_obmc slot=2] 72% flashing rduc primary
[xrdu_obmc slot=2] 76% staging rduc
[xrdu_obmc slot=2] 80% flashing rduc recovery
[xrdu_obmc slot=2] 92% verifying
[xrdu_obmc slot=2] 100% done
...
OK    xrdu_obmc node=node1 slot=2
OK    xrdu_rduc node=node1 slot=2
...

  16 components: 16 ok, 0 failed  (1451.5s)

Slots run in parallel, so lines from different slots interleave in practice.

A bare --image path is rejected for --device-type xrdu, and prefixed values are rejected for single-component types.

Upgrade procedures by device type

Follow these in the recommended order when upgrading more than one component. CARD → BIOS → BMC → XRDU

CARD (Thor NIC)

Pre-condition: the SambaNova driver daemon (snd) must be active on the host. SambaRack Manager waits for this automatically before flashing, so no manual check is required.

# Auto-download the POR image
snctl firmware upgrade --rack-id <rack_id> --device-type card

# Or flash a specific local image
snctl firmware upgrade --rack-id <rack_id> --device-type card --image /path/to/thor-firmware.pkg

No power cycle is required for CARD upgrades — bnxtnvm flashes take effect without a reboot. This is the one upgrade target that does not need a power operation afterward.

Verify the firmware version on every NIC interface:

snctl firmware version --rack-id <rack_id> --device-type card

BIOS

Pre-condition: the host must be powered on.

Applying the BIOS update and then immediately power-cycling the whole node can leave the host powered on but stuck before POST — it never reaches the OS. To apply the new BIOS reliably, cycle the host on its own first, confirm the SR-IOV setting, and only then power cycle the entire node. Follow every step below in order.

Confirm the host is powered on

snctl power state --node <node_id> --type host

# Power on if needed
snctl power on --node <node_id> --type host

Run the upgrade

# Auto-download the POR image
snctl firmware upgrade --rack-id <rack_id> --device-type bios

# Or flash a specific local image
snctl firmware upgrade --rack-id <rack_id> --device-type bios --image /path/to/bios.bin

Power off the node, then bring the host up alone

Omit --type so both the host and the XRDUs power off:

snctl power off --node <node_id>

Then power on the host by itself. The new BIOS is applied during POST — the console shows the BIOS settings being applied, and the host may reboot once on its own before booting to the OS.

snctl power on --node <node_id> --type host

Wait 10–15 minutes after power-on for the host to finish POST and boot to the OS before continuing.

Confirm SR-IOV is disabled

SR-IOV must be disabled in the BIOS. The relevant attribute is PCIS007. These checks run over Redfish, directly against the BMC.Gather the BMC connection details. The BMC IP and username come from the node’s inventory — read them from the BMC row:

snctl node show <node_id>

The BMC password comes from the secret store:

snctl secret get --device-type bmc --show-value

--show-value prints the password in clear text. Only use it in a secure environment, and never share the output.

Use these values for <bmc_ip>, <bmc_user>, and <bmc_password> below.Check the current value:

curl -s -k -u <bmc_user>:<bmc_password> \
  https://<bmc_ip>/redfish/v1/Systems/Self/Bios | jq '.Attributes.PCIS007'

Reads "Disabled" — continue to the next step.
Reads anything else — disable it below.

Disable SR-IOV (a successful request returns no output):

curl -s -k -u <bmc_user>:<bmc_password> \
  -X POST https://<bmc_ip>/redfish/v1/Systems/Self/Bios/SD \
  -H 'Content-Type: application/json' \
  -d '{"Attributes": {"PCIS007": "Disabled"}}'

Apply the change by cycling the host again:

snctl power off --node <node_id> --type host
snctl power on  --node <node_id> --type host

Wait 10–15 minutes for the host to boot, then re-check PCIS007 and confirm it now reads "Disabled" before continuing.

Power cycle the entire node

Only once SR-IOV is confirmed disabled. snctl power cycle always targets the whole node, so there is no --type flag.

snctl power cycle --node <node_id>

Verify the firmware version

snctl firmware version --rack-id <rack_id> --device-type bios

BMC

Pre-condition: the host must be powered on.

Run the upgrade

# Auto-download the POR image
snctl firmware upgrade --rack-id <rack_id> --device-type bmc

# Or flash a specific local image
snctl firmware upgrade --rack-id <rack_id> --device-type bmc --image /path/to/bmc.bin

Power cycle the node, then verify

snctl power cycle --node <node_id>
snctl firmware version --rack-id <rack_id> --device-type bmc

XRDU

Pre-condition: the host must be powered off — the opposite of the BIOS pre-condition.

Confirm the host is powered off

snctl power state --node <node_id> --type host

# Power off if needed
snctl power off --node <node_id> --type host

Run the upgrade

Both components (OBMC and RDU-C) are upgraded together.

# Auto-download the POR images for both components
snctl firmware upgrade --rack-id <rack_id> --device-type xrdu

# Or flash specific local images — both components required
snctl firmware upgrade --rack-id <rack_id> --device-type xrdu \
  --image obmc=/path/to/obmc.tgz --image rduc=/path/to/rduc.tgz

Remember that this powers off all XRDUs on the node, regardless of any --index you pass.

Power the node back on, then verify

snctl power on --node <node_id> --type node
snctl firmware version --rack-id <rack_id> --device-type xrdu

Next step

Populate a new installation in one command instead of adding devices by hand. See Inventory file schema.

Get started

Models

Features

Build

Resources

Firmware Updates

Key concepts

Prerequisites

Which device types support what

Choosing a target

(Optional) Slotted device types and `--index`

XRDUs flash two components

Where firmware images come from

Check firmware versions

Reading the output

Wide output

Upgrade firmware

Recommended order and pre-conditions

Examples

Progress output

Airgapped environments

Image path format

Upgrade procedures by device type

CARD (Thor NIC)

BIOS

BMC

XRDU

Next step

​Key concepts

​Prerequisites

​Which device types support what

​Choosing a target

​(Optional) Slotted device types and --index

​XRDUs flash two components

​Where firmware images come from

​Check firmware versions

​Reading the output

​Wide output

​Upgrade firmware

​Recommended order and pre-conditions

​Examples

​Progress output

​Airgapped environments

​Image path format

​Upgrade procedures by device type

​CARD (Thor NIC)

​BIOS

​BMC

​XRDU

​Next step

Key concepts

Prerequisites

Which device types support what

Choosing a target

(Optional) Slotted device types and `--index`

XRDUs flash two components

Where firmware images come from

Check firmware versions

Reading the output

Wide output

Upgrade firmware

Recommended order and pre-conditions

Examples

Progress output

Airgapped environments

Image path format

Upgrade procedures by device type

CARD (Thor NIC)

BIOS

BMC

XRDU

Next step