SN10-8R system administration guide (3 PDU)

Power is automatically applied to the DataScale SN10-8R components once the PDUs are plugged into the datacenter power and you close the circuit breakers. Figure 3 illustrates what a PDU circuit breaker group looks like (breaker switch 6 circled). Each PDU has a bank of three circuit breakers grouped together, and there are two groups on a PDU.

The DataScale SN10-H host modules and DataScale SN10-2 RDU modules boot into standby mode and wait to be manually powered on. The networking equipment in the rack does not go into standby mode; instead, it completely boots once power is established.

If power is provided to the networking components, but they are not powering on, see the product-specific documentation for how to power on the network device:

Boot the DataScale SN10-2 RDU modules in the system using one of the following methods:

Option 1: Use SSH to connect to the SN10-2 BMC

Option 2: Send a REST API call to the SN10-2 BMC

Boot the DataScale SN10-H host module using one of the following methods.

Option 1: Mechanical power on

Power on the SN10-H host module by pressing the power button located on the front panel of the SN10-H. This panel is located on the front left side of the server.

Option 2: Power on via IPMI

Run the following command from a system that has ipmitool installed and has access to the SN10-H host module’s BMC via the access network.

Option 3: Power on via WebUI

To power on via Web UI your system must meet the following requirements:

Follow these steps:

Shut down the SN10-H host module in each system by using one of the following methods:

Option 1: Shut down from the OS

Log in to the node via ssh as snuser1 and run the shutdown command.

This command does not shut down the system immediately but waits about a minute for users to save their work.

Option 2: Power off via IPMI

Run the following command from a system that has the following:

Option 3: Power off via Web UI

To power on via Web UI, your system must meet the following requirements:

Follow these steps:

Shut down the DataScale SN10-2 RDU modules in the node using one of the following methods:

Option 1: Use SSH to connect to the DataScale SN10-2 BMC

Option 2: Send a REST API call to the DataScale SN10-2 BMC

SambaNova provides the needed package repositories for Red Hat Enterprise Linux running on the DataScale SN10-8R. A partnership with Red Hat allows SambaNova to distribute a customized repository for the DataScale SN10-8R. Adding other repositories might cause issues with the operation of the SambaFlow software because there are some package and kernel version dependencies.

If the SambaNova software stack has problems running, SambaNova Support might request that you remove any packages that were not included from the SambaNova Red Hat Enterprise Linux repository or that you downgrade certain packages to a version that was certified.

To access the DataScale SN10-H host module:

DataScale SN10-H host module connectivity to the SambaNova repository is set up as part of the DataScale SN10-8R installation and relies on the site survey that your company completed before that step. As part of the initial installation, SambaNova provides a sambanova.repo file that contains the appropriate credentials and paths to your specific repository.

If you need to check the setup for the SambaNova OS repository, see KB article #1058.

SambaNova patch releases handle major upgrades to the DataScale SN10-H host module OS (for example, going from RHEL 8.2 to RHEL 8.5 or later). SambaNova provides a release notes file that contains any special notes and information about the commands you need to run to perform the upgrade.

To update the SambaFlow software packages, log in to the DataScale SN10-H host module(s) where the software packages need to be updated.

To view what packages are installed on the DataScale SN10-H host module, run the following command:

To view which SambaFlow packages have an update you can apply, run the following command:

To update the SambaFlow packages, examine the list provided in the output produced by the check-update command, and then run the following command to update a package and any of its package dependencies:

For example, if the output produced by the check-update command shows an update is available for the sambanova-runtime package, run the following command:

Repeat this step for each SambaFlow package that needs to be updated.

SambaNova provides the package repositories for Ubuntu running on the DataScale SN10-8R. A partnership with Ubuntu allows SambaNova to distribute a customized repository for the DataScale SN10-8R. Adding other repositories might cause issues with the operation of the SambaFlow software because there are some package and kernel version dependencies.

If the SambaFlow software stack has problems running, SambaNova Support may request that you remove any packages that were not installed from the SambaNova Ubuntu repository or that you downgrade certain packages to a version that was certified.

To access the DataScale SN10-H host module for the first time, from a system that can access the DataScale SN10-8R access network, use ssh as user snuser1 to log in to the DataScale SN10-H host module.

When you are prompted, enter the default password for snuser1. You can find default passwords in the "Default username and passwords for components" section of this document.

DataScale SN10-H host module connectivity to the SambaNova repository is set up as part of the DataScale SN10-8R installation and relies on the site survey that was completed prior to that step. As part of the initial installation, a sources.list file under the /etc/apt/ directory is provided, and the appropriate credentials for accessing your specific repository are configured in the /etc/apt/auth.conf file. If you need to check the setup for the SambaNova OS repository, see KB article #1057.

SambaNova patch releases handle major upgrades to the DataScale SN10-H host module OS (for example, going from 18.4 LTS to 20.04 LTS or kernel updates). A readme file contains any special notes and information about the commands to execute to perform the upgrade.

To update the SambaFlow software packages, log in to the DataScale SN10-H host module(s) where the software packages need to be updated.

To view what packages are installed on the DataScale SN10-H host module, run the following command:

To view which SambaNova packages have an update you can apply, run the following command:

To update all the packages that need to be updated, run the following command, which updates the packages and any package dependencies:

To update a specific package, replace samba[nf] with the name of a specific package. For example, to update sambanova-runtime, run the following command:

To update the SN10-H host module BIOS, your system must meet the following requirements:

You will first perform the BIOS update, and then update the OS. Follow these steps:

Performing the SN10-H BIOS update

Resetting the SN10-H OS

As a final step, you reset the SN10-H OS to complete the BIOS update.

If the DataScale SN10-H host module’s BMC is no longer responding or no longer accessible, or the DataScale SN10-H host module’s BMC password has been lost or forgotten, see the "Backing up and restoring components" section in this document.

You can quickly identify a system’s status and view diagnostic information and logs for the DataScale SN10-H BMC from the Web UI.

See KB article #1039, “Diagnostic Data Collection Tool(samba_diag),” in the SambaNova Support portal (https://support.sambanova.ai) for details on:

Updating the BMC and RDU-C firmware consistes of several tasks:

Preparing for update

Before you can start the update, you have to prepare, as follows:

Updating the DataScale SN10-2 BMC primary partition

Updating the DataScale SN10-2 BMC secondary/recovery partition

The re-imaging of the BMC will have removed the obmc-rdu-<version>.mtd and obmc-rdu-<version>.mtd.md5 files from /dev/shm/. You have to copy them back, as follows:

Once completed, move onto updating the DataScale SN10-2 RDU Controller (RDU-C) primary partition.

Updating the DataScale SN10-2 RDU-C primary partition

When secondary partition update has been completed, move on to updating the DataScale SN10-2 RDU Controller (RDU-C) primary partition.

Updating the DataScale SN10-2 RDU-C secondary/recovery partition

If any issues occur during the update of the DataScale SN10-2 BMC or RDU-C, contact SambaNova support

After the DataScale SN10-2 BMC and RDU-C have successfully been updated, it is safe to power on the DataScale SN10-2 and SN10-H modules. Powering on the DataScale SN10-8R discusses how to first safely power on the DataScale SN10-2 and then power on the DataScale SN10-H.

To change the default password for root on the DataScale SN10-2 BMC, follow these steps:

To support REST API calls that don’t use plain-text passwords, you can generate a secure token for the DataScale SN10-2 BMC root user.

DataScale SN10-2 BMC networking is configured as part of the DataScale SN10-8R delivery. It’s not usually necessary to modify the network configuration upon delivery, although there might be situations where the network has to be reconfigured later.

You can change the network settings by running the network-settings command.

Table 6 describes the command options.

See the "Administrative tasks for the baseboard management controller (BMC)" section of this document.

SambaNova does not alter the logs or log directories for Red Hat Enterprise Linux or Ubuntu. The majority of logs reside under the /var/log/ directory, along with others log tools such as journalctl.

The following SambaNova-related log files are also located in the /var/log/sambaflow/runtime/ directory:

Additional log events such as kernel logs (from the RDU driver module) go to dmesg(1).

Additional logs for the compilers are available in a user-specified directory that was specified at the time the models were compiled. These logs are fairly low level and might be requested by SambaNova Support to troubleshoot issues. For details see "Collecting diagnostic materials for SambaNova Support".

To check the status of the SND, run the systemctl status snd command. Below is sample output showing what the command might return:

You can start, stop, and restart the SND with the following commands:

To start the SND:

To stop the SND:

To restart the SND:

The SND CLI provides physical visibility into the entire DataScale SN10-8 system. This allows complete access to the RDU system for debugging, triage, and validation efforts.

The SND also responds to error events that occur on the RDU and on the entire DataScale SN10-2 RDU module.

All logs from the SND are written to /var/log/sambaflow/runtime/snd.log. This log provides a summary of the RDU resources available to the system and includes any hardware error events that occur. The information is useful for diagnosing and resolving hardware issues.

For problems that occur while compiling models, run the following command and examine the logs that are generated in the user-specified output directory:

For problems that occur while running models, see the following resources for details:

For operational issues with the third-party components in the DataScale SN10 rack, see the following vendor-specific product documentation. For issues that require additional support or for questions related to troubleshooting, open a support case through SambaNova Support.