DataScale SN10-8R power management

For proper operation of the DataScale SN10-8R and to prevent issues, be sure you power on and power off the system appropriately, as described below.

1. Warnings and general notes

The following notices apply to the DataScale SN10-8R.

Some components within the rack work at high voltage. To prevent personal injury and voiding of the warranty, do not attempt to service components except where noted.

To prevent the DataScale SN10-8R from failing and to prevent damage to its components, keep the front and rear rack doors closed during standard operation.

To prevent DataScale SN10-8R components from overheating, keep the front and rear of the rack clear of obstructions to allow proper airflow.

Before you begin, read the SambaNova Release notes, part of DataScale SN10-8R 2PDU hardware installation to ensure you understand any known issues or limitations that might apply to the system. If you do not read the release notes, you might incorrectly configure the system components or software, which might necessitate a factory reset.

Do not power off or reboot the DataScale SN10-8R components during a firmware update. Doing so might damage the DataScale SN10-8R components, and damaged components might not be recoverable. Perform a shutdown or reboot only after a firmware update has been completed.

When the PDUs are physically connected to the datacenter’s power receptacles and power is applied to the rack, all DataScale SN10-8R components begin to power on. These components’ fans initially run at full speed but eventually ramp down after the BMCs finish their boot sequence. Power is not immediately applied to the rack components because the breakers on the PDUs are turned off. You must manually turn on these breakers to begin feeding power to the DataScale SN10-8R components.

2. Powering on the DataScale SN10-8R

Power on the DataScale SN10-2 RDU modules before powering on the DataScale SN10-H host modules, as described in the following steps.

2.1. Task 1: Turn on the six circuit breakers for each PDU

When the PDUs are plugged into the datacenter power and you close the circuit breakers, power is automatically applied to the DataScale SN10-8R rack components.

Figure 1 shows what a PDU circuit breaker group looks like and shows breaker switch 6 circled. Each PDU has a bank of three circuit breakers grouped together. There are two groups on a PDU.

Figure 1. Circuit breakers on PDU

The DataScale SN10-H host modules and DataScale SN10-2 RDU modules boot into standby mode and wait to be manually powered on. The networking equipment in the rack does not go into standby mode; instead, it completely boots after power is established.

If power is provided to the networking components, but they are not powering on, see the product-specific documentation for how to power on the network device:

Lantronix SLC8000 serial console server:
https://cdn.lantronix.com/wp-content/uploads/pdf/SLC8000_UG.pdf
Juniper EX4300 access switch (for the access network):
https://www.juniper.net/documentation/product/en_US/ex4300
Juniper QFX5200 Ethernet high-bandwidth data switch (for the data network):
https://www.juniper.net/documentation/product/en_US/qfx5200

2.2. Task 2: Power on the DataScale SN10-2 modules

Boot the DataScale SN10-2 RDU modules in the system by using SSH to connect to the SN10-2 BMC, or by sending an API call to the SN10-2 BMC. This section includes steps for both options.

Option 1: Use SSH to connect to the SN10-2 BMC

From a system that has access to the DataScale SN10-8R access network:
1. Open a terminal session
2. Use the ssh command to securely connect to the first DataScale SN10-2 RDU module in each system.
  
  The first DataScale SN10-2 RDU module in each system is as follows:
  - System 1: SN10-2-1
  - System 2: SN10-2-5
  - System 3: SN10-2-9
  - System 4: SN10-2-13
    
    Here’s an example for system 1 given IP address subnet 10.0.1.0/26 for the access network:
    
    $ ssh root@10.0.1.13 root@10.0.1.5’s password: <Enter root password> root@xrdu:~#

Run the following xrduutil command to power on the system:

root@xrdu:~# xrduutil -U root -P <root_password> poweron

To ensure the DataScale SN10-2 RDU modules are up before you boot the DataScale SN10-H host module, run the following command to check the status of each of the DataScale SN10-2 RDU modules.
```
root@xrdu:~# xrduutil -U root -P <root_password> powerstate
Power is on for XRDU_0
Power is on for XRDU_1
Power is on for XRDU_2
Power is on for XRDU_3
```

Option 2: Send a REST API call to the SN10-2 BMC

Generate a token (recommended).

If you use the REST API, SambaNova recommends that you use token-based authentication so that plain-text passwords are not sent over the network for REST API commands. See Generating a secure API token for instructions.

Run the REST API power-on command for each DataScale SN10-2 RDU module. Run this command for each DataScale SN10-2 RDU module in each of the nodes, in no particular order.

Format:

$ curl -b cjar -k -H "X-Auth-Token: $token" -X PUT -d '\{"data":"xyz.openbmc_project.State.Chassis.Transition.On"}' https://<SN10-2_BMC_IP>/xyz/openbmc_project/state/chassis0/attr/RequestedPowerTransition

Example:

$ curl -b cjar -k -H "X-Auth-Token: $token" -X PUT -d '\{"data":"xyz.openbmc_project.State.Chassis.Transition.On"}' https://10.0.1.21/xyz/openbmc_project/state/chassis0/attr/RequestedPowerTransition

To ensure the DataScale SN10-2 RDU modules are up before you boot the SN10-H, run the following command against each of the DataScale SN10-2 RDU modules:

Format:

$ curl -b cjar -k -H "X-Auth-Token: $token" https://<SN10-2_BMC_IP>/xyz/openbmc_project/state/chassis0

Example:

$ curl -b cjar -k -H "X-Auth-Token: $token" https://10.10.0.21/xyz/openbmc_project/state/chassis0

Example output after an SN10-2 RDU module is powered on:

{
"data": {
"CurrentPowerState": "xyz.openbmc_project.State.Chassis.PowerState.On",
"LastStateChangeTime": 1591197275103,
"POHCounter": 75,
"RequestedPowerTransition": "xyz.openbmc_project.State.Chassis.Transition.On"
},
"message": "200 OK",
"status": "ok"
}

Option 3: Mechanical power-on

To power on the SN10-2 resources:

Press the power button located on the front panel of the SN10-2 for 5 seconds. This panel is located on the front left side of the system. The power button is identified as item 1 in SN10 front panel (annotated).
Wait for the system LED (callout item 2) to go from blinking to solid green light. When the system LED is no longer blinking, then the SN10-2 resources are being powered on.

Figure 2. SN10 front panel (annotated)

2.3. Task 3: Power the DataScale SN10-H host module

To ensure that the DataScale SN10-H host module populates the system device tree properly, power on the host module only after the DataScale SN10-2 RDU modules are powered on fully.

Boot the DataScale SN10-H host module using either mechanical power on, or power on via IPMI, or power on via the Web UI. This section discusses each option.

Option 1: Mechanical power on

Power on the SN10-H host module by pressing the power button located on the front panel of the SN10-H. This panel is located on the front left side of the server.

Power button

Option 2: Power on via IPMI

Run the following command from a system that has ipmitool installed and has access to the SN10-H host module’s BMC via the access network.

$ ipmitool -I lanplus -H <SN10-H_BMC_IP_Address> -U root -P <root password> power on

Option 3: Power on via Web UI

To power on via Web UI your system must meet the following requirements:

Access to the DataScale SN10-H host module’s BMC via the access network
One of the following supported web browsers:
- Chrome (latest version)
- Firefox (latest version)

Follow these steps:

In the browser’s address bar, enter the IP address of the SN10-H host module’s BMC.
Log in to the management console by entering the user credentials.
Click Sign me in.
From the BMC dashboard, select Power Control.
Select the Power On checkbox, and then click Perform Action.
Perform this boot sequence for all nodes in the DataScale SN10-8R. The order in which you bring up the nodes does not matter.

3. Gracefully shutting down the DataScale SN10-8R

To shut down the DataScale SN10-8R, but not completely power off the entire rack, perform the following steps for each of the nodes in the DataScale SN10-8R rack.

3.1. Shut down the SN10-H host module in each system

Shut down the host module using one of the following methods:

Option 1: Shut down from the OS

$ ssh snuser1@<SN10-H_OS_IP_Address>
snuser1@SN10-H1’s password: <password>
$ sudo shutdown

This command does not shut down the system immediately but waits about a minute for users to save their work.

Option 2: Power off via IPMI

Run the following command from a system that has the following:

Access to the SN10-H host module’s BMC via the access network
The ipmitool installed

$ ipmitool -I lanplus -H <SN10-H_BMC_IP_Address> -U root -P <root password> power off

Option 3: Power off via Web UI

To power on via Web UI, your system must meet the following requirements:

Access to the DataScale SN10-H host module’s BMC via the access network
One of the following supported web browsers:
- Chrome (latest version)
- Firefox (latest version)

Follow these steps:

In the browser’s address bar, enter the IP address of the SN10-H host module’s BMC.
Log in to the management console by entering the user credentials.
Click Sign me in.
From the BMC dashboard, select Power Control.
Select the Power Off checkbox in the Power Actions screen and click Perform Action.

3.2. Shut down the DataScale SN10-2 RDU modules

Shut down the DataScale SN10-2 RDU modules in the node using one of the following methods:

Option 1: Use SSH to connect to the DataScale SN10-2 BMC

From a system that has access to the DataScale SN10-8R access network, open a terminal session and use ssh to connect to the first DataScale SN10-2 in each node.

See Network administration for IP address info.

The first DataScale SN10-2 in each of the nodes is as follows:
- System 1: SN10-2-1 (SN10-H-1-XRDU0)
- System 2: SN10-2-5 (SN10-H-2-XRDU0)
- System 1: SN10-2-9 (SN10-H-3-XRDU0)
- System 2: SN10-2-13 (SN10-H-4-XRDU0)
  
  Example for system 1 given IP address subnet 10.0.1.0/26 for the access network:
  $ ssh root@10.0.1.13 root@10.0.1.5’s password: <Enter root password> root@xrdu:~#

Run the xrduutil poweroff command:

root@xrdu:~# xrduutil -U root -P <root_password> poweroff

Option 2: Send a REST API call to the DataScale SN10-2 BMC

Generate a token (recommended).

If you use the REST API, SambaNova recommends that you use token-based authentication so that plain-text passwords are not sent over the network for REST API commands. See Generating a secure API token for instructions.

Run the REST API power-off command for each of the DataScale SN10-2 RDU modules in each of the systems.

Format:

$ curl -b cjar -k -H "X-Auth-Token: $token" -X PUT -d '\{"data":"xyz.openbmc_project.State.Chassis.Transition.Off"}' https://<SN10-2_BMC_IP>/xyz/openbmc_project/state/chassis0/attr/RequestedPowerTransition

Example:

$ curl -b cjar -k -H "X-Auth-Token: $token" -X PUT -d '\{"data":"xyz.openbmc_project.State.Chassis.Transition.Off"}' https://10.0.1.21/xyz/openbmc_project/state/chassis0/attr/RequestedPowerTransition

Shut down the Juniper QFX5200 high-bandwidth data switch, the Lantronix SLC8000 serial console server, and the Juniper EX4300 access switch.

When you power down the entire DataScale SN10-8R, shut down the Juniper EX4300 access switch last, because this switch controls the final access to the system via the network.

See the following product-specific documentation for information on how to shut down each of these network devices:
- Lantronix SLC8000 serial console server:
  https://cdn.lantronix.com/wp-content/uploads/pdf/SLC8000_UG.pdf
- Juniper EX4300 access switch:
  https://www.juniper.net/documentation/product/en_US/ex4300
- Juniper QFX5200 Ethernet high-bandwidth data switch (for the data network):
  https://www.juniper.net/documentation/product/en_US/qfx5200
  
  After shutting down the Lantronix serial console server, the only method for powering this network device back on is to physically disconnect power to the device and then reconnect the power.