SN10-8R hardware installation guide :: SambaNova Documentation

1. Using this document

SN10 (3 PDU) guide

This document describes the guidelines and specific components for the SN10 (3 PDU) version of the SambaNova DataScale rack.

Overview: Provide site-specific requirements and installation details for the DataScale SN10 (3 PDU) system.

Audience: Datacenter facility personnel, network administrators, and system administrators.

1.1. Product document library

SambaNova product documentation can be found at https://docs.sambanova.ai. SambaNova product knowledge base articles can be found at: https://support.sambanova.ai. To provide documentation feedback, please use the support portal to rate the documentation, or to open a documentation support case ticket to request updates to the documents.

2. Overview of installing the DataScale SN10-8R system

There are several steps that need to be followed to ensure a proper deployment of the DataScale SN10-8R system. This document describes the required steps including:

Overview of DataScale SN10-8 hardware
DataScale SN10-8 cabling
Site preparation for DataScale SN10-8 rack installation
Hardware installation tasks
Spare parts kit

Each of these sections is described in detail and provides checklists to minimize the chance of running into site issues or incorrectly configuring the DataScale SN10-8 system.

2.1. Installation safety

This product shall be installed following the instructions described in this document.
This product must be installed, serviced, and operated only by skilled and qualified personnel who have the necessary knowledge and practical experience with electrical equipment and who understand the hazards that can arise when working on this type of equipment. This product is intended for use in a Restricted Access Location.
This equipment is to be used in controlled environments (an area where the humidity is maintained at levels that cannot cause condensation on the equipment, the contaminating dust is controlled, and the steady-state ambient temperature is within the range specified).

3. Overview of DataScale SN10-8 hardware

The DataScale SN10-8 system is self-contained in a standard 42 rack unit (RU) datacenter rack and may be purchased in various configurations depending on customer requirements, including any specific datacenter requirements that must be met. System population begins at the bottom of the rack with node 1 and increments up the rack. Network switches and other equipment are installed at the top of the rack.

A DataScale SN10-8 system consists of four DataScale SN10-2 RDU modules and an x86-based DataScale SN10-H server host module running either Red Hat^® Enterprise Linux^® or Ubuntu^® Linux. Each DataScale SN10-2 module contains two Reconfigurable Data Units™ (RDUs), for a total of eight RDUs per DataScale SN10-8 system, and they are managed by the SambaNova’s SambaFlow^TM software stack running on the host. Both the DataScale SN10-2 RDU module and the DataScale SN10-H module are 2RU chassis.

Switch equipment at the top of the rack provides a data network and an access network. Figure 1 and Table 1 identify the main components in the DataScale SN10-8.

Figure 1. DataScale SN10-8 rack components (Front View)

Table 1. DataScale SN10-8 rack components
No.	Component
1	System 1 SN10-8 (four SN10-2)
2	System 1 SN10-8 (SN10-H)
3	System 2 SN10-8 (four SN10-2)
4	System 2 SN10-8 (SN10-H)
5	System 3 SN10-8 (four SN10-2)
6	System 3 SN10-8 (SN10-H)
7	System 4 SN10-8 (four SN10-2)
8	System 4 SN10-8 (SN10-H)
9	Juniper^® QFX5200-32c Ethernet (default) Mellanox^® SB7800 IB high-bandwidth data switch
10	Juniper EX4300 access switch

4. DataScale SN10-8R cabling

The DataScale SN10-8R internal components are delivered pre-configured and cabled in the 42RU rack. This section is for informational purposes only and to help identify what is cabled internally. Customers are not expected to interact with the internal cabling of the DataScale SN10-8 rack.

The only cables that customers are expected to interact with are for network and power components that need to be connected to the datacenter to provide power and connectivity to the DataScale SN10-8R.

Figure 2 and Table 2 below identify the locations of the main components in the DataScale SN10-8R (rear facing).

Figure 2. DataScale SN10-8R rack (Rear View)

Table 2. DataScale SN10-8R component identification
No.	Component Description
1	SN10-2-1 (system 1)
2	SN10-2-2 (system 1)
3	SN10-H 1 (system 1)
4	SN10-2-3 (system 1)
5	SN10-2-4 (system 1)
6	SN10-2-5 (system 2)
7	SN10-2-6 (system 2)
8	SN10-H 2 (system 2)
9	SN10-2-7 (system 2)
10	SN10-2-8 (system 2)
11	SN10-2-9 (system 3)
12	SN10-2-10 (system 3)
13	SN10-H 3 (system 3)
14	SN10-2-11 (system 3)
15	SN10-2-12 (system 3)
16	SN10-2-13 (system 4)
17	SN10-2-14 (system 4)
18	SN10-H-4 (system 4)
19	SN10-2-15 (system 4)
20	SN10-2-16 (system 4)
21	Juniper QFX5200-32c Ethernet (default) Mellanox SB7800 IB (optional) high-bandwidth data switch
22	Juniper EX4300 access switch
23	Lantronix^® serial console server (behind Juniper EX4300)
0RU PDUs (not shown) are on the right side of the rack when facing the rack rear.

4.1. Cabling system components

The DataScale SN10-8R comes pre-configured and cabled with the purchased number of systems connected to the appropriate switches and switch ports.

Figure 3, Figure 4, Table 3, and Table 4 below identify the components that will either be connected to switches or inter-connected to other system components.

Figure 3. SN10-H (Rear View)

Table 3. SN10-H components (Rear View)
No.	Component	Connector/Cable Type
1	Two power inlets	C13 to C14 power cord
2	SN10-H Net0 1GbE LAN	RJ45, Cat5e or higher
3	SN10-H 10/100/1000 management LAN (labeled “M”)	RJ45, Cat5e or higher
4	DB9 serial connection	Rollover DB9F to RJ45 console cable
5	Host interface card (HIC)	QSFP-DD cable
6	High-bandwidth network ports	100GbE or EDR IB cable

Figure 4. SN10-2 (Rear View)

Table 4. SN10-2 components (Rear View)
No.	Component	Connector/Cable Type
1	SN10-2 COM serial port (COM)	RJ45, Cat5e or higher
2	SN10-2 10/100/1000 management LAN (MGMT)	RJ45, Cat5e or higher
3	Host HIC connections (8 pairs)	QSFP-DD cable
4	Two power inlets	C13 to C14 power cord
5	High-bandwidth network port connections	100GbE or EDR IB cable

4.2. Serial console server cabling

The serial console server is installed and cabled in the factory. This component is located in RU42, and it occupies the front portion of the same rack unit as the Juniper EX4300 access switch. Cables used for this are standard Ethernet RJ45 cables and rollover DB9F to RJ45 console cables for the DataScale SN10-H systems.

A dedicated non-tagged VLAN network drop, that is on the access/management network, is required for the serial console server. This connection goes to the serial console server management port indicated by the number 1 in Figure 5 below. This ensures there is still network connectivity to the serial console server if anything should happen to the Juniper EX4300 access switch.

Please note that ports for this device are not visible or accessible to the user due to the physical location of the unit in front of the access switch. This device is primarily intended for SambaNova service operations.

Figure 5. Lantronix serial console server (Rear View)

Table 5. Lantronix serial console server components (Rear View)
No.	Component	Connector/Cable Type
1	10/100/1000 management Network (labeled “1”)	RJ45, Cat5e or higher
2	Serial RJ45 connections (ports 1–32)	RJ45, Ethernet cable
3	Two power inlets	C13 to C14 power cord

Cabling requirements from the Lantronix serial console server to the other DataScale SN10-8 rack components are listed in Table 6 below.

Table 6. Serial console server connections
From Serial Console Server Port	To Component	Component Location	Component Port
1	Juniper EX4300 access switch	RU42	CON2
2	Juniper QFX5200 Ethernet (default) Mellanox SB7800 IB (optional) high-bandwidth data switch	RU41	CON (Juniper) 10101 (Mellanox)
3	PDU 1	Rear rack, right side, closest to front of rack	RJ45 just left of USB labeled “10101”
4	PDU 2	Rear rack, right side, between PDU1 and PDU3	RJ45 just left of USB labeled “10101”
5	PDU 3	Rear rack, right side, closest to rear of rack / door	RJ45 just left of USB labeled “10101”
6	-	-	-
7	SN10-2-16	RU39/RU40	COM
8	SN10-2-15	RU37/RU18	COM
9	SN10-H-4	RU35/RU36	DB9 male PCI card
10	SN10-2-14	RU33/RU34	COM
11	SN10-2-13	RU31/RU32	COM
12	SN10-2-12	RU29/RU30	COM
13	SN10-2-11	RU27/RU28	COM
14	SN10-H-3	RU25/RU26	DB9 male PCI card
15	SN10-2-10	RU23/RU24	COM
16	SN10-2-9	RU21/RU22	COM
17	SN10-2-8	RU19/RU20	COM
18	SN10-2-7	RU17/RU18	COM
19	SN10-H-2	RU15/RU16	DB9 male PCI card
20	SN10-2-6	RU13/RU14	COM
21	SN10-2-5	RU11/RU12	COM
22	SN10-2-4	RU9/RU10	COM
23	SN10-2-3	RU7/RU8	COM
24	SN10-H-1	RU5/RU6	DB9 male PCI card
25	SN10-2-2	RU3/RU4	COM
26	SN10-2-1	RU1/RU2	COM
27-32	-	-	-

4.3. Access switch cabling

The Juniper EX4300 switch used for the access networks is installed and cabled in the factory. This component is located in RU42 at the rear of the rack and is shared in the same rack unit as the Lantronix serial console server. Cables used for this are standard Cat6 Ethernet RJ45 cables. This switch can further be segregated using port-based VLANs to separate access to the management interfaces of the DataScale SN10-8 components and the SN10-Hs’ operating system(OS) interface. The customer is responsible to configure this added network segregation. Please refer to Table 8 for what device is connected to which port, and available ports that can be used if an additional network drop is required for the separate network. This port separation can be configured by SambaNova if requested during initial deployment.

Warning: When configuring the access switch to separate the management interfaces and the SN10-H OS interfaces with port-based VLANs, a separate network drop must be used for each network. Be aware that if the switch is reset to factory default for whatever reason, or the VLAN configuration is lost. This could result in spanning-tree protocols disabling one or both of the networks. By default, the access switch is configured for Rapid Spanning Tree Protocol (RSTP).

Figure 6. Juniper EX4300 access switch (Front View)

Figure 7. Juniper EX4300 access switch (Rear View)

Table 7. Juniper EX4300 access switch components
No.	Component	Connector/Cable Type
1	1GbE ports (Ports 0–47)	RJ45, Cat5e or higher
2	RJ45 console port	RJ45, Cat5e or higher
3	Network management port for em0	RJ45, Cat5e or higher
4	Two power inlets	C13 to C14 power cord

Cabling requirements from the Juniper EX4300 switch for the access network to the other DataScale SN10-8 rack components are listed in Table 8.

Table 8. Access network port cabling
From Access Switch Port	To Component	Component Location	Component Port
0	-	-	-
1	Juniper EX4300 access switch	RU42	MGMT (switch rear)
2	Juniper QFX5200-32c Ethernet (default) Mellanox SB7800 IB (optional) high-bandwidth data switch	RU41	C0 or 10101 (switch rear)
3	SN10-2-1	RU1/RU2	MGMT
4	SN10-2-2	RU3/RU4	MGMT
5	SN10-2-3	RU7/RU8	MGMT
6	SN10-2-4	RU9/RU10	MGMT
7	SN10-2-5	RU11/RU12	MGMT
8	SN10-2-6	RU13/RU14	MGMT
9	SN10-2-7	RU17/RU18	MGMT
10	SN10-2-8	RU19/RU20	MGMT
11	SN10-2-9	RU21/RU22	MGMT
12	SN10-2-10	RU23/RU24	MGMT
13	SN10-2-11	RU27/RU28	MGMT
14	SN10-2-12	RU29/RU10	MGMT
15	SN10-2-13	RU31/RU32	MGMT
16	SN10-2-14	RU33/RU34	MGMT
17	SN10-2-15	RU37/RU38	MGMT
18	SN10-2-16	RU39/RU40	MGMT
19-26	-	-	-
27	PDU 1	Rear rack, right side, closest to front of rack/system	RJ45 labeled “1”
28	PDU 2	Rear rack, right side, between PDU1 and PDU3	RJ45 Labeled “1”
29	PDU 3	Rear rack, right side, closest to rack rear/door	RJ45 Labeled “1”
30	SN10-H-1	RU5/RU6	Labeled “M”
31	SN10-H-2	RU15/RU16	Labeled “M”
32	SN10-H-3	RU25/RU26	Labeled “M”
33	SN10-H-4	RU35/RU36	Labeled “M”
34-42	-	-	-
43	SN10-H-1	RU5/RU6	Left port Net0
44	SN10-H-2	RU15/RU16	Left port Net0
45	SN10-H-3	RU25/RU26	Left port Net0
46	SN10-H-4	RU35/RU36	Left port Net0
47	Uplink to customer access network	-	-

4.4. High-bandwidth data switch cabling

The DataScale SN10-8R is configured with the Juniper QFX5200-32C 100GbE Ethernet switch by default, which can optionally be replaced with a Mellanox SB7800 – Switch-IB 2 EDR 100Gbps InfiniBand (IB) switch.

4.4.1. High-bandwidth Ethernet switch (default configuration)

The Juniper QFX5200-32C switch, which is the standard high-bandwidth data switch in the DataScale SN10-8R, is installed and cabled in the factory. This switch is located in RU41.

Figure 8. Juniper QFX5200-32C high-bandwidth Ethernet data switch (front view)

Figure 9. Juniper QFX5200-32C high-bandwidth Ethernet data switch (rear view)

Table 9. Juniper QFX5200-32C high-bandwidth Ethernet data switch components
No.	Component	Connector/Cable Type
1	100GbE ports (ports 0–31)	QSFP28 (or compatible)
2	Network management port for em0	RJ45, Cat5e or higher
3	RJ45 console port connection (CON)	RJ45, Ethernet serial cable
4	Two power inlets	C13 to C14 power cord

4.4.2. High-bandwidth InfiniBand switch (optional configuration)

The Mellanox SB7800 100Gbps EDR InfiniBand switch is an optional high-bandwidth data switch in the DataScale SN10-8R rack that replaces the Juniper QFX5200-32c Ethernet switch. This 36-port switch is installed and cabled in the factory. The default partition is 0X7FFF. This component is located in RU41.

Figure 10. Optional Mellanox SB7800 high-bandwidth data switch (front view)

Figure 11. Optional Mellanox SB7800 high-bandwidth switch (rear view)

Table 10. Mellanox SB7800 high-Bandwidth data switch components
No.	Component	Connector/Cable Type
1	100Gbps ports (ports 1–36)	QSFP28 (or compatible)
2	Network management port	RJ45, Cat5e or higher
3	RJ45 serial console port connection (10101)	RJ45, Ethernet serial cable
4	Two power inlets	C13 to C14 power cord

4.5. SN10-H and SN10-2 high-bandwidth network cabling

The high-bandwidth cabling for the SN10-Hs and SN10-2 systems is installed in the factory before delivery. The cabling used for this is high-bandwidth TwinAx cables.

Figure 12. SN10-H high-bandwidth network card port assignment

Figure 13. SN10-2 high-bandwidth network card port assignment

Cabling requirements from the Juniper QFX5200-32c Ethernet switch for the high-bandwidth data network to the other DataScale SN10-8R components are listed in the table below.

Table 11. High-bandwidth data network port cabling (Ethernet, default)
From Data Network Switch Port	To Component	Component Location	Component Port
0	SN10-2-1	RU1/RU2	PCIe slot 1 - B
1	SN10-2-2	RU3/RU4	PCIe slot 1 - B
2	SN10-H-1	RU5/RU6	PCIe slot 1 - B
3	SN10-2-3	RU7/RU8	PCIe slot 1 - B
4	SN10-2-4	RU9/RU10	PCIe slot 1 - B
5	SN10-2-5	RU11/RU12	PCIe slot 1 - B
6	SN10-2-6	RU13/RU14	PCIe slot 1 - B
7	SN10-H-2	RU15/RU16	PCIe slot 1 - B
8	SN10-2-7	RU17/RU18	PCIe slot 1 - B
9	SN10-2-8	RU19/RU20	PCIe slot 1 - B
10	SN10-2-9	RU21/RU23	PCIe slot 1 - B
11	SN10-2-10	RU23/RU24	PCIe slot 1 - B
12	SN10-H-3	RU25/RU26	PCIe slot 1 - B
13	SN10-2-11	RU27/RU28	PCIe slot 1 - B
14	SN10-2-12	RU29/RU30	PCIe slot 1 - B
15	SN10-2-13	RU31/RU32	PCIe slot 1 - B
16	SN10-2-14	RU33/RU34	PCIe slot 1 - B
17	SN10-H-4	RU35/RU36	PCIe slot 1 - B
18	SN10-2-15	RU37/RU38	PCIe slot 1 - B
19	SN10-2-16	RU39/RU40	PCIe slot 1 - B
20–30	Additional uplinks to customer data network	-	-
31	Main uplink to customer data network	-	-

Cabling requirements from the Mellanox SB7800 InfiniBand switch for the high-bandwidth data network, replacing the Juniper QGX5200 Ethernet switch, to the other DataScale SN10-8R rack components are listed in Table 12 below.

Table 12. High-bandwidth data network port cabling (InfiniBand; optional)
From Data Network Switch Port	To Component	Component Location	Component Port
1	SN10-2-1	RU1/RU2	PCIe slot 1 - B
2	SN10-2-2	RU3/RU4	PCIe slot 1 - B
3	SN10-H-1	RU5/RU6	PCIe slot 1 - B
4	SN10-2-3	RU7/RU8	PCIe slot 1 - B
5	SN10-2-4	RU9/RU10	PCIe slot 1 - B
6	SN10-2-5	RU11/RU12	PCIe slot 1 - B
7	SN10-2-6	RU13/RU14	PCIe slot 1 - B
8	SN10-H-2	RU15/RU16	PCIe slot 1 - B
9	SN10-2-7	RU17/RU18	PCIe slot 1 - B
10	SN10-2-8	RU19/RU20	PCIe slot 1 - B
11	SN10-2-9	RU21/RU23	PCIe slot 1 - B
12	SN10-2-10	RU23/RU24	PCIe slot 1 - B
13	SN10-H-3	RU25/RU26	PCIe slot 1 - B
14	SN10-2-11	RU27/RU28	PCIe slot 1 - B
15	SN10-2-12	RU29/RU30	PCIe slot 1 - B
16	SN10-2-13	RU31/RU32	PCIe slot 1 - B
17	SN10-2-14	RU33/RU34	PCIe slot 1 - B
18	SN10-H-4	RU35/RU36	PCIe slot 1 - B
19	SN10-2-15	RU37/RU38	PCIe slot 1 - B
20	SN10-2-16	RU39/RU40	PCIe slot 1 - B
21-35	Additional uplinks to customer data network	-	-
36	Main uplink to customer data network	-	-

4.6. Intra-system SN10-2 and SN10-H cabling

Intra-system cabling is installed in the factory before delivery. These cables are not intended to be user serviceable. Intra-system SN10-H-to-SN102 and SN10-2-to-SN10-2 cables are specially designed QSFP-DD terminated cable pairs.

Figure 14. SN10-H Intra-system connection cards and port assignment

Please note that the port numbering for a slot 3 cards is reversed. This is because the PCIe card orientation is different from the other cards.

Figure 15. SN10-2 Intra-System connection port assignment

For cabling between the SN10-H server and the SN10-2 modules in a system, the ports are grouped with a blue and green pair. For example, the SN10-2 module’s port 1 consists of an upper blue port 1 and lower green port 1. This is highlighted in Figure 15.

For the SN10-H, there are two pairs of color-coded ports per PCIe card slot, and only one of the pairs is used. For example, in slot 1 of an SN10-H, there are 1B and 1G for port pair 1 and 2B and 2G for port pair 2, but for this particular card, only port pair 1 is used. This is shown in Figure 16.

In general, cabling between the SN10-H and the SN10-2s in a system looks as follows:

Figure 16. Topology of connections between the SN10-Hs to SN10-2s in a system

Where identified, the source blue port from the identified PCIe slot in the host goes to the corresponding target blue port 1 in the identified SN10-2s. Similarly, the source green port from the identified PCIe slot in the host goes to the corresponding target green port in the identified SN10-2. For example, from the host’s PCIe slot 1, slot 1 – Blue (1B) goes to the second SN10-2 in the node: Port 1 – Blue (1B). Each cable connection is detailed in Table 13 below. Also note that the plug ends on the cables are also color-coded green or blue.

Table 13. Intra-system cabling between SN10-Hs and SN10-2s
From SN10-H	Location	From SN10-H Card - Port	To SN10-2	Location	To SN10-2 Port
SN10-H-1	RU5/RU6	Slot 1-1B	SN10-2-2	RU3/RU4	1B
SN10-H-1	RU5/RU6	Slot 1-1G	SN10-2-2	RU3/RU4	1G
SN10-H-1	RU5/RU6	Slot 2-1B	SN10-2-3	RU7/RU8	1B
SN10-H-1	RU5/RU6	Slot 2-1G	SN10-2-3	RU7/RU8	1G
SN10-H-1	RU5/RU6	Slot 3-2B	SN10-2-1	RU1/RU2	1B
SN10-H-1	RU5/RU6	Slot 3-2G	SN10-2-1	RU1/RU2	1G
SN10-H-1	RU5/RU6	Slot 4-1B	SN10-2-4	RU9/RU10	1B
SN10-H-1	RU5/RU6	Slot 4-1G	SN10-2-4	RU9/RU10	1G
SN10-H-2	RU15/RU16	Slot 1-1B	SN10-2-6	RU13/RU14	1B
SN10-H-2	RU15/RU16	Slot 1-1G	SN10-2-6	RU13/RU14	1G
SN10-H-2	RU15/RU16	Slot 2-1B	SN10-2-7	RU17/RU18	1B
SN10-H-2	RU15/RU16	Slot 2-1G	SN10-2-7	RU17/RU18	1G
SN10-H-2	RU15/RU16	Slot 3-2B	SN10-2-5	RU11/RU12	1B
SN10-H-2	RU15/RU16	Slot 3-2G	SN10-2-5	RU11/RU12	1G
SN10-H-2	RU15/RU16	Slot 4-1B	SN10-2-8	RU19/RU20	1B
SN10-H-2	RU15/RU16	Slot 4-1G	SN10-2-8	RU19/RU20	1G
SN10-H-3	RU25/RU26	Slot 1-1B	SN10-2-10	RU23/RU24	1B
SN10-H-3	RU25/RU26	Slot 1-1G	SN10-2-10	RU23/RU24	1G
SN10-H-3	RU25/RU26	Slot 2-1B	SN10-2-11	RU28/RU29	1B
SN10-H-3	RU25/RU26	Slot 2-1G	SN10-2-11	RU28/RU29	1G
SN10-H-3	RU25/RU26	Slot 3-2B	SN10-2-9	RU21/RU22	1B
SN10-H-3	RU25/RU26	Slot 3-2G	SN10-2-9	RU21/RU22	1G
SN10-H-3	RU25/RU26	Slot 4-1B	SN10-2-12	RU29/RU30	1B
SN10-H-3	RU25/RU26	Slot 4-1G	SN10-2-12	RU29/RU30	1G
SN10-H-4	RU35/RU36	Slot 1-1B	SN10-2-14	RU33/RU34	1B
SN10-H-4	RU35/RU36	Slot 1-1G	SN10-2-14	RU33/RU34	1G
SN10-H-4	RU35/RU36	Slot 2-1B	SN10-2-15	RU37/RU38	1B
SN10-H-4	RU35/RU36	Slot 2-1G	SN10-2-15	RU37/RU38	1G
SN10-H-4	RU35/RU36	Slot 3-2B	SN10-2-13	RU31/RU32	1B
SN10-H-4	RU35/RU36	Slot 3-2G	SN10-2-13	RU31/RU32	1G
SN10-H-4	RU35/RU36	Slot 4-1B	SN10-2-16	RU39/RU40	1B
SN10-H-4	RU35/RU36	Slot 4-1G	SN10-2-16	RU39/RU40	1G

4.7. SN10-2 to SN10-2 Intra-system cabling

For the intra-system cabling between SN10-2 systems, the ports are grouped with a blue and green pair. For example, port 1 consists of blue port 1 above and green port 1 beneath. This is highlighted in Figure 17.

In general, the intra-system cabling of a SN10-2 system to other SN10-2 systems is done as follows:

Topology of SN10-2 Interconnects in a Node

Figure 17. Topology of SN10-2 interconnects in a system

Where identified, the source blue port goes to the corresponding target blue port, and the source green port goes to the corresponding target green port. For example, from the first SN10-2 in a system, the Port 3 – Blue (3B) goes to the second SN10-2 in the system: Port 5 – Blue (5B). Each cable connection is detailed in the Table 14 below.

Table 14. SN10-2 to SN10-2 Intra-system cabling
From SN10-2	Location	Port	To SN10-2	Location	Port
SN10-2-1	RU1/RU2	3B	SN10-2-2	RU3/RU4	5B
SN10-2-1	RU1/RU2	3G	SN10-2-2	RU3/RU4	5G
SN10-2-1	RU1/RU2	5B	SN10-2-2	RU3/RU4	3B
SN10-2-1	RU1/RU2	5G	SN10-2-2	RU3/RU4	3G
SN10-2-1	RU1/RU2	6B	SN10-2-4	RU9/RU10	7B
SN10-2-1	RU1/RU2	6G	SN10-2-4	RU9/RU10	7G
SN10-2-1	RU1/RU2	7B	SN10-2-2	RU3/RU4	8B
SN10-2-1	RU1/RU2	7G	SN10-2-2	RU3/RU4	8G
SN10-2-1	RU1/RU2	8B	SN10-2-2	RU3/RU4	7B
SN10-2-1	RU1/RU2	8G	SN10-2-2	RU3/RU4	8G
SN10-2-2	RU3/RU4	6B	SN10-2-3	RU7/RU8	7B
SN10-2-2	RU3/RU4	6G	SN10-2-3	RU7/RU8	7G
SN10-2-3	RU7/RU8	3B	SN10-2-4	RU9/RU10	5B
SN10-2-3	RU7/RU8	3G	SN10-2-4	RU9/RU10	5G
SN10-2-3	RU7/RU8	5B	SN10-2-4	RU9/RU10	3B
SN10-2-3	RU7/RU8	5G	SN10-2-4	RU9/RU10	3G
SN10-2-3	RU7/RU8	6B	SN10-2-4	RU9/RU10	8B
SN10-2-3	RU7/RU8	6G	SN10-2-4	RU9/RU10	8G
SN10-2-3	RU7/RU8	8B	SN10-2-4	RU9/RU10	6B
SN10-2-3	RU7/RU8	8G	SN10-2-4	RU9/RU10	6G
SN10-2-5	RU11/RU12	3B	SN10-2-6	RU13/RU14	5B
SN10-2-5	RU11/RU12	3G	SN10-2-6	RU13/RU14	5G
SN10-2-5	RU11/RU12	5B	SN10-2-6	RU13/RU14	3B
SN10-2-5	RU11/RU12	5G	SN10-2-6	RU13/RU14	3G
SN10-2-5	RU11/RU12	6B	SN10-2-8	RU19/RU20	7B
SN10-2-5	RU11/RU12	6G	SN10-2-8	RU19/RU20	7G
SN10-2-5	RU11/RU12	7B	SN10-2-6	RU13/RU14	8B
SN10-2-5	RU11/RU12	7G	SN10-2-6	RU13/RU14	8G
SN10-2-5	RU11/RU12	8B	SN10-2-6	RU13/RU14	7B
SN10-2-5	RU11/RU12	8G	SN10-2-6	RU13/RU14	8G
SN10-2-6	RU13/RU14	6B	SN10-2-7	RU17/RU18	7B
SN10-2-6	RU13/RU14	6G	SN10-2-7	RU17/RU18	7G
SN10-2-7	RU17/RU18	3B	SN10-2-8	RU19/RU20	5B
SN10-2-7	RU17/RU18	3G	SN10-2-8	RU19/RU20	5G
SN10-2-7	RU17/RU18	5B	SN10-2-8	RU19/RU20	3B
SN10-2-7	RU17/RU18	5G	SN10-2-8	RU19/RU20	3G
SN10-2-7	RU17/RU18	6B	SN10-2-8	RU19/RU20	8B
SN10-2-7	RU17/RU18	6G	SN10-2-8	RU19/RU20	8G
SN10-2-7	RU17/RU18	8B	SN10-2-8	RU19/RU20	6B
SN10-2-7	RU17/RU18	8G	SN10-2-8	RU19/RU20	6G
SN10-2-9	RU21/RU22	3B	SN10-2-10	RU23/RU24	5B
SN10-2-9	RU21/RU22	3G	SN10-2-10	RU23/RU24	5G
SN10-2-9	RU21/RU22	5B	SN10-2-10	RU23/RU24	3B
SN10-2-9	RU21/RU22	5G	SN10-2-10	RU23/RU24	3G
SN10-2-9	RU21/RU22	6B	SN10-2-10	RU23/RU24	7B
SN10-2-9	RU21/RU22	6G	SN10-2-10	RU23/RU24	7G
SN10-2-9	RU21/RU22	7B	SN10-2-10	RU23/RU24	8B
SN10-2-9	RU21/RU22	7G	SN10-2-10	RU23/RU24	8G
SN10-2-9	RU21/RU22	8B	SN10-2-10	RU23/RU24	7B
SN10-2-9	RU21/RU22	8G	SN10-2-10	RU23/RU24	8G
SN10-2-10	RU23/RU24	6B	SN10-2-11	RU27/RU29	7B
SN10-2-10	RU23/RU24	6G	SN10-2-11	RU27/RU29	7G
SN10-2-11	RU27/RU28	3B	SN10-2-12	RU29/RU30	5B
SN10-2-11	RU27/RU28	3G	SN10-2-12	RU29/RU30	5G
SN10-2-11	RU27/RU28	5B	SN10-2-12	RU29/RU30	3B
SN10-2-11	RU27/RU28	5G	SN10-2-12	RU29/RU30	3G
SN10-2-11	RU27/RU28	6B	SN10-2-12	RU29/RU30	8B
SN10-2-11	RU27/RU28	6G	SN10-2-12	RU29/RU30	8G
SN10-2-11	RU27/RU28	8B	SN10-2-12	RU29/RU30	6B
SN10-2-11	RU27/RU28	8G	SN10-2-12	RU29/RU30	6G
SN10-2-13	RU31/RU32	3B	SN10-2-14	RU33/RU34	5B
SN10-2-13	RU31/RU32	3G	SN10-2-14	RU33/RU34	5G
SN10-2-13	RU31/RU32	5B	SN10-2-14	RU33/RU34	3B
SN10-2-13	RU31/RU32	5G	SN10-2-14	RU33/RU34	3G
SN10-2-13	RU31/RU32	6B	SN10-2-14	RU33/RU34	7B
SN10-2-13	RU31/RU32	6G	SN10-2-14	RU33/RU34	7G
SN10-2-13	RU31/RU32	7B	SN10-2-14	RU33/RU34	8B
SN10-2-13	RU31/RU32	7G	SN10-2-14	RU33/RU34	8G
SN10-2-13	RU31/RU32	8B	SN10-2-14	RU33/RU34	7B
SN10-2-13	RU31/RU32	8G	SN10-2-14	RU33/RU34	8G
SN10-2-14	RU33/RU34	6B	SN10-2-15	RU37/RU38	7B
SN10-2-14	RU33/RU34	6G	SN10-2-15	RU37/RU38	7G
SN10-2-15	RU37/RU38	3B	SN10-2-16	RU39/RU40	5B
SN10-2-15	RU37/RU38	3G	SN10-2-16	RU39/RU40	5G
SN10-2-15	RU37/RU38	5B	SN10-2-16	RU39/RU40	3B
SN10-2-15	RU37/RU38	5G	SN10-2-16	RU39/RU40	3G
SN10-2-15	RU37/RU38	6B	SN10-2-16	RU39/RU40	8B
SN10-2-15	RU37/RU38	6G	SN10-2-16	RU39/RU40	8G
SN10-2-15	RU37/RU38	8B	SN10-2-16	RU39/RU40	6B
SN10-2-15	RU37/RU38	8G	SN10-2-16	RU39/RU40	6G

4.8. Power cabling the DataScale SN10-8R

There are three power distribution units (PDUs) installed in the DataScale SN10-8R that provide redundant power in case of a single PDU failure. They are located on the right side of the rack when you are facing the rear of the rack. Table 15 below details the PDU identification and location within the rack.

Table 15. PDU location in DataScale SN10-8R
PDU Identification	Location in Rack (Facing Rack Rear)
PDU1	Right side of rack, closest to the rack front/closest to the systems
PDU2	Right side of rack, in between PDU1 and PDU3
PDU3	Right side of rack, closest to the rack rear/furthest from the systems

Depending on the number of systems in the DataScale SN10-8R, port population of the PDUs will vary, but the same outlets are always used for the same systems regardless of rack configuration. Table 16 below shows all the needed connections for a four-system rack, but it also applies to a rack with fewer populated systems. DataScale SN10-8R are delivered pre-installed with power cables from the factory.

Unused ports must NOT be used for any reason, and only the designated ports should be used. This prevent overloading of breaker circuits in the PDU and prevents unexpected outages.

Table 16. PDU to component power supply mapping
From Rack Unit	From Component	To PDU1 Port	To PDU2 Port	To PDU3 Port
42	Juniper EX4300 access switch PSU 1 (Left PSU when facing rear of switch)	2
42	Juniper EX4300 access switch PSU 2 (right PSU when facing rear of switch)			17
42	Lantronix console server PSU1 (left PSU when facing rear of Lantronix)		1
42	Lantronix console server PSU2 (right PSU when facing rear of Lantronix)			19
41	Juniper QFX5200-32c Eth (default) PSU 1 or Mellanox SB7800 IB (optional) PSU 1 (left PSU when facing rear of switch)	3
41	Juniper QFX5200-32c Eth (default) PSU 1 or Mellanox SB7800 IB (optional) PSU 1 (right PSU when facing rear of switch)		17
40	SN10-2-16 top PSU (four-system)		3
39	SN10-2-16 bottom PSU (four-system)			2
38	SN10-2-15 top PSU (four-system)	5
37	SN10-2-15 bottom PSU (four-system)		5
35	SN10-H-4 right PSU (four-system)			16
35	SN10-H-4 left PSU (four-system)	7
34	SN10-2-14 top PSU (four-system)		7
33	SN10-2-14 bottom PSU (four-system)			1
32	SN10-2-13 top PSU (four-system)	9
31	SN10-2-13 bottom PSU (four-system)		9
30	SN10-2-12 top PSU (three-system)			3
29	SN10-2-12 bottom PSU (three-system)	11
28	SN10-2-11 top PSU (three-system)		2
27	SN10-2-11 bottom PSU (three-system)			11
25	SN10-H-3 right PSU (three-system)	13
25	SN10-H-3 left PSU (three-system)		13
24	SN10-2-10 top PSU (three-system)			7
23	SN10-2-10 bottom PSU (three-system)	15
22	SN10-2-9 top PSU (three-system)		15
21	SN10-2-9 bottom PSU (three-system)			15
20	SN10-2-8 top PSU (two-system)	16
19	SN10-2-8 bottom PSU (two-system)		16
18	SN10-2-7 top PSU (two-system)	18
17	SN10-2-7 bottom PSU (two-system)		19
15	SN10-H-2 right PSU (two-system)			18
15	SN10-H-2 left PSU (two-system)		18
14	SN10-2-6 top PSU (two-system)	17
13	SN10-2-6 bottom PSU (two-system)		23
12	SN10-2-5 top PSU (two-system)			25
11	SN10-2-5 bottom PSU (two-system)	21
10	SN10-2-4 top PSU (one-system)		21
9	SN10-2-4 bottom PSU (one-system)			21
8	SN10-2-3 top PSU (one-system)	25
7	SN10-2-3 bottom PSU (one-system)			13
5	SN10-H-1 right PSU (one-system)		29
5	SN10-H-1 left PSU (one-system)			23
4	SN10-2-2 top PSU (one-system)	27
3	SN10-2-2 bottom PSU (one-system)		11
2	SN10-2-1 top PSU (one-system)			29
1	SN10-2-1 bottom PSU (one-system)	29

5. Site preparation for DataScale SN10-8R installation

Please ensure the following specifications are met regarding physical space, power, cooling, networking, and cabling when preparing a site for rack installation.

5.1. Physical specifications for each DataScale SN10-8R

Table 17. DataScale SN10-8R physical specification
Description	English	Metric
Height	78.5 in.	1,994 mm
Width	24 in.	610 mm
Depth	50 in.	1,270 mm
Packaged shipping height	88.5 in.	2,248 mm
Packaged shipping width	42 in.	1,067 mm
Packaged shipping depth	64 in.	1,626 mm

Table 18. DataScale SN10-8R required clearance
Description	English	Metric
Minimum clearance cabinet top to overhead infrastructure	24 in.	610 mm
Minimum clearance to front of rack for system installation and service	32 in.	812 mm
Minimum clearance at rack rear for service	24 in.	610 mm

Table 19. DataScale SN10-8R weight specification
Configuration	Weight		Shipping Weight
	English	Metric	English	Metric
DataScale SN10-8R Quarter Rack	755 lbs.	343 kg	1055 lbs.	480 kg
DataScale SN10-8R Half Rack	1065 lbs.	484 kg	1365 lbs.	620 kg
DataScale SN10-8R Full Rack	1665 lbs.	756 kg	2025 lbs.	919 kg

5.2. Power requirements

Power requirements depend on the specific rack you are installing. Your SambaNova representative will discuss power draw, facility power requirements, and grounding requirements when you fill out your site-specific forms.

5.3. Network requirements

The DataScale SN10-8R requires certain networks to be provided as well as to be managed and maintained. This section describes these networks and what is required from the customer.

5.3.1. Network architecture and layout

There are in general two networks that are used within the DataScale SN10-8R:

Data network
Access network

Figure 18. Network topology diagram

The data network is a high-bandwidth (100GbE) Ethernet network used for the RDUs and SN10-H to take in data from an external storage source and for inter-RDU communication. Optionally, the 100GbE Ethernet network switch can be replaced with a 100Gbps InfiniBand switch.

The access network is a 1GbE network used to log in to the DataScale SN10-8R components. This network connects to each component’s baseboard management controller (BMC) for power control, configuration, and monitoring/debugging, as well to perform any other service operations. This network also connects to the systems’ SN10-H operating system (OS) to perform day-to-day user operations on the systems. This could include such operations as streaming logs, performing software updates, and performing other day-to-day system administration tasks.

Optionally, customers may choose to partition the access switch using port-based VLANs to separate access to the management interfaces of the DataScale SN10-8R components and the SN10-H OS interfaces. The customer is responsible for configuring this added network segregation. Please refer to Table 8 for device to port mappings, and available ports that can be used if an additional network drop is required for the separate network.

These two networks should typically be allocated a separate subnet to ensure the isolation of traffic, to regulate access to these networks, and to not hinder the performance of the data network.

5.3.2. Network infrastructure requirements

A dedicated subnet and downlink should be allocated for each of the networks described in Section 5.3.1. A CIDR /26 subnet, or a subnet of 62 usable IP addresses, for the access network and a CIDR /27 subnet, or 30 usable IP addresses, for the data network provide sufficient IP addresses for each of the networks. There is an assumption that the first three IP addresses on each subnet will be dedicated for redundant customer network infrastructure (for example, gateway VIP, switch failover IPs, and so on).

In addition to the creation of these two subnets, three physical downlinks, two for the access network, and one for the data network need to be provided.

One downlink for the access network connects directly to the access switch
The second downlink for the access network connects to the serial console server in the top of the DataScale SN10-8R as described in Section 4.2.
The downlink for the data network connects to the data switch.

The high-bandwidth switch may require that the customer configure multiple connections using a network aggregation (LACP/LAG) group in order to provide the necessary level of bandwidth to and from the rack. By default, only a single port is configured as an uplink. Details for each of these networks are described in the table below.

Table 20. DataScale SN10-8R infrastructure network requirements
Network	Switch	Uplink Port	Port Type
Data (/27)	Juniper QFX5200 data Ethernet switch (default)	31	QSFP28
Data (/27)	Mellanox SB7800 data InfiniBand switch (optional, replacing QFX)	36	QSFP28
Access (/26)	Juniper 1G access switch	47	RJ45
Access (/26)	Serial console server	10/100/1000 management Network (labeled “1”)	RJ45

To avoid potential spanning tree issues when initially connecting the rack to a customer network, do NOT plug in multiple uplinks from the data switch to the data network. Only the single uplink on port 31 of the Juniper data switch should be used initially. The default configuration of the switch has not been configured for the customer’s specific spanning tree configuration and multiple ports may lead to spanning tree issues. Once local switch administrators have implemented proper local spanning tree configurations on the Juniper data switch, multiple uplinks can be used.

Similarly, if the optional Mellanox IB switch is used for the data network instead of the default Juniper Ethernet switch, ensure that only a single uplink on port 36 is connected to the IB switch unless the InfiniBand subnet manager (SM) has been appropriately configured to allow multiple uplinks for the IB switch.

5.4. DataScale SN10-8R cabling

5.4.1. Cabling datacenter networks requirements

Prior to the delivery of the DataScale SN10-8R rack, ensure that the datacenter network and power cabling have been prepared.

Route the required network outlets from the datacenter network infrastructure to where the DataScale SN10-8R will be installed.
Ensure that the configurations described in Section 5.3 of this document have been completed.
It is good practice to label the cables that are used to connect the two networks to the DataScale SN10-8R.

5.4.2. Cabling datacenter power requirements

Route the needed power drops to where the DataScale SN10-8R will be installed.
Ensure the power drops meet the power requirements, socket type, and socket quantity described in Section 5.2 of this document.

6. Hardware installation tasks

Once the SambaNova DataScale SN10-8R is rolled to its location on the datacenter floor, perform the following:

Inspect the rack for any damage that may have resulted from local handling or relocation within the customer facility.
Allow the system to reside on the datacenter floor for 24 hours under ambient environmental conditions before powering on the rack. This allows the components to acclimatize to the environment, prevents the effects of sudden cooling or heating, and removes the possibility of any condensation accumulation.
Attach the rack’s bus grounding bar to the facility’s ground, as described in [Grounding requirements].
Typically, the rack should be permanently attached to the datacenter floor in its intended location using the tie-down brackets and bolts. Users should consult their facilities staff regarding preparing the tie-down bolt holes and locations. If it is standard procedure to lower the rack feet in each corner of the rack, please do so here.
Connect the datacenter network cable drop to the Juniper high-bandwidth data switch to port 31 or to port 36 if the optional Mellanox high-bandwidth data switch is used, per the table in Section 5.3.2.
Connect one datacenter access network cable drop to the Juniper Access switch to port 47, per the table in Section 5.3.2.
Connect the second datacenter access network cable drop to the Lantronix serial console server management interface port, labeled “1”, per Table 5 earlier in this document.
Connect the three power plugs from the PDUs to the three 3-phase power sockets, as described in [Facility power requirements].
Inspect the rack during power-on to make sure there are no issues with the PDUs or other rack components. Determine whether any warning LEDs on the rack components are lit. If warning or error lights appear, please refer to the support documentation for information on how to resolve the issue. Documentation can be found at https://support.sambanova.ai.

The hardware installation tasks may be handled by the SambaNova onsite engineering team if that is requested.

7. Spare parts kit

DataScale SN10-8R are delivered with a spare parts kit that includes the following:

SN10-H Solid State Drive (SSD) including carrier
SN10-2 fan module

Store these spare parts in a secure place because they will be asked for if they are needed.

8. Disclaimers

All product names, trademarks, and registered trademarks are property of their respective owners.