Network Configuration

This chapter describes how to configure the network of ZetaData's DB nodes and Storage nodes.

Overview

ZetaData uses InfiniBand for fast network communication between DB nodes and Storage nodes. Follow the steps below to set up a ZetaData network using InfiniBand.

1. Install the InfiniBand driver.

2. Verify the operation of the InfiniBand driver.

3. Configure kernel parameters.

4. Configure a network (refer to the network configuration manual of each InfiniBand manufacturer).

Note

If there is no InfiniBand or there is a problem with usage, then the network can be configured by using the TCP protocol instead of the RDMA protocol.

Refer to the 'SSVR_USE_TCP parameter' in "Initialization Parameters" for instructions.

The figure below shows the IP address configuration for InfiniBand on the ZetaData node that will be used in the examples going forward.

Figure 1. ZetaData Node InfiniBand IP address configuration diagram

---

1. InfiniBand Driver Installation

To use InfiniBand, install a driver and library appropriate for the OS. Check the library by using each OS command and install it first.

The following is an example of installing a library for each OS.

Ubuntu

$ sudo apt-cache search libibverb

Install the package 'Development files for the libibverbs library' searched with the above command.

$ sudo apt-get install libibverbs-dev

RedHat series (RHEL, CentOS, Fedora)

$ yum install libibverbs-devel libibverbs-devel-static

The following is an example of installing the InfiniBand driver on Red Hat 7,8 and 9 provided by Mellanox.

1. Install the Melanox driver.

$ ./mlnxofedinstall

1. Create a network configuration file for InfiniBand. Below is an example of the configuration file on DB node 0.

$ echo "CONNECTED_MODE=yes
> TYPE=InfiniBand
> BOOTPROTO=none
> DEFROUTE=yes
> #IPV4_FAILURE_FATAL=no
> #IPV6INIT=yes
> #IPV6_AUTOCONF=yes
> #IPV6_DEFROUTE=yes
> #IPV6_FAILURE_FATAL=no
> NAME=ib0
> #UUID=6842318b-35d2-475a-8ae2-b3f9ec6b787a
> DEVICE=ib0
> ONBOOT=yes
> IPADDR=10.10.10.11
> NETMASK=255.255.255.0
> #PREFIX=32
> #IPV6_PEERDNS=yes
> #IPV6_PEERROUTES=yes" > /etc/sysconfig/network-scripts/ifcfg-ib0

1. Execute the opensmd service for switches that do not support the subnet manager function.

$ systemctl enable opensmd.service
$ service opensmd start

1. Restart the server.

---

2. Verification of the InfiniBand driver operation

The following is an example of verifying the operation of the InfiniBand driver provided by Mellanox.

DB node #0

$ ip addr show ib0
8: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc pfifo_fast \
state UP qlen 1024
link/infiniband 80:00:00:2c:fe:80:00:00:00:00:00:00:50:65:f3:ff:ff:88:81:d0 \
brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
inet 10.10.10.11/24 brd 10.10.10.255 scope global ib0
valid_lft forever preferred_lft forever
inet6 fe80::5265:f3ff:ff88:81d0/64 scope link
valid_lft forever preferred_lft forever
$ ibv_rc_pingpong
local address: LID 0x0006, QPN 0x008382, PSN 0xc85390, GID ::
remote address: LID 0x0002, QPN 0x02e388, PSN 0xb0671b, GID ::
8192000 bytes in 0.01 seconds = 10726.02 Mbit/sec
1000 iters in 0.01 seconds = 6.11 usec/iter

DB node #0

$ ip addr show ib0
8: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc pfifo_fast \
state UP qlen 1024
link/infiniband 80:00:00:2c:fe:80:00:00:00:00:00:00:50:65:f3:ff:ff:88:71:40 \
brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
inet 10.10.10.12/24 brd 10.10.10.255 scope global ib0
valid_lft forever preferred_lft forever
inet6 fe80::5265:f3ff:ff88:7140/64 scope link
valid_lft forever preferred_lft forever
$ ibv_rc_pingpong 10.10.10.11
local address: LID 0x0002, QPN 0x02e388, PSN 0xb0671b, GID ::
remote address: LID 0x0006, QPN 0x008382, PSN 0xc85390, GID ::
8192000 bytes in 0.00 seconds = 13317.62 Mbit/sec
1000 iters in 0.00 seconds = 4.92 usec/iter

Note

In the above example, only tests were performed between DB nodes using the ibv_rc_pinpong command, but it is recommended to perform InfiniBand driver operation tests from all nodes, including SSVR nodes, to each ZetaData node.