> For the complete documentation index, see [llms.txt](https://docs.tibero.com/zetadata.en/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.tibero.com/zetadata.en/network-configuration.md).
# Network Configuration
This chapter describes how to configure the network of ZetaData's DB nodes and Storage nodes.
## Overview
ZetaData uses InfiniBand for fast network communication between DB nodes and Storage nodes. Follow the steps below to set up a ZetaData network using InfiniBand.
1. Install the InfiniBand driver.
2. Verify the operation of the InfiniBand driver.
3. Configure kernel parameters.
4. Configure a network (refer to the network configuration manual of each InfiniBand manufacturer).
{% hint style="info" %}
Note
If there is no InfiniBand or there is a problem with usage, then the network can be configured by
\
using the TCP protocol instead of the RDMA protocol.
Refer to the 'SSVR\_USE\_TCP parameter' in "Initialization Parameters" for instructions.
{% endhint %}
The figure below shows the IP address configuration for InfiniBand on the ZetaData node that will be used in the examples going forward.
###
***
## **1.** InfiniBand Driver Installation
To use InfiniBand, install a driver and library appropriate for the OS. Check the library by using each OS
\
command and install it first.
The following is an example of installing a library for each OS.
Ubuntu
```sql
$ sudo apt-cache search libibverb
```
Install the package 'Development files for the libibverbs library' searched with the above command.
```sql
$ sudo apt-get install libibverbs-dev
```
RedHat series (RHEL, CentOS, Fedora)
```sql
$ yum install libibverbs-devel libibverbs-devel-static
```
The following is an example of installing the InfiniBand driver on Red Hat 7,8 and 9 provided by Mellanox.
1\. Install the Melanox driver.
```sql
$ ./mlnxofedinstall
```
2. Create a network configuration file for InfiniBand. Below is an example of the configuration file on DB node 0.
```
$ echo "CONNECTED_MODE=yes
> TYPE=InfiniBand
> BOOTPROTO=none
> DEFROUTE=yes
> #IPV4_FAILURE_FATAL=no
> #IPV6INIT=yes
> #IPV6_AUTOCONF=yes
> #IPV6_DEFROUTE=yes
> #IPV6_FAILURE_FATAL=no
> NAME=ib0
> #UUID=6842318b-35d2-475a-8ae2-b3f9ec6b787a
> DEVICE=ib0
> ONBOOT=yes
> IPADDR=10.10.10.11
> NETMASK=255.255.255.0
> #PREFIX=32
> #IPV6_PEERDNS=yes
> #IPV6_PEERROUTES=yes" > /etc/sysconfig/network-scripts/ifcfg-ib0
```
3. Execute the opensmd service for switches that do not support the subnet manager function.
```
$ systemctl enable opensmd.service
$ service opensmd start
```
4. Restart the server.
###
***
## **2.** Verification of the InfiniBand driver operation
The following is an example of verifying the operation of the InfiniBand driver provided by Mellanox.
#### DB node #0
```sql
$ ip addr show ib0
8: ib0: mtu 65520 qdisc pfifo_fast \
state UP qlen 1024
link/infiniband 80:00:00:2c:fe:80:00:00:00:00:00:00:50:65:f3:ff:ff:88:81:d0 \
brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
inet 10.10.10.11/24 brd 10.10.10.255 scope global ib0
valid_lft forever preferred_lft forever
inet6 fe80::5265:f3ff:ff88:81d0/64 scope link
valid_lft forever preferred_lft forever
$ ibv_rc_pingpong
local address: LID 0x0006, QPN 0x008382, PSN 0xc85390, GID ::
remote address: LID 0x0002, QPN 0x02e388, PSN 0xb0671b, GID ::
8192000 bytes in 0.01 seconds = 10726.02 Mbit/sec
1000 iters in 0.01 seconds = 6.11 usec/iter
```
#### DB node #0
```sql
$ ip addr show ib0
8: ib0: mtu 65520 qdisc pfifo_fast \
state UP qlen 1024
link/infiniband 80:00:00:2c:fe:80:00:00:00:00:00:00:50:65:f3:ff:ff:88:71:40 \
brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
inet 10.10.10.12/24 brd 10.10.10.255 scope global ib0
valid_lft forever preferred_lft forever
inet6 fe80::5265:f3ff:ff88:7140/64 scope link
valid_lft forever preferred_lft forever
$ ibv_rc_pingpong 10.10.10.11
local address: LID 0x0002, QPN 0x02e388, PSN 0xb0671b, GID ::
remote address: LID 0x0006, QPN 0x008382, PSN 0xc85390, GID ::
8192000 bytes in 0.00 seconds = 13317.62 Mbit/sec
1000 iters in 0.00 seconds = 4.92 usec/iter
```
{% hint style="info" %}
**Note**
In the above example, only tests were performed between DB nodes using the ibv\_rc\_pinpong command, but it is recommended to perform InfiniBand driver operation tests from all nodes, including SSVR nodes, to each ZetaData node.
{% endhint %}
###