Associating a Red Hat Enterprise Linux Operating Environment with a Secure Fabric

Assigning IP Addresses for Default InfiniBand Ports

Typically, the IP addresses for the default InfiniBand ports were assigned during initial operating system installation and configuration. For more information on assigning IP addresses for the default InfiniBand ports, refer to the ClearPath Forward Installation and Getting Started Guide.

Creating a Virtual IP over InfiniBand (IPoIB) Interface

When configuring a Red Hat LINUX operating environment for attachment to a secure fabric, a virtual IP over InfiniBand (IPoIB) interface must be manually created using the subnet tag (PKEY) for each secure fabric the Red Hat Linux operating environment will associate with. In addition, for the virtual IPoIB interface to operate in a high availability (active-backup failover) mode, both ports of the InfiniBand HCA card must each have a virtual IPoIB interface associated with the secure fabric, and the virtual IPoIB interfaces must be bonded in active-backup mode.

Notes:

The use of bonded virtual IPoIB interfaces together with the bonding of the default IPoIB interfaces in a Red Hat Linux operating environment is restricted in the current release. If the default IPoIB interfaces are bonded, be sure to reconfigure the default IPoIB interfaces to remove the bond before bonding the virtual IPoIB interfaces. For more information, refer to the ClearPath Forward Installation and Getting Started Guide.
For Red Hat Enterprise Linux (RHEL) versions prior to RHEL version 7.2, be sure to perform the configuration tasks in the Additional Configuration Tasks for RHEL versions prior to RHEL 7.2 subsection to make interfaces to the secure fabric operational.

To enable and persist a virtual IPoIB interface, and assign an IP address to the interface, create a configuration file for each interface in the /etc/sysconfig/network-scripts directory. The syntax for the name of the configuration file is:

ifcfg-ibX.YYYY

Where X is 0 or 1, depending on the InfiniBand HCA port; and YYYY is the secure fabric subnet tag (PKEY) in hexadecimal.

Note: The format of the subnet tag (PKEY) is 0xYYYY where the highest order bit is always 1. For example, a subnet tag (PKEY) of 2 would be 0x8002.

For example, to create configuration files for a bonded virtual IPoIB interface for the secure fabric with the subnet tag (PKEY) of 4000 (4 hex-digit 0x0FA0), log in as root, and then create the following files:

A file named ifcfg-ib0.8fa0 in the /etc/sysconfig/network-scripts directory with the following content:

DEVICE=ib0.8fa0
TYPE=Infiniband
ONBOOT=yes
PKEY=yes
PKEY_ID=4000
PHYSDEV=ib0
SLAVE=yes
MASTER=bond0.8fa0
USERCTL=no

A file named ifcfg-ib1.8fa0 in the /etc/sysconfig/network-scripts directory with the following content:

DEVICE=ib1.8fa0
TYPE=Infiniband
ONBOOT=yes
PKEY=yes
PKEY_ID=4000
PHYSDEV=ib1
SLAVE=yes
MASTER=bond0.8fa0
USERCTL=no

A file named ifcfg-bond0.8fa0 in the /etc/sysconfig/network-scripts directory with the following content:

DEVICE=bond0.8fa0
TYPE=Bond
IPADDR=172.40.7.1
PREFIX=16
ONBOOT=yes
BOOTPROTO=none
MTU=
USERCTL=no
BONDING_OPTS="mode=active-backup miimon=100"
NAME='bond0.8fa0'

Be sure to restart all InfiniBand interfaces and apply changes:

systemctl restart network
nmcli con reload

Defining Bonds in the Bond.conf File

Define bonds in the file /etc/modprobe.d/bond.conf.

Ensure the format of each entry is in the following form:

alias <bond device name> bonding

For example,

alias bond0 bonding
alias bond0.8fa0 bonding

Verifying Connection

If desired, verify that your Red Hat Linux operating environment is connected to the secure fabric by issuing a ping command to another operating environment (if available) participating in the secure fabric.

Note: Occasionally, after network services restart, a ping to the source address parameter may fail until you try to ping the source address of the companion port. For example: ping -I ib0 172.31.x.y may fail until you try ping -I ib1 172.31.x.y (where 172.31.x.y is a valid IP address on the subnet).

Additional Configuration Tasks for RHEL versions prior to RHEL 7.2

Log in to your Red Hat Linux operating environment as root, and in the interface configuration file for the bond0 interface ifcfg-bond0 in the /etc/sysconfig/network-scripts/ directory, change the ONBOOT value from yes to no.
```
ONBOOT=no
```
Create the virtual network interfaces for each of the secure fabrics that your Red Hat Linux operating environment will access: Execute the following commands for each secure fabric, using the subnet tag (PKEY) value of the secure fabric that your Red Hat Linux operating environment will access.
```
# echo <subnet tag> > /sys/class/net/ib0/create_child
# echo <subnet tag> > /sys/class/net/ib1/create_child
```
Where <subnet tag> is hexadecimal value with a leading 0x, for example, 0x401.
After creating the child interfaces for all associated secure fabrics, restart the network services:
```
# systemctl restart network
```
Issue the interface up command to activate the default IP-LAN (bond0) secure fabric interface:
```
# ifup bond0
```
Note: You will need to issue this command after each reboot of the operating system to activate the default IP-LAN secure fabric interface.