qeth devices, namely OSA-Express and HiperSockets, have a vast array of functionalities that is easy to get lost in. This entry illustrates some of the most commonly sought functionalities, while trying to avoid confusing the reader with too much background information.
TLDR:
- IBM
z14: For KVM, always use separate OSA devices on source and target for
LGM; For OVS, use a primary bridgeport with OSA, and VNIC
characteristics with HiperSockets.
- IBM z15: For KVM with
OVS, use VNIC characteristics for any qeth device; for KVM and MacVTap,
use VNIC characteristics if you want to use the same device on source
and target system in LGM scenarios.
Bridgeport Mode
Initially, the only way to enable promiscuous mode on OSA-Express adapters and HiperSockets was through the so-called bridgeport mode. The concept of the bridgeport mode distinguishes between between ports as follows:
- Primary bridgeport:
The primary bridgeport receives all traffic for all destination
addresses unknown to the device. Or, in other words: If the device
receives data for a destination unknown to it, instead of dropping it,
it will be forwarded to the current primary bridgeport instead. Which
further implies that as soon as an operating system registers a MAC
address with the device, traffic destined for that MAC address becomes
"invisible" to the bridgeport.
Note: Only a single operating system can use the primary bridgeport on an adapter at any time. - Secondary bridgeport:
Whenever the operating system that currently has the primary bridgeport
gives up on it, one of the secondary bridgeports will become the new
primary. An arbitrary number of operating systems can register as a
secondary bridgeport.
Bridgeport mode is available in Layer 2 mode only. Furthermore, HiperSockets devices need to be defined as external-bridged in IOCDS.
Use attributes in the device's sysfs directory as follows:
- bridge_role: Set the desired role (none, primary, or secondary), and query the current one.
- bridge_state: Query the current state (active or inactive)
Bridgeport
mode effectively provides a promiscuous mode. But note that in addition
to enabling the primary bridgeport mode, the respective interface has
to have the promiscuous mode set, still!
All in all, here is how usage of this feature typically looks like:
$ echo primary >/sys/devices/qeth/0.0.bd00/bridge_role
# verify that we got primary bridgeport, not secondary, and are active:
$ cat /sys/devices/qeth/0.0.bd00/bridge_state
active
$ cat /sys/devices/qeth/0.0.bd00/bridge_role
primary
# enable promiscuous mode on the interface
$ ip link set <interface> promisc on
The downside of this approach is that only a single operating system per device can enable the primary bridgeport mode, which scales only that far. Therefore, something better, with more functionality was introduced to the platform.
VNIC Characteristics
Introduced
with IBM z14/LinuxONE II for HiperSockets, and IBM z15/LinuxONE III for
OSA, the VNIC characteristics feature provides promiscuous mode for
multiple operating systems attached to the same device, and provides
additional functionality which can be very handy especially with KVM.
The VNIC characteristics can be controlled through a number of attributes located in an extra subdirectory called vnicc in the device's sysfs directory.
Let us focus on two main functionalities.
Promiscuous Mode
Technically, VNIC does not provide a traditional promiscuous mode (just like bridgeport mode did not in the literal sense), but rather emulates a self-learning switch. However, for users looking for a promiscuous mode that is usable in conjunction with a Linux bridge or an Open vSwitch, the end-result is the same.
To activate, set the attributes as follows:
echo 1>/sys/devices/qeth/0.0.bd00/vnicc/flooding
echo 1>/sys/devices/qeth/0.0.bd00/vnicc/mcast_flooding
echo 1>/sys/devices/qeth/0.0.bd00/vnicc/learning
Again, in addition to enabling the promiscuous mode on the device,
the respective interface has to have the promiscuous mode set, still:
ip link set <interface> promisc on
KVM Live Guest Migration
Providing connectivity to virtual servers running in KVM, administrators have two choices to provide connectivity:
- Via Open vSwitch:
Requires a promiscuous mode, see above. Virtual servers migrated
between the two Open vSwitches will have uninterrupted connectivity
thanks to the devices being configure in promiscuous mode, provided that
the networking architecture is set up accordingly. The two Open
vSwitches may or may not share the same networking device.
- Via MAC Address Takeover: This is only required in case both, the source and the target KVM host share the same device and use MacVTap to connect to it. While the traffic will still run through the same device, some handshaking has to take place to make sure that the MAC address is configured correctly, and traffic forwared to the target KVM host once migration has completed. This has to be authorized - otherwise, an attacker could divert traffic elsewhere.
Luckily, VNIC characteristics offers functionality for MAC address takeover, too. To enable, set the VNIC characteristics as follows:
On the source KVM host:
echo 1>/sys/devices/qeth/0.0.bd00/vnicc/takeover_learning
On the target KVM host:
echo 1>/sys/devices/qeth/0.0.bd00/vnicc/takeover_setvmac
Final Words
Note that bridgeport mode and VNIC characteristics are mutually exclusive! Meaning as soon as e.g. a single VNIC characteristics-related attribute is activated, bridgeport-related functionality is not available anymore until that VNIC-characteristic is disabled again.
Furthermore, check your Linux distribution's tools on how to persist the changes outlined above. On many distros, chzdev (comes with the s390-tools package) does the job, but not (yet) on all.
This
article only provides a brief overview. Both, promiscuous mode and the
VNIC characteristics have a lot more to it than what was covered in this
brief overview, which merely aims to provide just enough information to
get readers started with the most common usecases. For a deeper
understanding, check the respective sections in the Device Drivers, Features, and Commands book.
