KVM PCI device passthru with high performance and low overhead

 KVM supports passing through PCI devices to its guests. A typical libvirt xml statement looks like

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0012' bus='0x00' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'>
        <zpci uid='0x0001' fid='0x00000000'/>
      </address>
    </hostdev>

See KVM Virtual Server Management SC34-2752 for details.

With newer kernel and QEMU versions (e.g. those found in RHEL 10.2, SLES 16.1 and Ubuntu 25.10) it is now possible to improve performance and reduce overhead when doing so. 

You have to specify the parameter iommu.passthrough=1 on the guest kernel command line via zipl or /boot/loader/entries.

This will pin the guest in memory, but gets rid of the expensive RPCIT instructions for IO that are used by the guest to notify the host about memory used for IO to control host paging.

As of today KVM already pins memory in use by the guest for PCI I/O. With todays lazy free of dma memory large parts of guest memory will be pinned over time. So pinning the whole guest during startup only has a small downside in real life.

By checking in the guest

   cat /sys/bus/pci/devices/<device number>/iommu_group/type 

it is possible to check if everything worked. The file should say "identity".

Updated Publication: "Networking with PCI adapters and functions"

Coinciding with the switch to the new Network Express adapters, we have updated our documentation to include the new card here.

 From the introduction:

"This publication explores what PCI network adapters offer for network connections of Linux® on IBM Z® and IBM® LinuxONE. The publication applies to Network Express and RoCE Express2 or RoCE Express3 adapters."

Note that Network Express adapters now also support promiscuous mode as required by Open vSwitch in a KVM context! 

New Release: Ubuntu 25.04

Canonical released a new version of their Ubuntu server offering Ubuntu Server 25.04!

Highlights include IBM z17 support, in particular:

• Exploitation of the AI facility in the Telum II processor
• New Secure Execution and Crypto features
• Network Express support​
• Power consumption reporting​
  

See the announcement on the mailing list here, and the blog entry at Canonical with all Z-specific highlights here.

IBM z17 announced!!

Our latest offering in the IBM Z family, IBM z17, was announced yesterday. General availability will be June 18.

See the official announcement, the updated Linux support matrix, and the Technical Guide with all the insights you will need.

Migration fails with something like ..pckmo.. ?

 If you run into a situation where migration fails with something like

internal error: QEMU unexpectedly closed the monitor (vm='testguest'):
qemu-kvm: Some features requested in the CPU model are not available in the current configuration: pckmo-aes-256 pckmo-aes-192 pckmo-aes-128 pckmo-etdea-192 pckmo-etdea-128 pckmo-edea msa9_pckmo Consider a different accelerator, QEMU, or kernel version

This indicates that both host are configured differently regarding the CPACF key management Operations

So you can (preferred if the security scheme allows for that) configure both LPARs the same way, de-activate and re-activate the LPAR

or

you can change the CPU model of the guest to no longer have these pckmo functions. Change your guest xml from "host-model" to "host-model with some features disabled".

So shutdown the guest and change the XML from

  <cpu mode='host-model' check='partial'/>

to

  <cpu mode='host-model' check='partial'>
    <feature policy='disable' name='msa3'/>

    <feature policy='require' name='msa3-base'/>
    <feature policy='disable' name='msa9_pckmo'/>
  </cpu>

This guest can now be migrated to a host without key managment functions. The downside is that this guest no longer has access to the key management functions. This makes it harder to use encrypted swap devices with automatic key generation. Therefore, enabling these functions on both LPARs is preferred.

New Feature: Installation Assistant for Linux on IBM Z

Ever struggled to create configuration files for starting Linux on IBM Z and LinuxONE installations? Fear no more, we got you covered now: A new assistant available online will help you create parameter files!
Writing parameter files can be a challenge, with bugs triggering cycles with lengthy turnaround times. Our new installation assistant generates installer parameter files by walking you through a step-by-step process, where you answer simple questions to generate a parameter file. Comes with contextual help in every stage, so you can follow along what is happening!
Currently supports OSA and PCI networking devices, IPv4/v6, and VLAN installations.

Currently supports RHEL 9 and SLES 15 SP5 or later.
Access the assistant at https://ibm.github.io/liz/

New Video: Configuring Crypto Express Adapters for KVM Guests

A new video illustrating the steps to perform on a KVM host and in a virtual server configuration to make AP queues of cryptographic adapters available to a KVM guest can be found here.

virtio-blk scalability improvements

QEMU 9.0 and libvirt 10.0 introduced an important improvement for disk (virtio-blk) scalability. Until now, a virtio-blk device could use one iothread in the host (or the QEMU main thread).
For a while it was already possible to specify multiple virtio queues for a single virtio-block device, and the Linux guest driver was able to exploit thse concurrently:

<driver name='qemu' queues='3'>

All queues have been handled by one iothread. This introduced a limit for the maximum amount of IO requests per seconds per disk no matter how many queues have been defined.

Now it is possible to assign a queue to a given host iothread allowing for higher throughput per device:

<driver name='qemu' queues='3'>
  <iothreads>
    <iothread id='2'>
      <queue id='1'/>
    </iothread>
    <iothread id='3'>
      <queue id='0'/>
      <queue id='2'/>
    </iothread>
  </iothreads>
</driver>

Example with 3 queues to illustrate the possibility to have 2 queues on one iothread and one queue on another. In real life 2 or 4 queues make more sense.

Initial tests showed improved performance and reduced CPU cycles when going from 1 to 2 queues. More performance analysis needs to happen but this looks like a very promising improvement and going from 1 to 2 is almost a no-brainer. Adding more queues continues to improve the performance, but also increases the overall CPU consumption so this needs additional considerations.

Sharing iothreads across multiple disks continues to be possible. 

This feature is also being backported into some distributions like RHEL 9.4 or will be available via regular QEMU/libvirt upgrades.

IBM Secure Execution for Linux support for Crypto Express adapters

IBM Secure Execution for Linux -- the Linux Kernel Virtual Machine (KVM) based Confidential Computing technology for IBM LinuxONE and Linux on IBM Z -- now allows Secure Execution guests leverage secure passthrough access to up to 12 Crypto Express 8S adapter domains in accelerator or EP11 co-processor mode.

Customers who require the highest level of protection (FIPS 140-2 level 4 certified) for their cryptographic keys and thus for their sensitive data can now have their workloads deployed as Secure Execution KVM guests with access to Hardware Security Modules (HSMs) if the provider uses IBM z16 or LinuxONE 4 servers with Crypto Express 8S adapters. This combination provides business value for solutions around key and certificate management, multi-party computation and digital assets. But more use cases arise as confidential computing becomes more common and the need to leverage such highly certified HSM to protect AI models or provide data sovereignty across organizational and infrastructure boundaries grows.

To exploit this new function, IBM z16 or LinuxONE 4 severs with firmware bundles S30 and S31b are needed. To use a Crypto Express 8S adapter in EP11 mode the minimal EP11 firmware version loaded must be version 5.8.30.

IBM is working with Linux distribution partners to include the required Linux support for this function for both the KVM Hypervisor and the Secure Execution guests in future distribution releases. Linux support for this function is already available today with Ubuntu 24.04 (Noble Numbat).

This new capability showcases IBM’s commitment and previously stated direction to foster the use of confidential computing and expand the security value proposition of existing security and crypto solutions as the business needs of our customers and technical possibilities evolve.

For detailed information on how to use Crypto Express support see the Introducing IBM Secure Execution for Linux publication.

 

Authored by

• Reinhard Bündgen – buendgen@de.ibm.com
• Nicolas Mäding – nmaeding@de.ibm.com

New Release: Ubuntu 24.04

Canonical released a new version of their Ubuntu server offering Ubuntu Server 24.04!

 Highlights include

• HSM support for Secure Execution
• Further Crypto enhancements and extensions

See the announcement on the mailing list here, and the blog entry at Canonical with all Z-specific highlights here.

This release is very significant, since it marks a so-called LTS (Long Term Support) release, granting an extended service timeframe of up to 10 years, as illustrated here.

Important Note on Verifying Secure Execution Host Key Documents

The certificates of the host key signing keys that are needed to verify host key documents will expire on April 24, 2024 for IBM z15 and LinuxONE III and on March 29, 2024 for IBM z16 and LinuxONE 4. Due to a requirement from the Certificate Authority (DigiCert), the renewed certificates are equipped with a new Locality value (“Armonk” instead of “Poughkeepsie”). These renewed certificates cause the current versions of the genprotimg, pvattest, and pvsecret tools to fail the verification of host key documents.

The IBM Z team is preparing updates of the genprotimg, pvattest, and pvsecret tools to accept the new certificates and is working with Linux distribution partners to release the updated tools.

To build new Secure Execution images, attestation requests, or add-secret requests before the updated tools are available in Linux distributions, follow these steps:

Step 1:

Obtain the host key document, the host key signing key certificate, the intermediate certificate from the Certificate Authority, and the list of revoked host keys (CRL):

• For IBM z15 and LinuxONE III, see https://www.ibm.com/support/resourcelink/api/content/public/secure-execution-gen1.html

• For IBM z16 and LinuxONE 4, see https://www.ibm.com/support/resourcelink/api/content/public/secure-execution-gen2.html
  
  

Step 2: 

Download the script check_hostkeydoc from https://github.com/ibm-s390-linux/s390-tools/blob/master/genprotimg/samples/check_hostkeydoc.

Step 3:

Verify each host key document using the check_hostkeydoc script. For example, issue

# ./check_hostkeydoc HKD1234.crt ibm-z-host-key-signing.crt \ -c DigiCertCA.crt -r ibm-z-host-key.crl

This example verifies the host key document HKD1234.crt using the host key signing key certificate ibm-z-host-key-signing.crt, and the intermediate certificate of the Certificate Authority DigiCertCA.crt,as well as the list of revoked host keys ibm-z-host-key.crl.

After the host key documents are verified using the check_hostkeydoc script, you can safely call genprotimg, pvattest, or pvsecret with the –-no-verify option.

For a description about how to manually verify host key documents, see https://www.ibm.com/docs/en/linux-on-z?topic=execution-verify-host-key-document.

New Linux on IBM Z & LinuxONE Forum at Open Mainframe Project

The Open Mainframe Project has launched a new forum dedicated to Linux on Z. It can be found here, and is intended to complement existing facilities like the mailing lists hosted at Maris college. Any topic around Linux on Z, including virtualization as provided by z/VM and KVM, is fair game, and you may use it to ask quesitions, share useful hints and tips, or simply have a casual conversation about some aspect of the platform!

Red Hat Ansible Automation Platform available on IBM Z and LinuxONE

While Linux on IBM Z and LinuxONE could be used as a target for Ansible scripts ever since, the backend had to be run on other architectures. But no longer so: Starting today, the entire Red Hat Ansible Automation Platform is becoming available on IBM Z and LinuxONE!

See here for more details, and here for the formal announcement from Red Hat.

New Releases: RHEL 8.9 and RHEL 9.3 on IBM Z & LinuxONE

Both, Red Hat Enterprise Linux 8.9 and 9.1 are out! See the press release here, and Red Hat's blog entry here!

Both releases ship

• s390-tools v2.27 (renamed to s390utils)
• smc-tools v1.8.2
• openCryptoki v3.21

Further information can be found in the release notes for RHEL 8.9 and see RHEL 9.3.

New Release: Ubuntu 23.10

Canonical released a new version of their Ubuntu server offering Ubuntu Server 23.10!

See the announcement on the mailing list here, and the blog entry at Canonical with Z-specific highlights here.

2023 IBM TechXchange EMEA Client Workshop for Linux on IBM Z and LinuxONE

Interested in the latest news on Linux on IBM Z & LinuxONE? Come and meet us at the 2023 IBM TechXchange EMEA Client Workshop for Linux on IBM Z and LinuxONE on September 19-20 in Ehningen, Germany!

Register here.

KVM in Linux Distributions in 2Q 2023

Second quarter of 2023 was quite productive in terms of new Linux distribution releases and KVM related features shipped in there. Here they are, in chronological order.

Ubuntu 23.04 

The most recent Ubuntu release contains the following new KVM-related functionality:

• Interpretive vfio-pci support for ISM: Allows pass-through of ISM devices to KVM guests, enabling high-bandwith and low-latency network communications using SMC-D.
• Encrypted dump for Secure Execution: Enhances problem determination capabilities while not compromising the security of secure KVM guests.
• Bus id for subchannels: Allows you to identify passthrough CCW devices by their device bus id in the host without going through hoops.
• Driverctl now lists persisted overrides: Makes it easier to identify and manage passthrough devices.

RHEL 8.8 and 9.2

While being distinct releases, both share the same set of new functionality, which is:

• Interpretive vfio-pci support for ISM: Which allows pass-through of ISM devices to KVM guests, enabling high-bandwith and low-latency network communication using SMC-D.
• Encrypted dump for Secure Execution: Enhances problem determination capabilities while not compromising the security of secure KVM guests.
• Bus id for subchannels: Allows you to identify passthrough CCW devices by their device bus id in the host without going through hoops.
• Dynamic configuration updates  for vfio-ap: Allows you to hot plug and unplug Crypto domains of a Crypto passthrough configuration for running KVM guests.

SLES 15 SP5

Being a large service pack, there are numerous new features delivered:

• Interpretive vfio-pci support for ISM: Which allows pass-through of ISM devices to KVM guests, enabling high-bandwith and low-latency network communication using SMC-D.
• Encrypted dump for Secure Execution: Enhances problem determination capabilities while not compromising the security of secure KVM guests.
• Bus id for subchannels: Allows you to identify passthrough CCW devices by their device bus id in the host without going through hoops.
• Driverctl now lists persisted overrides: Makes it easier to identify and manage passthrough devices.
• Persistent configuration for vfio-ap: The s390-tools command zdev can now be used to persist Crypto passthrough configurations.
• Dynamic configuration updates  for vfio-ap: Allows you to hot plug and unplug Crypto domains of a Crypto passthrough configuration for running KVM guests.

• Remote attestation for Secure Execution: Provides cryptographic evidence of workload authenticity and integrity facilitates integration into common Confidential Computing frameworks.
• Support of long kernel command lines of up to 64 KB length, for example, allows you to specify plenty of I/O devices.

...and other minor improvements

WARNING: Updating from RHEL8.6 to RHEL8.7/RHEL8.8 may break Network Access with RoCE Express Adapters

RoCE interfaces may lose their IP settings due to an unexpected change of the network interface name.

The RoCE Express adapters can lose their IP settings due to an unexpected change of the network interface name if both of the following conditions are met:

• User upgrades from a RHEL 8.6 system or earlier.
• The RoCE card is enumerated by UID.

To workaround this problem:

Create the file /etc/systemd/network/98-rhel87-s390x.link with the following content:

[Match]
Architecture=s390x
KernelCommandLine=!net.naming-scheme=rhel-8.7

[Link]
NamePolicy=kernel database slot path
AlternativeNamesPolicy=database slot path
MACAddressPolicy=persistent

After rebooting the system for the changes to take effect, you can safely upgrade to RHEL 8.7 or later.

Note:
RoCE interfaces that are enumerated by function ID (FID, indicated by featuring the prefix ens in their interface names) are not unique and not affected by this issue. Set the kernel parameter net.naming-scheme=rhel-8.7. to switch to predictable interface names with the eno prefix. See the Networking with RoCE Express book for further details.

2023 Client Workshop for Linux on IBM Z and LinuxONE North America

Join us at the 2023 Client Workshop for Linux on IBM Z and LinuxONE for North America in-person at the IBM Innovation Studio in Poughkeepsie, NY!

 

Go here to register.

Dates and Times

• Dates: Wednesday, May 24th and Thursday, May 25th
• Times: 9:30 AM until 5:30 PM local time on both days

Address

• IBM Poughkeepsie
• 705 Development Court
• Poughkeepsie, NY

General Information 

On May 24th and May 25th, you have the possibility to get the latest news and technical information for Hybrid Cloud with Linux on IBM zSystems, LinuxONE, KVM, and z/VM. The training will be delivered onsite at the IBM Innovation Studio Poughkeepsie. You have the chance to interact directly with our IBM experts and join different small work group sessions on these two days.  

Enrollment and Costs/Fees 

The class is free of charge. Travel & Living is on your own. 

Target Audience

Linux on IBM zSystems and LinuxONE Clients, Partners and Software Vendors, and IBM zSystems Technical Sales

Topic Highlights

• Hybrid Cloud on IBM zSystems with Linux and z/OS
• Getting the most out of the Latest Features in Linux on IBM Z and LinuxONE
• The Gravity of Data - Co-locating z/OS and Linux
• IBM LinuxONE Security and Compliance Center Overview & Demo
• Common z/VM Hurdles and How to Overcome Them
• Update on z/VM Express System Installation
• z/VM Security Session
• IBM Z next middleware HW overview
• Quantum Deep Dive
• Red Hat OpenShift, Storage and Solutioning Values
• Turbonomic resource management with OpenShift on IBM zSystems
• Running Databases on LinuxONE
• Confidential computing - Confidential Containers
• Cloud Native DevSecOps using OpenShift Pipelines on IBM LinuxONE
• Various Workgroup Sessions

New Release: Ubuntu 23.04

Canonical released a new version of their Ubuntu server offering Ubuntu Server 23.04!

See the announcement on the mailing list here, and the blog entry at Canonical with Z-specific highlights here.