virtio-blk scalability improvements

QEMU 9.0 and libvirt 10.0 introduced an important improvement for disk (virtio-blk) scalability. Until now, a virtio-blk device could use one iothread in the host (or the QEMU main thread).
For a while it was already possible to specify multiple virtio queues for a single virtio-block device, and the Linux guest driver was able to exploit thse concurrently:

<driver name='qemu' queues='3'>

All queues have been handled by one iothread. This introduced a limit for the maximum amount of IO requests per seconds per disk no matter how many queues have been defined.

Now it is possible to assign a queue to a given host iothread allowing for higher throughput per device:

<driver name='qemu' queues='3'>
  <iothreads>
    <iothread id='2'>
      <queue id='1'/>
    </iothread>
    <iothread id='3'>
      <queue id='0'/>
      <queue id='2'/>
    </iothread>
  </iothreads>
</driver>

Example with 3 queues to illustrate the possibility to have 2 queues on one iothread and one queue on another. In real life 2 or 4 queues make more sense.

Initial tests showed improved performance and reduced CPU cycles when going from 1 to 2 queues. More performance analysis needs to happen but this looks like a very promising improvement and going from 1 to 2 is almost a no-brainer. Adding more queues continues to improve the performance, but also increases the overall CPU consumption so this needs additional considerations.

Sharing iothreads across multiple disks continues to be possible. 

This feature is also being backported into some distributions like RHEL 9.4 or will be available via regular QEMU/libvirt upgrades.

SLES 15 SP1 Released

SLES 15 SP1 is out! See the announcement and their release notes with Z-specific changes.
It ships the following code level updates:

• QEMU v3.1 (GA: v2.11)
• libvirt v5.1 (GA: v4.0)

See previous blog entries on QEMU v2.12, v3.0 and v3.1 for details on new features that become available with the QEMU package update.
Furthermore, SLES 15 SP1 introduces the kvm_stat tool, which can be used for guest event analysis.

QEMU v4.0 released

QEMU v4.0 is out. Besides a number of small enhancements, some items that we would like to highlight from a KVM on Z perspective:

• CPU models for z14 GA2 as follows:
     $ qemu-system-s390x -cpu help -enable-kvm | grep z14.2
     s390 z14.2-base      IBM z14 GA2           (static, migration-safe)
     s390 z14.2           IBM z14 GA2           (migration-safe)
• vfio-ap now supports hotplugging of vfio-ap devices.

QEMU v3.1 released

QEMU v3.1 is out. Besides a number of small enhancements, some items that we would like to highlight from a KVM on Z perspective:

• Huge Pages Support: KVM guests can now utilize 1MB pages. As this removes one layer of address translation for the guest backing, less page-faults need to be processed, and less translation lookaside buffer (TLB) entries are needed to hold translations. This, as well as the TLB improvements in z14, will improve KVM guest performance.
  To use:
  Create config file /etc/modprobe.d/kvmhpage.conf file with the following content to enable huge pages for KVM:
  
     options kvm hpage=1
  
  Furthermore, add the following line to /etc/sysctl.conf to reserve N huge pages:
  
     vm.nr_hugepages = N
  
  Alternatively, append the following statement to the kernel parameter line in case support is compiled into the kernel: kvm.hpage=1 hugepages=N.
  Note that means to add hugepages dynamically after boot exist, but with effects like memory fragmentation, it is preferable to define huge pages as early as possible.
  If successful, the file /proc/sys/vm/nr_hugepages should show N huge pages. See here for further documentation.
  Then, to enable huge pages for a guest, add the following element to the respective domain XML:
  
     <memoryBacking>
       <hugepages/>
     </memoryBacking>
  
  The use of huge pages in the host is orthogonal to the use of huge pages in the guest. Both will improve the performance independently by reducing the number of page faults and the number of page table walks after a TLB miss.
  The biggest performance improvement can be achieved by using huge pages in both, host and guest, e.g. with libhugetlbfs, as this will also make use of the larger 1M TLB entries in the hardware.
  Requires Linux kernel 4.19.
• vfio-ap: The Adjunct Processor (AP) facility is an IBM Z cryptographic facility comprised of three AP instructions and up to 256 cryptographic adapter cards. Each adapter card is partitioned into up to 85 domains, each of which provides cryptographic services. An AP queue is the means by which AP messages are sent to and received from an AP adapter. Each AP queue is connected to a particular domain within a particular adapter. vfio-ap enables assignment of a subset of AP adapters and domains to one or more guests such that each guest has exclusive access to a discrete set of AP queues.
  Here is a small sample script illustrating host setup:
  
     # load vfio-ap device driver
     modprobe vfio-ap
  
     # reserve domain 7 for use by KVM guests
     echo -0x7 > /sys/bus/ap/aqmask
     # to reserve all domains of an adapter, use the following
     # line instead (by uncommenting it), and replace NN with the
     # adapter number:
     # echo -0xNN > /sys/bus/ap/apmask
  
     # create a mediated device (mdev) to provide userspace access
     # to a device in a secure manner   UUID=e926839d-a0b4-4f9c-95d0-c9b34190c4ba
     echo $UUID > /sys/devices/vfio_ap/matrix/mdev_supported_types/ \
                  vfio_ap-passthrough/create
  
     # assign adapter, domain and control domain
     echo 0x3 > /sys/devices/vfio_ap/matrix/${UUID}/assign_adapter
     echo 0x7 > /sys/devices/vfio_ap/matrix/${UUID}/assign_domain
     echo 0x7 > /sys/devices/vfio_ap/matrix/${UUID}/ \
                assign_control_domain
  
  To make use of the AP device in a KVM guest, add the following element to the respective domain XML:
  
     <hostdev mode='subsystem' type='mdev' managed='no' model='vfio-ap'>
       <source>
         <address uuid='e926839d-a0b4-4f9c-95d0-c9b34190c4ba'/>
       </source>
     </hostdev>
  
  Once complete, use the passthrough device in a KVM guest just like a regular crypto adapter.
  Requires Linux kernel 4.20 and libvirt 4.9, and is also available in RHEL 8, Ubuntu 18.04 and SLES 15 SP1.

QEMU v3.0 released

QEMU v3.0 is out. Besides a number of small enhancements, some items that we would like to highlight from a KVM on Z perspective:

• A new CPU model representing IBM z14 Model ZR1 was added:
  14ZR1 (long name: IBM z14 Model ZR1 GA1).
• Re-use your existing infrastructure for LPAR installs by utilizing the newly added support for .INS files in network boot.

Getting Started: RHEL 7.5 Instructions added

Instructions for RHEL7.5 were added to the Getting Started with KVM on Z series.
See here for the actual page.

Ubuntu 18.04 released

Ubuntu Server 18.04 LTS is out! Support for IBM Z is available here.
It ships

• Linux kernel 4.15,
• QEMU v2.11, and
• libvirt v4.0.

As obvious from these package versions, support for IBM z14 is readily in place.

Since this is a so-called LTS (Long Term Support) release providing approx. 5 years of support (in contrast to the usual 9 months of non-LTS releases), it is of particular interest to Ubuntu users interested in a stable environment for production deployments.

White Paper: Exploiting HiperSockets in a KVM Environment Using IP Routing with Linux on Z

Our performance group has published a new white paper titled "Exploiting HiperSockets in a KVM Environment Using IP Routing with Linux on Z".
Abstract:

"The IBM Z platforms provide the HiperSockets technology feature for high-speed communications. This paper documents how to set up and configure KVM virtual machines to use HiperSockets with IP routing capabilities of the TCP/IP stack.
It provides a Network Performance comparison between various network configurations and illustrates how HiperSockets can achieve greater performance for many workload types, across a wide range of data-flow patterns, compared with using an OSA 10GbE card."

This white paper is available as .pdf and .html.