SLES 15 SP1 Released

SLES 15 SP1 is out! See the announcement and their release notes with Z-specific changes.
It ships the following code level updates:

• QEMU v3.1 (GA: v2.11)
• libvirt v5.1 (GA: v4.0)

See previous blog entries on QEMU v2.12, v3.0 and v3.1 for details on new features that become available with the QEMU package update.
Furthermore, SLES 15 SP1 introduces the kvm_stat tool, which can be used for guest event analysis.

QEMU v4.0 released

QEMU v4.0 is out. Besides a number of small enhancements, some items that we would like to highlight from a KVM on Z perspective:

• CPU models for z14 GA2 as follows:
     $ qemu-system-s390x -cpu help -enable-kvm | grep z14.2
     s390 z14.2-base      IBM z14 GA2           (static, migration-safe)
     s390 z14.2           IBM z14 GA2           (migration-safe)
• vfio-ap now supports hotplugging of vfio-ap devices.

QEMU v3.1 released

QEMU v3.1 is out. Besides a number of small enhancements, some items that we would like to highlight from a KVM on Z perspective:

• Huge Pages Support: KVM guests can now utilize 1MB pages. As this removes one layer of address translation for the guest backing, less page-faults need to be processed, and less translation lookaside buffer (TLB) entries are needed to hold translations. This, as well as the TLB improvements in z14, will improve KVM guest performance.
  To use:
  Create config file /etc/modprobe.d/kvmhpage.conf file with the following content to enable huge pages for KVM:
  
     options kvm hpage=1
  
  Furthermore, add the following line to /etc/sysctl.conf to reserve N huge pages:
  
     vm.nr_hugepages = N
  
  Alternatively, append the following statement to the kernel parameter line in case support is compiled into the kernel: kvm.hpage=1 hugepages=N.
  Note that means to add hugepages dynamically after boot exist, but with effects like memory fragmentation, it is preferable to define huge pages as early as possible.
  If successful, the file /proc/sys/vm/nr_hugepages should show N huge pages. See here for further documentation.
  Then, to enable huge pages for a guest, add the following element to the respective domain XML:
  
     <memoryBacking>
       <hugepages/>
     </memoryBacking>
  
  The use of huge pages in the host is orthogonal to the use of huge pages in the guest. Both will improve the performance independently by reducing the number of page faults and the number of page table walks after a TLB miss.
  The biggest performance improvement can be achieved by using huge pages in both, host and guest, e.g. with libhugetlbfs, as this will also make use of the larger 1M TLB entries in the hardware.
  Requires Linux kernel 4.19.
• vfio-ap: The Adjunct Processor (AP) facility is an IBM Z cryptographic facility comprised of three AP instructions and up to 256 cryptographic adapter cards. Each adapter card is partitioned into up to 85 domains, each of which provides cryptographic services. An AP queue is the means by which AP messages are sent to and received from an AP adapter. Each AP queue is connected to a particular domain within a particular adapter. vfio-ap enables assignment of a subset of AP adapters and domains to one or more guests such that each guest has exclusive access to a discrete set of AP queues.
  Here is a small sample script illustrating host setup:
  
     # load vfio-ap device driver
     modprobe vfio-ap
  
     # reserve domain 7 for use by KVM guests
     echo -0x7 > /sys/bus/ap/aqmask
     # to reserve all domains of an adapter, use the following
     # line instead (by uncommenting it), and replace NN with the
     # adapter number:
     # echo -0xNN > /sys/bus/ap/apmask
  
     # create a mediated device (mdev) to provide userspace access
     # to a device in a secure manner   UUID=e926839d-a0b4-4f9c-95d0-c9b34190c4ba
     echo $UUID > /sys/devices/vfio_ap/matrix/mdev_supported_types/ \
                  vfio_ap-passthrough/create
  
     # assign adapter, domain and control domain
     echo 0x3 > /sys/devices/vfio_ap/matrix/${UUID}/assign_adapter
     echo 0x7 > /sys/devices/vfio_ap/matrix/${UUID}/assign_domain
     echo 0x7 > /sys/devices/vfio_ap/matrix/${UUID}/ \
                assign_control_domain
  
  To make use of the AP device in a KVM guest, add the following element to the respective domain XML:
  
     <hostdev mode='subsystem' type='mdev' managed='no' model='vfio-ap'>
       <source>
         <address uuid='e926839d-a0b4-4f9c-95d0-c9b34190c4ba'/>
       </source>
     </hostdev>
  
  Once complete, use the passthrough device in a KVM guest just like a regular crypto adapter.
  Requires Linux kernel 4.20 and libvirt 4.9, and is also available in RHEL 8, Ubuntu 18.04 and SLES 15 SP1.

QEMU v3.0 released

QEMU v3.0 is out. Besides a number of small enhancements, some items that we would like to highlight from a KVM on Z perspective:

• A new CPU model representing IBM z14 Model ZR1 was added:
  14ZR1 (long name: IBM z14 Model ZR1 GA1).
• Re-use your existing infrastructure for LPAR installs by utilizing the newly added support for .INS files in network boot.

Getting Started: RHEL 7.5 Instructions added

Instructions for RHEL7.5 were added to the Getting Started with KVM on Z series.
See here for the actual page.

Ubuntu 18.04 released

Ubuntu Server 18.04 LTS is out! Support for IBM Z is available here.
It ships

• Linux kernel 4.15,
• QEMU v2.11, and
• libvirt v4.0.

As obvious from these package versions, support for IBM z14 is readily in place.

Since this is a so-called LTS (Long Term Support) release providing approx. 5 years of support (in contrast to the usual 9 months of non-LTS releases), it is of particular interest to Ubuntu users interested in a stable environment for production deployments.

QEMU v2.12 released

QEMU v2.12 is out. Here are the highlights from a KVM on Z perspective:

• Added support for an interactive bootloader. As always, we strongly recommend to use the existing support in libvirt.
  To enable/disable, add the following element to your guest definition:
  
     <os>
       <bootmenu enable=’yes|no’ timeout=’n’/>
       ...
     </os>
  
  The timeout parameter specifies a timeout in milliseconds after which the default entry is chosen.
  Alternatively, set attribute loadparm to PROMPT to enable the boot menu without timeout in the respective disk's element:
  
     <disk ...>
       <boot order=’1’ loadparm=’PROMPT’/>
       ...
     </disk>
  
  Example:
  To enable the boot menu for 32 seconds for a guest using a libvirt
  domain XML format follows:
  
     <domain type=’kvm’>
       <os>
         <bootmenu enable=’yes’ timeout=’32000’/>
         ...
       </os>
• Exposure of guest crash information: When a guest is started using libvirt and crashes due to disabled wait, wrong interrupts or a program check loop, libvirt will print the information to the guest’s log, typically located at /var/log/libvirt/qemu.
  E.g. a crash due to a disabled wait results in an entry as follows:
  
     s390: psw-mask=’0xXXXXXXXXXXXXXXXX’, psw-addr=’0xXXXXXXXXXX
           XXXXXX’,crash reason: disabled wait
  
  Requires  libvirt v4.2.
• Added support for guests with more than 8TB of memory.

White Paper: Exploiting HiperSockets in a KVM Environment Using IP Routing with Linux on Z

Our performance group has published a new white paper titled "Exploiting HiperSockets in a KVM Environment Using IP Routing with Linux on Z".
Abstract:

"The IBM Z platforms provide the HiperSockets technology feature for high-speed communications. This paper documents how to set up and configure KVM virtual machines to use HiperSockets with IP routing capabilities of the TCP/IP stack.
It provides a Network Performance comparison between various network configurations and illustrates how HiperSockets can achieve greater performance for many workload types, across a wide range of data-flow patterns, compared with using an OSA 10GbE card."

This white paper is available as .pdf and .html.

RHEL 7.5 with support for KVM on Z available

Red Hat Enterprise Linux 7.5 is out. From the release notes, available here:

Availability across multiple architectures
To further support customer choice in computing architecture, Red Hat Enterprise Linux 7.5 is simultaneously available across all supported architectures, including x86, IBM Power, IBM z Systems, and 64-bit Arm.

Support for IBM Z is available through the kernel-alt package, as indicated earlier here, which provides Linux kernel 4.14. QEMU ships v2.10 via package qemu-kvm-ma, and libvirt is updated to v3.9.0 for all platforms.
Thereby, all IBM z14 features as previously listed here are available.
Check these instructions on how to get started. 

SLES 12 SP3 Updates

SLES 12 SP3, released late last year, received a couple of mostly performance and security-related updates in support of IBM z14 and LinuxONE through the maintenance web updates.
In particular:

• Instruction Execution Protection
• SIMD Extensions
• STHYI Support
• Transparent activation of new hardware features: This will enable a set of performance-oriented hardware facilities of z14 that do not require hypervisor changes.
  This includes the following facilities:
• miscellaneous-instruction-extensions facility 2
• entropy-encoding compression facility
• order-preserving-compression facility

RHEL 7.5 Beta supports KVM on Z

The Red Hat Enterprise Linux 7.5 Beta ships with support for KVM on Z through the kernel-alt packages. This will essentially ship Linux kernel 4.14.
Here is the respective section from the release notes:

KVM virtualization is now supported on IBM z Systems. However, this feature is only available in the newly introduced user space based on kernel version 4.14, provided by the kernel-alt packages.

See here for further details.

QEMU v2.11 released

QEMU v2.11 is out. Here are the highlights from a KVM on Z perspective:

• TOD-Clock Epoch Extension Support: Extends the TOD clock beyond the year 2042.
• Setting sysctl vm.allocate_pgste is now superfluous.
• Netboot: The network boot firmware sets the client architecture option (93) in the DHCP request to 0x1f ("s390 Basic"). This allows a DHCP server to deliver the correct boot image for IBM Z guests. This is useful in situations where a single DHCP server has to provide network boot images for multiple architectures, e.g. for the purpose of installing operating systems.
• Added support for virtio-input-ccw and virtio-gpu-ccw. These newly supported devices lay the foundation for applications that require graphical interfaces, which thereby become usable from remote via VNC or SPICE.
  Here is a sample XML snippet for a guest definition:
  
      <input type='keyboard' bus='virtio'/>
      <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0'>
        <listen type='address' address='0.0.0.0'/>
      </graphics>
      <video>
        <model type='virtio' heads='1' primary='yes'/>
      </video>

DataCenter Insider Article

See here for an article (in German) in DataCenter Insider about KVM on z's history, usage concepts, support and z14 features.

QEMU v2.10 released

QEMU v2.10 is out. See here for a detailed list of all changes related to IBM Z. Here are some of the highlights from a KVM on z perspective:

• z14 CPU model
  Provides base support for all of the features introduced with the z14's new processor generation.
  See this previous blog post for details.
• Emulated 3270 Device
  This implementation has been tested with x3270 only. Your mileage with other clients might vary. For practical matters, the sclp console remains to be the recommended terminal type for KVM guests on Z.
  As this feature is still marked as experimental in QEMU, there is currently no libvirt support available. Therefore to configure a 3270 device on a guest (accessible on all IP addresses of the host at port 23), we must use libvirt’s ability to pass-through arbitrary QEMU command line arguments via domain XML (see here for furher details):

  <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>   …   <qemu:commandline>     <qemu:arg value="-chardev"/>     <qemu:arg value="socket,id=charconsole1,host=0.0.0.0,port=23,\                        nowait,server,tn3270"/>     <qemu:arg value="-device"/>     <qemu:arg value="x-terminal3270,chardev=charconsole1,\                       devno=fe.0.000a,id=terminal1"/>   </qemu:commandline>   … </domain>

  See here for further hints on how to use.
• Bootparameter Support
  Until now, when a guest had multiple kernels to boot with, one had to change the default kernel in /etc/zipl.conf and re-run zipl to boot with a different kernel next time.
  With this feature, we now support option -L in the chreipl command from the s390-tools package, which allows to switch to a different boot option dynamically, e.g. using

$ chreipl ccw -d 0.0.1234 -L 2

However, if somebody has set a new default kernel that does not boot, this will not help, since the guest will not start anymore in the first place.
However, with the newly introduced attribute loadparm, one can switch to a different (working) kernel within the host in a guest's XML:

<disk> ... <boot order='1' loadparm='2'> </disk>

Notes:
• chreipl changes will no longer be effective after the respective guest has been powered off.
• The loadparm attribute became available with libvirt v3.5 or higher. 

KVM on z14 features

While the latest addition to the IBM Z family has been announced, here is a list of features in support of specific features of the new hardware generation in past releases of the Linux kernel, QEMU and libvirt, all activated by default in the z14 CPU model:

• Instruction Execution Protection
  This feature provides KVM hypervisor support for the Instruction Execution Protection (IEP) facility in the z14. The IEP prevents code execution from memory regions marked as non-executable, improving the security model.
  Other than activating/deactivating this feature in the applicable the CPU models in QEMU (which holds true for most hardware-related features on IBM Z in general), there are no switches associated with this feature.
  Requires Linux kernel 4.11 in the KVM host and guests, as well as QEMU v2.10 (host only).
  In the z14 CPU model, the respective feature is:
    iep       Instruction-execution-protection facility
• SIMD Extensions
  Following up to the SIMD instructions as introduced with the previous z13 model, the new z14 provides further vector instructions, which can again be used in KVM guests.
  These new vector instructions can be used to improve decimal calculations as well as for implementing high performance variants of certain cryptographic operations.
  Requires Linux kernel 4.11 as well as QEMU v2.10 in the KVM host, and binaries or a respective Java Runtime Environment in guests using the new vector instructions.
  In the z14 CPU model, the respective feature is:
    vxpd      Vector packed decimal facility
    vxeh      Vector enhancements facility
• Keyless Guest Support
  This feature supports the so-called Keyless Subset (KSS) facility, a new feature of the z14 hardware. With the KSS facility enabled, a host is not required to perform the (costly) storage key initialization and management for KVM guests, unless a guest issues a storage key instruction.
  Requires Linux kernel 4.12 in the KVM host. As for the guests, note that starting with SLES12SP1, RHEL7.2 and Ubuntu 16.04, Linux on IBM Z does not issue any storage key operations anymore.
  This feature does not have a separate entry in the z14 CPU model.
  
• CPUMF Basic Sample Configuration Level Indication
  Basic mode samples as defined in "The Load-Program-Parameter and the CPU-Measurement Facilities" (SA23-2260) do not provide an indication whether the sample was taken in a KVM host or guest. Beginning with z14, the hardware provides an indication of the configuration level (level of SIE, e.g. LPAR or KVM). This item exploits this information to make the perf guest/host decision reliable.
  Requires Linux kernel 4.12 in the KVM host.
  There is no separate entry in the z14 CPU model, since this feature applies to the host only.
• Semaphore assist
  Improves performance of semaphore locks.
  Requires Linux kernel 4.7 and QEMU v2.10 in the KVM host. Exploitation in Linux kernels in guests is still in progress here, scheduled for 4.14.
  In the z14 CPU model, the respective feature is:
    sema      Semaphore-assist facility
• Guarded storage
  This feature is specifically aimed at Java Virtual Machines running in KVM guests to run with fewer and shorter pauses for garbage collection.
  Requires Linux kernel 4.12 and QEMU 2.10 in the KVM host, and a Java Runtime Environment with respective support in the guests.
  In the z14 CPU model, the respective feature is:
    gs        Guarded-storage facility
• MSA Updates
  z14 introduces 3 new Message Security Assists (MSA) for the following functionalities:
      MSA6: SHA3 hashing
      MSA7: A True Random  Number Generator (TRNG)
      MSA8: The CIPHER MESSAGE WITH AUTHENTICATION instruction,
                  which provides support for the Galois-counter-mode (GCM)
  MSA6 and MSA 7 require Linux kernel 4.7, while MSA8 requires Linux kernel 4.12. All require QEMU v2.10 in the KVM host. These features can be exploited in KVM guests' kernels and userspace applications independently (i.e. a KVM guest's userspace applications can take advantage of these features irrespective of the guest's kernel version).
  In the z14 CPU model, the respective features are:
    msa6      Message-security-assist-extension 6 facility
    msa7      Message-security-assist-extension 7 facility
    msa8      Message-security-assist-extension 8 facility
• Compression enhancements
  New instructions improve compression capabilities and performance.
  Requires Linux kernel 4.7 in the KVM host.
  In the z14 CPU model, the respective features are:
    opc       Order Preserving Compression facility
    eec       Entropy encoding compression facility
• Miscellaneous instructions
  Details on these instructions are to be published in the forthcoming z14 Principles of Operation (PoP).
  Requires Linux kernel 4.7 and QEMU 2.10 in the KVM host, and binaries that were compiled for the z14 instruction set using binutils v2.28 and gcc v7.1 in the guests.
  In the z14 CPU model, the respective feature is:
    minste2   Miscellaneous-instruction-extensions facility 2

Note: All versions specified are minimum versions.

Further features will be announced in future blog posts as usual as they find their way into the respective Open Source projects.
Also, don't forget to check this blog entry with further details on z14 in general and Linux on z in particular.

z14 announced

Today, IBM announced the new IBM Z model named z14. See here for the full press release.

We will look at features in support of the new IBM Z model in a separate blog entry soon.

Migrating from KVM for IBM z to Ubuntu

In case you are contemplating a migration from KVM for IBM z to Ubuntu 16.04 or later, you might find the instructions published on the Ubuntu Wiki here helpful.

Furthermore, Canonical published a nice article here that provides an overview of the many options how to deploy Ubuntu on IBM z Systems, including KVM.

libvirt v.3.5.0 released

libvirt v3.5.0 is now available for download at the libvirt project website.
A new z Systems-specific feature is the introduction of parameter loadparm for boot devices.
The loadparm parameter can be used to select a specific boot configuration from the zipl menu when IPL’ing/booting from a boot device, similar to what the chreipl command does in Linux on z.
Being able to select a configuration from multiple boot configurations provides more flexibility and allows to recover from error situations by booting a "last known good configuration".
To use, specify the loadparm attribute for a boot device in a guest's XML as follows:

   <disk>
      <boot order='1' loadparm='3'/>
      [...]
   </disk>

Note that this feature requires the forthcoming QEMU 2.10 release.

QEMU v2.9 released

QEMU v2.9 is out. Here are the highlights from a KVM on z perspective:

• Added support for virtio-crypto, providing a virtual crypto device as a front-end to the host's crypto capabilities on z Systems.
• Added multiqueue support for virtio-ccw devices, improving throughput in presence of multiple processors.
• Provided network boot enablement. See here for further details.

Linux kernel 4.11 released

Linux kernel 4.11 (available here) has been released, improving Kernel Samepage Merging (KSM) support for KVM on z:

KSM is an existing feature of the Linux kernel that allows the kernel to merge pages with identical content, indicated as MADV_MERGEABLE via madvise(), across different processes to save memory.
This z Systems specific extension improves the performance of userspace applications (e.g. QEMU) in case large amounts of empty/zeroed pages are merged.
This feature is disabled by default, enable as follows:

   $ echo 1 > /sys/kernel/mm/ksm/use_zero_pages