KVM on z14 features

While the latest addition to the IBM Z family has been announced, here is a list of features in support of specific features of the new hardware generation in past releases of the Linux kernel, QEMU and libvirt, all activated by default in the z14 CPU model:

• Instruction Execution Protection
  This feature provides KVM hypervisor support for the Instruction Execution Protection (IEP) facility in the z14. The IEP prevents code execution from memory regions marked as non-executable, improving the security model.
  Other than activating/deactivating this feature in the applicable the CPU models in QEMU (which holds true for most hardware-related features on IBM Z in general), there are no switches associated with this feature.
  Requires Linux kernel 4.11 in the KVM host and guests, as well as QEMU v2.10 (host only).
  In the z14 CPU model, the respective feature is:
    iep       Instruction-execution-protection facility
• SIMD Extensions
  Following up to the SIMD instructions as introduced with the previous z13 model, the new z14 provides further vector instructions, which can again be used in KVM guests.
  These new vector instructions can be used to improve decimal calculations as well as for implementing high performance variants of certain cryptographic operations.
  Requires Linux kernel 4.11 as well as QEMU v2.10 in the KVM host, and binaries or a respective Java Runtime Environment in guests using the new vector instructions.
  In the z14 CPU model, the respective feature is:
    vxpd      Vector packed decimal facility
    vxeh      Vector enhancements facility
• Keyless Guest Support
  This feature supports the so-called Keyless Subset (KSS) facility, a new feature of the z14 hardware. With the KSS facility enabled, a host is not required to perform the (costly) storage key initialization and management for KVM guests, unless a guest issues a storage key instruction.
  Requires Linux kernel 4.12 in the KVM host. As for the guests, note that starting with SLES12SP1, RHEL7.2 and Ubuntu 16.04, Linux on IBM Z does not issue any storage key operations anymore.
  This feature does not have a separate entry in the z14 CPU model.
  
• CPUMF Basic Sample Configuration Level Indication
  Basic mode samples as defined in "The Load-Program-Parameter and the CPU-Measurement Facilities" (SA23-2260) do not provide an indication whether the sample was taken in a KVM host or guest. Beginning with z14, the hardware provides an indication of the configuration level (level of SIE, e.g. LPAR or KVM). This item exploits this information to make the perf guest/host decision reliable.
  Requires Linux kernel 4.12 in the KVM host.
  There is no separate entry in the z14 CPU model, since this feature applies to the host only.
• Semaphore assist
  Improves performance of semaphore locks.
  Requires Linux kernel 4.7 and QEMU v2.10 in the KVM host. Exploitation in Linux kernels in guests is still in progress here, scheduled for 4.14.
  In the z14 CPU model, the respective feature is:
    sema      Semaphore-assist facility
• Guarded storage
  This feature is specifically aimed at Java Virtual Machines running in KVM guests to run with fewer and shorter pauses for garbage collection.
  Requires Linux kernel 4.12 and QEMU 2.10 in the KVM host, and a Java Runtime Environment with respective support in the guests.
  In the z14 CPU model, the respective feature is:
    gs        Guarded-storage facility
• MSA Updates
  z14 introduces 3 new Message Security Assists (MSA) for the following functionalities:
      MSA6: SHA3 hashing
      MSA7: A True Random  Number Generator (TRNG)
      MSA8: The CIPHER MESSAGE WITH AUTHENTICATION instruction,
                  which provides support for the Galois-counter-mode (GCM)
  MSA6 and MSA 7 require Linux kernel 4.7, while MSA8 requires Linux kernel 4.12. All require QEMU v2.10 in the KVM host. These features can be exploited in KVM guests' kernels and userspace applications independently (i.e. a KVM guest's userspace applications can take advantage of these features irrespective of the guest's kernel version).
  In the z14 CPU model, the respective features are:
    msa6      Message-security-assist-extension 6 facility
    msa7      Message-security-assist-extension 7 facility
    msa8      Message-security-assist-extension 8 facility
• Compression enhancements
  New instructions improve compression capabilities and performance.
  Requires Linux kernel 4.7 in the KVM host.
  In the z14 CPU model, the respective features are:
    opc       Order Preserving Compression facility
    eec       Entropy encoding compression facility
• Miscellaneous instructions
  Details on these instructions are to be published in the forthcoming z14 Principles of Operation (PoP).
  Requires Linux kernel 4.7 and QEMU 2.10 in the KVM host, and binaries that were compiled for the z14 instruction set using binutils v2.28 and gcc v7.1 in the guests.
  In the z14 CPU model, the respective feature is:
    minste2   Miscellaneous-instruction-extensions facility 2

Note: All versions specified are minimum versions.

Further features will be announced in future blog posts as usual as they find their way into the respective Open Source projects.
Also, don't forget to check this blog entry with further details on z14 in general and Linux on z in particular.

Linux kernel 4.11 released

Linux kernel 4.11 (available here) has been released, improving Kernel Samepage Merging (KSM) support for KVM on z:

KSM is an existing feature of the Linux kernel that allows the kernel to merge pages with identical content, indicated as MADV_MERGEABLE via madvise(), across different processes to save memory.
This z Systems specific extension improves the performance of userspace applications (e.g. QEMU) in case large amounts of empty/zeroed pages are merged.
This feature is disabled by default, enable as follows:

   $ echo 1 > /sys/kernel/mm/ksm/use_zero_pages

Linux kernel 4.8 released

Linux kernel 4.8 (available here) has been released, here are the highlights in support of KVM on z:

Nested Virtualization
This feature allows to start further KVM hosts within KVM guests, also called second level virtualization. As a prerequisite, it requires a recent, post-2.7 version of QEMU including the s390 CPU models (assumed to be included in the forthcoming QEMU 2.8 release).
Nested virtualization is currently turned off per default, and has to be enabled when loading the kvm module:

   [root ~]# modprobe kvm nested=1

or by appending kvm.nested=1 to the kernel command line.
When starting QEMU, make sure to chose the right machine ("s390-ccw-virtio-2.8" or higher) in the domain xml's os element for the guest:

   <os>
      <type arch='s390x' machine='s390-ccw-virtio-2.8'>hvm</type>
      <boot dev='hd'/>
   </os>

Finally, verify in a KVM guest that hosting further KVM guests is possible, as indicated by flag "sie" in /proc/cpuinfo as follows:

   [root ~]# cat /proc/cpuinfo | grep features
   features        : esan3 zarch stfle msa ldisp eimm dfp edat etf3eh
                     highgprs te vx sie

STHYI Instruction available for KVM Guests
Linux kernel 4.8 saw the inclusion of an implementation of the STHYI (Store Hypervisor Information) instruction. Using Linux kernel 4.8 for the host will make the STHYI instruction available to all KVM guests of that host.
Previously available on z/VM only, this instruction provides detailed information on CPU resources on various levels (machine, LPAR, hypervisor, guest, etc.).

Use qclib to access the information. Also see this blog post for further details regarding qclib and KVM on z.

Linux kernel 4.7 released

In Linux kernel 4.7 (available here), we improved usability of the kvm_stat tool for z Systems, introducing (among others) new option -p to filter by process ID.

Using kvm_stat, one can monitor e.g. the number of SIE exits or specific diags to investigate performance issues:

  kvm statistics - summary

   Event                                        Total Current
   kvm_s390_sie_enter                          621113      96
   kvm_s390_sie_exit                           621113      96
   kvm_vcpu_wakeup                             231054      69
   kvm_s390_intercept_instruction              187720      24
   kvm_halt_poll_ns                            131133      15
   kvm_s390_handle_diag                         64543      12
   kvm_userspace_exit                            1427      11
   kvm_s390_handle_sigp                        121746       1
   kvm_set_irq                                 154402       0
   kvm_s390_handle_sigp_pei                     18407       0
   kvm_s390_handle_stsi                            12       0
   kvm_s390_handle_stfl                             1       0

Use option -p to further investigate specific QEMU processes to identify the potential culprits:

  $ ./kvm_stat -p 4304 -1 | grep kvm_s390_sie
  kvm_s390_sie_enter                              3985      3985
  kvm_s390_sie_exit                               3985      3985
  kvm_s390_sie_fault                                 0         0