Monday, December 12, 2016

libvirt v.2.5.0 released

libvirt v2.5.0 is now available for download at the libvirt project website.
A highlight for the z Systems platform support in this release is scsi_host hostdev passthrough, improving throughput by providing a libvirt frontend for scsi_host devices.

Wednesday, November 16, 2016

New White Paper available: KVM Network Performance

A new white paper has been published: KVM Network Performance - Best Practices and Tuning Recommendations. It is available here, and explores different system configurations (running KVM guests), different networking configuration choices, as well as tuning recommendations for the KVM host and KVM guest environments to achieve greater network performance on the IBM z Systems platforms.

Tuesday, November 15, 2016

New Books available

Two new books are available as follows:
  • The new book KVM Virtual Server Management Tools is available here, covering the open source package virt-manager (see screenshot above) along with its supporting tools virt-install and virt-clone in a generic manner. I.e. it applies to both, KVM on z in general as well as the KVM for IBM z Systems product.
  • The Device Drivers, Features, and Commands for Linux as a KVM Guest book is now also available in a version specific to Ubuntu 16.04 here.
For a publications overview see here.

Friday, October 28, 2016

KVM for IBM z Systems v1.1.2 released

KVM for IBM z Systems v1.1.2 is out today! See here for the respective blog post on Mainframe Insights, and the following pages for the formal announcements: US, Canada, Asia-Pacific, Japan, Latin America and Europe.

It ships with QEMU v2.6 and libvirt v1.3.3. Here is a list of highlights from a KVM on z perspective:
  • Enhanced SCA support
    Exploit up to 248 CPUs per KVM guest
  • SIE capability
    Expose SIE availability in /proc/cpuinfo as follows:

       $ cat /proc/cpuinfo | grep features
       features        : esan3 zarch stfle msa ldisp eimm dfp edat etf3eh
                         highgprs te vx sie
  • STP host toleration
    Previously, Server Time Protocol (STP) had to be turned off in the host. In case of time differences, the TOD clock is adjusted in a smooth manner, avoiding jumps.
  • Improved removable media support
    Allow to boot KVM guests seamlessly from ISO9660 media.
  • OASIS VIRTIO v1.0 support
    Support OASIS OASIS standard for virtio devices.
  • CPU hotplugging
    Allow to add CPUs dynamically to a running guest
  • STHYI support: Added the z/VM-defined Store Hypervisor Information instruction (STHYI) for KVM guests. See also here.

Thursday, October 6, 2016

Linux kernel 4.8 released

Linux kernel 4.8 (available here) has been released, here are the highlights in support of KVM on z:

Nested Virtualization
This feature allows to start further KVM hosts within KVM guests, also called second level virtualization. As a prerequisite, it requires a recent, post-2.7 version of QEMU including the s390 CPU models (assumed to be included in the forthcoming QEMU 2.8 release).
Nested virtualization is currently turned off per default, and has to be enabled when loading the kvm module:

   [root ~]# modprobe kvm nested=1

or by appending kvm.nested=1 to the kernel command line.
When starting QEMU, make sure to chose the right machine ("s390-ccw-virtio-2.8" or higher) in the domain xml's os element for the guest:

      <type arch='s390x' machine='s390-ccw-virtio-2.8'>hvm</type>
      <boot dev='hd'/>

Finally, verify in a KVM guest that hosting further KVM guests is possible, as indicated by flag "sie" in /proc/cpuinfo as follows:

   [root ~]# cat /proc/cpuinfo | grep features
   features        : esan3 zarch stfle msa ldisp eimm dfp
edat etf3eh
                     highgprs te vx sie

STHYI Instruction available for KVM Guests
Linux kernel 4.8 saw the inclusion of an implementation of the STHYI (Store Hypervisor Information) instruction. Using Linux kernel 4.8 for the host will make the STHYI instruction available to all KVM guests of that host.
Previously available on z/VM only, this instruction provides detailed information on CPU resources on various levels (machine, LPAR, hypervisor, guest, etc.).

Use qclib to access the information. Also see this blog post for further details regarding qclib and KVM on z.

Thursday, September 15, 2016

QEMU v2.7 released

QEMU v2.7 is out! Here is a list of the highlights for KVM on z:
  • Support for PCI passthrough devices is now fully functional for the first time. While we consider support as experimental, since there is still work to be done to utilize hardware acceleration facilities as well as further testing pending, attaching PCI devices directly to guests is now possible. Also note that while hot-plugging PCI devices is supported, hot-unplugging is not. Further limitations, e.g. for live guest migration, apply for now.
    We strictly recommend to use libvirt for all configuring, including PCI devices. Therefore, a step-by-step description of how to attach a PCI device to a KVM guest on z Systems will follow when the respective code becomes available in one of the next libvirt releases.

  • Runtime instrumentation in KVM guests: IBM z13 and LinuxONE machines introduced a new hardware facility which allows to collect sampling data on code execution in CPUs at runtime. This data can be used by just-in-time compilers, like the one provided by IBM Java, to improve the generated code for better performance.
  • IPL from any subchannel set: Prior to QEMU v2.7, KVM guests could only IPL/boot from devices with a subchannel set ID of 0. Once IPL'd, they could attach disks in any subchannel set just fine, but the IPL/boot from devices in subchannel sets 1-3 was not possible. With the addition of this feature, KVM guests can IPL from devices in all 4 subchannel sets.
    Here is a sample domain definition indicating the change:

      <disk type=’file’ device=’disk’>
    <driver name=’qemu’ type=’qcow2’/>
      <source file=’/var/lib/libvirt/images/guest.qcow2’/>
        <target dev=’vda’ bus=’virtio’/>
        <address type=’ccw’ cssid=’0xab’ ssid=’0x2’

Wednesday, September 7, 2016

Open Positions in Development

Recently graduated (or about to do so) and looking for a job in KVM or Linux development on z Systems?
Then go to this page and search for
  • 59794BR for a position in KVM on z development ("Entwickler/in für Linux Virtualisierung auf IBM z Systems"), or
  • 59784BR for a position in Linux on z development ("Linux Entwickler für IBM z Systems (m/f)")

Friday, July 29, 2016

Linux kernel 4.7 released

In Linux kernel 4.7 (available here), we improved usability of the kvm_stat tool for z Systems, introducing (among others) new option -p to filter by process ID.

Using kvm_stat, one can monitor e.g. the number of SIE exits or specific diags to investigate performance issues:
  kvm statistics - summary

   Event                                        Total Current
   kvm_s390_sie_enter                          621113      96
   kvm_s390_sie_exit                           621113      96
   kvm_vcpu_wakeup                             231054      69
   kvm_s390_intercept_instruction              187720      24
   kvm_halt_poll_ns                            131133      15
   kvm_s390_handle_diag                         64543      12
   kvm_userspace_exit                            1427      11
   kvm_s390_handle_sigp                        121746       1
   kvm_set_irq                                 154402       0
   kvm_s390_handle_sigp_pei                     18407       0
   kvm_s390_handle_stsi                            12       0
   kvm_s390_handle_stfl                             1       0
Use option -p to further investigate specific QEMU processes to identify the potential culprits:
  $ ./kvm_stat -p 4304 -1 | grep kvm_s390_sie
  kvm_s390_sie_enter                              3985      3985
  kvm_s390_sie_exit                               3985      3985
  kvm_s390_sie_fault                                 0         0

Wednesday, July 13, 2016

Webinar: KVM for z Systems Performance

Abstract: KVM for IBM z Systems performance has not been presented at conferences before. This session covers performance aspects of the KVM host, virtual devices, and considerations for guest definitions to reach certain performance levels. It will also give examples how to monitor host and guest.
Note: It is not intended to compare KVM for IBM z Systems to other hypervisors.

Speaker: Martin Kammerer, Manager Linux on z Systems Performance Evaluation, IBM Germany

Audience: IT Managers, IT Architects, System Administrators

Date: Wednesday, July 20, 2016
Time: 11:00 AM EST / New York, 4:00 PM UK, 5:00 PM CET / Germany
Duration: 75 minutes

Registration: You can register here. After registering, you will receive a confirmation email containing information about joining the webinar.

Replay & Archive
: All sessions are recorded. For the archive as well as a replay and handout of this session (available by July 25, 2016), as well as all previous webcasts please visit

Contact: Please direct any questions to Stephanie Gherghe at

Friday, July 8, 2016

KVM on z Documentation References

The amount of documentation provided in the IBM Knowledge Center can at times be overwhelming. Therefore, this entry will provide some general guidance.
See here for a continuously updated version of this entry.

The documentation for KVM on z (see overall entry point here) is grouped under these categories:
For white papers, Redbooks, live virtual classes, and more, check out the materials related to virtualization on z Systems available here.

In addition, here is a document listing the recommended limits when running KVM on z Systems. It is aimed at the KVM for IBM z product, but also provides a good starting point when running other flavors of KVM on z.

Finally, documentation and pointers on ICM (IBM Cloud Manager, IBM's OpenStack-based product) can be found here.

Friday, June 24, 2016

KVM Forum 2016

The schedule of the KVM Forum 2016, to be held August 24-26 in Toronto, is out here.
We (KVM on z project) will be hosting three sessions:

Friday, June 10, 2016

qclib v1.2.0 released

New version qclib v1.2.0 was released on IBM developerWorks here.
Among others, this version adds additional counts for KVM hosts and guests as follows:
  • For KVM hosts: Counts for CPs, IFLs and CPUs (sum of CPs and IFLs)
  • For KVM guests: CP and IFL counts
Below is some (edited) output of the sample program qc_test (part of the library), highlighting the new attributes for a KVM host with 4 IFLs and a guest with 2 virtual IFLs. See the README for documentation and further details.
        ===== Layer 3: KVM-hypervisor ==========================
                 qc_layer_type [n/a]: KVM-hypervisor
             qc_layer_category [n/a]: HOST
             qc_layer_type_num [n/a]: 6
         qc_layer_category_num [n/a]: 2
         qc_control_program_id [S  ]: KVM/Linux
                 qc_adjustment [S  ]: 1000

              qc_num_cpu_total [  V]: 4
          qc_num_cpu_dedicated [SHV]: 0
             qc_num_cpu_shared [SHV]: 4
               qc_num_cp_total [ HV]: 0
           qc_num_cp_dedicated [ hV]: 0
              qc_num_cp_shared [ hV]: 0
              qc_num_ifl_total [SHV]: 4
          qc_num_ifl_dedicated [ShV]: 0
             qc_num_ifl_shared [ShV]: 4


          ===== Layer 4: KVM-guest
                   qc_layer_type [n/a]: KVM-guest
               qc_layer_category [n/a]: GUEST
               qc_layer_type_num [n/a]: 7
           qc_layer_category_num [n/a]: 1
                   qc_layer_name [S  ]: u1604
          qc_layer_extended_name [S  ]: u1604
                   qc_layer_uuid [S  ]: 407d134c-7d93-4a7b-bd27-4568ef10e85a

                qc_num_cpu_total [S  ]: 2
           qc_num_cpu_configured [S  ]: 2
              qc_num_cpu_standby [S  ]: 0
             qc_num_cpu_reserved [S  ]: 0
            qc_num_cpu_dedicated [S  ]: 0
               qc_num_cpu_shared [S  ]: 2
                qc_num_ifl_total [S  ]: 2
            qc_num_ifl_dedicated [S  ]: 0
               qc_num_ifl_shared [S  ]: 2
            qc_ifl_dispatch_type [SHV]: 3


    qclib is a C library providing access to system information on the various layers of virtualization on z Systems.

    KVM for IBM z Systems Feature Requests

    In case you are missing a feature in IBM's own Linux distribution KVM for IBM z Systems, you can now join the RFE (Request for Enhancement) community on developerWorks to file your request!
    Use the following query in the community to list all requests against KVM for IBM z Systems:

    Monday, June 6, 2016

    libvirt 1.3.5 released

    libvirt v1.3.5 is now available for download at the libvirt project website.
    The list of highlights improving z Systems platform support is as follows:
    • virt-host-validate: Added support for the s390 platform. This tool detects virtualization capabilities of a host, and can verify capabilities for a specific hypervisor such as QEMU.
    • Added a panic device model for s390. This new device model allows to represent the z Systems architecture's guest panic capabilities. A panic device is useful to receive guest panic events.

    Wednesday, June 1, 2016

    QEMU 2.6 released

    QEMU 2.6 has been released. In addition to the many improvements provided by the community, here is a list of the new features with most relevance for KVM on z:
    • On z Systems, it is now possible to IPL from virtio-scsi attached disks and disk images.
    • CPU hotplug: CPU hotplugging via command cpu-add now works on z Systems.
    • Postcopy: We made sure that this feature provided by the community is supported on z Systems from the first day after losing its experimental status.
    • The script now works on z Systems.
    • PCI passthrough support continues to see improvements, but is not yet ready for production use.


    Welcome to the KVM on z blog! The primary focus will be upstream news for open source projects Linux, QEMU and libvirt with relevance to KVM on IBM z Systems, but we will touch other areas as well.