Migration fails with something like ..pckmo.. ?

 If you run into a situation where migration fails with something like

internal error: QEMU unexpectedly closed the monitor (vm='testguest'):
qemu-kvm: Some features requested in the CPU model are not available in the current configuration: pckmo-aes-256 pckmo-aes-192 pckmo-aes-128 pckmo-etdea-192 pckmo-etdea-128 pckmo-edea msa9_pckmo Consider a different accelerator, QEMU, or kernel version

This indicates that both host are configured differently regarding the CPACF key management Operations

So you can (preferred if the security scheme allows for that) configure both LPARs the same way, de-activate and re-activate the LPAR

or

you can change the CPU model of the guest to no longer have these pckmo functions. Change your guest xml from "host-model" to "host-model with some features disabled".

So shutdown the guest and change the XML from

  <cpu mode='host-model' check='partial'/>

to

  <cpu mode='host-model' check='partial'>
    <feature policy='disable' name='msa3'/>

    <feature policy='require' name='msa3-base'/>
    <feature policy='disable' name='msa9_pckmo'/>
  </cpu>

This guest can now be migrated to a host without key managment functions. The downside is that this guest no longer has access to the key management functions. This makes it harder to use encrypted swap devices with automatic key generation. Therefore, enabling these functions on both LPARs is preferred.

New Feature: Installation Assistant for Linux on IBM Z

Ever struggled to create configuration files for starting Linux on IBM Z and LinuxONE installations? Fear no more, we got you covered now: A new assistant available online will help you create parameter files!
Writing parameter files can be a challenge, with bugs triggering cycles with lengthy turnaround times. Our new installation assistant generates installer parameter files by walking you through a step-by-step process, where you answer simple questions to generate a parameter file. Comes with contextual help in every stage, so you can follow along what is happening!
Currently supports OSA and PCI networking devices, IPv4/v6, and VLAN installations.

Currently supports RHEL 9 and SLES 15 SP5 or later.
Access the assistant at https://ibm.github.io/liz/

New Video: Configuring Crypto Express Adapters for KVM Guests

A new video illustrating the steps to perform on a KVM host and in a virtual server configuration to make AP queues of cryptographic adapters available to a KVM guest can be found here.

virtio-blk scalability improvements

QEMU 9.0 and libvirt 10.0 introduced an important improvement for disk (virtio-blk) scalability. Until now, a virtio-blk device could use one iothread in the host (or the QEMU main thread).
For a while it was already possible to specify multiple virtio queues for a single virtio-block device, and the Linux guest driver was able to exploit thse concurrently:

<driver name='qemu' queues='3'>

All queues have been handled by one iothread. This introduced a limit for the maximum amount of IO requests per seconds per disk no matter how many queues have been defined.

Now it is possible to assign a queue to a given host iothread allowing for higher throughput per device:

<driver name='qemu' queues='3'>
  <iothreads>
    <iothread id='2'>
      <queue id='1'/>
    </iothread>
    <iothread id='3'>
      <queue id='0'/>
      <queue id='2'/>
    </iothread>
  </iothreads>
</driver>

Example with 3 queues to illustrate the possibility to have 2 queues on one iothread and one queue on another. In real life 2 or 4 queues make more sense.

Initial tests showed improved performance and reduced CPU cycles when going from 1 to 2 queues. More performance analysis needs to happen but this looks like a very promising improvement and going from 1 to 2 is almost a no-brainer. Adding more queues continues to improve the performance, but also increases the overall CPU consumption so this needs additional considerations.

Sharing iothreads across multiple disks continues to be possible. 

This feature is also being backported into some distributions like RHEL 9.4 or will be available via regular QEMU/libvirt upgrades.

IBM Secure Execution for Linux support for Crypto Express adapters

IBM Secure Execution for Linux -- the Linux Kernel Virtual Machine (KVM) based Confidential Computing technology for IBM LinuxONE and Linux on IBM Z -- now allows Secure Execution guests leverage secure passthrough access to up to 12 Crypto Express 8S adapter domains in accelerator or EP11 co-processor mode.

Customers who require the highest level of protection (FIPS 140-2 level 4 certified) for their cryptographic keys and thus for their sensitive data can now have their workloads deployed as Secure Execution KVM guests with access to Hardware Security Modules (HSMs) if the provider uses IBM z16 or LinuxONE 4 servers with Crypto Express 8S adapters. This combination provides business value for solutions around key and certificate management, multi-party computation and digital assets. But more use cases arise as confidential computing becomes more common and the need to leverage such highly certified HSM to protect AI models or provide data sovereignty across organizational and infrastructure boundaries grows.

To exploit this new function, IBM z16 or LinuxONE 4 severs with firmware bundles S30 and S31b are needed. To use a Crypto Express 8S adapter in EP11 mode the minimal EP11 firmware version loaded must be version 5.8.30.

IBM is working with Linux distribution partners to include the required Linux support for this function for both the KVM Hypervisor and the Secure Execution guests in future distribution releases. Linux support for this function is already available today with Ubuntu 24.04 (Noble Numbat).

This new capability showcases IBM’s commitment and previously stated direction to foster the use of confidential computing and expand the security value proposition of existing security and crypto solutions as the business needs of our customers and technical possibilities evolve.

For detailed information on how to use Crypto Express support see the Introducing IBM Secure Execution for Linux publication.

 

Authored by

• Reinhard Bündgen – buendgen@de.ibm.com
• Nicolas Mäding – nmaeding@de.ibm.com

New Release: Ubuntu 24.04

Canonical released a new version of their Ubuntu server offering Ubuntu Server 24.04!

 Highlights include

• HSM support for Secure Execution
• Further Crypto enhancements and extensions

See the announcement on the mailing list here, and the blog entry at Canonical with all Z-specific highlights here.

This release is very significant, since it marks a so-called LTS (Long Term Support) release, granting an extended service timeframe of up to 10 years, as illustrated here.

Important Note on Verifying Secure Execution Host Key Documents

The certificates of the host key signing keys that are needed to verify host key documents will expire on April 24, 2024 for IBM z15 and LinuxONE III and on March 29, 2024 for IBM z16 and LinuxONE 4. Due to a requirement from the Certificate Authority (DigiCert), the renewed certificates are equipped with a new Locality value (“Armonk” instead of “Poughkeepsie”). These renewed certificates cause the current versions of the genprotimg, pvattest, and pvsecret tools to fail the verification of host key documents.

The IBM Z team is preparing updates of the genprotimg, pvattest, and pvsecret tools to accept the new certificates and is working with Linux distribution partners to release the updated tools.

To build new Secure Execution images, attestation requests, or add-secret requests before the updated tools are available in Linux distributions, follow these steps:

Step 1:

Obtain the host key document, the host key signing key certificate, the intermediate certificate from the Certificate Authority, and the list of revoked host keys (CRL):

• For IBM z15 and LinuxONE III, see https://www.ibm.com/support/resourcelink/api/content/public/secure-execution-gen1.html

• For IBM z16 and LinuxONE 4, see https://www.ibm.com/support/resourcelink/api/content/public/secure-execution-gen2.html
  
  

Step 2: 

Download the script check_hostkeydoc from https://github.com/ibm-s390-linux/s390-tools/blob/master/genprotimg/samples/check_hostkeydoc.

Step 3:

Verify each host key document using the check_hostkeydoc script. For example, issue

# ./check_hostkeydoc HKD1234.crt ibm-z-host-key-signing.crt \ -c DigiCertCA.crt -r ibm-z-host-key.crl

This example verifies the host key document HKD1234.crt using the host key signing key certificate ibm-z-host-key-signing.crt, and the intermediate certificate of the Certificate Authority DigiCertCA.crt,as well as the list of revoked host keys ibm-z-host-key.crl.

After the host key documents are verified using the check_hostkeydoc script, you can safely call genprotimg, pvattest, or pvsecret with the –-no-verify option.

For a description about how to manually verify host key documents, see https://www.ibm.com/docs/en/linux-on-z?topic=execution-verify-host-key-document.