Knowledge Series: CPU Models and Migration Considerations

Live guest migration with KVM on IBM Z requires some consideration: While migration to a later IBM Z model is unproblematic due to the platform's backwards compatibility, a guest exploiting a specific feature of a later generation model cannot be migrated to an earlier model, as the following figure illustrates.
Guest A, started on a zEC12 (and using its respective CPU model), can migrate towards a z14 without any issues. But Guest B, started on a z14, cannot be migrated towards a zEC12, due to the lack of various hardware features not present in the zEC12 CPU model as compared to z14. If that happened, Guest B would immediately freeze after migration to the zEC12.

But fear not: Support for CPU models in KVM can prevent migration in cases where compatibility issues are encountered by possibly checking all features in use on the source system against supported features on the target system. If the target system is lacking support for a feature that is strictly required for a KVM guest to run, the host aborts the migration with a respective error message - any attempt to migrate Guest B to the zEC12 would be rejected, and Guest B would continue to run on z14 instead.

While this is certainly a useful feature to prevent havoc from happening, the
administrator is still left with a guest that cannot be migrated to the desired target system. Therefore, a bit of care is required when defining guests.

There is a way out: If we define Guest B on the z14 using a zEC12 CPU model, only the back-level machine's CPU features will be required to run Guest B, and we can guarantee live guest migration capability among all models of at least the zEC12 CPU model generation!
To choose a CPU model, pick any of the following four base modes:
  • Safe settings
    • Host model: Expanded to a reproducible model
        <cpu mode='host-model'/>
    • Custom (variant 1): Choose a predefined model
        <cpu mode='custom'>
            <model>zEC12</model>
        </cpu>
  • Expert Settings
    • Host passthrough: Also prevents any CPU model checks prior to live guest migration, and is therefore discouraged.
        <cpu mode='host-passthrough'/>
    • Custom (variant 2): Modify a predefined model
        <cpu mode='custom'>
            <model>z13-base</model>
            <feature policy='disable' name='dfp'/>
            <feature policy='require' name='vx'/>
        </cpu>
Use any of the safe settings models for guests unless you have special requirements.
If not specified, KVM defaults to host-passthrough. Because it disables CPU model checks, we recommend to always specify host-model instead: It enables the same CPU model features that host-passthrough would, but provides CPU model checks prior to any live guest migration.

Note that not all Linux distributions providing KVM on Z support CPU models, e.g. SLES 12 SP3 and Ubuntu 16.04. Besides the obvious lack of protection against migration faults and the flexibility provided by CPU models, live guest migration from a KVM host with CPU model support to these distributions will not work! Hence, migration from SLES 12 SP4 (which does provide CPU model support, see here) to SLES 12 SP3 does not work.

Use virsh domcapabilities to identify a host's CPU model as well as all compatible models:
   $ virsh domcapabilities
   <domainCapabilities>
   [...]
     <cpu>
  
     <mode name='host-passthrough' supported='yes'/>
   
    <mode name='host-model' supported='yes'>
   
      <model fallback='forbid'>z13.2-base</model>
   
      <feature policy='require' name='aen'/>
         [...]
    </mode>
    <mode name='custom' supported='yes'>
      <model usable='yes'>z890.2</model>
      <model usable='yes'>z990.4</model>
      <model usable='no'>z14ZR1</model>
      <model usable='yes'>z10BC.2</model>
      <model usable='yes'>z196.2</model>
      <model usable='no'>z14</model>
      <model usable='yes'>z9BC-base</model>
      [...]

The respective XML snippets can be copied directly into guest domain definitions.

A comprehensive overview of all available CPU models can be retrieved from QEMU as follows:
   $ qemu-system-s390x -cpu help -enable-kvm
   [...]

   s390 z196-base       IBM zEnterprise 196 GA1             (static, migration-safe)
   s390 z196            IBM zEnterprise 196 GA1             (migration-safe)
   s390 z196.2-base     IBM zEnterprise 196 GA2             (static, migration-safe)
   s390 z196.2          IBM zEnterprise 196 GA2             (migration-safe)
   s390 z114-base       IBM zEnterprise 114 GA1             (static, migration-safe)
   s390 z114            IBM zEnterprise 114 GA1             (migration-safe)
   s390 zEC12-base      IBM zEnterprise EC12 GA1            (static, migration-safe)
   s390 zEC12           IBM zEnterprise EC12 GA1            (migration-safe)
   s390 zEC12.2-base    IBM zEnterprise EC12 GA2            (static, migration-safe)
   s390 zEC12.2         IBM zEnterprise EC12 GA2            (migration-safe)
   s390 zBC12-base      IBM zEnterprise BC12 GA1            (static, migration-safe)
   s390 zBC12           IBM zEnterprise BC12 GA1            (migration-safe)
   s390 z13-base        IBM z13 GA1                         (static, migration-safe)
   s390 z13             IBM z13 GA1                         (migration-safe)
   s390 z13.2-base      IBM z13 GA2                         (static, migration-safe)
   s390 z13.2           IBM z13 GA2                         (migration-safe)
   s390 z13s-base       IBM z13s GA1                        (static, migration-safe)
   s390 z13s            IBM z13s GA1                        (migration-safe)
   s390 z14-base        IBM z14 GA1                         (static, migration-safe)
   s390 z14             IBM z14 GA1                         (migration-safe)
The models *-base models, also described with keyword static, represent a base version of the respective machine generation without any additional features set. This is in contrast to the non-base versions, which will can add further features on top.

As some general guidance, here is a simplified table illustrating the migration compatibility between IBM Z machine generations:
zEC12 z13 z14 z14 ZR1
zEC12 - no no no
z13 yes - no no
z14 yes yes - yes
z14 ZR1 yes yes yes -
Note that z14 ZR1 and z14 models are completely compatible in terms of live guest migration.

1 comment:

  1. There are lots of good things, like this, to read in Stefan's blog

    ReplyDelete