QEMU 9.0 and libvirt 10.0 introduced an important improvement for disk (virtio-blk) scalability. Until now, a virtio-blk device could use one iothread in the host (or the QEMU main thread).
For a while it was already possible to specify multiple virtio queues for a single virtio-block device, and the Linux guest driver was able to exploit thse concurrently:
<driver name='qemu' queues='3'>All queues have been handled by one iothread. This introduced a limit for the maximum amount of IO requests per seconds per disk no matter how many queues have been defined.
Now it is possible to assign a queue to a given host iothread allowing for higher throughput per device:
<driver name='qemu' queues='3'>
<iothreads>
<iothread id='2'>
<queue id='1'/>
</iothread>
<iothread id='3'>
<queue id='0'/>
<queue id='2'/>
</iothread>
</iothreads>
</driver>Example with 3 queues to illustrate the possibility to have 2 queues on one iothread and one queue on another. In real life 2 or 4 queues make more sense.
Initial tests showed improved performance and reduced CPU cycles when going from 1 to 2 queues. More performance analysis needs to happen but this looks like a very promising improvement and going from 1 to 2 is almost a no-brainer. Adding more queues continues to improve the performance, but also increases the overall CPU consumption so this needs additional considerations.
Sharing iothreads across multiple disks continues to be possible.
This feature is also being backported into some distributions like RHEL 9.4 or will be available via regular QEMU/libvirt upgrades.
Very interesting! A queues='2' will now automatically use multiple iothreads or do we need to manually dispatch queues like in your 2nd example?
ReplyDeleteNo, you (or a management tool) need to do the manual placing. Otherwise every guest queue is being backed by the QEMU main io thread.
Delete