Repository navigation
KVM: live volume migration on Ceph RBD, and between RBD, NFS, SharedMountPoint and local pools - #14426
Draft
weizhouapache wants to merge 4 commits into
Draft
KVM: live volume migration on Ceph RBD, and between RBD, NFS, SharedMountPoint and local pools#14426weizhouapache wants to merge 4 commits into
weizhouapache wants to merge 4 commits into
Conversation
Everything that talks to Ceph needs the pool's monitors as a list the client can parse, and the rules are the same wherever that list is built: give each monitor the pool's port unless it names one already, and put a bare IPv6 address in square brackets so its own colons stay distinguishable from the separator before the port. The mon_host option of a disk's rbd: string wants the same list again, with ':' and ';' escaped, because those two separate the options of that string and its monitors. None of it depends on a storage pool, so collect it in RbdUtils as static methods with their unit tests, rather than leaving each caller to grow a copy that is right in a different set of cases.
- enable migrate volume between ceph - regenerate disk xml for virsh blockcopy - use SHALLOW flag instead of REUSE_EXT
test_13_migrate_volume_and_change_offering previously always stopped the VM before migrating an attached volume on KVM. Now that VolumeApiServiceImpl allows KVM to live-migrate a volume between two RBD pools (VolumeApiServiceImpl.isSourceAndDestOnRBD), keep the VM running and pass livemigrate=true for that RBD-to-RBD case, same as already done for VMware/XenServer. Any other KVM source/destination combination keeps the existing stop/migrate/start behavior.
…l pools Migrate a volume of a running Instance between an RBD pool and an NFS, SharedMountPoint or host-local pool, and between those file based pools, with a libvirt block copy on the host the Instance runs on, so that neither the Instance nor its host change. Migrate encrypted volumes between RBD and NFS, online and offline. Management server: - VolumeApiServiceImpl lets a KVM live migration through for these pool pairs, where it asked for migrateVirtualMachineWithVolumes before. A local pool has to be the one of the host the Instance runs on. - AncientDataMotionStrategy keeps the format of the volume right after the move: RAW on RBD, QCOW2 on a file based pool, and luks2 or luks as the encryption format, as the agent encrypts an RBD volume with librbd and any other volume with qemu. KVM agent: - LibvirtMigrateVolumeCommandWrapper builds the disk XML of the destination for the type of the destination pool, an RBD image or a qcow2 file, and copies the whole disk, so a volume cloned from a template does not depend on a backing file of the source pool. SHALLOW is only used from RBD to RBD. The source volume is removed after the pivot and the destination pool refreshed. - An encrypted volume gets its destination created with the encryption header first, and an offline copy goes through qemu-img with both secrets (LibvirtStorageAdaptor, KVMStorageProcessor). - LibvirtStorageVolumeXMLParser read the volume format with its arguments swapped and found none, so a migrated qcow2 file named like an RBD image was taken for a raw one and the Instance failed to start again. It reads the format of the volume now. - The ceph.conf which the copy of an encrypted volume writes gets its mon_host from RbdUtils.getRbdMonitors, which gives each monitor its port, leaves alone one which has it, and brackets a bare IPv6 address. Tests: - Unit tests for the pool pairs which are accepted, the local pool of another host, and the dispatch of the agent to the block copy. - test_volume_migration_rbd_nfs.py migrates ROOT and DATA volumes online, offline and detached, encrypted or not, and stops and starts the Instance after every move to check that it is still reachable. - test_volumes.py: the data disk of the encrypted volume migration test is found by its size.
boring-cyborg
Bot
added
component:api
component:integration-test
component:kvm
component:storage
Python
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #14426 +/- ##
==========================================
Coverage 20.39% 20.40%
- Complexity 21283 21326 +43
==========================================
Files 6481 6482 +1
Lines 588040 588429 +389
Branches 72021 72111 +90
==========================================
+ Hits 119957 120059 +102
- Misses 455052 455323 +271
- Partials 13031 13047 +16
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
2 of 14 tasks
winterhazel
self-requested a review
October 16, 2026 16:29
bernardodemarco
self-requested a review
October 16, 2026 16:29
Member
Author
|
@winterhazel @bernardodemarco I will revert some of the changes related to live volume migration between NFS, local, and SharedMountPoint pools. These changes use blockcopy for the migrations. Although they worked in my testing, I think it's better to keep the current behavior and leave this out of the scope of the PR for now. We can discuss it separately later. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
migrateVolumewithlivemigrate=trueon KVM was refused for every pair of pools, except for storage with its own handling (StorPool, PowerFlex). The error told the operator to usemigrateVirtualMachineWithVolumes, which moves the whole Instance to another host. This PR lets an operator move a single volume of a running Instance, without stopping the Instance and without moving it, between:Encrypted volumes can also be migrated between RBD and NFS, online and offline.
How it works:
VolumeApiServiceImplaccepts these pool pairs for a KVM live migration. A local pool has to be the one of the host the Instance is running on, as the copy runs there.AncientDataMotionStrategykeeps the volume's format right after the move (RAW on RBD, QCOW2 on a file pool) and its encryption format (luks2on RBD,lukselsewhere).LibvirtMigrateVolumeCommandWrapperbuilds the disk XML of the destination for the type of the destination pool (an RBD image or a qcow2 file) and runs a libvirt block copy on the running domain. The whole disk is copied, so a volume cloned from a template does not depend on a backing file of the source pool.SHALLOWis used only from RBD to RBD. After the pivot the source volume is removed. An encrypted volume gets its destination created with the encryption header first; an offline copy goes throughqemu-imgwith both secrets.LibvirtStorageVolumeXMLParserread the volume format with its two arguments swapped, so it never found one. After a migration from RBD, a qcow2 file still named like an RBD image was taken for a raw one, and libvirt then failed to start the Instance again. The parser now reads<format type='…'/>, sogetPhysicalDiskreports the format libvirt sees.The branch has four commits:
utils: add RbdUtils for the Ceph monitors of an RBD pool. This is the first commit of utils,kvm: build every Ceph monitor list with RbdUtils #14402, so the encrypted copy can build itsmon_hostwithRbdUtils.getRbdMonitors. It will drop out of this PR when utils,kvm: build every Ceph monitor list with RbdUtils #14402 is merged.KVM: live volume migration on ceph. RBD to RBD.smoke tests: exercise online volume migration for KVM on RBD.KVM: live migrate volumes between RBD, NFS, SharedMountPoint and local pools. Everything else above, with its unit tests andtest_volume_migration_rbd_nfs.py.Types of changes
Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale
Bug Severity
How Has This Been Tested?
Unit tests, on this branch:
RbdUtilsTest(17),VolumeApiServiceImplTest(167),AncientDataMotionStrategyTest(19),LibvirtMigrateVolumeCommandWrapperTest(12) andLibvirtStorageAdaptorTest(23), all passing. New cases cover the pool pairs that are accepted, the local pool of another host, and the dispatch of the agent to the block copy.A live matrix, on the same changes from my
4.23.0-cephbranch, which carries them together with the commits of my other Ceph PRs. Two KVM hosts; Ceph with two pools; two NFS pools; two SharedMountPoint pools; one local pool per host. One Instance with a ROOT and a DATA volume, moved together, 23 legs: RBD to RBD and back, RBD to and from NFS, SharedMountPoint and local, and every pair among NFS, SharedMountPoint and local. After each leg:Four of five full matrices passed. A 5 GiB DATA disk took about 45 to 95 seconds to copy, a small ROOT disk 3 to 9 seconds.
The fifth run failed once, on an RBD to RBD copy of the DATA disk. The copy finished and the Instance switched to the new image, but the removal of the old image by libvirt failed after about a minute ("failed to remove volume … No such file or directory"). The migration was reported as failed, so CloudStack kept the old pool while the Instance used the new one. I did not find the cause, and it did not happen again in the three runs that followed.
Not done: the Marvin smoke test
test_volume_migration_rbd_nfs.pyhas not been run by Marvin from this branch (Marvin does not work on that setup). Offline and detached migrations and encrypted volumes were not part of the final matrices.migrateVirtualMachineWithVolumeswas not retested.How did you try to break this feature and the system with this change?
apache/main, which does not yet have kvm: look RBD volumes up through librbd instead of refreshing the whole pool #14188 or kvm: remove the partial RBD image when a template copy fails #14393. Two edits of this work tomon_hostlines in methods from those PRs were left out, so expect a small conflict inLibvirtStorageAdaptorwhen they merge.