Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-03-22
20:32:26 sean-k-mooney in pre livemigrate i think
20:32:28 rouk :( and i hoped backporting those cpu feature changes would help me, heh.
20:35:15 sean-k-mooney this is where its failing https://github.com/openstack/nova/blob/3de7fb7c327db348d04d15d4cd3c4f811a336126/nova/virt/libvirt/driver.py#L8991-L9060
20:36:18 sean-k-mooney we ask libvirt if the xml is comparitble https://github.com/openstack/nova/blob/3de7fb7c327db348d04d15d4cd3c4f811a336126/nova/virt/libvirt/host.py#L1424
20:36:20 rouk a migration should survive adding features no? could it be made soft/warn for new features?
20:36:43 sean-k-mooney rouk: no the cpu flag cannot have addtion or removales in a live migration
20:37:23 rouk alright. then new features could be trimmed off?
20:37:51 rouk new features on migrate shouldnt happen... even if its qemu being dumb and retroactively changing things.
20:38:26 sean-k-mooney am yes but im trying to see how nova is in this state
20:38:45 sean-k-mooney we can generate the cpu xml we pass in 3 ways
20:38:52 sean-k-mooney https://github.com/openstack/nova/blob/3de7fb7c327db348d04d15d4cd3c4f811a336126/nova/virt/libvirt/driver.py#L9017-L9032
20:41:12 sean-k-mooney its only caled in 2 places https://github.com/openstack/nova/blob/3de7fb7c327db348d04d15d4cd3c4f811a336126/nova/virt/libvirt/driver.py#L8701-L8708 and here https://github.com/openstack/nova/blob/3de7fb7c327db348d04d15d4cd3c4f811a336126/nova/virt/libvirt/driver.py#L872-L888
20:42:37 rouk why would the vm be checked against target host features? if the vm doesnt have the feature, it shouldnt be checked on the vm side...
20:43:06 rouk nova doesnt care if the vm is missing a feature the host has
20:43:40 rouk if it fits within the features of the new host, it should pass.
20:43:46 rouk which, it does.
20:45:58 rouk if i was to change some of these checks to only care about the host having what the vm has, itd work, no?
20:46:11 rouk or would it migrate with host features and die
20:46:19 sean-k-mooney so nova does need to check the that host has all feature that the vm uses
20:46:30 sean-k-mooney you are right ti does not care about ones that are disabled
20:46:49 sean-k-mooney so nova should ignore onces that are disabeld
20:46:55 rouk yeah, vm needs to fit inside the host, not the other side.
20:47:29 sean-k-mooney right so libvirt did not previoulsy have disabeld feature we had to ignore until recenlty
20:47:35 sean-k-mooney and nova did not provide a way to disable them
20:48:07 rouk if these checks pass, will it migrate with the bad config, or the current vm config (which will work)?
20:48:25 rouk is it as simple as making these checks looser on the vm side?
20:48:28 sean-k-mooney it will migrate with teh current config
20:48:34 sean-k-mooney which should work
20:48:37 rouk yeah
20:48:41 sean-k-mooney well
20:48:46 sean-k-mooney actully not nessisarly
20:49:04 sean-k-mooney so hte issue we have is that how migration work is actully different then most people think
20:49:28 sean-k-mooney libvirt on the source house ask libvirt on the dest host to spawn a qemu instance using an xml we provide
20:49:28 rouk i got confused once i had to patch nova-ssh :p
20:49:56 sean-k-mooney so that qemu instance i like a norm new instance that was booted with a given xml
20:50:27 sean-k-mooney if libvirt addes flags to that qemu becuase it has a different cpu model definiton
20:50:47 sean-k-mooney the qne qemu tryes to do the migration form the source to dest it will fail
20:51:03 rouk so on move, vm-side missing features need to be added as disabled?
20:51:16 rouk instead of just missing
20:51:16 sean-k-mooney yes
20:51:31 sean-k-mooney i belive they would if they weere enabled in the model
20:51:56 rouk yeah, thats what i tried to do with those patches, cause when i manually tested migration, i could make it move by disabling features on the target
20:52:14 rouk whats the most elegant way to add missing as disabled on migrate?
20:53:23 rouk or should i be forking qemu till i can get vms onto new features organically?
20:53:25 rouk heh
20:53:35 rouk id... rather not do that.
20:53:35 sean-k-mooney so the current failure is here https://github.com/openstack/nova/blob/3de7fb7c327db348d04d15d4cd3c4f811a336126/nova/virt/libvirt/driver.py#L8702-L8706 i ruled out the other code path
20:53:52 sean-k-mooney rouk: you dont need to fork qemu
20:54:47 sean-k-mooney for testing you could comment out both checks on the dest host
20:55:13 rouk but wouldnt libvirt then add the feature on move?
20:55:58 sean-k-mooney am well the xml we provide wont reference them
20:56:10 sean-k-mooney but yes your right it might do it implcitly
20:56:20 rouk i can try, if you think its a worthy test
20:57:12 sean-k-mooney i think what will happne is the libvirt error will go away but you might get a qemu error when we actully call migrate
20:57:37 rouk probably, if nova isnt the one adding these in the first place.
20:57:42 sean-k-mooney what i was thinking was we could modify the migrate xml to remove/add the cpu feature based on the config
20:57:56 sean-k-mooney rouk: basically what you were orginally trying to do
20:58:01 sean-k-mooney but also on migrate
20:58:09 sean-k-mooney the orginal patch only did it on spawn
20:58:19 rouk yeah, but wont that be a problem for people in other cases?
20:58:42 rouk for me, sure, it fixes my problem
20:58:49 sean-k-mooney yep im sure ill get a downstream bug for this ill have to help fix in a rush
20:59:02 sean-k-mooney im tyrin g to think through 2 things currently
20:59:12 sean-k-mooney what would not be a horrible hack for you
20:59:33 sean-k-mooney and waht we could do to workaround the libvirt/qemu abi break more generally
21:00:07 rouk well, more generally, cant we just edit the migration xml on compare error, we have the missing features, and we know which direction the failure is.
21:00:40 rouk if the vm is at fault, and its a new feature on the host, disable it, it will get enabled next reboot.
21:00:41 sean-k-mooney maybe we are also currently reqorking how we do the cpu compare
21:01:03 sean-k-mooney https://review.opendev.org/c/openstack/nova/+/762330
21:01:05 rouk i might need a horrible hack though, depending on how long the fix will take.
21:01:08 sean-k-mooney although i dont think that will fix it
21:01:21 rouk every day this sits, new vms come up depending on new features, and old vms are stranded.
21:01:33 rouk got hosts with broken NICs i cant evict, heh
21:01:37 rouk thanks supermicro
21:02:45 sean-k-mooney ya so this will only affect exsiting instance now that you have updated all the contianers
21:02:46 rouk if i didnt have like 1/3rd of my capacity having nics all blow up at once.
21:03:07 sean-k-mooney so option 1 is cold migration or hardreboot + libve migration
21:03:08 rouk i would hardly mind this cpu change, cause id just uh... wait till everyone reboots.
21:03:14 sean-k-mooney not grate but it would work
21:03:25 rouk yeah, its about 1500 VMs to reboot.
21:03:35 rouk and ill get quite the tomatoes thrown at me
21:03:37 sean-k-mooney option 2 patch the cpu model xml and to use the old values
21:03:45 rouk its not in the xml.
21:04:01 rouk its added higher up, in qemu cpu.c
21:04:02 sean-k-mooney no one sec
21:04:18 sean-k-mooney i mean /usr/share/libvirt/cpu_map/x86_EPYC-IBPB.xml
21:04:24 rouk yeah, it doesnt mention these features.
21:04:37 sean-k-mooney yep you could add them and set them disabled
21:04:43 rouk ah
21:04:54 rouk didnt see an arg for disabled on any of the existing lines.
21:04:59 sean-k-mooney then make a copy of the file as normal with a new name
21:05:05 sean-k-mooney and update the nova.conf to use that for new vms
21:05:21 rouk will nova know to use the old name on migrate?
21:05:36 sean-k-mooney yes because that is in the xml which we dont update
21:05:47 rouk ah yeah
21:06:13 rouk its quite the hack, but... i can do it pretty trivially.
21:06:28 rouk just make it part of my nova build.
21:06:47 sean-k-mooney ya so basically mv <file>.xml <file>_v2.xml
21:06:57 sean-k-mooney well cp
21:07:04 sean-k-mooney and then edit <file.xml>

Earlier   Later