Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-03-22
20:43:06 rouk nova doesnt care if the vm is missing a feature the host has
20:43:40 rouk if it fits within the features of the new host, it should pass.
20:43:46 rouk which, it does.
20:45:58 rouk if i was to change some of these checks to only care about the host having what the vm has, itd work, no?
20:46:11 rouk or would it migrate with host features and die
20:46:19 sean-k-mooney so nova does need to check the that host has all feature that the vm uses
20:46:30 sean-k-mooney you are right ti does not care about ones that are disabled
20:46:49 sean-k-mooney so nova should ignore onces that are disabeld
20:46:55 rouk yeah, vm needs to fit inside the host, not the other side.
20:47:29 sean-k-mooney right so libvirt did not previoulsy have disabeld feature we had to ignore until recenlty
20:47:35 sean-k-mooney and nova did not provide a way to disable them
20:48:07 rouk if these checks pass, will it migrate with the bad config, or the current vm config (which will work)?
20:48:25 rouk is it as simple as making these checks looser on the vm side?
20:48:28 sean-k-mooney it will migrate with teh current config
20:48:34 sean-k-mooney which should work
20:48:37 rouk yeah
20:48:41 sean-k-mooney well
20:48:46 sean-k-mooney actully not nessisarly
20:49:04 sean-k-mooney so hte issue we have is that how migration work is actully different then most people think
20:49:28 rouk i got confused once i had to patch nova-ssh :p
20:49:28 sean-k-mooney libvirt on the source house ask libvirt on the dest host to spawn a qemu instance using an xml we provide
20:49:56 sean-k-mooney so that qemu instance i like a norm new instance that was booted with a given xml
20:50:27 sean-k-mooney if libvirt addes flags to that qemu becuase it has a different cpu model definiton
20:50:47 sean-k-mooney the qne qemu tryes to do the migration form the source to dest it will fail
20:51:03 rouk so on move, vm-side missing features need to be added as disabled?
20:51:16 sean-k-mooney yes
20:51:16 rouk instead of just missing
20:51:31 sean-k-mooney i belive they would if they weere enabled in the model
20:51:56 rouk yeah, thats what i tried to do with those patches, cause when i manually tested migration, i could make it move by disabling features on the target
20:52:14 rouk whats the most elegant way to add missing as disabled on migrate?
20:53:23 rouk or should i be forking qemu till i can get vms onto new features organically?
20:53:25 rouk heh
20:53:35 sean-k-mooney so the current failure is here https://github.com/openstack/nova/blob/3de7fb7c327db348d04d15d4cd3c4f811a336126/nova/virt/libvirt/driver.py#L8702-L8706 i ruled out the other code path
20:53:35 rouk id... rather not do that.
20:53:52 sean-k-mooney rouk: you dont need to fork qemu
20:54:47 sean-k-mooney for testing you could comment out both checks on the dest host
20:55:13 rouk but wouldnt libvirt then add the feature on move?
20:55:58 sean-k-mooney am well the xml we provide wont reference them
20:56:10 sean-k-mooney but yes your right it might do it implcitly
20:56:20 rouk i can try, if you think its a worthy test
20:57:12 sean-k-mooney i think what will happne is the libvirt error will go away but you might get a qemu error when we actully call migrate
20:57:37 rouk probably, if nova isnt the one adding these in the first place.
20:57:42 sean-k-mooney what i was thinking was we could modify the migrate xml to remove/add the cpu feature based on the config
20:57:56 sean-k-mooney rouk: basically what you were orginally trying to do
20:58:01 sean-k-mooney but also on migrate
20:58:09 sean-k-mooney the orginal patch only did it on spawn
20:58:19 rouk yeah, but wont that be a problem for people in other cases?
20:58:42 rouk for me, sure, it fixes my problem
20:58:49 sean-k-mooney yep im sure ill get a downstream bug for this ill have to help fix in a rush
20:59:02 sean-k-mooney im tyrin g to think through 2 things currently
20:59:12 sean-k-mooney what would not be a horrible hack for you
20:59:33 sean-k-mooney and waht we could do to workaround the libvirt/qemu abi break more generally
21:00:07 rouk well, more generally, cant we just edit the migration xml on compare error, we have the missing features, and we know which direction the failure is.
21:00:40 rouk if the vm is at fault, and its a new feature on the host, disable it, it will get enabled next reboot.
21:00:41 sean-k-mooney maybe we are also currently reqorking how we do the cpu compare
21:01:03 sean-k-mooney https://review.opendev.org/c/openstack/nova/+/762330
21:01:05 rouk i might need a horrible hack though, depending on how long the fix will take.
21:01:08 sean-k-mooney although i dont think that will fix it
21:01:21 rouk every day this sits, new vms come up depending on new features, and old vms are stranded.
21:01:33 rouk got hosts with broken NICs i cant evict, heh
21:01:37 rouk thanks supermicro
21:02:45 sean-k-mooney ya so this will only affect exsiting instance now that you have updated all the contianers
21:02:46 rouk if i didnt have like 1/3rd of my capacity having nics all blow up at once.
21:03:07 sean-k-mooney so option 1 is cold migration or hardreboot + libve migration
21:03:08 rouk i would hardly mind this cpu change, cause id just uh... wait till everyone reboots.
21:03:14 sean-k-mooney not grate but it would work
21:03:25 rouk yeah, its about 1500 VMs to reboot.
21:03:35 rouk and ill get quite the tomatoes thrown at me
21:03:37 sean-k-mooney option 2 patch the cpu model xml and to use the old values
21:03:45 rouk its not in the xml.
21:04:01 rouk its added higher up, in qemu cpu.c
21:04:02 sean-k-mooney no one sec
21:04:18 sean-k-mooney i mean /usr/share/libvirt/cpu_map/x86_EPYC-IBPB.xml
21:04:24 rouk yeah, it doesnt mention these features.
21:04:37 sean-k-mooney yep you could add them and set them disabled
21:04:43 rouk ah
21:04:54 rouk didnt see an arg for disabled on any of the existing lines.
21:04:59 sean-k-mooney then make a copy of the file as normal with a new name
21:05:05 sean-k-mooney and update the nova.conf to use that for new vms
21:05:21 rouk will nova know to use the old name on migrate?
21:05:36 sean-k-mooney yes because that is in the xml which we dont update
21:05:47 rouk ah yeah
21:06:13 rouk its quite the hack, but... i can do it pretty trivially.
21:06:28 rouk just make it part of my nova build.
21:06:47 sean-k-mooney ya so basically mv <file>.xml <file>_v2.xml
21:06:57 sean-k-mooney well cp
21:07:04 sean-k-mooney and then edit <file.xml>
21:07:16 sean-k-mooney you could do that as a layer in the nova-libvirt container
21:08:02 sean-k-mooney if you want new vms to use the new defintion just set cpu_model=eypc-ibpb-v2
21:08:17 sean-k-mooney new vms will get that but migrated ones wont until you hard reboot
21:08:55 sean-k-mooney basically with a kolla build overried file you would be patching in the versioning that libvirt should have done
21:10:28 sean-k-mooney rouk: 3 is we figure out how to do this in code which could take a little while to do
21:10:44 sean-k-mooney rouk: have you filed a bug?
21:11:14 sean-k-mooney rouk: its not really a nova bug but we could workaround it i think
21:13:09 sean-k-mooney we would need to create a functional repoducer first as while i undersatd why this happended its really not due to an external change that we cant contol
21:14:07 rouk i havnt filed a bug, no.
21:14:33 rouk but yeah, ill build a nova-libvirt with another model.
21:15:55 sean-k-mooney actully i think there is something else you could try
21:16:42 sean-k-mooney so they added teh "fix" for this to https://github.com/qemu/qemu/blob/1b507e55f8199eaad99744613823f6929e4d57c6/hw/i386/pc.c#L125-L147
21:16:59 rouk yeah whats this compat, i googled it on friday

Earlier   Later