Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-03-22
20:51:56 rouk yeah, thats what i tried to do with those patches, cause when i manually tested migration, i could make it move by disabling features on the target
20:52:14 rouk whats the most elegant way to add missing as disabled on migrate?
20:53:23 rouk or should i be forking qemu till i can get vms onto new features organically?
20:53:25 rouk heh
20:53:35 sean-k-mooney so the current failure is here https://github.com/openstack/nova/blob/3de7fb7c327db348d04d15d4cd3c4f811a336126/nova/virt/libvirt/driver.py#L8702-L8706 i ruled out the other code path
20:53:35 rouk id... rather not do that.
20:53:52 sean-k-mooney rouk: you dont need to fork qemu
20:54:47 sean-k-mooney for testing you could comment out both checks on the dest host
20:55:13 rouk but wouldnt libvirt then add the feature on move?
20:55:58 sean-k-mooney am well the xml we provide wont reference them
20:56:10 sean-k-mooney but yes your right it might do it implcitly
20:56:20 rouk i can try, if you think its a worthy test
20:57:12 sean-k-mooney i think what will happne is the libvirt error will go away but you might get a qemu error when we actully call migrate
20:57:37 rouk probably, if nova isnt the one adding these in the first place.
20:57:42 sean-k-mooney what i was thinking was we could modify the migrate xml to remove/add the cpu feature based on the config
20:57:56 sean-k-mooney rouk: basically what you were orginally trying to do
20:58:01 sean-k-mooney but also on migrate
20:58:09 sean-k-mooney the orginal patch only did it on spawn
20:58:19 rouk yeah, but wont that be a problem for people in other cases?
20:58:42 rouk for me, sure, it fixes my problem
20:58:49 sean-k-mooney yep im sure ill get a downstream bug for this ill have to help fix in a rush
20:59:02 sean-k-mooney im tyrin g to think through 2 things currently
20:59:12 sean-k-mooney what would not be a horrible hack for you
20:59:33 sean-k-mooney and waht we could do to workaround the libvirt/qemu abi break more generally
21:00:07 rouk well, more generally, cant we just edit the migration xml on compare error, we have the missing features, and we know which direction the failure is.
21:00:40 rouk if the vm is at fault, and its a new feature on the host, disable it, it will get enabled next reboot.
21:00:41 sean-k-mooney maybe we are also currently reqorking how we do the cpu compare
21:01:03 sean-k-mooney https://review.opendev.org/c/openstack/nova/+/762330
21:01:05 rouk i might need a horrible hack though, depending on how long the fix will take.
21:01:08 sean-k-mooney although i dont think that will fix it
21:01:21 rouk every day this sits, new vms come up depending on new features, and old vms are stranded.
21:01:33 rouk got hosts with broken NICs i cant evict, heh
21:01:37 rouk thanks supermicro
21:02:45 sean-k-mooney ya so this will only affect exsiting instance now that you have updated all the contianers
21:02:46 rouk if i didnt have like 1/3rd of my capacity having nics all blow up at once.
21:03:07 sean-k-mooney so option 1 is cold migration or hardreboot + libve migration
21:03:08 rouk i would hardly mind this cpu change, cause id just uh... wait till everyone reboots.
21:03:14 sean-k-mooney not grate but it would work
21:03:25 rouk yeah, its about 1500 VMs to reboot.
21:03:35 rouk and ill get quite the tomatoes thrown at me
21:03:37 sean-k-mooney option 2 patch the cpu model xml and to use the old values
21:03:45 rouk its not in the xml.
21:04:01 rouk its added higher up, in qemu cpu.c
21:04:02 sean-k-mooney no one sec
21:04:18 sean-k-mooney i mean /usr/share/libvirt/cpu_map/x86_EPYC-IBPB.xml
21:04:24 rouk yeah, it doesnt mention these features.
21:04:37 sean-k-mooney yep you could add them and set them disabled
21:04:43 rouk ah
21:04:54 rouk didnt see an arg for disabled on any of the existing lines.
21:04:59 sean-k-mooney then make a copy of the file as normal with a new name
21:05:05 sean-k-mooney and update the nova.conf to use that for new vms
21:05:21 rouk will nova know to use the old name on migrate?
21:05:36 sean-k-mooney yes because that is in the xml which we dont update
21:05:47 rouk ah yeah
21:06:13 rouk its quite the hack, but... i can do it pretty trivially.
21:06:28 rouk just make it part of my nova build.
21:06:47 sean-k-mooney ya so basically mv <file>.xml <file>_v2.xml
21:06:57 sean-k-mooney well cp
21:07:04 sean-k-mooney and then edit <file.xml>
21:07:16 sean-k-mooney you could do that as a layer in the nova-libvirt container
21:08:02 sean-k-mooney if you want new vms to use the new defintion just set cpu_model=eypc-ibpb-v2
21:08:17 sean-k-mooney new vms will get that but migrated ones wont until you hard reboot
21:08:55 sean-k-mooney basically with a kolla build overried file you would be patching in the versioning that libvirt should have done
21:10:28 sean-k-mooney rouk: 3 is we figure out how to do this in code which could take a little while to do
21:10:44 sean-k-mooney rouk: have you filed a bug?
21:11:14 sean-k-mooney rouk: its not really a nova bug but we could workaround it i think
21:13:09 sean-k-mooney we would need to create a functional repoducer first as while i undersatd why this happended its really not due to an external change that we cant contol
21:14:07 rouk i havnt filed a bug, no.
21:14:33 rouk but yeah, ill build a nova-libvirt with another model.
21:15:55 sean-k-mooney actully i think there is something else you could try
21:16:42 sean-k-mooney so they added teh "fix" for this to https://github.com/qemu/qemu/blob/1b507e55f8199eaad99744613823f6929e4d57c6/hw/i386/pc.c#L125-L147
21:16:59 rouk yeah whats this compat, i googled it on friday
21:17:01 sean-k-mooney you might be able to use a versioned machinve type for this
21:17:17 rouk i couldnt find any docs for how these compat versions work.
21:17:23 sean-k-mooney so i have nver looked at this before but i think it might eb related to machine tyeps
21:18:38 rouk i just have no idea what arg or config i need to do to activate this 3.1 compat
21:20:36 rouk i cant find a single note of documentation on it
21:21:49 sean-k-mooney so these are the machine type i have on centos they are disto specific
21:21:51 sean-k-mooney http://paste.openstack.org/show/803804/
21:22:04 sean-k-mooney im going to check a ubuntu vm
21:22:25 rouk oh... that notation, i saw some stuff for that in virt-manager before.
21:22:33 rouk wonder if it has it listed
21:23:44 sean-k-mooney http://paste.openstack.org/show/803805/
21:23:53 sean-k-mooney that is ubuntu 20.04
21:24:34 sean-k-mooney i wonder if you can use pc-i440fx-3.1
21:24:38 rouk pc-i440fx-3.1 Standard PC (i440FX + PIIX, 1996)
21:24:40 rouk yeah
21:24:54 rouk can nova set machine type... sec
21:25:01 sean-k-mooney yep
21:25:19 sean-k-mooney 1 of two ways. in the image which wont help and in the nova.conf which should
21:25:31 rouk yeah, libvirt.hw_machine_type
21:25:37 rouk can do that, that sounds a lot cleaner.
21:25:39 sean-k-mooney https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.hw_machine_type
21:25:59 sean-k-mooney yep so add x86_64=pc-i440fx-3.1
21:26:03 rouk ill get that tested after some food, been at this issue most of the day, heh.
21:26:55 rouk thanks for all the help, even though its not nova's fault, at least the knowledge that this dumb editing can happen from qemu means nova can defend against it.
21:27:31 sean-k-mooney ya. if using the version machine type fixes it then great
21:27:52 rouk maybe that needs to be a default thing thats recommended to set, idk
21:27:53 sean-k-mooney that would at least give use time to think about what to do in nova if anything in the futrue
21:28:06 sean-k-mooney well there are two camps

Earlier   Later