| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-03-22 | |||
| 20:52:14 | rouk | whats the most elegant way to add missing as disabled on migrate? | |
| 20:53:23 | rouk | or should i be forking qemu till i can get vms onto new features organically? | |
| 20:53:25 | rouk | heh | |
| 20:53:35 | rouk | id... rather not do that. | |
| 20:53:35 | sean-k-mooney | so the current failure is here https://github.com/openstack/nova/blob/3de7fb7c327db348d04d15d4cd3c4f811a336126/nova/virt/libvirt/driver.py#L8702-L8706 i ruled out the other code path | |
| 20:53:52 | sean-k-mooney | rouk: you dont need to fork qemu | |
| 20:54:47 | sean-k-mooney | for testing you could comment out both checks on the dest host | |
| 20:55:13 | rouk | but wouldnt libvirt then add the feature on move? | |
| 20:55:58 | sean-k-mooney | am well the xml we provide wont reference them | |
| 20:56:10 | sean-k-mooney | but yes your right it might do it implcitly | |
| 20:56:20 | rouk | i can try, if you think its a worthy test | |
| 20:57:12 | sean-k-mooney | i think what will happne is the libvirt error will go away but you might get a qemu error when we actully call migrate | |
| 20:57:37 | rouk | probably, if nova isnt the one adding these in the first place. | |
| 20:57:42 | sean-k-mooney | what i was thinking was we could modify the migrate xml to remove/add the cpu feature based on the config | |
| 20:57:56 | sean-k-mooney | rouk: basically what you were orginally trying to do | |
| 20:58:01 | sean-k-mooney | but also on migrate | |
| 20:58:09 | sean-k-mooney | the orginal patch only did it on spawn | |
| 20:58:19 | rouk | yeah, but wont that be a problem for people in other cases? | |
| 20:58:42 | rouk | for me, sure, it fixes my problem | |
| 20:58:49 | sean-k-mooney | yep im sure ill get a downstream bug for this ill have to help fix in a rush | |
| 20:59:02 | sean-k-mooney | im tyrin g to think through 2 things currently | |
| 20:59:12 | sean-k-mooney | what would not be a horrible hack for you | |
| 20:59:33 | sean-k-mooney | and waht we could do to workaround the libvirt/qemu abi break more generally | |
| 21:00:07 | rouk | well, more generally, cant we just edit the migration xml on compare error, we have the missing features, and we know which direction the failure is. | |
| 21:00:40 | rouk | if the vm is at fault, and its a new feature on the host, disable it, it will get enabled next reboot. | |
| 21:00:41 | sean-k-mooney | maybe we are also currently reqorking how we do the cpu compare | |
| 21:01:03 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/762330 | |
| 21:01:05 | rouk | i might need a horrible hack though, depending on how long the fix will take. | |
| 21:01:08 | sean-k-mooney | although i dont think that will fix it | |
| 21:01:21 | rouk | every day this sits, new vms come up depending on new features, and old vms are stranded. | |
| 21:01:33 | rouk | got hosts with broken NICs i cant evict, heh | |
| 21:01:37 | rouk | thanks supermicro | |
| 21:02:45 | sean-k-mooney | ya so this will only affect exsiting instance now that you have updated all the contianers | |
| 21:02:46 | rouk | if i didnt have like 1/3rd of my capacity having nics all blow up at once. | |
| 21:03:07 | sean-k-mooney | so option 1 is cold migration or hardreboot + libve migration | |
| 21:03:08 | rouk | i would hardly mind this cpu change, cause id just uh... wait till everyone reboots. | |
| 21:03:14 | sean-k-mooney | not grate but it would work | |
| 21:03:25 | rouk | yeah, its about 1500 VMs to reboot. | |
| 21:03:35 | rouk | and ill get quite the tomatoes thrown at me | |
| 21:03:37 | sean-k-mooney | option 2 patch the cpu model xml and to use the old values | |
| 21:03:45 | rouk | its not in the xml. | |
| 21:04:01 | rouk | its added higher up, in qemu cpu.c | |
| 21:04:02 | sean-k-mooney | no one sec | |
| 21:04:18 | sean-k-mooney | i mean /usr/share/libvirt/cpu_map/x86_EPYC-IBPB.xml | |
| 21:04:24 | rouk | yeah, it doesnt mention these features. | |
| 21:04:37 | sean-k-mooney | yep you could add them and set them disabled | |
| 21:04:43 | rouk | ah | |
| 21:04:54 | rouk | didnt see an arg for disabled on any of the existing lines. | |
| 21:04:59 | sean-k-mooney | then make a copy of the file as normal with a new name | |
| 21:05:05 | sean-k-mooney | and update the nova.conf to use that for new vms | |
| 21:05:21 | rouk | will nova know to use the old name on migrate? | |
| 21:05:36 | sean-k-mooney | yes because that is in the xml which we dont update | |
| 21:05:47 | rouk | ah yeah | |
| 21:06:13 | rouk | its quite the hack, but... i can do it pretty trivially. | |
| 21:06:28 | rouk | just make it part of my nova build. | |
| 21:06:47 | sean-k-mooney | ya so basically mv <file>.xml <file>_v2.xml | |
| 21:06:57 | sean-k-mooney | well cp | |
| 21:07:04 | sean-k-mooney | and then edit <file.xml> | |
| 21:07:16 | sean-k-mooney | you could do that as a layer in the nova-libvirt container | |
| 21:08:02 | sean-k-mooney | if you want new vms to use the new defintion just set cpu_model=eypc-ibpb-v2 | |
| 21:08:17 | sean-k-mooney | new vms will get that but migrated ones wont until you hard reboot | |
| 21:08:55 | sean-k-mooney | basically with a kolla build overried file you would be patching in the versioning that libvirt should have done | |
| 21:10:28 | sean-k-mooney | rouk: 3 is we figure out how to do this in code which could take a little while to do | |
| 21:10:44 | sean-k-mooney | rouk: have you filed a bug? | |
| 21:11:14 | sean-k-mooney | rouk: its not really a nova bug but we could workaround it i think | |
| 21:13:09 | sean-k-mooney | we would need to create a functional repoducer first as while i undersatd why this happended its really not due to an external change that we cant contol | |
| 21:14:07 | rouk | i havnt filed a bug, no. | |
| 21:14:33 | rouk | but yeah, ill build a nova-libvirt with another model. | |
| 21:15:55 | sean-k-mooney | actully i think there is something else you could try | |
| 21:16:42 | sean-k-mooney | so they added teh "fix" for this to https://github.com/qemu/qemu/blob/1b507e55f8199eaad99744613823f6929e4d57c6/hw/i386/pc.c#L125-L147 | |
| 21:16:59 | rouk | yeah whats this compat, i googled it on friday | |
| 21:17:01 | sean-k-mooney | you might be able to use a versioned machinve type for this | |
| 21:17:17 | rouk | i couldnt find any docs for how these compat versions work. | |
| 21:17:23 | sean-k-mooney | so i have nver looked at this before but i think it might eb related to machine tyeps | |
| 21:18:38 | rouk | i just have no idea what arg or config i need to do to activate this 3.1 compat | |
| 21:20:36 | rouk | i cant find a single note of documentation on it | |
| 21:21:49 | sean-k-mooney | so these are the machine type i have on centos they are disto specific | |
| 21:21:51 | sean-k-mooney | http://paste.openstack.org/show/803804/ | |
| 21:22:04 | sean-k-mooney | im going to check a ubuntu vm | |
| 21:22:25 | rouk | oh... that notation, i saw some stuff for that in virt-manager before. | |
| 21:22:33 | rouk | wonder if it has it listed | |
| 21:23:44 | sean-k-mooney | http://paste.openstack.org/show/803805/ | |
| 21:23:53 | sean-k-mooney | that is ubuntu 20.04 | |
| 21:24:34 | sean-k-mooney | i wonder if you can use pc-i440fx-3.1 | |
| 21:24:38 | rouk | pc-i440fx-3.1 Standard PC (i440FX + PIIX, 1996) | |
| 21:24:40 | rouk | yeah | |
| 21:24:54 | rouk | can nova set machine type... sec | |
| 21:25:01 | sean-k-mooney | yep | |
| 21:25:19 | sean-k-mooney | 1 of two ways. in the image which wont help and in the nova.conf which should | |
| 21:25:31 | rouk | yeah, libvirt.hw_machine_type | |
| 21:25:37 | rouk | can do that, that sounds a lot cleaner. | |
| 21:25:39 | sean-k-mooney | https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.hw_machine_type | |
| 21:25:59 | sean-k-mooney | yep so add x86_64=pc-i440fx-3.1 | |
| 21:26:03 | rouk | ill get that tested after some food, been at this issue most of the day, heh. | |
| 21:26:55 | rouk | thanks for all the help, even though its not nova's fault, at least the knowledge that this dumb editing can happen from qemu means nova can defend against it. | |
| 21:27:31 | sean-k-mooney | ya. if using the version machine type fixes it then great | |
| 21:27:52 | rouk | maybe that needs to be a default thing thats recommended to set, idk | |
| 21:27:53 | sean-k-mooney | that would at least give use time to think about what to do in nova if anything in the futrue | |
| 21:28:06 | sean-k-mooney | well there are two camps | |
| 21:28:17 | sean-k-mooney | one camp is alwasy uses the versioned machine type | |