| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-03-22 | |||
| 20:05:23 | rouk | "very old vms" is pre ussuri | |
| 20:05:35 | rouk | "somewhat old" is post ussuri, pre update. | |
| 20:05:44 | sean-k-mooney | ok | |
| 20:06:25 | rouk | all of ussuri had libvirt 4.0+, which is where the features got retroactively added | |
| 20:06:43 | rouk | unless kolla changed base distro... which is... well i can check. | |
| 20:07:28 | rouk | dang, i only have newer versions running, id have to check if kolla has a history somewhere. | |
| 20:08:01 | sean-k-mooney | kolla has different images per disto | |
| 20:08:18 | sean-k-mooney | are you on centos or ubuntu or debian | |
| 20:08:30 | sean-k-mooney | it wont chagne the os version witin a release | |
| 20:08:31 | rouk | we are on ubuntu, which is based on 18.04 right now for ussuri, havnt jumped version. | |
| 20:08:55 | sean-k-mooney | so one thing i notice is <cpu mode='custom' match='exact' check='full'> | |
| 20:09:05 | sean-k-mooney | nova does not sett check=full | |
| 20:09:22 | sean-k-mooney | that might be part of the issue here | |
| 20:09:26 | rouk | i tried to find any code in libvirt or nova for that, and couldnt find either. im... not sure what changed. | |
| 20:09:28 | rouk | it used to be partial. | |
| 20:09:51 | sean-k-mooney | this is a default change in libvirt | |
| 20:10:00 | rouk | and i confirmed, vms that used partial, do migrate | |
| 20:10:05 | rouk | even if theyre ancient | |
| 20:10:21 | sean-k-mooney | so libvirt is from the ubuntu cloud archive in your case | |
| 20:10:41 | sean-k-mooney | kolla just uses the libvirt form uca for the given version | |
| 20:11:16 | rouk | i dont think ubuntu tampered with libvirt defaults... but ive seen worse, but also that would be a very strange mid-release change... | |
| 20:12:30 | rouk | so, what i dont get is i merged e0a8cd7ca8a907a3d178c759212e1685d0fa35c6 and c60f4df8b19b75c8c98ec570b2a506aece9a5a34 | |
| 20:12:52 | rouk | and backported them correctly, to try and just -npt to solve it, and migration still adds it. | |
| 20:13:04 | rouk | even though new vms properly have it explicitly disabled. | |
| 20:13:48 | rouk | im not sure what i should be doing? other than a reboot, what can i do? custom libvirt config override? this issue will affect more than me. | |
| 20:14:07 | sean-k-mooney | can you do "virsh dumpxml <vm> --update-cpu --migratable" on one of the instances that cannot live migrate | |
| 20:14:37 | rouk | i have played with that, but yes, i can get one i havnt touched and get you the before/after etc, sec. | |
| 20:15:21 | sean-k-mooney | rouk: so we get the xml form libvirt similar to ^ | |
| 20:15:33 | sean-k-mooney | but we do not update the cpu flags part | |
| 20:15:52 | sean-k-mooney | im wondiering if we are getting those flags form libvirt in that call | |
| 20:16:09 | sean-k-mooney | the patch that we have for removing cpu flag only takes effect for new instances | |
| 20:16:35 | rouk | one way to find out, when i saw the commit from qemu that retroactively changed epyc/opteron i just sighed, cause now we got vms stuck with old and new. | |
| 20:16:48 | rouk | took a couple weeks after rollout to notice. | |
| 20:16:58 | sean-k-mooney | ya | |
| 20:17:06 | sean-k-mooney | they should have versioned the model definiton | |
| 20:17:19 | sean-k-mooney | they should never update them in place | |
| 20:17:22 | rouk | yep. | |
| 20:19:30 | rouk | http://paste.openstack.org/show/5ozPeL4s49EoSssXRvK7/ | |
| 20:19:40 | rouk | not the result i expected | |
| 20:20:16 | sean-k-mooney | thats similar to what i expected | |
| 20:20:31 | rouk | so wheres it being added if thats nova's starting point? | |
| 20:20:35 | sean-k-mooney | --update-cpu should only change thigns for host model | |
| 20:20:57 | sean-k-mooney | possible form the cpu basline check on the dest | |
| 20:21:45 | rouk | is there any way i can trick nova into not seeing these new features somewhere? | |
| 20:22:22 | rouk | or any other output you want | |
| 20:24:49 | sean-k-mooney | the only way to trick it would be to copy the file and revert the change | |
| 20:25:13 | sean-k-mooney | im trying to get virsh cpu-baseline to work | |
| 20:25:37 | rouk | the edits are in C, so it would be a recompile to fix it seems. | |
| 20:26:26 | rouk | but, the only way this could be responsible is if ubuntu jumped from 3.x to 4.x, as this qemu change is only in 4.0+ | |
| 20:26:28 | sean-k-mooney | no you would jst need to edit the files in /usr/share/libvirt/cpu_maps/*.xml | |
| 20:26:44 | sean-k-mooney | /usr/share/libvirt/cpu_map/x86_EPYC-IBPB.xml in your case | |
| 20:26:49 | rouk | https://patchwork.kernel.org/project/qemu-devel/patch/20190121155051.5628-1-vkuznets@redhat.com/ these qemu changes arent related? | |
| 20:27:33 | rouk | nrip nor npt are in my xml | |
| 20:27:42 | rouk | its not part of the qemu cpu_map | |
| 20:27:53 | rouk | s/qemu/libvirt | |
| 20:28:26 | rouk | its being added higher up, in qemu itself? | |
| 20:28:43 | sean-k-mooney | i think what we do is basically http://paste.openstack.org/show/803802/ | |
| 20:29:19 | sean-k-mooney | or rather what libvirt does and then its comparing the baseline cpus between the source and dest host | |
| 20:30:11 | rouk | it is present there, yeah. | |
| 20:30:51 | sean-k-mooney | could you try "virsh capabilities > /tmp/caps.xml ; virsh cpu-baseline /tmp/caps.xml --migratable --features; rm -f /tmp/caps.xml" | |
| 20:31:22 | rouk | <feature policy='require' name='nrip-save'/> | |
| 20:31:24 | rouk | yeah its there. | |
| 20:31:43 | sean-k-mooney | so that is where its coming from | |
| 20:31:56 | sean-k-mooney | let me see if i can find the nova code | |
| 20:32:04 | sean-k-mooney | so this is not form the xml update | |
| 20:32:13 | sean-k-mooney | its form the eariler cpu compatiablity check | |
| 20:32:26 | sean-k-mooney | in pre livemigrate i think | |
| 20:32:28 | rouk | :( and i hoped backporting those cpu feature changes would help me, heh. | |
| 20:35:15 | sean-k-mooney | this is where its failing https://github.com/openstack/nova/blob/3de7fb7c327db348d04d15d4cd3c4f811a336126/nova/virt/libvirt/driver.py#L8991-L9060 | |
| 20:36:18 | sean-k-mooney | we ask libvirt if the xml is comparitble https://github.com/openstack/nova/blob/3de7fb7c327db348d04d15d4cd3c4f811a336126/nova/virt/libvirt/host.py#L1424 | |
| 20:36:20 | rouk | a migration should survive adding features no? could it be made soft/warn for new features? | |
| 20:36:43 | sean-k-mooney | rouk: no the cpu flag cannot have addtion or removales in a live migration | |
| 20:37:23 | rouk | alright. then new features could be trimmed off? | |
| 20:37:51 | rouk | new features on migrate shouldnt happen... even if its qemu being dumb and retroactively changing things. | |
| 20:38:26 | sean-k-mooney | am yes but im trying to see how nova is in this state | |
| 20:38:45 | sean-k-mooney | we can generate the cpu xml we pass in 3 ways | |
| 20:38:52 | sean-k-mooney | https://github.com/openstack/nova/blob/3de7fb7c327db348d04d15d4cd3c4f811a336126/nova/virt/libvirt/driver.py#L9017-L9032 | |
| 20:41:12 | sean-k-mooney | its only caled in 2 places https://github.com/openstack/nova/blob/3de7fb7c327db348d04d15d4cd3c4f811a336126/nova/virt/libvirt/driver.py#L8701-L8708 and here https://github.com/openstack/nova/blob/3de7fb7c327db348d04d15d4cd3c4f811a336126/nova/virt/libvirt/driver.py#L872-L888 | |
| 20:42:37 | rouk | why would the vm be checked against target host features? if the vm doesnt have the feature, it shouldnt be checked on the vm side... | |
| 20:43:06 | rouk | nova doesnt care if the vm is missing a feature the host has | |
| 20:43:40 | rouk | if it fits within the features of the new host, it should pass. | |
| 20:43:46 | rouk | which, it does. | |
| 20:45:58 | rouk | if i was to change some of these checks to only care about the host having what the vm has, itd work, no? | |
| 20:46:11 | rouk | or would it migrate with host features and die | |
| 20:46:19 | sean-k-mooney | so nova does need to check the that host has all feature that the vm uses | |
| 20:46:30 | sean-k-mooney | you are right ti does not care about ones that are disabled | |
| 20:46:49 | sean-k-mooney | so nova should ignore onces that are disabeld | |
| 20:46:55 | rouk | yeah, vm needs to fit inside the host, not the other side. | |
| 20:47:29 | sean-k-mooney | right so libvirt did not previoulsy have disabeld feature we had to ignore until recenlty | |
| 20:47:35 | sean-k-mooney | and nova did not provide a way to disable them | |
| 20:48:07 | rouk | if these checks pass, will it migrate with the bad config, or the current vm config (which will work)? | |
| 20:48:25 | rouk | is it as simple as making these checks looser on the vm side? | |
| 20:48:28 | sean-k-mooney | it will migrate with teh current config | |
| 20:48:34 | sean-k-mooney | which should work | |
| 20:48:37 | rouk | yeah | |
| 20:48:41 | sean-k-mooney | well | |
| 20:48:46 | sean-k-mooney | actully not nessisarly | |
| 20:49:04 | sean-k-mooney | so hte issue we have is that how migration work is actully different then most people think | |
| 20:49:28 | sean-k-mooney | libvirt on the source house ask libvirt on the dest host to spawn a qemu instance using an xml we provide | |
| 20:49:28 | rouk | i got confused once i had to patch nova-ssh :p | |