| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-03-22 | |||
| 19:07:06 | sean-k-mooney | ricolin: master is not intended to run on bionic by the way | |
| 19:07:36 | sean-k-mooney | it might be compatiable but its not part of the offical testing runtimes anymore | |
| 19:08:04 | sean-k-mooney | we have 1/2 jobs that use it for reasons but the arm jobs should be on focal | |
| 19:09:08 | sean-k-mooney | ricolin: ussuri was the last release to use bionic | |
| 19:09:41 | sean-k-mooney | all jobs for victoria wallaby and master shoudl be useding 20.04/focal if they are ubuntu based | |
| 19:12:11 | sean-k-mooney | ricolin: is there a reason the aarch64 job was using bionic | |
| 19:14:06 | sean-k-mooney | ricolin: stephenfin we have seen issue with libs incorrectly monkeypactching codecs.py in the past | |
| 19:15:35 | sean-k-mooney | this looks like we are hitting https://github.com/openstack/nova/commit/b862f6ff35d1611d0d63623a6254fc889012bfb9 | |
| 19:16:20 | sean-k-mooney | where blockdiag was messing with codecs.getreader | |
| 19:34:19 | rouk | after updating to the latest stable/ussuri nova version, live migrations are broken due to cpu features being added that shouldnt be. | |
| 19:35:04 | rouk | i have merged the patches in master which allow disabling cpu features, but even that doesnt make migrations work. | |
| 19:35:30 | rouk | even after disabling these features, live migrate fights back with libvirt.libvirtError: operation failed: guest CPU doesn't match specification: extra features: npt,nrip-save | |
| 19:36:53 | rouk | how do i stop nova from adding features on migration and breaking things? | |
| 19:40:49 | rouk | seems like a pretty big breaking change that made it into a patch... | |
| 19:42:54 | sean-k-mooney | rouk: that is not a breaking change in nova | |
| 19:43:13 | sean-k-mooney | that is a breaking change in your kernel | |
| 19:43:31 | sean-k-mooney | rouk: nova does not add those features | |
| 19:43:31 | rouk | it only appeared after patching nova a week ago to stable/ussuri... | |
| 19:43:41 | rouk | and is present on old-kernel boxes. | |
| 19:44:01 | openstackgerrit | Merged openstack/nova stable/train: Prevent archiving of pci_devices records because of 'instance_uuid' https://review.opendev.org/c/openstack/nova/+/760978 | |
| 19:44:03 | sean-k-mooney | nova does not add those features | |
| 19:44:19 | sean-k-mooney | rouk: so i think you got some other change you did not intend | |
| 19:44:44 | rouk | it never used to, no. but these new changes that came in to "fix" live migrations, are adding features | |
| 19:45:06 | sean-k-mooney | which change are you refering too | |
| 19:45:16 | rouk | sec while i grab the commit | |
| 19:49:00 | sean-k-mooney | this ? https://github.com/openstack/nova/commit/b6c473159ec45e0aa715edd45cde28f77484a5f7 | |
| 19:49:24 | rouk | there was one a bit earlier | |
| 19:49:48 | sean-k-mooney | thre is no nova code to add those extrapsec explitly | |
| 19:49:57 | sean-k-mooney | *extra features | |
| 19:50:21 | sean-k-mooney | can you share how you have configured the cpu_mode/cpu_model/extra cpu flags | |
| 19:50:28 | sean-k-mooney | in your nova.conf | |
| 19:50:57 | rouk | i have model, i added extra flags to try and fix this, as i merged support for - syntax to remove features. | |
| 19:51:09 | rouk | but, just epyc-ibpb | |
| 19:51:35 | sean-k-mooney | are all you servers amd eypc ? | |
| 19:51:38 | rouk | yep. | |
| 19:52:31 | rouk | https://patchwork.kernel.org/project/qemu-devel/patch/20190121155051.5628-1-vkuznets@redhat.com/ | |
| 19:52:44 | rouk | which, this happened a while ago, which added these as features retoactively in qemu for the model. | |
| 19:52:55 | sean-k-mooney | thise appears to be the defintion of that model | |
| 19:52:56 | sean-k-mooney | http://paste.openstack.org/show/803793/ | |
| 19:53:17 | rouk | and then this got picked up by nova, which then added these as required features on migrate | |
| 19:53:36 | sean-k-mooney | ok so this is not a nova bug so | |
| 19:53:47 | sean-k-mooney | those requiremtns are coming form libvirt | |
| 19:54:03 | rouk | so how do we stop them from retoactively being added on migrate from nova? | |
| 19:54:12 | sean-k-mooney | did you update the qemu/libvirt version when you updated ussuir | |
| 19:54:29 | sean-k-mooney | have you confrim that is what is happening | |
| 19:54:36 | sean-k-mooney | do the vms actully have them? | |
| 19:54:52 | sean-k-mooney | if hte vm was hard rebooted it could have had teh feature exposted to it | |
| 19:54:52 | rouk | we were on 4.0+ (where this qemu change happened) the whole time during ussuri, migrations only broke recently. | |
| 19:55:06 | rouk | hard rebooting the vm does fix it, by adding the feature | |
| 19:55:10 | rouk | i dont want to reboot an entire cloud. | |
| 19:55:37 | sean-k-mooney | fair but do you have the nova fix i mentioned above | |
| 19:55:51 | sean-k-mooney | https://github.com/openstack/nova/commit/b6c473159ec45e0aa715edd45cde28f77484a5f7 | |
| 19:56:05 | rouk | yes, i am on stable/ussuri as of a week ago. | |
| 19:56:11 | rouk | built from git. | |
| 19:57:24 | sean-k-mooney | ok so what it sounds like is the current vm definiton which is based on epyc-ibpb | |
| 19:57:34 | sean-k-mooney | was generated before the model was updated | |
| 19:57:59 | sean-k-mooney | and now that its migrating to the new host the info we get from the dest libvirt is expecting the new defintions | |
| 19:58:21 | sean-k-mooney | and its failing as a result of the abi break that libvirt/qemu did to the model | |
| 19:58:37 | rouk | libvirt migrations usually dont add features during migrate, usually you wait for reboot for that., | |
| 19:58:54 | sean-k-mooney | correct it cant | |
| 19:58:58 | sean-k-mooney | and i dont think nova is | |
| 19:59:08 | sean-k-mooney | i think this is happening lower down the stack | |
| 19:59:26 | rouk | how would libvirt add its own policy | |
| 19:59:33 | rouk | cpu check mode is now full, instead of partial before. | |
| 19:59:53 | rouk | so that also changed. vms before the latest patch had no feature policy mentioned, just the model | |
| 20:00:32 | rouk | here, let me paste the 3 iterations i have observed | |
| 20:00:38 | sean-k-mooney | rouk: we ask libvirt to dump the migratable xml here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/migration.py#L58 | |
| 20:01:15 | sean-k-mooney | rouk: just so you are aware the xml that libvirt uses and dispalas is not the same one that nova gives it | |
| 20:01:30 | sean-k-mooney | rouk: libvirt parses the xml we give it and add extra info to it | |
| 20:01:49 | rouk | yeah, gotta use --live, etc | |
| 20:02:29 | sean-k-mooney | am if you have nova-compute in debug mode we will print the xml that we set to libvirt | |
| 20:02:40 | sean-k-mooney | but the one that is shown in any virsh output is not the one we gave it | |
| 20:02:51 | sean-k-mooney | its the one after it updates it and files in things like guest pci devices | |
| 20:03:15 | rouk | http://paste.openstack.org/show/opnX0PSXurwehQCWsSbg/ | |
| 20:03:15 | sean-k-mooney | the migrate xml we use is one that we retive from libvirt | |
| 20:03:20 | sean-k-mooney | then we modify it | |
| 20:03:41 | sean-k-mooney | with https://github.com/openstack/nova/blob/master/nova/virt/libvirt/migration.py#L59-L69 | |
| 20:03:47 | rouk | so the --migratable doesnt have those cpu flags required. | |
| 20:03:47 | sean-k-mooney | but we dont modify the cpu flags | |
| 20:04:01 | sean-k-mooney | but are they listed | |
| 20:04:04 | rouk | nope. | |
| 20:04:08 | rouk | see paste, first block. | |
| 20:04:09 | sean-k-mooney | just looking at you link now | |
| 20:05:05 | sean-k-mooney | so looking at the paste | |
| 20:05:20 | sean-k-mooney | the very old vms are the ones booted and not rebooted since the model defineiton was updated | |
| 20:05:23 | rouk | "very old vms" is pre ussuri | |
| 20:05:35 | rouk | "somewhat old" is post ussuri, pre update. | |
| 20:05:44 | sean-k-mooney | ok | |
| 20:06:25 | rouk | all of ussuri had libvirt 4.0+, which is where the features got retroactively added | |
| 20:06:43 | rouk | unless kolla changed base distro... which is... well i can check. | |
| 20:07:28 | rouk | dang, i only have newer versions running, id have to check if kolla has a history somewhere. | |
| 20:08:01 | sean-k-mooney | kolla has different images per disto | |
| 20:08:18 | sean-k-mooney | are you on centos or ubuntu or debian | |
| 20:08:30 | sean-k-mooney | it wont chagne the os version witin a release | |
| 20:08:31 | rouk | we are on ubuntu, which is based on 18.04 right now for ussuri, havnt jumped version. | |
| 20:08:55 | sean-k-mooney | so one thing i notice is <cpu mode='custom' match='exact' check='full'> | |
| 20:09:05 | sean-k-mooney | nova does not sett check=full | |
| 20:09:22 | sean-k-mooney | that might be part of the issue here | |
| 20:09:26 | rouk | i tried to find any code in libvirt or nova for that, and couldnt find either. im... not sure what changed. | |
| 20:09:28 | rouk | it used to be partial. | |