| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-08-07 | |||
| 09:39:47 | lyarwood | kashyap: it doesn't look like we stash them anywhere | |
| 09:40:06 | lyarwood | kashyap: we appear to be grabbing a fresh copy from libvirt everytime we need them | |
| 09:40:35 | kashyap | lyarwood: Hmm. Got a code pointer? | |
| 09:42:20 | lyarwood | kashyap: https://paste.centos.org/view/86fa4860 | |
| 09:44:21 | kashyap | lyarwood: Excellent! That helped | |
| 09:44:31 | kashyap | lyarwood: Context: | |
| 09:45:04 | kashyap | lyarwood: During FFU, some hosts report the CPU feature 'arch-facilities' (to make matters worse, it's called differently on EL8: 'arch-capabilities') | |
| 09:45:42 | kashyap | lyarwood: So migration from EL7 to EL8 fails (on appropriate hardware) with: "libvirt.libvirtError: internal error: Unknown CPU feature arch-facilities" | |
| 09:45:55 | kashyap | lyarwood: To 'workaround' that, we need to disable the 'arch-facilities' on the source host | |
| 09:46:02 | lyarwood | kashyap: isn't that a libvirt bug? | |
| 09:46:15 | kashyap | s/disable/comment-out | |
| 09:46:22 | lyarwood | kashyap: ah wait, is it the way we generate the xml on the src? | |
| 09:46:29 | kashyap | lyarwood: No, it's not :-( I talked w/ Jiri Denemark today | |
| 09:46:51 | kashyap | In short, we (Nova) need to improve on how we're doing CPU checks. Some checks are redundant (as libvirt does them anyway) | |
| 09:46:56 | lyarwood | kashyap: we can handle that in https://github.com/openstack/nova/blob/9ecefeb836964c52a5a2969b15c82b11c51d32ab/nova/virt/libvirt/migration.py#L56-L70 | |
| 09:47:02 | kashyap | lyarwood: Yeah, indeed. The guest XML itself doesn't have 'arch-facilities' | |
| 09:50:32 | kashyap | lyarwood: /me clicks | |
| 09:52:15 | kashyap | lyarwood: So, let me explain the sequence of the current workaround: | |
| 09:59:05 | kashyap | 1. Double-check none of the EL7 instance XML contain 'arch-facilities' CPU feature. Normally, the guests do _not_ have them. | |
| 09:59:08 | kashyap | 2. Run `virsh capabilities | grep arch-facilities` — it will be present | |
| 09:59:11 | kashyap | 3. Comment out 'arch-facilties' in /usr/share/libvirt/cpu_share.xml | |
| 09:59:14 | kashyap | 4. Restart 'libvirtd' and 'nova-compute' service (both are needed) | |
| 09:59:16 | kashyap | 5. Then run `virsh capabilities`, again: it will now be absent (expected) | |
| 09:59:19 | kashyap | 6. Now migrate the guest from EL7 to EL8: it _must_ succeed. | |
| 09:59:29 | kashyap | (The above is when migrating only from source hardware that supports 'arch-facilities' CPU feature.) | |
| 10:01:57 | lyarwood | urgh we need to fix this in Nova tbh | |
| 10:02:14 | lyarwood | we can check on the dest if arch-capabilities is set | |
| 10:02:26 | lyarwood | if it is, use it in the migration xml we send there | |
| 10:03:05 | lyarwood | if it isn't but arch-facilities is set on the src, remove that from the migration xml | |
| 10:03:12 | lyarwood | does that sound right? | |
| 10:05:40 | kashyap | lyarwood: Reading the scroll | |
| 10:05:49 | lyarwood | brb coffee | |
| 10:06:15 | kashyap | lyarwood: Right, that does sound correct. I'm gonna file a bug for it | |
| 10:06:43 | kashyap | lyarwood: Oh, but note: 'arch-facilties' is marked as *non-migratable* | |
| 10:10:35 | lyarwood | /o\ | |
| 10:10:42 | lyarwood | so should we always remove it? | |
| 10:12:50 | kashyap | lyarwood: I'd change the plan to this: just drop 'arch-facilities' and let libvirt handle it — this works, Jiri from libvirt confirms | |
| 10:13:44 | lyarwood | kashyap: ack that's simple enough then | |
| 10:15:58 | lyarwood | kashyap: as you're out on PTO next week I'd be happy to handle that once you've written up the bug | |
| 10:22:17 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Follow up for Ibf8dca4bd57b3bddb39955b53cc03564506f5754 https://review.opendev.org/745095 | |
| 10:25:37 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Pass context, instance to '_create_guest' https://review.opendev.org/741286 | |
| 10:25:37 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Add helpers for suspend, resume and reboot of server https://review.opendev.org/741285 | |
| 10:25:38 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Add emulated TPM support to Nova https://review.opendev.org/631363 | |
| 10:25:38 | openstackgerrit | Stephen Finucane proposed openstack/nova master: api: Reject non-spawn operations for vTPM https://review.opendev.org/741500 | |
| 10:25:39 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add type hints to 'nova.compute.manager' https://review.opendev.org/742863 | |
| 10:25:39 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Add docs for vTPM support https://review.opendev.org/739213 | |
| 10:25:40 | openstackgerrit | Stephen Finucane proposed openstack/nova master: privsep: Add support for recursive chown, move_tree operations https://review.opendev.org/742864 | |
| 10:25:40 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Don't unset Instance.old_flavor, new_flavor until necessary https://review.opendev.org/741995 | |
| 10:25:41 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add support for resize and cold migration of emulated TPM files https://review.opendev.org/639934 | |
| 10:25:41 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add type hints to 'nova.virt.libvirt.utils' https://review.opendev.org/742865 | |
| 10:30:58 | kashyap | lyarwood: A quick one, without checking the code, the migrate XML - is it the one we send to _compare_cpu() method or to libvirt's migrate API? | |
| 10:31:27 | kashyap | lyarwood: I'll let you know once I've got the bug up... | |
| 10:32:58 | lyarwood | kashyap: libvirt's migrate API | |
| 10:33:05 | lyarwood | kashyap: is this failing in _compare_cpu? | |
| 10:33:25 | kashyap | lyarwood: Yep, the failure is indeed from _compare_cpu() | |
| 10:34:25 | kashyap | This whole thing needs replacing; see my earlier-mentinoned spec on newer APIs. I've got a draft patch branch for it somewhere (https://opendev.org/openstack/nova-specs/commit/70811da221035044e27) | |
| 10:34:43 | kashyap | But the "full replacing" can wait. One step at a time | |
| 10:35:22 | kashyap | lyarwood: Oh, sigh; 'arch-facilities' is a EL-only thing :-( | |
| 10:35:46 | lyarwood | \o/ | |
| 10:35:47 | kashyap | lyarwood: So, I don't think it makes sense for upstream; perhaps a EL-8 downstream-specific patch is needed | |
| 10:35:51 | lyarwood | ggwp libvirt | |
| 10:36:42 | kashyap | lyarwood: Yeah, annoying: | |
| 10:36:47 | kashyap | [quote] | |
| 10:36:48 | kashyap | confusion about the correct naming. | |
| 10:36:48 | kashyap | RHEL version of QEMU. Let's update the test files to avoid possible | |
| 10:36:48 | kashyap | arch-facilities. Apparently some CPU test files were gathered with the | |
| 10:36:48 | kashyap | The kernel calls this feature arch_capabilities and RHEL/CentOS 7.* use | |
| 10:36:50 | kashyap | [/quote] | |
| 10:36:54 | kashyap | From here: https://libvirt.org/git/?p=libvirt.git;a=commit;h=511df17aec | |
| 10:36:58 | kashyap | Err, wrong link | |
| 10:37:13 | kashyap | No, that's correct. (Self, slow down) | |
| 11:52:41 | sean-k-mooney | kashyap: we do cache the host capablities i belive too. i think i added that | |
| 11:53:02 | sean-k-mooney | kashyap: but this is a downstream only bug anyway so not really an issue with caching | |
| 11:54:10 | sean-k-mooney | well i think we cache the domain caps | |
| 11:54:31 | sean-k-mooney | which is slitghly different | |
| 11:56:46 | sean-k-mooney | we cache both https://github.com/openstack/nova/blob/stable/train/nova/virt/libvirt/host.py#L693-L843 | |
| 11:57:16 | kashyap | Yeah, domain caps is different from host capabilities | |
| 11:57:29 | kashyap | 'virsh capabilities' vs. 'virsh domcapabilities' | |
| 11:57:32 | kashyap | Confusing | |
| 11:57:33 | sean-k-mooney | yep | |
| 11:57:35 | sean-k-mooney | i know | |
| 11:57:37 | sean-k-mooney | we cache both | |
| 11:59:09 | sean-k-mooney | looks like we have cached the host capabilities since danpb added the function | |
| 11:59:43 | sean-k-mooney | libvirt caches the data too in the drivers so its not going to chagne without restart the libvirt deamon anyway | |
| 12:05:44 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add new default roles in FIP policies https://review.opendev.org/742030 | |
| 12:06:36 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Pass the actual target in FIP policy https://review.opendev.org/742570 | |
| 12:33:35 | gibi | and now I clogged the gate with all the open, and now approved, policy patches | |
| 12:33:38 | gibi | happy Friday :D | |
| 12:34:16 | sean-k-mooney | :) | |
| 13:36:46 | mnaser | i'm trying to clean up our configs and start using the service catalog for auto discovery of services in nova | |
| 13:36:53 | mnaser | (to stop using glance api_servers) | |
| 13:37:11 | mnaser | "NOTE: The preferred mechanism for endpoint discovery is via keystoneauth1 loading options. Only use api_servers if you need multiple endpoints and are unable to use a load balancer for some reason." | |
| 13:37:17 | mnaser | does this mean i would fill this up like the neutron section? | |
| 13:37:24 | mnaser | or is there a common section? the docs don't seem clera there | |
| 13:40:23 | sean-k-mooney | are you using haproxy infront of glance api | |
| 13:40:39 | sean-k-mooney | or are you listing multiple glance api_servers | |
| 13:41:35 | sean-k-mooney | mnaser: you will have to fill it in if you are using nova as a loadbalncer for the glance api but ideally you would not use that and instead just deploy haproxy | |
| 13:41:57 | sean-k-mooney | wicch you proably are already doing for other services | |
| 13:43:53 | mnaser | sean-k-mooney: we already have haproxy :) i mainly just dont wanna hardcode the api_servers value and let nova do service discovery | |
| 13:45:06 | mnaser | so pretty much just want nova to use the value inside the service catalog | |