| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-02-22 | |||
| 08:36:54 | opendevreview | melanie witt proposed openstack/nova master: Support encrypted backing files for qcow2 https://review.opendev.org/c/openstack/nova/+/907961 | |
| 08:36:54 | opendevreview | melanie witt proposed openstack/nova master: Support cross cell resize with ephemeral encryption for qcow2 https://review.opendev.org/c/openstack/nova/+/909595 | |
| 08:36:55 | opendevreview | melanie witt proposed openstack/nova master: libvirt: Introduce support for raw with LUKS https://review.opendev.org/c/openstack/nova/+/884313 | |
| 08:36:55 | opendevreview | melanie witt proposed openstack/nova master: libvirt: Introduce support for rbd with LUKS https://review.opendev.org/c/openstack/nova/+/889912 | |
| 09:17:07 | noonedeadpunk | hey folks. I guess I finally realized what is the usecase of root volume detach that was bothering me and which raised discussions here and there. And that is - volume resize. As in order to resize the disk, it must be in detached state. And you can't detach root volume. | |
| 09:38:44 | tkajinam | bauzas, hi. I just noticed I made a mistake in os-vif patch while discussion in another project and submitted a fix for it. I wonder we can merge this before we create os-vif release for caracal... https://review.opendev.org/c/openstack/os-vif/+/909682 | |
| 09:39:04 | bauzas | tkajinam: I'm pretty busy those days, but I can take a look | |
| 09:41:23 | tkajinam | bauzas, thanks and sorry for bothering you. I've left a same comment in a release patch for os-vif... | |
| 09:57:18 | bauzas | tkajinam: that's cool, I'll review it today | |
| 10:05:04 | sean-k-mooney[m] | gibi: i think we can fix the edge cases and backport but we might want to keep a revirt ready incase we need it. my inclinations is it would be ok to have as a known issue in the context of a beta release but if we cant backport the bug fixes in time we might revert for ga | |
| 10:08:00 | sean-k-mooney[m] | tkajinam: i assume you would also like https://review.opendev.org/c/openstack/os-vif/+/909341 | |
| 10:09:33 | sean-k-mooney[m] | bauzas: im +2 on both the os vif changes so let include them in the os-vif release | |
| 10:12:02 | sean-k-mooney[m] | @gibi the isolate issue in particalar is a low probability of impacting people, the general live migration issue however is much more impactful. so we should try and land rene’s fix for that. | |
| 10:17:17 | sean-k-mooney[m] | gibi: looking at the second poc patch it looks like its also affecting live migration with pinned cpus | |
| 10:17:50 | sean-k-mooney[m] | that is very surprising since i tought bauzas had tested that and it was covered | |
| 10:18:39 | bauzas | I'm on a meeting | |
| 10:19:05 | bauzas | and I miss context, what are you guys discussing ? | |
| 10:34:22 | opendevreview | Doug Szumski proposed openstack/nova master: Revert "[libvirt] Live migration fails when config_drive_format=iso9660" https://review.opendev.org/c/openstack/nova/+/909122 | |
| 10:34:37 | gibi | sean-k-mooney[m]: OK i will open a revert with a hold | |
| 10:36:50 | gibi | bauzas: yet another bug in the power mgmt feature | |
| 10:37:24 | bauzas | which is ? | |
| 10:38:06 | opendevreview | Doug Szumski proposed openstack/nova master: Revert "[libvirt] Live migration fails when config_drive_format=iso9660" https://review.opendev.org/c/openstack/nova/+/909122 | |
| 10:39:25 | sean-k-mooney[m] | bauzas: there are two, livemigation is broken and emultor thread policy=isolate | |
| 10:39:52 | bauzas | ack, any bug report to provide me ? | |
| 10:39:58 | gibi | sean-k-mooney[m]: the liv migration one is not specific to power mgmt isnt it? | |
| 10:40:15 | bauzas | sean-k-mooney: fwiw, I haven't tested live-migration | |
| 10:40:29 | sean-k-mooney[m] | i tought you had | |
| 10:40:37 | sean-k-mooney[m] | but that explains why it was missed | |
| 10:41:10 | sean-k-mooney[m] | there is a spereate live migration issue that is unrelated to power management | |
| 10:41:26 | sean-k-mooney[m] | but there is also a power management live migration bug | |
| 10:41:37 | sean-k-mooney[m] | we are not turning on the cores as part of pre live migrate | |
| 10:41:55 | sean-k-mooney[m] | so thats just actully broken | |
| 10:42:22 | sean-k-mooney[m] | the related issue is when cpu_shared_set is used and we dont have a numa topology | |
| 10:42:31 | sean-k-mooney[m] | we dont update the cores of the vm for the desination | |
| 10:42:39 | sean-k-mooney[m] | we have code up for review to fix that | |
| 10:43:03 | sean-k-mooney[m] | that means when combined with mixed shared and dedicated cpus on the same host | |
| 10:43:20 | sean-k-mooney[m] | its possible ot migrate a floating instnace to a core that is offline | |
| 10:43:43 | sean-k-mooney[m] | if the cpu_shared_set and cpu_dedicated_set are not the same on all hosts | |
| 10:44:04 | sean-k-mooney[m] | or rather the source and dest host | |
| 10:44:57 | sean-k-mooney[m] | gibi: the more i think about it the more im leaning towards truning it off by default again and trying to renable it after beta instead | |
| 10:45:00 | bauzas | I think I checked that we were turning the cores on on the right internal method that's called for *any* guest start | |
| 10:45:16 | bauzas | but maybe live-migration uses another path | |
| 10:45:28 | sean-k-mooney[m] | bauzas: the vm on the dest is started by libvirt not nova | |
| 10:45:39 | bauzas | and then paused, right? | |
| 10:46:32 | sean-k-mooney[m] | correct its started in the paused state and then libvirt unpauases it when it when we swap form executing on the source to the dest | |
| 10:46:45 | sean-k-mooney[m] | so we need to power them on in pre-livemigration | |
| 10:46:56 | gibi | sean-k-mooney[m]: ack, we can land the revert and I reopen the tracker Jira | |
| 10:47:17 | bauzas | sean-k-mooney: I see then | |
| 10:47:20 | sean-k-mooney[m] | on the plus side whitebox caught htis | |
| 10:47:47 | bauzas | when is the target guest defined and paused ? | |
| 10:47:50 | sean-k-mooney[m] | on the down side we are not yet running whitebox on nova | |
| 10:48:00 | bauzas | I thought it was when we were calling qemu migrate | |
| 10:48:06 | gibi | would be nice to get bug reports i can link to. | |
| 10:48:22 | sean-k-mooney[m] | bauzas: its defiend and paused by libvirt when we call migrateToUri3 | |
| 10:48:44 | bauzas | which is not done in pre-livemigrate | |
| 10:48:52 | sean-k-mooney[m] | no | |
| 10:48:54 | bauzas | but I see the reason on turning on the core before | |
| 10:49:00 | bauzas | then in premigrate | |
| 10:49:02 | bauzas | gotcha | |
| 10:49:03 | sean-k-mooney[m] | its done after pre-livemigate | |
| 10:49:28 | sean-k-mooney[m] | but its the call to libvirt on the source host that cause libvirt to create the vm in the paused sate on the dest | |
| 10:49:48 | bauzas | yeah, same point than for the mdev things we discussed :) | |
| 10:50:06 | bauzas | bad news is that internals of live-migration was blind for me in Antelope timle | |
| 10:50:11 | bauzas | now, this is no longer the case | |
| 10:50:13 | sean-k-mooney[m] | yes its the same code paths as the mdevs | |
| 10:50:37 | gibi | https://github.com/openstack-k8s-operators/nova-operator/pull/702 revert is up | |
| 10:51:02 | sean-k-mooney[m] | by the way nova-next is blocked | |
| 10:51:09 | sean-k-mooney[m] | im going to fix that shortly | |
| 10:51:26 | sean-k-mooney[m] | we are messing with the port bidnign profile for some reason in the post hook | |
| 10:51:29 | sean-k-mooney[m] | we should not be doing that | |
| 10:54:15 | sean-k-mooney[m] | gibi: approved. im going to grab coffee and ill work on unblocking the gate when i get back | |
| 10:54:48 | opendevreview | Doug Szumski proposed openstack/nova master: Revert "[libvirt] Live migration fails when config_drive_format=iso9660" https://review.opendev.org/c/openstack/nova/+/909122 | |
| 11:04:57 | bauzas | sean-k-mooney: gibi: we could add a note in our upstream doc saying we don't support live-migration yet due to a bug | |
| 11:19:03 | gibi | I would rather fix the bugs instead | |
| 11:21:25 | sean-k-mooney | bauzas: same im not ok with doc fixes like that in general i would prefer an api block or an actual fix | |
| 11:21:47 | bauzas | cool then | |
| 11:22:16 | sean-k-mooney | bauzas: artom has started workign on a fix | |
| 11:22:26 | bauzas | gtk | |
| 11:22:38 | sean-k-mooney | so lets just help him do that and we can file an upstream bug to track it properly | |
| 12:11:08 | opendevreview | sean mooney proposed openstack/nova master: [S-RBAC] adapt nova-next for port's binding:profile field change https://review.opendev.org/c/openstack/nova/+/909859 | |
| 12:11:18 | sean-k-mooney | bauzas: gibi ^ that will fix the gate blocker | |
| 12:29:39 | opendevreview | Merged openstack/os-vif master: Drop wrong stacklevel https://review.opendev.org/c/openstack/os-vif/+/909682 | |
| 12:30:45 | gibi | +2 with some followup thinking | |
| 12:31:56 | artom | gibi, sean-k-mooney, there's still something weird going on, https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/909785 isn't fully passing | |
| 12:32:14 | artom | It's _better_ with the two POC patches and Uggla's cpu_shared_set live migration patches, but still red | |
| 12:55:21 | sean-k-mooney | gibi: we can likely refactor this to use python to set the extra atributes on the port | |
| 12:55:29 | sean-k-mooney | just not right now | |
| 12:56:21 | sean-k-mooney | we can however just drop a python file into the hooks dir that import novas service user stuff and uses nova's code to set the my_key data | |
| 13:04:57 | gibi | sean-k-mooney: ack, i definitely wont block on this | |
| 13:07:52 | sean-k-mooney | i belvie we have unit tests that make sure we dont clobber but i agree the integration test was nice ot have | |
| 13:08:20 | gibi | artom: Feb 22 03:45:17.085381 np0036833327 nova-compute[103912]: ERROR oslo_messaging.rpc.server NotImplementedError: Cannot load 'emulator_pins' in the base class | |
| 13:08:53 | sean-k-mooney | ya so that depending on a patch that is not in the patch chain i belive | |
| 13:09:11 | sean-k-mooney | gibi: i think the whitebox job is using two depend on again nova | |
| 13:09:51 | sean-k-mooney | https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/909785 | |
| 13:09:53 | sean-k-mooney | yep | |
| 13:10:33 | sean-k-mooney | or not | |
| 13:10:47 | gibi | it seem the numa_info has no emulator_pin field | |
| 13:10:51 | sean-k-mooney | i tought it might be in https://review.opendev.org/c/openstack/nova/+/877773/8 | |
| 13:11:36 | gibi | a functional test in that patch probably can reproduce this issue | |
| 13:11:44 | sean-k-mooney | the emulator thread are ment to be included in cpu_pins | |