| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-02-06 | |||
| 18:06:58 | tkajinam | sean-k-mooney, thanks ! | |
| 20:08:04 | melwitt | sean-k-mooney: this is a fix for the nova-ovs-hybrid-plug job, which is at a 100% fail rn afaict https://review.opendev.org/c/openstack/nova/+/908182 | |
| 20:14:09 | sean-k-mooney | melwitt: cool ill have a look quickly. iw as not really aware we did evacuate testing in that job | |
| 20:14:37 | sean-k-mooney | i tought i turned off cinder and swift in that job | |
| 20:14:42 | melwitt | yeah I didn't really know either | |
| 20:15:20 | sean-k-mooney | well it passed so ok +2w | |
| 20:15:53 | melwitt | yes cinder was disabled but the resource setup phase of the evacuate test stuff boots a bfv server, which fails bc no cinder endpoint. but how was it ever working before that? I don't see the reason | |
| 20:16:09 | sean-k-mooney | that job is specificaly for testing ml2/ovs so we tried to keep it small and not run things that were unrelated to the networking backend | |
| 20:16:16 | melwitt | I saw that the cinder disable patch was from september last year so I don't get it | |
| 20:16:45 | sean-k-mooney | did we recently add the evacuate hook or something | |
| 20:18:02 | melwitt | the evacuate hook has been around for a long time. and the patch that disabled cinder actually addressed some things related to evacuate https://review.opendev.org/c/openstack/nova/+/893540 | |
| 20:18:31 | sean-k-mooney | https://github.com/openstack/nova/commit/c486cc89dc11c2698cbf19fe8b2e477666331dee#diff-a27241a0d23e0378e836ec8459debe520a3ac0a18b8dbd7acc6e4cefc15ee75c | |
| 20:18:47 | melwitt | it's just it didn't guard around creation of a bfv server in setup_evacuate_resources.sh so I dunno why it didn't fail at that time, I would have thought it should have | |
| 20:18:53 | sean-k-mooney | so in the change we did to disabel cinder we tried to make it skip the bfv tests | |
| 20:19:04 | melwitt | right | |
| 20:19:49 | sean-k-mooney | ya so that should have alwasy just not work or passed | |
| 20:19:59 | sean-k-mooney | there must be somethign else happening but ok | |
| 20:22:33 | sean-k-mooney | i mean you directly modifed the failign code here but --poll on nova client is the same as --wait right | |
| 20:22:36 | sean-k-mooney | https://github.com/openstack/nova/commit/e96ac439d314358b52a17333fb13bd15a8a8e2c0 | |
| 20:23:23 | sean-k-mooney | actuly your change also merged before the cidner disabel one | |
| 20:23:48 | sean-k-mooney | melwitt: while you are around did you have time to respin the ephmeral encyption series | |
| 20:24:01 | sean-k-mooney | if so ill pull it down tomorrow and try and review it again | |
| 20:26:30 | melwitt | sean-k-mooney: I did. I'm going to respin once more today because there's something wonky with the encrypted backing files patch but it should be ready to pull down tomorrow | |
| 20:27:33 | melwitt | yeah --poll is the same as --wait in osc | |
| 20:28:23 | sean-k-mooney | cool | |
| 20:41:58 | melwitt | looking at a random passing nova-ovs-hybrid-plug run from a few days ago, it was failing to create the bfv server but it wasn't fatal for some reason https://zuul.opendev.org/t/openstack/build/6156ed6d9d1644d492300b2c33716ce2/log/job-output.txt#29530-29534 | |
| 20:44:18 | sean-k-mooney | odd im not sure why that woudl not have consitent behavior | |
| 20:44:59 | sean-k-mooney | maybe osc changerd its return code | |
| 20:45:14 | sean-k-mooney | i.e. in the past it might have returned 0 even if wait had an error | |
| 20:49:59 | melwitt | oh hm yeah | |
| 21:26:39 | opendevreview | keerthivasan proposed openstack/nova master: Adding sysctl changes for post-copy migration https://review.opendev.org/c/openstack/nova/+/908201 | |
| 21:50:33 | opendevreview | keerthivasan proposed openstack/nova master: Adding sysctl changes for post-copy migration https://review.opendev.org/c/openstack/nova/+/908201 | |
| 22:07:42 | opendevreview | Merged openstack/nova master: run-evacuate-hook: Check cinder before creating BFV server https://review.opendev.org/c/openstack/nova/+/908182 | |
| #openstack-nova - 2024-02-07 | |||
| 01:33:28 | opendevreview | melanie witt proposed openstack/nova master: libvirt: Introduce support for raw with LUKS https://review.opendev.org/c/openstack/nova/+/884313 | |
| 01:33:28 | opendevreview | melanie witt proposed openstack/nova master: WIP Support encrypted backing files for qcow2 https://review.opendev.org/c/openstack/nova/+/907961 | |
| 01:33:29 | opendevreview | melanie witt proposed openstack/nova master: libvirt: Introduce support for rbd with LUKS https://review.opendev.org/c/openstack/nova/+/889912 | |
| 02:13:14 | opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: AMD SEV-ES support https://review.opendev.org/c/openstack/nova-specs/+/907702 | |
| 02:15:55 | opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: AMD SEV-ES support https://review.opendev.org/c/openstack/nova-specs/+/907702 | |
| 02:17:28 | opendevreview | Takashi Kajinami proposed openstack/os-resource-classes master: Update bug tracker url https://review.opendev.org/c/openstack/os-resource-classes/+/908219 | |
| 02:18:24 | opendevreview | Takashi Kajinami proposed openstack/os-traits master: Update bug tracker url https://review.opendev.org/c/openstack/os-traits/+/908220 | |
| 02:24:09 | opendevreview | Takashi Kajinami proposed openstack/osc-placement master: Update bug tracker url https://review.opendev.org/c/openstack/osc-placement/+/908221 | |
| 06:02:25 | opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: AMD SEV-ES support https://review.opendev.org/c/openstack/nova-specs/+/907702 | |
| 06:03:41 | melwitt | sean-k-mooney: I did the respin, it seems like things are working except stable rescue in some cases. I'm debugging that | |
| 06:24:27 | opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: AMD SEV-ES support https://review.opendev.org/c/openstack/nova-specs/+/907702 | |
| 10:27:39 | bauzas | okay, so I can officially say that nova-lvm has an issue :- | |
| 11:21:39 | whoami-rajat | bauzas, hey, do you mean nova-lvm or nova-cinder-lvm? not sure if nova directly uses lvm without cinder | |
| 11:27:25 | frickler | iiuc that's the nova-lvm job, which does NOVA_BACKEND="lvm", so not involving cinder | |
| 11:28:27 | whoami-rajat | okay, i remember nova team mentioning about some issues with the cinder-lvm backend so good to know it's not that | |
| 11:55:24 | opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: AMD SEV-ES support https://review.opendev.org/c/openstack/nova-specs/+/907702 | |
| 12:08:16 | opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: AMD statless firmware support https://review.opendev.org/c/openstack/nova-specs/+/908297 | |
| 12:08:38 | opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: Statless firmware support https://review.opendev.org/c/openstack/nova-specs/+/908297 | |
| 12:24:26 | greatgatsby_ | Hello. Is it possible to modify OS-EXT-SVR-ATTR:host via the nova command or OSC if it's incorrect (after compute crash)? Trying to avoid modifying via the database. | |
| 12:36:21 | bauzas | whoami-rajat: frickler: yeah was at lunch, correct this is nova-lvm jobn | |
| 13:26:39 | opendevreview | Sylvain Bauza proposed openstack/nova master: Fix verifying all the alloc requests from a multi-create https://review.opendev.org/c/openstack/nova/+/846786 | |
| 13:54:46 | opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: AMD SEV-ES support https://review.opendev.org/c/openstack/nova-specs/+/907702 | |
| 14:01:47 | opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: Statless firmware support https://review.opendev.org/c/openstack/nova-specs/+/908297 | |
| 14:02:28 | opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: AMD SEV-ES support https://review.opendev.org/c/openstack/nova-specs/+/907702 | |
| 14:03:45 | opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: Statless firmware support https://review.opendev.org/c/openstack/nova-specs/+/908297 | |
| 14:44:30 | opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: Stateless firmware support https://review.opendev.org/c/openstack/nova-specs/+/908297 | |
| 16:09:23 | bauzas | dansmith: saw your comment on https://review.opendev.org/c/openstack/nova/+/904209/14/nova/virt/libvirt/driver.py#9838 | |
| 16:09:43 | bauzas | dansmith: you're right, we could have problems, that's why I wrote this in the spec : | |
| 16:10:17 | bauzas | "will persist that list of target mediated devices in some internal dictionary field of the LibvirtDriver instance, keyed by the instance UUID. " https://specs.openstack.org/openstack/nova-specs/specs/2024.1/approved/libvirt-mdev-live-migrate.html#proposed-change | |
| 16:10:51 | bauzas | but yeah, in case the operator restart the compute service, then the dict could be wrong | |
| 16:11:03 | bauzas | or the other way if we have a rabbit issue | |
| 16:11:11 | bauzas | so I don't know what to say here | |
| 16:11:40 | bauzas | that's why we discussed this in the PTG, we said we will try to remember those by a dict | |
| 16:12:49 | bauzas | https://etherpad.opendev.org/p/nova-caracal-ptg#L585 | |
| 16:13:04 | bauzas | anyway, let's dicuss this after our meeting | |
| 16:19:26 | zigo | It felt super fast in the host, not so much in the VMs... | |
| 16:23:17 | dansmith | bauzas: yeah, and L576 says "Dan Smith not very convinced" :) | |
| 16:23:40 | dansmith | so I think this is well in bounds for being concerned now that we're looking at the implementation | |
| 16:24:06 | bauzas | dansmith: let's find an alternative after our meeting :) | |
| 16:24:14 | bauzas | let's try* to find (tbc) | |
| 16:24:15 | dansmith | I'm certainly not trying to cause you trouble, I'm just expressing concern | |
| 16:24:19 | dansmith | ack | |
| 17:09:08 | bauzas | dansmith: I'm swamped in another meeting but my point is that I wonder if we should persist the internal dict | |
| 17:09:42 | bauzas | in case of a rabbit issue, people just restart their computes | |
| 17:10:09 | bauzas | that said, if we have a current live-migration, it would be eventually stopped, right? | |
| 17:10:13 | bauzas | or not ? | |
| 17:10:13 | dansmith | right, but if they do, they leak resources in placement right? | |
| 17:10:59 | bauzas | because of the target allocations ? | |
| 17:11:28 | dansmith | yeah | |
| 17:40:40 | bauzas | dansmith: sorry was in a meeting | |
| 17:41:11 | bauzas | so, what I can do is to provide some logs, for sure | |
| 17:42:41 | bauzas | then, I'll try to test with my hardware environment what would arrive if the operator tries to restart a compute while a live-migration is currently running | |
| 17:43:37 | bauzas | for the allocations, it would be like any other feature | |
| 17:54:23 | dansmith | bauzas: what if we had the compute service look for and clean up any allocations against its RP for instances that don't exist on it? basically "things that were reserved here, but are gone now"? | |
| 17:54:27 | dansmith | maybe we already do that and I'm missing it? | |
| 17:54:48 | dansmith | but if so, then a restart would *actually* clean up everything because the in-memory bit would be dumped and we'd clean up the stale reservations | |
| 17:54:52 | bauzas | dansmith: there are two things | |
| 17:55:17 | bauzas | there are VGPU allocations and there are mdev "reservations" | |
| 17:56:02 | bauzas | the internal dict is just here for making sure we don't pass a mdev to a new instance if that mdev is currently used by a live-migrating instance | |
| 17:56:14 | bauzas | so we just 'reserve' it | |
| 17:57:00 | bauzas | that said, and that's something I can verify, if the tarrget domain is created just when we start the live-migration, then we don't need to 'reserve' the mdev | |
| 17:58:16 | bauzas | maybe the context you miss is that a compute only knows which mdevs are used by instances by looking at the guest XMLs | |
| 17:59:18 | bauzas | my concern I had is that if we don't have a domain for a target guest yet while live-migrating, then the compute would not know that the related mdevs are currently used for that one | |
| 17:59:29 | bauzas | hence the internal dict | |
| 18:09:22 | dansmith | bauzas: thanks, I understand the difference between the dict reservation and the allocations | |
| 18:09:36 | dansmith | however, the allocations are not cleaned up by a restart (right?) even though the dict reservations are | |
| 18:09:52 | dansmith | I don't really like the latter requiring a restart, but the former seems very unfortunate to me | |