| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-02-06 | |||
| 17:49:17 | tkajinam | I wonder if anyone has ever seen this unicode error https://paste.opendev.org/show/byXf2AJyRTKB3Ck8VE9L/ | |
| 17:52:23 | tkajinam | maybe something has changed in libvirt and is causing that wired result | |
| 17:57:49 | opendevreview | melanie witt proposed openstack/nova master: run-evacuate-hook: Check cinder before creating BFV server https://review.opendev.org/c/openstack/nova/+/908182 | |
| 18:02:25 | melwitt | bauzas: hopefully ^ will work. it's weird though bc cinder was disabled in the job months ago yet it only started failing recently? | |
| 18:03:16 | sean-k-mooney | tkajinam: yep its an upstream libvirt bug | |
| 18:03:22 | sean-k-mooney | its fixed in libvirt 10 | |
| 18:03:46 | sean-k-mooney | tkajinam: we are trying to get that fixed in cetos9stream | |
| 18:04:02 | tkajinam | sean-k-mooney, ah, ok. if you are aware of it then that's nice. | |
| 18:04:07 | sean-k-mooney | tkajinam: one workaround would be to enabel the virt-preview corp repo | |
| 18:04:22 | tkajinam | I started hitting this a few hours ago, since libvirt 9.10.0 was released in c9s mirror | |
| 18:04:40 | tkajinam | in CI | |
| 18:05:15 | sean-k-mooney | we started seeing it last week in other cis that dont have a mirror | |
| 18:06:00 | melwitt | for the nova-lvm failures, so far I'm seeing a lot of glanceclient.exc.HTTPBadGateway: HTTP 502 | |
| 18:06:01 | sean-k-mooney | so i guess the other workaround is to clamp to libvirt <9.10.0 | |
| 18:06:16 | tkajinam | yeah. there may be some delay between upstream c9s mirror and opendev infra mirror | |
| 18:06:28 | opendevreview | Merged openstack/python-novaclient master: Update python classifier in setup.cfg https://review.opendev.org/c/openstack/python-novaclient/+/904617 | |
| 18:06:55 | tkajinam | I'll try that pin. afaik libvirt is not installed in the default image so pinning it may be easy | |
| 18:06:58 | tkajinam | sean-k-mooney, thanks ! | |
| 20:08:04 | melwitt | sean-k-mooney: this is a fix for the nova-ovs-hybrid-plug job, which is at a 100% fail rn afaict https://review.opendev.org/c/openstack/nova/+/908182 | |
| 20:14:09 | sean-k-mooney | melwitt: cool ill have a look quickly. iw as not really aware we did evacuate testing in that job | |
| 20:14:37 | sean-k-mooney | i tought i turned off cinder and swift in that job | |
| 20:14:42 | melwitt | yeah I didn't really know either | |
| 20:15:20 | sean-k-mooney | well it passed so ok +2w | |
| 20:15:53 | melwitt | yes cinder was disabled but the resource setup phase of the evacuate test stuff boots a bfv server, which fails bc no cinder endpoint. but how was it ever working before that? I don't see the reason | |
| 20:16:09 | sean-k-mooney | that job is specificaly for testing ml2/ovs so we tried to keep it small and not run things that were unrelated to the networking backend | |
| 20:16:16 | melwitt | I saw that the cinder disable patch was from september last year so I don't get it | |
| 20:16:45 | sean-k-mooney | did we recently add the evacuate hook or something | |
| 20:18:02 | melwitt | the evacuate hook has been around for a long time. and the patch that disabled cinder actually addressed some things related to evacuate https://review.opendev.org/c/openstack/nova/+/893540 | |
| 20:18:31 | sean-k-mooney | https://github.com/openstack/nova/commit/c486cc89dc11c2698cbf19fe8b2e477666331dee#diff-a27241a0d23e0378e836ec8459debe520a3ac0a18b8dbd7acc6e4cefc15ee75c | |
| 20:18:47 | melwitt | it's just it didn't guard around creation of a bfv server in setup_evacuate_resources.sh so I dunno why it didn't fail at that time, I would have thought it should have | |
| 20:18:53 | sean-k-mooney | so in the change we did to disabel cinder we tried to make it skip the bfv tests | |
| 20:19:04 | melwitt | right | |
| 20:19:49 | sean-k-mooney | ya so that should have alwasy just not work or passed | |
| 20:19:59 | sean-k-mooney | there must be somethign else happening but ok | |
| 20:22:33 | sean-k-mooney | i mean you directly modifed the failign code here but --poll on nova client is the same as --wait right | |
| 20:22:36 | sean-k-mooney | https://github.com/openstack/nova/commit/e96ac439d314358b52a17333fb13bd15a8a8e2c0 | |
| 20:23:23 | sean-k-mooney | actuly your change also merged before the cidner disabel one | |
| 20:23:48 | sean-k-mooney | melwitt: while you are around did you have time to respin the ephmeral encyption series | |
| 20:24:01 | sean-k-mooney | if so ill pull it down tomorrow and try and review it again | |
| 20:26:30 | melwitt | sean-k-mooney: I did. I'm going to respin once more today because there's something wonky with the encrypted backing files patch but it should be ready to pull down tomorrow | |
| 20:27:33 | melwitt | yeah --poll is the same as --wait in osc | |
| 20:28:23 | sean-k-mooney | cool | |
| 20:41:58 | melwitt | looking at a random passing nova-ovs-hybrid-plug run from a few days ago, it was failing to create the bfv server but it wasn't fatal for some reason https://zuul.opendev.org/t/openstack/build/6156ed6d9d1644d492300b2c33716ce2/log/job-output.txt#29530-29534 | |
| 20:44:18 | sean-k-mooney | odd im not sure why that woudl not have consitent behavior | |
| 20:44:59 | sean-k-mooney | maybe osc changerd its return code | |
| 20:45:14 | sean-k-mooney | i.e. in the past it might have returned 0 even if wait had an error | |
| 20:49:59 | melwitt | oh hm yeah | |
| 21:26:39 | opendevreview | keerthivasan proposed openstack/nova master: Adding sysctl changes for post-copy migration https://review.opendev.org/c/openstack/nova/+/908201 | |
| 21:50:33 | opendevreview | keerthivasan proposed openstack/nova master: Adding sysctl changes for post-copy migration https://review.opendev.org/c/openstack/nova/+/908201 | |
| 22:07:42 | opendevreview | Merged openstack/nova master: run-evacuate-hook: Check cinder before creating BFV server https://review.opendev.org/c/openstack/nova/+/908182 | |
| #openstack-nova - 2024-02-07 | |||
| 01:33:28 | opendevreview | melanie witt proposed openstack/nova master: libvirt: Introduce support for raw with LUKS https://review.opendev.org/c/openstack/nova/+/884313 | |
| 01:33:28 | opendevreview | melanie witt proposed openstack/nova master: WIP Support encrypted backing files for qcow2 https://review.opendev.org/c/openstack/nova/+/907961 | |
| 01:33:29 | opendevreview | melanie witt proposed openstack/nova master: libvirt: Introduce support for rbd with LUKS https://review.opendev.org/c/openstack/nova/+/889912 | |
| 02:13:14 | opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: AMD SEV-ES support https://review.opendev.org/c/openstack/nova-specs/+/907702 | |
| 02:15:55 | opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: AMD SEV-ES support https://review.opendev.org/c/openstack/nova-specs/+/907702 | |
| 02:17:28 | opendevreview | Takashi Kajinami proposed openstack/os-resource-classes master: Update bug tracker url https://review.opendev.org/c/openstack/os-resource-classes/+/908219 | |
| 02:18:24 | opendevreview | Takashi Kajinami proposed openstack/os-traits master: Update bug tracker url https://review.opendev.org/c/openstack/os-traits/+/908220 | |
| 02:24:09 | opendevreview | Takashi Kajinami proposed openstack/osc-placement master: Update bug tracker url https://review.opendev.org/c/openstack/osc-placement/+/908221 | |
| 06:02:25 | opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: AMD SEV-ES support https://review.opendev.org/c/openstack/nova-specs/+/907702 | |
| 06:03:41 | melwitt | sean-k-mooney: I did the respin, it seems like things are working except stable rescue in some cases. I'm debugging that | |
| 06:24:27 | opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: AMD SEV-ES support https://review.opendev.org/c/openstack/nova-specs/+/907702 | |
| 10:27:39 | bauzas | okay, so I can officially say that nova-lvm has an issue :- | |
| 11:21:39 | whoami-rajat | bauzas, hey, do you mean nova-lvm or nova-cinder-lvm? not sure if nova directly uses lvm without cinder | |
| 11:27:25 | frickler | iiuc that's the nova-lvm job, which does NOVA_BACKEND="lvm", so not involving cinder | |
| 11:28:27 | whoami-rajat | okay, i remember nova team mentioning about some issues with the cinder-lvm backend so good to know it's not that | |
| 11:55:24 | opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: AMD SEV-ES support https://review.opendev.org/c/openstack/nova-specs/+/907702 | |
| 12:08:16 | opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: AMD statless firmware support https://review.opendev.org/c/openstack/nova-specs/+/908297 | |
| 12:08:38 | opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: Statless firmware support https://review.opendev.org/c/openstack/nova-specs/+/908297 | |
| 12:24:26 | greatgatsby_ | Hello. Is it possible to modify OS-EXT-SVR-ATTR:host via the nova command or OSC if it's incorrect (after compute crash)? Trying to avoid modifying via the database. | |
| 12:36:21 | bauzas | whoami-rajat: frickler: yeah was at lunch, correct this is nova-lvm jobn | |
| 13:26:39 | opendevreview | Sylvain Bauza proposed openstack/nova master: Fix verifying all the alloc requests from a multi-create https://review.opendev.org/c/openstack/nova/+/846786 | |
| 13:54:46 | opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: AMD SEV-ES support https://review.opendev.org/c/openstack/nova-specs/+/907702 | |
| 14:01:47 | opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: Statless firmware support https://review.opendev.org/c/openstack/nova-specs/+/908297 | |
| 14:02:28 | opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: AMD SEV-ES support https://review.opendev.org/c/openstack/nova-specs/+/907702 | |
| 14:03:45 | opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: Statless firmware support https://review.opendev.org/c/openstack/nova-specs/+/908297 | |
| 14:44:30 | opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: Stateless firmware support https://review.opendev.org/c/openstack/nova-specs/+/908297 | |
| 16:09:23 | bauzas | dansmith: saw your comment on https://review.opendev.org/c/openstack/nova/+/904209/14/nova/virt/libvirt/driver.py#9838 | |
| 16:09:43 | bauzas | dansmith: you're right, we could have problems, that's why I wrote this in the spec : | |
| 16:10:17 | bauzas | "will persist that list of target mediated devices in some internal dictionary field of the LibvirtDriver instance, keyed by the instance UUID. " https://specs.openstack.org/openstack/nova-specs/specs/2024.1/approved/libvirt-mdev-live-migrate.html#proposed-change | |
| 16:10:51 | bauzas | but yeah, in case the operator restart the compute service, then the dict could be wrong | |
| 16:11:03 | bauzas | or the other way if we have a rabbit issue | |
| 16:11:11 | bauzas | so I don't know what to say here | |
| 16:11:40 | bauzas | that's why we discussed this in the PTG, we said we will try to remember those by a dict | |
| 16:12:49 | bauzas | https://etherpad.opendev.org/p/nova-caracal-ptg#L585 | |
| 16:13:04 | bauzas | anyway, let's dicuss this after our meeting | |
| 16:19:26 | zigo | It felt super fast in the host, not so much in the VMs... | |
| 16:23:17 | dansmith | bauzas: yeah, and L576 says "Dan Smith not very convinced" :) | |
| 16:23:40 | dansmith | so I think this is well in bounds for being concerned now that we're looking at the implementation | |
| 16:24:06 | bauzas | dansmith: let's find an alternative after our meeting :) | |
| 16:24:14 | bauzas | let's try* to find (tbc) | |
| 16:24:15 | dansmith | I'm certainly not trying to cause you trouble, I'm just expressing concern | |
| 16:24:19 | dansmith | ack | |
| 17:09:08 | bauzas | dansmith: I'm swamped in another meeting but my point is that I wonder if we should persist the internal dict | |
| 17:09:42 | bauzas | in case of a rabbit issue, people just restart their computes | |
| 17:10:09 | bauzas | that said, if we have a current live-migration, it would be eventually stopped, right? | |
| 17:10:13 | bauzas | or not ? | |
| 17:10:13 | dansmith | right, but if they do, they leak resources in placement right? | |
| 17:10:59 | bauzas | because of the target allocations ? | |
| 17:11:28 | dansmith | yeah | |
| 17:40:40 | bauzas | dansmith: sorry was in a meeting | |