| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-02-06 | |||
| 16:16:27 | bauzas | \o/ | |
| 16:17:03 | fwiesel | As you can see, if you cannot boot with a nova root disk, the tests are mostly red. I am now trying to figure out, why that happens. | |
| 16:17:22 | fwiesel | I haven't raised a bug yet, because I am not sure, that it is an issue in our setup. | |
| 16:17:56 | bauzas | ok | |
| 16:18:05 | fwiesel | That's from my side. | |
| 16:18:11 | fwiesel | Any questions? | |
| 16:18:58 | bauzas | nope from me | |
| 16:19:19 | fwiesel | Then back to you. | |
| 16:19:42 | bauzas | ok, last topic | |
| 16:19:48 | bauzas | #topic Open discussion | |
| 16:19:52 | bauzas | none in the agenda | |
| 16:19:56 | bauzas | any question ? | |
| 16:21:00 | bauzas | looks not | |
| 16:21:05 | bauzas | ok, thanks all | |
| 16:21:07 | opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2024/nova.2024-02-06-16.00.log.html | |
| 16:21:07 | opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2024/nova.2024-02-06-16.00.txt | |
| 16:21:07 | opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2024/nova.2024-02-06-16.00.html | |
| 16:21:07 | opendevmeet | Meeting ended Tue Feb 6 16:21:07 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | |
| 16:21:07 | bauzas | #endmeeting | |
| 16:21:21 | fwiesel | Thanks a lot. | |
| 16:22:05 | elodilles | thanks o/ | |
| 16:48:08 | opendevreview | Merged openstack/nova master: reno: Update master for unmaintained/yoga https://review.opendev.org/c/openstack/nova/+/908040 | |
| 16:48:11 | opendevreview | Merged openstack/placement master: reno: Update master for unmaintained/yoga https://review.opendev.org/c/openstack/placement/+/908066 | |
| 17:32:19 | melwitt | bauzas: looking at the nova-ovs-hybrid-plug job, not a fail reason I was expecting "keystoneauth1.exceptions.catalog.EndpointNotFound: publicURL endpoint for volumev3 service not found". I'll try to find what/how that's happening | |
| 17:32:42 | bauzas | cool | |
| 17:36:19 | melwitt | lol, I guess this would be why https://github.com/openstack/nova/commit/c486cc89dc11c2698cbf19fe8b2e477666331dee | |
| 17:37:47 | melwitt | cinder was removed from the job but a boot from volume test is still running | |
| 17:41:38 | melwitt | ah, it's not a test per se, but during resource setup it tries to create a bfv instance | |
| 17:49:17 | tkajinam | I wonder if anyone has ever seen this unicode error https://paste.opendev.org/show/byXf2AJyRTKB3Ck8VE9L/ | |
| 17:52:23 | tkajinam | maybe something has changed in libvirt and is causing that wired result | |
| 17:57:49 | opendevreview | melanie witt proposed openstack/nova master: run-evacuate-hook: Check cinder before creating BFV server https://review.opendev.org/c/openstack/nova/+/908182 | |
| 18:02:25 | melwitt | bauzas: hopefully ^ will work. it's weird though bc cinder was disabled in the job months ago yet it only started failing recently? | |
| 18:03:16 | sean-k-mooney | tkajinam: yep its an upstream libvirt bug | |
| 18:03:22 | sean-k-mooney | its fixed in libvirt 10 | |
| 18:03:46 | sean-k-mooney | tkajinam: we are trying to get that fixed in cetos9stream | |
| 18:04:02 | tkajinam | sean-k-mooney, ah, ok. if you are aware of it then that's nice. | |
| 18:04:07 | sean-k-mooney | tkajinam: one workaround would be to enabel the virt-preview corp repo | |
| 18:04:22 | tkajinam | I started hitting this a few hours ago, since libvirt 9.10.0 was released in c9s mirror | |
| 18:04:40 | tkajinam | in CI | |
| 18:05:15 | sean-k-mooney | we started seeing it last week in other cis that dont have a mirror | |
| 18:06:00 | melwitt | for the nova-lvm failures, so far I'm seeing a lot of glanceclient.exc.HTTPBadGateway: HTTP 502 | |
| 18:06:01 | sean-k-mooney | so i guess the other workaround is to clamp to libvirt <9.10.0 | |
| 18:06:16 | tkajinam | yeah. there may be some delay between upstream c9s mirror and opendev infra mirror | |
| 18:06:28 | opendevreview | Merged openstack/python-novaclient master: Update python classifier in setup.cfg https://review.opendev.org/c/openstack/python-novaclient/+/904617 | |
| 18:06:55 | tkajinam | I'll try that pin. afaik libvirt is not installed in the default image so pinning it may be easy | |
| 18:06:58 | tkajinam | sean-k-mooney, thanks ! | |
| 20:08:04 | melwitt | sean-k-mooney: this is a fix for the nova-ovs-hybrid-plug job, which is at a 100% fail rn afaict https://review.opendev.org/c/openstack/nova/+/908182 | |
| 20:14:09 | sean-k-mooney | melwitt: cool ill have a look quickly. iw as not really aware we did evacuate testing in that job | |
| 20:14:37 | sean-k-mooney | i tought i turned off cinder and swift in that job | |
| 20:14:42 | melwitt | yeah I didn't really know either | |
| 20:15:20 | sean-k-mooney | well it passed so ok +2w | |
| 20:15:53 | melwitt | yes cinder was disabled but the resource setup phase of the evacuate test stuff boots a bfv server, which fails bc no cinder endpoint. but how was it ever working before that? I don't see the reason | |
| 20:16:09 | sean-k-mooney | that job is specificaly for testing ml2/ovs so we tried to keep it small and not run things that were unrelated to the networking backend | |
| 20:16:16 | melwitt | I saw that the cinder disable patch was from september last year so I don't get it | |
| 20:16:45 | sean-k-mooney | did we recently add the evacuate hook or something | |
| 20:18:02 | melwitt | the evacuate hook has been around for a long time. and the patch that disabled cinder actually addressed some things related to evacuate https://review.opendev.org/c/openstack/nova/+/893540 | |
| 20:18:31 | sean-k-mooney | https://github.com/openstack/nova/commit/c486cc89dc11c2698cbf19fe8b2e477666331dee#diff-a27241a0d23e0378e836ec8459debe520a3ac0a18b8dbd7acc6e4cefc15ee75c | |
| 20:18:47 | melwitt | it's just it didn't guard around creation of a bfv server in setup_evacuate_resources.sh so I dunno why it didn't fail at that time, I would have thought it should have | |
| 20:18:53 | sean-k-mooney | so in the change we did to disabel cinder we tried to make it skip the bfv tests | |
| 20:19:04 | melwitt | right | |
| 20:19:49 | sean-k-mooney | ya so that should have alwasy just not work or passed | |
| 20:19:59 | sean-k-mooney | there must be somethign else happening but ok | |
| 20:22:33 | sean-k-mooney | i mean you directly modifed the failign code here but --poll on nova client is the same as --wait right | |
| 20:22:36 | sean-k-mooney | https://github.com/openstack/nova/commit/e96ac439d314358b52a17333fb13bd15a8a8e2c0 | |
| 20:23:23 | sean-k-mooney | actuly your change also merged before the cidner disabel one | |
| 20:23:48 | sean-k-mooney | melwitt: while you are around did you have time to respin the ephmeral encyption series | |
| 20:24:01 | sean-k-mooney | if so ill pull it down tomorrow and try and review it again | |
| 20:26:30 | melwitt | sean-k-mooney: I did. I'm going to respin once more today because there's something wonky with the encrypted backing files patch but it should be ready to pull down tomorrow | |
| 20:27:33 | melwitt | yeah --poll is the same as --wait in osc | |
| 20:28:23 | sean-k-mooney | cool | |
| 20:41:58 | melwitt | looking at a random passing nova-ovs-hybrid-plug run from a few days ago, it was failing to create the bfv server but it wasn't fatal for some reason https://zuul.opendev.org/t/openstack/build/6156ed6d9d1644d492300b2c33716ce2/log/job-output.txt#29530-29534 | |
| 20:44:18 | sean-k-mooney | odd im not sure why that woudl not have consitent behavior | |
| 20:44:59 | sean-k-mooney | maybe osc changerd its return code | |
| 20:45:14 | sean-k-mooney | i.e. in the past it might have returned 0 even if wait had an error | |
| 20:49:59 | melwitt | oh hm yeah | |
| 21:26:39 | opendevreview | keerthivasan proposed openstack/nova master: Adding sysctl changes for post-copy migration https://review.opendev.org/c/openstack/nova/+/908201 | |
| 21:50:33 | opendevreview | keerthivasan proposed openstack/nova master: Adding sysctl changes for post-copy migration https://review.opendev.org/c/openstack/nova/+/908201 | |
| 22:07:42 | opendevreview | Merged openstack/nova master: run-evacuate-hook: Check cinder before creating BFV server https://review.opendev.org/c/openstack/nova/+/908182 | |
| #openstack-nova - 2024-02-07 | |||
| 01:33:28 | opendevreview | melanie witt proposed openstack/nova master: libvirt: Introduce support for raw with LUKS https://review.opendev.org/c/openstack/nova/+/884313 | |
| 01:33:28 | opendevreview | melanie witt proposed openstack/nova master: WIP Support encrypted backing files for qcow2 https://review.opendev.org/c/openstack/nova/+/907961 | |
| 01:33:29 | opendevreview | melanie witt proposed openstack/nova master: libvirt: Introduce support for rbd with LUKS https://review.opendev.org/c/openstack/nova/+/889912 | |
| 02:13:14 | opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: AMD SEV-ES support https://review.opendev.org/c/openstack/nova-specs/+/907702 | |
| 02:15:55 | opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: AMD SEV-ES support https://review.opendev.org/c/openstack/nova-specs/+/907702 | |
| 02:17:28 | opendevreview | Takashi Kajinami proposed openstack/os-resource-classes master: Update bug tracker url https://review.opendev.org/c/openstack/os-resource-classes/+/908219 | |
| 02:18:24 | opendevreview | Takashi Kajinami proposed openstack/os-traits master: Update bug tracker url https://review.opendev.org/c/openstack/os-traits/+/908220 | |
| 02:24:09 | opendevreview | Takashi Kajinami proposed openstack/osc-placement master: Update bug tracker url https://review.opendev.org/c/openstack/osc-placement/+/908221 | |
| 06:02:25 | opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: AMD SEV-ES support https://review.opendev.org/c/openstack/nova-specs/+/907702 | |
| 06:03:41 | melwitt | sean-k-mooney: I did the respin, it seems like things are working except stable rescue in some cases. I'm debugging that | |
| 06:24:27 | opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: AMD SEV-ES support https://review.opendev.org/c/openstack/nova-specs/+/907702 | |
| 10:27:39 | bauzas | okay, so I can officially say that nova-lvm has an issue :- | |
| 11:21:39 | whoami-rajat | bauzas, hey, do you mean nova-lvm or nova-cinder-lvm? not sure if nova directly uses lvm without cinder | |
| 11:27:25 | frickler | iiuc that's the nova-lvm job, which does NOVA_BACKEND="lvm", so not involving cinder | |
| 11:28:27 | whoami-rajat | okay, i remember nova team mentioning about some issues with the cinder-lvm backend so good to know it's not that | |
| 11:55:24 | opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: AMD SEV-ES support https://review.opendev.org/c/openstack/nova-specs/+/907702 | |
| 12:08:16 | opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: AMD statless firmware support https://review.opendev.org/c/openstack/nova-specs/+/908297 | |
| 12:08:38 | opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: Statless firmware support https://review.opendev.org/c/openstack/nova-specs/+/908297 | |
| 12:24:26 | greatgatsby_ | Hello. Is it possible to modify OS-EXT-SVR-ATTR:host via the nova command or OSC if it's incorrect (after compute crash)? Trying to avoid modifying via the database. | |
| 12:36:21 | bauzas | whoami-rajat: frickler: yeah was at lunch, correct this is nova-lvm jobn | |
| 13:26:39 | opendevreview | Sylvain Bauza proposed openstack/nova master: Fix verifying all the alloc requests from a multi-create https://review.opendev.org/c/openstack/nova/+/846786 | |
| 13:54:46 | opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: AMD SEV-ES support https://review.opendev.org/c/openstack/nova-specs/+/907702 | |