| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-05-19 | |||
| 17:25:53 | melwitt | yeah from what I saw O_EXCL not exotic | |
| 17:43:46 | melwitt | I have a couple of vtpm live migration follow up items, one small patch with corrected config help text https://review.opendev.org/c/openstack/nova/+/987312 and one small patch spec amendment to clarify secret access control language https://review.opendev.org/c/openstack/nova-specs/+/987310 for those reading the delivered spec as a reference, if anyone could help review | |
| 17:53:46 | sean-k-mooney | melwitt: by the way there is a somewat related but to that swap/epmereal issues tha twe shoudl evengually fix | |
| 17:54:43 | sean-k-mooney | when we created ephmeral and i belive swap disk with the qcow backend we share a backing file based on the size | |
| 17:55:35 | sean-k-mooney | and at leasat for the ephermal disk we can format the qcow for a given vm | |
| 17:56:12 | sean-k-mooney | but if im not mistaken that can casue filesysetm currption when we cold migrate if we resue a backign file form a diffent host | |
| 17:58:44 | sean-k-mooney | basiclly we shoudl just never use backign files for swap and ephmeral | |
| 17:58:54 | sean-k-mooney | https://blueprints.launchpad.net/nova/+spec/allow-disabling-ephemeral-disk-formatting was sort of related to that | |
| 17:59:07 | sean-k-mooney | but there was bug for the actual issue somewhere | |
| 18:01:42 | sean-k-mooney | melwitt: this https://bugs.launchpad.net/nova/+bug/2095173 | |
| 18:02:08 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/940900 | |
| 18:03:08 | sean-k-mooney | melwitt: the swap thing just reminede me we hever actly fixed that | |
| 18:19:56 | melwitt | sean-k-mooney: ah yeah, I am aware of that general issue. good to have the links, I had not seen a bug report for it | |
| 19:19:30 | opendevreview | Stephen Finucane proposed openstack/os-vif master: typing: Updates for typed oslo.config https://review.opendev.org/c/openstack/os-vif/+/989206 | |
| 19:20:02 | opendevreview | Stephen Finucane proposed openstack/os-vif master: Remove linux bridge plugin https://review.opendev.org/c/openstack/os-vif/+/941586 | |
| 19:32:45 | opendevreview | Stephen Finucane proposed openstack/os-vif master: typing: Updates for typed oslo.config https://review.opendev.org/c/openstack/os-vif/+/989206 | |
| 19:32:57 | opendevreview | Stephen Finucane proposed openstack/os-vif master: Remove linux bridge plugin https://review.opendev.org/c/openstack/os-vif/+/941586 | |
| 19:36:18 | opendevreview | Stephen Finucane proposed openstack/nova master: Use static pages for eom release release notes https://review.opendev.org/c/openstack/nova/+/989211 | |
| 19:36:18 | opendevreview | Stephen Finucane proposed openstack/nova master: Use static pages for eol release release notes https://review.opendev.org/c/openstack/nova/+/989210 | |
| 19:36:19 | opendevreview | Stephen Finucane proposed openstack/nova master: docs: Speed up release notes builds https://review.opendev.org/c/openstack/nova/+/989212 | |
| 19:38:15 | stephenfin | melwitt: That should significantly speed up release notes builds. I recall you complaining about that a long time ago | |
| 19:38:18 | stephenfin | ☝️ | |
| 20:27:49 | melwitt | stephenfin: IIRC I added parallel build a long time ago for that :) thanks for the heads up, I will look at them | |
| #openstack-nova - 2026-05-20 | |||
| 01:43:32 | opendevreview | Goutham Pacha Ravi proposed openstack/nova-specs master: Add virtiofs cold migration spec https://review.opendev.org/c/openstack/nova-specs/+/985738 | |
| 10:47:25 | opendevreview | Merged openstack/os-vif master: typing: Updates for typed oslo.config https://review.opendev.org/c/openstack/os-vif/+/989206 | |
| 10:52:27 | opendevreview | Merged openstack/os-vif master: Remove linux bridge plugin https://review.opendev.org/c/openstack/os-vif/+/941586 | |
| 13:05:42 | opendevreview | Elod Illes proposed openstack/nova master: Do not log metadata proxy shared secret https://review.opendev.org/c/openstack/nova/+/988501 | |
| 13:06:10 | elodilles | sean-k-mooney: i've extended a unit test ^^^ | |
| 13:07:39 | sean-k-mooney | self.assertIn('mismatched_signature', str(warning_calls)) | |
| 13:08:04 | sean-k-mooney | shoudl that not be assertNotIn | |
| 13:08:20 | sean-k-mooney | oh | |
| 13:08:22 | sean-k-mooney | no | |
| 13:08:27 | sean-k-mooney | that the value that was passed in teh header | |
| 13:08:37 | sean-k-mooney | ok and signed is the cofnig value | |
| 13:08:55 | sean-k-mooney | elodilles: ya ok that correct +2 | |
| 13:10:46 | sean-k-mooney | you could argure you shoudl not log either value but i think at that poitn it start to be a litte hard to debug | |
| 13:11:07 | sean-k-mooney | elodilles: im ok with this as a ok compromise for operational debugablity | |
| 13:12:30 | sean-k-mooney | at some point we shoudl split that test up into multiple tests but that out of scope of your change | |
| 13:16:32 | elodilles | ACK, thanks :) | |
| 13:21:34 | opendevreview | Masanori Ueno proposed openstack/nova master: NUMA live-migration: ensure allocation_ratio is respected https://review.opendev.org/c/openstack/nova/+/989378 | |
| 13:23:32 | opendevreview | Masanori Ueno proposed openstack/nova master: NUMA live-migration: ensure allocation_ratio is respected https://review.opendev.org/c/openstack/nova/+/989378 | |
| 13:24:13 | sean-k-mooney | elodilles: by the way the os-vif changes finally merged if you want to proceed with the release and have not already done so | |
| 13:24:36 | sean-k-mooney | we might need to update teh sha. if you arelady did the m1 release there is no rush to release again | |
| 13:25:29 | elodilles | sean-k-mooney: oh, cool, i've been waiting for the patch to merge. but then it did :) let me update the hash on the rel patch | |
| 13:29:55 | elodilles | sean-k-mooney: hash and version number is updated -> https://review.opendev.org/c/openstack/releases/+/988061 | |
| 13:32:02 | sean-k-mooney | that looks correct to me, i have commented the same on the patch | |
| 14:24:47 | elodilles | Uggla: when you have some time, could you please take a quick look at the os-vif release patch? Sean and Stephen are both +1'd it already ;) https://review.opendev.org/c/openstack/releases/+/988061 | |
| 14:25:52 | Uggla | elodilles, so I guess Stephen includes what he wanted in os-vif right ? | |
| 14:27:34 | elodilles | yepp | |
| 14:27:53 | elodilles | that patch has merged | |
| 14:44:02 | auniyal | https://review.opendev.org/c/openstack/requirements/+/988412 | |
| 14:44:45 | auniyal | gmaan this might be the reason for tempest bug - attachment issue | |
| 14:58:42 | auniyal | :( seems nope | |
| 15:46:55 | opendevreview | Takashi Kajinami proposed openstack/nova master: Drop description for ancient horizon https://review.opendev.org/c/openstack/nova/+/989419 | |
| 15:48:03 | opendevreview | Takashi Kajinami proposed openstack/nova master: Drop description for ancient horizon https://review.opendev.org/c/openstack/nova/+/989419 | |
| 16:57:49 | gmaan | auniyal: thanks, let's chat on cinder channel as it is not related to nova | |
| 17:00:12 | Uggla | sean-k-mooney, does the PCI tracker only read the libvirt pci device "inventory" at startup ? | |
| 17:02:49 | sean-k-mooney | it technially doe snot read it at all | |
| 17:03:03 | sean-k-mooney | the pci tracker isslef is virt dirver indepentent | |
| 17:03:30 | sean-k-mooney | and each of the virt driver reasd the pci infromation internally and present ti to the tracker in a normalised form | |
| 17:03:52 | sean-k-mooney | but yes its effectivly cached | |
| 17:04:02 | sean-k-mooney | we dont supprot changing the device wheil nova-comptue is running | |
| 17:04:08 | Uggla | I mean update_devices_from_hypervisor_resources is only called at boot ? | |
| 17:04:28 | Uggla | s/boot/nova startup/ | |
| 17:04:29 | sean-k-mooney | we do refesh it as part of update aviabel resouces | |
| 17:04:43 | sean-k-mooney | but wiht cached data or with data that shoudl be cached | |
| 17:06:23 | Uggla | sean-k-mooney thx. | |
| 17:22:01 | melwitt | compute node shows as temporarily disabled, | |
| 17:22:01 | melwitt | gmaan: since your comment on the nova-vtpm job patch, I went looking through the NoValidHost failures again (the reason inspiring the patch) and found that it looks like what happens in that case is that a "Connection event '0' reason 'Connection to libvirt lost: 1'" happens on a compute node which causes the COMPUTE_STATUS_DISABLED trait to be added to that compute and _that's_ what ends up causing the NoValidHost ... as the other | |
| 17:22:29 | melwitt | so now I'm questioning whether setting tempest_concurrency=1 would actually help anything ... seems like maybe not | |
| 17:23:42 | melwitt | I'm not sure how or why it could be more common to cause a failure in nova-vtpm ... perhaps because there are so few tests total and two of them involve resizes which have to go to the other host | |
| 17:29:50 | sean-k-mooney | melwitt: if the libvirt connection is lost | |
| 17:30:05 | sean-k-mooney | melwitt: we mark the compute service as disabled automaticlly | |
| 17:30:13 | sean-k-mooney | and renableit when it comes back | |
| 17:30:43 | melwitt | sean-k-mooney: yes that's what I meant is I can see why it is causing the NoValidHost, if it happens at an inopportune time | |
| 17:31:10 | sean-k-mooney | so the question is why is that disconenct happening are there OOM events | |
| 17:31:24 | sean-k-mooney | that noramly only happens if libvirt restarts | |
| 17:31:51 | melwitt | hm ok, let me check that | |
| 17:32:50 | sean-k-mooney | the nova-vtpm job looks pretty stabel by the way | |
| 17:33:26 | sean-k-mooney | i assuem your debuging https://zuul.openstack.org/build/61181a17a11a41feb198f6d817337ca1 | |
| 17:33:53 | melwitt | yeah.. agreed. at the time I proposed the patch to reduce concurrency to 1, it was failing with NoValidHost more often and I had thought it was in part due to tests running in parallel. but now that I see this libvirt connection loss thing, I am questioning that thought | |
| 17:34:28 | melwitt | no I'm using https://zuul.opendev.org/t/openstack/build/1efc16477b414adbaf673b00520aa5cf bc that's where I see NoValidHost. the one you linked looked to be a different problem unless I missed something | |
| 17:34:29 | sean-k-mooney | well it could be concrance related | |
| 17:34:46 | melwitt | for the libvirt connection lost? | |
| 17:35:08 | sean-k-mooney | well if the conenction loos was because of an OOM | |
| 17:35:14 | sean-k-mooney | but i doen see that in the logs for that job | |
| 17:35:31 | melwitt | gotcha. yeah I'm not seeing something like that either so far | |
| 17:38:00 | sean-k-mooney | libvirt.libvirtError: internal error: client socket is closed | |
| 17:38:17 | sean-k-mooney | that is werid | |
| 17:39:05 | sean-k-mooney | taht is saying that nova could not create the vm in this case becaus the libvirt socket was closed form libivrts side | |
| 17:39:26 | sean-k-mooney | ot at least we recived a disconenct on the socket we had opened to talk to it | |
| 17:40:02 | sean-k-mooney | in this case it hapepns to be coming form libvirt_secret.undefine() | |
| 17:40:17 | sean-k-mooney | based on the trace back https://zuul.opendev.org/t/openstack/build/1efc16477b414adbaf673b00520aa5cf/log/compute-host/logs/screen-n-cpu.txt#5203 | |
| 17:43:53 | melwitt | this is the one I found that caused a NoValidHost https://zuul.opendev.org/t/openstack/build/1efc16477b414adbaf673b00520aa5cf/log/controller/logs/screen-n-cpu.txt#4543 | |
| 17:47:54 | sean-k-mooney | so i think the libvirt log time stampe are off by an hour form novas | |
| 17:47:58 | sean-k-mooney | 2026-05-12 22:08:17.608+0100: 29111: debug : virThreadJobClear:118 : Thread 29111 (rpc-libvirtd) finished job remoteDispatchNodeGetCPUMap with ret=0 │ | |
| 17:48:00 | sean-k-mooney | │2026-05-12 22:08:29.663+0100: 29119: debug : virThreadJobSet:93 : Thread 29119 (prio-rpc-libvirtd) is now running job remoteDispatchConnectGetLibVersion | |
| 17:48:10 | sean-k-mooney | but there appare to be a dead space in the log at that point | |
| 17:50:52 | sean-k-mooney | oh i was lookign at the compute | |
| 17:52:49 | sean-k-mooney | there is a 20 secodn gap just before then | |