| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-05-18 | |||
| 17:35:33 | sean-k-mooney | 2026-05-18 13:15:38.441225 | compute1 | { | |
| 17:35:35 | sean-k-mooney | 2026-05-18 13:15:38.441296 | compute1 | "msg": "ssh: connect to host 23.253.56.19 port 22: Connection timed out\r\nrsync: connection unexpectedly closed (0 bytes received so far) [Receiver]\nrsync error: unexplained error (code 255) at io.c(232) [Receiver=3.2.7]\n", | |
| 17:35:37 | sean-k-mooney | 2026-05-18 13:15:38.441357 | compute1 | "rc": 255 | |
| 17:35:39 | sean-k-mooney | 2026-05-18 13:15:38.441417 | compute1 | } | |
| 17:35:48 | sean-k-mooney | that i think was the actual reason it fiald but ill check again when it reports back | |
| 17:36:32 | sean-k-mooney | ah the live migration job in the recheck has already passed https://zuul.opendev.org/t/openstack/build/ea322beec7b145c49f2316d6b1b5ec7e | |
| 17:37:07 | sean-k-mooney | so ya it was just a network blip on teh ci vms | |
| 17:41:03 | sean-k-mooney | lajoskatona: if your intreested in whta th job was doing we run this as a post playbook in the job https://github.com/openstack/nova/blob/master/playbooks/nova-live-migration/post-run.yaml which invokes this role to test evacuate https://github.com/openstack/nova/blob/master/roles/run-evacuate-hook/tasks/main.yaml | |
| 17:52:07 | lajoskatona | sean-k-mooney: thanks, I check these good to look exotic non-neutron jobs sometimes :-) | |
| 17:52:40 | sean-k-mooney | so form a neutron perspective | |
| 17:52:50 | sean-k-mooney | nova-live-migration tests live migration with ovn | |
| 17:53:04 | sean-k-mooney | nova-next tests live migraiton with ml2/ovs and the ovs firewall driver | |
| 17:53:18 | sean-k-mooney | and nova-alt-config tests ml2/ovs with the iptabels firewall driver | |
| 17:53:25 | sean-k-mooney | all with local nova storage | |
| 17:53:40 | sean-k-mooney | and nova-live-migration-ceph is ovn + cpeh for the vm storage | |
| 17:54:31 | sean-k-mooney | so we try to cover the most common cofnigurations | |
| 18:20:35 | blanson[m] | Hello guys, quick question. I'm seeing a seemingly "normal" alert on my rabbitmq cluster for nova because scheduler_fanout stream queue il pilling up with nova_instance_info messages ? from what I could read in the code this is normal, expected behavior ? Is it up to the operator to tweaks retention on this queue to not have it pile up to infinity ? | |
| 18:21:34 | blanson[m] | https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2428 <- this specifically from my understanding produces those messages which are never consumed ? or is something broken on my end ? | |
| 19:03:16 | sean-k-mooney | soh those are consumed by the schduelr if you have that enabled | |
| 19:08:52 | blanson[m] | I see, but it's not consumed by default ? in which case retention policy should be fine ? | |
| 19:35:40 | sean-k-mooney | blanson[m]: sorry was disctrcted | |
| 19:36:30 | sean-k-mooney | blanson[m]: i woudl have to confirm but i belive this is contole by https://docs.openstack.org/nova/latest/configuration/config.html#filter_scheduler.track_instance_changes | |
| 19:36:52 | sean-k-mooney | blanson[m]: and by defautl these are both sent by the compute nodes and consubed by the schduler | |
| 19:37:09 | blanson[m] | I arrived to the same bit of code by digging, I will give that a try to see if that indeed starts consumming messages, thank you ! | |
| 19:37:18 | sean-k-mooney | however if you have multiple cells then it may not work | |
| 19:40:21 | sean-k-mooney | blanson[m]: the other thing that is related to this si you shoudl only enable notificaion if there is a consumer in general | |
| 19:41:07 | sean-k-mooney | if you have notificaion enabel dits ecpected tha tyou either have a consumer for those or you have implemetin a retaion policy on rabbit ectra to discard them after a given treashold | |
| 19:42:11 | sean-k-mooney | blanson[m]: actully looking again https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.scheduler_instance_sync_interval i sthe confgi to contol this and you can disble it by seting that to -1 | |
| 19:48:14 | blanson[m] | I will have a look into the track_instance_change, but my next plan was to indeed disable the whole thing otherwise ! thank you very much ! | |
| 20:28:48 | blanson[m] | well, apparently that's an oslo_messaging/rabbitmq issue. I had the correct consumer configuration on the scheduler side, but messages in stream fanout queues are never ack'd. as per https://docs.openstack.org/openstack-ansible/latest/user/messaging/messaging.html#managing-rabbitmq-stream-policy tthose keep piling up until 500MB unless there is a retention policy on the rabbitmq side. TIL ! | |
| 22:30:49 | opendevreview | sean mooney proposed openstack/nova master: weights: fix TypeError in weight debug log serialization https://review.opendev.org/c/openstack/nova/+/904060 | |
| 22:30:50 | opendevreview | sean mooney proposed openstack/nova master: weights: add regression test for JSON log TypeError (bug 2028518) https://review.opendev.org/c/openstack/nova/+/989029 | |
| 23:37:37 | opendevreview | Merged openstack/nova master: Add regression test for bug #2134375 https://review.opendev.org/c/openstack/nova/+/984900 | |
| #openstack-nova - 2026-05-19 | |||
| 00:09:06 | opendevreview | Sam Morrison proposed openstack/nova master: Filter out deleted instances when looking for build timouts https://review.opendev.org/c/openstack/nova/+/880125 | |
| 01:14:55 | opendevreview | Merged openstack/nova master: Fix port remain while error on update port with multi network https://review.opendev.org/c/openstack/nova/+/983672 | |
| 07:26:24 | opendevreview | Artem Vasilyev proposed openstack/nova master: Fix race between resize and update_available_resource for dedicated CPUs https://review.opendev.org/c/openstack/nova/+/984048 | |
| 07:26:33 | opendevreview | Artem Vasilyev proposed openstack/nova master: Add reproducer for bug #2147776 https://review.opendev.org/c/openstack/nova/+/984007 | |
| 10:41:32 | opendevreview | Joan Gilabert proposed openstack/nova-specs master: Repropose and update cyborg vGPU (mdev) support https://review.opendev.org/c/openstack/nova-specs/+/967515 | |
| 11:39:08 | opendevreview | Maximilian Stinsky proposed openstack/nova master: Remove experimental warning from UnifiedLimitsDriver https://review.opendev.org/c/openstack/nova/+/989102 | |
| 12:07:34 | opendevreview | Stephen Finucane proposed openstack/os-vif master: zuul: Disable cinder jobs https://review.opendev.org/c/openstack/os-vif/+/989113 | |
| 12:13:12 | stephenfin | sean-k-mooney: Is that what you were suggesting? ^ | |
| 12:13:25 | sean-k-mooney | more or less yes | |
| 12:13:56 | sean-k-mooney | i have 3 or 4 other memory optimisation patches in flight to neturon and devstack too | |
| 12:14:05 | sean-k-mooney | to help out but that is a quick and easy win | |
| 12:15:43 | sean-k-mooney | if we disable it in the vars section i dont think we need to replciate that ot subnodes but it will fail if we do and we can add it | |
| 12:16:15 | sean-k-mooney | i think all the os-vif jobs are single node anyway | |
| 12:42:17 | opendevreview | Matt Crees proposed openstack/nova master: Don't drop pci_dev network caps when VF is in use https://review.opendev.org/c/openstack/nova/+/982660 | |
| 12:42:18 | opendevreview | Matt Crees proposed openstack/nova master: Add test_update_devices_preserves_vf_network_caps https://review.opendev.org/c/openstack/nova/+/989121 | |
| 13:07:18 | opendevreview | Matt Crees proposed openstack/nova master: Add test_update_devices_preserves_vf_network_caps https://review.opendev.org/c/openstack/nova/+/989121 | |
| 13:07:18 | opendevreview | Matt Crees proposed openstack/nova master: Don't drop pci_dev network caps when VF is in use https://review.opendev.org/c/openstack/nova/+/982660 | |
| 15:11:33 | opendevreview | Merged openstack/os-vif master: zuul: Disable cinder jobs https://review.opendev.org/c/openstack/os-vif/+/989113 | |
| 15:11:55 | opendevreview | sean mooney proposed openstack/nova master: Add regression test for unified limits resource bug https://review.opendev.org/c/openstack/nova/+/975859 | |
| 15:11:55 | opendevreview | sean mooney proposed openstack/nova master: Fix unified limits to include all resource types https://review.opendev.org/c/openstack/nova/+/975872 | |
| 15:35:44 | sean-k-mooney | melwitt: i think i adressed all your comment on https://review.opendev.org/q/topic:%22bug/2140631%22 let me know if i missed anything | |
| 15:36:11 | melwitt | sean-k-mooney: ok nice, I will take a look | |
| 15:40:01 | opendevreview | Matt Crees proposed openstack/nova master: Add test_update_devices_preserves_vf_network_caps https://review.opendev.org/c/openstack/nova/+/989121 | |
| 15:40:01 | opendevreview | Matt Crees proposed openstack/nova master: Don't drop pci_dev network caps when VF is in use https://review.opendev.org/c/openstack/nova/+/982660 | |
| 16:01:55 | Uggla | melwitt, dansmith if you can have a look at 987704: Fix resize to unpermitted flavors | https://review.opendev.org/c/openstack/nova/+/987704, that should not take long | |
| 16:02:46 | dansmith | ? | |
| 16:08:20 | melwitt | Uggla: it looks like that is merged already? | |
| 16:09:40 | Uggla | melwitt, dansmith oops sory wrong CC here: I mean 988501: Do not log metadata proxy shared secret | https://review.opendev.org/c/openstack/nova/+/988501 | |
| 16:17:05 | melwitt | Uggla: ah ok, sure I will look | |
| 16:24:45 | sean-k-mooney | https://bugs.launchpad.net/nova/+bug/2150094 is a secuirty weakness more then a security vulnerablity | |
| 16:25:07 | sean-k-mooney | but we shoudl avoid logging it in general | |
| 16:26:30 | sean-k-mooney | im not sure that rotaion of the metadata_proxy_shared_secret is strictly require but it does not hurt | |
| 16:52:09 | sean-k-mooney | elodilles: mind addign a unit test for the log message just to assert the secret is not in it. https://review.opendev.org/c/openstack/nova/+/988501 | |
| 17:07:02 | melwitt | sean-k-mooney: thought you might find this interesting -- this is a bug that first appeared in downstream CI intermittently where I found the root cause ... but I figured it was probably too rare to prioritize. then it showed up again recently in a customer case so I proposed a fix for it https://review.opendev.org/c/openstack/nova/+/988547 you might be good to comment on it bc you know a lot about NFS | |
| 17:07:48 | sean-k-mooney | less then you might think and more then i ever wanted | |
| 17:07:59 | sean-k-mooney | but sure i can take a look | |
| 17:08:06 | melwitt | heheh | |
| 17:08:24 | sean-k-mooney | that sound liek a nfs lockign issue | |
| 17:09:22 | sean-k-mooney | melwitt: on enit. @mock.patch('os.open', | |
| 17:09:32 | sean-k-mooney | i belive we have mock open helper | |
| 17:09:42 | sean-k-mooney | because mocking builting like open is a littel od | |
| 17:09:47 | sean-k-mooney | it might be better to use that | |
| 17:10:05 | sean-k-mooney | ill read the rest but just somthign i notced initally | |
| 17:10:11 | melwitt | yeah in debugging the CI issue we put debug printing _everywhere_ in it and were able to confirm a false positive of the exists() call -- returned true when it was not there | |
| 17:10:22 | melwitt | oh gotcha | |
| 17:10:37 | melwitt | thanks, that sounds familiar. I'll go look for it | |
| 17:12:17 | sean-k-mooney | we might have got rid of our helper since mock not does if for you | |
| 17:12:26 | sean-k-mooney | we often do @mock.patch('builtins.open', new_callable=mock.mock_open()) | |
| 17:12:36 | sean-k-mooney | but mocing os.open will probaly work | |
| 17:13:46 | sean-k-mooney | melwitt: i just know that used to be a bit flaky in older version fo python so im alwasy a litte carful of it failing in the gate if we dotn do that correctly | |
| 17:14:16 | melwitt | yeah, it's a good point | |
| 17:17:11 | opendevreview | Maximilian Stinsky proposed openstack/nova master: Remove experimental warning from UnifiedLimitsDriver https://review.opendev.org/c/openstack/nova/+/989102 | |
| 17:17:34 | sean-k-mooney | melwitt: so the fix si to add O_CREAT|O_EXCL | |
| 17:17:55 | sean-k-mooney | to bypass the cache | |
| 17:19:49 | melwitt | that is the idea. I'm not aware if there could be any possible pitfall with that or rather if there is some way that it would not work | |
| 17:22:17 | sean-k-mooney | well the constrati woudl be if one of the open modes was not supproted by a given version of nfs or the kernel | |
| 17:22:29 | sean-k-mooney | i dont think you are usign anyting exotic in this patch | |
| 17:23:23 | opendevreview | Merged openstack/nova master: Reproduce bug/2152697 https://review.opendev.org/c/openstack/nova/+/988769 | |
| 17:23:34 | sean-k-mooney | we did have to do soem O_DIRECT check for a time but apprently that was added in kernel 2.4.10 so we may not need that anymore | |
| 17:24:08 | sean-k-mooney | O_EXCL Ensure that this call creates the file: if this flag is | |
| 17:24:10 | sean-k-mooney | specified in conjunction with O_CREAT, and path already | |
| 17:24:12 | sean-k-mooney | exists, then open() fails with the error EEXIST. | |
| 17:24:23 | sean-k-mooney | ok so that is the behvior we want for the probe | |
| 17:25:04 | sean-k-mooney | On NFS, O_EXCL is supported only when using NFSv3 or later | |
| 17:25:06 | sean-k-mooney | on kernel 2.6 or later. | |
| 17:25:17 | sean-k-mooney | ya i think we are good with those min versions in 2026 | |
| 17:25:53 | melwitt | yeah from what I saw O_EXCL not exotic | |