| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-05-06 | |||
| 15:11:18 | mnaser | well im out of it lol | |
| 15:11:57 | sean-k-mooney | apprently we have both now | |
| 15:12:09 | sean-k-mooney | i tought we removed the old disable behivior entirly | |
| 15:12:17 | sean-k-mooney | https://docs.openstack.org/nova/latest/configuration/config.html#filter_scheduler.build_failure_weight_multiplier | |
| 15:12:45 | sean-k-mooney | you can set that to a smaller value but it was inteneded to force the host to be avoided unless all other wer efully | |
| 15:13:23 | sean-k-mooney | we still have https://docs.openstack.org/nova/latest/configuration/config.html#compute.consecutive_build_service_disable_threshold in the docs | |
| 15:14:13 | mnaser | looks like that is still in use, but wont disable sean-k-mooney | |
| 15:14:15 | mnaser | seeing https://opendev.org/openstack/nova/commit/91e29079a0eac825c5f4fe793cf607cb1771467d | |
| 15:15:17 | sean-k-mooney | yep | |
| 15:15:28 | sean-k-mooney | it is used to contol if you send info to the weigher or not | |
| 15:16:19 | mnaser | unfortuantely the by product is there is no easy way of knowing if a host is being heavily weighed down | |
| 15:16:27 | mnaser | vs seeing it as disabled is pretty obvious to an operato | |
| 15:20:06 | sean-k-mooney | operator feedback was why we change this | |
| 15:20:24 | sean-k-mooney | i tought the old behavior was the correct one | |
| 15:20:57 | sean-k-mooney | i.e. disabling the service ideally with the reason to something the operator would consume | |
| 15:23:49 | sean-k-mooney | mnaser: by the way you shoudl be abel to force that host to be used currently | |
| 15:24:02 | sean-k-mooney | it just wont get selected byt the schduler if there are other options | |
| 15:24:11 | mnaser | sean-k-mooney: well this explains the whole story why when guilhermesp_ tried to spin up a vm on it | |
| 15:24:17 | mnaser | it started accepting requests right away | |
| 15:24:21 | sean-k-mooney | one thing i had considered in the past was having this time out in some way | |
| 15:24:24 | mnaser | it probably reset the failure counter to 0 and its all good to go | |
| 15:24:34 | sean-k-mooney | yep | |
| 15:24:36 | sean-k-mooney | it would have | |
| 15:25:19 | mnaser | this doesnt explain why i was getting the port plugging timeouts that caused this but | |
| 15:25:31 | mnaser | at least this wont just bring a whole bunch of systems out of being scheduled for an extended period of time | |
| 15:25:39 | sean-k-mooney | well the neutron agent on that host might have hadd issues | |
| 15:26:09 | sean-k-mooney | and perhpase they have now been resovled? | |
| 15:26:19 | mnaser | sean-k-mooney: yeah but it's happening across different systems so its a common thing | |
| 15:26:23 | mnaser | and on a stuck box i saw this once | |
| 15:26:24 | mnaser | Agent rpc_loop - iteration:1192598 - ports processed. Elapsed:6685.721 | |
| 15:26:36 | mnaser | 1.85h for the rpc loop? :) | |
| 15:26:46 | sean-k-mooney | nice | |
| 15:26:50 | sean-k-mooney | must be a record | |
| 15:26:59 | mnaser | lol yup | |
| 15:27:17 | mnaser | so i think tehre is something in the interaction between ovs and n-ovs-agent that just started the domino effect | |
| 15:27:27 | sean-k-mooney | what os-vif driver are you usign by the way | |
| 15:27:34 | sean-k-mooney | the navie one or vsctl | |
| 15:28:02 | mnaser | i believe vsctl right now | |
| 15:28:10 | mnaser | i think we should probably flip that over to native at some point soon | |
| 15:28:11 | sean-k-mooney | ya so the reason i brought os-vif up is the ovs python bindings have a bug which i hope is now fixed | |
| 15:28:56 | sean-k-mooney | basically the ovs python bindings actuly made un monkeypatched poll socket calls | |
| 15:29:13 | sean-k-mooney | that could lock up nova when there was a disconnect until it reconnected | |
| 15:29:51 | sean-k-mooney | i think neutron have work around that but depned on the release | |
| 15:30:35 | mnaser | yeah i managed to pull a gmr when neutron was stuck for ~2h | |
| 15:30:50 | sean-k-mooney | mnaser: what release | |
| 15:31:00 | sean-k-mooney | im just checkinng for the neutron workaround | |
| 15:31:39 | mnaser | victoria, i think we're waiting for a few things before going to wallaby (but this is wallaby nova + victoria n-ovs-agent) | |
| 15:32:54 | mnaser | this is part of the culprit https://www.irccloud.com/pastebin/nxc0Uknt/ | |
| 15:33:25 | sean-k-mooney | https://review.opendev.org/c/openstack/neutron/+/794892 | |
| 15:33:53 | sean-k-mooney | i wonder if they only fixed ovn | |
| 15:34:33 | opendevreview | ribaudr proposed openstack/nova master: Allow unshelve to a specific host https://review.opendev.org/c/openstack/nova/+/831507 | |
| 15:34:49 | sean-k-mooney | i dont see it here https://github.com/openstack/neutron/blob/57f9822a8b398e92d1efebd85a1bdd4f9b27b046/neutron/agent/ovsdb/impl_idl.py | |
| 15:36:13 | sean-k-mooney | os-vif does it but only if you are using the native imple | |
| 15:36:26 | sean-k-mooney | https://github.com/openstack/os-vif/blob/master/vif_plug_ovs/ovsdb/impl_idl.py#L62-L99= | |
| 15:37:27 | mnaser | sean-k-mooney: i mean technically since neutron polls so often it might as well be a keepalived | |
| 15:38:39 | sean-k-mooney | so this is worign aroudn a bug in the ovs python bindings | |
| 15:38:57 | sean-k-mooney | we basically trun off the heartbeat | |
| 15:39:02 | sean-k-mooney | and rely on tcp keepalive | |
| 15:39:38 | sean-k-mooney | https://bugs.launchpad.net/os-vif/+bug/1929446 | |
| 15:41:01 | sean-k-mooney | the issue has been fixed in python-ovs 2.16 | |
| 15:41:08 | sean-k-mooney | based on https://review.opendev.org/c/openstack/ovsdbapp/+/818446 | |
| 15:41:24 | sean-k-mooney | the os-vif and neutron fixes were mitigations | |
| 15:41:30 | mnaser | i see | |
| 15:43:12 | opendevreview | Merged openstack/nova master: Add oslo.limit options to generated nova.conf https://review.opendev.org/c/openstack/nova/+/831101 | |
| 15:43:21 | opendevreview | Merged openstack/nova master: Log the exception returned from a cell during API.get() https://review.opendev.org/c/openstack/nova/+/840260 | |
| 15:43:31 | opendevreview | Merged openstack/nova master: Enable live_migration_events in nova-ovs-hybrid-plug https://review.opendev.org/c/openstack/nova/+/840446 | |
| 15:43:44 | opendevreview | Merged openstack/nova master: pre-commit: Sync hacking version, bump plugins https://review.opendev.org/c/openstack/nova/+/836639 | |
| 15:44:28 | sean-k-mooney | zuul just woke up i see | |
| 15:45:03 | mnaser | my guess is a slow job in a commit that slowed down the merge gate :p | |
| 15:48:36 | opendevreview | Balazs Gibizer proposed openstack/nova stable/yoga: Enable live_migration_events in nova-ovs-hybrid-plug https://review.opendev.org/c/openstack/nova/+/840931 | |
| 15:48:49 | opendevreview | ribaudr proposed openstack/nova master: Allow unshelve to a specific host https://review.opendev.org/c/openstack/nova/+/831507 | |
| 15:53:11 | opendevreview | Balazs Gibizer proposed openstack/nova master: Add Python 3.10 functional jobs https://review.opendev.org/c/openstack/nova/+/839029 | |
| 15:53:48 | opendevreview | ribaudr proposed openstack/python-novaclient master: Microversion 2.91: Support specifying destination host to unshelve https://review.opendev.org/c/openstack/python-novaclient/+/831651 | |
| 16:00:07 | gmann | gibi: melwitt: releasenotes for dropping py3.6|7 https://review.opendev.org/c/openstack/nova/+/840786 | |
| 16:01:04 | gibi | gmann: thanks. +A | |
| 16:01:15 | gmann | gibi: thanks | |
| 16:42:11 | opendevreview | Merged openstack/nova master: Add releasenote about dropping pythin 3.6|7 support https://review.opendev.org/c/openstack/nova/+/840786 | |
| 18:09:38 | opendevreview | Artom Lifshitz proposed openstack/nova-specs master: Domain names in metadata https://review.opendev.org/c/openstack/nova-specs/+/840974 | |
| 19:21:37 | opendevreview | Merged openstack/nova master: Adapt bindep ubuntu-jammy https://review.opendev.org/c/openstack/nova/+/840822 | |
| 20:28:08 | opendevreview | Mohammed Naser proposed openstack/nova master: Switch libvirt event timeout message to warning https://review.opendev.org/c/openstack/nova/+/840985 | |
| 20:31:35 | opendevreview | Mohammed Naser proposed openstack/nova master: Switch libvirt event timeout message to warning https://review.opendev.org/c/openstack/nova/+/840985 | |
| 21:19:53 | opendevreview | Mohammed Naser proposed openstack/nova master: Fix race condition in _get_pci_passthrough_devices https://review.opendev.org/c/openstack/nova/+/840993 | |
| #openstack-nova - 2022-05-07 | |||
| 10:58:02 | opendevreview | sean mooney proposed openstack/nova master: [WIP] Add VDPA support for suspend and livemigrate https://review.opendev.org/c/openstack/nova/+/841012 | |
| 11:38:35 | opendevreview | sean mooney proposed openstack/nova master: enable blocked VDPA move operations https://review.opendev.org/c/openstack/nova/+/832330 | |
| 11:38:35 | opendevreview | sean mooney proposed openstack/nova master: [WIP] Add VDPA support for suspend and livemigrate https://review.opendev.org/c/openstack/nova/+/841012 | |
| 18:44:58 | opendevreview | sean mooney proposed openstack/nova master: [WIP] Add VDPA support for suspend and livemigrate https://review.opendev.org/c/openstack/nova/+/841012 | |
| 18:44:59 | opendevreview | sean mooney proposed openstack/nova master: [WIP] add sorce dev parsing for vdpa interfaces https://review.opendev.org/c/openstack/nova/+/841016 | |
| 18:44:59 | opendevreview | sean mooney proposed openstack/nova master: [WIP] fix suspend for non hostdev sriov ports https://review.opendev.org/c/openstack/nova/+/841017 | |
| #openstack-nova - 2022-05-08 | |||
| 04:59:56 | opendevreview | Merged openstack/python-novaclient master: Drop lower-constraints.txt and its testing https://review.opendev.org/c/openstack/python-novaclient/+/840019 | |
| 14:14:45 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM: log number of green(thread|let)s periodically https://review.opendev.org/c/openstack/nova/+/841040 | |
| 14:26:31 | opendevreview | Balazs Gibizer proposed openstack/nova stable/wallaby: DNM: log number of green(thread|let)s periodically https://review.opendev.org/c/openstack/nova/+/841041 | |
| 16:34:01 | opendevreview | Merged openstack/nova master: Debug Nova APIs call failures https://review.opendev.org/c/openstack/nova/+/806683 | |
| #openstack-nova - 2022-05-09 | |||
| 08:24:28 | gibi | sean-k-mooney: when you are up. We can go through the open questions in the PCI tracing spec | |
| 09:10:21 | ralonsoh | gibi, qq if you know that. For live-migration, all libvirt should have the same "virsh secret-list" ? | |
| 09:14:15 | gibi | ralonsoh: o/ good question. I guess you are looking at a situation with encrypted volumes | |
| 09:15:00 | ralonsoh | gibi, yes, I've deployed two nodes with devstack and ceph | |
| 09:18:12 | gibi | hm, I dont seem to find any code in nova that would move the secret | |
| 09:27:41 | gibi | I'm not a sure how this works but seems like barbican is involved too | |
| 09:31:52 | gibi | ralonsoh: as melwitt took over the ephemeral encryption work from lyarwood I expect that she might know more about the volume encryption case as well | |
| 09:32:20 | ralonsoh | gibi, thanks, I'm going to try to deploy this env without encription, if possible | |