| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-05-06 | |||
| 13:54:11 | sean-k-mooney | so i dont think openstack genergates thsi today | |
| 13:54:45 | artom | Yeah, there's nothing that I could find in https://docs.openstack.org/nova/latest/user/metadata.html#openstack-format-metadata | |
| 13:54:54 | sean-k-mooney | i would have expected use to keep local host name as just the host naem and pulblic_hostname as i guess the fqdu in the interface section | |
| 13:56:33 | artom | So I don't think we generate any of that today | |
| 13:56:57 | sean-k-mooney | right we dont | |
| 13:57:07 | sean-k-mooney | which is what you were ment to be implementing | |
| 13:57:30 | artom | So what happens for the openstack formatted metadata? | |
| 13:57:37 | sean-k-mooney | so we coudl do this on a per port basis sicne we decide ot not take domain as a server paramter | |
| 13:57:49 | sean-k-mooney | artom: we extend it with this same info | |
| 13:57:57 | sean-k-mooney | with a new version number | |
| 13:58:10 | sean-k-mooney | well date sicne we use date for the version | |
| 14:06:25 | sean-k-mooney | bauzas: do you want to wait for the next team meetign for people ot review https://review.opendev.org/c/openstack/project-config/+/837595 or will i ask infra to review | |
| 14:13:05 | bauzas | sean-k-mooney: not really, I accepted the names | |
| 14:13:13 | bauzas | but if you want, we can wait until the meeting | |
| 14:13:25 | sean-k-mooney | i would be happy to proceed as is | |
| 14:13:57 | sean-k-mooney | i told infra we would wait already so lets leave it | |
| 14:14:01 | sean-k-mooney | we are not in a rush | |
| 14:15:02 | bauzas | as you want | |
| 15:09:05 | mnaser | wel | |
| 15:09:12 | mnaser | i've made some progress with the whole "nothing spawns on a box" story | |
| 15:09:24 | mnaser | `WeighedHost [host: (<snip>) ram: 515266MB disk: 1662976MB io_ops: 0 instances: 0, weight: -999997.0]` | |
| 15:09:44 | sean-k-mooney | the build weigher i woudl assume | |
| 15:10:06 | sean-k-mooney | specificly the failed build weigher | |
| 15:10:22 | mnaser | oh crap | |
| 15:10:31 | mnaser | that became a weigher | |
| 15:10:37 | mnaser | instead of the host being disabled?? | |
| 15:11:14 | mnaser | ..4 years ago... | |
| 15:11:18 | mnaser | well im out of it lol | |
| 15:11:57 | sean-k-mooney | apprently we have both now | |
| 15:12:09 | sean-k-mooney | i tought we removed the old disable behivior entirly | |
| 15:12:17 | sean-k-mooney | https://docs.openstack.org/nova/latest/configuration/config.html#filter_scheduler.build_failure_weight_multiplier | |
| 15:12:45 | sean-k-mooney | you can set that to a smaller value but it was inteneded to force the host to be avoided unless all other wer efully | |
| 15:13:23 | sean-k-mooney | we still have https://docs.openstack.org/nova/latest/configuration/config.html#compute.consecutive_build_service_disable_threshold in the docs | |
| 15:14:13 | mnaser | looks like that is still in use, but wont disable sean-k-mooney | |
| 15:14:15 | mnaser | seeing https://opendev.org/openstack/nova/commit/91e29079a0eac825c5f4fe793cf607cb1771467d | |
| 15:15:17 | sean-k-mooney | yep | |
| 15:15:28 | sean-k-mooney | it is used to contol if you send info to the weigher or not | |
| 15:16:19 | mnaser | unfortuantely the by product is there is no easy way of knowing if a host is being heavily weighed down | |
| 15:16:27 | mnaser | vs seeing it as disabled is pretty obvious to an operato | |
| 15:20:06 | sean-k-mooney | operator feedback was why we change this | |
| 15:20:24 | sean-k-mooney | i tought the old behavior was the correct one | |
| 15:20:57 | sean-k-mooney | i.e. disabling the service ideally with the reason to something the operator would consume | |
| 15:23:49 | sean-k-mooney | mnaser: by the way you shoudl be abel to force that host to be used currently | |
| 15:24:02 | sean-k-mooney | it just wont get selected byt the schduler if there are other options | |
| 15:24:11 | mnaser | sean-k-mooney: well this explains the whole story why when guilhermesp_ tried to spin up a vm on it | |
| 15:24:17 | mnaser | it started accepting requests right away | |
| 15:24:21 | sean-k-mooney | one thing i had considered in the past was having this time out in some way | |
| 15:24:24 | mnaser | it probably reset the failure counter to 0 and its all good to go | |
| 15:24:34 | sean-k-mooney | yep | |
| 15:24:36 | sean-k-mooney | it would have | |
| 15:25:19 | mnaser | this doesnt explain why i was getting the port plugging timeouts that caused this but | |
| 15:25:31 | mnaser | at least this wont just bring a whole bunch of systems out of being scheduled for an extended period of time | |
| 15:25:39 | sean-k-mooney | well the neutron agent on that host might have hadd issues | |
| 15:26:09 | sean-k-mooney | and perhpase they have now been resovled? | |
| 15:26:19 | mnaser | sean-k-mooney: yeah but it's happening across different systems so its a common thing | |
| 15:26:23 | mnaser | and on a stuck box i saw this once | |
| 15:26:24 | mnaser | Agent rpc_loop - iteration:1192598 - ports processed. Elapsed:6685.721 | |
| 15:26:36 | mnaser | 1.85h for the rpc loop? :) | |
| 15:26:46 | sean-k-mooney | nice | |
| 15:26:50 | sean-k-mooney | must be a record | |
| 15:26:59 | mnaser | lol yup | |
| 15:27:17 | mnaser | so i think tehre is something in the interaction between ovs and n-ovs-agent that just started the domino effect | |
| 15:27:27 | sean-k-mooney | what os-vif driver are you usign by the way | |
| 15:27:34 | sean-k-mooney | the navie one or vsctl | |
| 15:28:02 | mnaser | i believe vsctl right now | |
| 15:28:10 | mnaser | i think we should probably flip that over to native at some point soon | |
| 15:28:11 | sean-k-mooney | ya so the reason i brought os-vif up is the ovs python bindings have a bug which i hope is now fixed | |
| 15:28:56 | sean-k-mooney | basically the ovs python bindings actuly made un monkeypatched poll socket calls | |
| 15:29:13 | sean-k-mooney | that could lock up nova when there was a disconnect until it reconnected | |
| 15:29:51 | sean-k-mooney | i think neutron have work around that but depned on the release | |
| 15:30:35 | mnaser | yeah i managed to pull a gmr when neutron was stuck for ~2h | |
| 15:30:50 | sean-k-mooney | mnaser: what release | |
| 15:31:00 | sean-k-mooney | im just checkinng for the neutron workaround | |
| 15:31:39 | mnaser | victoria, i think we're waiting for a few things before going to wallaby (but this is wallaby nova + victoria n-ovs-agent) | |
| 15:32:54 | mnaser | this is part of the culprit https://www.irccloud.com/pastebin/nxc0Uknt/ | |
| 15:33:25 | sean-k-mooney | https://review.opendev.org/c/openstack/neutron/+/794892 | |
| 15:33:53 | sean-k-mooney | i wonder if they only fixed ovn | |
| 15:34:33 | opendevreview | ribaudr proposed openstack/nova master: Allow unshelve to a specific host https://review.opendev.org/c/openstack/nova/+/831507 | |
| 15:34:49 | sean-k-mooney | i dont see it here https://github.com/openstack/neutron/blob/57f9822a8b398e92d1efebd85a1bdd4f9b27b046/neutron/agent/ovsdb/impl_idl.py | |
| 15:36:13 | sean-k-mooney | os-vif does it but only if you are using the native imple | |
| 15:36:26 | sean-k-mooney | https://github.com/openstack/os-vif/blob/master/vif_plug_ovs/ovsdb/impl_idl.py#L62-L99= | |
| 15:37:27 | mnaser | sean-k-mooney: i mean technically since neutron polls so often it might as well be a keepalived | |
| 15:38:39 | sean-k-mooney | so this is worign aroudn a bug in the ovs python bindings | |
| 15:38:57 | sean-k-mooney | we basically trun off the heartbeat | |
| 15:39:02 | sean-k-mooney | and rely on tcp keepalive | |
| 15:39:38 | sean-k-mooney | https://bugs.launchpad.net/os-vif/+bug/1929446 | |
| 15:41:01 | sean-k-mooney | the issue has been fixed in python-ovs 2.16 | |
| 15:41:08 | sean-k-mooney | based on https://review.opendev.org/c/openstack/ovsdbapp/+/818446 | |
| 15:41:24 | sean-k-mooney | the os-vif and neutron fixes were mitigations | |
| 15:41:30 | mnaser | i see | |
| 15:43:12 | opendevreview | Merged openstack/nova master: Add oslo.limit options to generated nova.conf https://review.opendev.org/c/openstack/nova/+/831101 | |
| 15:43:21 | opendevreview | Merged openstack/nova master: Log the exception returned from a cell during API.get() https://review.opendev.org/c/openstack/nova/+/840260 | |
| 15:43:31 | opendevreview | Merged openstack/nova master: Enable live_migration_events in nova-ovs-hybrid-plug https://review.opendev.org/c/openstack/nova/+/840446 | |
| 15:43:44 | opendevreview | Merged openstack/nova master: pre-commit: Sync hacking version, bump plugins https://review.opendev.org/c/openstack/nova/+/836639 | |
| 15:44:28 | sean-k-mooney | zuul just woke up i see | |
| 15:45:03 | mnaser | my guess is a slow job in a commit that slowed down the merge gate :p | |
| 15:48:36 | opendevreview | Balazs Gibizer proposed openstack/nova stable/yoga: Enable live_migration_events in nova-ovs-hybrid-plug https://review.opendev.org/c/openstack/nova/+/840931 | |
| 15:48:49 | opendevreview | ribaudr proposed openstack/nova master: Allow unshelve to a specific host https://review.opendev.org/c/openstack/nova/+/831507 | |
| 15:53:11 | opendevreview | Balazs Gibizer proposed openstack/nova master: Add Python 3.10 functional jobs https://review.opendev.org/c/openstack/nova/+/839029 | |
| 15:53:48 | opendevreview | ribaudr proposed openstack/python-novaclient master: Microversion 2.91: Support specifying destination host to unshelve https://review.opendev.org/c/openstack/python-novaclient/+/831651 | |