| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-09-12 | |||
| 11:58:23 | sean-k-mooney | ack | |
| 11:58:59 | sean-k-mooney | [os_vif_ovs]/ovsdb_interface=vsctl | |
| 11:59:24 | sean-k-mooney | setting that in your nova.conf will also workaround it | |
| 12:00:58 | noonedeadpunk | ah, undocumented option?:) | |
| 12:01:12 | noonedeadpunk | let me try it out then | |
| 12:01:22 | sean-k-mooney | well its not undcoumented | |
| 12:01:32 | sean-k-mooney | its an os_vif option | |
| 12:01:33 | sean-k-mooney | not nova | |
| 12:02:23 | noonedeadpunk | well, yeah, but keystone_authtoken and oslo are included. And with that as operator I would expect see rest as well... | |
| 12:02:25 | noonedeadpunk | Anyway) | |
| 12:02:53 | zigo | Looks like my issue is related to https://github.com/eventlet/eventlet/issues/730 | |
| 12:03:13 | zigo | I need to fix eventlet in Debian with py 3.10 first, and then see what's going on with oslo.concurrency ... | |
| 12:03:57 | sean-k-mooney | noonedeadpunk: we could render them i guess i was expecting them to be in https://docs.openstack.org/os-vif/latest/index.html but im not seeing them | |
| 12:04:12 | sean-k-mooney | noonedeadpunk: we have explictly list namespaces if we want them to show up | |
| 12:05:57 | sean-k-mooney | noonedeadpunk: this is where its defiend by the way https://github.com/openstack/os-vif/blob/master/vif_plug_ovs/ovs.py#L71-L82 | |
| 12:06:15 | sean-k-mooney | these are the valid options https://github.com/openstack/os-vif/blob/771dfffcd90dcd7c8c95c41744092f5ad4917be3/vif_plug_ovs/ovsdb/api.py#L18-L21 | |
| 12:07:38 | noonedeadpunk | nice, thanks! | |
| 12:08:31 | sean-k-mooney | noonedeadpunk: if your using ml2/ovs you should also be enabling https://github.com/openstack/os-vif/blob/master/vif_plug_ovs/ovs.py#L96-L99 | |
| 12:08:57 | sean-k-mooney | ignore per_port_bridge | |
| 12:08:59 | sean-k-mooney | i should remove that | |
| 12:09:56 | noonedeadpunk | til about `isolate_vif` | |
| 12:10:43 | noonedeadpunk | and yes, it your note does make sense | |
| 12:30:43 | noonedeadpunk | sean-k-mooney: `[os_vif_ovs]/ovsdb_interface = vsctl` seems not to solve the issue. | |
| 12:32:32 | noonedeadpunk | no ovsdbapp logs though | |
| 12:33:08 | sean-k-mooney | ok then its likely not beauce of the agent looking up | |
| 12:33:33 | sean-k-mooney | you mentioned you have disabled the pthread for heatbeat yes | |
| 12:34:06 | sean-k-mooney | do you you iptbale firewall or openvswtich | |
| 12:34:10 | noonedeadpunk | yeah, I did, for all neutron-ovs-agents, nova-compute/scheduler/conductor | |
| 12:34:40 | noonedeadpunk | huh. | |
| 12:34:53 | sean-k-mooney | iptables we add the linux bridge and veth pair ovs firewall we add the tap directly to ovs | |
| 12:35:06 | noonedeadpunk | I just realized I likely missed disabling for neutron-server | |
| 12:35:20 | noonedeadpunk | as I assumed it's running in uwsgi, but likely it's not in this deployment | |
| 12:35:28 | sean-k-mooney | well neutron-server | |
| 12:35:36 | sean-k-mooney | ah i was going to sayy its proably wsgi | |
| 12:36:03 | sean-k-mooney | ya worth checking | |
| 12:36:13 | sean-k-mooney | that has been reverted to off by default | |
| 12:36:22 | sean-k-mooney | not sure if its been released yet on stable branches | |
| 12:36:23 | noonedeadpunk | nah, I disabled heartbeat_in_pthread for neutron-server as well | |
| 12:37:21 | noonedeadpunk | firewall_driver = iptables_hybrid | |
| 12:37:46 | noonedeadpunk | yeah, I guess there's an lxb in place | |
| 12:49:30 | sean-k-mooney | yep | |
| 12:49:59 | sean-k-mooney | ok on sec | |
| 12:51:25 | sean-k-mooney | https://github.com/openstack/nova/commit/0b0f40d1b308b29da537859b72080488560c23d4 | |
| 12:51:33 | noonedeadpunk | Tbh I still kind of blame pthread, as actual error is `oslo_messaging.rpc.server eventlet.timeout.Timeout: 300 seconds` which is exactly what would happen because of pthreads iirc | |
| 12:51:43 | noonedeadpunk | and it's intermittent as well | |
| 12:52:07 | noonedeadpunk | but only revert is affected which is weird given it's pthread | |
| 12:52:26 | sean-k-mooney | https://bugs.launchpad.net/nova/+bug/1952003 | |
| 12:52:50 | sean-k-mooney | so we when back and foth with this due to a few inflight change at once | |
| 12:52:58 | sean-k-mooney | but you are probaly hitting that | |
| 12:53:32 | sean-k-mooney | we shoudl not be waiting for the network vif plugged event if you have a specic set of patches. | |
| 12:54:37 | sean-k-mooney | noonedeadpunk: do you have https://github.com/openstack/nova/commit/66c7f00e1d9d7c0eebe46eb4b24b2b21f7413789 | |
| 12:56:01 | gibi | zigo: ack, let me know if I can help somehow | |
| 12:56:23 | sean-k-mooney | when we adressed https://bugs.launchpad.net/nova/+bug/1895220 it intoduced https://bugs.launchpad.net/nova/+bug/1952003 which orginially fixed https://bugs.launchpad.net/nova/+bug/1832028 and https://bugs.launchpad.net/nova/+bug/1833902 | |
| 12:57:49 | sean-k-mooney | noonedeadpunk: https://github.com/openstack/nova/commit/0b0f40d1b308b29da537859b72080488560c23d4 is in yoga | |
| 12:58:07 | noonedeadpunk | sean-k-mooney: I do have https://github.com/openstack/nova/commit/66c7f00e1d9d7c0eebe46eb4b24b2b21f7413789 | |
| 12:58:41 | noonedeadpunk | I guess I don't have I3cb39a9ec2c260f422b3c48122b9db512cdd799b though, as it's Xena | |
| 12:58:42 | sean-k-mooney | noonedeadpunk: what about https://review.opendev.org/c/openstack/nova/+/828414 | |
| 12:58:53 | sean-k-mooney | we backproted it | |
| 12:59:03 | sean-k-mooney | but only 5 months ago | |
| 12:59:16 | noonedeadpunk | Nah, we did not do this minor upgrade | |
| 12:59:21 | noonedeadpunk | Let me check it out | |
| 12:59:55 | sean-k-mooney | im not sure if we have done a release since then | |
| 13:00:24 | noonedeadpunk | Gerrit says you did :p | |
| 13:00:41 | noonedeadpunk | But we run 24.0.1.dev10, and it's included in 24.1.1 | |
| 13:00:43 | sean-k-mooney | no it say we merge it | |
| 13:00:54 | sean-k-mooney | where did you see that in gerrit | |
| 13:01:14 | noonedeadpunk | three dots in upper right corner -> included in | |
| 13:01:44 | sean-k-mooney | oh wow didnt know that was a thing | |
| 13:02:26 | sean-k-mooney | https://github.com/openstack/releases/commit/ac4be06827ec7a450233244d8c5cae8834b95ffc | |
| 13:02:29 | noonedeadpunk | it was there even in gerrit 2 | |
| 13:02:37 | sean-k-mooney | but yes we did that on 21st jun | |
| 13:03:01 | sean-k-mooney | never used it i normally just check the release in github | |
| 13:04:47 | sean-k-mooney | noonedeadpunk: so ya soory i think its https://bugs.launchpad.net/nova/+bug/1952003 | |
| 13:05:17 | sean-k-mooney | on the pluse side if it is then you just need to do the minor update when you have time too | |
| 13:18:28 | noonedeadpunk | ah, bug report is indeed super familiar | |
| 13:29:17 | zigo | gibi: Building Eventlet, I still get this: | |
| 13:29:17 | zigo | https://paste.opendev.org/show/bIinQaPTAy81Uac3ZPHS/ | |
| 13:29:17 | zigo | After a lot of head-scratching, I can't get how to fix it (note: I already cherry-picked https://github.com/eventlet/eventlet/pull/754/commits/cd2532168e33d892de625f9fc831bf0951f4e937 the collections.abc.Iterable one, and another about ssl_version=ssl.PROTOCOL_TLSv1_2). | |
| 13:29:58 | zigo | The send_method object contains what, in fact? | |
| 13:30:11 | zigo | I see it's self.fd.send_method or something ... | |
| 13:43:33 | gibi | I looked at it but I have no ideas either on that | |
| 14:14:18 | noonedeadpunk | sean-k-mooney: seems that patch revert does work, thanks a lot! | |
| 14:16:17 | sean-k-mooney | we had 3 or 4 supper niche edgcases that we resolve and unfortetly it took use a while to relase that that was nolonger required after we fixed that previous bug | |
| 14:16:34 | sean-k-mooney | noonedeadpunk: im glad this is working for you | |
| 14:34:56 | zigo | gibi: In oslo.concurrency, I tried reverting "Fix fair internal lock used from eventlet.spawn_n" and it was still stuck, so now I'm trying to revert "Prove that spawn_n with fair lock is broken" ... | |
| 14:35:35 | sean-k-mooney | zigo: just an an fyi that fix is needed | |
| 14:35:49 | sean-k-mooney | zigo: without it none of the fair logs in nova actully work | |
| 14:36:07 | zigo | You mean the "Fix fair internal lock used from eventlet.spawn_n" ? | |
| 14:36:14 | sean-k-mooney | yes | |
| 14:36:24 | sean-k-mooney | that is required for correctness | |
| 14:36:26 | zigo | Right, though it doesn't seem to be the brokenness ... | |
| 14:36:31 | sean-k-mooney | ack | |
| 14:39:43 | zigo | It passed ... | |
| 14:40:27 | zigo | Removing https://review.opendev.org/c/openstack/oslo.concurrency/+/855713 fixed my issue. | |
| 14:42:49 | gibi | as I noted earlier you are probably running those test ^^ without monkey patching hence the they stuck | |
| 14:43:20 | gibi | you have no better option now but removing those tests | |
| 14:43:38 | gibi | but you still have to keep the fix form "Fix fair internal lock used from eventlet.spawn_n"as sean-k-mooney noted | |
| 14:44:30 | zigo | I did. | |
| 14:45:09 | zigo | gibi: Is it possible that I'm running into this problem because I'm not doing `TEST_EVENTLET=0 lockutils-wrapper` before stestr run ? | |
| 14:45:23 | gibi | zigo: yes, I think so | |