Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-06-10
11:17:13 sean-k-mooney the first
11:17:32 lyarwood kk the short term workaround is just to bump the rpc timeout I guess
11:17:45 lyarwood just in the LM jobs
11:18:21 sean-k-mooney or rather os-vif usese ovsdbapp which create a connection to ovs and then it kicks off a pooling loop that monitors ovs for the addtion and removal of ports
11:18:32 sean-k-mooney os-vif never uses that feature of ovsdbapp
11:18:48 sean-k-mooney it is used by the neutron l2 agent which uses ovsdbapp directly
11:18:57 sean-k-mooney to know when we add and remove vm ports
11:19:15 sean-k-mooney but ovsdbapp appears to not have an obvios way to turn it off
11:19:39 lyarwood ah so we do this once via os-vif and ovsdbapp keeps polling in the background forever?
11:20:19 sean-k-mooney yep
11:20:24 lyarwood ewwww
11:20:28 sean-k-mooney im sure you have seen it in the debug logs
11:20:29 lyarwood so we don't even need this?!
11:20:34 sean-k-mooney correct
11:20:36 lyarwood yeah I just assumed we needed it
11:20:39 lyarwood christ
11:20:41 lyarwood :D
11:20:51 sean-k-mooney so there is a better workaround
11:20:53 sean-k-mooney https://github.com/openstack/os-vif/blob/master/vif_plug_ovs/ovs.py#L65-L75
11:21:05 sean-k-mooney go back to the old cli based backend
11:22:24 sean-k-mooney sorry i proably have to expand on that more
11:23:00 sean-k-mooney [os_vif_ovs]/ovsdb_interface=vsctl
11:23:07 lyarwood tbh if the native approach is broken and stealing this much time from actual n-cpu requests I'd suggest we do that
11:23:10 sean-k-mooney set that in the nova.conf
11:23:33 sean-k-mooney well the native implemation is much much faster espaically at scale
11:24:20 sean-k-mooney but for right now we could go back to the old one which i was ment to delete last cycle until i can fix the native implementation
11:26:36 lyarwood question is do we do this for all CI envs or just the live migration ones?
11:27:03 sean-k-mooney at the scale we operate at in the ci its safe to do it for all if we want too
11:27:20 sean-k-mooney the performace delta only becomes appearent if you have 100s of ports
11:27:29 gibi I'm not be surpised if this polling interferese with out eventlet monkey patching as internally it also patches some eventlet things
11:27:50 lyarwood fun
11:27:54 sean-k-mooney gibi: well os-vif intentionally does not use eventlets
11:28:16 sean-k-mooney although its always or almost always loaded into an enve that is monkypatched already
11:28:21 gibi /usr/lib/python3/dist-packages/ovs/poller.py
11:28:29 gibi i mean
11:28:30 gibi https://github.com/openvswitch/ovs/blob/210c4cba9bc69412473a2fee8e9b6f023150e6e6/python/ovs/poller.py#L270
11:28:36 gibi it does have eventlet patching
11:29:09 sean-k-mooney that is in the ovs python binding but ya
11:29:12 gibi as far as I see it actually escaping monkey patching
11:29:44 gibi monkey patched by eventlet or gevent library, it gets the original
11:29:44 gibi "If select.poll is
11:29:45 gibi select.poll and returns an object of it"
11:29:46 sean-k-mooney we woudl want the pooling if it cant be disabled to be on a real pthread
11:30:18 sean-k-mooney https://github.com/openvswitch/ovs/blob/210c4cba9bc69412473a2fee8e9b6f023150e6e6/python/ovs/poller.py#L59-L63
11:30:45 gibi ohh this is nice https://github.com/openvswitch/ovs/blob/210c4cba9bc69412473a2fee8e9b6f023150e6e6/python/ovs/poller.py#L59-L63
11:30:52 gibi hehe, found the same thing :SD
11:31:04 sean-k-mooney ya that sound very familar
11:31:11 opendevmeet Launchpad bug 1929446 in OpenStack Compute (nova) "check_can_live_migrate_source taking > 60 seconds in CI" [Medium,Triaged]
11:31:11 lyarwood sean-k-mooney: can you update https://bugs.launchpad.net/nova/+bug/1929446 to point to os-vif and update the bug title?
11:32:25 sean-k-mooney i guess but its really in ovs or ovsdbapp. im going to read through the poller implementation and see if we can tweak our usage
11:32:48 lyarwood right but any changes and/or fixes will end up in os-vif right?
11:33:40 sean-k-mooney not nessisarly it could be in ovsdbapp but it wont be in nova
11:33:59 sean-k-mooney if we can fix it in os-vif i might just do it there
11:34:21 lyarwood ack cool sorry the no changes required in nova part was more my point :)
11:34:31 sean-k-mooney yep
11:35:52 sean-k-mooney this is where ovsdbapp is using that poller implementaion https://github.com/openstack/ovsdbapp/blob/master/ovsdbapp/backend/ovs_idl/connection.py#L105
11:36:23 sean-k-mooney we create an instanice of that oconnection object here https://github.com/openstack/os-vif/blob/master/vif_plug_ovs/ovsdb/impl_idl.py#L31-L33
11:37:26 sean-k-mooney ovsdbapp is trying to run the poller in a seperate thread https://github.com/openstack/ovsdbapp/blob/master/ovsdbapp/backend/ovs_idl/connection.py#L91
11:37:36 sean-k-mooney but that is monkypatched
11:37:57 sean-k-mooney really we want that to be a pthread
11:38:18 gibi sean-k-mooney: on hacky thing we can do is to monkey patch ovs.poller.Poller class from os-vif to be an empty implementation
11:39:00 sean-k-mooney i was considering doing something like that
11:39:40 sean-k-mooney i mean i could proablu just use mock to replace it
11:39:47 gibi yeah
11:42:44 sean-k-mooney ok ill see if i can play with this quickly but i think we should really fix this in ovsdbapp by allowing the connect to be created without poolling
11:43:48 sean-k-mooney https://github.com/openstack/ovsdbapp/blob/master/ovsdbapp/backend/ovs_idl/connection.py#L98-L102
11:43:52 sean-k-mooney this does concern me a bit
11:44:32 lyarwood https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure we really need to clean this out
11:45:10 sean-k-mooney but yes ovs is defently un monkypatching https://github.com/openvswitch/ovs/blob/210c4cba9bc69412473a2fee8e9b6f023150e6e6/python/ovs/poller.py#L270
11:55:03 opendevmeet Launchpad bug 1793364 in Cinder "mysql db opportunistic unit tests timing out intermittently in the gate (bad thread switch?)" [High,Confirmed]
11:55:03 lyarwood gibi: is https://bugs.launchpad.net/nova/+bug/1793364 resolved now by https://review.opendev.org/q/topic:%22bp%252Fcompact-db-migrations-wallaby%22+(status:open%20OR%20status:merged) ?
11:55:12 lyarwood your comment suggests it was supposed to be
11:55:55 gibi lyarwood: it helped but I think we saw timeouts even after the compation
11:56:06 gibi let me find the info
11:56:21 sean-k-mooney i think we did too
11:56:26 lyarwood kk do we want to keep this open or move it to incomplete and update it with fresh logs etc the next time it happens?
11:56:37 gibi https://review.opendev.org/c/openstack/nova/+/774889/1#message-db5303e41b22ebfb7b33067d815a12ece2df510b
11:56:57 gibi lyarwood: ack
11:57:07 gibi I will check logstash for fresh occurences
12:00:35 masterpe[m] If I run ./nova-manage placement heal_allocations on Train I get the error: "Compute host scpuko57 could not be found." But it is in the "openstack hypervisor list" and "openstack compute service list"
12:02:10 opendevmeet Launchpad bug 1823251 in OpenStack Compute (nova) "Spike in TestNovaMigrationsMySQL.test_walk_versions/test_innodb_tables failures since April 1 2019 on limestone-regionone" [High,Confirmed]
12:02:10 gibi I made 1793364 duplicate of https://bugs.launchpad.net/nova/+bug/1823251 (as the newer report has more info) and added a link to the log of a recent occurence
12:03:26 gibi I have https://review.opendev.org/c/openstack/nova/+/775094 for more logs (now rechecked) but honestly I looked at this problem so many times that I don't think I can solve it.
12:24:34 opendevreview sean mooney proposed openstack/os-vif master: [WIP] mock ovs.poller.Poller https://review.opendev.org/c/openstack/os-vif/+/795770
12:25:14 sean-k-mooney gibi: that passes the os-vif functional tests which actully create ports in ovs and locally i had it assert that the Poller was called
12:25:27 sean-k-mooney but im not sure if this will actully fix the issue
12:25:56 sean-k-mooney ill take a look at the tempest run when its done and we can see if the repeating dbug message is still present or not
12:29:21 gibi sean-k-mooney: cool
12:29:21 kashyap lyarwood: </me back after eclipse hunting> Hey. Have you got a bug filed, or shall I file one?
12:29:41 kashyap [OT] A couple of pictures of projections, if you missed it: https://kashyapc.fedorapeople.org/partial_solar_eclipse_2021/
12:29:47 lyarwood kashyap: I don't yet so feel free to write one up if you have time
12:29:59 kashyap lyarwood: Sure; I'll do it right now
12:30:16 kashyap But would be good to see if we can reproduce this at least twice...
12:32:09 gibi kashyap: I think we hit it 7 times in the last 7 days http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22libvirt.libvirtError%3A%20unable%20to%20connect%20to%20server%20at%5C%22
12:32:57 kashyap gibi: Ah, that's good to know.
12:33:13 kashyap So the trigger is block-migrating a puased instance
12:33:35 kashyap (Where "block-migrating" == migrating the instance along with its storage)
13:21:38 opendevmeet Launchpad bug 1912310 in OpenStack Compute (nova) "libvirt.libvirtError: unable to connect to server at " [Medium,Confirmed]
13:21:38 lyarwood kashyap: https://bugs.launchpad.net/nova/+bug/1912310 found an older bug we could mark as a duplicate if you've already written a fresh bug up
13:22:35 kashyap lyarwood: Ah, good find; I'm just drafting it locally

Earlier   Later