Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-06-10
10:55:17 gibi this is the recent grenade job failure on https://review.opendev.org/c/openstack/nova/+/795533
10:55:57 gibi but I think this 'failed to reach available status (current detaching)' and 'failed to reach available status (current in-use)' are pretty common failures
10:56:21 lyarwood right it's still the detach logic
10:56:59 lyarwood https://zuul.opendev.org/t/openstack/build/24485b8c450740c5946c39dcf310a746/log/controller/logs/screen-n-cpu.txt#25665
10:57:16 lyarwood I had a few hits of this last week and wanted to dump the instance console on failure
10:57:31 lyarwood led me down a rabbit hole that I've not had time to revisit this week
10:57:48 lyarwood https://review.opendev.org/c/openstack/tempest/+/794757 was my initial attempt
10:58:01 gibi ohh
10:58:13 gibi thanks for the info
10:58:23 gibi I went to the cinder side
10:58:26 gibi and lost
11:00:24 gibi lyarwood: what I saw that the tempest sent a volume detachment https://zuul.opendev.org/t/openstack/build/24485b8c450740c5946c39dcf310a746/log/job-output.txt#58720 and that led to the volume being in detaching state
11:02:05 gibi and that succeeded in nova https://zuul.opendev.org/t/openstack/build/24485b8c450740c5946c39dcf310a746/log/controller/logs/screen-n-cpu.txt#23456
11:03:37 gibi ohh it does not
11:03:48 gibi it oly succeded from the persistent domain
11:04:10 lyarwood right req-6f7d27cf-3d82-4e8d-ad25-7f53249a05a0 fails to detach the volume from the live domain
11:04:17 gibi yeah, now I see
11:04:54 lyarwood I traced this all through previously and couldn't see any issues with n-cpu, libvirt or even QEMU tbh
11:05:04 lyarwood so I wanted to see what the state of the guest OS was
11:05:14 lyarwood before I reported a bug to the QEMU folks
11:05:18 lyarwood and/or cirros
11:07:33 lyarwood https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ec1/795533/5/check/nova-live-migration/ec1e40c/testr_results.html - the latest run has failed in nova-live-migration
11:07:58 lyarwood with a timeout during the initial POST, odd.
11:08:57 lyarwood oh I wonder if it's https://bugs.launchpad.net/nova/+bug/1929446
11:08:57 opendevmeet Launchpad bug 1929446 in OpenStack Compute (nova) "check_can_live_migrate_source taking > 60 seconds in CI" [Medium,Triaged]
11:09:14 lyarwood sean-k-mooney: ^ did you get anywhere with that?
11:09:21 bauzas fwiw, the mypy removal is green from the CI https://review.opendev.org/c/openstack/nova/+/795744
11:11:11 lyarwood It's still likely to fail in the actual gate with all of these failures however right?
11:12:45 sean-k-mooney i think its stil ovsdbapp but i did not see a way to stop the polling in the lib
11:13:32 sean-k-mooney i can take another look im wondering if we need to move the polling to the prive sep deamon or into a real pthread
11:16:16 lyarwood sean-k-mooney: I still don't understand the timing here
11:16:42 lyarwood sean-k-mooney: is there a long running os-vif thing in the background or is it related to check_can_live_migrate_source?
11:17:13 sean-k-mooney the first
11:17:32 lyarwood kk the short term workaround is just to bump the rpc timeout I guess
11:17:45 lyarwood just in the LM jobs
11:18:21 sean-k-mooney or rather os-vif usese ovsdbapp which create a connection to ovs and then it kicks off a pooling loop that monitors ovs for the addtion and removal of ports
11:18:32 sean-k-mooney os-vif never uses that feature of ovsdbapp
11:18:48 sean-k-mooney it is used by the neutron l2 agent which uses ovsdbapp directly
11:18:57 sean-k-mooney to know when we add and remove vm ports
11:19:15 sean-k-mooney but ovsdbapp appears to not have an obvios way to turn it off
11:19:39 lyarwood ah so we do this once via os-vif and ovsdbapp keeps polling in the background forever?
11:20:19 sean-k-mooney yep
11:20:24 lyarwood ewwww
11:20:28 sean-k-mooney im sure you have seen it in the debug logs
11:20:29 lyarwood so we don't even need this?!
11:20:34 sean-k-mooney correct
11:20:36 lyarwood yeah I just assumed we needed it
11:20:39 lyarwood christ
11:20:41 lyarwood :D
11:20:51 sean-k-mooney so there is a better workaround
11:20:53 sean-k-mooney https://github.com/openstack/os-vif/blob/master/vif_plug_ovs/ovs.py#L65-L75
11:21:05 sean-k-mooney go back to the old cli based backend
11:22:24 sean-k-mooney sorry i proably have to expand on that more
11:23:00 sean-k-mooney [os_vif_ovs]/ovsdb_interface=vsctl
11:23:07 lyarwood tbh if the native approach is broken and stealing this much time from actual n-cpu requests I'd suggest we do that
11:23:10 sean-k-mooney set that in the nova.conf
11:23:33 sean-k-mooney well the native implemation is much much faster espaically at scale
11:24:20 sean-k-mooney but for right now we could go back to the old one which i was ment to delete last cycle until i can fix the native implementation
11:26:36 lyarwood question is do we do this for all CI envs or just the live migration ones?
11:27:03 sean-k-mooney at the scale we operate at in the ci its safe to do it for all if we want too
11:27:20 sean-k-mooney the performace delta only becomes appearent if you have 100s of ports
11:27:29 gibi I'm not be surpised if this polling interferese with out eventlet monkey patching as internally it also patches some eventlet things
11:27:50 lyarwood fun
11:27:54 sean-k-mooney gibi: well os-vif intentionally does not use eventlets
11:28:16 sean-k-mooney although its always or almost always loaded into an enve that is monkypatched already
11:28:21 gibi /usr/lib/python3/dist-packages/ovs/poller.py
11:28:29 gibi i mean
11:28:30 gibi https://github.com/openvswitch/ovs/blob/210c4cba9bc69412473a2fee8e9b6f023150e6e6/python/ovs/poller.py#L270
11:28:36 gibi it does have eventlet patching
11:29:09 sean-k-mooney that is in the ovs python binding but ya
11:29:12 gibi as far as I see it actually escaping monkey patching
11:29:44 gibi "If select.poll is
11:29:44 gibi monkey patched by eventlet or gevent library, it gets the original
11:29:45 gibi select.poll and returns an object of it"
11:29:46 sean-k-mooney we woudl want the pooling if it cant be disabled to be on a real pthread
11:30:18 sean-k-mooney https://github.com/openvswitch/ovs/blob/210c4cba9bc69412473a2fee8e9b6f023150e6e6/python/ovs/poller.py#L59-L63
11:30:45 gibi ohh this is nice https://github.com/openvswitch/ovs/blob/210c4cba9bc69412473a2fee8e9b6f023150e6e6/python/ovs/poller.py#L59-L63
11:30:52 gibi hehe, found the same thing :SD
11:31:04 sean-k-mooney ya that sound very familar
11:31:11 lyarwood sean-k-mooney: can you update https://bugs.launchpad.net/nova/+bug/1929446 to point to os-vif and update the bug title?
11:31:11 opendevmeet Launchpad bug 1929446 in OpenStack Compute (nova) "check_can_live_migrate_source taking > 60 seconds in CI" [Medium,Triaged]
11:32:25 sean-k-mooney i guess but its really in ovs or ovsdbapp. im going to read through the poller implementation and see if we can tweak our usage
11:32:48 lyarwood right but any changes and/or fixes will end up in os-vif right?
11:33:40 sean-k-mooney not nessisarly it could be in ovsdbapp but it wont be in nova
11:33:59 sean-k-mooney if we can fix it in os-vif i might just do it there
11:34:21 lyarwood ack cool sorry the no changes required in nova part was more my point :)
11:34:31 sean-k-mooney yep
11:35:52 sean-k-mooney this is where ovsdbapp is using that poller implementaion https://github.com/openstack/ovsdbapp/blob/master/ovsdbapp/backend/ovs_idl/connection.py#L105
11:36:23 sean-k-mooney we create an instanice of that oconnection object here https://github.com/openstack/os-vif/blob/master/vif_plug_ovs/ovsdb/impl_idl.py#L31-L33
11:37:26 sean-k-mooney ovsdbapp is trying to run the poller in a seperate thread https://github.com/openstack/ovsdbapp/blob/master/ovsdbapp/backend/ovs_idl/connection.py#L91
11:37:36 sean-k-mooney but that is monkypatched
11:37:57 sean-k-mooney really we want that to be a pthread
11:38:18 gibi sean-k-mooney: on hacky thing we can do is to monkey patch ovs.poller.Poller class from os-vif to be an empty implementation
11:39:00 sean-k-mooney i was considering doing something like that
11:39:40 sean-k-mooney i mean i could proablu just use mock to replace it
11:39:47 gibi yeah
11:42:44 sean-k-mooney ok ill see if i can play with this quickly but i think we should really fix this in ovsdbapp by allowing the connect to be created without poolling
11:43:48 sean-k-mooney https://github.com/openstack/ovsdbapp/blob/master/ovsdbapp/backend/ovs_idl/connection.py#L98-L102
11:43:52 sean-k-mooney this does concern me a bit
11:44:32 lyarwood https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure we really need to clean this out

Earlier   Later