| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-06-10 | |||
| 11:20:39 | lyarwood | christ | |
| 11:20:41 | lyarwood | :D | |
| 11:20:51 | sean-k-mooney | so there is a better workaround | |
| 11:20:53 | sean-k-mooney | https://github.com/openstack/os-vif/blob/master/vif_plug_ovs/ovs.py#L65-L75 | |
| 11:21:05 | sean-k-mooney | go back to the old cli based backend | |
| 11:22:24 | sean-k-mooney | sorry i proably have to expand on that more | |
| 11:23:00 | sean-k-mooney | [os_vif_ovs]/ovsdb_interface=vsctl | |
| 11:23:07 | lyarwood | tbh if the native approach is broken and stealing this much time from actual n-cpu requests I'd suggest we do that | |
| 11:23:10 | sean-k-mooney | set that in the nova.conf | |
| 11:23:33 | sean-k-mooney | well the native implemation is much much faster espaically at scale | |
| 11:24:20 | sean-k-mooney | but for right now we could go back to the old one which i was ment to delete last cycle until i can fix the native implementation | |
| 11:26:36 | lyarwood | question is do we do this for all CI envs or just the live migration ones? | |
| 11:27:03 | sean-k-mooney | at the scale we operate at in the ci its safe to do it for all if we want too | |
| 11:27:20 | sean-k-mooney | the performace delta only becomes appearent if you have 100s of ports | |
| 11:27:29 | gibi | I'm not be surpised if this polling interferese with out eventlet monkey patching as internally it also patches some eventlet things | |
| 11:27:50 | lyarwood | fun | |
| 11:27:54 | sean-k-mooney | gibi: well os-vif intentionally does not use eventlets | |
| 11:28:16 | sean-k-mooney | although its always or almost always loaded into an enve that is monkypatched already | |
| 11:28:21 | gibi | /usr/lib/python3/dist-packages/ovs/poller.py | |
| 11:28:29 | gibi | i mean | |
| 11:28:30 | gibi | https://github.com/openvswitch/ovs/blob/210c4cba9bc69412473a2fee8e9b6f023150e6e6/python/ovs/poller.py#L270 | |
| 11:28:36 | gibi | it does have eventlet patching | |
| 11:29:09 | sean-k-mooney | that is in the ovs python binding but ya | |
| 11:29:12 | gibi | as far as I see it actually escaping monkey patching | |
| 11:29:44 | gibi | monkey patched by eventlet or gevent library, it gets the original | |
| 11:29:44 | gibi | "If select.poll is | |
| 11:29:45 | gibi | select.poll and returns an object of it" | |
| 11:29:46 | sean-k-mooney | we woudl want the pooling if it cant be disabled to be on a real pthread | |
| 11:30:18 | sean-k-mooney | https://github.com/openvswitch/ovs/blob/210c4cba9bc69412473a2fee8e9b6f023150e6e6/python/ovs/poller.py#L59-L63 | |
| 11:30:45 | gibi | ohh this is nice https://github.com/openvswitch/ovs/blob/210c4cba9bc69412473a2fee8e9b6f023150e6e6/python/ovs/poller.py#L59-L63 | |
| 11:30:52 | gibi | hehe, found the same thing :SD | |
| 11:31:04 | sean-k-mooney | ya that sound very familar | |
| 11:31:11 | opendevmeet | Launchpad bug 1929446 in OpenStack Compute (nova) "check_can_live_migrate_source taking > 60 seconds in CI" [Medium,Triaged] | |
| 11:31:11 | lyarwood | sean-k-mooney: can you update https://bugs.launchpad.net/nova/+bug/1929446 to point to os-vif and update the bug title? | |
| 11:32:25 | sean-k-mooney | i guess but its really in ovs or ovsdbapp. im going to read through the poller implementation and see if we can tweak our usage | |
| 11:32:48 | lyarwood | right but any changes and/or fixes will end up in os-vif right? | |
| 11:33:40 | sean-k-mooney | not nessisarly it could be in ovsdbapp but it wont be in nova | |
| 11:33:59 | sean-k-mooney | if we can fix it in os-vif i might just do it there | |
| 11:34:21 | lyarwood | ack cool sorry the no changes required in nova part was more my point :) | |
| 11:34:31 | sean-k-mooney | yep | |
| 11:35:52 | sean-k-mooney | this is where ovsdbapp is using that poller implementaion https://github.com/openstack/ovsdbapp/blob/master/ovsdbapp/backend/ovs_idl/connection.py#L105 | |
| 11:36:23 | sean-k-mooney | we create an instanice of that oconnection object here https://github.com/openstack/os-vif/blob/master/vif_plug_ovs/ovsdb/impl_idl.py#L31-L33 | |
| 11:37:26 | sean-k-mooney | ovsdbapp is trying to run the poller in a seperate thread https://github.com/openstack/ovsdbapp/blob/master/ovsdbapp/backend/ovs_idl/connection.py#L91 | |
| 11:37:36 | sean-k-mooney | but that is monkypatched | |
| 11:37:57 | sean-k-mooney | really we want that to be a pthread | |
| 11:38:18 | gibi | sean-k-mooney: on hacky thing we can do is to monkey patch ovs.poller.Poller class from os-vif to be an empty implementation | |
| 11:39:00 | sean-k-mooney | i was considering doing something like that | |
| 11:39:40 | sean-k-mooney | i mean i could proablu just use mock to replace it | |
| 11:39:47 | gibi | yeah | |
| 11:42:44 | sean-k-mooney | ok ill see if i can play with this quickly but i think we should really fix this in ovsdbapp by allowing the connect to be created without poolling | |
| 11:43:48 | sean-k-mooney | https://github.com/openstack/ovsdbapp/blob/master/ovsdbapp/backend/ovs_idl/connection.py#L98-L102 | |
| 11:43:52 | sean-k-mooney | this does concern me a bit | |
| 11:44:32 | lyarwood | https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure we really need to clean this out | |
| 11:45:10 | sean-k-mooney | but yes ovs is defently un monkypatching https://github.com/openvswitch/ovs/blob/210c4cba9bc69412473a2fee8e9b6f023150e6e6/python/ovs/poller.py#L270 | |
| 11:55:03 | opendevmeet | Launchpad bug 1793364 in Cinder "mysql db opportunistic unit tests timing out intermittently in the gate (bad thread switch?)" [High,Confirmed] | |
| 11:55:03 | lyarwood | gibi: is https://bugs.launchpad.net/nova/+bug/1793364 resolved now by https://review.opendev.org/q/topic:%22bp%252Fcompact-db-migrations-wallaby%22+(status:open%20OR%20status:merged) ? | |
| 11:55:12 | lyarwood | your comment suggests it was supposed to be | |
| 11:55:55 | gibi | lyarwood: it helped but I think we saw timeouts even after the compation | |
| 11:56:06 | gibi | let me find the info | |
| 11:56:21 | sean-k-mooney | i think we did too | |
| 11:56:26 | lyarwood | kk do we want to keep this open or move it to incomplete and update it with fresh logs etc the next time it happens? | |
| 11:56:37 | gibi | https://review.opendev.org/c/openstack/nova/+/774889/1#message-db5303e41b22ebfb7b33067d815a12ece2df510b | |
| 11:56:57 | gibi | lyarwood: ack | |
| 11:57:07 | gibi | I will check logstash for fresh occurences | |
| 12:00:35 | masterpe[m] | If I run ./nova-manage placement heal_allocations on Train I get the error: "Compute host scpuko57 could not be found." But it is in the "openstack hypervisor list" and "openstack compute service list" | |
| 12:02:10 | opendevmeet | Launchpad bug 1823251 in OpenStack Compute (nova) "Spike in TestNovaMigrationsMySQL.test_walk_versions/test_innodb_tables failures since April 1 2019 on limestone-regionone" [High,Confirmed] | |
| 12:02:10 | gibi | I made 1793364 duplicate of https://bugs.launchpad.net/nova/+bug/1823251 (as the newer report has more info) and added a link to the log of a recent occurence | |
| 12:03:26 | gibi | I have https://review.opendev.org/c/openstack/nova/+/775094 for more logs (now rechecked) but honestly I looked at this problem so many times that I don't think I can solve it. | |
| 12:24:34 | opendevreview | sean mooney proposed openstack/os-vif master: [WIP] mock ovs.poller.Poller https://review.opendev.org/c/openstack/os-vif/+/795770 | |
| 12:25:14 | sean-k-mooney | gibi: that passes the os-vif functional tests which actully create ports in ovs and locally i had it assert that the Poller was called | |
| 12:25:27 | sean-k-mooney | but im not sure if this will actully fix the issue | |
| 12:25:56 | sean-k-mooney | ill take a look at the tempest run when its done and we can see if the repeating dbug message is still present or not | |
| 12:29:21 | gibi | sean-k-mooney: cool | |
| 12:29:21 | kashyap | lyarwood: </me back after eclipse hunting> Hey. Have you got a bug filed, or shall I file one? | |
| 12:29:41 | kashyap | [OT] A couple of pictures of projections, if you missed it: https://kashyapc.fedorapeople.org/partial_solar_eclipse_2021/ | |
| 12:29:47 | lyarwood | kashyap: I don't yet so feel free to write one up if you have time | |
| 12:29:59 | kashyap | lyarwood: Sure; I'll do it right now | |
| 12:30:16 | kashyap | But would be good to see if we can reproduce this at least twice... | |
| 12:32:09 | gibi | kashyap: I think we hit it 7 times in the last 7 days http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22libvirt.libvirtError%3A%20unable%20to%20connect%20to%20server%20at%5C%22 | |
| 12:32:57 | kashyap | gibi: Ah, that's good to know. | |
| 12:33:13 | kashyap | So the trigger is block-migrating a puased instance | |
| 12:33:35 | kashyap | (Where "block-migrating" == migrating the instance along with its storage) | |
| 13:21:38 | opendevmeet | Launchpad bug 1912310 in OpenStack Compute (nova) "libvirt.libvirtError: unable to connect to server at " [Medium,Confirmed] | |
| 13:21:38 | lyarwood | kashyap: https://bugs.launchpad.net/nova/+bug/1912310 found an older bug we could mark as a duplicate if you've already written a fresh bug up | |
| 13:22:35 | kashyap | lyarwood: Ah, good find; I'm just drafting it locally | |
| 13:22:43 | opendevmeet | Launchpad bug 1797185 in OpenStack Compute (nova) "live migration intermittently fails in CI with "Connection refused" during guest transfer" [Low,Confirmed] | |
| 13:22:43 | lyarwood | https://bugs.launchpad.net/nova/+bug/1797185 is another | |
| 13:22:48 | kashyap | Sigh | |
| 13:23:33 | kashyap | lyarwood: What I wonder is - how can we check _why_ the destination is unreahable... | |
| 13:23:41 | kashyap | Is it something peculiar to our upstream CI; or something else | |
| 13:26:10 | lyarwood | I honestly think it's a bug in the libvirt python bindings where they miss that the migration has already failed, try to poll it's progress and don't handle the fact that it's no longer there correctly | |
| 13:26:23 | lyarwood | the unreachable error is a red herring IMHO | |
| 13:27:47 | kashyap | lyarwood: Yeah; I asked Michal from libvirt; and he suggested Jiri Denemark | |
| 13:28:07 | kashyap | He isn't around; but I'll ask him to comment on this once he's back | |
| 13:28:24 | kashyap | lyarwood: Also, you only speak of Python bindings - why won't it be a bug in the C API itself? | |
| 13:28:37 | kashyap | Asking out of ignorance, not challenging :) | |
| 13:28:49 | bauzas | hola | |
| 13:29:00 | bauzas | lyarwood: gibican we know how many jobs have problems ? | |
| 13:29:04 | bauzas | gibi: ^ | |
| 13:29:35 | bauzas | if we have problems with fixing those problems, can we make those specific jobs non-voting until we fix the main issues ? | |