| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-06-10 | |||
| 11:28:29 | gibi | i mean | |
| 11:28:30 | gibi | https://github.com/openvswitch/ovs/blob/210c4cba9bc69412473a2fee8e9b6f023150e6e6/python/ovs/poller.py#L270 | |
| 11:28:36 | gibi | it does have eventlet patching | |
| 11:29:09 | sean-k-mooney | that is in the ovs python binding but ya | |
| 11:29:12 | gibi | as far as I see it actually escaping monkey patching | |
| 11:29:44 | gibi | "If select.poll is | |
| 11:29:44 | gibi | monkey patched by eventlet or gevent library, it gets the original | |
| 11:29:45 | gibi | select.poll and returns an object of it" | |
| 11:29:46 | sean-k-mooney | we woudl want the pooling if it cant be disabled to be on a real pthread | |
| 11:30:18 | sean-k-mooney | https://github.com/openvswitch/ovs/blob/210c4cba9bc69412473a2fee8e9b6f023150e6e6/python/ovs/poller.py#L59-L63 | |
| 11:30:45 | gibi | ohh this is nice https://github.com/openvswitch/ovs/blob/210c4cba9bc69412473a2fee8e9b6f023150e6e6/python/ovs/poller.py#L59-L63 | |
| 11:30:52 | gibi | hehe, found the same thing :SD | |
| 11:31:04 | sean-k-mooney | ya that sound very familar | |
| 11:31:11 | lyarwood | sean-k-mooney: can you update https://bugs.launchpad.net/nova/+bug/1929446 to point to os-vif and update the bug title? | |
| 11:31:11 | opendevmeet | Launchpad bug 1929446 in OpenStack Compute (nova) "check_can_live_migrate_source taking > 60 seconds in CI" [Medium,Triaged] | |
| 11:32:25 | sean-k-mooney | i guess but its really in ovs or ovsdbapp. im going to read through the poller implementation and see if we can tweak our usage | |
| 11:32:48 | lyarwood | right but any changes and/or fixes will end up in os-vif right? | |
| 11:33:40 | sean-k-mooney | not nessisarly it could be in ovsdbapp but it wont be in nova | |
| 11:33:59 | sean-k-mooney | if we can fix it in os-vif i might just do it there | |
| 11:34:21 | lyarwood | ack cool sorry the no changes required in nova part was more my point :) | |
| 11:34:31 | sean-k-mooney | yep | |
| 11:35:52 | sean-k-mooney | this is where ovsdbapp is using that poller implementaion https://github.com/openstack/ovsdbapp/blob/master/ovsdbapp/backend/ovs_idl/connection.py#L105 | |
| 11:36:23 | sean-k-mooney | we create an instanice of that oconnection object here https://github.com/openstack/os-vif/blob/master/vif_plug_ovs/ovsdb/impl_idl.py#L31-L33 | |
| 11:37:26 | sean-k-mooney | ovsdbapp is trying to run the poller in a seperate thread https://github.com/openstack/ovsdbapp/blob/master/ovsdbapp/backend/ovs_idl/connection.py#L91 | |
| 11:37:36 | sean-k-mooney | but that is monkypatched | |
| 11:37:57 | sean-k-mooney | really we want that to be a pthread | |
| 11:38:18 | gibi | sean-k-mooney: on hacky thing we can do is to monkey patch ovs.poller.Poller class from os-vif to be an empty implementation | |
| 11:39:00 | sean-k-mooney | i was considering doing something like that | |
| 11:39:40 | sean-k-mooney | i mean i could proablu just use mock to replace it | |
| 11:39:47 | gibi | yeah | |
| 11:42:44 | sean-k-mooney | ok ill see if i can play with this quickly but i think we should really fix this in ovsdbapp by allowing the connect to be created without poolling | |
| 11:43:48 | sean-k-mooney | https://github.com/openstack/ovsdbapp/blob/master/ovsdbapp/backend/ovs_idl/connection.py#L98-L102 | |
| 11:43:52 | sean-k-mooney | this does concern me a bit | |
| 11:44:32 | lyarwood | https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure we really need to clean this out | |
| 11:45:10 | sean-k-mooney | but yes ovs is defently un monkypatching https://github.com/openvswitch/ovs/blob/210c4cba9bc69412473a2fee8e9b6f023150e6e6/python/ovs/poller.py#L270 | |
| 11:55:03 | lyarwood | gibi: is https://bugs.launchpad.net/nova/+bug/1793364 resolved now by https://review.opendev.org/q/topic:%22bp%252Fcompact-db-migrations-wallaby%22+(status:open%20OR%20status:merged) ? | |
| 11:55:03 | opendevmeet | Launchpad bug 1793364 in Cinder "mysql db opportunistic unit tests timing out intermittently in the gate (bad thread switch?)" [High,Confirmed] | |
| 11:55:12 | lyarwood | your comment suggests it was supposed to be | |
| 11:55:55 | gibi | lyarwood: it helped but I think we saw timeouts even after the compation | |
| 11:56:06 | gibi | let me find the info | |
| 11:56:21 | sean-k-mooney | i think we did too | |
| 11:56:26 | lyarwood | kk do we want to keep this open or move it to incomplete and update it with fresh logs etc the next time it happens? | |
| 11:56:37 | gibi | https://review.opendev.org/c/openstack/nova/+/774889/1#message-db5303e41b22ebfb7b33067d815a12ece2df510b | |
| 11:56:57 | gibi | lyarwood: ack | |
| 11:57:07 | gibi | I will check logstash for fresh occurences | |
| 12:00:35 | masterpe[m] | If I run ./nova-manage placement heal_allocations on Train I get the error: "Compute host scpuko57 could not be found." But it is in the "openstack hypervisor list" and "openstack compute service list" | |
| 12:02:10 | gibi | I made 1793364 duplicate of https://bugs.launchpad.net/nova/+bug/1823251 (as the newer report has more info) and added a link to the log of a recent occurence | |
| 12:02:10 | opendevmeet | Launchpad bug 1823251 in OpenStack Compute (nova) "Spike in TestNovaMigrationsMySQL.test_walk_versions/test_innodb_tables failures since April 1 2019 on limestone-regionone" [High,Confirmed] | |
| 12:03:26 | gibi | I have https://review.opendev.org/c/openstack/nova/+/775094 for more logs (now rechecked) but honestly I looked at this problem so many times that I don't think I can solve it. | |
| 12:24:34 | opendevreview | sean mooney proposed openstack/os-vif master: [WIP] mock ovs.poller.Poller https://review.opendev.org/c/openstack/os-vif/+/795770 | |
| 12:25:14 | sean-k-mooney | gibi: that passes the os-vif functional tests which actully create ports in ovs and locally i had it assert that the Poller was called | |
| 12:25:27 | sean-k-mooney | but im not sure if this will actully fix the issue | |
| 12:25:56 | sean-k-mooney | ill take a look at the tempest run when its done and we can see if the repeating dbug message is still present or not | |
| 12:29:21 | kashyap | lyarwood: </me back after eclipse hunting> Hey. Have you got a bug filed, or shall I file one? | |
| 12:29:21 | gibi | sean-k-mooney: cool | |
| 12:29:41 | kashyap | [OT] A couple of pictures of projections, if you missed it: https://kashyapc.fedorapeople.org/partial_solar_eclipse_2021/ | |
| 12:29:47 | lyarwood | kashyap: I don't yet so feel free to write one up if you have time | |
| 12:29:59 | kashyap | lyarwood: Sure; I'll do it right now | |
| 12:30:16 | kashyap | But would be good to see if we can reproduce this at least twice... | |
| 12:32:09 | gibi | kashyap: I think we hit it 7 times in the last 7 days http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22libvirt.libvirtError%3A%20unable%20to%20connect%20to%20server%20at%5C%22 | |
| 12:32:57 | kashyap | gibi: Ah, that's good to know. | |
| 12:33:13 | kashyap | So the trigger is block-migrating a puased instance | |
| 12:33:35 | kashyap | (Where "block-migrating" == migrating the instance along with its storage) | |
| 13:21:38 | lyarwood | kashyap: https://bugs.launchpad.net/nova/+bug/1912310 found an older bug we could mark as a duplicate if you've already written a fresh bug up | |
| 13:21:38 | opendevmeet | Launchpad bug 1912310 in OpenStack Compute (nova) "libvirt.libvirtError: unable to connect to server at " [Medium,Confirmed] | |
| 13:22:35 | kashyap | lyarwood: Ah, good find; I'm just drafting it locally | |
| 13:22:43 | lyarwood | https://bugs.launchpad.net/nova/+bug/1797185 is another | |
| 13:22:43 | opendevmeet | Launchpad bug 1797185 in OpenStack Compute (nova) "live migration intermittently fails in CI with "Connection refused" during guest transfer" [Low,Confirmed] | |
| 13:22:48 | kashyap | Sigh | |
| 13:23:33 | kashyap | lyarwood: What I wonder is - how can we check _why_ the destination is unreahable... | |
| 13:23:41 | kashyap | Is it something peculiar to our upstream CI; or something else | |
| 13:26:10 | lyarwood | I honestly think it's a bug in the libvirt python bindings where they miss that the migration has already failed, try to poll it's progress and don't handle the fact that it's no longer there correctly | |
| 13:26:23 | lyarwood | the unreachable error is a red herring IMHO | |
| 13:27:47 | kashyap | lyarwood: Yeah; I asked Michal from libvirt; and he suggested Jiri Denemark | |
| 13:28:07 | kashyap | He isn't around; but I'll ask him to comment on this once he's back | |
| 13:28:24 | kashyap | lyarwood: Also, you only speak of Python bindings - why won't it be a bug in the C API itself? | |
| 13:28:37 | kashyap | Asking out of ignorance, not challenging :) | |
| 13:28:49 | bauzas | hola | |
| 13:29:00 | bauzas | lyarwood: gibican we know how many jobs have problems ? | |
| 13:29:04 | bauzas | gibi: ^ | |
| 13:29:35 | bauzas | if we have problems with fixing those problems, can we make those specific jobs non-voting until we fix the main issues ? | |
| 13:29:39 | lyarwood | kashyap: it could be that as well but n-cpu is calling the python bindings and they are raising the error here so it easily be an issue there | |
| 13:30:20 | bauzas | fwiw, I saw that https://review.opendev.org/c/openstack/nova/+/791506 passed the check pipeline but we don't know yet whether the gate one will accept it | |
| 13:30:34 | kashyap | lyarwood: Ah, okay; you're just speaking of the error trace in our case. Reasonable | |
| 13:31:08 | kashyap | lyarwood: Can you copy/paste your comment from the review to start with in there? | |
| 13:31:18 | kashyap | (In the latest bug you filed 2021 Jan) | |
| 13:31:30 | lyarwood | bauzas: There are various intermittent failures at the moment, I'm trying to clean up the gate-failures bug tag to get a better handle on what is failing and how often | |
| 13:32:02 | lyarwood | bauzas: we could move them to non-voting but this has been going on for weeks so I'd rather we try to get a better handle on this first | |
| 13:32:04 | bauzas | okay, fair enough | |
| 13:32:27 | lyarwood | I honestly think we haven't been landing enough to notice recently | |
| 13:32:28 | bauzas | lyarwood: I guess my point is that we're still having *all* the changes getting -1 because of the mypy issue | |
| 13:32:51 | bauzas | so I want the gate back asap | |
| 13:33:06 | lyarwood | yup fair | |
| 13:33:07 | bauzas | we have two possibilities, the main fix https://review.opendev.org/c/openstack/nova/+/791506 | |
| 13:33:38 | bauzas | but I also prepared an alternative change that's simplier for the gate and doesn't hit the transient issues we know https://review.opendev.org/c/openstack/nova/+/795744 | |
| 13:34:17 | bauzas | so, I guess, I'd recommend to wait for the main change to merge, but if we get a Zuul -2 from the gate on it, I'd honestly recommend to let https://review.opendev.org/c/openstack/nova/+/795744 go | |
| 13:42:23 | lyarwood | sean-k-mooney: https://bugs.launchpad.net/nova/+bug/1863889/comments/3 another possible hit of the ovsdbapp issue | |
| 13:42:23 | opendevmeet | Launchpad bug 1863889 in OpenStack Compute (nova) "Revert resize problem in neutron-tempest-dvr-ha-multinode-full" [Medium,Confirmed] | |
| 13:44:06 | sean-k-mooney | lyarwood: im talking to otherwiseguy and ralonsoh in #openstack-neutron about it now but ill let them know | |
| 13:44:28 | lyarwood | it's just another random timeout FWIW | |