| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-05-18 | |||
| 11:25:36 | kashyap | Yeah | |
| 11:25:44 | gibi | it is related to another SSHABLE tempest fix in https://review.opendev.org/c/openstack/tempest/+/817772 | |
| 11:33:28 | tobias-urdin | sean-k-mooney: please have a quick look at https://review.opendev.org/c/openstack/nova/+/838976 when you have time, i need to go afk for a while but will be back later today | |
| 11:35:43 | sean-k-mooney | sure | |
| 11:36:05 | sean-k-mooney | oh this is for the nodedev name change | |
| 11:36:09 | sean-k-mooney | ya im aware of that | |
| 11:36:26 | tobias-urdin | ack :) | |
| 11:37:19 | gibi | I've just finished reviewing that | |
| 11:37:40 | gibi | I have some comments and a request for tests | |
| 11:38:39 | sean-k-mooney | just seeing them as i click | |
| 11:47:31 | sean-k-mooney | gibi: replied inline | |
| 11:51:29 | gibi | thanks | |
| 12:44:54 | opendevreview | Merged openstack/nova stable/train: [stable-only] Drop lower-constraints job https://review.opendev.org/c/openstack/nova/+/838037 | |
| 12:49:29 | opendevreview | Mohammed Naser proposed openstack/nova master: Fix race condition in _get_pci_passthrough_devices https://review.opendev.org/c/openstack/nova/+/840993 | |
| 13:07:25 | sean-k-mooney | gibi: are you working on https://bugs.launchpad.net/nova/+bug/1971760 i was going to try and find time today to push a patch to make spawn_n actully be spwan and see if that helps | |
| 13:08:31 | gibi | sean-k-mooney: I have an env where I try to reproduce the leak itself but I had no time to kick that env enough. I'm not sure I can reproduce the leak in a reproducible way | |
| 13:08:43 | gibi | so it is hard to test any changesd | |
| 13:09:21 | sean-k-mooney | i think it will hapen if a thread that is invoked with spawn_n raises an excpeiton | |
| 13:09:35 | sean-k-mooney | that is not caugt before the entry porint of the spawn_n call | |
| 13:09:43 | gibi | I tried that | |
| 13:09:48 | gibi | it does not create the leak | |
| 13:10:07 | gibi | I tried vif plug timeout | |
| 13:10:21 | sean-k-mooney | well my other tought was maybe itst related to https://bugs.launchpad.net/oslo.messaging/+bug/1949964 | |
| 13:10:42 | gibi | I havent looked at ^^ yet | |
| 13:11:24 | gibi | so feel free to propose a patch but it will be hard to prove it solved the issue except if mnaser are willing to take that patch to his env and let it running for a while | |
| 13:16:10 | sean-k-mooney | i was hoping to be able to tweak https://github.com/eventlet/eventlet/issues/731#issue-1032856809 to repoduce it but ya i just want to see if we do https://github.com/eventlet/eventlet/issues/731#issuecomment-968135262 will it help | |
| 13:16:45 | sean-k-mooney | well will it work with nova | |
| 13:17:12 | sean-k-mooney | i was considering putting it behind a workaround config option so we could get feedback | |
| 13:18:35 | gibi | yeah either we need a way to reproduce or we need to make this optional and ask for feedback | |
| 13:18:56 | gibi | overall I don't see problems replacing our spawn_n calls with spawn | |
| 13:19:04 | gibi | It should not cause any additional issue | |
| 13:19:11 | gibi | but it might not fix the current one | |
| 13:19:12 | gibi | :) | |
| 13:19:33 | sean-k-mooney | :) ya that is kind of what i was thinking too | |
| 13:20:05 | sean-k-mooney | it should not make things worse it just might not have any effect at all | |
| 13:21:51 | gibi | yepp | |
| 13:22:37 | gibi | there is a small memory / cpu overhead in case of spawn as it does wrap the greenlet into a GreenThread object but we don't have that much greenlets that it causes issues | |
| 13:23:02 | gibi | except when we start leaking them :D | |
| 13:35:46 | opendevreview | sean mooney proposed openstack/nova master: [DNM] allow mokey patching spawn_n to spawn https://review.opendev.org/c/openstack/nova/+/842359 | |
| 13:36:12 | opendevreview | sean mooney proposed openstack/nova master: [DNM] allow monkey patching spawn_n to spawn https://review.opendev.org/c/openstack/nova/+/842359 | |
| 13:36:58 | sean-k-mooney | i have not test ^ and it currently defaults to enabled | |
| 13:37:06 | sean-k-mooney | but we will see what the ci thinks | |
| 13:37:48 | sean-k-mooney | gibi shoudl i also add you greenlet state reporting | |
| 13:38:07 | gibi | you can pull the patch top of it just for data | |
| 13:38:40 | gibi | but it does not prove anything as CI is basically doing a lot of operations then stops, so there is now time for the numbers to settle to a baseline | |
| 13:39:04 | sean-k-mooney | ya but the data could be interesting to compare | |
| 13:39:18 | sean-k-mooney | shall i cherry pick your patch so? | |
| 13:39:31 | sean-k-mooney | or rebase on it | |
| 13:41:02 | gibi | just cherry pick top of yours | |
| 13:42:03 | opendevreview | sean mooney proposed openstack/nova master: DNM: log number of green(thread|let)s periodically https://review.opendev.org/c/openstack/nova/+/841040 | |
| 13:42:12 | sean-k-mooney | cool | |
| 13:42:20 | sean-k-mooney | so looking at the output of the previous run | |
| 13:42:29 | sean-k-mooney | we see both greenthreadds and greenlets | |
| 13:42:41 | sean-k-mooney | and we are not expecting to only see greenthreads with my patch | |
| 13:43:31 | gibi | yepp we expect only greenthreads and no naked greenlets | |
| 13:44:11 | gibi | greenlets are implemented in a C extension greenthreads are proper python objects implemented by eventlet | |
| 13:44:45 | sean-k-mooney | so looking at the output from the test run | |
| 13:45:00 | sean-k-mooney | teh greenthrad sayed pretty constant | |
| 13:45:13 | sean-k-mooney | but the greenlets were more or less slowly increaing over time | |
| 13:45:59 | gibi | I see same behavior locally too, but most of the time after couple of minutes of idle time the greenlets also decreased back to baseline. | |
| 13:46:04 | sean-k-mooney | https://termbin.com/dzg3 | |
| 13:46:18 | gibi | in CI there is no couple of minutes of idle time | |
| 13:46:25 | frickler | gibi: kashyap: fyi I had another fix for this in tempest recentish https://review.opendev.org/c/openstack/tempest/+/835382 | |
| 13:46:27 | gibi | but it would be interesting to see what happens there | |
| 13:46:47 | frickler | this = volume attachments | |
| 13:47:08 | sean-k-mooney | ah for tagged attachments | |
| 13:47:12 | gibi | frickler: yepp, that helps too. thanks. we need to track down all the detach scenarios | |
| 13:47:14 | sean-k-mooney | ya | |
| 13:47:41 | gibi | all volume detach operation is potenitally affected | |
| 13:48:29 | sean-k-mooney | and attach since we do detach as a cleanup action | |
| 13:48:40 | gibi | sean-k-mooney: good point, yes | |
| 13:49:10 | sean-k-mooney | you know technially this could affect nics too | |
| 13:49:25 | sean-k-mooney | i dont think we have ever seen it affect them | |
| 13:50:02 | sean-k-mooney | but both are just virtual pci devices form qemus point of view | |
| 13:50:12 | sean-k-mooney | just one is virtio-blk and the other is virtio-net | |
| 13:51:18 | gibi | yeah, I never see this in case of interfaces | |
| 13:51:28 | gibi | probably something is different down in the stack | |
| 13:51:28 | ricolin | sean-k-mooney: stephenfin | |
| 13:51:38 | gibi | either qemu dev handling or the guest OS dev handling | |
| 13:51:48 | gibi | nova today uses the same codepath | |
| 13:52:13 | sean-k-mooney | gibi: ya. i guess network attach/detach is more common and porably better tested | |
| 13:52:33 | ricolin | if you got some time, please help to review https://review.opendev.org/c/openstack/nova-specs/+/840310 as mnaser already laeve some comments would like to have your feedback:) | |
| 13:52:46 | gibi | also probably force pulling out a disk is more problematic from data consistency perspective than pulling a netdev | |
| 13:53:22 | sean-k-mooney | right the netdev is mostly stateless | |
| 13:54:14 | sean-k-mooney | ricolin: i think its stephenfin's feedback we need really | |
| 13:54:40 | sean-k-mooney | i dont like exposing the aw_bits but i understand that ye have a need for it | |
| 13:55:07 | ricolin | sean-k-mooney: cool, thanks:) | |
| 13:55:25 | sean-k-mooney | i think hw_viommu_model is the write absraction for opting in as we use the same pattern for contoling the graphic deviecs or nic models | |
| 14:03:17 | sean-k-mooney | gibi: hum https://review.opendev.org/c/openstack/tempest/+/842140 still failed the same way | |
| 14:03:23 | sean-k-mooney | frickler: ^ any idea why | |
| 14:05:13 | gibi | sean-k-mooney: yeah, this is where my tempest knowledge ends. probably something is different in the network setup of this test | |
| 14:05:20 | frickler | sean-k-mooney: I'll take a look later, in a meeting now | |
| 14:05:51 | sean-k-mooney | gibi: possible i can see in the tempest output it waited for it to go form resize verify to active | |
| 14:05:56 | sean-k-mooney | and tehn it tries to ssh in | |
| 14:06:02 | sean-k-mooney | but it didn not work | |
| 14:07:20 | sean-k-mooney | "fixed_ip_address": "10.1.0.10" | |
| 14:07:22 | sean-k-mooney | ip-route:10.1.0.0/28 dev eth0 scope link src 10.1.0.10 | |
| 14:07:54 | sean-k-mooney | ah | |
| 14:08:11 | sean-k-mooney | ## ping -c 5 10.1.0.1 | |
| 14:08:13 | sean-k-mooney | PING 10.1.0.1 (10.1.0.1): 56 data bytes | |