| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-05-18 | |||
| 13:16:10 | sean-k-mooney | i was hoping to be able to tweak https://github.com/eventlet/eventlet/issues/731#issue-1032856809 to repoduce it but ya i just want to see if we do https://github.com/eventlet/eventlet/issues/731#issuecomment-968135262 will it help | |
| 13:16:45 | sean-k-mooney | well will it work with nova | |
| 13:17:12 | sean-k-mooney | i was considering putting it behind a workaround config option so we could get feedback | |
| 13:18:35 | gibi | yeah either we need a way to reproduce or we need to make this optional and ask for feedback | |
| 13:18:56 | gibi | overall I don't see problems replacing our spawn_n calls with spawn | |
| 13:19:04 | gibi | It should not cause any additional issue | |
| 13:19:11 | gibi | but it might not fix the current one | |
| 13:19:12 | gibi | :) | |
| 13:19:33 | sean-k-mooney | :) ya that is kind of what i was thinking too | |
| 13:20:05 | sean-k-mooney | it should not make things worse it just might not have any effect at all | |
| 13:21:51 | gibi | yepp | |
| 13:22:37 | gibi | there is a small memory / cpu overhead in case of spawn as it does wrap the greenlet into a GreenThread object but we don't have that much greenlets that it causes issues | |
| 13:23:02 | gibi | except when we start leaking them :D | |
| 13:35:46 | opendevreview | sean mooney proposed openstack/nova master: [DNM] allow mokey patching spawn_n to spawn https://review.opendev.org/c/openstack/nova/+/842359 | |
| 13:36:12 | opendevreview | sean mooney proposed openstack/nova master: [DNM] allow monkey patching spawn_n to spawn https://review.opendev.org/c/openstack/nova/+/842359 | |
| 13:36:58 | sean-k-mooney | i have not test ^ and it currently defaults to enabled | |
| 13:37:06 | sean-k-mooney | but we will see what the ci thinks | |
| 13:37:48 | sean-k-mooney | gibi shoudl i also add you greenlet state reporting | |
| 13:38:07 | gibi | you can pull the patch top of it just for data | |
| 13:38:40 | gibi | but it does not prove anything as CI is basically doing a lot of operations then stops, so there is now time for the numbers to settle to a baseline | |
| 13:39:04 | sean-k-mooney | ya but the data could be interesting to compare | |
| 13:39:18 | sean-k-mooney | shall i cherry pick your patch so? | |
| 13:39:31 | sean-k-mooney | or rebase on it | |
| 13:41:02 | gibi | just cherry pick top of yours | |
| 13:42:03 | opendevreview | sean mooney proposed openstack/nova master: DNM: log number of green(thread|let)s periodically https://review.opendev.org/c/openstack/nova/+/841040 | |
| 13:42:12 | sean-k-mooney | cool | |
| 13:42:20 | sean-k-mooney | so looking at the output of the previous run | |
| 13:42:29 | sean-k-mooney | we see both greenthreadds and greenlets | |
| 13:42:41 | sean-k-mooney | and we are not expecting to only see greenthreads with my patch | |
| 13:43:31 | gibi | yepp we expect only greenthreads and no naked greenlets | |
| 13:44:11 | gibi | greenlets are implemented in a C extension greenthreads are proper python objects implemented by eventlet | |
| 13:44:45 | sean-k-mooney | so looking at the output from the test run | |
| 13:45:00 | sean-k-mooney | teh greenthrad sayed pretty constant | |
| 13:45:13 | sean-k-mooney | but the greenlets were more or less slowly increaing over time | |
| 13:45:59 | gibi | I see same behavior locally too, but most of the time after couple of minutes of idle time the greenlets also decreased back to baseline. | |
| 13:46:04 | sean-k-mooney | https://termbin.com/dzg3 | |
| 13:46:18 | gibi | in CI there is no couple of minutes of idle time | |
| 13:46:25 | frickler | gibi: kashyap: fyi I had another fix for this in tempest recentish https://review.opendev.org/c/openstack/tempest/+/835382 | |
| 13:46:27 | gibi | but it would be interesting to see what happens there | |
| 13:46:47 | frickler | this = volume attachments | |
| 13:47:08 | sean-k-mooney | ah for tagged attachments | |
| 13:47:12 | gibi | frickler: yepp, that helps too. thanks. we need to track down all the detach scenarios | |
| 13:47:14 | sean-k-mooney | ya | |
| 13:47:41 | gibi | all volume detach operation is potenitally affected | |
| 13:48:29 | sean-k-mooney | and attach since we do detach as a cleanup action | |
| 13:48:40 | gibi | sean-k-mooney: good point, yes | |
| 13:49:10 | sean-k-mooney | you know technially this could affect nics too | |
| 13:49:25 | sean-k-mooney | i dont think we have ever seen it affect them | |
| 13:50:02 | sean-k-mooney | but both are just virtual pci devices form qemus point of view | |
| 13:50:12 | sean-k-mooney | just one is virtio-blk and the other is virtio-net | |
| 13:51:18 | gibi | yeah, I never see this in case of interfaces | |
| 13:51:28 | gibi | probably something is different down in the stack | |
| 13:51:28 | ricolin | sean-k-mooney: stephenfin | |
| 13:51:38 | gibi | either qemu dev handling or the guest OS dev handling | |
| 13:51:48 | gibi | nova today uses the same codepath | |
| 13:52:13 | sean-k-mooney | gibi: ya. i guess network attach/detach is more common and porably better tested | |
| 13:52:33 | ricolin | if you got some time, please help to review https://review.opendev.org/c/openstack/nova-specs/+/840310 as mnaser already laeve some comments would like to have your feedback:) | |
| 13:52:46 | gibi | also probably force pulling out a disk is more problematic from data consistency perspective than pulling a netdev | |
| 13:53:22 | sean-k-mooney | right the netdev is mostly stateless | |
| 13:54:14 | sean-k-mooney | ricolin: i think its stephenfin's feedback we need really | |
| 13:54:40 | sean-k-mooney | i dont like exposing the aw_bits but i understand that ye have a need for it | |
| 13:55:07 | ricolin | sean-k-mooney: cool, thanks:) | |
| 13:55:25 | sean-k-mooney | i think hw_viommu_model is the write absraction for opting in as we use the same pattern for contoling the graphic deviecs or nic models | |
| 14:03:17 | sean-k-mooney | gibi: hum https://review.opendev.org/c/openstack/tempest/+/842140 still failed the same way | |
| 14:03:23 | sean-k-mooney | frickler: ^ any idea why | |
| 14:05:13 | gibi | sean-k-mooney: yeah, this is where my tempest knowledge ends. probably something is different in the network setup of this test | |
| 14:05:20 | frickler | sean-k-mooney: I'll take a look later, in a meeting now | |
| 14:05:51 | sean-k-mooney | gibi: possible i can see in the tempest output it waited for it to go form resize verify to active | |
| 14:05:56 | sean-k-mooney | and tehn it tries to ssh in | |
| 14:06:02 | sean-k-mooney | but it didn not work | |
| 14:07:20 | sean-k-mooney | "fixed_ip_address": "10.1.0.10" | |
| 14:07:22 | sean-k-mooney | ip-route:10.1.0.0/28 dev eth0 scope link src 10.1.0.10 | |
| 14:07:54 | sean-k-mooney | ah | |
| 14:08:11 | sean-k-mooney | ## ping -c 5 10.1.0.1 | |
| 14:08:13 | sean-k-mooney | PING 10.1.0.1 (10.1.0.1): 56 data bytes | |
| 14:08:15 | sean-k-mooney | --- 10.1.0.1 ping statistics --- | |
| 14:08:17 | sean-k-mooney | 5 packets transmitted, 0 packets received, 100% packet loss | |
| 14:08:20 | sean-k-mooney | the vm cannot ping its gateway | |
| 14:10:14 | sean-k-mooney | gibi: my guess is security groups | |
| 14:12:06 | bauzas | gmann: to clarify the new release naming thingy, it would be something like "2022.2 Arbitrary" for the AA release ? | |
| 14:12:28 | bauzas | where "Arbitrary" be chosen by the Foundation folks | |
| 14:12:29 | bauzas | right? | |
| 14:13:02 | sean-k-mooney | where "Arbitrary" is aardvark so say it dansmith :) | |
| 14:13:24 | sean-k-mooney | havnt actully read the most recent status of this | |
| 14:13:33 | sean-k-mooney | so also interested | |
| 14:13:39 | dansmith | I would love them to choose such a name to make my point, but yes, it's going to be aardvark :) | |
| 14:13:57 | dansmith | the name will be the name, the version will be 2022.2 | |
| 14:17:43 | sean-k-mooney | gibi: it might be becasue the server is not being created with teh validation resouces initally | |
| 14:17:53 | bauzas | dansmith: if it was named a-ha, it could take on me | |
| 14:17:58 | sean-k-mooney | gibi: so it might not hwave an sshkey | |
| 14:18:13 | gibi | sean-k-mooney: OK, that I can fix | |
| 14:18:21 | dansmith | bauzas: you win today | |
| 14:18:23 | sean-k-mooney | gibi: we are not calling https://review.opendev.org/c/openstack/tempest/+/842140/4/tempest/api/compute/volumes/test_attach_volume.py#45= | |
| 14:18:25 | gibi | thanks for looking at it | |
| 14:18:53 | sean-k-mooney | here https://review.opendev.org/c/openstack/tempest/+/842140/4/tempest/api/compute/volumes/test_attach_volume.py#384 | |
| 14:19:07 | bauzas | dansmith: heh, thanks for the explanation anyway | |
| 14:19:26 | sean-k-mooney | so we are not waiting for the server to be sshable before we attach the multi attach volume | |
| 14:20:14 | sean-k-mooney | a is for aardvark is an amican thing we normally say ant or similar | |
| 14:20:38 | dansmith | it's perfect because it starts with AA | |
| 14:20:44 | sean-k-mooney | yep | |