| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-11-22 | |||
| 19:03:37 | mnaser | sean-k-mooney: i'm still doing my research, but also, i suspect this affects n-cond too | |
| 19:03:49 | mnaser | and anything rabbit related, it seems like the notification blocks the main process | |
| 19:04:50 | mnaser | or maybe when the queue of unsent messages gets so big, the whole process bogs down | |
| 19:05:01 | mnaser | or it has a limit of threads it will bubble up to and then the whole process stops responding | |
| 19:05:54 | sean-k-mooney | its possible that the eventlet thread pool will file up eventually | |
| 19:06:12 | sean-k-mooney | hopefully this is something i can detech ast part of the health check work | |
| 19:06:13 | mnaser | afaik i think the default timeout or retry is set to 0 with notifications | |
| 19:06:28 | sean-k-mooney | well notificaiotn are off by default | |
| 19:06:38 | sean-k-mooney | or rather we use the noop driver | |
| 19:06:43 | mnaser | right yes, but if you turn them on, retries=0 so retry forever | |
| 19:07:01 | sean-k-mooney | i woudl expect 0 to be retry never | |
| 19:07:05 | sean-k-mooney | and -1 be retry for ever | |
| 19:07:12 | mnaser | 0 is retry forever in notifier i think let me duble check | |
| 19:07:22 | mnaser | sean-k-mooney: btw i suggest looking at how we do health checks in openstack-helm, it has some neat things where it actually makes an rpc call to the local instance and make sure we get an error back saying "not valid call" | |
| 19:07:28 | mnaser | there's some neat stuff there that might draw inspiration | |
| 19:07:51 | mnaser | sean-k-mooney: https://opendev.org/openstack/openstack-helm/src/branch/master/neutron/templates/bin/_health-probe.py.tpl | |
| 19:08:00 | sean-k-mooney | mnaser: i wanted to do active probes but the direction at the ptg was that was not ok | |
| 19:08:22 | mnaser | this one pretty much runs the check when it's asked | |
| 19:08:32 | sean-k-mooney | maybe after the intial work si done we can add a probe endpoint but it will intially be based on cached sate | |
| 19:08:48 | sean-k-mooney | mnaser: ya that is what i was going to do but it was rejected when i proposed it | |
| 19:08:55 | mnaser | https://opendev.org/openstack/openstack-helm/src/branch/master/nova/templates/bin/_health-probe.py.tpl is how its done for nova | |
| 19:09:22 | sean-k-mooney | mnaser: well that is writing to the nova message bus | |
| 19:09:37 | sean-k-mooney | so that is not allowed by anything that is not apart fo nova | |
| 19:09:57 | mnaser | yes it might not be very clean but it works(tm) | |
| 19:10:15 | sean-k-mooney | sure and will void any downstream support you have with your vendor | |
| 19:10:32 | mnaser | fair enough, my downstream support is me =P | |
| 19:10:49 | sean-k-mooney | but ya probing the queue was one of the thing i wanted to do | |
| 19:11:03 | sean-k-mooney | we migth add a way to do that at somepoint | |
| 19:11:05 | mnaser | btw, you were right, -1 is indefinite, and it defaults to that => https://opendev.org/openstack/oslo.messaging/src/branch/master/oslo_messaging/notify/notifier.py#L55-L58 | |
| 19:11:20 | sean-k-mooney | ack | |
| 19:11:50 | mnaser | i guess oslo messaging doesnt have a timeout | |
| 19:19:27 | sean-k-mooney | ya im not sure | |
| 19:19:32 | opendevreview | Artom Lifshitz proposed openstack/nova master: api-ref: Adjust BFV rescue non-support note. https://review.opendev.org/c/openstack/nova/+/818823 | |
| 19:19:39 | sean-k-mooney | likely you shoul change the default to be say 10 or simialr | |
| 19:19:44 | sean-k-mooney | in OSA | |
| 20:04:29 | opendevreview | Merged openstack/nova stable/xena: Add a WA flag waiting for vif-plugged event during reboot https://review.opendev.org/c/openstack/nova/+/818515 | |
| 20:16:50 | mnaser | sean-k-mooney: well it sounds like maybe that's not a great default value i guess | |
| 20:18:24 | sean-k-mooney | mnaser: i assume notifcaiton are off in osa by default. if it enabeld an no default is spcifed for retry i woudl proably default to 0,1 or 3 but not -1 | |
| 20:18:47 | sean-k-mooney | or just make it an error | |
| 20:18:48 | mnaser | sean-k-mooney: yeah im thinking more of a more sane oslo.messaging defaults | |
| 20:18:57 | sean-k-mooney | require it to be set | |
| 20:19:19 | sean-k-mooney | well again it depends on your setup you might rely on notificaions | |
| 20:19:37 | sean-k-mooney | but if you do then you als need to have monitoring in place to know that ere are rabbit issues | |
| 20:19:43 | sean-k-mooney | and correct that | |
| 20:20:02 | mnaser | sean-k-mooney: yeah but to me it sounds like notifications failing should not result in nova falling apart | |
| 20:21:12 | sean-k-mooney | well it should not but that might just mean that -1 for retry is not a vaild value | |
| 20:21:34 | sean-k-mooney | -1 presumable mean you must keep every notificaiotn in memory | |
| 20:21:37 | sean-k-mooney | untill its sent | |
| 20:22:41 | sean-k-mooney | with a copertive threading model like evently if you have enough notificiton eventlet pendign that will eventurally degrade the performance of the service | |
| 20:23:38 | mnaser | yeah im trying to repro right now | |
| 21:21:22 | opendevreview | Stanislav Dmitriev proposed openstack/nova master: Retry image download if it's corrupted https://review.opendev.org/c/openstack/nova/+/818503 | |
| 22:06:47 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Add PCI VPD Capability Handling https://review.opendev.org/c/openstack/nova/+/808199 | |
| 22:06:48 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Support remote-managed SmartNIC DPU ports https://review.opendev.org/c/openstack/nova/+/812111 | |
| #openstack-nova - 2021-11-23 | |||
| 04:02:58 | mnaser | sean-k-mooney: https://bugs.launchpad.net/nova/+bug/1917645?comments=all there's literally no way to workaround this issue, if that rabbitmq is down, the whole clouds melts down, even with retry=0 | |
| 08:49:10 | opendevreview | Nicolas Parquet proposed openstack/nova master: Add v2.91 microversion, allowing @ and dot (.) characters in keypair name https://review.opendev.org/c/openstack/nova/+/781076 | |
| 09:04:44 | opendevreview | Nicolas Parquet proposed openstack/nova master: Add v2.91 microversion, allowing @ and dot (.) characters in keypair name https://review.opendev.org/c/openstack/nova/+/781076 | |
| 09:14:41 | gibi | mnaser: hi, about notification blocking the main thread, there is a bug from belmiro https://bugs.launchpad.net/nova/+bug/1917645 | |
| 09:15:18 | gibi | I intended to look into at some point, but never had the time | |
| 09:17:01 | gibi | now I feel again I should look | |
| 09:46:12 | gibi | stephenfin: I have a bad feeling about https://review.opendev.org/c/openstack/nova/+/817746/comment/6690000b_3989ece7/ but if there is no better way then I will +A it | |
| 09:46:41 | bauzas | gibi: looking at https://docs.openstack.org/nova/latest/contributor/ptl-guide.html#milestone-1 | |
| 09:46:50 | bauzas | gibi: wondering if all this fish is still needed ? | |
| 09:47:48 | gibi | bauzas: launchpad bookkeeping is really just bookkeeping. If you don't do it then nothing will fail | |
| 09:48:04 | bauzas | gibi: you did this for Xena ? | |
| 09:48:06 | gibi | bauzas: the lib releases are probably proposed by the release team automatically | |
| 09:48:09 | bauzas | if so, I'll do it | |
| 09:48:20 | bauzas | gibi: yeah I'm looking at the releases gerrit | |
| 09:48:30 | bauzas | gibi: I wasn't asked to look at some changes | |
| 09:49:33 | gibi | bauzas: it seems that I only released xena-3 in launchpad not 1, 2 or rc1 | |
| 09:49:52 | bauzas | hah, OK | |
| 09:49:55 | bauzas | I'll look at that | |
| 09:50:49 | gibi | bauzas: I think the release team only proposes a release if there was real content on the branch since the last release | |
| 09:51:02 | bauzas | yeah, I only see https://review.opendev.org/c/openstack/releases/+/818415 | |
| 09:51:12 | bauzas | elodilles: ^ correct ? | |
| 09:51:28 | gibi | bauzas: we have os-vif release proposed https://review.opendev.org/c/openstack/releases/+/818415 | |
| 09:51:37 | bauzas | jinxed :p | |
| 09:51:44 | gibi | but not python-novaclient | |
| 09:52:29 | bauzas | yup | |
| 09:52:33 | gibi | os-vif had some meaningfull change since 2.6 | |
| 09:52:49 | bauzas | yes | |
| 09:52:49 | gibi | novaclient does not | |
| 09:52:54 | bauzas | correct | |
| 09:53:22 | gibi | and I thin placement's os-traits and os-resource-classes are marked as independent libs so there we don't need a release | |
| 09:53:28 | gibi | per milestone | |
| 09:53:42 | bauzas | this is correct | |
| 09:54:24 | bauzas | https://releases.openstack.org/independent.html#os-traits | |
| 09:54:33 | bauzas | https://releases.openstack.org/independent.html#os-resource-classes | |
| 09:54:40 | bauzas | they are independant | |
| 09:56:52 | gibi | what I did at m1 is to move the bps targeted to m1 to m2 | |
| 09:58:16 | gibi | but as far as see you opted not to target bps to milestones at all, so that retargeting is not needed | |
| 09:58:28 | gibi | the list https://blueprints.launchpad.net/nova/yoga seem OK | |
| 10:02:20 | elodilles | bauzas: gibi is right :) at milestone-1 only such projects had a generated release patch that had some real content in their master branch since the last release | |
| 10:17:46 | bauzas | elodilles: thanks for explaining :) | |
| 10:33:30 | elodilles | :) | |
| 11:10:00 | opendevreview | Merged openstack/nova master: db: Replace use of Executable.scalar(), Executable.execute() https://review.opendev.org/c/openstack/nova/+/804878 | |
| 11:17:39 | bauzas | lyarwood: haven't seen that https://specs.openstack.org/openstack/nova-specs/specs/yoga/approved/ephemeral-encryption-libvirt.html shares the same BP than https://specs.openstack.org/openstack/nova-specs/specs/yoga/approved/ephemeral-encryption.html | |
| 11:17:58 | bauzas | lyarwood: could you please create another BP for the libvirt one so I could mark it for yoga ? | |
| 11:18:20 | bauzas | lyarwood: also, could you please modify the spec to use the new BP ? | |
| 11:26:44 | opendevreview | Merged openstack/nova master: db: Replace use of 'autoload' parameter https://review.opendev.org/c/openstack/nova/+/805734 | |
| 11:26:53 | opendevreview | Merged openstack/nova master: db: Replace use of legacy select() calling style https://review.opendev.org/c/openstack/nova/+/805735 | |