| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-11-22 | |||
| 14:07:49 | sean-k-mooney | kashyap: the highest profile use is proably SKA the Square Kilometer Array telescope | |
| 14:08:05 | kashyap | Cool; good to know :) | |
| 14:08:44 | Henriqueof | sean-k-mooney: Really? Until now I thought kolla-ansible was one of the most popular deployment tool. | |
| 14:10:40 | sean-k-mooney | Henriqueof: it is yes | |
| 14:11:04 | sean-k-mooney | Henriqueof: im not sure how much market share it has vs tripleo,openstack charms and openstack ansible | |
| 14:11:16 | sean-k-mooney | but those are the big 4 deployment tools | |
| 14:14:06 | sean-k-mooney | looking at https://www.openstack.org/analytics | |
| 14:14:14 | sean-k-mooney | if you go to Deployment Decisions | |
| 14:14:27 | Henriqueof | Yeah, it is very straight forward and stable tool so I never felt the nned to experiment with the others. | |
| 14:14:34 | sean-k-mooney | 29% of respondence used kolla ansible | |
| 14:15:21 | sean-k-mooney | which is about the same as juju/triplo/OSA combined | |
| 14:15:48 | sean-k-mooney | that does not tell you how big the deployment are however | |
| 14:16:27 | sean-k-mooney | so ther might be more respondnce using kolla-ansible but that does not mean there are more servers managed by it but it at least give some indeicaionts of it popularity | |
| 14:21:15 | kashyap | stephenfin: Hey, have you ever used this? - sphinxcontrib-spelling | |
| 14:21:27 | kashyap | [https://sphinxcontrib-spelling.readthedocs.io/en/latest/] | |
| 14:23:28 | sean-k-mooney | kashyap: i would expect it would have issues with the terms we use like extra-spec | |
| 14:24:02 | kashyap | Right; but still I wonder is it overall a net win or not | |
| 14:24:03 | sean-k-mooney | we likely could include a dictionary with those but that might get tedious | |
| 14:24:36 | kashyap | sean-k-mooney: Yes, it can use project-specific dictionaries | |
| 15:25:46 | opendevreview | Merged openstack/nova-specs master: Repropose Add libvirt support for flavor and image defined ephemeral encryption https://review.opendev.org/c/openstack/nova-specs/+/810868 | |
| 15:38:08 | opendevreview | Dan Smith proposed openstack/nova master: Revert project-specific APIs for servers https://review.opendev.org/c/openstack/nova/+/816206 | |
| 15:38:08 | opendevreview | Dan Smith proposed openstack/nova master: Allow per-context rule in error messages https://review.opendev.org/c/openstack/nova/+/816865 | |
| 15:38:40 | dansmith | gmann: johnthetubaguy[m]: Removed the WIPs from these ^ as I'm assuming there are no more fundamental concerns | |
| 15:39:23 | gmann | dansmith: ack, I will check today. thanks | |
| 15:48:14 | dansmith | can we get this merged? https://review.opendev.org/c/openstack/nova/+/817030 | |
| 15:48:29 | dansmith | it's already being used to debug gate and real VIF plugging event failures | |
| 15:49:54 | gibi | dansmith: done | |
| 15:51:54 | dansmith | gibi: thanks | |
| 16:51:33 | kashyap | In CirrOS latest 0.5.2, where is this file? /etc/cirros-init/config? | |
| 16:51:46 | kashyap | Is it moved to somewhere else? /me didn't find it in a quick libguestfs inspection | |
| 16:57:49 | kashyap | Actually, ignore me. It's still there. | |
| 17:35:50 | opendevreview | Merged openstack/nova master: Log instance event wait times https://review.opendev.org/c/openstack/nova/+/817030 | |
| 18:39:05 | opendevreview | Merged openstack/nova master: nova-manage: Always get BDMs using get_by_volume_and_instance https://review.opendev.org/c/openstack/nova/+/811716 | |
| 18:56:29 | mnaser | hi y'all | |
| 18:56:57 | mnaser | has anyone ran into an issue where the api stops responding if the notification transport is failing? | |
| 18:57:36 | mnaser | i.e. oslo_messaging_notificaitons/transport_url = rabbit://foobar , where foobar goes down, and the DEFAULT/transport_url still is up, but i guess the threads all get blocked till it grinds down to a halt? | |
| 18:58:17 | mnaser | i've repro'd on a customer environment that is deployed by OSA but i'm trying to get a devstack up right now and get GMR to see how it hands | |
| 18:58:20 | mnaser | s/hands/hangs/ | |
| 18:59:40 | sean-k-mooney | it might be related to the heartbeat | |
| 18:59:51 | sean-k-mooney | or the wsgi server | |
| 19:00:18 | sean-k-mooney | if you are using mod_wsgi under apptach each worker will only ever service 1 api request at a time | |
| 19:01:04 | sean-k-mooney | we may monkey patch the api but that will never allow the apache process to service a second request in parallel as that is managed by apache | |
| 19:01:48 | sean-k-mooney | if all the api workers are trying to do somethign that needs rabit then it will stop responding until the request or rpc timeout fires and it retruns an error | |
| 19:02:30 | sean-k-mooney | i dont know if uwsgi is better in that regard | |
| 19:03:02 | mnaser | sean-k-mooney: OSA deploys with uwsgi | |
| 19:03:37 | mnaser | sean-k-mooney: i'm still doing my research, but also, i suspect this affects n-cond too | |
| 19:03:49 | mnaser | and anything rabbit related, it seems like the notification blocks the main process | |
| 19:04:50 | mnaser | or maybe when the queue of unsent messages gets so big, the whole process bogs down | |
| 19:05:01 | mnaser | or it has a limit of threads it will bubble up to and then the whole process stops responding | |
| 19:05:54 | sean-k-mooney | its possible that the eventlet thread pool will file up eventually | |
| 19:06:12 | sean-k-mooney | hopefully this is something i can detech ast part of the health check work | |
| 19:06:13 | mnaser | afaik i think the default timeout or retry is set to 0 with notifications | |
| 19:06:28 | sean-k-mooney | well notificaiotn are off by default | |
| 19:06:38 | sean-k-mooney | or rather we use the noop driver | |
| 19:06:43 | mnaser | right yes, but if you turn them on, retries=0 so retry forever | |
| 19:07:01 | sean-k-mooney | i woudl expect 0 to be retry never | |
| 19:07:05 | sean-k-mooney | and -1 be retry for ever | |
| 19:07:12 | mnaser | 0 is retry forever in notifier i think let me duble check | |
| 19:07:22 | mnaser | sean-k-mooney: btw i suggest looking at how we do health checks in openstack-helm, it has some neat things where it actually makes an rpc call to the local instance and make sure we get an error back saying "not valid call" | |
| 19:07:28 | mnaser | there's some neat stuff there that might draw inspiration | |
| 19:07:51 | mnaser | sean-k-mooney: https://opendev.org/openstack/openstack-helm/src/branch/master/neutron/templates/bin/_health-probe.py.tpl | |
| 19:08:00 | sean-k-mooney | mnaser: i wanted to do active probes but the direction at the ptg was that was not ok | |
| 19:08:22 | mnaser | this one pretty much runs the check when it's asked | |
| 19:08:32 | sean-k-mooney | maybe after the intial work si done we can add a probe endpoint but it will intially be based on cached sate | |
| 19:08:48 | sean-k-mooney | mnaser: ya that is what i was going to do but it was rejected when i proposed it | |
| 19:08:55 | mnaser | https://opendev.org/openstack/openstack-helm/src/branch/master/nova/templates/bin/_health-probe.py.tpl is how its done for nova | |
| 19:09:22 | sean-k-mooney | mnaser: well that is writing to the nova message bus | |
| 19:09:37 | sean-k-mooney | so that is not allowed by anything that is not apart fo nova | |
| 19:09:57 | mnaser | yes it might not be very clean but it works(tm) | |
| 19:10:15 | sean-k-mooney | sure and will void any downstream support you have with your vendor | |
| 19:10:32 | mnaser | fair enough, my downstream support is me =P | |
| 19:10:49 | sean-k-mooney | but ya probing the queue was one of the thing i wanted to do | |
| 19:11:03 | sean-k-mooney | we migth add a way to do that at somepoint | |
| 19:11:05 | mnaser | btw, you were right, -1 is indefinite, and it defaults to that => https://opendev.org/openstack/oslo.messaging/src/branch/master/oslo_messaging/notify/notifier.py#L55-L58 | |
| 19:11:20 | sean-k-mooney | ack | |
| 19:11:50 | mnaser | i guess oslo messaging doesnt have a timeout | |
| 19:19:27 | sean-k-mooney | ya im not sure | |
| 19:19:32 | opendevreview | Artom Lifshitz proposed openstack/nova master: api-ref: Adjust BFV rescue non-support note. https://review.opendev.org/c/openstack/nova/+/818823 | |
| 19:19:39 | sean-k-mooney | likely you shoul change the default to be say 10 or simialr | |
| 19:19:44 | sean-k-mooney | in OSA | |
| 20:04:29 | opendevreview | Merged openstack/nova stable/xena: Add a WA flag waiting for vif-plugged event during reboot https://review.opendev.org/c/openstack/nova/+/818515 | |
| 20:16:50 | mnaser | sean-k-mooney: well it sounds like maybe that's not a great default value i guess | |
| 20:18:24 | sean-k-mooney | mnaser: i assume notifcaiton are off in osa by default. if it enabeld an no default is spcifed for retry i woudl proably default to 0,1 or 3 but not -1 | |
| 20:18:47 | sean-k-mooney | or just make it an error | |
| 20:18:48 | mnaser | sean-k-mooney: yeah im thinking more of a more sane oslo.messaging defaults | |
| 20:18:57 | sean-k-mooney | require it to be set | |
| 20:19:19 | sean-k-mooney | well again it depends on your setup you might rely on notificaions | |
| 20:19:37 | sean-k-mooney | but if you do then you als need to have monitoring in place to know that ere are rabbit issues | |
| 20:19:43 | sean-k-mooney | and correct that | |
| 20:20:02 | mnaser | sean-k-mooney: yeah but to me it sounds like notifications failing should not result in nova falling apart | |
| 20:21:12 | sean-k-mooney | well it should not but that might just mean that -1 for retry is not a vaild value | |
| 20:21:34 | sean-k-mooney | -1 presumable mean you must keep every notificaiotn in memory | |
| 20:21:37 | sean-k-mooney | untill its sent | |
| 20:22:41 | sean-k-mooney | with a copertive threading model like evently if you have enough notificiton eventlet pendign that will eventurally degrade the performance of the service | |
| 20:23:38 | mnaser | yeah im trying to repro right now | |
| 21:21:22 | opendevreview | Stanislav Dmitriev proposed openstack/nova master: Retry image download if it's corrupted https://review.opendev.org/c/openstack/nova/+/818503 | |
| 22:06:47 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Add PCI VPD Capability Handling https://review.opendev.org/c/openstack/nova/+/808199 | |
| 22:06:48 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Support remote-managed SmartNIC DPU ports https://review.opendev.org/c/openstack/nova/+/812111 | |
| #openstack-nova - 2021-11-23 | |||
| 04:02:58 | mnaser | sean-k-mooney: https://bugs.launchpad.net/nova/+bug/1917645?comments=all there's literally no way to workaround this issue, if that rabbitmq is down, the whole clouds melts down, even with retry=0 | |
| 08:49:10 | opendevreview | Nicolas Parquet proposed openstack/nova master: Add v2.91 microversion, allowing @ and dot (.) characters in keypair name https://review.opendev.org/c/openstack/nova/+/781076 | |