Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-11-22
15:38:08 opendevreview Dan Smith proposed openstack/nova master: Allow per-context rule in error messages https://review.opendev.org/c/openstack/nova/+/816865
15:38:08 opendevreview Dan Smith proposed openstack/nova master: Revert project-specific APIs for servers https://review.opendev.org/c/openstack/nova/+/816206
15:38:40 dansmith gmann: johnthetubaguy[m]: Removed the WIPs from these ^ as I'm assuming there are no more fundamental concerns
15:39:23 gmann dansmith: ack, I will check today. thanks
15:48:14 dansmith can we get this merged? https://review.opendev.org/c/openstack/nova/+/817030
15:48:29 dansmith it's already being used to debug gate and real VIF plugging event failures
15:49:54 gibi dansmith: done
15:51:54 dansmith gibi: thanks
16:51:33 kashyap In CirrOS latest 0.5.2, where is this file? /etc/cirros-init/config?
16:51:46 kashyap Is it moved to somewhere else? /me didn't find it in a quick libguestfs inspection
16:57:49 kashyap Actually, ignore me. It's still there.
17:35:50 opendevreview Merged openstack/nova master: Log instance event wait times https://review.opendev.org/c/openstack/nova/+/817030
18:39:05 opendevreview Merged openstack/nova master: nova-manage: Always get BDMs using get_by_volume_and_instance https://review.opendev.org/c/openstack/nova/+/811716
18:56:29 mnaser hi y'all
18:56:57 mnaser has anyone ran into an issue where the api stops responding if the notification transport is failing?
18:57:36 mnaser i.e. oslo_messaging_notificaitons/transport_url = rabbit://foobar , where foobar goes down, and the DEFAULT/transport_url still is up, but i guess the threads all get blocked till it grinds down to a halt?
18:58:17 mnaser i've repro'd on a customer environment that is deployed by OSA but i'm trying to get a devstack up right now and get GMR to see how it hands
18:58:20 mnaser s/hands/hangs/
18:59:40 sean-k-mooney it might be related to the heartbeat
18:59:51 sean-k-mooney or the wsgi server
19:00:18 sean-k-mooney if you are using mod_wsgi under apptach each worker will only ever service 1 api request at a time
19:01:04 sean-k-mooney we may monkey patch the api but that will never allow the apache process to service a second request in parallel as that is managed by apache
19:01:48 sean-k-mooney if all the api workers are trying to do somethign that needs rabit then it will stop responding until the request or rpc timeout fires and it retruns an error
19:02:30 sean-k-mooney i dont know if uwsgi is better in that regard
19:03:02 mnaser sean-k-mooney: OSA deploys with uwsgi
19:03:37 mnaser sean-k-mooney: i'm still doing my research, but also, i suspect this affects n-cond too
19:03:49 mnaser and anything rabbit related, it seems like the notification blocks the main process
19:04:50 mnaser or maybe when the queue of unsent messages gets so big, the whole process bogs down
19:05:01 mnaser or it has a limit of threads it will bubble up to and then the whole process stops responding
19:05:54 sean-k-mooney its possible that the eventlet thread pool will file up eventually
19:06:12 sean-k-mooney hopefully this is something i can detech ast part of the health check work
19:06:13 mnaser afaik i think the default timeout or retry is set to 0 with notifications
19:06:28 sean-k-mooney well notificaiotn are off by default
19:06:38 sean-k-mooney or rather we use the noop driver
19:06:43 mnaser right yes, but if you turn them on, retries=0 so retry forever
19:07:01 sean-k-mooney i woudl expect 0 to be retry never
19:07:05 sean-k-mooney and -1 be retry for ever
19:07:12 mnaser 0 is retry forever in notifier i think let me duble check
19:07:22 mnaser sean-k-mooney: btw i suggest looking at how we do health checks in openstack-helm, it has some neat things where it actually makes an rpc call to the local instance and make sure we get an error back saying "not valid call"
19:07:28 mnaser there's some neat stuff there that might draw inspiration
19:07:51 mnaser sean-k-mooney: https://opendev.org/openstack/openstack-helm/src/branch/master/neutron/templates/bin/_health-probe.py.tpl
19:08:00 sean-k-mooney mnaser: i wanted to do active probes but the direction at the ptg was that was not ok
19:08:22 mnaser this one pretty much runs the check when it's asked
19:08:32 sean-k-mooney maybe after the intial work si done we can add a probe endpoint but it will intially be based on cached sate
19:08:48 sean-k-mooney mnaser: ya that is what i was going to do but it was rejected when i proposed it
19:08:55 mnaser https://opendev.org/openstack/openstack-helm/src/branch/master/nova/templates/bin/_health-probe.py.tpl is how its done for nova
19:09:22 sean-k-mooney mnaser: well that is writing to the nova message bus
19:09:37 sean-k-mooney so that is not allowed by anything that is not apart fo nova
19:09:57 mnaser yes it might not be very clean but it works(tm)
19:10:15 sean-k-mooney sure and will void any downstream support you have with your vendor
19:10:32 mnaser fair enough, my downstream support is me =P
19:10:49 sean-k-mooney but ya probing the queue was one of the thing i wanted to do
19:11:03 sean-k-mooney we migth add a way to do that at somepoint
19:11:05 mnaser btw, you were right, -1 is indefinite, and it defaults to that => https://opendev.org/openstack/oslo.messaging/src/branch/master/oslo_messaging/notify/notifier.py#L55-L58
19:11:20 sean-k-mooney ack
19:11:50 mnaser i guess oslo messaging doesnt have a timeout
19:19:27 sean-k-mooney ya im not sure
19:19:32 opendevreview Artom Lifshitz proposed openstack/nova master: api-ref: Adjust BFV rescue non-support note. https://review.opendev.org/c/openstack/nova/+/818823
19:19:39 sean-k-mooney likely you shoul change the default to be say 10 or simialr
19:19:44 sean-k-mooney in OSA
20:04:29 opendevreview Merged openstack/nova stable/xena: Add a WA flag waiting for vif-plugged event during reboot https://review.opendev.org/c/openstack/nova/+/818515
20:16:50 mnaser sean-k-mooney: well it sounds like maybe that's not a great default value i guess
20:18:24 sean-k-mooney mnaser: i assume notifcaiton are off in osa by default. if it enabeld an no default is spcifed for retry i woudl proably default to 0,1 or 3 but not -1
20:18:47 sean-k-mooney or just make it an error
20:18:48 mnaser sean-k-mooney: yeah im thinking more of a more sane oslo.messaging defaults
20:18:57 sean-k-mooney require it to be set
20:19:19 sean-k-mooney well again it depends on your setup you might rely on notificaions
20:19:37 sean-k-mooney but if you do then you als need to have monitoring in place to know that ere are rabbit issues
20:19:43 sean-k-mooney and correct that
20:20:02 mnaser sean-k-mooney: yeah but to me it sounds like notifications failing should not result in nova falling apart
20:21:12 sean-k-mooney well it should not but that might just mean that -1 for retry is not a vaild value
20:21:34 sean-k-mooney -1 presumable mean you must keep every notificaiotn in memory
20:21:37 sean-k-mooney untill its sent
20:22:41 sean-k-mooney with a copertive threading model like evently if you have enough notificiton eventlet pendign that will eventurally degrade the performance of the service
20:23:38 mnaser yeah im trying to repro right now
21:21:22 opendevreview Stanislav Dmitriev proposed openstack/nova master: Retry image download if it's corrupted https://review.opendev.org/c/openstack/nova/+/818503
22:06:47 opendevreview Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Add PCI VPD Capability Handling https://review.opendev.org/c/openstack/nova/+/808199
22:06:48 opendevreview Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Support remote-managed SmartNIC DPU ports https://review.opendev.org/c/openstack/nova/+/812111
#openstack-nova - 2021-11-23
04:02:58 mnaser sean-k-mooney: https://bugs.launchpad.net/nova/+bug/1917645?comments=all there's literally no way to workaround this issue, if that rabbitmq is down, the whole clouds melts down, even with retry=0
08:49:10 opendevreview Nicolas Parquet proposed openstack/nova master: Add v2.91 microversion, allowing @ and dot (.) characters in keypair name https://review.opendev.org/c/openstack/nova/+/781076
09:04:44 opendevreview Nicolas Parquet proposed openstack/nova master: Add v2.91 microversion, allowing @ and dot (.) characters in keypair name https://review.opendev.org/c/openstack/nova/+/781076
09:14:41 gibi mnaser: hi, about notification blocking the main thread, there is a bug from belmiro https://bugs.launchpad.net/nova/+bug/1917645
09:15:18 gibi I intended to look into at some point, but never had the time
09:17:01 gibi now I feel again I should look
09:46:12 gibi stephenfin: I have a bad feeling about https://review.opendev.org/c/openstack/nova/+/817746/comment/6690000b_3989ece7/ but if there is no better way then I will +A it
09:46:41 bauzas gibi: looking at https://docs.openstack.org/nova/latest/contributor/ptl-guide.html#milestone-1
09:46:50 bauzas gibi: wondering if all this fish is still needed ?
09:47:48 gibi bauzas: launchpad bookkeeping is really just bookkeeping. If you don't do it then nothing will fail
09:48:04 bauzas gibi: you did this for Xena ?
09:48:06 gibi bauzas: the lib releases are probably proposed by the release team automatically
09:48:09 bauzas if so, I'll do it
09:48:20 bauzas gibi: yeah I'm looking at the releases gerrit
09:48:30 bauzas gibi: I wasn't asked to look at some changes
09:49:33 gibi bauzas: it seems that I only released xena-3 in launchpad not 1, 2 or rc1
09:49:52 bauzas hah, OK
09:49:55 bauzas I'll look at that
09:50:49 gibi bauzas: I think the release team only proposes a release if there was real content on the branch since the last release
09:51:02 bauzas yeah, I only see https://review.opendev.org/c/openstack/releases/+/818415
09:51:12 bauzas elodilles: ^ correct ?
09:51:28 gibi bauzas: we have os-vif release proposed https://review.opendev.org/c/openstack/releases/+/818415

Earlier   Later