Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-03-19
16:44:58 bogdando sean-k-mooney: while that is totally logical, that was not clear to me after reading https://bugs.launchpad.net/nova/+bug/1882094
16:45:49 bogdando that "Removing and moving the DB connection establishment out of our WSGI init script" could be understood like it connects to DB even before config gets validated
16:46:02 dansmith of course config :)
16:46:20 sean-k-mooney bogdando: it could but that not what the code does
16:47:29 bogdando well, sometimes it firstly needs alive MQ connection before to start verifying something else in a config
16:47:43 bogdando not saying that's exactly a case here
16:47:44 sean-k-mooney the api does not
16:48:09 sean-k-mooney so this call to init_global_data is waht calls config.parse_args https://github.com/openstack/nova/blob/master/nova/api/openstack/wsgi_app.py#L128
16:48:18 sean-k-mooney here https://github.com/openstack/nova/blob/master/nova/api/openstack/wsgi_app.py#L101
16:49:17 sean-k-mooney the first time the db is acccesed is here https://github.com/openstack/nova/blob/master/nova/api/openstack/wsgi_app.py#L64
16:49:54 sean-k-mooney which is after we return from init_global_data
16:50:19 sean-k-mooney dansmith: by hte way this is why we are not getting the config error the second time https://github.com/openstack/nova/blob/master/nova/api/openstack/wsgi_app.py#L92-L93
16:50:47 bogdando at least my silly question brought some useful considerations
16:50:47 dansmith sean-k-mooney: ah interesting :)
16:53:06 sean-k-mooney i could proably tweak run_once for this edge case too but im slighly reluctant to broaden the scope of that
16:53:23 bogdando @sean-k-mooney so, that means we can hit that initialization again with an invalid config
16:53:32 bogdando a rogue database connection :)
16:53:51 sean-k-mooney bogdando: yes but rememebr python is ment to die if you dont cache the excption
16:54:02 sean-k-mooney mod_wsgi is the thing that is breaking expections :)
16:54:14 bogdando my point is now this try catch becomes even more nuanced
16:55:24 sean-k-mooney that run_once was added for the orginal bug
16:55:46 bogdando yea, mod_wsgi is the thing that is breaking excEPtions
16:55:48 sean-k-mooney it was there to make sure we dint try and re parse the config amoung other things if it succeed the first time
16:55:51 sean-k-mooney becuase that is an error
16:57:27 sean-k-mooney bogdando: extra context we have shipt the run_once approch for 5 years and backported it to all our downstream release that are still supproted.
16:57:56 sean-k-mooney so while we can condier changing this we have several years fo experince that this work good enough
16:58:09 sean-k-mooney so im trying to be tactical without createing more tech debt
17:00:27 opendevreview sean mooney proposed openstack/nova master: [WIP] raise_forever on wsgi app init https://review.opendev.org/c/openstack/nova/+/945028
17:00:34 bogdando puppet used to be much more restrictive on configs management
17:00:48 bogdando than eating free form snippets
17:01:10 sean-k-mooney pupet was a terribel user experince
17:01:22 bogdando but it was guarding us against this behavior
17:01:30 sean-k-mooney to be clear we are only having this converstaion because healthcheck are not merged
17:02:02 sean-k-mooney we have an implmemtned feature in review for over 2 years that was intned to solve this and other problems
17:02:12 bogdando good point
17:02:42 sean-k-mooney https://review.opendev.org/q/topic:%22per-process-healthchecks%22+status:open
17:10:29 opendevreview sean mooney proposed openstack/nova master: [WIP] raise_forever on wsgi app init https://review.opendev.org/c/openstack/nova/+/945028
17:10:51 sean-k-mooney i just rebased that to master
17:12:05 sean-k-mooney and here is the test patch https://github.com/openstack-k8s-operators/nova-operator/pull/943
17:12:29 sean-k-mooney im going to leave it there for today but i will look at the result tomrrow
17:19:15 opendevreview Mina Shahsavan pour proposed openstack/nova master: Volume Affinity Filter https://review.opendev.org/c/openstack/nova/+/945031
17:26:16 opendevreview Balazs Gibizer proposed openstack/nova master: Ignore metadata tags in pci/stats _find_pool logic https://review.opendev.org/c/openstack/nova/+/944277
17:53:27 dansmith bauzas: Uggla good to approve stuff for master=flamingo yes?
18:28:43 dansmith aight, stable/2025.1 is present so I'm just gonna do it
19:10:30 sean-k-mooney dansmith: do you have any idea what "reservation id" is for a nova instance
19:12:47 sean-k-mooney hum
19:12:51 sean-k-mooney "The reservation id for the server. This is an id that can be useful in tracking groups of servers created with multiple create, that will all have the same reservation_id. By default, it appears in the response for administrative users only."
19:13:25 sean-k-mooney ok apprently that show up even if you dont use multi create
19:14:18 sean-k-mooney i dont think i have seen that in horizon before but i guess i dont use the admin project most of the time
19:18:14 dansmith sean-k-mooney: that may be a holdover from being able to support the ec2 api
19:18:32 sean-k-mooney it was added in 2.3 but maybe
19:18:41 sean-k-mooney i was wondering if it was somehow blazar related
19:18:47 sean-k-mooney just off the name
19:19:19 dansmith orly okay
19:25:06 opendevreview Merged openstack/nova master: Fix missing backtick in configuration option help https://review.opendev.org/c/openstack/nova/+/931253
19:26:11 opendevreview Merged openstack/nova master: Reproduce bug/2098496 https://review.opendev.org/c/openstack/nova/+/941673
22:04:08 opendevreview Merged openstack/nova master: Ignore metadata tags in pci/stats _find_pool logic https://review.opendev.org/c/openstack/nova/+/944277
#openstack-nova - 2025-03-20
07:48:10 opendevreview Adam Kijak proposed openstack/nova stable/2024.1: Fix race condition when attaching a volume and deleting the instance https://review.opendev.org/c/openstack/nova/+/945067
09:17:05 bauzas dansmith: sean-k-mooney: yeah the reservation id is some older field related to ec2 API, nothing related to blazar
09:17:37 bauzas it was existing *before* climate :) (ie. blazar older name)
10:09:59 bauzas gmann: dansmith: when you're up, as a reminder, there is some work to be done on grenade as we can't merge https://review.opendev.org/c/openstack/nova/+/944018 yet for our non-SLURP support enveloppe
10:27:34 sean-k-mooney bauzas: ack, i use horizon im just rarely logged in as admin so i was like what is this feild i never see
10:29:26 bauzas more than a decade ago when we incepted climate, we were wondering the interest of that field and whether it could be useful
10:29:38 bauzas eventually, we said meh
10:30:26 bauzas but afaik, you get the same reservation id for multicreate, that's one way for you to match the relationship (particularly given the request_id isn't public)
10:31:02 bauzas that has to be confirmed, I'm a bit unsure of that
10:37:26 sean-k-mooney ya apprently its just a way to corralate instance create by multi create (and happens to be set on all instnaces probly because of common code path) im not really sure why that useful
10:37:48 sean-k-mooney it would be more useful if we recored the orginal request id but ti probably predates that?
10:38:23 sean-k-mooney at least with the resquest id we could easilly corralate the request to logs if they still existed and it would work across all services
10:39:07 sean-k-mooney bauzas: the resuqest id is set in the header of the request by the user so it is public
10:39:18 sean-k-mooney bauzas: we generate one if its not present using middelware
10:39:54 sean-k-mooney but you as a normal user are free to set it, you just dont have access to the logs to make use of it. the only api that takes it as an input it the instance action log
10:44:08 sean-k-mooney dansmith: by the way my ci run with the invokeation fo kill took the sys.exit path... so either the mod_wsgi docs are wrong about the constant i need to check or it does not work for some other reason
10:44:53 sean-k-mooney dansmith: on the pluse side the raise_forever version worked as expected and cause the k8s pods to fail the ready check and that got reported back up into the CRs
10:45:32 sean-k-mooney so im going to rework https://review.opendev.org/c/openstack/nova/+/945028 with a workaorund config flag to opt into that behavior if you have external monitoring that will detect the error
10:47:54 sean-k-mooney bauzas: by the way i think we have our own variant of teh grenade multi node job
10:48:18 sean-k-mooney bauzas: so if we wanted we could overried the base branch temporally until it get changed for everything else
10:48:48 sean-k-mooney we jsut need to export the wirte varible in the job def
10:49:11 sean-k-mooney or wait like 2 weeks until it gets fixed in greneade either is fine
12:17:37 opendevreview Mina Shahsavan pour proposed openstack/nova master: Volume Affinity Filter https://review.opendev.org/c/openstack/nova/+/945031
12:57:49 opendevreview Balazs Gibizer proposed openstack/nova stable/2025.1: Reproduce bug/2098496 https://review.opendev.org/c/openstack/nova/+/945095
12:57:50 opendevreview Balazs Gibizer proposed openstack/nova stable/2025.1: Ignore metadata tags in pci/stats _find_pool logic https://review.opendev.org/c/openstack/nova/+/945096
12:59:13 opendevreview Balazs Gibizer proposed openstack/nova stable/2024.2: Reproduce bug/2098496 https://review.opendev.org/c/openstack/nova/+/945098
12:59:14 opendevreview Balazs Gibizer proposed openstack/nova stable/2024.2: Ignore metadata tags in pci/stats _find_pool logic https://review.opendev.org/c/openstack/nova/+/945099
12:59:55 opendevreview Balazs Gibizer proposed openstack/nova stable/2024.1: Reproduce bug/2098496 https://review.opendev.org/c/openstack/nova/+/945100
12:59:56 opendevreview Balazs Gibizer proposed openstack/nova stable/2024.1: Ignore metadata tags in pci/stats _find_pool logic https://review.opendev.org/c/openstack/nova/+/945101
13:01:26 opendevreview Balazs Gibizer proposed openstack/nova stable/2023.2: Reproduce bug/2098496 https://review.opendev.org/c/openstack/nova/+/945102
13:01:27 opendevreview Balazs Gibizer proposed openstack/nova stable/2023.2: Ignore metadata tags in pci/stats _find_pool logic https://review.opendev.org/c/openstack/nova/+/945103
13:32:22 sean-k-mooney feck it anyway, of cores i cant use a workaround config flag to decied how to handel invalid config errors
13:32:47 sean-k-mooney s/cores/course/
13:33:45 sean-k-mooney i wrote some nice help text and everything, oh well i guess i can use some of that for the release note
13:38:14 dansmith sean-k-mooney: cool
14:03:03 opendevreview sean mooney proposed openstack/nova master: raise_forever on wsgi app init https://review.opendev.org/c/openstack/nova/+/945028
14:10:13 sean-k-mooney dansmith: ^ i think is in a pretty good state ill kick off one more recheck via the operator repo to to get fresh result with mod_wsgi but it should be good to review
14:10:20 dansmith I've got it open
17:21:30 gmann bauzas: yes, today I got go ahead from release team about all repo are branched now and I will start the QA tooling (including grenade) setup for old/new releases
17:33:11 bauzas gmann: cool excellent
17:57:21 zigo sean-k-mooney: Hey there! Do you happen to know how far we've been from having the virtiofs driver in Manila? Is it in Epoxy, or should we wait another release ?!?
17:59:01 opendevreview sean mooney proposed openstack/nova master: wrap wsgi_app.init_application with latch_error_on_raise https://review.opendev.org/c/openstack/nova/+/945028
17:59:03 sean-k-mooney zigo: its in

Earlier   Later