Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-09-24
14:29:01 sean-k-mooney priteau: by the way the template also tells you to run udo sosreport -o openstack_nova --batch
14:29:12 sean-k-mooney which i dont think i have ever seen peopl actully do
14:29:20 priteau Who follows instructions? :)
14:29:34 priteau Would you like people to use `pip3 list | grep nova` as an alternative?
14:30:08 sean-k-mooney not nessisarly but its helpful if they clearly state that they used train or the serise name
14:30:16 opendevreview Lucian Petrut proposed openstack/nova master: api: enable oslo.reports when using uwsgi https://review.opendev.org/c/openstack/nova/+/810922
14:30:27 sean-k-mooney knowing the disto and or package version is nice too
14:32:05 priteau I imagine knowing the release tag can be quite useful
14:32:15 opendevreview Lucian Petrut proposed openstack/nova master: api: enable oslo.reports when using uwsgi https://review.opendev.org/c/openstack/nova/+/810922
14:32:44 gibi just yesterday I troubleshooted a deployment with nova_compute version 22.2.3 :D
14:33:08 lpetrut hi, I'm hitting some nova api deadlocks and noticed that oslo.reports isn't enabled when using uwsgi so I've submitted a small commit: https://review.opendev.org/c/openstack/nova/+/810922
14:33:16 gibi (note that we only released 22.2.2 upstream)
14:33:48 sean-k-mooney gibi: ya i was going to say was this in the gate :)
14:34:19 sean-k-mooney cause otherwise they are going to have fun when we do the next stable release
14:34:50 gibi sean-k-mooney: it was downstream. I think what they did is they took what was unreleased from stable/victoria and created 22.2.3 out of it downstream
14:35:10 sean-k-mooney i see
14:35:26 gibi which is problematic as you said
14:35:32 gibi lpetrut: seems useful
14:35:59 gibi lpetrut: thanks
14:36:33 sean-k-mooney huh
14:37:19 sean-k-mooney maybe that is why the GMR were not working instead of what we tought with the signal being intercpted by uwsgi/mod_wsgi
14:37:42 gibi sean-k-mooney: or we need both :)
14:37:52 sean-k-mooney lpetrut: i assume you tested this and it logs the GMR to the log properly on kill -usr2
14:38:24 sean-k-mooney gibi: ya i was wonderign if that would only work if you set the signal to the python interpreter instnace
14:39:17 gibi unfortunately this is pretty complicated to test upstream.
14:39:22 lpetrut sean-k-mooney: the signal still gets intercepted but I'm using a file listener
14:39:47 gibi I mean automatically testing it
14:39:48 sean-k-mooney lpetrut: intercepted by uwsgi and not passed to nova right
14:40:04 sean-k-mooney lpetrut: oh are you poking a file to trigger it
14:40:14 sean-k-mooney instead of sig_usr2 or soemthing
14:40:38 lpetrut yep, I'm setting something like oslo_reports.file_event_handler = /opt/stack/logs/trigger
14:40:48 sean-k-mooney gibi: we could proably add a func test but we would have to expand the test scope
14:40:57 sean-k-mooney lpetrut: ok i dont think we technially support that in nova
14:41:07 sean-k-mooney but it certenly works around the issue
14:41:27 lpetrut it already works with most nova services, they key is to pass the config when calling the gmr hook
14:41:30 sean-k-mooney so this is really a mini feature rather then a bug
14:41:34 gibi sean-k-mooney: do we run nova-api in uwsgi in func test?
14:41:56 sean-k-mooney gibi: no but we coudl do somehting like neutorn fullstack tests
14:42:03 gibi sean-k-mooney: ack, that is a possibility
14:42:04 sean-k-mooney it would be a different type of test
14:42:20 gibi sean-k-mooney: or add this to nova-next post test hook
14:42:33 sean-k-mooney gibi: ya that too
14:43:00 gibi bauzas, sean-k-mooney: btw I confirm that neutron also leaks inventories if the bw or pps config is removed
14:43:16 sean-k-mooney lpetrut: did you want to backport udo sosreport -o openstack_nova --batch
14:43:24 sean-k-mooney * https://review.opendev.org/c/openstack/nova/+/810922/2/nova/api/openstack/wsgi_app.py
14:43:52 sean-k-mooney to me this is really a specless blueprint
14:43:58 sean-k-mooney so not something we woudl backport
14:44:18 sean-k-mooney i think its a nice change to merge so no real objection to the patch
14:45:02 sean-k-mooney just not sure this is a bug and a spec is way to heavy weight so not sure how to track this
14:45:23 sean-k-mooney to me its really just a trivial fix but it proably should have a release note
14:46:45 lpetrut yeah, it's hard to label it as a bug in order to allow backports but that's ok. a release note makes sense, I can also mention the fact that uwsgi may intercept SIGUSR2, in which case a file trigger may be configured
14:47:35 sean-k-mooney lpetrut: ya if you add a release note and maybe add a doc for the intercept i would be +1 on it
14:48:07 lpetrut awesome, thanks. is there a specific doc that you have in mind?
14:48:28 sean-k-mooney we have a doc for GMR i think in the contibutor section
14:49:17 sean-k-mooney https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/doc/source/reference/gmr.rst
14:49:20 sean-k-mooney ah its in refernce
14:49:44 sean-k-mooney can you update https://github.com/openstack/nova/blob/master/doc/source/reference/gmr.rst#generating-a-gmr
14:50:07 sean-k-mooney with the exmaple of the file trigger
14:50:14 lpetrut definitely, thanks for the link
14:51:00 lpetrut the "TextGuruMeditation.setup_autorun(version)" hook sample should also be updated. if we don't pass the config, gmr will not be aware of the [oslo_reports] config opts
14:56:32 sean-k-mooney ya i dont think we have really updated it since it was added
14:57:26 sean-k-mooney i would suggest updating the existig singal based exmaple to use nova-compute and then adding the file example for nova-api and makeign any other changes that you think are needed
14:59:54 lpetrut sounds good
15:15:07 bauzas gibi: ack, so we need to discuss this during the PTG
15:15:52 gibi yepp
15:15:57 gibi added notes to the bad
15:16:00 gibi pad
15:16:29 lpetrut one minor nit: gmr.setup_autorun takes a "service_name" parameter which is used when constructing the report filename. when missing, it's trying to retrieve it from the stack trace but it seems to always end up with "thread.py", so the reports are named something like "thread.py_gurumeditation_20210924141722". since none of the other service pass this parameter, I'm thinking about doing the same for nova-uwsgi for consistency reasons.
15:19:54 sean-k-mooney lpetrut: or you could fix them all
15:20:05 sean-k-mooney lpetrut: you should be able to use the service binary name
15:20:29 sean-k-mooney so service_obj.binary
15:20:33 lpetrut that works as well
15:21:16 sean-k-mooney that way if you have multiple service on the same host using the same file tirrger they wont overright
15:21:31 sean-k-mooney although the timestamp is unlikely to collide in anycase
15:24:03 lpetrut right, it's also more user friendly since it's easier to tell which is the originating service
15:25:24 sean-k-mooney well when not using the file backend it dumps to the service log so that not been an issue before but for dumping the GMR to a file its something we shoudl definetly address
15:41:31 gibi sean-k-mooney: btw, I tried your echo 0 > numa_node trick and it works like a charm. I can now confirm that live migration with SRIOV + NUMA works and the numa topology is properly recalculated
15:44:45 sean-k-mooney thats an old trick i have been using for ever
15:45:57 gibi this knowledge is gold
15:48:47 sean-k-mooney if you ignore the horrible hack that it is you can actully write udev rules to allwo you to assocaite the device with other numa nodes on the same socket if you enabel cluster on die or amds numa_per_socket>1
15:48:47 opendevreview Lucian Petrut proposed openstack/nova master: api: enable oslo.reports when using uwsgi https://review.opendev.org/c/openstack/nova/+/810922
15:55:04 sean-k-mooney ok going to finish there o/ talk to ye in a week
16:36:32 fungi elodilles: has there been any progress on discussions of whether to discontinue lower-constraints jobs on nova's stable/ussuri branch? at this point, nothing (including an outstanding security fix) can merge there, so it's probably time to start talking about early eol instead
16:37:44 fungi at least if we're up front with users that we're no longer fixing known vulnerabilities on that branch, the vmt can go forward with announcing fixes on the nova branches which are still receiving patches
17:08:24 opendevreview Rodolfo Alonso proposed openstack/nova master: Set "cache_ok=True" in "TypeDecorator" inheriting classes https://review.opendev.org/c/openstack/nova/+/807359
17:58:17 opendevreview Artom Lifshitz proposed openstack/nova master: Gracefully power off guest on instance delete https://review.opendev.org/c/openstack/nova/+/808474
17:58:17 opendevreview Artom Lifshitz proposed openstack/nova master: "Regression" test for server delete https://review.opendev.org/c/openstack/nova/+/810951
17:58:18 opendevreview Artom Lifshitz proposed openstack/nova master: WIP: Make os_shutdown_timer a proper image property https://review.opendev.org/c/openstack/nova/+/810952
18:02:57 opendevreview Artom Lifshitz proposed openstack/nova master: "Regression" test for server delete https://review.opendev.org/c/openstack/nova/+/810951
18:02:57 opendevreview Artom Lifshitz proposed openstack/nova master: Gracefully power off guest on instance delete https://review.opendev.org/c/openstack/nova/+/808474
18:02:58 opendevreview Artom Lifshitz proposed openstack/nova master: WIP: Make os_shutdown_timer a proper image property https://review.opendev.org/c/openstack/nova/+/810952
18:57:41 elodilles fungi: there is a patch that sets the lower-constraints job non-voting, but unfortunately it needed an impressive amount of rechecks and it is still not merged so far ( https://review.opendev.org/c/openstack/nova/+/809955 )
18:59:06 elodilles also there is another patch from gibi that pins the setuptools ( https://review.opendev.org/c/openstack/nova/+/810461 )
19:04:22 fungi note that the lower-constraints job was already not passing on stable/ussuri when setuptools updated
19:12:41 elodilles how do you mean?
19:45:32 elodilles what I remember is that the gate was not blocked until the gate started to use setuptools 58.0.4.
19:45:50 elodilles even lower-constraints jobs were passing
19:48:23 elodilles and the passing setuptools pinning patch also proves that there is no other issue (for now) than the one that was introduced by the latest setuptools release
20:06:53 fungi elodilles: oh, you're right, we got setuptools 58 in tox jobs when virtualenv 20.8.0 was released on 2021-09-16 at 12:52 utc and change 806628 was unlucky enough to be approved a few hours later at 16:23 utc that same day
20:10:16 fungi change 810461 could probably instead do requires = virtualenv<20.8 if you wanted to be more flexible

Earlier   Later