| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-08-15 | |||
| 13:49:09 | gibi | I tried to see how close we were the limit in 19.2.0 but I cannot roll back to that version any more locally | |
| 13:49:42 | dansmith | so tools like this have to work pretty hard to not load the entire image into memory so I think it's probably a good idea for us to have that limit | |
| 13:50:15 | gibi | yeah, I affraid that this is a wack a mole just bumping the limit blindly | |
| 13:50:17 | dansmith | and TBH, I'm surprised that it needs to be 1G to run qemu-img info, which probably means a bug that this is catching | |
| 13:50:53 | gibi | the Ceph folks so far was responsive in the bz. So hopefully they can come up with a way to troubleshoot it further | |
| 13:50:58 | dansmith | unless there's a very large extent size on a volume or something, causing us to buffer that large | |
| 13:51:23 | dansmith | I would think making it configurable would be a good idea | |
| 13:51:26 | jssfr | and/or a non-raw format | |
| 13:52:16 | dansmith | jssfr: it's hard but not impossible to support the non-raw formats in constant memory, but that's an easy place for bugs to creep in of course | |
| 13:52:24 | jssfr | yep. | |
| 13:52:31 | dansmith | I was thinking more the LUKS chunk size, since this is encrypted | |
| 13:52:39 | jssfr | ah, I was thinking rbd block size | |
| 13:52:40 | dansmith | but either way, that sort of thing is exactly why having a limit is a good idea | |
| 13:52:54 | dansmith | could be either and/or some relation between rbd and LUKS here | |
| 13:52:59 | jssfr | (which is typically 4 MiB. if you make it read 256 different blocks, mix in some librbd caching...) | |
| 13:53:13 | jssfr | agreed, limits good. | |
| 13:53:21 | jssfr | (in particular when remembering why that qemu-info call is there in the first place.) | |
| 13:53:36 | jssfr | (ah the fun of rolling out fixes for that CVE while in a train to a customer meeting :-)) | |
| 13:57:36 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM: Reproduce bug/2116852 https://review.opendev.org/c/openstack/nova/+/957419 | |
| 14:04:51 | opendevreview | Balazs Gibizer proposed openstack/nova stable/2024.2: DNM: Reproduce bug/2116852 with decreasing mem limit https://review.opendev.org/c/openstack/nova/+/957541 | |
| 14:05:13 | gibi | let see how close we are to the 1G limit on Jammy with Ceph 17 ^^ | |
| 14:11:28 | sean-k-mooney | gibi: i think any limit like that shoudl really be configureabl and not hard coded | |
| 14:12:22 | sean-k-mooney | i dont recall the history but im i also kidn of agree that we dont want to kust keep bumping it either | |
| 14:13:57 | sean-k-mooney | personally i would make that and the cpu times config options, leave the time as is and bump the memory to maybe 2-4G | |
| 14:14:16 | sean-k-mooney | gibi: the thing im wonderingis we are setting adress_space | |
| 14:14:35 | sean-k-mooney | is that what we actully creare about or do we care about resident memory limit | |
| 14:15:36 | sean-k-mooney | if qemu-info uses 10TB of adress space but never uses more then 100MB of commit memory that perficly fine so we myigh be setting the wrong limit | |
| 14:17:35 | sean-k-mooney | hum https://github.com/openstack/oslo.concurrency/blob/master/oslo_concurrency/processutils.py#L125-L134 | |
| 14:18:56 | sean-k-mooney | so https://github.com/openstack/oslo.concurrency/blob/master/oslo_concurrency/prlimit.py#L25-L68 prlimti supprot limiting on RSS | |
| 14:19:10 | sean-k-mooney | that actully what we likely care about | |
| 14:19:40 | sean-k-mooney | so we shoudl likely be using resident_set_size instead of of adress_space here https://github.com/openstack/nova/blob/e39bac965a2d947d32325446ef8427dd5f6c85c7/nova/privsep/qemu.py#L36 | |
| 14:22:54 | sean-k-mooney | gibi: if i recall when we were looking at this there was some refactoring happeing invovlg moving to differnt stackless coretine implemations and how they used allcoators so while that may have increase memrory usage my supisiton is it may have increase adresssapce usage but not nessisarly RSS | |
| 14:26:10 | sean-k-mooney | gibi: this is where we first added this https://github.com/openstack/nova/commit/068d851561addfefb2b812d91dc2011077cb6e1d for https://bugs.launchpad.net/nova/+bug/1449062 | |
| 14:26:36 | sean-k-mooney | gibi: so we added it to wrkaournd bugs in qemu-img | |
| 14:30:45 | sean-k-mooney | i also agree with dan that qemu-img should not need that much memroy in genreal so before bumping the limit i would move it form adress space to rss and see if that is enough | |
| 15:02:38 | opendevreview | Balazs Gibizer proposed openstack/nova master: Warn on long task wait time for executor https://review.opendev.org/c/openstack/nova/+/952666 | |
| 15:02:38 | opendevreview | Balazs Gibizer proposed openstack/nova master: [vncproxy]Handle ssl.wrap_socket removal in py312 https://review.opendev.org/c/openstack/nova/+/955915 | |
| 15:26:45 | opendevreview | Markus Hentsch proposed openstack/nova master: Adapt to Cinder and Glance changes for image encryption https://review.opendev.org/c/openstack/nova/+/926326 | |
| 15:47:28 | opendevreview | Markus Hentsch proposed openstack/nova master: Adapt to Cinder and Glance changes for image encryption https://review.opendev.org/c/openstack/nova/+/926326 | |
| 15:48:19 | opendevreview | Markus Hentsch proposed openstack/nova master: Adapt to Cinder and Glance changes for image encryption https://review.opendev.org/c/openstack/nova/+/926326 | |
| 19:26:55 | dansmith | melwitt: thanks for hitting that image encryption one.. meant to ping you | |
| 19:48:40 | melwitt | np. I wondered why it didn't start gate jobs, looks like it Depends-On a cinder change that isn't approved yet | |
| #openstack-nova - 2025-08-16 | |||
| 02:21:00 | opendevreview | melanie witt proposed openstack/nova master: DNM vtpm tempest https://review.opendev.org/c/openstack/nova/+/957477 | |
| 04:14:11 | opendevreview | Ghanshyam proposed openstack/nova master: Add service role in Nova policy https://review.opendev.org/c/openstack/nova/+/957578 | |
| 04:18:36 | opendevreview | Ghanshyam proposed openstack/nova master: Add service role in Nova policy https://review.opendev.org/c/openstack/nova/+/957578 | |
| 18:13:18 | opendevreview | melanie witt proposed openstack/nova master: DNM vtpm tempest https://review.opendev.org/c/openstack/nova/+/957477 | |
| 20:08:18 | opendevreview | Ghanshyam proposed openstack/nova master: Add service role in Nova policy https://review.opendev.org/c/openstack/nova/+/957596 | |
| 20:09:53 | opendevreview | Ghanshyam proposed openstack/nova master: Add service role in Nova policy https://review.opendev.org/c/openstack/nova/+/957578 | |
| #openstack-nova - 2025-08-18 | |||
| 08:11:09 | priteau | Good morning Nova. Would a core reviewer be able to approve this backport? Thanks! https://review.opendev.org/c/openstack/nova/+/945096 | |
| 08:37:07 | gibi | priteau: o/ I would but that is my backport :) | |
| 08:43:49 | priteau | I will try again later, it's still early for the US folks :) | |
| 08:44:55 | gibi | yeah. bauzas is still on PTO | |
| 08:45:02 | gibi | and US is asleep | |
| 08:55:58 | opendevreview | Balazs Gibizer proposed openstack/nova stable/2024.2: DNM: Reproduce bug/2116852 with decreasing mem limit https://review.opendev.org/c/openstack/nova/+/957541 | |
| 09:08:22 | opendevreview | Balazs Gibizer proposed openstack/nova master: Warn on long task wait time for executor https://review.opendev.org/c/openstack/nova/+/952666 | |
| 09:08:22 | opendevreview | Balazs Gibizer proposed openstack/nova master: [vncproxy]Handle ssl.wrap_socket removal in py312 https://review.opendev.org/c/openstack/nova/+/955915 | |
| 13:00:45 | opendevreview | Takashi Kajinami proposed openstack/nova master: Remove unused validator for uri element https://review.opendev.org/c/openstack/nova/+/951084 | |
| 13:05:35 | tkajinam | it has been a while since https://review.opendev.org/c/openstack/nova/+/879021 got one +2. I wonder if it can be merged. | |
| 13:15:12 | opendevreview | Balazs Gibizer proposed openstack/nova stable/2024.2: DNM: Reproduce bug/2116852 with decreasing mem limit https://review.opendev.org/c/openstack/nova/+/957541 | |
| 15:15:47 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Fix validators for hw:cpu_max_* extra specs https://review.opendev.org/c/openstack/nova/+/957732 | |
| 15:21:19 | opendevreview | Stephen Finucane proposed openstack/nova stable/2025.1: api: Fix validators for hw:cpu_max_* extra specs https://review.opendev.org/c/openstack/nova/+/957739 | |
| 15:25:05 | opendevreview | Stephen Finucane proposed openstack/nova stable/2024.2: api: Fix validators for hw:cpu_max_* extra specs https://review.opendev.org/c/openstack/nova/+/957744 | |
| 15:25:29 | opendevreview | Stephen Finucane proposed openstack/nova stable/2024.1: api: Fix validators for hw:cpu_max_* extra specs https://review.opendev.org/c/openstack/nova/+/957745 | |
| 16:05:38 | opendevreview | sean mooney proposed openstack/nova master: restrict swap volume to cinder https://review.opendev.org/c/openstack/nova/+/957757 | |
| 16:06:23 | opendevreview | sean mooney proposed openstack/nova stable/2025.1: restrict swap volume to cinder https://review.opendev.org/c/openstack/nova/+/957759 | |
| 16:06:52 | opendevreview | sean mooney proposed openstack/nova stable/2024.2: restrict swap volume to cinder https://review.opendev.org/c/openstack/nova/+/957762 | |
| 16:07:23 | opendevreview | sean mooney proposed openstack/nova stable/2024.1: restrict swap volume to cinder https://review.opendev.org/c/openstack/nova/+/957764 | |
| 16:11:20 | opendevreview | sean mooney proposed openstack/nova master: restrict swap volume to cinder https://review.opendev.org/c/openstack/nova/+/957757 | |
| 16:11:42 | opendevreview | sean mooney proposed openstack/nova master: restrict swap volume to cinder https://review.opendev.org/c/openstack/nova/+/957757 | |
| 16:13:54 | opendevreview | sean mooney proposed openstack/nova stable/2025.1: restrict swap volume to cinder https://review.opendev.org/c/openstack/nova/+/957759 | |
| 16:14:42 | opendevreview | sean mooney proposed openstack/nova stable/2024.2: restrict swap volume to cinder https://review.opendev.org/c/openstack/nova/+/957762 | |
| 16:15:29 | opendevreview | sean mooney proposed openstack/nova stable/2024.1: restrict swap volume to cinder https://review.opendev.org/c/openstack/nova/+/957764 | |
| 17:22:44 | melwitt | gibi: hi, sorry there is one more thread on the vtpm series that would appreciate your thought. it's about the default_tpm_secret_security config option + the user confirmation mechanism https://review.opendev.org/c/openstack/nova/+/942502/13/nova/scheduler/request_filter.py#477 | |
| 17:23:11 | gibi | melwitt: OK i will check | |
| 17:24:31 | melwitt | gibi: thanks 🙏 the last few comments are where you are mentioned for input | |
| 22:18:56 | opendevreview | Merged openstack/nova master: api: Deprecate v2 API https://review.opendev.org/c/openstack/nova/+/954102 | |
| #openstack-nova - 2025-08-19 | |||
| 01:00:43 | opendevreview | Merged openstack/nova master: restrict swap volume to cinder https://review.opendev.org/c/openstack/nova/+/957757 | |
| 05:38:34 | gibi | elodilles: when you are up could you look at https://review.opendev.org/q/topic:%22bug/2112187%22+project:openstack/nova backports. It is important as it is a security hardening bug | |
| 07:02:45 | sean-k-mooney[m] | gibi thanks for updating tempest, is that third test new? i tought ran tempest full - the encyption adn cinder backup tests on friday so im not sure why i didnt see it fail | |
| 07:07:11 | sean-k-mooney[m] | hum no i must not have set the approate tempest config locally and it must have skiped | |
| 07:07:48 | sean-k-mooney[m] | i proably didnt explictly define the multiatch volume type of something along those lines | |
| 07:17:18 | gibi | sean-k-mooney[m]: yeah it is not new | |
| 09:08:06 | opendevreview | Merged openstack/nova stable/2025.1: restrict swap volume to cinder https://review.opendev.org/c/openstack/nova/+/957759 | |
| 09:16:31 | opendevreview | Max proposed openstack/nova master: fix: leftover volume_attachment on instance delete https://review.opendev.org/c/openstack/nova/+/934984 | |
| 12:02:24 | sean-k-mooney | gibi: did you see my comment last week about using RSS instead of adress_sapce for the prlimit for the qemu-img commands by the way? | |
| 12:04:43 | sean-k-mooney | priteau: approved | |
| 12:05:48 | priteau | Thank you sean-k-mooney :) | |
| 12:41:56 | opendevreview | Kamil Sambor proposed openstack/nova master: Switch nova-conductor to use ThreadPoolExecutor https://review.opendev.org/c/openstack/nova/+/957088 | |
| 13:18:08 | gibi | sean-k-mooney: I saw and made a note abuot it | |
| 13:18:39 | gibi | I have to get back to it eventually | |
| 13:27:16 | sean-k-mooney | gibi: i assum you have not pushed a poc patch to bump the limit yet? i could try and push one to swap ti to rss and depend on the DNM ci job revert patch you have | |
| 13:32:53 | gibi | sean-k-mooney: I only pinpointed the address space limit increase we need on master +5MB. And I tried to see how much address space we used before the new ceph but I cannot get that data as i) I cannot roll back the ceph version in ubuntu as the package repo already cleaned ii) the older ubuntu based jobs does not want to run the specific test for me even though I think I set everything. But I had | |
| 13:32:59 | gibi | not time to dig deep into RSS at all | |
| 13:33:39 | gibi | the problem with RSS is similar, we don't know the baseline usage. | |
| 13:34:05 | gibi | so we cannot judge the that the increase is just small and OK, or it is a huge bump from the previous version | |
| 13:34:34 | gibi | if you have time you sure you can play with it to get an RSS number we need on master | |
| 13:35:13 | sean-k-mooney | well i was orginally thinking fo using the same 1G limit to start with but on RSS not adress space | |
| 13:35:35 | sean-k-mooney | and then cosndiering if we need to factor it out to a tuneable config option later | |
| 13:36:10 | sean-k-mooney | the orgianl intent of the limit was to workaroudn some pathological beahivor in old version of qemu-img with untrusted input | |
| 13:36:13 | gibi | I suggest to at least try locally with different volume sizes to see if the RSS is depending on the size of the volume | |