| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-08-19 | |||
| 09:16:31 | opendevreview | Max proposed openstack/nova master: fix: leftover volume_attachment on instance delete https://review.opendev.org/c/openstack/nova/+/934984 | |
| 12:02:24 | sean-k-mooney | gibi: did you see my comment last week about using RSS instead of adress_sapce for the prlimit for the qemu-img commands by the way? | |
| 12:04:43 | sean-k-mooney | priteau: approved | |
| 12:05:48 | priteau | Thank you sean-k-mooney :) | |
| 12:41:56 | opendevreview | Kamil Sambor proposed openstack/nova master: Switch nova-conductor to use ThreadPoolExecutor https://review.opendev.org/c/openstack/nova/+/957088 | |
| 13:18:08 | gibi | sean-k-mooney: I saw and made a note abuot it | |
| 13:18:39 | gibi | I have to get back to it eventually | |
| 13:27:16 | sean-k-mooney | gibi: i assum you have not pushed a poc patch to bump the limit yet? i could try and push one to swap ti to rss and depend on the DNM ci job revert patch you have | |
| 13:32:53 | gibi | sean-k-mooney: I only pinpointed the address space limit increase we need on master +5MB. And I tried to see how much address space we used before the new ceph but I cannot get that data as i) I cannot roll back the ceph version in ubuntu as the package repo already cleaned ii) the older ubuntu based jobs does not want to run the specific test for me even though I think I set everything. But I had | |
| 13:32:59 | gibi | not time to dig deep into RSS at all | |
| 13:33:39 | gibi | the problem with RSS is similar, we don't know the baseline usage. | |
| 13:34:05 | gibi | so we cannot judge the that the increase is just small and OK, or it is a huge bump from the previous version | |
| 13:34:34 | gibi | if you have time you sure you can play with it to get an RSS number we need on master | |
| 13:35:13 | sean-k-mooney | well i was orginally thinking fo using the same 1G limit to start with but on RSS not adress space | |
| 13:35:35 | sean-k-mooney | and then cosndiering if we need to factor it out to a tuneable config option later | |
| 13:36:10 | sean-k-mooney | the orgianl intent of the limit was to workaroudn some pathological beahivor in old version of qemu-img with untrusted input | |
| 13:36:13 | gibi | I suggest to at least try locally with different volume sizes to see if the RSS is depending on the size of the volume | |
| 13:36:32 | gibi | sean-k-mooney: I affraid we might have a new pathological behavior :) | |
| 13:37:02 | gibi | today the test resizes an 1G volume to a 2G volume | |
| 13:37:02 | sean-k-mooney | well i think sinc ethey are now recfacotring things to use stackless corutiens in the newer version of c++ | |
| 13:37:20 | gibi | so the test is not really realistic. Out there the volumes are lot bigger | |
| 13:37:22 | sean-k-mooney | that the amoutn of adress space has increase but not nessisarly the amount of commit memory | |
| 13:37:32 | gibi | yeah I agree about that ^^ | |
| 13:37:36 | gibi | so RSS is a good direction | |
| 13:37:44 | sean-k-mooney | gibi: in principal this does not matter abot voluems corroct | |
| 13:37:56 | sean-k-mooney | we coudl do some local testign with files of diffent size | |
| 13:38:12 | sean-k-mooney | or do you think the volumes actully play an imporant role | |
| 13:38:28 | sean-k-mooney | oh actully this iss in the rbd code path | |
| 13:38:39 | sean-k-mooney | so we likely do need ceph right? | |
| 13:38:42 | gibi | either the volume, or the fact that it is luksv1 encrypted, or the fact that the test case is resizing plays a role as other test cases does not fail | |
| 13:39:05 | sean-k-mooney | ack so ya we woudl need to test a few diffent ways. | |
| 13:39:19 | gibi | File "/opt/stack/tempest/tempest/api/volume/admin/test_encrypted_volumes_extend.py", line 41, in test_extend_attached_encrypted_volume_luksv1 | |
| 13:39:21 | sean-k-mooney | i wonder fi there is a way to log this info | |
| 13:39:24 | gibi | this is the test case that triggers it | |
| 13:39:55 | sean-k-mooney | we porobaly shoudl park this for now but sinc ehte ceph folks are interested i also dont want to loose too much momentem | |
| 13:40:04 | gibi | we call a bunch of qemu-img info during tempest, but only this test trigger the out of memory for qemu-img info | |
| 13:40:31 | sean-k-mooney | ya so dans comment that the encyption may impact it seams valid | |
| 13:40:36 | gibi | yeah I have a bz open on ceph and they are responsive but we don't have a clear non tempest repro for them | |
| 13:41:11 | sean-k-mooney | i was wondering if we coudl create a smaller non openstack repdocuer too | |
| 13:41:18 | sean-k-mooney | that why i was wondering about a file | |
| 13:41:40 | gibi | I copied the volume out of ceph as a file and run the same command it did not failed | |
| 13:41:41 | sean-k-mooney | but we could likely jsut see if we can manually repoduce this with ceph and qemu directly | |
| 13:41:51 | sean-k-mooney | i just dont happen to have a ceph install to hand currently | |
| 13:41:53 | gibi | so rbd is in the picture | |
| 13:42:02 | sean-k-mooney | ack | |
| 13:42:32 | sean-k-mooney | so im wondering is it as simple as create a raw image wiht luks, upload that to ceph as a voluem | |
| 13:42:50 | sean-k-mooney | extend the volume and messure the adress space and rss | |
| 13:42:56 | sean-k-mooney | fo qemu image as you grow it | |
| 13:43:19 | gibi | yeah that can be tried | |
| 13:43:39 | gibi | realistically for me it is not earlier than next week when I can try | |
| 13:43:54 | sean-k-mooney | ya i was goign to say the same | |
| 13:43:57 | sean-k-mooney | post FF | |
| 13:44:32 | sean-k-mooney | i can do the quick RSS hack and push but digging deeper will take more time then ill have before FF or perhaps RC1 is done | |
| 13:45:09 | gibi | ack | |
| 14:07:36 | dansmith | gibi: I wonder if you could look at this conversation: https://review.opendev.org/c/openstack/nova/+/952630/12/nova/conductor/manager.py#322 | |
| 14:07:52 | dansmith | I really hate this idea that we have to do a mini data migration (from instance to reqspec) every time we do a move operation | |
| 14:08:18 | dansmith | it feels fragile to me, like we could end up missing that step in some cases, and/or get other weird rebuild/evac/whatever behavior later, and after hours of debugging, | |
| 14:08:36 | dansmith | realize that one instance worked because it had been migrated once (and thus got the reqspec change) and another failed because it had never been migrated | |
| 14:09:23 | gibi | dansmith: OK, added to my list along with the other place https://review.opendev.org/c/openstack/nova/+/942502/13/nova/scheduler/request_filter.py#477 melwitt ping me about yesterday | |
| 14:09:27 | dansmith | (in this case I mean s/migrated/moved in some way/) | |
| 14:09:39 | dansmith | gibi: ack, thanks | |
| 14:14:09 | dansmith | gibi: that's a similar theme to the question above.. all about how we define, choose, and expose the default policy we choose for new instances when they don't otherwise request one | |
| 15:01:49 | Uggla | Hello, nova meeting in ~1h. | |
| 15:05:46 | jssfr | Uggla, I might not make it, but I'd like to bring https://review.opendev.org/c/openstack/nova/+/955657 to the table -- I managed to (hopefully) bring the patch into shape as discussed two weeks ago and I'd appreciate any reviews or feedback :) | |
| 15:06:01 | jssfr | I'll read up on IRC discussion if I'm pinged here, too (but probably I'm going to be preparing dinner at the meeting time :-X) :) | |
| 16:00:25 | Uggla | #startmeeting nova | |
| 16:00:25 | opendevmeet | Meeting started Tue Aug 19 16:00:25 2025 UTC and is due to finish in 60 minutes. The chair is Uggla. Information about MeetBot at http://wiki.debian.org/MeetBot. | |
| 16:00:25 | opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | |
| 16:00:25 | opendevmeet | The meeting name has been set to 'nova' | |
| 16:00:28 | gibi | o/ | |
| 16:00:30 | tkajinam | o/ | |
| 16:00:33 | Uggla | Hello everyone | |
| 16:00:33 | gmaan | o/ | |
| 16:01:05 | Uggla | Awaiting a couple of minute so people can join. | |
| 16:02:06 | Uggla | #topic Bugs (stuck/critical) | |
| 16:02:06 | elodilles | o/ | |
| 16:02:16 | Uggla | #info No Critical bug | |
| 16:02:25 | Uggla | #topic Gate status | |
| 16:02:32 | Uggla | #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs | |
| 16:02:39 | Uggla | #link https://etherpad.opendev.org/p/nova-ci-failures-minimal | |
| 16:02:45 | Uggla | #link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&branch=stable%2F*&branch=master&pipeline=periodic-weekly&skip=0 Nova&Placement periodic jobs status | |
| 16:02:52 | Uggla | #info Please look at the gate failures and file a bug report with the gate-failure tag. | |
| 16:02:58 | Uggla | #info Please try to provide a meaningful comment when you recheck | |
| 16:03:35 | Uggla | I checked quickly the report and don't see anything scary. | |
| 16:04:38 | Uggla | #topic tempest-with-latest-microversion job status | |
| 16:04:42 | gibi | we have a security adjacent bug switched to public yesterday. Fix landed on master and stable backports are progressing. https://bugs.launchpad.net/nova/+bug/2112187 | |
| 16:04:49 | Uggla | gmaan, do you want to say anything about it ? | |
| 16:04:57 | gmaan | sorry, nothing from me. | |
| 16:05:25 | Uggla | thanks gmaan | |
| 16:05:34 | Uggla | gibi, sorry I was a bit fast. | |
| 16:05:53 | gibi | I just wanted to raise attention to that bug and the related fixes | |
| 16:06:16 | gibi | we need to do a tempest dance to skip some tests, land the nova fix, then re-enable tests with fixes applied | |
| 16:06:25 | elodilles | regarding the sec bug fix: do we want to do stable releases once those fixes landed? | |
| 16:06:35 | gibi | elodilles: yes we want | |
| 16:06:57 | elodilles | gibi: ACK, then I'll prepare some release patches | |
| 16:07:00 | gibi | thanks | |
| 16:07:05 | elodilles | np | |
| 16:07:32 | Uggla | thx elodilles | |
| 16:07:39 | gibi | Uggla: back to you | |
| 16:08:45 | Uggla | #topic Release Planning | |