| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-09-12 | |||
| 14:09:54 | sean-k-mooney | memory_tracker low_point: 1900960 | |
| 14:09:59 | gibi | it is cputime from ps | |
| 14:10:10 | gibi | https://github.com/openstack/devstack/blob/f6d8dab0e885b8de8c0f44388d538da7d4f9b7ec/tools/memory_tracker.sh#L88C66-L88C73 | |
| 14:10:20 | sean-k-mooney | so the low point was almost 2G fo aviabel ram | |
| 14:11:10 | opendevreview | Julien LE JEUNE proposed openstack/nova master: nova-condutor puts instance in error state https://review.opendev.org/c/openstack/nova/+/901655 | |
| 14:11:33 | sean-k-mooney | teh resident memory of nova-comptue seams to be around 165mb so if we round up to say 200MB | |
| 14:12:15 | sean-k-mooney | we can proably run 20-25 fake agent on the contoler and i would guess 50+ on the 8G subnodes based on ram | |
| 14:13:15 | sean-k-mooney | my question woudl then be what those the cpu load look like but you proably can simulate a 200 node cluster if needed | |
| 14:14:02 | sean-k-mooney | if you were to try that i woudl proably do 4 dedicatec comptue and disable the nova-comptues on the contoler entirly | |
| 14:15:43 | sean-k-mooney | gibi interesting i wonder if we shoudl make the colelcted parmaters configurable via an devstack environment var | |
| 14:16:25 | gibi | soo many options:) | |
| 14:16:38 | sean-k-mooney | ill stop suggesting them | |
| 14:18:29 | sean-k-mooney | your current patch has times 400 and concurrency 40 + 10 fake computes | |
| 14:18:37 | sean-k-mooney | and you turned on atop | |
| 14:18:46 | sean-k-mooney | so lets see what that shows | |
| 14:19:31 | sean-k-mooney | ... post failure | |
| 14:19:39 | sean-k-mooney | rally: error: unrecognized arguments: -n 2 | xargs rally task trends --out /opt/stack/.rally/results/trends.html --tasks | |
| 14:20:31 | opendevreview | Sylvain Bauza proposed openstack/nova master: Support multiple allocations for vGPUs https://review.opendev.org/c/openstack/nova/+/845757 | |
| 14:22:31 | gibi | ahh probably ansible command vs shell. I use pipes | |
| 14:27:08 | sean-k-mooney | ah right proably | |
| 14:27:28 | sean-k-mooney | ya your using command | |
| 14:27:33 | sean-k-mooney | so you need shell for that to work | |
| 14:29:24 | opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | |
| 15:06:05 | opendevreview | Florian proposed openstack/nova master: Add check for PCIe devices attach limit for volume and ports https://review.opendev.org/c/openstack/nova/+/955584 | |
| 15:11:04 | gibi | sean-k-mooney: dansmith: 20 fake fits in one node https://zuul.opendev.org/t/openstack/build/d865972fcf374ebb813496cec36be473/logs memory low point is around 700MB | |
| 15:21:24 | sean-k-mooney | gibi: that is for a contoller right dedicated comptue could run more. mind if i create a seperete patch just to play with that aspect and a custom nodeset | |
| 15:22:25 | sean-k-mooney | i dont want to modify your but i ws thinkign of puting one on top and tweakign the nodeset def and subnodes to see if we coudl get to the 100-200 fake node range | |
| 16:07:09 | sean-k-mooney | @gibi with the increase comptues and iteration and concurrance | |
| 16:07:21 | sean-k-mooney | the tests look like tehy took about the same amoutn of tiem | |
| 16:07:46 | sean-k-mooney | well no | |
| 16:08:02 | sean-k-mooney | it went form around 207 to 270 seconds | |
| 16:11:21 | sean-k-mooney | the median time is shockingly consitent across all 4 runs | |
| 16:13:24 | sean-k-mooney | the max time doubles as we doubled the number of instances and the number of hosts. | |
| 16:13:51 | sean-k-mooney | that i do find a littel interstign as i was expectign that to cancle out espcially since you doubled the concurrency as well | |
| 16:14:59 | sean-k-mooney | gibi: you could consider upping the max confurrent build per host form the default 10 to see it that reduces that but its seams to be linear | |
| 16:15:32 | sean-k-mooney | so if you cahnge that form 10 to say 100 i wonder if that woudl affect anything | |
| 16:26:06 | zigo | Got this building Nova Flamingo: https://paste.opendev.org/show/bpVRJbZbVFoY5mieellm/ | |
| 16:26:10 | zigo | Any idea what's going on? | |
| 16:27:28 | zigo | Oh, should be my patch... :P | |
| 16:28:27 | sean-k-mooney | we have been removing the disperate imports of eventlet from the difent moduels and centralising them | |
| 16:28:50 | zigo | It's my fault, no worries. | |
| 16:29:50 | zigo | I had a patch adding greenthread.sleep(0), from past fix in Epoxy, removed it, should be ok this time. | |
| 16:33:46 | sean-k-mooney | gibi: https://gist.github.com/SeanMooney/27bb304653173a393c72d4cf4eca98e2 i got gemini to so some basic analasy fo the raw memory tracker data | |
| 16:37:32 | sean-k-mooney | i actuly need to check if i gave it the right data | |
| 16:39:00 | opendevreview | Tobias Urdin proposed openstack/nova master: Prevent leaking host info when HostMappingNotFound https://review.opendev.org/c/openstack/nova/+/959296 | |
| 16:39:07 | opendevreview | Tobias Urdin proposed openstack/nova master: Prevent leaking host info when HostMappingNotFound https://review.opendev.org/c/openstack/nova/+/959296 | |
| 16:39:14 | sean-k-mooney | the memory tracker data seasm to end at Sep 12 14:44:52 | |
| 16:39:26 | sean-k-mooney | were as the second rally run starts at 2025-09-12T14:48:54 | |
| 16:39:49 | sean-k-mooney | so i dont think we actully have good data for that | |
| 18:17:34 | opendevreview | Merged openstack/nova stable/2025.1: Fix 'nova-manage image_property set' command https://review.opendev.org/c/openstack/nova/+/958834 | |
| 21:14:57 | sean-k-mooney | gibi: i gave up on gmini and got claude to write a script to process the atop data by converting it to json an then procesing that | |
| 21:15:06 | sean-k-mooney | this is the inial output https://paste.opendev.org/show/brIoumAiGGb9GHx2eXzv/ | |
| 21:15:58 | sean-k-mooney | im still not sure tha tis correct either but ill push it somewhere on monday after i have checked if it sane | |
| #openstack-nova - 2025-09-13 | |||
| 20:55:21 | opendevreview | Thomas Goirand proposed openstack/nova-specs master: Restrict booting from volume with disk https://review.opendev.org/c/openstack/nova-specs/+/960513 | |
| #openstack-nova - 2025-09-14 | |||
| 10:41:31 | ivveh | hello folks, i have read https://docs.openstack.org/nova/latest/admin/manage-shares.html and it says that: "Due to current virtiofs implementation in Qemu / libvirt, the following Nova features are blocked for VMs with shares attached" and I have also seen post where this limitation blaim is put on KVM/QEMU (that it can't attach a filesystem to a running VM. Assuming the vm has memfd&shared mem enabled, this is not true and yo | |
| 10:42:05 | ivveh | just looking for clarification, if everyone is misinterpreting this statement or what is going on? | |
| 10:43:25 | ivveh | because there is nothing hindering from KVM/QEMU side to execute "virsh attach-device <instance> some.xml --live --config" which contains xml config for a filesystem with a socket coming from virtiofsd | |
| 10:43:48 | ivveh | (while the instance is running).. you can also detach-device as well | |
| 10:44:45 | ivveh | and since this was implemented not so long ago and this has worked for quite some time in qemu, im just wondering.. is it pointing perhaps to a very old version of QEMU/KVM? | |
| #openstack-nova - 2025-09-15 | |||
| 07:37:35 | opendevreview | Julien LE JEUNE proposed openstack/nova master: Adds regression test for bug LP#2044235 https://review.opendev.org/c/openstack/nova/+/960349 | |
| 08:21:59 | opendevreview | Julien LE JEUNE proposed openstack/nova master: Adds regression test for bug LP#2044235 https://review.opendev.org/c/openstack/nova/+/960349 | |
| 09:01:27 | opendevreview | Julien LE JEUNE proposed openstack/nova master: nova-conductor puts instance in error state https://review.opendev.org/c/openstack/nova/+/901655 | |
| 09:47:46 | opendevreview | Sylvain Bauza proposed openstack/nova master: Support multiple allocations for vGPUs https://review.opendev.org/c/openstack/nova/+/845757 | |
| 10:40:16 | sean-k-mooney | ivveh so the current state is as follows, when we were adding the feature to nova orginally the qemu feature to supprot attach was not complete so it was put out of scope of the nova feature wit the intent to support atach in the future. the nova support for this took longer then intended and since then libvirt and qemu have released supprot for it but nova has not been | |
| 10:40:18 | sean-k-mooney | updated to supprot it | |
| 10:41:26 | sean-k-mooney | ivveh so the tl;dr is it used ot be a qemu limiation and not its a nova one as we have not updated to use the new qemu fatures. for partcial reasons however the limitation si still in effefct for openstack but its something we would like to eventually adress. | |
| 10:49:35 | opendevreview | Rajesh Tailor proposed openstack/nova master: Fix string format specifier https://review.opendev.org/c/openstack/nova/+/961211 | |
| 11:31:24 | ratailor | sean-k-mooney, gibi bauzas could you please review clean backports, one more +2 is required. https://review.opendev.org/q/project:openstack/nova+branch:stable/2024.2+owner:ratailor@redhat.com | |
| 11:48:07 | sean-k-mooney | gibi's on pto until wednesday i belvie but i can take a look at the latter two now. it looks like elod and gibi have already approved the first 2 | |
| 11:48:24 | sean-k-mooney | oh elod already didi it | |
| 11:49:57 | sean-k-mooney | since elod has appvoed the 2024.2 version ill quickly pre review the 2024.1 backports | |
| 11:51:06 | jlejeune | hello sean-k-mooney, did you have time to check my regression test: https://review.opendev.org/c/openstack/nova/+/960349 ? | |
| 11:52:53 | sean-k-mooney | i saw that you change to a plain Excption you could also have used a TestEXcption but my main concern with that is you likely can just catch every excption in the main code. | |
| 11:53:55 | sean-k-mooney | this might be fine https://review.opendev.org/c/openstack/nova/+/901655/8/nova/compute/manager.py#9095 because its in pre live migraiton | |
| 11:54:29 | sean-k-mooney | but the futher we get into the migraiton tbe more clean up that is required and the more careful and specific we need to be with what we catch | |
| 11:55:33 | sean-k-mooney | you are at least converting the generic error to a specific on in that expcption handeler which is good | |
| 11:56:25 | jlejeune | yes it's because it happens in pre live migration check that I allowed myself to do it | |
| 11:57:28 | sean-k-mooney | can you rebase your fix patch on the latest revsion of the repoducer so we can see them working togehter | |
| 11:58:41 | jlejeune | sure, doing | |
| 12:00:25 | sean-k-mooney | your reusign the exsitng exception.MigrationPreCheckError that we have for the late tenatn anti affinity check which imples this exception case shoudl already be handeld and it shoudl rollback any work done so i think that is a good chioce and it is also another precheck failure so its semanticly valid | |
| 12:00:50 | opendevreview | Julien LE JEUNE proposed openstack/nova master: nova-conductor puts instance in error state https://review.opendev.org/c/openstack/nova/+/901655 | |
| 12:02:33 | sean-k-mooney | jlejeune: by the way, whle we prefer functional repoduces if wee cant simulate the exact condtions in the fucntional suite we also do use unit test repodcues form time to time. | |
| 12:03:39 | jlejeune | hm ok good to know | |
| 12:04:21 | sean-k-mooney | so if other want to see more explicte testign with a timeout excption, we could add one that directly calls check_can_live_migrate_destination and mke it mock check_can_live_migrate_source to raise the timeout excption you orgianlly hit | |
| 12:04:46 | sean-k-mooney | then you can show yoru fix convet it to a MigrationPreCheckError | |
| 12:05:12 | sean-k-mooney | the functional test you already have will continue to assert the overall instnace state stays active | |
| 12:05:20 | jlejeune | it was what I have done at first on the unit test :) | |
| 12:05:23 | sean-k-mooney | and the unit test will assert that timeouts are properly handeled | |
| 12:05:36 | sean-k-mooney | yep | |
| 12:05:54 | jlejeune | I can replace the test.Exception by the olso message timeout if you prefer | |
| 12:06:00 | jlejeune | in the unit tets | |
| 12:06:41 | sean-k-mooney | im not sure that is requird. | |
| 12:06:55 | sean-k-mooney | im more saying f other think we should have a more explcit repoducer | |
| 12:07:18 | sean-k-mooney | i woudl proably do that via pulling in part fo the unit test you wote | |
| 12:07:51 | sean-k-mooney | but instead assert that the excption is not handled properly locally. | |
| 12:08:19 | ratailor | sean-k-mooney, do we need to backport this once it merges on master. https://review.opendev.org/c/openstack/nova/+/961211 | |
| 12:08:27 | sean-k-mooney | i.e. that it is not properly converted to the MigrationPreCheckError error by check_can_live_migrate_destination | |
| 12:09:06 | sean-k-mooney | it depend on what the side effect is. | |
| 12:09:14 | sean-k-mooney | we proably should backprot it because its trivial | |
| 12:09:25 | sean-k-mooney | but i dont see a bug for it | |