Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2026-05-15
13:27:56 gibi which only exists with VMs that are created (or rebooted) since the IOThread feature
13:37:08 sean-k-mooney im surpised our multinode grenade job didnt pick that up
13:37:16 sean-k-mooney but ok that shoudl not be hard to fixx
13:37:29 sean-k-mooney we just need ot gate that code by the xml having it
13:43:03 gibi I guess our grenade does not have pinned VMs
13:43:19 sean-k-mooney oh is this only for pinned vms
13:43:29 sean-k-mooney oh no
13:43:35 sean-k-mooney its not pined exaclytu
13:43:46 sean-k-mooney its that we are not using cpu_share_set or cpu_dedicated_set
13:43:59 sean-k-mooney so since we are not using either of them we never generate teh element
13:44:06 gibi nope
13:44:13 sean-k-mooney cpu_shared_set woudl be enough to catch this i think
13:44:29 sean-k-mooney gibi: are you plannign to work on a patch to make it conditional?
13:44:44 gibi I'm working on a functional reproducer
13:45:07 sean-k-mooney ack cool im happy to review both so add me or ping me when its up
13:45:22 gibi during live migration we update the src XML, for a pre-existing VM, the src XML does not have any iothreads or iothreadpin field
13:45:52 sean-k-mooney yep we have had this tyep of issue before
13:46:09 sean-k-mooney i think we have exsing functional tests that simulate somethign similar
13:51:51 opendevreview Balazs Gibizer proposed openstack/nova master: Reproduce bug/2152697 https://review.opendev.org/c/openstack/nova/+/988769
13:52:01 gibi sean-k-mooney: this is the reproducer ^^
13:53:48 sean-k-mooney am `del list(conn._vms.values())[0]._def["iothreads"]`
13:53:58 sean-k-mooney ok i see why that works
13:54:43 sean-k-mooney the list is convertign the the iterator int a lis that containes refence to the actul objects in the _vms dict
13:54:55 sean-k-mooney so we are deleting trhough the list
13:55:14 sean-k-mooney coudl we do this a littel iffenrtly
13:55:44 sean-k-mooney vm_domain = next(conn._vms.values())
13:55:54 gibi we can use next yes
13:56:06 sean-k-mooney del vm_domain._def["..."]
13:56:38 sean-k-mooney i can see why what you have works on a second or third reading but its a littel non obvious
13:57:09 sean-k-mooney gibi: over all the repoducer is more or less doign what i expect
13:57:27 sean-k-mooney as noted however this shoudl also manifest if you have jsut set cpu_shared_set and are not using dedicated
13:57:47 sean-k-mooney so it migh be nice to add a second test for floating vms
13:58:30 sean-k-mooney otherwise this looks like a reasonabel repoducer. it was more or less what i was expecting you to write
13:59:45 gibi I tried an instance with shared CPUs before and it did not hit the same issue
13:59:53 gibi the problematic code is under a condition
13:59:54 gibi https://github.com/openstack/nova/commit/53a613d9948826ec9a4cd4a502f7a5d1b2dc87d7#diff-1f1f3f935853a67b1239220cd9f8c28278734732c5c28e8d238f62860391b1ecR260-R262
13:59:57 gibi about pinning
14:00:10 sean-k-mooney we do pin shared cpus
14:00:21 sean-k-mooney if an only if cpu_share_set is defiend
14:01:40 sean-k-mooney oh
14:01:56 sean-k-mooney actully in that case we only set vcpu cpuset=...
14:02:07 sean-k-mooney so we pin it but we dont generate the element
14:02:22 sean-k-mooney ok then ya i see this would only impacte pinned vms that were created before iothread
14:05:10 opendevreview Balazs Gibizer proposed openstack/nova master: Fix live migration with pinned VM and iothreads https://review.opendev.org/c/openstack/nova/+/988774
14:08:21 sean-k-mooney gibi: +2 on the repoducer +1 on the fix. ill see if i can try and trest this on monday and loop back to the review after the ci has ran
14:08:40 sean-k-mooney gibi: we might want to add a release note but that my only feedback really right now
14:08:52 sean-k-mooney maybe also a unit test for the branchign bevhior in the migration module
14:09:20 sean-k-mooney just to test _update_numa_xml
14:09:27 gibi yeah I can do that
14:17:25 opendevreview Balazs Gibizer proposed openstack/nova master: Fix live migration with pinned VM and iothreads https://review.opendev.org/c/openstack/nova/+/988774
14:17:25 opendevreview Balazs Gibizer proposed openstack/nova master: Reproduce bug/2152697 https://review.opendev.org/c/openstack/nova/+/988769
14:20:43 opendevreview Balazs Gibizer proposed openstack/nova master: Fix live migration with pinned VM and iothreads https://review.opendev.org/c/openstack/nova/+/988774
14:20:49 gibi now with reno and unit test :)
14:21:15 gibi ^^ cc dansmith as we chatted about it downstream
14:24:30 sean-k-mooney gibi: so one nuance about your fix
14:24:49 sean-k-mooney it will work but you intentally choose not to add an iothread to the running instnace
14:24:58 sean-k-mooney on live migrate
14:25:30 sean-k-mooney we could also do that perhasp in a followup patch
14:26:34 sean-k-mooney im personally fine with not doing that and just saying you need to reboot to get the iothread jsut notign that qemu/libvirt techniall supprot adding and removign them on a runing instance
14:27:36 gibi I think it is safer not to try to add it during live migration
14:27:47 sean-k-mooney ya that was my feelign too
14:27:52 sean-k-mooney just wanted to raise the possiblity
14:28:01 gibi so if somebody explicity want this then that is a new small feature to me
14:29:56 sean-k-mooney cool well still +1 lets see what ci says and ill loop back
14:31:11 sean-k-mooney gibi: nice find by the way idd you just come across it or was it somethign you noticed in a ci failreu or something
14:31:26 sean-k-mooney oh you were backporting this downstream right
14:31:38 gibi yepp I tested my backport downstream
14:31:53 gibi there it is simpler to thest the upgrade
14:32:16 gibi by just changing the nova-compute container image
14:32:42 gibi as we had bugs before with live migration and pinned CPUs, I tested the combination of those :)
14:32:56 sean-k-mooney ack i need to revive and rewrite my ci change
14:33:01 opendevreview Masanori Ueno proposed openstack/nova master: DNM: Add functional test for NUMA live migration overcommit bug https://review.opendev.org/c/openstack/nova/+/988777
14:33:03 sean-k-mooney that enabeld cpu pining upstream
14:33:20 gibi yeah I have a todo to add some iothread tests to whitebox
14:33:27 sean-k-mooney i never got aroudn to making the calulation of cpu_share_set and cpu_dedicated_set dynmic
14:33:36 sean-k-mooney whitebox woudl be good too ya
16:38:02 dansmith gibi_off: sean-k-mooney: but... people might want to use live migration to get iothreads without a guest restart
16:38:12 dansmith I mean, I know the risk is higher but.. I _know_ people will want that :D
17:10:32 sean-k-mooney dansmith: thats why i broguht it up. it would be nice if that worked but that is why i was suggesting perhaps a followup
17:10:49 sean-k-mooney i knwo there is an api to add an remove them at runtime
17:10:57 sean-k-mooney but i dont knwo if you can do this when live migrating
17:11:01 dansmith ack
19:37:04 melwitt gmaan: sorry to bug you with this but I'm trying to close out a quota bug fix I worked on awhile back that has got one +2 https://review.opendev.org/q/topic:%22bug/2131272%22 the CI results are old but I just rechecked it, if you might have a chance to look at it. it is not urgent
19:45:53 gmaan melwitt: ack, I will take a look
19:46:50 melwitt thanks gmaan
20:26:32 gmaan melwitt: one question. do we allow to set user quota >project quota OR set project quota < user quota ? if yes then that is the problem right?
20:30:48 gmaan so this is case when both are equal and multi users. got it but just wondering if we handle the above case ^^ (i assume yes)
20:49:42 melwitt gmaan: I think it is not allowed to set an individual user's quota to be larger than the quota of their project. And I don't think it's allowed to set project quota lower than an individual user's quota in the project but I am not as sure about that one. /me checks if we have tests for this and if not, it's probably worth adding a couple
20:53:24 gmaan yeah, I was thinking to have test for those cases. but anyways that is separate things came up in my mind. bug#2131272 fix lgtm, waiting for CI results
20:57:26 melwitt gmaan: yeah I think it's a good point. I might add the tests to this patch if I find we don't already have them, since we are waiting anyway
20:58:26 melwitt I would think they should be easy small func tests that just try to set the quota and assert it's rejected in those scenarios
21:23:03 opendevreview melanie witt proposed openstack/nova master: Fix swap disk creation skipped on NFS during cold migration https://review.opendev.org/c/openstack/nova/+/988547
21:24:27 gmaan ++ thanks
21:27:03 opendevreview melanie witt proposed openstack/nova master: Use tempest_concurrency=1 for nova-vtpm job https://review.opendev.org/c/openstack/nova/+/984864
21:42:29 opendevreview Merged openstack/nova master: Reproducer for bug 2131272 https://review.opendev.org/c/openstack/nova/+/967147
21:49:31 melwitt I have a crazy old 4 character bug fix with one +2 on it if anyone wants an easy review https://review.opendev.org/c/openstack/nova/+/904155
22:25:40 opendevreview melanie witt proposed openstack/nova master: Fix usage count when user-scoped quota is set https://review.opendev.org/c/openstack/nova/+/967148
22:31:34 gmaan +w
23:07:27 melwitt thanks gmaan !
23:10:06 opendevreview Merged openstack/nova master: Implementing get_num_instances for ironic virt driver https://review.opendev.org/c/openstack/nova/+/955685

Earlier   Later