Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-06-06
13:04:11 sean-k-mooney gibi: i assume we cant pass an arge to do the join egerly
13:05:09 elodilles hi folks, i know thay you are busy with other things, but there is a small "tool" change i'd appreciate you would review and merge if possible (yeah, it's a small fix for backport validator, related to unmaintained branches :/): https://review.opendev.org/c/openstack/nova/+/949628
13:05:17 elodilles s/thay/that/
13:06:17 gibi elodilles: +2
13:06:19 sean-k-mooney looks fine to me i guess
13:06:43 sean-k-mooney it is one of the first times we have had to skip eols
13:06:54 gibi sean-k-mooney: we might, but the server/view code is entered from different places so we should only conditionally do the join if we know it will be used
13:07:05 stephenfin gibi: sean-k-mooney: thanks for the help https://github.com/stephenfin/nova/commit/2630daf37c8adf4b443c80e66139ea6285c04eae
13:07:24 stephenfin will push all those once the bulk of what's already up on gerrit has merged
13:08:17 elodilles thanks gibi sean-k-mooney o/
13:35:42 sean-k-mooney gibi: im on a call but we have someting like "expected_filds" or something like that to contole which filed are lazy loaded on the instance object
13:35:57 sean-k-mooney i dont knwo if we can use that in soem way
13:37:37 sean-k-mooney i was thinkig of https://github.com/openstack/nova/blob/master/nova/objects/instance.py#L77C5-L101
13:46:49 opendevreview Balazs Gibizer proposed openstack/nova master: Run nova-api and -metadata in threaded mode https://review.opendev.org/c/openstack/nova/+/951957
13:48:33 gibi sean-k-mooney: yeah we can ask for the join, the problem is that there are multiple ways reaching that calls and some will use the bdms later some won't. Also the way how this is built up make is pretty hard to untangle them. It is not impossible to refactor this but I decided not to try it now
13:48:59 gibi btw if https://review.opendev.org/c/openstack/nova/+/951957 works then that is a nice surprise and an easy win
13:49:32 gibi locally it works for me but I need the full tempest run from nova-next to believe in it
14:00:50 sean-k-mooney gibi: well i had workign versions last cycel so it really wont take much
14:01:36 sean-k-mooney gibi: as in the api was the one of the first thigns i thied to move so with the infra you have added im expict that to not be supper complx to get working
14:02:20 sean-k-mooney gibi: and yes scater gatther is the only use of eventlet spwan in them
14:02:28 sean-k-mooney that why i suggested starting with the api
14:03:11 sean-k-mooney gibi: i know i need to go back adn review the most recent version but i think schduler api and metadta are doabel by m2
14:05:43 gibi yeah now that I see api working locally I believe that it is doable for m2
14:06:47 sean-k-mooney it might be worth askign clark to look at https://review.opendev.org/c/openstack/devstack/+/948436 again
14:07:17 sean-k-mooney although it will be a few days in any case before we get that far in the series
14:09:25 gibi good point. I pinged clark in the review now and I can ping him on IRC next week
15:58:09 opendevreview Merged openstack/nova master: [tool] Fix backport validator for non-SLURP https://review.opendev.org/c/openstack/nova/+/949628
16:03:19 opendevreview Elod Illes proposed openstack/nova stable/2025.1: [tool] Fix backport validator for non-SLURP https://review.opendev.org/c/openstack/nova/+/951968
16:04:36 opendevreview Elod Illes proposed openstack/nova stable/2024.2: [tool] Fix backport validator for non-SLURP https://review.opendev.org/c/openstack/nova/+/951969
16:05:14 opendevreview Elod Illes proposed openstack/nova stable/2024.1: [tool] Fix backport validator for non-SLURP https://review.opendev.org/c/openstack/nova/+/951970
16:29:22 noonedeadpunk I'm trying here to setup hugepages for one of my computes, but instance gets filtered out at NUMATopologyFilter with no good candidates https://github.com/openstack/nova/blob/master/nova/scheduler/filters/numa_topology_filter.py#L115-L121
16:31:31 noonedeadpunk compute object seems to be having mempages https://paste.openstack.org/show/bRHPMmumA43t0KHk186K/
16:33:02 noonedeadpunk I wonder if that's might be also due to PCI Passthrough attempt, as that's I guess what might not work with numa placement nicely?
16:34:55 noonedeadpunk flavor defined as `hw:cpu_policy='dedicated', hw:mem_page_size='2MB', hw:vif_multiqueue_enabled='true', trait:CUSTOM_GPU='required', pci_passthrough:alias='A10_FULL:1'`
16:35:17 noonedeadpunk and obviously once I unset hw:mem_page_size - scheduling passing nicely
16:35:58 opendevreview sean mooney proposed openstack/nova master: [DNM] testing py3.13 eventlet bug workaround https://review.opendev.org/c/openstack/nova/+/951749
16:57:47 sean-k-mooney you can have both enabled
16:58:20 sean-k-mooney however if you have a numa vm by default we require the pci device request by the guest to come form the same numa node as the guest cpus/memory
16:58:30 sean-k-mooney noonedeadpunk: you can diabel that in your pci config
17:00:37 sean-k-mooney noonedeadpunk: in your pci alias you can set https://docs.openstack.org/nova/latest/configuration/config.html#pci.alias numa_policy=preferred
17:00:52 sean-k-mooney noonedeadpunk: you can also set that in the flavor which is generally simpler
17:02:04 sean-k-mooney i would recomend setting hw:pci_numa_affinity_policy=preferred or hw:pci_numa_affinity_policy=socket
17:02:25 sean-k-mooney preferred is the most relaxed policy
17:02:54 sean-k-mooney i woudl also suggest using hw:mem_page_size=large
17:05:36 noonedeadpunk large is 1G?
17:06:11 noonedeadpunk let me try setting policies then...
17:06:52 noonedeadpunk as I tried couple of options, like hw:numa_nodes=1
17:12:37 noonedeadpunk nah, didn't fly somehow... I guess best thing to do for me now - go for a weekend and check on that with fresh eyes after it :)
17:12:43 gmaan about 2.0 removal. I am not sure what is benefits if we keep v2.1 as min version. 2.0 is just a endpoint without any change from v2.1 so no overhead of maintenance. This will just break user using 2.0 endpoint without giving any gain
17:21:33 sean-k-mooney noonedeadpunk: large is anything that isnt small i.e. anything other then 4k default pagesize
17:22:10 noonedeadpunk aha, ok
17:22:27 sean-k-mooney noonedeadpunk: if you have not turned on debug yet do that on monday in the schduler
17:22:39 sean-k-mooney you will need that to discover why
17:22:45 noonedeadpunk I did that on scheduler
17:22:51 sean-k-mooney ack
17:23:04 noonedeadpunk that's where I grabbed cpu capabilities
17:23:10 noonedeadpunk *compute
17:23:37 sean-k-mooney so your flavor was askign for pinned cpus and hugepages
17:23:43 noonedeadpunk yeah
17:23:48 sean-k-mooney did you actully configure cpu_dedicated_set ?
17:23:49 noonedeadpunk and PCI device
17:24:10 sean-k-mooney in the comptue section of the nova.conf
17:24:14 sean-k-mooney on the compute node
17:24:27 noonedeadpunk nope, I did not, I just set reserved_host_cpus = 2 and that's it
17:24:41 noonedeadpunk but it has no VMs running, as is in the isolated aggregate
17:24:47 sean-k-mooney ok well that probaly the issue
17:24:55 sean-k-mooney what release are you testing with
17:25:01 noonedeadpunk Caracal
17:25:42 noonedeadpunk when I drop hugepages from properties - it does allocate pinned cpus though
17:26:06 sean-k-mooney do you have https://docs.openstack.org/nova/2024.1/configuration/config.html#workarounds.disable_fallback_pcpu_query defiend
17:26:10 noonedeadpunk or you saying that without hugepages it does not go through the numa filter...
17:26:30 noonedeadpunk nope
17:26:38 sean-k-mooney ok so its falling abck
17:27:16 sean-k-mooney so any of hugpeages, pinning or an explcit numa topolgoy will result in the schduelr usign the numa filter to validate the host
17:27:52 sean-k-mooney how much hugepages did you allocate on the host? and how much ram are you asking for in the vm
17:28:14 noonedeadpunk ok, here where the problem can actually be....
17:28:28 noonedeadpunk as I kind of took example from docs for now for some POC
17:28:39 noonedeadpunk so it's 1024*2MB
17:28:59 noonedeadpunk and I'm asking for *really* big instance of 200Gb RAM
17:29:55 noonedeadpunk So what I should be doing - is to split available ram into blocks and that num_pages * page_size == ram?
17:30:38 sean-k-mooney would you mind pasting the output of virsh cabalities
17:30:49 sean-k-mooney i can advise based on the size of the host with that
17:31:09 noonedeadpunk https://paste.openstack.org/show/bYi3OvxXJbP6Grx275O7/
17:31:22 noonedeadpunk but we plan to run there like... 4 VMs max?
17:32:44 sean-k-mooney ok so you have 2 numa nodes
17:33:21 noonedeadpunk yup
17:33:23 noonedeadpunk no HT
17:33:39 sean-k-mooney and you allcoate 2 GB of 2MB hugepages per numa node
17:34:00 sean-k-mooney <pages unit='KiB' size='2048'>1024</pages>
17:34:25 noonedeadpunk Right... And I guess that's where the problem is
17:34:25 sean-k-mooney noonedeadpunk: try creating a 1G vm that asks for 1 cpu with hw:mem_page_size=large
17:34:44 sean-k-mooney if that works all you need ot do is decied how you want to partion the host ram
17:34:53 sean-k-mooney i.e. how much to reserve exclusivly for hugepages
17:36:14 noonedeadpunk so just to double check where I was wrong. When I set `hw:mem_page_size='2MB'` in flavor - what needs to happen is that flavor ram / hw:mem_page_size = amount of available on compute hugepages?
17:36:26 sean-k-mooney looks like you have 512GB total which 500 is usable
17:36:44 noonedeadpunk yeah, sounds about right
17:36:59 noonedeadpunk so 256 on single numa node
17:37:47 sean-k-mooney ya i woudl start conservity ly and allocage 480GB of hugepages on that system total
17:37:48 noonedeadpunk but actually seeing virsh capabiltities I'm really understanding where my logic went off now
17:37:58 noonedeadpunk ++

Earlier   Later