Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-03-13
15:33:19 gibi and boot a VM requesting 1 VF, then delete the VM, and boot again the second VM will get 2 VFs instead of one
15:33:34 dansmith oh right right sorry
15:34:07 gibi the way the pci stats device pooling interacts with the placement driven PCI allocation leads some broken assumptions
15:34:43 gibi causing that the two VFs are broken up into two pools but both refer to the same placement RP
15:35:10 gibi while the other side of the code assumes that devices from a same RP are always in the same pool
15:35:54 dansmith I totally remember the conversation now, yep
15:36:07 gibi now both side of the code is improved. i) we avoid breaking up the pool ii) even if the pool is broken up in the future the other side of the code will be more graceful
15:36:27 gibi and not just allocate the same amount from each pool
15:45:04 dansmith okay I skimmed the repro thing.. I need to go learn about the pooling as I have a big gap there.. I'm also surprised this affects placement, but I guess because of some healing
15:45:42 dansmith but the "doesn't end up like we started" part of that makes sense of course
15:46:37 sean-k-mooney the pooling of the pci devices?
15:46:41 gibi sean-k-mooney: yepp
15:47:18 gibi dansmith: exactly I also think it affects placement due to healing, I will assert that in the repro..
15:47:33 sean-k-mooney nova pci tracker kid of wors like placment but before placement was a thing. we summerised the "invetories" of devices into pools and that what we expose to the schduler
15:47:49 sean-k-mooney rather then all the devices on each host
15:47:55 dansmith I assume the pools are because of numa nodes or something?
15:48:08 sean-k-mooney not exactly
15:48:25 sean-k-mooney they are partly related ot that
15:48:39 sean-k-mooney btu they can be splcit on other thign liek neutron phsynet
15:49:06 gibi or rp_uui :)
15:49:11 gibi rp_uuid
15:49:28 sean-k-mooney ya that was grafted on after the fact for pci in placement
15:49:46 gibi dansmith: (confirmed locally that indeed the healing part breaks the placement view)
15:50:23 sean-k-mooney before a pool of VFs (think inventory of vfs) was group by the vendor id and product id, and i think parent and or phsynet?
15:51:04 dansmith but.. is there a pool object somewhere? seems like maybe they're just generated on the fly when we go to process a request?
15:51:12 sean-k-mooney yes
15:51:17 sean-k-mooney its stored in the comptue node table
15:51:31 sean-k-mooney that where the schdluer gets them form to add to the host state object
15:52:24 dansmith oh yeah I see
15:52:50 dansmith but this pci/stats thing (which seems not very related to stats) defines Pool as a dict
15:53:03 sean-k-mooney no its an ovo
15:53:07 sean-k-mooney https://github.com/openstack/nova/blob/master/nova/objects/pci_device_pool.py
15:53:12 dansmith I see that
15:53:27 sean-k-mooney well its stored as a json blob if that what your asking
15:53:28 dansmith https://review.opendev.org/c/openstack/nova/+/944277/1/nova/pci/stats.py line 39
15:53:41 dansmith ^ is what I'm talking about
15:54:41 sean-k-mooney we convert it here https://github.com/openstack/nova/blob/master/nova/objects/compute_node.py#L321-L326
15:55:04 sean-k-mooney which is called in save on the compute node here https://github.com/openstack/nova/blob/master/nova/objects/compute_node.py#L360
15:55:22 gibi stats pool concept works with dict yes, the DB model works with jsonified ovos
15:55:52 dansmith and the PciDeviceStats seems to call a PciDevicePoolList "stats"
15:56:14 sean-k-mooney ya
15:56:15 dansmith which it then converts to a dict which it calls "pools"
15:56:20 dansmith wtf?
15:56:34 sean-k-mooney the nameing is shall we say in consitent
15:56:59 dansmith I was going to pick a different adjective.. possibly an emoji
15:57:22 sean-k-mooney the db colume is called pci_stats
15:57:24 sean-k-mooney https://github.com/openstack/nova/blob/master/nova/db/main/models.py#L227-L229
15:57:44 dansmith smh
15:57:52 sean-k-mooney the stats name came for the fact its a summary
15:58:00 sean-k-mooney i.e. a count and some metadata
15:58:05 dansmith so PciDevicePool has a product and vendor but not an address,
15:58:06 sean-k-mooney rather then each device
15:58:13 sean-k-mooney correct
15:58:29 dansmith so I assume we take the alias map and generate all the PciDevicePool objects we need with the appropriate count, even if count=1 because they just specified one address?
15:58:33 sean-k-mooney its a count of how many of that type of device (product/vendor) are avaible
15:58:50 sean-k-mooney yes
15:59:08 sean-k-mooney if the pci dev spec only allows 1 device you get a pool or 1
15:59:12 sean-k-mooney *of 1
15:59:16 dansmith this code is, I think, a very nice concrete example of why I normally just gloss over anytime PCI stuff comes up and try to ignore it :)
15:59:28 dansmith ack
15:59:53 sean-k-mooney i actully like this code partly becasue to me it very placement like
16:00:03 sean-k-mooney well conceptually
16:00:03 dansmith does the grouping per physnet, per rp, etc, etc happen in compute node so we report the proper splitting to scheduler?
16:00:18 sean-k-mooney dansmith: yes it does
16:00:41 dansmith sean-k-mooney: I mean the name confusion and the converting everything to mis-named dicts to work on :)
16:00:42 sean-k-mooney its part of the resouce summary view we create as a perodic
16:00:48 dansmith okay makes sense
16:01:17 sean-k-mooney dansmith: some of this code i think predate when you added nova objects
16:01:31 sean-k-mooney so ya the conventiosn were not as well established
16:02:22 dansmith sean-k-mooney: I don't think it predates objects, but I know it's old
16:02:51 sean-k-mooney perhasp it was 2012 ish
16:03:33 sean-k-mooney not imporant now i ugess
16:03:43 gibi another "complication" that pool filtering happens twice by this code, once from the PCIPassthroughFilter and once from the pci claim on the compute. But it happens a bit differently as the former only handles counts, while the latter handles real device addresses to allocat
16:04:05 sean-k-mooney it actully happens 3 times
16:04:56 gibi true, twice in the scheduler
16:05:23 sean-k-mooney it happens in both the pci passthoguh filter and again in the numa topology filter
16:06:19 sean-k-mooney the pci filter really just checks that there are enough devices of the correct type
16:06:30 sean-k-mooney the numa filter need to also take into acocunt the numa constraits
17:30:12 opendevreview Dan Smith proposed openstack/nova master: Support "one-time-use" PCI devices https://review.opendev.org/c/openstack/nova/+/943816
17:30:13 opendevreview Dan Smith proposed openstack/nova master: Add one-time-use devices docs and reno https://review.opendev.org/c/openstack/nova/+/944262
17:38:24 opendevreview Balazs Gibizer proposed openstack/nova master: Reproduce bug/2098496 https://review.opendev.org/c/openstack/nova/+/941673
17:38:25 opendevreview Balazs Gibizer proposed openstack/nova master: Ignore metadata tags in pci/stats _find_pool logic https://review.opendev.org/c/openstack/nova/+/944277
19:59:39 atmark hello, does ceph driver for nova support rbd namespaces?
#openstack-nova - 2025-03-14
07:37:07 gokhan . I want to be sure if this is expected behaviour with ceph.
07:37:07 gokhan hello folks, we are using nova with ceph backend. Sometimes we are getting "nova-compute: Got no allocation candidates from the Placement API. This could be due to insufficient resources or a temporary occurrence as compute nodes start up" warnings. when I look at the resource provider inventory list, hosts DISK_GB value is not correct. Only one host shows corresct value as ceph pool capacity. https://paste.openstack.org/show/b9g8Js7V63bdf7rKuLqg/
07:58:37 frickler gokhan: that does sound like a bug to me, what versions of nova and ceph? probably best create a bug report
08:02:33 gokhan frickler, nova version is 29.2.0 and ceph is 18.2.2. But in my other deployments which has same version, DISK_GB on every hosts is correct.
08:06:41 gokhan frickler, ohh sorry ceph versions sre different. İt is working with ceph 18.2.4 but with ceph 18.2.2 it is not working. may be there is a known issue
10:50:20 ishanwar Hi Everyone!
10:50:38 ishanwar I am planning to take a shot at this LHF https://bugs.launchpad.net/nova/+bug/2027156
10:51:15 ishanwar Can someone help me understand why we invented to use to the neutron quota API here ? Please dont mind me question I am new to nova dev
10:51:32 ishanwar intend
10:52:02 ishanwar @Sean-k-mooney @bauzas
10:59:47 sean-k-mooney ishanwar: am short version is today we are pulling the full port object with all the extentions and then counting them in python
11:00:08 sean-k-mooney to see if we exceed the quota the project has for total ports
11:00:22 sean-k-mooney but neutron has an api that just returns this as a number
11:00:45 sean-k-mooney so its much faster to use the dedciated api and less work for both nova and neutron if we do
11:01:14 sean-k-mooney so this is just a performance optimisation ot reduce the amount of data sent over the wire and reduce the database load on the neutron side
11:02:33 sean-k-mooney ishanwar: so the idea is, 1 try and use the neutron api exteion for quota details, if that works great we can skip the extra work of counting the ports ourselves, if not fall back to what we do today
11:06:09 ishanwar Hi Sean

Earlier   Later