| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-09-14 | |||
| 09:52:44 | zigo | My bad. | |
| 09:53:05 | sean-k-mooney | i have almost the same qustion for watcher :P although i had seen that mention at least this side of the pandemic | |
| 09:53:28 | zigo | To the contrary of freezer, there was some commits for this cycle. | |
| 09:54:30 | sean-k-mooney | there was like 8 commit in watcher most procedural | |
| 09:54:43 | sean-k-mooney | but i guess its stil offically supported | |
| 09:56:06 | sean-k-mooney | tobias-urdin: you were making some changes to watcher eiarler in the year. is that a project you have use of personaly | |
| 09:56:50 | sean-k-mooney | just wondering if patches were created how likely is it that they would be merged/reviewd | |
| 11:31:14 | opendevreview | Amit Uniyal proposed openstack/nova master: WIP:Adds device tagging functional tests https://review.opendev.org/c/openstack/nova/+/895162 | |
| 11:51:09 | tobias-urdin | sean-k-mooney: we do not use it, nothing i actively work on, i just did some drive-by fixes for oslo.messaging or fixing ci probably | |
| 11:57:50 | sean-k-mooney | tobias-urdin: ack ya that is what i was assumign i was just wondering if there were any active contibutors left | |
| 12:24:30 | bauzas | ralonsoh: while https://review.opendev.org/c/openstack/neutron/+/893447 is now merged (the neutron revert), we still have CI failures today https://zuul.opendev.org/t/openstack/build/01dd3befff5e42cfb0c0f667b8e7e30b | |
| 12:28:54 | bauzas | elodilles: I guess you know the current situation in Nova | |
| 12:29:06 | bauzas | we still need to await a few merges before RC1 | |
| 12:30:53 | bauzas | hmpfff https://zuul.opendev.org/t/openstack/builds?job_name=nova-live-migration&skip=0 | |
| 12:31:05 | bauzas | 30% of failures I'd say | |
| 12:44:09 | mgariepy | is it possible to force nova a device_type for pci devices ? | |
| 12:44:55 | sean-k-mooney | no | |
| 12:45:51 | mgariepy | i wiant to only do some pci passthrough on a couple of A100 | |
| 12:46:14 | sean-k-mooney | you can filter by the pci adress | |
| 12:46:39 | sean-k-mooney | with support for regxes or bash globs if you dont want to just use the adress | |
| 12:46:51 | mgariepy | https://github.com/openstack/nova/blob/stable/2023.1/nova/pci/stats.py#L682C1-L683C1 | |
| 12:47:52 | sean-k-mooney | yes if the alais is not device_type pf then we filter them out when considering candiates | |
| 12:48:10 | sean-k-mooney | that does not mean you cant reqeust PF for passthrough | |
| 12:48:35 | sean-k-mooney | you jsut have to set the device_type to type-PF in the pci ailais | |
| 12:49:22 | sean-k-mooney | so inestead of type-pci or type-vf https://github.com/openstack/nova/blob/stable/2023.1/nova/pci/request.py#L34 you woudl set type-PF | |
| 12:50:50 | sean-k-mooney | the alias and device list work toghtere to selec tthe device. the alias is how you describe the request and the device_list (pci whitelist) is how you declar what device may be allocated | |
| 12:51:36 | mgariepy | alias = { "name": "nvidia-a100-smx4-40gb", "product_id": "20b0", "vendor_id": "10de", 'device_type': 'type-PF'} | |
| 12:51:46 | mgariepy | restart the scheduler, and still get the same error. | |
| 12:52:25 | sean-k-mooney | you need to set this on all the compute too just an fyi hte have to match | |
| 12:52:40 | sean-k-mooney | but what is the error | |
| 12:53:10 | mgariepy | on it's also set on the gpu compute. | |
| 12:53:49 | sean-k-mooney | have you checked the pci devices tabel to ensure the a100s are correctly tracked | |
| 12:54:51 | sean-k-mooney | wether they show up as type-PF or type-pci will depend on if they report supprot for creating VF | |
| 12:55:05 | sean-k-mooney | that may depend on if you have enbale mig mode of not | |
| 12:55:27 | sean-k-mooney | for t4 it depeneded on the firmware version | |
| 12:55:48 | sean-k-mooney | i think a100 woudl alays report sriov supprot but its worth checking | |
| 12:56:12 | mgariepy | Placement PCI resource view: Placement PCI view on gpu-hpc21145: RP(gpu-hpc21145_0000:01:00.0, CUSTOM_PCI_10DE_20B0=1, traits=COMPUTE_MANAGED_PCI_DEVICE) | |
| 12:56:38 | sean-k-mooney | oh your using the pci in placemnt feature | |
| 12:57:01 | mgariepy | yep all i tried was also failing :) haha | |
| 12:57:08 | mgariepy | on 2023.1 | |
| 12:57:19 | sean-k-mooney | well this should be fine | |
| 12:57:32 | sean-k-mooney | so is it passign placment and failing in the pci filter? | |
| 12:57:40 | sean-k-mooney | or is it failing to get results form placemnt | |
| 12:59:23 | elodilles | bauzas: ACK. you can -1 the patch until it'll be updated: https://review.opendev.org/c/openstack/releases/+/894693 | |
| 12:59:52 | bauzas | oh missed it from my mails | |
| 13:00:01 | mgariepy | fails at running : _filter_pools_for_unrequested_pfs() | |
| 13:00:04 | bauzas | I was wondering why I wasn't seen it | |
| 13:00:44 | mgariepy | dev_type vs device_type ? | |
| 13:00:52 | mgariepy | or it's abstracted somewhere ? | |
| 13:01:18 | sean-k-mooney | its device_type in the alias | |
| 13:01:44 | ralonsoh | bauzas, yes, I'm going to mark this test as unstable again | |
| 13:01:50 | sean-k-mooney | https://github.com/openstack/nova/blob/stable/2023.1/nova/pci/request.py#L95C10-L104 that is the jsonschma fo rthe validation | |
| 13:02:00 | ralonsoh | we need to find a better way to set the subport as active after the live migration | |
| 13:02:01 | sean-k-mooney | but i think its dev type in the object | |
| 13:02:27 | mgariepy | let me logs the request and pool. to see if something seems odds. | |
| 13:02:38 | sean-k-mooney | https://github.com/openstack/nova/blob/stable/2023.1/nova/pci/request.py#L143C13-L145 | |
| 13:02:51 | bauzas | ralonsoh: ack | |
| 13:02:58 | sean-k-mooney | mgariepy: i would check the pci_devices table in teh cell db | |
| 13:03:13 | sean-k-mooney | and make sure the device has the expected type ectra | |
| 13:04:19 | mgariepy | nova.pci_devices tells me that the device is type-PF | |
| 13:05:02 | ralonsoh | bauzas, https://review.opendev.org/c/openstack/tempest/+/895167 | |
| 13:09:59 | sean-k-mooney | mgariepy: what does the pci whitelist look like for that host | |
| 13:11:09 | sean-k-mooney | based on the alias above it looks co unless you added physnet or remote managed or something im not sure why it would be remvoed | |
| 13:11:50 | sean-k-mooney | its the pci filter not the numa filter that is removing it right? | |
| 13:11:52 | mgariepy | https://paste.openstack.org/show/bij0SgRpSEVCTILyzIzf/ | |
| 13:12:10 | mgariepy | seems to be the pci filter | |
| 13:13:02 | sean-k-mooney | hum that all looks correct to me | |
| 13:14:15 | mgariepy | https://paste.openstack.org/show/bP2oY2IOe4lNvj6wGB8w/ | |
| 13:14:51 | mgariepy | what is resolving the alias ? is it dont by the scheduler directly ? | |
| 13:15:27 | sean-k-mooney | that depends on the request | |
| 13:15:47 | mgariepy | my flavor has alias stuff in in. | |
| 13:15:49 | sean-k-mooney | but no i think its doen in the compute or the api | |
| 13:16:53 | sean-k-mooney | https://docs.openstack.org/nova/latest/admin/pci-passthrough.html#configure-nova-api | |
| 13:17:07 | sean-k-mooney | so for ne vm request this is usd by the nova-api not the scheduler | |
| 13:17:30 | sean-k-mooney | and for rezize its used form teh compute | |
| 13:17:45 | sean-k-mooney | so if you are udataing this you need to restart the nova-api not the scheduler | |
| 13:18:26 | sean-k-mooney | the scheudle reciveds a fully popultated RequestSepc object which has the pci resuets embeeded in it | |
| 13:19:38 | mgariepy | but me resquest doesn't receive the device_type :/ | |
| 13:23:12 | sean-k-mooney | did you restart the nova-api processes after updating hte alias | |
| 13:24:02 | mgariepy | i did restart the nova slice. let me try juste putting the same config accross all the cluster to see if it changes something. | |
| 13:32:19 | mgariepy | what driver shoud be loaded fo the pci dev ? | |
| 13:32:56 | mgariepy | pci-stub or vfio-pci ? | |
| 13:32:58 | sean-k-mooney | the driver should not matter to the schduler | |
| 13:33:13 | sean-k-mooney | that woudl only matter on the compute node but yes either of those shoudl work | |
| 13:33:18 | mgariepy | ok | |
| 13:33:36 | sean-k-mooney | the important part is that the framebuffer is not initallised by xorg/wayland as that woudl prevent the passthough | |
| 13:33:53 | mgariepy | ha ok. | |
| 13:34:13 | sean-k-mooney | vfio-pci is what kvm/qemu will use when its being passthough through | |
| 13:34:26 | sean-k-mooney | so prebinding to that is generally not a bad idea but often not required | |
| 13:34:46 | mgariepy | i did some stuff with passthrough with el-cheapo gamer gpus. but with older openstack release. | |
| 13:35:50 | sean-k-mooney | thre is nothing really that has changed that would impact this | |
| 13:36:06 | sean-k-mooney | the alias and whitelist/device list syntax is the same | |
| 13:36:22 | sean-k-mooney | and your reporting that its fialing in the pci filter os its not related to the placment part | |
| 13:36:31 | sean-k-mooney | since it passes that to get to the point where its failing | |
| 13:37:09 | mgariepy | yeah i'll retest in like 30 minutes.. meeting time ;) | |
| 15:04:23 | mgariepy | sean-k-mooney, it works now Thanks a log for your help. | |
| 15:04:47 | mgariepy | with the type-PF and all my server with the same config it's almost magical :D | |
| 15:21:51 | sean-k-mooney | awsome | |
| 15:21:52 | opendevreview | Sylvain Bauza proposed openstack/placement master: Update 2023.2 reqs to support os-traits 3.0.0 as min version https://review.opendev.org/c/openstack/placement/+/895186 | |
| 15:22:21 | sean-k-mooney | the reqeust was proably being accpeted by a not restarted instace | |