| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-12-09 | |||
| 11:55:04 | sean-k-mooney | DominikDanelski[m] so when yyou make the requeted change you shoudl mark it as done. if the ocmment is just geneal feedback but does not requrie a change you can mark it as acknolaged or you can reply and tick resolved in the ui to show the comment is nolonger relevnet | |
| 11:55:49 | sean-k-mooney | for that comment specificly i will get an emial when you updated the patch but not nessialy on a comment | |
| 11:56:15 | sean-k-mooney | the reason i asked you ot use a explcit flavor is its is not obvious why the excption should be raised without finding what the defualt flaovr is | |
| 11:56:51 | sean-k-mooney | you do not have any comment explainging that so as a review i woudl have to go digging intot hte test infra to know if the test should raise or not | |
| 11:57:31 | noonedeadpunk | hey folks. after upgrade to Epoxy I've started seing stack trace when attempting to create a gpu-passthrough instance, and I somehow link it to the https://opendev.org/openstack/nova/commit/78be1679312768383b684fa70ca4d2f5c4e35fa9#diff-8255edc78feecdbe2d14ab15e4580b4b6de916d2 | |
| 11:58:20 | noonedeadpunk | the problem is that self.roots is empty here: https://opendev.org/openstack/nova/src/commit/8a4b000216c7a6c2673af78d7eb7f9bf938dc867/nova/compute/provider_tree.py#L436 | |
| 11:59:40 | noonedeadpunk | from the placement view - resource provider is obviously there: https://paste.openstack.org/show/bn6ekRrWbg9ggNpuGx4h/ | |
| 11:59:59 | sean-k-mooney | self.roots is not refenced in teh commit you linked | |
| 11:59:59 | noonedeadpunk | But I get `ValueError: No such provider 623a0277-62e4-43bc-acae-964dab595905` on the nova-compute side as a result | |
| 12:00:13 | sean-k-mooney | it is used in the code but it is not in the changed code | |
| 12:00:36 | sean-k-mooney | noonedeadpunk: you dont have any downstream backprot do you? | |
| 12:00:51 | noonedeadpunk | no, I don't | |
| 12:00:52 | sean-k-mooney | it kind of soundsl like you mihgt have missed a patch | |
| 12:00:54 | sean-k-mooney | ack | |
| 12:01:07 | noonedeadpunk | but. stack trace goes through the new _remove_managed_rps_from_tree_not_in_view | |
| 12:01:09 | noonedeadpunk | https://paste.openstack.org/show/bqjIms96L7A7ODUPU33W/ | |
| 12:01:47 | sean-k-mooney | that does not refence self.roots either | |
| 12:02:11 | sean-k-mooney | do you have pci in placment enabled in this deployjment | |
| 12:02:29 | noonedeadpunk | Isn't that default? | |
| 12:02:34 | sean-k-mooney | no | |
| 12:02:49 | noonedeadpunk | jsut let me quick check.... | |
| 12:02:52 | sean-k-mooney | but once its enabled we dont support turning it off | |
| 12:03:08 | noonedeadpunk | I was quite sure it is | |
| 12:03:30 | noonedeadpunk | shoot | |
| 12:03:32 | noonedeadpunk | thanks | |
| 12:03:34 | noonedeadpunk | it was fast | |
| 12:03:38 | sean-k-mooney | we may have turned it on in the last release or so | |
| 12:03:42 | noonedeadpunk | I bet somebody has removed it... | |
| 12:03:52 | sean-k-mooney | but i dont know if we did that yet or not | |
| 12:04:02 | noonedeadpunk | I have report_in_placement = True but not pci_in_placement | |
| 12:04:16 | sean-k-mooney | still false https://docs.openstack.org/nova/latest/configuration/config.html#pci.report_in_placement | |
| 12:04:17 | noonedeadpunk | sorry for this stupid thing... | |
| 12:04:36 | sean-k-mooney | no thats actully the option that i wanted | |
| 12:04:40 | sean-k-mooney | so there are two part | |
| 12:04:52 | sean-k-mooney | on the compute you set report_in_placement = True | |
| 12:05:07 | sean-k-mooney | and in the schdluer you set the other flag to include it in the placment query | |
| 12:05:50 | noonedeadpunk | Seems I don;t have override for `pci_in_placement` at all | |
| 12:05:50 | sean-k-mooney | if you have report_in_placement=true it shoudl enabel this codepath but the error is basiclly saying it cant find the resouce provdier | |
| 12:06:10 | opendevreview | Lajos Katona proposed openstack/nova master: WIP: Use SDK for Neutron Ports https://review.opendev.org/c/openstack/nova/+/969298 | |
| 12:06:22 | noonedeadpunk | but if it's a scheduler part - scheduler seems to be able to find and supply the resource id | |
| 12:06:59 | noonedeadpunk | let me fix scheduler anyway... | |
| 12:07:10 | sean-k-mooney | noonedeadpunk: can you check if the resouce provider for that compute exists in placement | |
| 12:07:31 | noonedeadpunk | so the id of in the stack trace is a child | |
| 12:07:39 | noonedeadpunk | https://paste.openstack.org/show/bn6ekRrWbg9ggNpuGx4h/ | |
| 12:07:47 | sean-k-mooney | ack ya it shoudl be the chile for the pci device | |
| 12:08:04 | sean-k-mooney | right so its os-compute-gpu01-az2_0000:25:00.0 | |
| 12:08:12 | noonedeadpunk | yeah | |
| 12:08:16 | sean-k-mooney | the inventory for the gpu | |
| 12:08:37 | sean-k-mooney | so can you confirm that that is in you pci devspec and that the adress is correct | |
| 12:09:11 | noonedeadpunk | 25:00.0 3D controller: NVIDIA Corporation | |
| 12:09:14 | noonedeadpunk | from lspci | |
| 12:09:41 | sean-k-mooney | ya that looks normal | |
| 12:10:28 | sean-k-mooney | i belive self._remove_managed_rps_from_tree_not_in_view(provider_tree) is intended to be used to remove RP for pci device that are nolonger in the whitelist | |
| 12:10:48 | sean-k-mooney | so that is why i was suggestign checkign the devspec in the nova.conf on the compute to confirm that is correct | |
| 12:12:34 | sean-k-mooney | there could ligitimatly be a bug i just want to rule out config errors | |
| 12:13:15 | noonedeadpunk | so one thing I am suspicious about.... | |
| 12:13:47 | noonedeadpunk | Is that we have a custom trait on top of the resource class in there | |
| 12:14:00 | noonedeadpunk | and this custom trait is applied only to the root device | |
| 12:14:20 | noonedeadpunk | https://paste.openstack.org/show/bhtdPwgzYHpzCdoIGe32/ | |
| 12:14:24 | sean-k-mooney | am i dont think that shoudl break anythign but maybe | |
| 12:15:26 | noonedeadpunk | ah, well, it's passed to children as well | |
| 12:15:33 | sean-k-mooney | we shoudl be adding COMPUTE_MANAGED_PCI_DEVICE as well | |
| 12:15:39 | sean-k-mooney | can you check fi you see both? | |
| 12:16:38 | noonedeadpunk | yes, I see CUSTOM_GPU_PASSTHROUGH and COMPUTE_MANAGED_PCI_DEVICE traits on both parent and childs | |
| 12:16:48 | noonedeadpunk | but not the resource_class | |
| 12:17:05 | sean-k-mooney | well the resouce class shoul be the resouce class in the invetnotry | |
| 12:17:11 | noonedeadpunk | But resource class should not be in traits iirc | |
| 12:17:18 | sean-k-mooney | correct | |
| 12:17:27 | sean-k-mooney | if you do an inventory show on the rp you shoudl see it there | |
| 12:17:42 | noonedeadpunk | openstack resource class show CUSTOM_A10_FULL returns the name, yes | |
| 12:20:57 | noonedeadpunk | I kinda failing to understand at what point `self.roots_by_uuid` is expected to be populated | |
| 12:22:16 | sean-k-mooney | https://opendev.org/openstack/nova/src/commit/8a4b000216c7a6c2673af78d7eb7f9bf938dc867/nova/compute/provider_tree.py#L292 | |
| 12:22:17 | noonedeadpunk | it does not look like it's cached, or populated during instance create request | |
| 12:22:47 | noonedeadpunk | but according to trace it's not called? | |
| 12:22:50 | sean-k-mooney | it can be populated in othere wase too | |
| 12:23:10 | sean-k-mooney | the tace doe not actully show a specific error | |
| 12:23:20 | sean-k-mooney | just that the child rp was nto found | |
| 12:23:43 | sean-k-mooney | i assume this si repoabale i.e. if yuou restart the nova comptue agent it happens again | |
| 12:24:02 | noonedeadpunk | well, I used epdb, and self.roots is dict_values([]) | |
| 12:24:10 | noonedeadpunk | yes, sure | |
| 12:24:43 | sean-k-mooney | its returnnign the values mapping of the dict yes | |
| 12:25:17 | noonedeadpunk | and self.roots_by_uuid is {} | |
| 12:25:27 | sean-k-mooney | this shoudl be updated in memory as part fo init_host | |
| 12:25:31 | noonedeadpunk | so it feels that it's not populated in fact... | |
| 12:26:23 | sean-k-mooney | so during init_host we end up running update_aviable_resocues which populates the resouce tacker and caches the provieder tree | |
| 12:27:28 | sean-k-mooney | this should be poplulated before we ever get to _build_and_run_instance | |
| 12:29:36 | noonedeadpunk | unless class is not re-inited at some point.... | |
| 12:36:28 | opendevreview | Dominik proposed openstack/nova master: Regression test for Placement allocations remaining during failed schedule https://review.opendev.org/c/openstack/nova/+/969251 | |
| 12:36:50 | opendevreview | Dominik proposed openstack/nova master: Remove Placement allocations in the broken build cleanup https://review.opendev.org/c/openstack/nova/+/968446 | |
| 12:38:36 | DominikDanelski[m] | sean-k-mooney: All right, I added a comment explaining why the exact flavour doesn't matter, so now all discussion is resolved. | |
| 12:39:33 | noonedeadpunk | sean-k-mooney: so I reverted 78be1679312768383b684fa70ca4d2f5c4e35fa9 and instance become spawning normally :( | |
| 12:40:16 | noonedeadpunk | will submit a bug report in the meanwhile I guess | |
| 12:40:38 | sean-k-mooney | an do they have allcoation against the placemetn RPs for the gpu usage? | |
| 12:41:00 | sean-k-mooney | i..e there were no other sideffects? | |
| 12:41:21 | sean-k-mooney | im wondering if we are misisng a fix for one of the ohter caching bugs related to OTU devices | |
| 12:41:28 | sean-k-mooney | in the sable branch | |
| 12:41:45 | sean-k-mooney | gibi and dan were fixing some other bugs around the same time | |
| 12:44:10 | noonedeadpunk | it looks fine? https://paste.openstack.org/show/bj8ZZgxzHSYHZF2hGFvg/ | |
| 12:45:00 | noonedeadpunk | hm, maybe I am missing some backport... | |
| 12:45:47 | noonedeadpunk | nah, I think we're on current HEAD of 2025.1 | |