| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-09-24 | |||
| 13:08:33 | gibi | or is it tracked on the compute RP? | |
| 13:21:58 | opendevreview | sean mooney proposed openstack/nova master: [DMN] test removal of CAP_DAC_OVERRIDE https://review.opendev.org/c/openstack/nova/+/810906 | |
| 13:22:17 | bauzas | gibi: my bad, those are just inventories from the same RP | |
| 13:22:19 | sean-k-mooney | gibi: i think its on the compute node currently | |
| 13:22:31 | gibi | yepp that what was I assumed | |
| 13:22:33 | bauzas | sean-k-mooney: yeah, https://github.com/openstack/nova/blob/62406b5728077afa9cd38d5c5d510bba64c43bd7/nova/virt/libvirt/driver.py#L8379 | |
| 13:22:33 | sean-k-mooney | with 1 inventory per namespace | |
| 13:22:50 | sean-k-mooney | *namespace size | |
| 13:23:06 | bauzas | then, maybe bandwidth-aware resource providers could be impacted or is that only a VGPU thing ? | |
| 13:23:23 | bauzas | actually, this doesn't come from the virt driver | |
| 13:23:28 | gibi | bauzas: bwm RPs are managed by neutron agents | |
| 13:23:32 | bauzas | yeah | |
| 13:23:38 | sean-k-mooney | what are you currently talking about by the way | |
| 13:23:39 | bauzas | so I guess this is maybe unrelate | |
| 13:23:43 | gibi | to be precies RP and inventory is managed by neutron, allocation managed by nova | |
| 13:24:05 | bauzas | sean-k-mooney: https://bugs.launchpad.net/nova/+bug/1944031 | |
| 13:24:06 | sean-k-mooney | for bandwith and pps inventories? | |
| 13:24:08 | gibi | sean-k-mooney: https://bugs.launchpad.net/nova/+bug/1944031 | |
| 13:24:14 | bauzas | shoooooot | |
| 13:24:17 | sean-k-mooney | ah right that | |
| 13:24:58 | sean-k-mooney | well for things on the root rp its simple to fix | |
| 13:25:09 | sean-k-mooney | for nested RPs without an owner trait on the RP its harder | |
| 13:25:40 | sean-k-mooney | e.g. if RP had ownwer_nova and its not in the tree nova just compute then it should be removed | |
| 13:25:59 | sean-k-mooney | without that its very hard to figure out it the RP is owned by nova or by cyborg/neutorn | |
| 13:27:06 | sean-k-mooney | if vgpus were tracked in the db we woudl also have another way to figure this out but since we dont have them in the resouce table or pci_devices table | |
| 13:27:20 | sean-k-mooney | we cant use the nova db to determin if it was an rp we created | |
| 13:27:25 | bauzas | sean-k-mooney: yeah tbh, I think it should be the responsibility of the virt driver to compare the provider tree before and after the update and eventually remove unused RPs | |
| 13:27:53 | sean-k-mooney | bauzas: it should but we dont have the info in placment to be able to do that correctly today | |
| 13:28:30 | bauzas | sean-k-mooney: this could be done in https://github.com/openstack/nova/blob/62406b5728077afa9cd38d5c5d510bba64c43bd7/nova/virt/libvirt/driver.py#L8472 | |
| 13:28:43 | sean-k-mooney | bauzas: not today | |
| 13:28:51 | bauzas | of course | |
| 13:28:59 | bauzas | but we have the actual state in provider_tree | |
| 13:29:04 | sean-k-mooney | bauzas: you can know if the nested RP is own by neutron or cyborg | |
| 13:29:05 | bauzas | that's given | |
| 13:29:13 | sean-k-mooney | so it not safe to remove it | |
| 13:29:35 | bauzas | sean-k-mooney: we namespace the RPs | |
| 13:29:42 | sean-k-mooney | we do not | |
| 13:29:50 | gibi | what we can do is to assume that if the RP had VGPU inventory before then it is owned by nova | |
| 13:29:53 | bauzas | for GPUs ? we totally do | |
| 13:29:53 | gibi | :) | |
| 13:30:07 | gibi | or we use the name of the RP | |
| 13:30:21 | sean-k-mooney | gibi: only if we intend to make it so cyborg cannot use the vgpu inventory type | |
| 13:30:25 | bauzas | the problem is that this naming could be used by *something else* | |
| 13:31:10 | bauzas | but if we consider that https://github.com/openstack/nova/blob/62406b5728077afa9cd38d5c5d510bba64c43bd7/nova/virt/libvirt/driver.py#L8505 is only used by the virt driver, then you can look at all existing RPs matching this | |
| 13:31:25 | sean-k-mooney | any resilance on nameing or RC usage is fragile | |
| 13:31:35 | gibi | yep it is fragile I agree | |
| 13:31:41 | bauzas | agreed tho | |
| 13:31:47 | gibi | RP naming is a bit better than RC I think | |
| 13:32:00 | gibi | so fixing this as a bug I would go with relying on RP naming | |
| 13:32:09 | sean-k-mooney | if we prefixed them all with nova_ then i would strongly agree | |
| 13:32:20 | sean-k-mooney | fight now however we may have collitions although its unlikely | |
| 13:32:29 | sean-k-mooney | *right | |
| 13:32:39 | sean-k-mooney | gibi: as a bug fix using the name template | |
| 13:32:54 | sean-k-mooney | would proably be ok if we implement a proper solution in yoga | |
| 13:33:45 | bauzas | honestly, operators are not intended to change their config everyday | |
| 13:33:54 | sean-k-mooney | e.g. in nova prefix all nova RPs with nova_ or add a mapping table in our db where we store the uuids of all RPs we own | |
| 13:34:01 | gibi | sean-k-mooney: yeah, thinking about it in Yoga PTG would be a good first step | |
| 13:34:03 | bauzas | we could somehow use the audit command to find the orphaned RPs | |
| 13:34:20 | sean-k-mooney | a nova manage command ya would work | |
| 13:34:31 | bauzas | it would have my preference honestly | |
| 13:34:32 | sean-k-mooney | using osc they can also fix it today | |
| 13:34:38 | sean-k-mooney | via osc-placment | |
| 13:34:40 | bauzas | sure, hence the low | |
| 13:34:46 | bauzas | the low status I mean | |
| 13:34:51 | bauzas | this is absolutely not a problem | |
| 13:35:03 | bauzas | this is just, you're creating garbage | |
| 13:35:15 | sean-k-mooney | well it can break schduling | |
| 13:35:19 | sean-k-mooney | so it can be a problem | |
| 13:35:35 | bauzas | give me a sec, verifying the inventories | |
| 13:35:45 | sean-k-mooney | but we do not nessisaly need to automically fix it if we provide a tool to do it when you change the config | |
| 13:36:27 | sean-k-mooney | gibi: just an fyi im on PTO next week just in case your looking for me | |
| 13:36:28 | bauzas | hmmm, you're right | |
| 13:36:35 | bauzas | inventories are left | |
| 13:36:38 | bauzas | [root@hab-19 devstack]# openstack resource provider inventory list 7853e09b-5f5c-4b79-99c6-a550c56eb7e0 | |
| 13:36:38 | bauzas | | resource_class | allocation_ratio | min_unit | max_unit | reserved | step_size | total | used | | |
| 13:36:38 | bauzas | +----------------+------------------+----------+----------+----------+-----------+-------+------+ | |
| 13:36:38 | bauzas | | VGPU | 1.0 | 1 | 1 | 0 | 1 | 1 | 0 | | |
| 13:36:39 | bauzas | +----------------+------------------+----------+----------+----------+-----------+-------+------+ | |
| 13:36:41 | bauzas | dammit | |
| 13:36:50 | gibi | sean-k-mooney: thanks for the heads up, enjoy your time off | |
| 13:37:03 | bauzas | so instances could pick those resources | |
| 13:37:14 | sean-k-mooney | bauzas: ya the inventories would because when you remove the confi the virt driver will just not update the rp or its inventories again | |
| 13:37:29 | bauzas | yup | |
| 13:37:30 | sean-k-mooney | this is also a problem for provider.yaml by the way | |
| 13:37:49 | sean-k-mooney | if we remove something form provider.yaml it will never get removed form placment | |
| 13:38:12 | bauzas | definitely worth a PTG discussion then | |
| 13:38:23 | sean-k-mooney | bauzas: we could perhaps write a file to disk in the nova state dir | |
| 13:38:25 | bauzas | sean-k-mooney: fancy adding a 10th bullet point or want me to write it ? | |
| 13:38:45 | bauzas | I mean, I couldf | |
| 13:38:54 | sean-k-mooney | that way the agent could read it on start up and use that to figure out if something was removed | |
| 13:39:02 | sean-k-mooney | if we did not want to store it in the db | |
| 13:39:26 | sean-k-mooney | bauzas: am i can i guess i have another to add anyway | |
| 13:39:45 | opendevreview | Balazs Gibizer proposed openstack/nova master: Store old_flavor already on source host during resize https://review.opendev.org/c/openstack/nova/+/810909 | |
| 13:39:48 | sean-k-mooney | just assume the stuff i add is an aggreate of stuff we have mentioned in downstream or upstream meetings/discussions | |
| 13:42:56 | gibi | bauzas, sean-k-mooney: I will check what today's neutron does if I remove the bw config. I guess we have the same missing cleanup problem in neutron too. If we have, then I would argue for a solution that is not nova specific or at least easy to apply to neutron too | |
| 13:44:38 | bauzas | gibi: that's why a audit tool seems important | |
| 13:44:59 | bauzas | but the working logic is owned by neutron or the virt driver | |
| 13:45:14 | bauzas | so I don't know how a tool could be prescriptive | |
| 13:45:28 | gibi | bauzas: RP cleanup is OK to be left for the nova-manage but inventory cleanup is needed to be automatic to avoid wrong scheduling | |
| 13:45:42 | bauzas | agreed | |