| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-05-16 | |||
| 16:55:29 | bauzas | none from the conductor or the compute ? | |
| 16:56:01 | melwitt | bauzas: I think only n-api. I'll double check | |
| 16:56:24 | bauzas | I quickly looked at the code, I only found the scheduler and the api were importing nova.quota | |
| 16:58:44 | melwitt | bauzas: there's a "recheck" in nova-conductor that enforces "strict quota" like races | |
| 16:59:06 | bauzas | then I need to also update nova_cell1.conf | |
| 17:00:41 | bauzas | Unable to initialize OpenStackSDK session: An auth plugin is required to determine endpoint URL: keystoneauth1.exceptions.auth_plugins.MissingAuthPlugin | |
| 17:00:48 | bauzas | humpf, I probably need to set something else | |
| 17:01:29 | bauzas | melwitt: do you know what I missed ? ^ | |
| 17:01:56 | bauzas | (that's the API exception I get when wanting to create a new instance) | |
| 17:02:02 | melwitt | looking.. | |
| 17:02:48 | bauzas | oh, I have nothing | |
| 17:02:55 | bauzas | devstack doesn't automatically set it | |
| 17:04:00 | melwitt | yeah.. maybe something switched to using the sdk since then? I don't remember seeing anything like this | |
| 17:04:18 | melwitt | I need to lookup what's needed for sdk. though it seems surprising it wouldn't be done automatically in devstack like you said | |
| 17:08:46 | melwitt | oh huh that's actually coming from oslo.limit. so likely something needed in an [oslo_limit] conf section. still looking | |
| 17:09:03 | bauzas | yeah, still getting the api exception | |
| 17:09:21 | bauzas | after restarting n-api (and setting [keystone] section)à | |
| 17:10:04 | melwitt | https://docs.openstack.org/oslo.limit/latest/user/usage.html#configuration | |
| 17:10:09 | bauzas | https://docs.openstack.org/oslo.limit/latest/user/usage.html#configuration | |
| 17:10:13 | bauzas | heh | |
| 17:11:14 | melwitt | I'm just confused how/where this got missed. and I'll will add stuff to the quota docs | |
| 17:12:17 | melwitt | ok, here it is in devstack, it should have configured it for you https://github.com/openstack/devstack/blob/85b8d2ccab53bb2b38090c8801404072fae2f7ab/lib/nova#L817 | |
| 17:12:26 | dansmith | do we not have a job with unified limits enabled? | |
| 17:12:32 | melwitt | (I'll still add to docs) | |
| 17:12:39 | melwitt | we do, it's on in nova-next | |
| 17:13:13 | dansmith | ack, so it must be in devstack unless we're hacking it all on in our job config right? | |
| 17:14:02 | melwitt | yeah I just linked the devstack stuff above | |
| 17:14:25 | melwitt | bauzas maybe didn't do the config through devstack or something (?) | |
| 17:14:27 | dansmith | I know, I'm just talking about the assertion of it being missed or not supported | |
| 17:14:28 | dansmith | right | |
| 17:14:33 | melwitt | NOVA_USE_UNIFIED_LIMITS=True should do the stuff | |
| 17:14:52 | melwitt | oh, sure | |
| 17:15:42 | bauzas | no, I just modified nova.conf directly | |
| 17:16:27 | bauzas | I don't want to set quotas for all resources but only VGPU | |
| 17:16:51 | bauzas | I know it will mean that all the quotas limits will be unlimitied :) | |
| 17:17:16 | melwitt | ok cool. I'll propose a patch to add the oslo_limit config info to the docs, that's what I missed | |
| 17:18:02 | melwitt | bauzas: make sure you set a limit for DISK_GB otherwise it will treat it as 0 limit | |
| 17:18:27 | melwitt | that's the only new "core" quota resource | |
| 17:18:59 | bauzas | melwitt: oh ok | |
| 17:19:20 | melwitt | oh wait, actually you have to set all of them bc you didn't do in devstack. but you can just copy what's in devstack | |
| 17:20:01 | melwitt | https://github.com/openstack/devstack/blob/85b8d2ccab53bb2b38090c8801404072fae2f7ab/lib/nova#L789-L812 | |
| 17:20:20 | melwitt | and then this to make the nova service user able to read the quotas https://github.com/openstack/devstack/blob/85b8d2ccab53bb2b38090c8801404072fae2f7ab/lib/nova#L827-L829 | |
| 17:24:21 | bauzas | yup, I'm done now, I just need to tweak the registered limits | |
| 17:24:24 | bauzas | Project 1471d08833d141c583b5f04344476ebd is over a limit for [Resource class:DISK_GB is over limit of 0 due to current usage 80 and delta 40, Resource class:MEMORY_MB is over limit of 0 due to current usage 16384 and delta 8192, Resource class:VCPU is over limit of 0 due to current usage 2 and delta 1, Resource class:VGPU is over limit of 0 due to current usage 2 and delta 1, Resource servers is over limit of 0 due to current usa | |
| 17:24:26 | bauzas | ge 2 and delta 1] (HTTP 403) (Request-ID: req-980c41e8-6f18-4e12-984b-7eb5daa7d9e8) | |
| 17:24:53 | bauzas | so, I not only need to add a limit for DISK_GB but also the other classes | |
| 17:29:09 | melwitt | yes. any class that's going to be requested has to have a registered limit else oslo.limit treats it as zero | |
| 17:30:02 | bauzas | yup I know it :) | |
| 17:30:04 | melwitt | (which is why we were discussing at the PTG about how to make an unlimited option in nova) | |
| 17:30:27 | bauzas | yeah my brain fried | |
| 17:31:13 | melwitt | :) | |
| 17:31:13 | bauzas | sometimes my brain can't reconcile two things :) | |
| 17:35:08 | bauzas | yay that works | |
| 17:35:31 | bauzas | I have everything now for my demo, I just need to record it :) | |
| 17:35:51 | bauzas | my next nights => bye bye | |
| 17:36:01 | melwitt | \o/ | |
| 17:36:05 | bauzas | thanks melwitt for the quick resolution and the help | |
| 17:36:38 | bauzas | CERN will be more happy to test unified limits for VGPU if they haven't done it yet :) | |
| 17:36:52 | melwitt | np. you can ping me if you have questions or run into any other issues | |
| 17:37:10 | melwitt | that would be super :) | |
| 17:37:18 | bauzas | no, unified limits is waaaay simplier that f**** nvidia kernel usage :) | |
| 17:38:04 | melwitt | haha! yeah, I guess if you can nvidia then you can do anything 😆 | |
| 17:39:11 | bauzas | at least now I know more about cuda and numba :) | |
| 17:39:47 | bauzas | like the fact we need to use a specific cuda release related to the mdev type | |
| 17:40:23 | bauzas | (and the kernel nvidia version) | |
| 18:11:46 | sean-k-mooney | i mean its not surpsing the have spit there mdev by vdi vs comptue usage in there skus forever | |
| #openstack-nova - 2024-05-17 | |||
| 02:25:47 | opendevreview | melanie witt proposed openstack/nova master: docs: Add more information about unified limits https://review.opendev.org/c/openstack/nova/+/919913 | |
| 03:32:53 | opendevreview | melanie witt proposed openstack/nova master: docs: Add more information about unified limits https://review.opendev.org/c/openstack/nova/+/919913 | |
| 04:05:34 | opendevreview | melanie witt proposed openstack/nova master: docs: Add more information about unified limits https://review.opendev.org/c/openstack/nova/+/919913 | |
| 04:26:56 | opendevreview | melanie witt proposed openstack/nova master: docs: Add more information about unified limits https://review.opendev.org/c/openstack/nova/+/919913 | |
| 07:41:22 | opendevreview | Robert Hoffmann proposed openstack/nova master: Fix backups hanging in creating state https://review.opendev.org/c/openstack/nova/+/919923 | |
| 10:05:31 | opendevreview | Robert Hoffmann proposed openstack/nova master: Fix backups hanging in creating state https://review.opendev.org/c/openstack/nova/+/919923 | |
| 13:13:30 | opendevreview | Mauricio Faria de Oliveira proposed openstack/nova master: libvirt: Filter out disabled guest CPU features in _get_cpu_feature_traits() https://review.opendev.org/c/openstack/nova/+/919949 | |
| 14:10:22 | opendevreview | Andrew Bonney proposed openstack/nova master: [WIP] Set port dns_name when unshelving instances https://review.opendev.org/c/openstack/nova/+/919961 | |
| 17:56:14 | opendevreview | sean mooney proposed openstack/nova master: add functional repoducer for bug 2048837 https://review.opendev.org/c/openstack/nova/+/919979 | |
| 17:56:14 | opendevreview | sean mooney proposed openstack/nova master: [WIP] ensure correct cleanup of multi-attach volumes https://review.opendev.org/c/openstack/nova/+/916322 | |
| 17:57:09 | sean-k-mooney | i spend the better part of 6 hours trying to get that repoduce to pass ^ and then i realsied the reason i could not get it to pass was my fix didnt work... | |
| 17:57:29 | sean-k-mooney | on the plus side i now have a repoducer for the orginal bug... | |
| 17:59:39 | melwitt | too real | |
| 18:05:19 | sean-k-mooney | melwitt: you dont happen to knwo if we stash if a volume is a multi attach volume in the bdms do you | |
| 18:07:03 | melwitt | sean-k-mooney: there isn't a dedicated field for it that I see. I dunno if it could possibly be in the connection_info though | |
| 18:08:18 | sean-k-mooney | ack no worries | |
| 18:09:20 | sean-k-mooney | it is in the conection info yes | |
| 18:09:54 | sean-k-mooney | actully we do cache that in the bdms right | |
| 18:10:21 | melwitt | yeah. sorry, connection_info is a field on the bdm. I was just saying there isn't a field that is only for multiattach | |
| 18:11:33 | sean-k-mooney | right so the simple way to fix the bug is to take a lock on the volume when delete the instnace | |
| 18:11:36 | sean-k-mooney | basiclly @utils.synchronized(f'disconnect_volume_{volume_id}') | |
| 18:12:32 | sean-k-mooney | but instead of doing that in the virt driver in _disconnect_volume | |
| 18:12:56 | melwitt | ok I see | |
| 18:12:57 | sean-k-mooney | i need to do that in _delete_instance in the compute manger | |
| 18:13:10 | sean-k-mooney | but i only need to do that for multi attach volumes | |
| 18:14:46 | sean-k-mooney | i was orginally tring to do the locking as fine grained as possible | |
| 18:15:17 | sean-k-mooney | but that wont work | |
| 18:15:25 | melwitt | oh yeah. so you're thinking you can check bdm.connection_info to see if it's multiattach? | |
| 18:15:56 | sean-k-mooney | ya | |
| 18:16:38 | melwitt | seems like that would work 🤔 | |
| 18:16:38 | sean-k-mooney | the other option is to basiclaly reuse or reinvent https://github.com/openstack/nova/blob/ab3ca1e2059f951b44d723c15a5065c6532540da/nova/virt/libvirt/volume/mount.py#L33 | |
| 18:17:00 | sean-k-mooney | the _HostMountStateManager basicaly impelmentes refernce counting for host mounts | |
| 18:17:18 | sean-k-mooney | but we dont use that for anything other the nfs | |
| 18:19:22 | sean-k-mooney | the problem is this logic https://github.com/openstack/nova/blob/ab3ca1e2059f951b44d723c15a5065c6532540da/nova/virt/libvirt/driver.py#L2045-L2065 | |