| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-05-16 | |||
| 17:14:02 | melwitt | yeah I just linked the devstack stuff above | |
| 17:14:25 | melwitt | bauzas maybe didn't do the config through devstack or something (?) | |
| 17:14:27 | dansmith | I know, I'm just talking about the assertion of it being missed or not supported | |
| 17:14:28 | dansmith | right | |
| 17:14:33 | melwitt | NOVA_USE_UNIFIED_LIMITS=True should do the stuff | |
| 17:14:52 | melwitt | oh, sure | |
| 17:15:42 | bauzas | no, I just modified nova.conf directly | |
| 17:16:27 | bauzas | I don't want to set quotas for all resources but only VGPU | |
| 17:16:51 | bauzas | I know it will mean that all the quotas limits will be unlimitied :) | |
| 17:17:16 | melwitt | ok cool. I'll propose a patch to add the oslo_limit config info to the docs, that's what I missed | |
| 17:18:02 | melwitt | bauzas: make sure you set a limit for DISK_GB otherwise it will treat it as 0 limit | |
| 17:18:27 | melwitt | that's the only new "core" quota resource | |
| 17:18:59 | bauzas | melwitt: oh ok | |
| 17:19:20 | melwitt | oh wait, actually you have to set all of them bc you didn't do in devstack. but you can just copy what's in devstack | |
| 17:20:01 | melwitt | https://github.com/openstack/devstack/blob/85b8d2ccab53bb2b38090c8801404072fae2f7ab/lib/nova#L789-L812 | |
| 17:20:20 | melwitt | and then this to make the nova service user able to read the quotas https://github.com/openstack/devstack/blob/85b8d2ccab53bb2b38090c8801404072fae2f7ab/lib/nova#L827-L829 | |
| 17:24:21 | bauzas | yup, I'm done now, I just need to tweak the registered limits | |
| 17:24:24 | bauzas | Project 1471d08833d141c583b5f04344476ebd is over a limit for [Resource class:DISK_GB is over limit of 0 due to current usage 80 and delta 40, Resource class:MEMORY_MB is over limit of 0 due to current usage 16384 and delta 8192, Resource class:VCPU is over limit of 0 due to current usage 2 and delta 1, Resource class:VGPU is over limit of 0 due to current usage 2 and delta 1, Resource servers is over limit of 0 due to current usa | |
| 17:24:26 | bauzas | ge 2 and delta 1] (HTTP 403) (Request-ID: req-980c41e8-6f18-4e12-984b-7eb5daa7d9e8) | |
| 17:24:53 | bauzas | so, I not only need to add a limit for DISK_GB but also the other classes | |
| 17:29:09 | melwitt | yes. any class that's going to be requested has to have a registered limit else oslo.limit treats it as zero | |
| 17:30:02 | bauzas | yup I know it :) | |
| 17:30:04 | melwitt | (which is why we were discussing at the PTG about how to make an unlimited option in nova) | |
| 17:30:27 | bauzas | yeah my brain fried | |
| 17:31:13 | melwitt | :) | |
| 17:31:13 | bauzas | sometimes my brain can't reconcile two things :) | |
| 17:35:08 | bauzas | yay that works | |
| 17:35:31 | bauzas | I have everything now for my demo, I just need to record it :) | |
| 17:35:51 | bauzas | my next nights => bye bye | |
| 17:36:01 | melwitt | \o/ | |
| 17:36:05 | bauzas | thanks melwitt for the quick resolution and the help | |
| 17:36:38 | bauzas | CERN will be more happy to test unified limits for VGPU if they haven't done it yet :) | |
| 17:36:52 | melwitt | np. you can ping me if you have questions or run into any other issues | |
| 17:37:10 | melwitt | that would be super :) | |
| 17:37:18 | bauzas | no, unified limits is waaaay simplier that f**** nvidia kernel usage :) | |
| 17:38:04 | melwitt | haha! yeah, I guess if you can nvidia then you can do anything 😆 | |
| 17:39:11 | bauzas | at least now I know more about cuda and numba :) | |
| 17:39:47 | bauzas | like the fact we need to use a specific cuda release related to the mdev type | |
| 17:40:23 | bauzas | (and the kernel nvidia version) | |
| 18:11:46 | sean-k-mooney | i mean its not surpsing the have spit there mdev by vdi vs comptue usage in there skus forever | |
| #openstack-nova - 2024-05-17 | |||
| 02:25:47 | opendevreview | melanie witt proposed openstack/nova master: docs: Add more information about unified limits https://review.opendev.org/c/openstack/nova/+/919913 | |
| 03:32:53 | opendevreview | melanie witt proposed openstack/nova master: docs: Add more information about unified limits https://review.opendev.org/c/openstack/nova/+/919913 | |
| 04:05:34 | opendevreview | melanie witt proposed openstack/nova master: docs: Add more information about unified limits https://review.opendev.org/c/openstack/nova/+/919913 | |
| 04:26:56 | opendevreview | melanie witt proposed openstack/nova master: docs: Add more information about unified limits https://review.opendev.org/c/openstack/nova/+/919913 | |
| 07:41:22 | opendevreview | Robert Hoffmann proposed openstack/nova master: Fix backups hanging in creating state https://review.opendev.org/c/openstack/nova/+/919923 | |
| 10:05:31 | opendevreview | Robert Hoffmann proposed openstack/nova master: Fix backups hanging in creating state https://review.opendev.org/c/openstack/nova/+/919923 | |
| 13:13:30 | opendevreview | Mauricio Faria de Oliveira proposed openstack/nova master: libvirt: Filter out disabled guest CPU features in _get_cpu_feature_traits() https://review.opendev.org/c/openstack/nova/+/919949 | |
| 14:10:22 | opendevreview | Andrew Bonney proposed openstack/nova master: [WIP] Set port dns_name when unshelving instances https://review.opendev.org/c/openstack/nova/+/919961 | |
| 17:56:14 | opendevreview | sean mooney proposed openstack/nova master: add functional repoducer for bug 2048837 https://review.opendev.org/c/openstack/nova/+/919979 | |
| 17:56:14 | opendevreview | sean mooney proposed openstack/nova master: [WIP] ensure correct cleanup of multi-attach volumes https://review.opendev.org/c/openstack/nova/+/916322 | |
| 17:57:09 | sean-k-mooney | i spend the better part of 6 hours trying to get that repoduce to pass ^ and then i realsied the reason i could not get it to pass was my fix didnt work... | |
| 17:57:29 | sean-k-mooney | on the plus side i now have a repoducer for the orginal bug... | |
| 17:59:39 | melwitt | too real | |
| 18:05:19 | sean-k-mooney | melwitt: you dont happen to knwo if we stash if a volume is a multi attach volume in the bdms do you | |
| 18:07:03 | melwitt | sean-k-mooney: there isn't a dedicated field for it that I see. I dunno if it could possibly be in the connection_info though | |
| 18:08:18 | sean-k-mooney | ack no worries | |
| 18:09:20 | sean-k-mooney | it is in the conection info yes | |
| 18:09:54 | sean-k-mooney | actully we do cache that in the bdms right | |
| 18:10:21 | melwitt | yeah. sorry, connection_info is a field on the bdm. I was just saying there isn't a field that is only for multiattach | |
| 18:11:33 | sean-k-mooney | right so the simple way to fix the bug is to take a lock on the volume when delete the instnace | |
| 18:11:36 | sean-k-mooney | basiclly @utils.synchronized(f'disconnect_volume_{volume_id}') | |
| 18:12:32 | sean-k-mooney | but instead of doing that in the virt driver in _disconnect_volume | |
| 18:12:56 | melwitt | ok I see | |
| 18:12:57 | sean-k-mooney | i need to do that in _delete_instance in the compute manger | |
| 18:13:10 | sean-k-mooney | but i only need to do that for multi attach volumes | |
| 18:14:46 | sean-k-mooney | i was orginally tring to do the locking as fine grained as possible | |
| 18:15:17 | sean-k-mooney | but that wont work | |
| 18:15:25 | melwitt | oh yeah. so you're thinking you can check bdm.connection_info to see if it's multiattach? | |
| 18:15:56 | sean-k-mooney | ya | |
| 18:16:38 | melwitt | seems like that would work 🤔 | |
| 18:16:38 | sean-k-mooney | the other option is to basiclaly reuse or reinvent https://github.com/openstack/nova/blob/ab3ca1e2059f951b44d723c15a5065c6532540da/nova/virt/libvirt/volume/mount.py#L33 | |
| 18:17:00 | sean-k-mooney | the _HostMountStateManager basicaly impelmentes refernce counting for host mounts | |
| 18:17:18 | sean-k-mooney | but we dont use that for anything other the nfs | |
| 18:19:22 | sean-k-mooney | the problem is this logic https://github.com/openstack/nova/blob/ab3ca1e2059f951b44d723c15a5065c6532540da/nova/virt/libvirt/driver.py#L2045-L2065 | |
| 18:19:36 | sean-k-mooney | if we have 2 vms on the same host using the same multi attach volume | |
| 18:19:46 | sean-k-mooney | and you delete both of them at the same time | |
| 18:19:57 | melwitt | ah ok | |
| 18:20:16 | sean-k-mooney | then because we are looking at the host's instance list until know if we should do the disconenct there is a race | |
| 18:21:18 | sean-k-mooney | so to fix it we either need a refence count and delete it when that hits 0 or we need to just seriase the delete | |
| 18:21:32 | sean-k-mooney | i think the latter is simpler and safer | |
| 18:21:54 | melwitt | I agree | |
| 18:23:01 | sean-k-mooney | ok now that i know someone else does not think this is an insane approch i think im going to call it a day | |
| 18:23:18 | melwitt | haha :) | |
| 18:23:54 | sean-k-mooney | the worst part was i didnt even thing the issue was with my fix. i was confived the issue was with my repoducer test for so long | |
| 18:24:38 | melwitt | I know the feeling 😝 | |
| 19:26:11 | opendevreview | melanie witt proposed openstack/nova master: docs: Add more information about unified limits https://review.opendev.org/c/openstack/nova/+/919913 | |
| 20:51:19 | opendevreview | melanie witt proposed openstack/nova master: Support create with ephemeral encryption for qcow2 https://review.opendev.org/c/openstack/nova/+/870932 | |
| 20:51:19 | opendevreview | melanie witt proposed openstack/nova master: Consolidate vTPM and ephemeral encryption secret creation https://review.opendev.org/c/openstack/nova/+/912094 | |
| 20:51:20 | opendevreview | melanie witt proposed openstack/nova master: libvirt: Add optional 'description' kwarg to create_secret() https://review.opendev.org/c/openstack/nova/+/919986 | |
| 23:42:27 | opendevreview | melanie witt proposed openstack/nova master: Support (resize|cold migration) with ephemeral encryption for qcow2 https://review.opendev.org/c/openstack/nova/+/870933 | |
| 23:42:28 | opendevreview | melanie witt proposed openstack/nova master: Support live migration with ephemeral encryption for qcow2 https://review.opendev.org/c/openstack/nova/+/905512 | |
| 23:42:29 | opendevreview | melanie witt proposed openstack/nova master: Support rebuild with ephemeral encryption for qcow2 https://review.opendev.org/c/openstack/nova/+/870939 | |
| 23:42:31 | opendevreview | melanie witt proposed openstack/nova master: Add backing_encryption_secret_uuid to BlockDeviceMapping https://review.opendev.org/c/openstack/nova/+/907960 | |
| 23:42:31 | opendevreview | melanie witt proposed openstack/nova master: Add encryption support to qemu-img rebase https://review.opendev.org/c/openstack/nova/+/870936 | |
| 23:42:32 | opendevreview | melanie witt proposed openstack/nova master: Support encrypted backing files for qcow2 https://review.opendev.org/c/openstack/nova/+/907961 | |
| 23:42:34 | opendevreview | melanie witt proposed openstack/nova master: Validate key manager create access in API https://review.opendev.org/c/openstack/nova/+/919989 | |
| 23:42:34 | opendevreview | melanie witt proposed openstack/nova master: Support rescue with ephemeral encryption for qcow2 https://review.opendev.org/c/openstack/nova/+/873675 | |
| 23:42:35 | opendevreview | melanie witt proposed openstack/nova master: libvirt: "Auto heal" ephemeral encryption secrets on guest launch https://review.opendev.org/c/openstack/nova/+/919991 | |
| 23:42:35 | opendevreview | melanie witt proposed openstack/nova master: Clean up unused ephemeral encryption libvirt secrets https://review.opendev.org/c/openstack/nova/+/919990 | |
| 23:42:36 | opendevreview | melanie witt proposed openstack/nova master: Support encrypted source images for qcow2 https://review.opendev.org/c/openstack/nova/+/919992 | |