| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-05-16 | |||
| 17:20:20 | melwitt | and then this to make the nova service user able to read the quotas https://github.com/openstack/devstack/blob/85b8d2ccab53bb2b38090c8801404072fae2f7ab/lib/nova#L827-L829 | |
| 17:24:21 | bauzas | yup, I'm done now, I just need to tweak the registered limits | |
| 17:24:24 | bauzas | Project 1471d08833d141c583b5f04344476ebd is over a limit for [Resource class:DISK_GB is over limit of 0 due to current usage 80 and delta 40, Resource class:MEMORY_MB is over limit of 0 due to current usage 16384 and delta 8192, Resource class:VCPU is over limit of 0 due to current usage 2 and delta 1, Resource class:VGPU is over limit of 0 due to current usage 2 and delta 1, Resource servers is over limit of 0 due to current usa | |
| 17:24:26 | bauzas | ge 2 and delta 1] (HTTP 403) (Request-ID: req-980c41e8-6f18-4e12-984b-7eb5daa7d9e8) | |
| 17:24:53 | bauzas | so, I not only need to add a limit for DISK_GB but also the other classes | |
| 17:29:09 | melwitt | yes. any class that's going to be requested has to have a registered limit else oslo.limit treats it as zero | |
| 17:30:02 | bauzas | yup I know it :) | |
| 17:30:04 | melwitt | (which is why we were discussing at the PTG about how to make an unlimited option in nova) | |
| 17:30:27 | bauzas | yeah my brain fried | |
| 17:31:13 | bauzas | sometimes my brain can't reconcile two things :) | |
| 17:31:13 | melwitt | :) | |
| 17:35:08 | bauzas | yay that works | |
| 17:35:31 | bauzas | I have everything now for my demo, I just need to record it :) | |
| 17:35:51 | bauzas | my next nights => bye bye | |
| 17:36:01 | melwitt | \o/ | |
| 17:36:05 | bauzas | thanks melwitt for the quick resolution and the help | |
| 17:36:38 | bauzas | CERN will be more happy to test unified limits for VGPU if they haven't done it yet :) | |
| 17:36:52 | melwitt | np. you can ping me if you have questions or run into any other issues | |
| 17:37:10 | melwitt | that would be super :) | |
| 17:37:18 | bauzas | no, unified limits is waaaay simplier that f**** nvidia kernel usage :) | |
| 17:38:04 | melwitt | haha! yeah, I guess if you can nvidia then you can do anything 😆 | |
| 17:39:11 | bauzas | at least now I know more about cuda and numba :) | |
| 17:39:47 | bauzas | like the fact we need to use a specific cuda release related to the mdev type | |
| 17:40:23 | bauzas | (and the kernel nvidia version) | |
| 18:11:46 | sean-k-mooney | i mean its not surpsing the have spit there mdev by vdi vs comptue usage in there skus forever | |
| #openstack-nova - 2024-05-17 | |||
| 02:25:47 | opendevreview | melanie witt proposed openstack/nova master: docs: Add more information about unified limits https://review.opendev.org/c/openstack/nova/+/919913 | |
| 03:32:53 | opendevreview | melanie witt proposed openstack/nova master: docs: Add more information about unified limits https://review.opendev.org/c/openstack/nova/+/919913 | |
| 04:05:34 | opendevreview | melanie witt proposed openstack/nova master: docs: Add more information about unified limits https://review.opendev.org/c/openstack/nova/+/919913 | |
| 04:26:56 | opendevreview | melanie witt proposed openstack/nova master: docs: Add more information about unified limits https://review.opendev.org/c/openstack/nova/+/919913 | |
| 07:41:22 | opendevreview | Robert Hoffmann proposed openstack/nova master: Fix backups hanging in creating state https://review.opendev.org/c/openstack/nova/+/919923 | |
| 10:05:31 | opendevreview | Robert Hoffmann proposed openstack/nova master: Fix backups hanging in creating state https://review.opendev.org/c/openstack/nova/+/919923 | |
| 13:13:30 | opendevreview | Mauricio Faria de Oliveira proposed openstack/nova master: libvirt: Filter out disabled guest CPU features in _get_cpu_feature_traits() https://review.opendev.org/c/openstack/nova/+/919949 | |
| 14:10:22 | opendevreview | Andrew Bonney proposed openstack/nova master: [WIP] Set port dns_name when unshelving instances https://review.opendev.org/c/openstack/nova/+/919961 | |
| 17:56:14 | opendevreview | sean mooney proposed openstack/nova master: [WIP] ensure correct cleanup of multi-attach volumes https://review.opendev.org/c/openstack/nova/+/916322 | |
| 17:56:14 | opendevreview | sean mooney proposed openstack/nova master: add functional repoducer for bug 2048837 https://review.opendev.org/c/openstack/nova/+/919979 | |
| 17:57:09 | sean-k-mooney | i spend the better part of 6 hours trying to get that repoduce to pass ^ and then i realsied the reason i could not get it to pass was my fix didnt work... | |
| 17:57:29 | sean-k-mooney | on the plus side i now have a repoducer for the orginal bug... | |
| 17:59:39 | melwitt | too real | |
| 18:05:19 | sean-k-mooney | melwitt: you dont happen to knwo if we stash if a volume is a multi attach volume in the bdms do you | |
| 18:07:03 | melwitt | sean-k-mooney: there isn't a dedicated field for it that I see. I dunno if it could possibly be in the connection_info though | |
| 18:08:18 | sean-k-mooney | ack no worries | |
| 18:09:20 | sean-k-mooney | it is in the conection info yes | |
| 18:09:54 | sean-k-mooney | actully we do cache that in the bdms right | |
| 18:10:21 | melwitt | yeah. sorry, connection_info is a field on the bdm. I was just saying there isn't a field that is only for multiattach | |
| 18:11:33 | sean-k-mooney | right so the simple way to fix the bug is to take a lock on the volume when delete the instnace | |
| 18:11:36 | sean-k-mooney | basiclly @utils.synchronized(f'disconnect_volume_{volume_id}') | |
| 18:12:32 | sean-k-mooney | but instead of doing that in the virt driver in _disconnect_volume | |
| 18:12:56 | melwitt | ok I see | |
| 18:12:57 | sean-k-mooney | i need to do that in _delete_instance in the compute manger | |
| 18:13:10 | sean-k-mooney | but i only need to do that for multi attach volumes | |
| 18:14:46 | sean-k-mooney | i was orginally tring to do the locking as fine grained as possible | |
| 18:15:17 | sean-k-mooney | but that wont work | |
| 18:15:25 | melwitt | oh yeah. so you're thinking you can check bdm.connection_info to see if it's multiattach? | |
| 18:15:56 | sean-k-mooney | ya | |
| 18:16:38 | sean-k-mooney | the other option is to basiclaly reuse or reinvent https://github.com/openstack/nova/blob/ab3ca1e2059f951b44d723c15a5065c6532540da/nova/virt/libvirt/volume/mount.py#L33 | |
| 18:16:38 | melwitt | seems like that would work 🤔 | |
| 18:17:00 | sean-k-mooney | the _HostMountStateManager basicaly impelmentes refernce counting for host mounts | |
| 18:17:18 | sean-k-mooney | but we dont use that for anything other the nfs | |
| 18:19:22 | sean-k-mooney | the problem is this logic https://github.com/openstack/nova/blob/ab3ca1e2059f951b44d723c15a5065c6532540da/nova/virt/libvirt/driver.py#L2045-L2065 | |
| 18:19:36 | sean-k-mooney | if we have 2 vms on the same host using the same multi attach volume | |
| 18:19:46 | sean-k-mooney | and you delete both of them at the same time | |
| 18:19:57 | melwitt | ah ok | |
| 18:20:16 | sean-k-mooney | then because we are looking at the host's instance list until know if we should do the disconenct there is a race | |
| 18:21:18 | sean-k-mooney | so to fix it we either need a refence count and delete it when that hits 0 or we need to just seriase the delete | |
| 18:21:32 | sean-k-mooney | i think the latter is simpler and safer | |
| 18:21:54 | melwitt | I agree | |
| 18:23:01 | sean-k-mooney | ok now that i know someone else does not think this is an insane approch i think im going to call it a day | |
| 18:23:18 | melwitt | haha :) | |
| 18:23:54 | sean-k-mooney | the worst part was i didnt even thing the issue was with my fix. i was confived the issue was with my repoducer test for so long | |
| 18:24:38 | melwitt | I know the feeling 😝 | |
| 19:26:11 | opendevreview | melanie witt proposed openstack/nova master: docs: Add more information about unified limits https://review.opendev.org/c/openstack/nova/+/919913 | |
| 20:51:19 | opendevreview | melanie witt proposed openstack/nova master: Consolidate vTPM and ephemeral encryption secret creation https://review.opendev.org/c/openstack/nova/+/912094 | |
| 20:51:19 | opendevreview | melanie witt proposed openstack/nova master: Support create with ephemeral encryption for qcow2 https://review.opendev.org/c/openstack/nova/+/870932 | |
| 20:51:20 | opendevreview | melanie witt proposed openstack/nova master: libvirt: Add optional 'description' kwarg to create_secret() https://review.opendev.org/c/openstack/nova/+/919986 | |
| 23:42:27 | opendevreview | melanie witt proposed openstack/nova master: Support (resize|cold migration) with ephemeral encryption for qcow2 https://review.opendev.org/c/openstack/nova/+/870933 | |
| 23:42:28 | opendevreview | melanie witt proposed openstack/nova master: Support live migration with ephemeral encryption for qcow2 https://review.opendev.org/c/openstack/nova/+/905512 | |
| 23:42:29 | opendevreview | melanie witt proposed openstack/nova master: Support rebuild with ephemeral encryption for qcow2 https://review.opendev.org/c/openstack/nova/+/870939 | |
| 23:42:31 | opendevreview | melanie witt proposed openstack/nova master: Add encryption support to qemu-img rebase https://review.opendev.org/c/openstack/nova/+/870936 | |
| 23:42:31 | opendevreview | melanie witt proposed openstack/nova master: Add backing_encryption_secret_uuid to BlockDeviceMapping https://review.opendev.org/c/openstack/nova/+/907960 | |
| 23:42:32 | opendevreview | melanie witt proposed openstack/nova master: Support encrypted backing files for qcow2 https://review.opendev.org/c/openstack/nova/+/907961 | |
| 23:42:34 | opendevreview | melanie witt proposed openstack/nova master: Support rescue with ephemeral encryption for qcow2 https://review.opendev.org/c/openstack/nova/+/873675 | |
| 23:42:34 | opendevreview | melanie witt proposed openstack/nova master: Validate key manager create access in API https://review.opendev.org/c/openstack/nova/+/919989 | |
| 23:42:35 | opendevreview | melanie witt proposed openstack/nova master: Clean up unused ephemeral encryption libvirt secrets https://review.opendev.org/c/openstack/nova/+/919990 | |
| 23:42:35 | opendevreview | melanie witt proposed openstack/nova master: libvirt: "Auto heal" ephemeral encryption secrets on guest launch https://review.opendev.org/c/openstack/nova/+/919991 | |
| 23:42:36 | opendevreview | melanie witt proposed openstack/nova master: Support encrypted source images for qcow2 https://review.opendev.org/c/openstack/nova/+/919992 | |
| #openstack-nova - 2024-05-19 | |||
| 13:49:54 | opendevreview | Takashi Kajinami proposed openstack/osc-placement master: Replace simplejson by built-in json https://review.opendev.org/c/openstack/osc-placement/+/919999 | |
| 14:34:57 | opendevreview | Dmitriy Rabotyagov proposed openstack/nova master: Handle MAC dash-addresses during migration https://review.opendev.org/c/openstack/nova/+/919760 | |
| #openstack-nova - 2024-05-20 | |||
| 11:56:10 | wncslln | hey nova o/, I was looking at some low hanging fruits bugs to solve, I've found this one https://bugs.launchpad.net/nova/+bug/2065693 | |
| 11:56:32 | wncslln | since I'm a beginner, someone could show me where I should fix, if is possible? | |
| 11:57:35 | wncslln | I undestood that foreign key may does not exist in another table | |
| 12:04:14 | sean-k-mooney | wncslln: that might not be the best bug to start with | |
| 12:05:17 | sean-k-mooney | this tempest test is trying to delete an server that has not been completed spawning yet | |
| 12:06:01 | sean-k-mooney | what failing is recordign the delete event on the instancee because the instnace does not currently exist in the instnace table | |
| 12:06:23 | sean-k-mooney | that is causing the foreign key to fail | |
| 12:06:47 | sean-k-mooney | so this is not an issue with the db schema | |
| 12:07:44 | sean-k-mooney | whats broken here is that we shoudl gracefuly handle that issue and either reject the delete at the api or complete the delete without an error | |
| 12:10:56 | sean-k-mooney | neither are trivial to do correctly. we likely do no want to reject the request as a user should always be able to delete an insance for billing reasons | |
| 12:11:19 | sean-k-mooney | so while a 409 conflict could be valid in other case it woudl be undesireable in this case | |
| 12:11:58 | sean-k-mooney | the simplest possibel fix in this case would be ot catch the excption to record the instance action and log/ignore it then proceeed with the delete | |
| 12:13:29 | sean-k-mooney | that would be a valid resolution to the bug but if the instance does not exist in the cell db it also means we cant mark it as deleted which means we have no way to interup the request to spawn it. | |