| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-03-26 | |||
| 21:27:23 | sean-k-mooney | i should be enabling the db mocks here but im not calling the db | |
| 21:27:30 | sean-k-mooney | so its not need yet | |
| 21:27:41 | sean-k-mooney | the second it would call the db we need to include the db fixture | |
| 22:00:53 | melwitt | sean-k-mooney: nice, thanks | |
| #openstack-nova - 2024-03-27 | |||
| 05:46:47 | manuvakery1 | Hey .. I am getting ConflictNovaUsingAttachment while dettaching volume from VM. I do have the service_user enabled in nova.conf as per https://docs.openstack.org/cinder/latest/configuration/block-storage/service-token.html | |
| 07:30:38 | sahid | bauzas, sean-k-mooney o/ | |
| 07:30:51 | sahid | all good for https://review.opendev.org/c/openstack/nova/+/896512 ? | |
| 07:30:57 | sahid | any action that I have to take? | |
| 07:34:05 | bauzas | sahid : the action is for me, I need to modify the blueprint to be accepted and add the series in our etherpad :-) | |
| 07:50:19 | sahid | bauzas: perfect ! thanks | |
| 08:01:02 | sahid | guys, quick question, we have noticed that some VMs that are using volumes do not have the right number of iscsi path, normally they should all have 8 paths | |
| 08:01:18 | sahid | but we can notice that some have only 4 or 3 or 7 paths | |
| 08:33:31 | opendevreview | ribaudr proposed openstack/nova master: Test live migration between hosts with differnet cpu_shared_sets https://review.opendev.org/c/openstack/nova/+/913744 | |
| 08:33:31 | opendevreview | ribaudr proposed openstack/nova master: Fix: Live migrating to a host with cpu_shared_set configured will now update the VM's configuration accordingly. https://review.opendev.org/c/openstack/nova/+/903706 | |
| 09:59:42 | bauzas | sahid: I'm not a specialist but I think it depends on the machine type | |
| 10:00:05 | bauzas | nevermind, looked too quickly | |
| 16:08:41 | melwitt | dansmith: thanks for looking at the spec. I haven't gone through it yet in depth but wanted to mention it might be easier to read(at least the tables) to look at the docs preview https://793e4b83b6aee9c90173-a0541882d2023eca9a1cc07087449de0.ssl.cf2.rackcdn.com/907654/7/check/openstack-tox-docs/55aebd2/docs/specs/2024.2/approved/ephemeral-storage-encryption.html | |
| 16:09:09 | dansmith | melwitt: oh yeah, that was the *only* way I could make sense of it :D | |
| 16:09:30 | dansmith | I was remarking about choosing which line to complain about in the source :) | |
| 16:09:40 | melwitt | oh, gotcha | |
| 16:15:03 | melwitt | sean-k-mooney: just a heads up that I did go through and test the systemd way of enabling gpu virtual functions and it worked for me https://review.opendev.org/c/openstack/nova/+/910041 | |
| 16:25:56 | bauzas | melwitt: I can shortly quick-approve back the persistent mdevs bp | |
| 16:26:03 | bauzas | so I could look at your patch | |
| 16:26:27 | melwitt | bauzas: ok, that would be great, thanks | |
| 16:26:28 | bauzas | we agreed last cycle it was a specless bp, so I feel brave enough to approve it again without asking other folks | |
| 16:27:30 | melwitt | specless bp ftw | |
| 16:27:47 | bauzas | done | |
| 16:31:44 | melwitt | thanks | |
| 16:31:55 | bauzas | melwitt: can you confirm that besides persisting the sriov-manage VFs, all the mdevs were persisted after reboot without modifying some systemd files or udev rules ? | |
| 16:32:49 | bauzas | melwitt: I have concern with documenting some specific nvidia command in our upstream docs, so I'll just leave a -1 about that tomorrow, but I just wanted to make sure it works | |
| 16:35:12 | melwitt | bauzas: yes it worked for me. to be clear, I could boot a vm with vgpu, reboot the host, sriov-manage -e (via systemd service file), then 'server start' the vm and it retained the mdev in its xml and started up fine | |
| 16:35:20 | bauzas | \o/ | |
| 16:40:52 | bauzas | melwitt: -1d with my comment https://review.opendev.org/c/openstack/nova/+/910041 | |
| 16:41:55 | melwitt | bauzas: ok thanks | |
| 16:44:07 | bauzas | melwitt: if you read the nvidia doc link I gave in the review, you'll see this paragraph : | |
| 16:44:09 | bauzas | " Note: If you are using a GPU that supports SR-IOV, the mdev device file persists after a host reboot only if you perform Step 1 before rebooting any VM that is configured with a vGPU on the | |
| 16:44:11 | bauzas | GPU. You cannot use the mdevctl command to make the mdev device file for a MIG-backed vGPU persistent. The mdev device file for a MIG-backed vGPU is not retained after the host is rebooted because MIG | |
| 16:44:13 | bauzas | instances are no longer available. " | |
| 16:44:32 | bauzas | tl;dr: this works for time-sliced vGPUs but if you enable MIG types, you're screwed | |
| 16:44:50 | dansmith | wow | |
| 16:45:17 | bauzas | because all of this is fuckingly on the userspacez | |
| 16:45:21 | bauzas | thanks nvidia | |
| 16:48:07 | melwitt | bauzas: oh, I see. I don't know what MIG types means but I'll look it up :) | |
| 16:48:46 | bauzas | melwitt: that's another mechanism to slice your GPU, this time using capable hardware resources | |
| 16:49:18 | bauzas | eg. a A100 can be sliced either by concurrency (time-sliced types) or physically (using MIG graphical instances) | |
| 16:49:29 | melwitt | ah ok | |
| 16:54:33 | bauzas | MIGs are only available with another "new" licence, which isn't particularly cheap 'AI Enterprise' | |
| 16:54:45 | bauzas | that's why you didn't see it :) | |
| 16:54:52 | bauzas | seen* | |
| 16:56:12 | melwitt | AI $$$ | |
| 16:56:50 | bauzas | well, you indirectly support nvidia to become a new GAFAM | |
| 16:58:07 | bauzas | (which indirectly helps me, since I have a position on the nasdaq index :) ) | |
| 17:21:36 | opendevreview | Stephen Finucane proposed openstack/nova-specs master: Add openapi spec https://review.opendev.org/c/openstack/nova-specs/+/909448 | |
| 17:22:33 | opendevreview | Stephen Finucane proposed openstack/nova-specs master: Add openapi spec https://review.opendev.org/c/openstack/nova-specs/+/909448 | |
| 18:32:06 | opendevreview | Merged openstack/nova master: Fix nova-manage image_property show unexpected keyword https://review.opendev.org/c/openstack/nova/+/880557 | |
| 19:15:02 | melwitt | dansmith: I replied to your comments on the spec except the one about glance + secrets. I had thought I read something in docs or specs related to that at some point but I can't find it. I wanted to look around a little more before replying on that so I posted everything else in the meantime | |
| 19:16:39 | dansmith | oh yeah, it's a thing, but the link is broken | |
| 19:17:00 | dansmith | it's also completely unmerged and still needs some discussion I think.. not sure how much review it has really gotten | |
| 19:17:04 | dansmith | it too has been pending for many cycles | |
| 19:17:23 | dansmith | I reached out to those folks this morning and I think we'll get a good cross-project session on it at PTG | |
| 19:17:47 | dansmith | it's quite different from what you have proposed here too, AFAICT from a skim | |
| 19:18:14 | melwitt | yeah, that paragraph has been in there forever so I'm not sure what happened to the link | |
| 19:18:51 | melwitt | ok that sounds good re: PTG | |
| 19:18:58 | dansmith | potentially a presumptive link that never became a 200 because it was never merged, or perhaps a de-approval | |
| #openstack-nova - 2024-03-28 | |||
| 10:30:13 | opendevreview | Stephen Finucane proposed openstack/nova-specs master: Add openapi spec https://review.opendev.org/c/openstack/nova-specs/+/909448 | |
| 15:22:45 | opendevreview | Rajesh Tailor proposed openstack/nova master: Add regression test case for bug 1978573 https://review.opendev.org/c/openstack/nova/+/914653 | |
| 16:41:54 | ygk_12345 | anyone online here ? | |
| 17:09:49 | auniyal | hey ygk_12345, you can just ask | |
| 20:16:05 | opendevreview | Artom Lifshitz proposed openstack/nova master: DNM/POC: demonstrate bug with NUMA-aware vswitches https://review.opendev.org/c/openstack/nova/+/914712 | |
| 20:16:35 | opendevreview | Artom Lifshitz proposed openstack/nova master: DNM/POC: demonstrate bug with NUMA-aware vswitches https://review.opendev.org/c/openstack/nova/+/914712 | |
| #openstack-nova - 2024-03-29 | |||
| 04:06:36 | opendevreview | Jang YeHoon proposed openstack/nova master: Modify the auth_url field in the service_user section https://review.opendev.org/c/openstack/nova/+/914743 | |
| 15:26:03 | ozzzo | We're rebuilding our kolla-ansible clusters from Train to Wallaby, and on the rebuilt clusters VM builds take longer. On a Train cluster, I see a 2-second delay in the nova log after "Creating image"; on Wallaby clusters I see a 9-second delay. | |
| 15:26:47 | ozzzo | I suspect that the hypervisor may be redownloading the image for every build. Is there a configuration item that controls whether the hypervisor saves the image locally so that it doesn't have to be redownloaded? | |
| 15:36:50 | dansmith | using qcow images? | |
| 15:37:04 | dansmith | or rather, qcow for instance disks I mean | |
| 15:37:52 | dansmith | ozzzo: ^ | |
| 15:38:17 | ozzzo | I think so, lemme look at an image | |
| 15:39:23 | ozzzo | yes disk_format is qcow2 | |
| 15:39:26 | dansmith | it's not the image, | |
| 15:39:31 | dansmith | it's the instance disk | |
| 15:39:48 | ozzzo | how can I check that? | |
| 15:40:22 | dansmith | if you don't have any instance disk-related config (which I presume you know would about) then you're using qcow disks | |
| 15:40:48 | dansmith | in that case, you kinda have to be not re-downloading things I think because all the disks are using the same backing file anyway | |
| 15:41:20 | dansmith | you can run `qemu-img info` on disks in /var/lib/instances and they should all be pointing the backing file to the same thing in _base if they're using the same image | |
| 15:42:09 | ozzzo | ok I'll try that | |
| 15:42:35 | dansmith | better to just look at the logs (perhaps the glance log) to see if one compute is downloading the same image though | |
| 15:42:48 | dansmith | because regardless, that's the thing you're concerned about | |
| 15:43:55 | dansmith | presumably preallocate_images is not set in your config, that could be another thing | |
| 16:01:55 | ozzzo | Do I need to turn on debug logging to see the compute downloading the image? | |
| 16:06:31 | dansmith | probably? I mean, depends on how it's setup | |
| 16:06:44 | dansmith | I always have debug logging turned on so I don't even know what is logged without it :) | |
| 16:07:08 | dansmith | you should be able to see all the previous attempts if it's logging enough | |
| 16:07:43 | dansmith | I don't have my running system handy but the request logging you're looking for is paths that look like /images/UUID/data | |
| 17:19:11 | opendevreview | Takashi Kajinami proposed openstack/nova master: Retry image download when 500 Internal Server Error is received https://review.opendev.org/c/openstack/nova/+/914761 | |
| 18:55:05 | opendevreview | melanie witt proposed openstack/nova stable/2024.1: Fix nova-manage image_property show unexpected keyword https://review.opendev.org/c/openstack/nova/+/914671 | |
| 18:57:25 | opendevreview | melanie witt proposed openstack/nova stable/2023.2: Fix nova-manage image_property show unexpected keyword https://review.opendev.org/c/openstack/nova/+/914765 | |
| 18:58:25 | opendevreview | melanie witt proposed openstack/nova stable/2023.1: Fix nova-manage image_property show unexpected keyword https://review.opendev.org/c/openstack/nova/+/914766 | |
| 18:59:24 | opendevreview | melanie witt proposed openstack/nova stable/zed: Fix nova-manage image_property show unexpected keyword https://review.opendev.org/c/openstack/nova/+/914767 | |
| #openstack-nova - 2024-03-30 | |||
| 02:59:26 | opendevreview | Takashi Kajinami proposed openstack/nova master: Retry image download when 500 Internal Server Error is received https://review.opendev.org/c/openstack/nova/+/914761 | |
| 03:11:17 | opendevreview | Takashi Kajinami proposed openstack/nova master: Retry image download when 500 Internal Server Error is received https://review.opendev.org/c/openstack/nova/+/914761 | |