Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2024-03-27
10:00:05 bauzas nevermind, looked too quickly
16:08:41 melwitt dansmith: thanks for looking at the spec. I haven't gone through it yet in depth but wanted to mention it might be easier to read(at least the tables) to look at the docs preview https://793e4b83b6aee9c90173-a0541882d2023eca9a1cc07087449de0.ssl.cf2.rackcdn.com/907654/7/check/openstack-tox-docs/55aebd2/docs/specs/2024.2/approved/ephemeral-storage-encryption.html
16:09:09 dansmith melwitt: oh yeah, that was the *only* way I could make sense of it :D
16:09:30 dansmith I was remarking about choosing which line to complain about in the source :)
16:09:40 melwitt oh, gotcha
16:15:03 melwitt sean-k-mooney: just a heads up that I did go through and test the systemd way of enabling gpu virtual functions and it worked for me https://review.opendev.org/c/openstack/nova/+/910041
16:25:56 bauzas melwitt: I can shortly quick-approve back the persistent mdevs bp
16:26:03 bauzas so I could look at your patch
16:26:27 melwitt bauzas: ok, that would be great, thanks
16:26:28 bauzas we agreed last cycle it was a specless bp, so I feel brave enough to approve it again without asking other folks
16:27:30 melwitt specless bp ftw
16:27:47 bauzas done
16:31:44 melwitt thanks
16:31:55 bauzas melwitt: can you confirm that besides persisting the sriov-manage VFs, all the mdevs were persisted after reboot without modifying some systemd files or udev rules ?
16:32:49 bauzas melwitt: I have concern with documenting some specific nvidia command in our upstream docs, so I'll just leave a -1 about that tomorrow, but I just wanted to make sure it works
16:35:12 melwitt bauzas: yes it worked for me. to be clear, I could boot a vm with vgpu, reboot the host, sriov-manage -e (via systemd service file), then 'server start' the vm and it retained the mdev in its xml and started up fine
16:35:20 bauzas \o/
16:40:52 bauzas melwitt: -1d with my comment https://review.opendev.org/c/openstack/nova/+/910041
16:41:55 melwitt bauzas: ok thanks
16:44:07 bauzas melwitt: if you read the nvidia doc link I gave in the review, you'll see this paragraph :
16:44:09 bauzas " Note: If you are using a GPU that supports SR-IOV, the mdev device file persists after a host reboot only if you perform Step 1 before rebooting any VM that is configured with a vGPU on the
16:44:11 bauzas GPU. You cannot use the mdevctl command to make the mdev device file for a MIG-backed vGPU persistent. The mdev device file for a MIG-backed vGPU is not retained after the host is rebooted because MIG
16:44:13 bauzas instances are no longer available. "
16:44:32 bauzas tl;dr: this works for time-sliced vGPUs but if you enable MIG types, you're screwed
16:44:50 dansmith wow
16:45:17 bauzas because all of this is fuckingly on the userspacez
16:45:21 bauzas thanks nvidia
16:48:07 melwitt bauzas: oh, I see. I don't know what MIG types means but I'll look it up :)
16:48:46 bauzas melwitt: that's another mechanism to slice your GPU, this time using capable hardware resources
16:49:18 bauzas eg. a A100 can be sliced either by concurrency (time-sliced types) or physically (using MIG graphical instances)
16:49:29 melwitt ah ok
16:54:33 bauzas MIGs are only available with another "new" licence, which isn't particularly cheap 'AI Enterprise'
16:54:45 bauzas that's why you didn't see it :)
16:54:52 bauzas seen*
16:56:12 melwitt AI $$$
16:56:50 bauzas well, you indirectly support nvidia to become a new GAFAM
16:58:07 bauzas (which indirectly helps me, since I have a position on the nasdaq index :) )
17:21:36 opendevreview Stephen Finucane proposed openstack/nova-specs master: Add openapi spec https://review.opendev.org/c/openstack/nova-specs/+/909448
17:22:33 opendevreview Stephen Finucane proposed openstack/nova-specs master: Add openapi spec https://review.opendev.org/c/openstack/nova-specs/+/909448
18:32:06 opendevreview Merged openstack/nova master: Fix nova-manage image_property show unexpected keyword https://review.opendev.org/c/openstack/nova/+/880557
19:15:02 melwitt dansmith: I replied to your comments on the spec except the one about glance + secrets. I had thought I read something in docs or specs related to that at some point but I can't find it. I wanted to look around a little more before replying on that so I posted everything else in the meantime
19:16:39 dansmith oh yeah, it's a thing, but the link is broken
19:17:00 dansmith it's also completely unmerged and still needs some discussion I think.. not sure how much review it has really gotten
19:17:04 dansmith it too has been pending for many cycles
19:17:23 dansmith I reached out to those folks this morning and I think we'll get a good cross-project session on it at PTG
19:17:47 dansmith it's quite different from what you have proposed here too, AFAICT from a skim
19:18:14 melwitt yeah, that paragraph has been in there forever so I'm not sure what happened to the link
19:18:51 melwitt ok that sounds good re: PTG
19:18:58 dansmith potentially a presumptive link that never became a 200 because it was never merged, or perhaps a de-approval
#openstack-nova - 2024-03-28
10:30:13 opendevreview Stephen Finucane proposed openstack/nova-specs master: Add openapi spec https://review.opendev.org/c/openstack/nova-specs/+/909448
15:22:45 opendevreview Rajesh Tailor proposed openstack/nova master: Add regression test case for bug 1978573 https://review.opendev.org/c/openstack/nova/+/914653
16:41:54 ygk_12345 anyone online here ?
17:09:49 auniyal hey ygk_12345, you can just ask
20:16:05 opendevreview Artom Lifshitz proposed openstack/nova master: DNM/POC: demonstrate bug with NUMA-aware vswitches https://review.opendev.org/c/openstack/nova/+/914712
20:16:35 opendevreview Artom Lifshitz proposed openstack/nova master: DNM/POC: demonstrate bug with NUMA-aware vswitches https://review.opendev.org/c/openstack/nova/+/914712
#openstack-nova - 2024-03-29
04:06:36 opendevreview Jang YeHoon proposed openstack/nova master: Modify the auth_url field in the service_user section https://review.opendev.org/c/openstack/nova/+/914743
15:26:03 ozzzo We're rebuilding our kolla-ansible clusters from Train to Wallaby, and on the rebuilt clusters VM builds take longer. On a Train cluster, I see a 2-second delay in the nova log after "Creating image"; on Wallaby clusters I see a 9-second delay.
15:26:47 ozzzo I suspect that the hypervisor may be redownloading the image for every build. Is there a configuration item that controls whether the hypervisor saves the image locally so that it doesn't have to be redownloaded?
15:36:50 dansmith using qcow images?
15:37:04 dansmith or rather, qcow for instance disks I mean
15:37:52 dansmith ozzzo: ^
15:38:17 ozzzo I think so, lemme look at an image
15:39:23 ozzzo yes disk_format is qcow2
15:39:26 dansmith it's not the image,
15:39:31 dansmith it's the instance disk
15:39:48 ozzzo how can I check that?
15:40:22 dansmith if you don't have any instance disk-related config (which I presume you know would about) then you're using qcow disks
15:40:48 dansmith in that case, you kinda have to be not re-downloading things I think because all the disks are using the same backing file anyway
15:41:20 dansmith you can run `qemu-img info` on disks in /var/lib/instances and they should all be pointing the backing file to the same thing in _base if they're using the same image
15:42:09 ozzzo ok I'll try that
15:42:35 dansmith better to just look at the logs (perhaps the glance log) to see if one compute is downloading the same image though
15:42:48 dansmith because regardless, that's the thing you're concerned about
15:43:55 dansmith presumably preallocate_images is not set in your config, that could be another thing
16:01:55 ozzzo Do I need to turn on debug logging to see the compute downloading the image?
16:06:31 dansmith probably? I mean, depends on how it's setup
16:06:44 dansmith I always have debug logging turned on so I don't even know what is logged without it :)
16:07:08 dansmith you should be able to see all the previous attempts if it's logging enough
16:07:43 dansmith I don't have my running system handy but the request logging you're looking for is paths that look like /images/UUID/data
17:19:11 opendevreview Takashi Kajinami proposed openstack/nova master: Retry image download when 500 Internal Server Error is received https://review.opendev.org/c/openstack/nova/+/914761
18:55:05 opendevreview melanie witt proposed openstack/nova stable/2024.1: Fix nova-manage image_property show unexpected keyword https://review.opendev.org/c/openstack/nova/+/914671
18:57:25 opendevreview melanie witt proposed openstack/nova stable/2023.2: Fix nova-manage image_property show unexpected keyword https://review.opendev.org/c/openstack/nova/+/914765
18:58:25 opendevreview melanie witt proposed openstack/nova stable/2023.1: Fix nova-manage image_property show unexpected keyword https://review.opendev.org/c/openstack/nova/+/914766
18:59:24 opendevreview melanie witt proposed openstack/nova stable/zed: Fix nova-manage image_property show unexpected keyword https://review.opendev.org/c/openstack/nova/+/914767
#openstack-nova - 2024-03-30
02:59:26 opendevreview Takashi Kajinami proposed openstack/nova master: Retry image download when 500 Internal Server Error is received https://review.opendev.org/c/openstack/nova/+/914761
03:11:17 opendevreview Takashi Kajinami proposed openstack/nova master: Retry image download when 500 Internal Server Error is received https://review.opendev.org/c/openstack/nova/+/914761
#openstack-nova - 2024-04-01
13:14:29 opendevreview Rajesh Tailor proposed openstack/nova master: Restore original AZ when unshelve fails https://review.opendev.org/c/openstack/nova/+/911108
#openstack-nova - 2024-04-02
06:39:11 gibi o/
09:56:47 gibi bauzas: did you intentionally set the nova meeting time to 4th of April instead of 2nd of April?
09:56:51 gibi https://wiki.openstack.org/wiki/Meetings/Nova
10:05:07 bauzas gibi: sorry nope, miscalculated it ;-)
10:05:29 bauzas Still on Tuesday 4pm UTC
10:09:03 gibi bauzas: ack, thanks for confirming
14:53:10 bauzas folks (in Europe), as a reminder, the nova meeting will still be at 1600UTC, which means now (after the daylight change this weekend) that it will be in ~1hour
15:57:23 bauzas shit, I'm on a meeting now so I forgot to do the meeting agenda
16:00:31 opendevmeet The meeting name has been set to 'nova'
16:00:31 opendevmeet Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:31 opendevmeet Meeting started Tue Apr 2 16:00:31 2024 UTC and is due to finish in 60 minutes. The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:31 bauzas #startmeeting nova
16:00:35 bauzas hey folks
16:00:37 elodilles o/

Earlier   Later