| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-03-26 | |||
| 09:15:49 | opendevreview | ribaudr proposed openstack/nova master: FUP Remove unnecessary PCI check https://review.opendev.org/c/openstack/nova/+/944105 | |
| 09:15:50 | opendevreview | ribaudr proposed openstack/nova master: FUP Add a warning to make non-explicit live migration request debugging easier https://review.opendev.org/c/openstack/nova/+/944133 | |
| 09:15:50 | opendevreview | ribaudr proposed openstack/nova master: FUP improve comment accuracy and variable naming for tag removal https://review.opendev.org/c/openstack/nova/+/943124 | |
| 09:15:51 | opendevreview | ribaudr proposed openstack/nova master: FUP: Improve libvirt fixture for hostdevs https://review.opendev.org/c/openstack/nova/+/945232 | |
| 09:15:51 | opendevreview | ribaudr proposed openstack/nova master: FUP Update pci-passthrough and virtual-gpu documentation https://review.opendev.org/c/openstack/nova/+/944153 | |
| 09:15:52 | opendevreview | ribaudr proposed openstack/nova master: FUP improve and add integration tests for PCI SR-IOV servers https://review.opendev.org/c/openstack/nova/+/944106 | |
| 09:26:26 | opendevreview | huanhongda proposed openstack/nova master: [WIP]Only cleanup local deleted instance https://review.opendev.org/c/openstack/nova/+/945546 | |
| 10:16:22 | opendevreview | Stefan K proposed openstack/nova-specs master: Add Cloud Hypervisor support spec https://review.opendev.org/c/openstack/nova-specs/+/945549 | |
| 10:28:31 | opendevreview | Stefan K proposed openstack/nova-specs master: Add Cloud Hypervisor support spec https://review.opendev.org/c/openstack/nova-specs/+/945549 | |
| 10:40:41 | opendevreview | Stefan K proposed openstack/nova-specs master: Add Cloud Hypervisor support spec https://review.opendev.org/c/openstack/nova-specs/+/945549 | |
| 11:41:10 | opendevreview | Stefan K proposed openstack/nova-specs master: Add Cloud Hypervisor support spec https://review.opendev.org/c/openstack/nova-specs/+/945549 | |
| 12:37:47 | opendevreview | melanie witt proposed openstack/nova master: Call volume detach rollback API if detach fails https://review.opendev.org/c/openstack/nova/+/880399 | |
| 13:00:36 | simondodsley | elodilles thank you for the response. I read this that I can create the backports for the 3 patches for Caracal and then the Nova team can decide... Correct me if I am wrong | |
| 13:21:00 | elodilles | simondodsley: that sounds good to me | |
| 14:07:32 | opendevreview | Simon Dodsley proposed openstack/nova stable/2024.1: Fix device_type=lun with boot_index https://review.opendev.org/c/openstack/nova/+/945594 | |
| 14:08:27 | opendevreview | Simon Dodsley proposed openstack/nova stable/2024.1: Avoid setting serial on raw LUN devices https://review.opendev.org/c/openstack/nova/+/945595 | |
| 14:09:05 | opendevreview | Simon Dodsley proposed openstack/nova stable/2024.1: Enable virtio-scsi in nova-next https://review.opendev.org/c/openstack/nova/+/945596 | |
| 15:19:39 | zigo | Does anyone have a clue of what's going on? | |
| 15:19:39 | zigo | When attempting to launch tempest (to validate Epoxy), I'm getting this: https://paste.openstack.org/show/bAvPPkNaAQ5Ftw46O6xj/ | |
| 15:20:31 | dansmith | the only clue I have is that it looks entirely unrelated to nova | |
| 15:20:42 | dansmith | heat, tempest, keystone, maybe.. but not nova :) | |
| 15:21:02 | opendevreview | Rajesh Tailor proposed openstack/nova master: Fix disable memballoon device https://review.opendev.org/c/openstack/nova/+/945621 | |
| 15:25:18 | zigo | Looks like an issue in heat-tempest-plugin. | |
| 15:25:20 | zigo | Sorry ... :) | |
| 15:25:24 | zigo | Will ask over there. | |
| 15:25:32 | zigo | Removing the heat plugin fixed it. | |
| 15:43:24 | opendevreview | Dan Smith proposed openstack/nova master: Extend invalidate_rp to only invalidate cache https://review.opendev.org/c/openstack/nova/+/944148 | |
| 15:43:25 | opendevreview | Dan Smith proposed openstack/nova master: Support "one-time-use" PCI devices https://review.opendev.org/c/openstack/nova/+/943816 | |
| 15:43:25 | opendevreview | Dan Smith proposed openstack/nova master: Invalidate PCI-in-placement cached RPs during claim https://review.opendev.org/c/openstack/nova/+/944149 | |
| 15:43:26 | opendevreview | Dan Smith proposed openstack/nova master: DNM: Test nova with placement alloc fix https://review.opendev.org/c/openstack/nova/+/945626 | |
| 15:43:26 | opendevreview | Dan Smith proposed openstack/nova master: Add one-time-use devices docs and reno https://review.opendev.org/c/openstack/nova/+/944262 | |
| 16:59:04 | opendevreview | Rajesh Tailor proposed openstack/nova master: Add support for showing instance-action finish_time https://review.opendev.org/c/openstack/nova/+/928933 | |
| 18:34:52 | dansmith | sean-k-mooney: I guess the thing limiting us from testing pci passthrough upstream is the host iommu stuff, right? | |
| 18:35:01 | dansmith | I guess I'm not sure now if nested kvm is actually required or not | |
| 18:36:12 | sean-k-mooney | we can enable viommu to enable that | |
| 18:36:38 | sean-k-mooney | or rather | |
| 18:37:07 | sean-k-mooney | to be able to allocate device to second level vm the first level vm need a viommu | |
| 18:37:36 | sean-k-mooney | the l0 host does not need an iommu configured unless you are doing doubel passthough | |
| 18:37:47 | dansmith | so you mean the host (i.e. the cloud provider) would need to enable that for our instances, not that we could do that in our own images with grub | |
| 18:38:14 | sean-k-mooney | right we can enable it in the iamges we upload to them via nodepool | |
| 18:38:31 | dansmith | why are we not already doing that? | |
| 18:38:37 | sean-k-mooney | so we can turn on teh viommu in the ci image and enabel igb | |
| 18:38:41 | sean-k-mooney | it kills performance | |
| 18:38:49 | dansmith | ah | |
| 18:38:57 | sean-k-mooney | well if you actully use the iommu emulation | |
| 18:39:12 | sean-k-mooney | if its on and you do nothing im not sure what the overhead is | |
| 18:39:13 | dansmith | for regular jobs that don't use it at all would it matter? | |
| 18:39:17 | dansmith | okay | |
| 18:39:41 | dansmith | also, I've switched to using the virtio rng and it works very well and doesn't crash my first level guest each time.. any reason not to just use that for our test device? | |
| 18:40:07 | sean-k-mooney | none we just need a device that would not hard fault | |
| 18:40:21 | sean-k-mooney | if that work then we could do it today with our exisitng proviers | |
| 18:40:34 | dansmith | ack, the QXL kernel driver crashes in the first-level guest when it gets returned to it, but the rng seems to work fine | |
| 18:40:35 | sean-k-mooney | welll not teh xen rax nodes | |
| 18:40:39 | sean-k-mooney | but most of the others | |
| 18:41:12 | dansmith | okay maybe I'll see if that's doable | |
| 18:41:13 | sean-k-mooney | well i suspect the virtio-gpu might far better i used ot use virtio-blk device or virtio-net in the past | |
| 18:41:35 | sean-k-mooney | so what we need to so is add a new image and lable in node pool | |
| 18:41:51 | sean-k-mooney | then defien a new zuul nodeset that uses that and swawp say nova next to use that | |
| 18:42:07 | dansmith | why better than the rng? I just figure either of the other types are more likely to be complicated and more failure-prone | |
| 18:42:38 | sean-k-mooney | oh it was more i could add several neturon prots or cinder volumes if i needed more | |
| 18:42:57 | sean-k-mooney | but ya no reason not to use the rng | |
| 18:43:28 | sean-k-mooney | its mainlly needed to avoid slow boot up when there is low entropy | |
| 18:43:47 | sean-k-mooney | but once the first level vm is up the kernel hsould have enough to conitnue without it | |
| 18:44:19 | dansmith | oh I see, that would give us some extra devices I guess okay.. either way, if we get it all working it probably doesn't matter which we use and/or we could add aliases for all of them so jobs can configure their flavors | |
| 18:44:32 | sean-k-mooney | dansmith: so this is how you add image metadata vai node pool https://github.com/openstack/project-config/blob/master/nodepool/nb07.opendev.org.yaml#L26-L29 | |
| 18:45:31 | dansmith | ack | |
| 18:47:38 | sean-k-mooney | i belive our stadnar images are all defiend here https://github.com/openstack/project-config/blob/master/nodepool/nodepool.yaml#L191-L461 | |
| 18:47:48 | sean-k-mooney | the other wones were arm sepcific versions | |
| 18:48:40 | sean-k-mooney | i dont think we need to modify the image build jsut the upload to each cloud | |
| 18:48:50 | sean-k-mooney | or a cloud i guess to start with | |
| 18:50:13 | sean-k-mooney | dansmith: im sure you at least partly famiarly with this but the nodeset used for the jobs are typeiclly defined in devstack although we can define those in nova | |
| 18:50:41 | sean-k-mooney | we jsut need to refence the lable provided by node pool (the label is basicly and image an flavor mapped to each provider) | |
| 18:51:13 | sean-k-mooney | so you would basicly copy paste https://github.com/openstack/devstack/blob/master/.zuul.yaml#L154-L182 | |
| 18:51:46 | sean-k-mooney | and change label: ubuntu-noble ro label: ubuntu-noble-with-iommu unless we just update ubuntu-noble to turn it on | |
| 18:52:03 | sean-k-mooney | in which case you coudl skip that stpe but i woudl impact everyone | |
| 18:57:27 | dansmith | vaguely, I don't think I've ever had to mess with it directly, but thanks for the pointers | |
| 18:57:48 | dansmith | I wonder if we could ask the virt team if there's any harm in just enabling viommu if it's not used.. if so that would make it easy | |
| 19:06:19 | sean-k-mooney | so this is the chart i remember seeing before https://kvm-forum.qemu.org/2021/vIOMMU%20KVM%20Forum%202021%20-%20v4.pdf#page=16 | |
| 19:06:45 | sean-k-mooney | not that really talkign about the affect of enabling the viommu on a pci device that is passhth though to the guest | |
| 19:06:59 | sean-k-mooney | be it an ssd or a nic | |
| 19:07:32 | sean-k-mooney | and we can see the overhead is a 60-80% performance loss | |
| 19:07:57 | sean-k-mooney | but i dont know if we will see any performance hit for virual device like virtio-net or virito-blk | |
| 19:08:10 | sean-k-mooney | as such in ci it might not matter | |
| 19:09:15 | sean-k-mooney | that also with kernel 5.12, 6.14 just came out so this has proably impvoed over the last 4 years | |
| 19:10:37 | sean-k-mooney | the qemu they were using to test does not evne meet our current min requirements for nova and we are not exactly agressive with moving that forward | |
| 19:11:49 | sean-k-mooney | i think it would be wroth just trying it locally with some basic test with iperf and fio | |
| 19:12:29 | sean-k-mooney | and if you dont see any horible impact we could talk to the infra folks about trying it on one image in say the vexhost cloud or ovh adn see how it goes | |
| 19:19:45 | dansmith | ack, I should be able to do that.. maybe just tweaking the pool we run nova-next on or something and seeing if it looks to be performing the same for a week would be enough | |
| 19:21:09 | sean-k-mooney | yep | |
| 19:28:45 | dansmith | sean-k-mooney: so, I thought the iommu changes I was making in my first-level guest were to share/expose the actual iommu to the guest, | |
| 19:29:07 | sean-k-mooney | no | |
| 19:29:08 | dansmith | but reading danpb's post I'm thinking what I've been using is actually just the viommu stuff that nova would be enabling | |
| 19:29:17 | sean-k-mooney | yes | |
| 19:29:34 | dansmith | don't mind me, first day on the job people! | |
| 19:29:35 | sean-k-mooney | nova and qemu does not care if the host has an iommu or even if its enabeld | |
| 19:29:43 | dansmith | ack | |
| 19:30:08 | dansmith | so, also from reading danpb's post it seems like the non-passthrough devices should be unchanged with it enabled, but I'll do some testing to make sure | |
| 19:30:37 | sean-k-mooney | ack, that what i was expecting/hoping but i was unsure | |
| 19:31:43 | sean-k-mooney | the iommu can levage some host hardware accalration if its supproted but like vtpm its a purly vritual impelmetion by default | |
| 19:32:42 | sean-k-mooney | if i rememebr correctly we do enable caching by defualt https://libvirt.org/formatdomain.html#iommu-devices | |