| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-03-26 | |||
| 18:37:47 | dansmith | so you mean the host (i.e. the cloud provider) would need to enable that for our instances, not that we could do that in our own images with grub | |
| 18:38:14 | sean-k-mooney | right we can enable it in the iamges we upload to them via nodepool | |
| 18:38:31 | dansmith | why are we not already doing that? | |
| 18:38:37 | sean-k-mooney | so we can turn on teh viommu in the ci image and enabel igb | |
| 18:38:41 | sean-k-mooney | it kills performance | |
| 18:38:49 | dansmith | ah | |
| 18:38:57 | sean-k-mooney | well if you actully use the iommu emulation | |
| 18:39:12 | sean-k-mooney | if its on and you do nothing im not sure what the overhead is | |
| 18:39:13 | dansmith | for regular jobs that don't use it at all would it matter? | |
| 18:39:17 | dansmith | okay | |
| 18:39:41 | dansmith | also, I've switched to using the virtio rng and it works very well and doesn't crash my first level guest each time.. any reason not to just use that for our test device? | |
| 18:40:07 | sean-k-mooney | none we just need a device that would not hard fault | |
| 18:40:21 | sean-k-mooney | if that work then we could do it today with our exisitng proviers | |
| 18:40:34 | dansmith | ack, the QXL kernel driver crashes in the first-level guest when it gets returned to it, but the rng seems to work fine | |
| 18:40:35 | sean-k-mooney | welll not teh xen rax nodes | |
| 18:40:39 | sean-k-mooney | but most of the others | |
| 18:41:12 | dansmith | okay maybe I'll see if that's doable | |
| 18:41:13 | sean-k-mooney | well i suspect the virtio-gpu might far better i used ot use virtio-blk device or virtio-net in the past | |
| 18:41:35 | sean-k-mooney | so what we need to so is add a new image and lable in node pool | |
| 18:41:51 | sean-k-mooney | then defien a new zuul nodeset that uses that and swawp say nova next to use that | |
| 18:42:07 | dansmith | why better than the rng? I just figure either of the other types are more likely to be complicated and more failure-prone | |
| 18:42:38 | sean-k-mooney | oh it was more i could add several neturon prots or cinder volumes if i needed more | |
| 18:42:57 | sean-k-mooney | but ya no reason not to use the rng | |
| 18:43:28 | sean-k-mooney | its mainlly needed to avoid slow boot up when there is low entropy | |
| 18:43:47 | sean-k-mooney | but once the first level vm is up the kernel hsould have enough to conitnue without it | |
| 18:44:19 | dansmith | oh I see, that would give us some extra devices I guess okay.. either way, if we get it all working it probably doesn't matter which we use and/or we could add aliases for all of them so jobs can configure their flavors | |
| 18:44:32 | sean-k-mooney | dansmith: so this is how you add image metadata vai node pool https://github.com/openstack/project-config/blob/master/nodepool/nb07.opendev.org.yaml#L26-L29 | |
| 18:45:31 | dansmith | ack | |
| 18:47:38 | sean-k-mooney | i belive our stadnar images are all defiend here https://github.com/openstack/project-config/blob/master/nodepool/nodepool.yaml#L191-L461 | |
| 18:47:48 | sean-k-mooney | the other wones were arm sepcific versions | |
| 18:48:40 | sean-k-mooney | i dont think we need to modify the image build jsut the upload to each cloud | |
| 18:48:50 | sean-k-mooney | or a cloud i guess to start with | |
| 18:50:13 | sean-k-mooney | dansmith: im sure you at least partly famiarly with this but the nodeset used for the jobs are typeiclly defined in devstack although we can define those in nova | |
| 18:50:41 | sean-k-mooney | we jsut need to refence the lable provided by node pool (the label is basicly and image an flavor mapped to each provider) | |
| 18:51:13 | sean-k-mooney | so you would basicly copy paste https://github.com/openstack/devstack/blob/master/.zuul.yaml#L154-L182 | |
| 18:51:46 | sean-k-mooney | and change label: ubuntu-noble ro label: ubuntu-noble-with-iommu unless we just update ubuntu-noble to turn it on | |
| 18:52:03 | sean-k-mooney | in which case you coudl skip that stpe but i woudl impact everyone | |
| 18:57:27 | dansmith | vaguely, I don't think I've ever had to mess with it directly, but thanks for the pointers | |
| 18:57:48 | dansmith | I wonder if we could ask the virt team if there's any harm in just enabling viommu if it's not used.. if so that would make it easy | |
| 19:06:19 | sean-k-mooney | so this is the chart i remember seeing before https://kvm-forum.qemu.org/2021/vIOMMU%20KVM%20Forum%202021%20-%20v4.pdf#page=16 | |
| 19:06:45 | sean-k-mooney | not that really talkign about the affect of enabling the viommu on a pci device that is passhth though to the guest | |
| 19:06:59 | sean-k-mooney | be it an ssd or a nic | |
| 19:07:32 | sean-k-mooney | and we can see the overhead is a 60-80% performance loss | |
| 19:07:57 | sean-k-mooney | but i dont know if we will see any performance hit for virual device like virtio-net or virito-blk | |
| 19:08:10 | sean-k-mooney | as such in ci it might not matter | |
| 19:09:15 | sean-k-mooney | that also with kernel 5.12, 6.14 just came out so this has proably impvoed over the last 4 years | |
| 19:10:37 | sean-k-mooney | the qemu they were using to test does not evne meet our current min requirements for nova and we are not exactly agressive with moving that forward | |
| 19:11:49 | sean-k-mooney | i think it would be wroth just trying it locally with some basic test with iperf and fio | |
| 19:12:29 | sean-k-mooney | and if you dont see any horible impact we could talk to the infra folks about trying it on one image in say the vexhost cloud or ovh adn see how it goes | |
| 19:19:45 | dansmith | ack, I should be able to do that.. maybe just tweaking the pool we run nova-next on or something and seeing if it looks to be performing the same for a week would be enough | |
| 19:21:09 | sean-k-mooney | yep | |
| 19:28:45 | dansmith | sean-k-mooney: so, I thought the iommu changes I was making in my first-level guest were to share/expose the actual iommu to the guest, | |
| 19:29:07 | sean-k-mooney | no | |
| 19:29:08 | dansmith | but reading danpb's post I'm thinking what I've been using is actually just the viommu stuff that nova would be enabling | |
| 19:29:17 | sean-k-mooney | yes | |
| 19:29:34 | dansmith | don't mind me, first day on the job people! | |
| 19:29:35 | sean-k-mooney | nova and qemu does not care if the host has an iommu or even if its enabeld | |
| 19:29:43 | dansmith | ack | |
| 19:30:08 | dansmith | so, also from reading danpb's post it seems like the non-passthrough devices should be unchanged with it enabled, but I'll do some testing to make sure | |
| 19:30:37 | sean-k-mooney | ack, that what i was expecting/hoping but i was unsure | |
| 19:31:43 | sean-k-mooney | the iommu can levage some host hardware accalration if its supproted but like vtpm its a purly vritual impelmetion by default | |
| 19:32:42 | sean-k-mooney | if i rememebr correctly we do enable caching by defualt https://libvirt.org/formatdomain.html#iommu-devices | |
| 19:34:57 | sean-k-mooney | ya we do https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L7987-L8019 | |
| 19:35:33 | sean-k-mooney | so if the host support intel vt-d it can use it | |
| #openstack-nova - 2025-03-27 | |||
| 08:24:51 | opendevreview | Rajesh Tailor proposed openstack/nova master: Fix disable memballoon device https://review.opendev.org/c/openstack/nova/+/945621 | |
| 08:30:24 | opendevreview | Stefan K proposed openstack/nova-specs master: Add Cloud Hypervisor support spec https://review.opendev.org/c/openstack/nova-specs/+/945549 | |
| 10:34:11 | opendevreview | Rajesh Tailor proposed openstack/nova master: Fix disable memballoon device https://review.opendev.org/c/openstack/nova/+/945621 | |
| 11:14:53 | opendevreview | Merged openstack/nova stable/2023.2: Reproducer for bug 2098892 https://review.opendev.org/c/openstack/nova/+/942655 | |
| 11:19:04 | opendevreview | Merged openstack/nova stable/2023.2: libvirt: Fix regression of listDevices() return type https://review.opendev.org/c/openstack/nova/+/942656 | |
| 12:12:53 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Adjust validation helpers for a single-method future https://review.opendev.org/c/openstack/nova/+/936365 | |
| 12:12:54 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Stop using wsgi.Controller.api_version to switch between API versions https://review.opendev.org/c/openstack/nova/+/936366 | |
| 12:12:54 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Only check minimum API version https://review.opendev.org/c/openstack/nova/+/940325 | |
| 12:12:55 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add new, simpler api_version decorator https://review.opendev.org/c/openstack/nova/+/936367 | |
| 12:12:55 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Only run format checks on strings https://review.opendev.org/c/openstack/nova/+/936368 | |
| 12:12:56 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Simplify parameter types https://review.opendev.org/c/openstack/nova/+/936369 | |
| 12:12:56 | opendevreview | Stephen Finucane proposed openstack/nova master: tests: Ensure all APIs have a response body schema https://review.opendev.org/c/openstack/nova/+/924598 | |
| 12:12:57 | opendevreview | Stephen Finucane proposed openstack/nova master: doc: Add missing API samples https://review.opendev.org/c/openstack/nova/+/937012 | |
| 12:12:57 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for hosts APIs https://review.opendev.org/c/openstack/nova/+/937047 | |
| 12:12:59 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for instance actions https://review.opendev.org/c/openstack/nova/+/937048 | |
| 12:12:59 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for hypervisors APIs (1/3) https://review.opendev.org/c/openstack/nova/+/937245 | |
| 12:13:01 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for hypervisors APIs (2/3) https://review.opendev.org/c/openstack/nova/+/937246 | |
| 12:13:01 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for hypervisors APIs (3/3) https://review.opendev.org/c/openstack/nova/+/937247 | |
| 12:13:03 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for server IPs APIs https://review.opendev.org/c/openstack/nova/+/937608 | |
| 12:13:03 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for remote consoles https://review.opendev.org/c/openstack/nova/+/940114 | |
| 12:13:05 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for keypairs APIs https://review.opendev.org/c/openstack/nova/+/940115 | |
| 12:13:05 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for image metadata APIs https://review.opendev.org/c/openstack/nova/+/940299 | |
| 12:13:07 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for server topology API https://review.opendev.org/c/openstack/nova/+/940300 | |
| 12:13:07 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for server diagnostics API https://review.opendev.org/c/openstack/nova/+/940506 | |
| 12:13:09 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Add response body schemas for server external events API https://review.opendev.org/c/openstack/nova/+/940508 | |
| 12:18:40 | opendevreview | Merged openstack/nova master: FUP Remove unnecessary PCI check https://review.opendev.org/c/openstack/nova/+/944105 | |
| 12:24:02 | opendevreview | Merged openstack/nova master: FUP improve comment accuracy and variable naming for tag removal https://review.opendev.org/c/openstack/nova/+/943124 | |
| 12:35:18 | opendevreview | Merged openstack/nova master: FUP Add a warning to make non-explicit live migration request debugging easier https://review.opendev.org/c/openstack/nova/+/944133 | |
| 12:35:24 | opendevreview | Merged openstack/nova master: FUP Update pci-passthrough and virtual-gpu documentation https://review.opendev.org/c/openstack/nova/+/944153 | |
| 12:35:30 | opendevreview | Merged openstack/nova master: FUP: Improve libvirt fixture for hostdevs https://review.opendev.org/c/openstack/nova/+/945232 | |
| 14:19:33 | Vii | Hello. Openstack 2024.2, I shelved the instance (I don't know if it's important on the host that was "compute service set --disable [hostname] nova-compute"). Now I can't unshelve this instance. There's an error in the nova logs: https://paste.openstack.org/show/bshRPNzpjA1ckFkGoa21/ does anyone have any ideas where the error might be? | |
| 14:20:46 | dansmith | Vii: it's timing out talking to neutron | |
| 15:09:22 | opendevreview | ribaudr proposed openstack/nova master: FUP improve and add integration tests for PCI SR-IOV servers https://review.opendev.org/c/openstack/nova/+/944106 | |
| 15:55:57 | opendevreview | Dan Smith proposed openstack/nova master: Extend invalidate_rp to only invalidate cache https://review.opendev.org/c/openstack/nova/+/944148 | |
| 15:55:57 | opendevreview | Dan Smith proposed openstack/nova master: Invalidate PCI-in-placement cached RPs during claim https://review.opendev.org/c/openstack/nova/+/944149 | |
| 15:55:58 | opendevreview | Dan Smith proposed openstack/nova master: Support "one-time-use" PCI devices https://review.opendev.org/c/openstack/nova/+/943816 | |