| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-11-16 | |||
| 12:10:11 | dmitriis | gibi: tyvm for the feedback | |
| 12:10:50 | gibi | dmitriis: you are welcome. it is a well written spec, thanks for putting in the effort | |
| 12:22:45 | sean-k-mooney | i dont know if i set send on my respocne to the last verions of it | |
| 12:23:02 | sean-k-mooney | gibi you are correct the pci passhtoug filter will be suffiect without the prefilter | |
| 12:23:29 | sean-k-mooney | but the prefiletr can help reduce the set if we only report the new trait on host that have off path devices | |
| 12:23:47 | sean-k-mooney | and the capablity to use them of course | |
| 12:24:04 | sean-k-mooney | ill re review that specs later today | |
| 12:26:06 | gibi | sean-k-mooney: yes, exactly my argument, the prefilter is not mandatory but it is good to have | |
| 13:00:21 | dmitriis | sean-k-mooney: ack, ty for confirming | |
| 13:02:28 | sean-k-mooney | i dont currently have access to hardware to test what you have done but i may have access before the end of the cycle. if i do i might reach out to you and try and test it end to end although i dont know if i will have time to do that or not | |
| 13:24:48 | elodilles | bauzas: i'll update now the meeting wiki #stable section if that is not interfering with you right now | |
| 13:26:55 | dmitriis | sean-k-mooney: btw, fnordahl and I have done end-to-end testing of this in a lab. Here's a PPA https://launchpad.net/~fnordahl/+archive/ubuntu/smartnic-enablement that was used in the process (the WIP reviews are in use there). It has https://listman.redhat.com/archives/libvir-list/2021-November/msg00431.html included as well - I am trying to get | |
| 13:26:55 | dmitriis | someone to review it sooner than later. | |
| 13:28:20 | dmitriis | it doesn't yet have the prefilter and compute capability parts that were recently added to the spec but I will work on updating the WIP review soon with that and on raising a relevant os-traits change | |
| 13:30:44 | dmitriis | We had a VM booted with a floating IP assigned which we then connected to via a router. The flows were offloaded into the ConnectX-6 chip present on BF2. | |
| 13:31:52 | sean-k-mooney | i think i have ping that patch to people dowstream already but ill let the vert team know | |
| 13:31:57 | sean-k-mooney | *virt | |
| 13:32:11 | dmitriis | ack, tyvm | |
| 13:33:28 | dmitriis | sean-k-mooney: besides testing overlays we also tried using VLAN provider networks. That worked as well but the only thing to note there is that collocating VMs with ports attached to overlay networks via PCI devices with the ones that are directly attached to VLAN networks is going to be problematic with the current whitelist based lookup | |
| 13:33:28 | dmitriis | implementation. | |
| 13:34:06 | dmitriis | entries in the whitelist get a physnet tag (either null for overlay networks or a physnet label) | |
| 13:34:28 | sean-k-mooney | correct they do | |
| 13:34:36 | dmitriis | but there is only one vendor/device id pair | |
| 13:34:36 | sean-k-mooney | and technially null was never intended to be supported | |
| 13:35:08 | sean-k-mooney | we never had a nova feature to support overlays with pci devices | |
| 13:35:23 | sean-k-mooney | they exploted a lack of null checking and it happend to work | |
| 13:36:02 | sean-k-mooney | dmitriis: anyway back to your point why is that problematic | |
| 13:36:04 | dmitriis | sean-k-mooney: heh, yes, I wasn't aware of the history but the hardware offload docs explicitly mention that null needs to be used https://docs.openstack.org/neutron/latest/admin/config-ovs-offload.html#configure-nodes-vxlan-configuration | |
| 13:36:23 | sean-k-mooney | dmitriis: yes that was never intended to work | |
| 13:36:37 | sean-k-mooney | but peopel now have it in produciton | |
| 13:37:20 | sean-k-mooney | dmitriis: https://bugs.launchpad.net/nova/+bug/1915282 | |
| 13:37:20 | dmitriis | sean-k-mooney: IIRC PCI requests come with a specific physnet parameter (or null). So when PCI stats are looked at, this parameter is used for lookup | |
| 13:37:28 | dmitriis | let me find that code again | |
| 13:37:59 | sean-k-mooney | yes we end up passing python NONE in the pci request | |
| 13:38:14 | sean-k-mooney | becaue the physent of a vxlan or geneve netowrk is not set | |
| 13:38:31 | sean-k-mooney | that wil match the null phsynet specified in the whitelist | |
| 13:39:04 | dmitriis | ((Pdb)) request | |
| 13:39:04 | dmitriis | InstancePCIRequest(alias_name=<?>,count=1,is_new=<?>,numa_policy=<?>,request_id=c3a87cba-323a-4203-bca7-0916927dcd5b,requester_id='28ea5b12-729c-46b4-b441-518fe786ea10',spec=[{physical_network=None,remote_managed='True'}]) | |
| 13:39:10 | dmitriis | I had something like this ^ | |
| 13:39:20 | sean-k-mooney | yep | |
| 13:39:26 | sean-k-mooney | that shoudl work | |
| 13:39:37 | sean-k-mooney | that is the python None | |
| 13:39:43 | sean-k-mooney | not its not quoted | |
| 13:39:48 | sean-k-mooney | *note | |
| 13:40:03 | sean-k-mooney | that will match physical_network=null | |
| 13:40:05 | dmitriis | ah, maybe that's an old note that I have. It was since fixed to use a string | |
| 13:40:50 | sean-k-mooney | you have to use "'physical_network':null" not "'physical_network':'null'" in the whitelist | |
| 13:41:32 | sean-k-mooney | like this passthrough_whitelist={ "vendor_id":"15b3", "product_id":"101e", "physical_network":null } | |
| 13:42:06 | sean-k-mooney | that enabels a connectx-6 dx for overlay networking | |
| 13:43:04 | sean-k-mooney | dmitriis: if you want to have some VF for vlan/flat and other for geneve tunnels you need to use the adress field to partion the vfs into groups | |
| 13:44:22 | dmitriis | sean-k-mooney: I suppose that could be one way to do it | |
| 13:44:32 | sean-k-mooney | dmitriis: this is because tunnels was never ment to be supported at all so we never impleted a way to allow a device to be part of multile physnets | |
| 13:45:20 | sean-k-mooney | dmitriis: if it was not for the fact that this was used as production we would have closed this as a secuirty bug and blocked the use of null the detail are in the bug | |
| 13:46:03 | dmitriis | sean-k-mooney: yeah, makes sense. I think that documenting this and suggesting address-based partitioning as a workaround is viable for now | |
| 13:46:52 | sean-k-mooney | dmitriis: the tl;dr is we use a json parser to parse the whitelist and in json unquoted null is mapped to the python None object which just happens to be what we get when we parse the phsynet form networks that dont have one | |
| 13:47:46 | sean-k-mooney | which is why physical_network=None in the pci request will actully match | |
| 13:48:09 | sean-k-mooney | sicne that is also python None object not the stirng 'None' | |
| 13:49:50 | sean-k-mooney | fyi hte docs for the whitelist are not greate but incase yo udont know we support both bash style globs and python regex expression in the adress filed | |
| 13:50:07 | sean-k-mooney | and we supprot both in either the sting or dict form | |
| 13:50:12 | sean-k-mooney | https://docs.openstack.org/nova/latest/configuration/config.html#pci.passthrough_whitelist has some examples | |
| 13:50:29 | dmitriis | sean-k-mooney: I recall some other place in Nova where I had to use a string instead (trying to find where so maybe I wrongly brought this up here). | |
| 13:51:28 | sean-k-mooney | there might be if you find it let me knwo and i might know the history or it might just be a bug | |
| 13:52:48 | dmitriis | sean-k-mooney: that's what we used in the lab | |
| 13:52:48 | dmitriis | passthrough_whitelist = [{"vendor_id": "15b3", "product_id": "101e", "physical_network": null, "remote_managed": "true"}] | |
| 13:53:13 | dmitriis | and for physnets: passthrough_whitelist = [{"vendor_id": "15b3", "product_id": "101e", "physical_network": "physnet1", "remote_managed": "true"}] | |
| 13:53:59 | sean-k-mooney | not at the same time right | |
| 13:54:23 | sean-k-mooney | if you add the adress filed you could use both but both look valid to me the first for geneve and the second for flat/vlan | |
| 13:55:07 | sean-k-mooney | huh interesting | |
| 13:55:51 | sean-k-mooney | the VFs for the connectx-6 on the bluefiled 2 have the same vendor and product id as a normal connectx-6 | |
| 13:59:59 | bauzas | elodilles: sure, please do it, I'll just update the wikipage after you | |
| 14:06:46 | dmitriis | sean-k-mooney: ack on the address field usage. | |
| 14:07:57 | dmitriis | sean-k-mooney: I don't have a separate ConnectX-6 at hand but BF2 has ConnectX-6 in it. Let me check the PCI ID DB - I think I've seen different ids but maybe that's for something else. | |
| 14:11:10 | elodilles | bauzas: done, thanks (i might overused the info and link markers o:) feel free to edit :)) | |
| 14:11:29 | bauzas | elodilles: ack, thanks | |
| 14:13:07 | dmitriis | sean-k-mooney: so the PF is different but VFs look like the ones from a "regular" ConnectX-6. | |
| 14:13:07 | dmitriis | PF: | |
| 14:13:07 | dmitriis | 82:00.0 Ethernet controller: Mellanox Technologies MT42822 BlueField-2 integrated ConnectX-6 Dx network controller (rev 01) | |
| 14:13:07 | dmitriis | 82:00.0 0200: 15b3:a2d6 (rev 01) | |
| 14:13:07 | dmitriis | Subsystem: 15b3:0061 | |
| 14:13:23 | dmitriis | VF: | |
| 14:13:23 | dmitriis | 82:00.3 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01) | |
| 14:13:24 | dmitriis | 82:00.3 0200: 15b3:101e (rev 01) | |
| 14:13:24 | dmitriis | Subsystem: 15b3:0061 | |
| 14:14:15 | dmitriis | so careful "remote_managed" tagging is needed | |
| 14:14:44 | sean-k-mooney | ack good to know | |
| 14:29:21 | sean-k-mooney | dmitriis: you can use the adress of the pf and vendor id of the VF to whitelist all the VFs that belog to that pf | |
| 14:29:24 | sean-k-mooney | just so you know | |
| 14:29:44 | sean-k-mooney | dmitriis: that behavior is not well known | |
| 14:32:37 | dmitriis | sean-k-mooney: didn't know that (not surprisingly), thanks for the info. | |
| 14:33:14 | dmitriis | sean-k-mooney: btw, BF2 does bonding at the ARM CPU side transparently to the hypervisor | |
| 14:34:37 | dmitriis | and there's an option to hide the inactive PF for the hypervisor side: https://docs.mellanox.com/display/BlueFieldSWv24011082/BlueField%20Link%20Aggregation | |
| 14:35:27 | dmitriis | that makes it easier for OpenStack deployers/operators since only one PF needs to be taken into account | |
| 14:42:16 | dmitriis | sean-k-mooney: so this is the place where I had to use a string (instead of bool, not None so my reference was not correct) https://review.opendev.org/c/openstack/nova/+/812111/3/nova/network/neutron.py#2295 - that's where a device spec is dynamically generated (not based on flavor or image properties). | |
| 14:45:41 | sean-k-mooney | im on a call but ill look it up after thanks | |
| 14:47:31 | dmitriis | ack | |
| 15:03:22 | Adri2000 | hi, I've got a race condition issue on ussuri and victoria when resizing an instance... specifically this is with /var/lib/nova/instances on NFS, and the following happens sometimes when resizing an instance where a cold migration is triggered: `qemu-img resize` will be run on the new compute node, before the old compute node has fully released the lock on the disk file; this | |
| 15:03:25 | Adri2000 | will put the instance in ERROR state. does that ring a bell to anyone? | |
| 15:03:36 | Adri2000 | ERROR nova.compute.manager [req-...] [instance: 6ca672fd-8746-441f-bbca-6baa3234bb5e] Setting instance vm_state to ERROR: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command. Command: qemu-img resize /var/lib/nova/instances/6ca672fd-8746-441f-bbca-6baa3234bb5e/disk Exit code: 1 Stdout: '' | |
| 15:03:43 | Adri2000 | Stderr: "qemu-img: Could not open '/var/lib/nova/instances/6ca672fd-8746-441f-bbca-6baa3234bb5e/disk': Could not open '/var/lib/nova/instances/6ca672fd-8746-441f-bbca-6baa3234bb5e/disk': Permission denied\n" | |
| 15:04:09 | sean-k-mooney | Adri2000: are you using nfsv3 | |