| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-11-16 | |||
| 13:36:23 | sean-k-mooney | dmitriis: yes that was never intended to work | |
| 13:36:37 | sean-k-mooney | but peopel now have it in produciton | |
| 13:37:20 | dmitriis | sean-k-mooney: IIRC PCI requests come with a specific physnet parameter (or null). So when PCI stats are looked at, this parameter is used for lookup | |
| 13:37:20 | sean-k-mooney | dmitriis: https://bugs.launchpad.net/nova/+bug/1915282 | |
| 13:37:28 | dmitriis | let me find that code again | |
| 13:37:59 | sean-k-mooney | yes we end up passing python NONE in the pci request | |
| 13:38:14 | sean-k-mooney | becaue the physent of a vxlan or geneve netowrk is not set | |
| 13:38:31 | sean-k-mooney | that wil match the null phsynet specified in the whitelist | |
| 13:39:04 | dmitriis | InstancePCIRequest(alias_name=<?>,count=1,is_new=<?>,numa_policy=<?>,request_id=c3a87cba-323a-4203-bca7-0916927dcd5b,requester_id='28ea5b12-729c-46b4-b441-518fe786ea10',spec=[{physical_network=None,remote_managed='True'}]) | |
| 13:39:04 | dmitriis | ((Pdb)) request | |
| 13:39:10 | dmitriis | I had something like this ^ | |
| 13:39:20 | sean-k-mooney | yep | |
| 13:39:26 | sean-k-mooney | that shoudl work | |
| 13:39:37 | sean-k-mooney | that is the python None | |
| 13:39:43 | sean-k-mooney | not its not quoted | |
| 13:39:48 | sean-k-mooney | *note | |
| 13:40:03 | sean-k-mooney | that will match physical_network=null | |
| 13:40:05 | dmitriis | ah, maybe that's an old note that I have. It was since fixed to use a string | |
| 13:40:50 | sean-k-mooney | you have to use "'physical_network':null" not "'physical_network':'null'" in the whitelist | |
| 13:41:32 | sean-k-mooney | like this passthrough_whitelist={ "vendor_id":"15b3", "product_id":"101e", "physical_network":null } | |
| 13:42:06 | sean-k-mooney | that enabels a connectx-6 dx for overlay networking | |
| 13:43:04 | sean-k-mooney | dmitriis: if you want to have some VF for vlan/flat and other for geneve tunnels you need to use the adress field to partion the vfs into groups | |
| 13:44:22 | dmitriis | sean-k-mooney: I suppose that could be one way to do it | |
| 13:44:32 | sean-k-mooney | dmitriis: this is because tunnels was never ment to be supported at all so we never impleted a way to allow a device to be part of multile physnets | |
| 13:45:20 | sean-k-mooney | dmitriis: if it was not for the fact that this was used as production we would have closed this as a secuirty bug and blocked the use of null the detail are in the bug | |
| 13:46:03 | dmitriis | sean-k-mooney: yeah, makes sense. I think that documenting this and suggesting address-based partitioning as a workaround is viable for now | |
| 13:46:52 | sean-k-mooney | dmitriis: the tl;dr is we use a json parser to parse the whitelist and in json unquoted null is mapped to the python None object which just happens to be what we get when we parse the phsynet form networks that dont have one | |
| 13:47:46 | sean-k-mooney | which is why physical_network=None in the pci request will actully match | |
| 13:48:09 | sean-k-mooney | sicne that is also python None object not the stirng 'None' | |
| 13:49:50 | sean-k-mooney | fyi hte docs for the whitelist are not greate but incase yo udont know we support both bash style globs and python regex expression in the adress filed | |
| 13:50:07 | sean-k-mooney | and we supprot both in either the sting or dict form | |
| 13:50:12 | sean-k-mooney | https://docs.openstack.org/nova/latest/configuration/config.html#pci.passthrough_whitelist has some examples | |
| 13:50:29 | dmitriis | sean-k-mooney: I recall some other place in Nova where I had to use a string instead (trying to find where so maybe I wrongly brought this up here). | |
| 13:51:28 | sean-k-mooney | there might be if you find it let me knwo and i might know the history or it might just be a bug | |
| 13:52:48 | dmitriis | passthrough_whitelist = [{"vendor_id": "15b3", "product_id": "101e", "physical_network": null, "remote_managed": "true"}] | |
| 13:52:48 | dmitriis | sean-k-mooney: that's what we used in the lab | |
| 13:53:13 | dmitriis | and for physnets: passthrough_whitelist = [{"vendor_id": "15b3", "product_id": "101e", "physical_network": "physnet1", "remote_managed": "true"}] | |
| 13:53:59 | sean-k-mooney | not at the same time right | |
| 13:54:23 | sean-k-mooney | if you add the adress filed you could use both but both look valid to me the first for geneve and the second for flat/vlan | |
| 13:55:07 | sean-k-mooney | huh interesting | |
| 13:55:51 | sean-k-mooney | the VFs for the connectx-6 on the bluefiled 2 have the same vendor and product id as a normal connectx-6 | |
| 13:59:59 | bauzas | elodilles: sure, please do it, I'll just update the wikipage after you | |
| 14:06:46 | dmitriis | sean-k-mooney: ack on the address field usage. | |
| 14:07:57 | dmitriis | sean-k-mooney: I don't have a separate ConnectX-6 at hand but BF2 has ConnectX-6 in it. Let me check the PCI ID DB - I think I've seen different ids but maybe that's for something else. | |
| 14:11:10 | elodilles | bauzas: done, thanks (i might overused the info and link markers o:) feel free to edit :)) | |
| 14:11:29 | bauzas | elodilles: ack, thanks | |
| 14:13:07 | dmitriis | Subsystem: 15b3:0061 | |
| 14:13:07 | dmitriis | 82:00.0 0200: 15b3:a2d6 (rev 01) | |
| 14:13:07 | dmitriis | 82:00.0 Ethernet controller: Mellanox Technologies MT42822 BlueField-2 integrated ConnectX-6 Dx network controller (rev 01) | |
| 14:13:07 | dmitriis | PF: | |
| 14:13:07 | dmitriis | sean-k-mooney: so the PF is different but VFs look like the ones from a "regular" ConnectX-6. | |
| 14:13:23 | dmitriis | 82:00.3 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01) | |
| 14:13:23 | dmitriis | VF: | |
| 14:13:24 | dmitriis | Subsystem: 15b3:0061 | |
| 14:13:24 | dmitriis | 82:00.3 0200: 15b3:101e (rev 01) | |
| 14:14:15 | dmitriis | so careful "remote_managed" tagging is needed | |
| 14:14:44 | sean-k-mooney | ack good to know | |
| 14:29:21 | sean-k-mooney | dmitriis: you can use the adress of the pf and vendor id of the VF to whitelist all the VFs that belog to that pf | |
| 14:29:24 | sean-k-mooney | just so you know | |
| 14:29:44 | sean-k-mooney | dmitriis: that behavior is not well known | |
| 14:32:37 | dmitriis | sean-k-mooney: didn't know that (not surprisingly), thanks for the info. | |
| 14:33:14 | dmitriis | sean-k-mooney: btw, BF2 does bonding at the ARM CPU side transparently to the hypervisor | |
| 14:34:37 | dmitriis | and there's an option to hide the inactive PF for the hypervisor side: https://docs.mellanox.com/display/BlueFieldSWv24011082/BlueField%20Link%20Aggregation | |
| 14:35:27 | dmitriis | that makes it easier for OpenStack deployers/operators since only one PF needs to be taken into account | |
| 14:42:16 | dmitriis | sean-k-mooney: so this is the place where I had to use a string (instead of bool, not None so my reference was not correct) https://review.opendev.org/c/openstack/nova/+/812111/3/nova/network/neutron.py#2295 - that's where a device spec is dynamically generated (not based on flavor or image properties). | |
| 14:45:41 | sean-k-mooney | im on a call but ill look it up after thanks | |
| 14:47:31 | dmitriis | ack | |
| 15:03:22 | Adri2000 | hi, I've got a race condition issue on ussuri and victoria when resizing an instance... specifically this is with /var/lib/nova/instances on NFS, and the following happens sometimes when resizing an instance where a cold migration is triggered: `qemu-img resize` will be run on the new compute node, before the old compute node has fully released the lock on the disk file; this | |
| 15:03:25 | Adri2000 | will put the instance in ERROR state. does that ring a bell to anyone? | |
| 15:03:36 | Adri2000 | ERROR nova.compute.manager [req-...] [instance: 6ca672fd-8746-441f-bbca-6baa3234bb5e] Setting instance vm_state to ERROR: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command. Command: qemu-img resize /var/lib/nova/instances/6ca672fd-8746-441f-bbca-6baa3234bb5e/disk Exit code: 1 Stdout: '' | |
| 15:03:43 | Adri2000 | Stderr: "qemu-img: Could not open '/var/lib/nova/instances/6ca672fd-8746-441f-bbca-6baa3234bb5e/disk': Could not open '/var/lib/nova/instances/6ca672fd-8746-441f-bbca-6baa3234bb5e/disk': Permission denied\n" | |
| 15:04:09 | sean-k-mooney | Adri2000: are you using nfsv3 | |
| 15:04:37 | Adri2000 | sean-k-mooney: `/var/lib/nova/instances type nfs4 (rw,relatime,vers=4.1...` | |
| 15:05:10 | sean-k-mooney | ok nfsv3 has locking issues v4.1 improves the situration but recommend v4.2+ | |
| 15:05:21 | sean-k-mooney | lyarwood: does ^ seem familar to you | |
| 15:07:54 | sean-k-mooney | Adri2000: i belive there are some tunabel in the mount option that can be used to help resolve this | |
| 15:11:37 | sean-k-mooney | Adri2000: are you using raw images? | |
| 15:11:57 | bauzas | gibi: I'll have to hardstop our meeting by 5:50pm our TZ | |
| 15:12:15 | gibi | bauzas: ack | |
| 15:12:27 | bauzas | in case we have to continue discussing, could you be chairing it ? | |
| 15:13:32 | sean-k-mooney | dmitriis: oh there | |
| 15:13:42 | sean-k-mooney | str(self._is_remote_managed(vnic_type)), | |
| 15:13:53 | sean-k-mooney | dmitriis: ya that makes sense | |
| 15:13:57 | Adri2000 | sean-k-mooney: qcow3 images. one nfs option I have currently is local_lock=none, maybe I should look into this one. | |
| 15:14:18 | sean-k-mooney | dmitriis: technicaly the tags are defiend to be of type String | |
| 15:14:31 | sean-k-mooney | so its a dict of string to string | |
| 15:15:12 | sean-k-mooney | Adri2000: ack the reason i asked is apparently the lockign behavior is differnt in qemu for raw vs qcow | |
| 15:15:16 | dmitriis | sean-k-mooney: ack | |
| 15:17:26 | sean-k-mooney | dmitriis: https://github.com/openstack/nova/blob/master/nova/pci/devspec.py#L262-L263 | |
| 15:18:27 | dmitriis | sean-k-mooney: yep, makes sense | |
| 15:18:51 | dmitriis | sean-k-mooney: btw, the ovn-vif repo is now up under ovn-org https://github.com/ovn-org/ovn-vif | |
| 15:19:06 | sean-k-mooney | yes i saw your comment | |
| 15:19:10 | sean-k-mooney | just looking at the code | |
| 15:19:17 | dmitriis | ack | |
| 15:19:44 | sean-k-mooney | am i right in assuming we do not want to allow these device to be used for flavor based pci passthouhg | |
| 15:20:38 | dmitriis | sean-k-mooney: yes, they won't be of much use without being plugged appropriately. Not the VFs at least. | |
| 15:21:00 | sean-k-mooney | ya | |
| 15:21:11 | sean-k-mooney | im wondering if we should explictly block that | |
| 15:21:44 | sean-k-mooney | unfortunetly i dont see a trivial way to do that | |
| 15:22:05 | sean-k-mooney | although we might already do that | |