Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-11-16
13:46:52 sean-k-mooney dmitriis: the tl;dr is we use a json parser to parse the whitelist and in json unquoted null is mapped to the python None object which just happens to be what we get when we parse the phsynet form networks that dont have one
13:47:46 sean-k-mooney which is why physical_network=None in the pci request will actully match
13:48:09 sean-k-mooney sicne that is also python None object not the stirng 'None'
13:49:50 sean-k-mooney fyi hte docs for the whitelist are not greate but incase yo udont know we support both bash style globs and python regex expression in the adress filed
13:50:07 sean-k-mooney and we supprot both in either the sting or dict form
13:50:12 sean-k-mooney https://docs.openstack.org/nova/latest/configuration/config.html#pci.passthrough_whitelist has some examples
13:50:29 dmitriis sean-k-mooney: I recall some other place in Nova where I had to use a string instead (trying to find where so maybe I wrongly brought this up here).
13:51:28 sean-k-mooney there might be if you find it let me knwo and i might know the history or it might just be a bug
13:52:48 dmitriis passthrough_whitelist = [{"vendor_id": "15b3", "product_id": "101e", "physical_network": null, "remote_managed": "true"}]
13:52:48 dmitriis sean-k-mooney: that's what we used in the lab
13:53:13 dmitriis and for physnets: passthrough_whitelist = [{"vendor_id": "15b3", "product_id": "101e", "physical_network": "physnet1", "remote_managed": "true"}]
13:53:59 sean-k-mooney not at the same time right
13:54:23 sean-k-mooney if you add the adress filed you could use both but both look valid to me the first for geneve and the second for flat/vlan
13:55:07 sean-k-mooney huh interesting
13:55:51 sean-k-mooney the VFs for the connectx-6 on the bluefiled 2 have the same vendor and product id as a normal connectx-6
13:59:59 bauzas elodilles: sure, please do it, I'll just update the wikipage after you
14:06:46 dmitriis sean-k-mooney: ack on the address field usage.
14:07:57 dmitriis sean-k-mooney: I don't have a separate ConnectX-6 at hand but BF2 has ConnectX-6 in it. Let me check the PCI ID DB - I think I've seen different ids but maybe that's for something else.
14:11:10 elodilles bauzas: done, thanks (i might overused the info and link markers o:) feel free to edit :))
14:11:29 bauzas elodilles: ack, thanks
14:13:07 dmitriis Subsystem: 15b3:0061
14:13:07 dmitriis 82:00.0 0200: 15b3:a2d6 (rev 01)
14:13:07 dmitriis 82:00.0 Ethernet controller: Mellanox Technologies MT42822 BlueField-2 integrated ConnectX-6 Dx network controller (rev 01)
14:13:07 dmitriis PF:
14:13:07 dmitriis sean-k-mooney: so the PF is different but VFs look like the ones from a "regular" ConnectX-6.
14:13:23 dmitriis 82:00.3 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
14:13:23 dmitriis VF:
14:13:24 dmitriis Subsystem: 15b3:0061
14:13:24 dmitriis 82:00.3 0200: 15b3:101e (rev 01)
14:14:15 dmitriis so careful "remote_managed" tagging is needed
14:14:44 sean-k-mooney ack good to know
14:29:21 sean-k-mooney dmitriis: you can use the adress of the pf and vendor id of the VF to whitelist all the VFs that belog to that pf
14:29:24 sean-k-mooney just so you know
14:29:44 sean-k-mooney dmitriis: that behavior is not well known
14:32:37 dmitriis sean-k-mooney: didn't know that (not surprisingly), thanks for the info.
14:33:14 dmitriis sean-k-mooney: btw, BF2 does bonding at the ARM CPU side transparently to the hypervisor
14:34:37 dmitriis and there's an option to hide the inactive PF for the hypervisor side: https://docs.mellanox.com/display/BlueFieldSWv24011082/BlueField%20Link%20Aggregation
14:35:27 dmitriis that makes it easier for OpenStack deployers/operators since only one PF needs to be taken into account
14:42:16 dmitriis sean-k-mooney: so this is the place where I had to use a string (instead of bool, not None so my reference was not correct) https://review.opendev.org/c/openstack/nova/+/812111/3/nova/network/neutron.py#2295 - that's where a device spec is dynamically generated (not based on flavor or image properties).
14:45:41 sean-k-mooney im on a call but ill look it up after thanks
14:47:31 dmitriis ack
15:03:22 Adri2000 hi, I've got a race condition issue on ussuri and victoria when resizing an instance... specifically this is with /var/lib/nova/instances on NFS, and the following happens sometimes when resizing an instance where a cold migration is triggered: `qemu-img resize` will be run on the new compute node, before the old compute node has fully released the lock on the disk file; this
15:03:25 Adri2000 will put the instance in ERROR state. does that ring a bell to anyone?
15:03:36 Adri2000 ERROR nova.compute.manager [req-...] [instance: 6ca672fd-8746-441f-bbca-6baa3234bb5e] Setting instance vm_state to ERROR: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command. Command: qemu-img resize /var/lib/nova/instances/6ca672fd-8746-441f-bbca-6baa3234bb5e/disk Exit code: 1 Stdout: ''
15:03:43 Adri2000 Stderr: "qemu-img: Could not open '/var/lib/nova/instances/6ca672fd-8746-441f-bbca-6baa3234bb5e/disk': Could not open '/var/lib/nova/instances/6ca672fd-8746-441f-bbca-6baa3234bb5e/disk': Permission denied\n"
15:04:09 sean-k-mooney Adri2000: are you using nfsv3
15:04:37 Adri2000 sean-k-mooney: `/var/lib/nova/instances type nfs4 (rw,relatime,vers=4.1...`
15:05:10 sean-k-mooney ok nfsv3 has locking issues v4.1 improves the situration but recommend v4.2+
15:05:21 sean-k-mooney lyarwood: does ^ seem familar to you
15:07:54 sean-k-mooney Adri2000: i belive there are some tunabel in the mount option that can be used to help resolve this
15:11:37 sean-k-mooney Adri2000: are you using raw images?
15:11:57 bauzas gibi: I'll have to hardstop our meeting by 5:50pm our TZ
15:12:15 gibi bauzas: ack
15:12:27 bauzas in case we have to continue discussing, could you be chairing it ?
15:13:32 sean-k-mooney dmitriis: oh there
15:13:42 sean-k-mooney str(self._is_remote_managed(vnic_type)),
15:13:53 sean-k-mooney dmitriis: ya that makes sense
15:13:57 Adri2000 sean-k-mooney: qcow3 images. one nfs option I have currently is local_lock=none, maybe I should look into this one.
15:14:18 sean-k-mooney dmitriis: technicaly the tags are defiend to be of type String
15:14:31 sean-k-mooney so its a dict of string to string
15:15:12 sean-k-mooney Adri2000: ack the reason i asked is apparently the lockign behavior is differnt in qemu for raw vs qcow
15:15:16 dmitriis sean-k-mooney: ack
15:17:26 sean-k-mooney dmitriis: https://github.com/openstack/nova/blob/master/nova/pci/devspec.py#L262-L263
15:18:27 dmitriis sean-k-mooney: yep, makes sense
15:18:51 dmitriis sean-k-mooney: btw, the ovn-vif repo is now up under ovn-org https://github.com/ovn-org/ovn-vif
15:19:06 sean-k-mooney yes i saw your comment
15:19:10 sean-k-mooney just looking at the code
15:19:17 dmitriis ack
15:19:44 sean-k-mooney am i right in assuming we do not want to allow these device to be used for flavor based pci passthouhg
15:20:38 dmitriis sean-k-mooney: yes, they won't be of much use without being plugged appropriately. Not the VFs at least.
15:21:00 sean-k-mooney ya
15:21:11 sean-k-mooney im wondering if we should explictly block that
15:21:44 sean-k-mooney unfortunetly i dont see a trivial way to do that
15:22:05 sean-k-mooney although we might already do that
15:22:20 dmitriis sean-k-mooney: I guess we could exclude devices from search results if remote_managed is present but not requestedd
15:22:46 sean-k-mooney yep
15:22:58 sean-k-mooney i was just going to provide an exmaple
15:23:04 sean-k-mooney we do this in other cases already
15:23:33 sean-k-mooney https://github.com/openstack/nova/blob/master/nova/pci/stats.py#L411-L433
15:23:46 sean-k-mooney That filteres out PF if you did not ask for one
15:24:08 sean-k-mooney dmitriis: so you can copy past https://github.com/openstack/nova/blob/master/nova/pci/stats.py#L520-L535
15:24:27 sean-k-mooney and then add a new function that will filter out remote managed device if not requested
15:24:49 dmitriis sean-k-mooney: https://review.opendev.org/c/openstack/nova/+/812111/3/nova/network/neutron.py#2295
15:25:00 sean-k-mooney i did this recently when i added support for vdpa https://github.com/openstack/nova/blob/master/nova/pci/stats.py#L540
15:25:01 dmitriis actually, I'm explicitly passing remote_managed=False
15:25:28 sean-k-mooney that wont work
15:25:55 dmitriis sean-k-mooney: even with this? https://review.opendev.org/c/openstack/nova/+/812111/3/nova/pci/stats.py#111
15:25:58 sean-k-mooney it will break existing deploymetn on upgrade as there existing device wont have remote_managed=False
15:26:07 sean-k-mooney and it woudl only apply for pci recuts form ports
15:26:52 sean-k-mooney dmitriis: that would work but we might end up doign a data migration of all existing rows
15:27:34 sean-k-mooney dmitriis: ok we can review this as part of the code review rather then the spec.
15:27:51 sean-k-mooney im just finsihing reading it now and ill approve it shortly
15:27:52 dmitriis sean-k-mooney: ack, I am open to adding a filter as you suggested
15:28:35 sean-k-mooney either would work but one involes updateing every row in the pci device tabel with remote_managed=false :)
15:28:48 dmitriis right, I would certainly like to avoid introducing a change that would break with a stale state in PCI stats
15:28:50 sean-k-mooney the important thing is there is not a gap in the design
15:28:55 dmitriis agreed
15:43:38 sean-k-mooney dmitriis: ok i captured some of my tought in this converstaion in the spec but +2 +w from me
15:45:12 sean-k-mooney dmitriis: feel free to ping me to review the implemetaion too. i do not have +2 right on the code repo but ill try and spend some time reviewing it end to end next week
15:46:14 dmitriis sean-k-mooney: tyvm. I'll try to get the code updated with some of the latest changes by then. Still have to extend func tests to cover more cases but there are some already.

Earlier   Later