Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-11-16
10:25:58 jkulik we've downstream extended the server-groups api to allow adding servers after the fact. we opted for not allowing to add servers if this would be against the server-group's rules
10:25:59 gibi looking
10:27:53 gibi kashyap: I think you got hit by https://bugs.launchpad.net/neutron/+bug/1940425
10:28:06 jkulik it helps customers if they already spawned an instance and forgot the server-group and now want to spawn another instance in some affinity to the existing one
10:28:08 gibi the stack trace is the same
10:28:42 gibi jkulik, jengbers: so both of you would like the same behavior, you should team up proposing this upstream :)
10:28:46 kashyap gibi: Oh, thank you
10:29:10 kashyap gibi: Now what? ... Should I do a "recheck 1940425"?
10:29:25 kashyap Or pray to the ju-ju at the bottom of the sea? Or...
10:29:26 gibi kashyap: yepp, recheck bug 1940425
10:29:46 jkulik https://github.com/sapcc/nova/commit/7220be3968ee1dd257c9add88228cc5bb9857795 is the main commit downstream for us
10:30:16 jkulik gibi: yes, we talked internally already about proposing this upstream, but small team, much work :/
10:30:18 gibi kashyap: I added your run to the bug maybe that way we can get attention to the failure as it is still happening
10:30:21 kashyap gibi: Thx for the quick spot
10:31:03 gibi jkulik: no pressure, I know that type of frustration
10:35:46 bauzas jkulik: we tried discussing this in the past upstream, but operators are very afraid by the races conditions it creates
10:36:39 bauzas jkulik: problem is, in a distributed service model like Nova, you can't get a valid answer whether you can do it, as when you validate, you don't ask the nova-compute service
10:37:48 jkulik bauzas: that reminds me ... we wanted to change the DB to disallow having a server in multiple server-groups to help with races. we haven't done that, yet. thanks :D
10:38:20 bauzas in theory we should hold new instance creations per compute once you ask for adding a new instance to the group
10:40:02 jkulik our problem is a little different, still, as we use VMware and not libvirt. thus, we have a lot of hidden hypervisors as nova only sees the cluster. therefore, hard anti-affinity doesn't really matter for us that much
10:40:48 jkulik customers want to make sure they run on different hypervisors and thus we sync the server-groups to the VMware clusters. VMware then migrates VMs around to make sure the rules apply.
10:42:46 jkulik i.e. most of our customers depend on soft-anti-affinity, which is a Weigher in nova-scheduler anyways
11:43:54 kashyap bauzas: Alright, added it to the Open Discussion here: https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting
11:44:48 opendevreview Rajat Dhasmana proposed openstack/nova-specs master: Add spec for volume backed server rebuild https://review.opendev.org/c/openstack/nova-specs/+/809621
12:10:11 dmitriis gibi: tyvm for the feedback
12:10:50 gibi dmitriis: you are welcome. it is a well written spec, thanks for putting in the effort
12:22:45 sean-k-mooney i dont know if i set send on my respocne to the last verions of it
12:23:02 sean-k-mooney gibi you are correct the pci passhtoug filter will be suffiect without the prefilter
12:23:29 sean-k-mooney but the prefiletr can help reduce the set if we only report the new trait on host that have off path devices
12:23:47 sean-k-mooney and the capablity to use them of course
12:24:04 sean-k-mooney ill re review that specs later today
12:26:06 gibi sean-k-mooney: yes, exactly my argument, the prefilter is not mandatory but it is good to have
13:00:21 dmitriis sean-k-mooney: ack, ty for confirming
13:02:28 sean-k-mooney i dont currently have access to hardware to test what you have done but i may have access before the end of the cycle. if i do i might reach out to you and try and test it end to end although i dont know if i will have time to do that or not
13:24:48 elodilles bauzas: i'll update now the meeting wiki #stable section if that is not interfering with you right now
13:26:55 dmitriis sean-k-mooney: btw, fnordahl and I have done end-to-end testing of this in a lab. Here's a PPA https://launchpad.net/~fnordahl/+archive/ubuntu/smartnic-enablement that was used in the process (the WIP reviews are in use there). It has https://listman.redhat.com/archives/libvir-list/2021-November/msg00431.html included as well - I am trying to get
13:26:55 dmitriis someone to review it sooner than later.
13:28:20 dmitriis it doesn't yet have the prefilter and compute capability parts that were recently added to the spec but I will work on updating the WIP review soon with that and on raising a relevant os-traits change
13:30:44 dmitriis We had a VM booted with a floating IP assigned which we then connected to via a router. The flows were offloaded into the ConnectX-6 chip present on BF2.
13:31:52 sean-k-mooney i think i have ping that patch to people dowstream already but ill let the vert team know
13:31:57 sean-k-mooney *virt
13:32:11 dmitriis ack, tyvm
13:33:28 dmitriis sean-k-mooney: besides testing overlays we also tried using VLAN provider networks. That worked as well but the only thing to note there is that collocating VMs with ports attached to overlay networks via PCI devices with the ones that are directly attached to VLAN networks is going to be problematic with the current whitelist based lookup
13:33:28 dmitriis implementation.
13:34:06 dmitriis entries in the whitelist get a physnet tag (either null for overlay networks or a physnet label)
13:34:28 sean-k-mooney correct they do
13:34:36 dmitriis but there is only one vendor/device id pair
13:34:36 sean-k-mooney and technially null was never intended to be supported
13:35:08 sean-k-mooney we never had a nova feature to support overlays with pci devices
13:35:23 sean-k-mooney they exploted a lack of null checking and it happend to work
13:36:02 sean-k-mooney dmitriis: anyway back to your point why is that problematic
13:36:04 dmitriis sean-k-mooney: heh, yes, I wasn't aware of the history but the hardware offload docs explicitly mention that null needs to be used https://docs.openstack.org/neutron/latest/admin/config-ovs-offload.html#configure-nodes-vxlan-configuration
13:36:23 sean-k-mooney dmitriis: yes that was never intended to work
13:36:37 sean-k-mooney but peopel now have it in produciton
13:37:20 sean-k-mooney dmitriis: https://bugs.launchpad.net/nova/+bug/1915282
13:37:20 dmitriis sean-k-mooney: IIRC PCI requests come with a specific physnet parameter (or null). So when PCI stats are looked at, this parameter is used for lookup
13:37:28 dmitriis let me find that code again
13:37:59 sean-k-mooney yes we end up passing python NONE in the pci request
13:38:14 sean-k-mooney becaue the physent of a vxlan or geneve netowrk is not set
13:38:31 sean-k-mooney that wil match the null phsynet specified in the whitelist
13:39:04 dmitriis ((Pdb)) request
13:39:04 dmitriis InstancePCIRequest(alias_name=<?>,count=1,is_new=<?>,numa_policy=<?>,request_id=c3a87cba-323a-4203-bca7-0916927dcd5b,requester_id='28ea5b12-729c-46b4-b441-518fe786ea10',spec=[{physical_network=None,remote_managed='True'}])
13:39:10 dmitriis I had something like this ^
13:39:20 sean-k-mooney yep
13:39:26 sean-k-mooney that shoudl work
13:39:37 sean-k-mooney that is the python None
13:39:43 sean-k-mooney not its not quoted
13:39:48 sean-k-mooney *note
13:40:03 sean-k-mooney that will match physical_network=null
13:40:05 dmitriis ah, maybe that's an old note that I have. It was since fixed to use a string
13:40:50 sean-k-mooney you have to use "'physical_network':null" not "'physical_network':'null'" in the whitelist
13:41:32 sean-k-mooney like this passthrough_whitelist={ "vendor_id":"15b3", "product_id":"101e", "physical_network":null }
13:42:06 sean-k-mooney that enabels a connectx-6 dx for overlay networking
13:43:04 sean-k-mooney dmitriis: if you want to have some VF for vlan/flat and other for geneve tunnels you need to use the adress field to partion the vfs into groups
13:44:22 dmitriis sean-k-mooney: I suppose that could be one way to do it
13:44:32 sean-k-mooney dmitriis: this is because tunnels was never ment to be supported at all so we never impleted a way to allow a device to be part of multile physnets
13:45:20 sean-k-mooney dmitriis: if it was not for the fact that this was used as production we would have closed this as a secuirty bug and blocked the use of null the detail are in the bug
13:46:03 dmitriis sean-k-mooney: yeah, makes sense. I think that documenting this and suggesting address-based partitioning as a workaround is viable for now
13:46:52 sean-k-mooney dmitriis: the tl;dr is we use a json parser to parse the whitelist and in json unquoted null is mapped to the python None object which just happens to be what we get when we parse the phsynet form networks that dont have one
13:47:46 sean-k-mooney which is why physical_network=None in the pci request will actully match
13:48:09 sean-k-mooney sicne that is also python None object not the stirng 'None'
13:49:50 sean-k-mooney fyi hte docs for the whitelist are not greate but incase yo udont know we support both bash style globs and python regex expression in the adress filed
13:50:07 sean-k-mooney and we supprot both in either the sting or dict form
13:50:12 sean-k-mooney https://docs.openstack.org/nova/latest/configuration/config.html#pci.passthrough_whitelist has some examples
13:50:29 dmitriis sean-k-mooney: I recall some other place in Nova where I had to use a string instead (trying to find where so maybe I wrongly brought this up here).
13:51:28 sean-k-mooney there might be if you find it let me knwo and i might know the history or it might just be a bug
13:52:48 dmitriis sean-k-mooney: that's what we used in the lab
13:52:48 dmitriis passthrough_whitelist = [{"vendor_id": "15b3", "product_id": "101e", "physical_network": null, "remote_managed": "true"}]
13:53:13 dmitriis and for physnets: passthrough_whitelist = [{"vendor_id": "15b3", "product_id": "101e", "physical_network": "physnet1", "remote_managed": "true"}]
13:53:59 sean-k-mooney not at the same time right
13:54:23 sean-k-mooney if you add the adress filed you could use both but both look valid to me the first for geneve and the second for flat/vlan
13:55:07 sean-k-mooney huh interesting
13:55:51 sean-k-mooney the VFs for the connectx-6 on the bluefiled 2 have the same vendor and product id as a normal connectx-6
13:59:59 bauzas elodilles: sure, please do it, I'll just update the wikipage after you
14:06:46 dmitriis sean-k-mooney: ack on the address field usage.
14:07:57 dmitriis sean-k-mooney: I don't have a separate ConnectX-6 at hand but BF2 has ConnectX-6 in it. Let me check the PCI ID DB - I think I've seen different ids but maybe that's for something else.
14:11:10 elodilles bauzas: done, thanks (i might overused the info and link markers o:) feel free to edit :))
14:11:29 bauzas elodilles: ack, thanks
14:13:07 dmitriis sean-k-mooney: so the PF is different but VFs look like the ones from a "regular" ConnectX-6.
14:13:07 dmitriis PF:

Earlier   Later