Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-11-16
10:03:39 sean-k-mooney but its kind fo hard to extend them for the reason above
10:03:59 gibi sean-k-mooney: for that we need to solve jengbers' problem and also extend the logic to support multiple groups per instance (or nested groups
10:04:02 gibi )
10:04:16 gibi both is painfully missing but hard to solve
10:05:23 gibi bauzas: I'm done with the spec sweep. I could not really comment on the ironic one https://review.opendev.org/c/openstack/nova-specs/+/815789 and it seems nobody commented yet
10:05:37 gibi the rest of the specs has feedback
10:05:40 sean-k-mooney ya. on second tought we should have "server aggreates" in parallel to server groups. you knwo so we can aggreate servers and give that aggreate of servers a name and even have some metadta that can be shared like this server it the primary of the aggreate and just not have it related to vm placement at all
10:05:46 bauzas gibi: I still have 3 specs to look at
10:05:55 sean-k-mooney that way we can pretend server groups dont exist :)
10:05:55 bauzas gibi: but OK, and thanks for the fish
10:06:24 gibi sean-k-mooney: :D
10:14:06 jengbers gibi, sean-k-mooney: If it was only possible for admins, that could work, because they can also migrate servers, but for users it seems quite hard.
10:15:42 gibi jengbers: yeah that could work. Feel free to propose a spec about the new API to get wider discussion around it
10:16:26 jengbers On the other hand, they can power of and start an instance. I guess that would mean it is started on a different hypervisor.
10:20:54 sean-k-mooney really there are 2 paths we could take. 1 allow normall user ask nova to cold/live migrate an instance to be consitent with a server group that it is currently not a member of and then allow them to add the server to the group after, rejecting the request if the policy is volated
10:21:33 sean-k-mooney or 2 we can have the server group add triger the migration as part fo the request
10:24:39 kashyap Is this failing for anyone else too?
10:24:41 kashyap tempest.api.compute.admin.test_live_migration.LiveAutoBlockMigrationV225Test.test_live_migration_with_trunk [108.238299s] ... FAILED
10:25:41 gibi kashyap: could you link the test run?
10:25:56 kashyap gibi: https://zuul.opendev.org/t/openstack/build/632f8ed30e9a4a04a32648843f227ef3
10:25:58 jkulik we've downstream extended the server-groups api to allow adding servers after the fact. we opted for not allowing to add servers if this would be against the server-group's rules
10:25:59 gibi looking
10:27:53 gibi kashyap: I think you got hit by https://bugs.launchpad.net/neutron/+bug/1940425
10:28:06 jkulik it helps customers if they already spawned an instance and forgot the server-group and now want to spawn another instance in some affinity to the existing one
10:28:08 gibi the stack trace is the same
10:28:42 gibi jkulik, jengbers: so both of you would like the same behavior, you should team up proposing this upstream :)
10:28:46 kashyap gibi: Oh, thank you
10:29:10 kashyap gibi: Now what? ... Should I do a "recheck 1940425"?
10:29:25 kashyap Or pray to the ju-ju at the bottom of the sea? Or...
10:29:26 gibi kashyap: yepp, recheck bug 1940425
10:29:46 jkulik https://github.com/sapcc/nova/commit/7220be3968ee1dd257c9add88228cc5bb9857795 is the main commit downstream for us
10:30:16 jkulik gibi: yes, we talked internally already about proposing this upstream, but small team, much work :/
10:30:18 gibi kashyap: I added your run to the bug maybe that way we can get attention to the failure as it is still happening
10:30:21 kashyap gibi: Thx for the quick spot
10:31:03 gibi jkulik: no pressure, I know that type of frustration
10:35:46 bauzas jkulik: we tried discussing this in the past upstream, but operators are very afraid by the races conditions it creates
10:36:39 bauzas jkulik: problem is, in a distributed service model like Nova, you can't get a valid answer whether you can do it, as when you validate, you don't ask the nova-compute service
10:37:48 jkulik bauzas: that reminds me ... we wanted to change the DB to disallow having a server in multiple server-groups to help with races. we haven't done that, yet. thanks :D
10:38:20 bauzas in theory we should hold new instance creations per compute once you ask for adding a new instance to the group
10:40:02 jkulik our problem is a little different, still, as we use VMware and not libvirt. thus, we have a lot of hidden hypervisors as nova only sees the cluster. therefore, hard anti-affinity doesn't really matter for us that much
10:40:48 jkulik customers want to make sure they run on different hypervisors and thus we sync the server-groups to the VMware clusters. VMware then migrates VMs around to make sure the rules apply.
10:42:46 jkulik i.e. most of our customers depend on soft-anti-affinity, which is a Weigher in nova-scheduler anyways
11:43:54 kashyap bauzas: Alright, added it to the Open Discussion here: https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting
11:44:48 opendevreview Rajat Dhasmana proposed openstack/nova-specs master: Add spec for volume backed server rebuild https://review.opendev.org/c/openstack/nova-specs/+/809621
12:10:11 dmitriis gibi: tyvm for the feedback
12:10:50 gibi dmitriis: you are welcome. it is a well written spec, thanks for putting in the effort
12:22:45 sean-k-mooney i dont know if i set send on my respocne to the last verions of it
12:23:02 sean-k-mooney gibi you are correct the pci passhtoug filter will be suffiect without the prefilter
12:23:29 sean-k-mooney but the prefiletr can help reduce the set if we only report the new trait on host that have off path devices
12:23:47 sean-k-mooney and the capablity to use them of course
12:24:04 sean-k-mooney ill re review that specs later today
12:26:06 gibi sean-k-mooney: yes, exactly my argument, the prefilter is not mandatory but it is good to have
13:00:21 dmitriis sean-k-mooney: ack, ty for confirming
13:02:28 sean-k-mooney i dont currently have access to hardware to test what you have done but i may have access before the end of the cycle. if i do i might reach out to you and try and test it end to end although i dont know if i will have time to do that or not
13:24:48 elodilles bauzas: i'll update now the meeting wiki #stable section if that is not interfering with you right now
13:26:55 dmitriis sean-k-mooney: btw, fnordahl and I have done end-to-end testing of this in a lab. Here's a PPA https://launchpad.net/~fnordahl/+archive/ubuntu/smartnic-enablement that was used in the process (the WIP reviews are in use there). It has https://listman.redhat.com/archives/libvir-list/2021-November/msg00431.html included as well - I am trying to get
13:26:55 dmitriis someone to review it sooner than later.
13:28:20 dmitriis it doesn't yet have the prefilter and compute capability parts that were recently added to the spec but I will work on updating the WIP review soon with that and on raising a relevant os-traits change
13:30:44 dmitriis We had a VM booted with a floating IP assigned which we then connected to via a router. The flows were offloaded into the ConnectX-6 chip present on BF2.
13:31:52 sean-k-mooney i think i have ping that patch to people dowstream already but ill let the vert team know
13:31:57 sean-k-mooney *virt
13:32:11 dmitriis ack, tyvm
13:33:28 dmitriis sean-k-mooney: besides testing overlays we also tried using VLAN provider networks. That worked as well but the only thing to note there is that collocating VMs with ports attached to overlay networks via PCI devices with the ones that are directly attached to VLAN networks is going to be problematic with the current whitelist based lookup
13:33:28 dmitriis implementation.
13:34:06 dmitriis entries in the whitelist get a physnet tag (either null for overlay networks or a physnet label)
13:34:28 sean-k-mooney correct they do
13:34:36 dmitriis but there is only one vendor/device id pair
13:34:36 sean-k-mooney and technially null was never intended to be supported
13:35:08 sean-k-mooney we never had a nova feature to support overlays with pci devices
13:35:23 sean-k-mooney they exploted a lack of null checking and it happend to work
13:36:02 sean-k-mooney dmitriis: anyway back to your point why is that problematic
13:36:04 dmitriis sean-k-mooney: heh, yes, I wasn't aware of the history but the hardware offload docs explicitly mention that null needs to be used https://docs.openstack.org/neutron/latest/admin/config-ovs-offload.html#configure-nodes-vxlan-configuration
13:36:23 sean-k-mooney dmitriis: yes that was never intended to work
13:36:37 sean-k-mooney but peopel now have it in produciton
13:37:20 sean-k-mooney dmitriis: https://bugs.launchpad.net/nova/+bug/1915282
13:37:20 dmitriis sean-k-mooney: IIRC PCI requests come with a specific physnet parameter (or null). So when PCI stats are looked at, this parameter is used for lookup
13:37:28 dmitriis let me find that code again
13:37:59 sean-k-mooney yes we end up passing python NONE in the pci request
13:38:14 sean-k-mooney becaue the physent of a vxlan or geneve netowrk is not set
13:38:31 sean-k-mooney that wil match the null phsynet specified in the whitelist
13:39:04 dmitriis ((Pdb)) request
13:39:04 dmitriis InstancePCIRequest(alias_name=<?>,count=1,is_new=<?>,numa_policy=<?>,request_id=c3a87cba-323a-4203-bca7-0916927dcd5b,requester_id='28ea5b12-729c-46b4-b441-518fe786ea10',spec=[{physical_network=None,remote_managed='True'}])
13:39:10 dmitriis I had something like this ^
13:39:20 sean-k-mooney yep
13:39:26 sean-k-mooney that shoudl work
13:39:37 sean-k-mooney that is the python None
13:39:43 sean-k-mooney not its not quoted
13:39:48 sean-k-mooney *note
13:40:03 sean-k-mooney that will match physical_network=null
13:40:05 dmitriis ah, maybe that's an old note that I have. It was since fixed to use a string
13:40:50 sean-k-mooney you have to use "'physical_network':null" not "'physical_network':'null'" in the whitelist
13:41:32 sean-k-mooney like this passthrough_whitelist={ "vendor_id":"15b3", "product_id":"101e", "physical_network":null }
13:42:06 sean-k-mooney that enabels a connectx-6 dx for overlay networking
13:43:04 sean-k-mooney dmitriis: if you want to have some VF for vlan/flat and other for geneve tunnels you need to use the adress field to partion the vfs into groups
13:44:22 dmitriis sean-k-mooney: I suppose that could be one way to do it
13:44:32 sean-k-mooney dmitriis: this is because tunnels was never ment to be supported at all so we never impleted a way to allow a device to be part of multile physnets
13:45:20 sean-k-mooney dmitriis: if it was not for the fact that this was used as production we would have closed this as a secuirty bug and blocked the use of null the detail are in the bug
13:46:03 dmitriis sean-k-mooney: yeah, makes sense. I think that documenting this and suggesting address-based partitioning as a workaround is viable for now
13:46:52 sean-k-mooney dmitriis: the tl;dr is we use a json parser to parse the whitelist and in json unquoted null is mapped to the python None object which just happens to be what we get when we parse the phsynet form networks that dont have one
13:47:46 sean-k-mooney which is why physical_network=None in the pci request will actully match

Earlier   Later