| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-03-23 | |||
| 11:45:45 | bauzas | gibi: this way, you wouldn't change the RPs after the compute service is restarted | |
| 11:45:58 | sean-k-mooney | there used to be a limiation around this but im tryign to recall what it was | |
| 11:46:07 | gibi | bauzas: these RPs are not handled by nova, so this reshape will happen in neutron :) | |
| 11:46:15 | bauzas | ah, shit, right | |
| 11:46:16 | gibi | bauzas: but besides that yes | |
| 11:47:28 | sean-k-mooney | allowing RPs to be reparented would be a much cleaner approch | |
| 11:47:36 | sean-k-mooney | i have asked for this in the past | |
| 11:47:49 | sean-k-mooney | to avoid the need to reshap | |
| 11:48:06 | bauzas | no | |
| 11:48:20 | bauzas | because you need to modify them just after restarting the service | |
| 11:48:25 | bauzas | and not after | |
| 11:48:35 | bauzas | or candidates would be different | |
| 11:48:47 | sean-k-mooney | bauzas: we have disucssed this in the past and i have nver been satifed with the answeres as to why changeing the parent is not allowed | |
| 11:49:55 | gibi | sean-k-mooney: agree reparenting in my specific case would be cleaner, a generic reparenting is dangerous in the other hand | |
| 11:50:10 | gibi | like moving an RP from one tree to other | |
| 11:50:18 | sean-k-mooney | i dont think its any more dangours then a reshap | |
| 11:50:18 | gibi | that would definitely break existing allocations | |
| 11:50:34 | sean-k-mooney | it may or it may not | |
| 11:50:43 | sean-k-mooney | there are several cases where it would not | |
| 11:51:25 | sean-k-mooney | gibi: setting the parent is what i was tinking of before not the name by the way | |
| 11:51:34 | sean-k-mooney | e.g. where we could only do it once | |
| 11:51:47 | gibi | sean-k-mooney: yeah, parent can only be set from null to non null | |
| 11:52:13 | sean-k-mooney | gibi: the bandwith RPs shoudl really be using the pci device addres not the netdev name by the way | |
| 11:52:52 | sean-k-mooney | i stongly advocated for not using the netdev name in the RP name when this was being first added | |
| 11:52:53 | gibi | sean-k-mooney: nova uses PCI address, neutron uses netdev name | |
| 11:53:34 | sean-k-mooney | right which is problematic | |
| 11:53:39 | gibi | as neutron handleds these RPs it got the netdev name and nova does a translation. | |
| 11:53:49 | sean-k-mooney | right i know | |
| 11:53:59 | sean-k-mooney | but i did not wnat it to use the netdev name in neutron at all | |
| 11:54:13 | gibi | thats a wide change ^^ | |
| 11:54:24 | gibi | as neturon used that even before qos | |
| 11:54:43 | sean-k-mooney | correct which has been broken for sriov for a long time | |
| 11:55:20 | sean-k-mooney | by the way i was not suggestign that neutron stops usign devname eveywhere | |
| 11:55:28 | sean-k-mooney | that is a sperate thing | |
| 11:55:41 | sean-k-mooney | just that we model device in placment with teh pci address | |
| 11:56:16 | sean-k-mooney | when we start modeling sriov vfs in placmnet we will need to use the PF pci address as part of the name for the VF inventores | |
| 11:56:47 | sean-k-mooney | other but that will be challing with qos | |
| 11:57:42 | sean-k-mooney | we shoudl not have different behavour with nic vs other pci devices so i dont see using the devname as a valid option for nova reporting pci device in the future | |
| 11:59:06 | gibi | sean-k-mooney: I agree that when we model the PFs in placement we need to do it right. and make the QoS related RPs aligned. | |
| 12:13:52 | gibi | sean-k-mooney: it is not impossile rename the device RPs in placement to use pci address, we just need to add some netdev -> PCI address translation code in the neturon sriov agent, and keep conditional logic in nova to look for old netdev names and new pci address names for an extra cycle to support rolling upgrade | |
| 12:14:41 | sean-k-mooney | how did the RP get move by the way | |
| 12:14:45 | sean-k-mooney | you mentioned a bugfix? | |
| 12:14:54 | sean-k-mooney | i assume that has been reverted? | |
| 12:15:17 | sean-k-mooney | and yes that is one way to adress that in the future | |
| 12:15:29 | gibi | sean-k-mooney: this fixed a bug https://review.opendev.org/c/openstack/neutron/+/696600 but broke the tree | |
| 12:15:40 | sean-k-mooney | the other way woul dbe to add a kind fo symlink or alias feature to placment | |
| 12:15:49 | sean-k-mooney | so we can refer to the same RP with different names | |
| 12:16:47 | sean-k-mooney | ah i rememebr this | |
| 12:17:27 | sean-k-mooney | gibi: what is currently broken by having them under the compute node RP | |
| 12:18:25 | gibi | sean-k-mooney: if you upgrade from Train to Ussuri in a way that you had QoS configured already in Train then the Ussuri neutron will error out | |
| 12:18:38 | gibi | if you deploy a new Ussuri then no visible problem seen | |
| 12:18:45 | sean-k-mooney | got it | |
| 12:18:56 | sean-k-mooney | so we dont have min bandwith testing in grenade | |
| 12:18:59 | gibi | this is why the problem went undetected for so long | |
| 12:19:02 | sean-k-mooney | well neutron does not | |
| 12:19:23 | gibi | I can even think that schduling works after the upgrade | |
| 12:19:33 | gibi | but i have to reproduce it and try | |
| 12:19:43 | sean-k-mooney | i dont think it woudl break schduling | |
| 12:19:48 | gibi | new deployments are not affected just have a wrong tree sturcture | |
| 12:19:50 | sean-k-mooney | nova is not currently relying on the structure | |
| 12:20:16 | sean-k-mooney | i proposed using that stucure because i wante dot model other networking requirments | |
| 12:20:27 | sean-k-mooney | like make port count support by a vswtich | |
| 12:20:49 | sean-k-mooney | or trait on a agent provider for offloads or network type supprot e.g. vxlan | |
| 12:21:00 | sean-k-mooney | then evneutally add that to the query with same subtree | |
| 12:21:24 | sean-k-mooney | but we did nto have same subtree at the time so the agent RPs are not actuly adding any benifit currently | |
| 12:22:05 | sean-k-mooney | anyway so the fix for this would go into neutron right | |
| 12:22:11 | gibi | yes and yes | |
| 12:22:37 | gibi | and the whole think came out as I started looking into modeling OVS packet processing capacity on the ovs agent RP | |
| 12:23:05 | sean-k-mooney | as in packet per second? | |
| 12:23:11 | gibi | yepp | |
| 12:23:38 | sean-k-mooney | you an i both know that that depends on the type of packet you use and the ip pipline/match rules | |
| 12:24:06 | sean-k-mooney | its very hard to get that right without a lot of testing on your excat hardware | |
| 12:24:30 | sean-k-mooney | i assume neutron will be dumb and just have the operator say the capastity is X in a config like bandwith | |
| 12:24:34 | gibi | I'm not well educated in OVS | |
| 12:24:56 | gibi | yes the inventory would be config driven like bandwidth | |
| 12:25:26 | sean-k-mooney | gibi: the pps number is differnt for vlan network vs vxlan network on the same ovs on the same host | |
| 12:25:55 | sean-k-mooney | e.g. l3 tunneled networks use more cpu cycle to decap and encap then vlan | |
| 12:26:23 | sean-k-mooney | in the case of ovs-dpdk vm to vm traffic on the same host uses more cycle then vm to physical network | |
| 12:26:38 | sean-k-mooney | since vm to vm can levgerage as many hardware offloads | |
| 12:27:01 | sean-k-mooney | so depening on the direction of traffic and the overlay use the capsity will change | |
| 12:27:33 | sean-k-mooney | you can obvioulsy deploy and test for this but it also will change based on other factors | |
| 12:27:56 | sean-k-mooney | such as security group rules implemnte in contrack and the traffic profile. | |
| 12:28:43 | sean-k-mooney | e.g. setting up and tearing downs lots of tcp connection requires all the inital packets to go through contrack to do the state tracking | |
| 12:29:11 | sean-k-mooney | so the pps will be lower with lots of new connection being established vs steady state | |
| 12:29:29 | sean-k-mooney | multicast/broadcast vs unicast also is a factor | |
| 12:29:47 | sean-k-mooney | gibi: so im glad figuring that out will be the operators problem | |
| 12:33:54 | lyarwood | https://review.opendev.org/c/openstack/nova/+/768466 - anyone able to review this change introducing a nova-live-migration-ceph job? Everything has finally merged so we should be good to go now. | |
| 12:34:15 | lyarwood | I'm still working on the grenade change ontop of it | |
| 12:36:51 | gibi | sean-k-mooney: thanks for the info, I don't have this deep networking knowledge, so I appreciate these details | |
| 12:37:22 | gibi | sean-k-mooney: I agree that configuring this will be really deployment specific | |
| 12:38:45 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Simplify device_path check in _detach_encryptor https://review.opendev.org/c/openstack/nova/+/778463 | |
| 12:38:55 | gibi | sean-k-mooney: as far as I understand some of our big deployers tend use a unified network setup, like all vlan. Also the these deployers tend to do deep performance testing with their config, so I assume they will figure out a pps number for their specific deployment and traffic patterns. | |
| 12:38:56 | sean-k-mooney | i worked with other teams at intel that worked on ovs and vsperf in opnfv | |
| 12:39:31 | sean-k-mooney | gibi: yep they do | |
| 12:39:45 | sean-k-mooney | but for public clouds this gets harder as you dont know the workload | |
| 12:39:53 | gibi | yes | |
| 12:39:58 | sean-k-mooney | i assume as part of this there is an enforcemnt elemnt | |
| 12:40:12 | sean-k-mooney | e.g. a max pps qos policy | |
| 12:40:30 | sean-k-mooney | i think tc can enforce that | |
| 12:40:34 | gibi | yeah the max pps is already proposed https://review.opendev.org/c/openstack/neutron-specs/+/779940/1/specs/wallaby/qos_pps_rule.rst | |