Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-11-13
12:39:10 gibi anyhow I stop here. I just noticed it and felt strange, but I can ignore it and be safe that it will keep working as before
14:20:03 opendevreview Balazs Gibizer proposed openstack/nova master: Compute manager to use thread pools selectively https://review.opendev.org/c/openstack/nova/+/966016
14:20:04 opendevreview Balazs Gibizer proposed openstack/nova master: Run nova-compute in native threading mode https://review.opendev.org/c/openstack/nova/+/965467
14:20:04 opendevreview Balazs Gibizer proposed openstack/nova master: Libvirt event handling without eventlet https://review.opendev.org/c/openstack/nova/+/965949
14:21:22 gibi gmaan: FYI https://review.opendev.org/c/openstack/nova/+/966016 proposes changing the default value of some of our config options. After the discussion of dansmith and sean-k-mooney we think this is OK from compatibility perspective via having an upgrade relnote. Do you agree?
14:32:43 opendevreview Balazs Gibizer proposed openstack/nova master: Libvirt event handling without eventlet https://review.opendev.org/c/openstack/nova/+/965949
14:32:43 opendevreview Balazs Gibizer proposed openstack/nova master: Compute manager to use thread pools selectively https://review.opendev.org/c/openstack/nova/+/966016
14:32:44 opendevreview Balazs Gibizer proposed openstack/nova master: Run nova-compute in native threading mode https://review.opendev.org/c/openstack/nova/+/965467
14:33:21 opendevreview Dan Smith proposed openstack/nova master: Test nova-next with >1 parallel migrations https://review.opendev.org/c/openstack/nova/+/966447
15:08:00 gibi stephenfin: after a bit of deliberation and a failed trial I'm OK with the direction in 967005
15:08:04 gibi 967005
15:08:13 gibi https://review.opendev.org/c/openstack/nova/+/967005
15:08:22 gibi (wtf happens with my copy paste buffer)
15:09:35 gibi stephenfin: will you fix the 700 failed unit test or looking for a volunteer to take over?
15:10:07 dansmith gibi: I still haven't looked at your patch to fix.. are you saying that one is not working and this is the one we need to pursue?
15:10:23 gibi nope, this is a followup top of mine
15:10:31 dansmith okay
15:10:36 gibi to prevent later patches re-introducing the issue
15:11:01 gibi ~ the generic issue of calling driver methods before driver.init_host
15:11:04 stephenfin gibi: sounds good: I had tried an alternative involving a metaclass but it was feeling...complicated
15:11:55 gibi metaclass, I haven't thought about it. Woudl that sidestep our inheritance pattern?
15:12:22 gibi anyhow
15:12:28 gibi I'm OK not going there :)
15:12:32 opendevreview Stephen Finucane proposed openstack/nova master: virt: Ensure init_host is always called https://review.opendev.org/c/openstack/nova/+/967051
15:12:42 stephenfin gibi: that's the patch (incomplete)
15:14:23 gibi ahh OK, so that basically automating injecting a wrapper to the normal virt calls. I think that will not work as the virt interface function is not called by the implementation as most of the virt inteface calls raising NotImplementedError
15:15:37 stephenfin yeah, that's what I'd figured out before lunch: we'd need to use the metaclass on the actual implementation rather than the base class. And ideally only wrap methods defined in the base class (not that drivers should have other public methods)
15:15:48 gibi yeah
15:16:04 gibi then I think we felt the same pain
15:16:37 gibi anyhow let me know if you need help with the unit test or need a re-review if you jump on the unit test by yourself
16:07:51 vsaienko hey sean-k-mooney: maybe it will be interesting there are performance tests for different scenarios (asap2, vdpa with packed ring 8 queues, virtio single queue, virtio packed + 8 queues) https://jumpshare.com/s/xHU2UTQoKoeKwAWDFsa6 https://jumpshare.com/s/mLQ1u29xOyMftaULc0lJ so far VDPA does not show super significant improvements on 10Gb.
16:08:25 vsaienko scenarios are iperf tests, arguments in each scenario description
16:18:40 sean-k-mooney i think that is in line with expectation
16:19:38 vsaienko I thought that vdpa will be near same as sriov
16:19:41 sean-k-mooney for tcp at least
16:20:02 vsaienko maybe it will be closer on higher rates 50Gbps or 100Gbps
16:20:19 sean-k-mooney vdpa can by connectx6 dx is not doign vdpa fully in hardware
16:21:11 vsaienko nvidiay says they support hardware vdpa on connect 6dx
16:21:18 sean-k-mooney can you expalin the labelign by the way? when you saw swtich deve you mean hardware offloead ovs with a driect assigned VF i assume
16:22:05 sean-k-mooney vsaienko: yes but its not done fully in an asic it does soem fo the data trasformation using firware/cpu comptue on the nic
16:23:08 vsaienko switchdev - sriov + asap2 hardware offload, vdpa-packed - its packed ring + 8 queues, virtio def - is default flavor, no multiqueue or packed ring, virtio-packed - packed ring + 8 queues
16:23:20 vsaienko vdpa with hardware offload as well
16:23:45 vsaienko between hosts VXLAN network, mtu 1450 (no jumboframes) on all configurations
16:23:46 sean-k-mooney ok because the defautl is usign vdpa so its not correct to call it virtio
16:24:06 sean-k-mooney ah you also adding vxlan overhead
16:24:45 sean-k-mooney so is this inter host traffic i.e. 2 vms on diffent hosts or 2 vms on the same host
16:25:01 vsaienko virtio def is default neutron port =normal which is virtio
16:25:16 sean-k-mooney its technaily kernel vhost
16:25:18 vsaienko all scenarios between VMs on different hosts
16:26:01 sean-k-mooney virtio woudl be the virto stack in ovs without the kernel vhost-net module offloading it
16:27:29 sean-k-mooney this looks to me like you are hittig a code path that is not supproted for hardware offload
16:28:00 sean-k-mooney for example the conenction tracker used for security groups is not fully offloadable to hardware
16:28:08 sean-k-mooney are you using ml2/ovs?
16:28:35 vsaienko its ovs, no qos no security groups, in TC I see that traffic is offloaded
16:29:25 sean-k-mooney well in https://jumpshare.com/s/xHU2UTQoKoeKwAWDFsa6
16:29:50 sean-k-mooney you are not hitting line rate env in the swtichdev case which is effectivly jsut using sriov for the dataplane
16:29:59 sean-k-mooney with ovs providing the contol plane
16:30:10 vsaienko yes
16:30:29 sean-k-mooney that to me suggest you are hittign bottel necks in teh trafic generateion as well
16:31:13 vsaienko not sure between hosts on same hypervisor iperf can show more, its new version with multithreads
16:31:35 sean-k-mooney well ierf it not really godo for doing real testing
16:31:43 sean-k-mooney you can use -p to have it use multipel cores
16:32:02 sean-k-mooney but if you want ot test this properly you eigher need a hardware traffic generator or somethign like trex
16:32:17 vsaienko yes, there are scenarios with p1 - its -P 1, p30 - -P 30
16:32:23 sean-k-mooney https://trex-tgn.cisco.com/
16:33:13 sean-k-mooney right but for 1400 by packets you shoudl be able to hit 10G even with 1 thread
16:34:03 sean-k-mooney if the hsot are other wise idel with 1400 by packets even standar kernel ovs shoudl be able to hit 10G
16:34:26 sean-k-mooney so to me the performac eof this host/vms is very much below what i woudl expect
16:36:00 sean-k-mooney vsaienko: what version of opesntack are you using
16:36:20 vsaienko but why with sriov it provides 6+gbps
16:36:27 vsaienko its epoxy
16:36:42 sean-k-mooney so 6GBPS is also too low for sriov
16:36:42 vsaienko 3.x openvswitch
16:37:01 vsaienko maybe indeed its a limitation of iperf
16:37:26 sean-k-mooney so if your using epoxy you shoudl have https://github.com/openstack/os-vif/blob/master/vif_plug_ovs/ovs.py#L107-L126
16:37:45 sean-k-mooney but can you check in ovs what the qos policy on the ovs port is set too
16:37:53 sean-k-mooney and confirm its linux-noop
16:37:57 vsaienko qos plugin is disabled on my env
16:38:06 sean-k-mooney that is not what im asking
16:38:27 vsaienko but thanks for pointing this out, I haven't seen this option
16:38:30 sean-k-mooney in new version fo systemd or the kernel the default qdisg was changed to fq_codel
16:38:55 sean-k-mooney now we added this option to make sure tha tthat was disabled on teh tap devices
16:39:08 vsaienko interesting
16:39:11 vsaienko let me check this
16:39:11 sean-k-mooney because if we dont ti results in massive perfroamce hti
16:39:21 vsaienko it should be default
16:39:24 sean-k-mooney yep
16:39:47 sean-k-mooney so we need ot check that 1 the policy in ovs set by os-vif is linux-noop
16:40:23 sean-k-mooney but we shoudl also check that the representor netdev added to ovs does not have a qos policy applied with tc
16:40:53 vsaienko its not explicitly set, so default is picked
16:41:39 sean-k-mooney right so im askign you to conrimf this because this was never tested for vdpa or hardware offloaded ovs
16:41:46 sean-k-mooney it shoudl get appled but it may not be
16:42:01 sean-k-mooney and if its not beign set in the ovs db then it coudl reduce the performacne
16:43:43 vsaienko https://paste.openstack.org/show/bZeQE1UxDhnNWIpJmiCb/
16:44:31 sean-k-mooney yep that the problem
16:44:35 sean-k-mooney its using fqcodel
16:44:57 sean-k-mooney create /etc/sysctl.conf.d/99_qdisk.conf and add net.core.default_qdisc = pfifo
16:45:24 sean-k-mooney thatthen apply it and try doing it again with a new vm
16:45:39 vsaienko ok thanks, let me try
16:46:33 sean-k-mooney well that might work but lets confirm something

Earlier   Later