Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-10-22
10:22:28 gibi in my local env I see plug/unplug event during nova hard reboot for VF ports but not for PF ports so probably this is the case
10:23:06 sean-k-mooney[m] yes so you might need to make an excption in your workaround patch
10:23:40 sean-k-mooney[m] perhaps change it form a boolean to a list of vnic_types
10:24:10 sean-k-mooney[m] odl only support vnic_type normal and vhost_user
10:24:20 sean-k-mooney[m] well vhost-user
10:24:52 gibi I think the doc in the patch still correct when we say set the flag only for ml2/ovs or networking-odl
10:25:19 gibi I might extend that with mech_sriov + vnic_type direct
10:25:31 sean-k-mooney[m] right but if you filter by vnic type you can use it when you have odl and sriov on the same host
10:26:05 gibi yeah, I can ignore direct-physical ports when waiting for plug
10:26:23 sean-k-mooney[m] ya i guess that also works
10:26:49 gibi sean-k-mooney[m]: what would be your way to filter?
10:28:37 sean-k-mooney[m] if we make the config option a list of vnic_types to wait for on hard reboot we just do
10:28:59 sean-k-mooney[m] if vif.vnic_type in CONF.wait_on_reboot: …
10:29:42 gibi hm yeah that is also a way
10:29:59 sean-k-mooney[m] im not sure if we need to have different behavor for other vnic types like the cyborg ones
10:30:24 sean-k-mooney[m] or baremetal though thtat is used only by ironic
10:31:41 sean-k-mooney[m] i wouold have to look at the spec again but when we are using cyborg provided smart nics neutron still sends the events right?
10:35:06 gibi hm neutron seems to support accelerator-direct with mech_sriov, and direct means a VF so I assume there is plug time events
10:35:51 sean-k-mooney[m] i would assume so too but its not actully mentioned in https://specs.openstack.org/openstack/nova-specs/specs/xena/implemented/sriov-smartnic-support.html
10:38:00 sean-k-mooney[m] gibi so you could either limit this to the case we know work (normal,direct,vhost-user) or you could filet out the case we know wont work (direct-physical,acclerator-direct-physical)
10:38:36 gibi yeah
10:38:45 gibi as the config today needs to be opt-in
10:38:55 gibi I guess opting in to supported vnic types are better
10:39:20 gibi when you say vhost-user why is not enough to simply filter for vnic_type direct?
10:40:03 sean-k-mooney[m] in that case the list would be normal,direct,macvtap,acclerator-direct,vhost-user
10:40:16 sean-k-mooney[m] well direct shoud work right
10:40:47 sean-k-mooney[m] hardware offloaded ovs with ml2/ovs support direct and will send plug time events and the sriov nic agent should also
10:41:10 sean-k-mooney[m] and vhost-user should also work with ml2/ovs and ml2/odl
10:41:35 sean-k-mooney[m] the sriov nica agent should support macvtap plug time events
10:42:16 sean-k-mooney[m] ml2/ovs should also send them for vdpa
10:42:17 gibi for the deployer probably it is easier to just list vnic_types and not go into details like vhost-user
10:42:30 sean-k-mooney[m] vhost-user is a vnic type
10:42:40 gibi sean-k-mooney[m]: is it?
10:42:43 sean-k-mooney[m] yes
10:42:48 sean-k-mooney[m] its not a vif_type
10:43:22 gibi blob/6d8e830859cd4ac9708701b8e344fdc68cbcaebb/neutron/plugins/ml2/drivers/mech_sriov/mech_driver/mech_driver.py#L164
10:43:28 gibi sorry
10:43:30 gibi wrong buffe4r
10:43:34 gibi https://github.com/openstack/neutron-lib/blob/f01b2e9025d33aeff3bf22ea2568bda036878819/neutron_lib/api/definitions/portbindings.py#L131
10:43:38 sean-k-mooney[m] so i think if you want to hard code it just filter out direct-physical,baremetal and acclerator-direct-physical
10:43:39 gibi se here are the vnic_types
10:44:26 gibi I don't see vhost-user as vnic type in that list
10:45:17 sean-k-mooney[m] oh sorry maybe your right i have not looked at dpdk in 2 years or more
10:45:53 sean-k-mooney[m] i might be miss rememebvring let me check the ml2/driver but i guess its vnic_normal
10:46:04 gibi I thin it is mapped to normal yes
10:46:22 gibi anyhow I think we are in agreement to have this filtering based on vnic_type
10:46:33 gibi I think I will amend the current patch with that
10:46:47 sean-k-mooney[m] ya you are right it is
10:47:42 sean-k-mooney[m] so we should just skip waiting then for *-physical and baremetal
10:48:38 sean-k-mooney[m] looking at the other vnic_types i dont think vnic_type smartnic is used with ovs or odl
10:48:57 gibi OK, I will discuss this the the downstream folks to and see if they prefer a configurable vnic_type or they are OK with a hardcode
10:48:59 sean-k-mooney[m] i think that is used by ironic
10:49:09 sean-k-mooney[m] ack
10:49:20 gibi yes, smartnic is ironic afaik
10:49:29 gibi so we can filter out that too
10:49:49 sean-k-mooney[m] ya most likely
10:50:10 sean-k-mooney[m] you could make the config an exclude list and default to the set we know wont work
10:50:20 sean-k-mooney[m] actully no
10:50:28 sean-k-mooney[m] that would enable it by default which we do not want
10:51:48 sean-k-mooney[m] ok ill be afk for 20 mins or so chat to you later
10:52:01 gibi ack, thanks!
11:13:14 frickler kashyap: couple of more findings: a) no change with the Nehalem cpu settings patch from clarkb
11:13:30 frickler b) same issue with qemu-6.1 compiled from source
11:13:44 kashyap frickler: Hi
11:14:13 frickler c) the delta doesn'n really increase with large flavors, i.e. with 512M or 1G, the cirros process still stays at 600M
11:15:00 kashyap frickler: The Nehalem thing here is not relevant (unless you're using CentOS9).
11:15:14 kashyap frickler: Good to know that you've actually tested it w/ compiled with source
11:15:24 frickler the latter is likely why this issue isn't more widely seen. it just affects CIs that try to start a larger number of small instances
11:15:44 frickler ... in a limited memory environment
11:17:19 kashyap frickler: Right. So, this is TCG - this is not super amazingly well-tested upstream. Because a lot of folks use hardware accel. That said:
11:18:02 kashyap frickler: Can you please file an upstream QEMU bug here (do you have a GitLab account?) - https://gitlab.com/qemu-project/qemu/-/issues
11:19:19 kashyap frickler: That'll help me investigate the issue with a TCG dev.
11:21:53 kashyap Also, please include the bits you posted yesterday. (https://paste.opendev.org/raw/810150/)
11:22:25 kashyap frickler: I wonder if we can replicate this outside of OpenStack CI: like artificially triggering a script that'll start a ton of CirrOS instances?
11:23:01 frickler kashyap: I'll do the bug report, though likely not today, I'll let you know then
11:23:31 kashyap Thanks! Do mention the buggy version where you saw it first. And also the 6.1 compiled-from-source test.
11:23:34 kashyap It'll help with bisecting.
11:24:08 frickler kashyap: well I replicated with a local devstack deployment. I can also try to just create an instance with virt-manager
11:24:34 kashyap Yes, that'll be more preferable, if possible.
11:25:24 frickler kashyap: ok, thx for your feedback so far
11:25:55 kashyap No problem. These TCG bugs (if it is indeed a bug) are hard to suss out.
11:37:05 sean-k-mooney1 frickler: you are seeing this just with a normal boot right
11:37:15 sean-k-mooney1 you dont need to boot many vms to trigger it
11:37:27 sean-k-mooney1 you are seeing the large memory usage with just a singel instnace
11:37:44 sean-k-mooney1 so this should not be hard to replicate right
11:37:53 kashyap sean-k-mooney1: I don't think it's just one boot. He said "CIs that try to start a larger no. of small instances"
11:37:57 sean-k-mooney1 frickler: out of interest do you have swap avaiable in these hosts
11:38:11 kashyap sean-k-mooney1: Good question ;-) The "ghost of swap"...
11:38:29 sean-k-mooney1 kashyap: its only an issue for cis because those small instance that used to fit nolonger do
11:38:56 sean-k-mooney1 kashyap: when i first spoke to frickler about this i think they mentioned it hapens for any small vms created
11:39:46 sean-k-mooney i.e. in ci the vm used to take say 128mb for ram is now using 600 so we cant run 4 of them in parallel anymore
11:40:36 sean-k-mooney so what im wondiring is in environment without swap are we seeing more resident memory usage
11:40:42 frickler sean-k-mooney: yes, I see this with a single instance. the CI example is just where we noticed it first, with jobs OOMing with tempest running parallel tests
11:41:35 sean-k-mooney frickler: we had a very weird customer issue where we saw python process have very large resident memory usage when no swap was presnet but when it was allocated they did not have high memory usage and also did not use any swap
11:42:27 sean-k-mooney it was like have swap avaiabel stop the memory allcoator preallcoating the memory
11:42:59 frickler hmm, indeed I have no swap on my test host. but we do have swap enabled on CI instances
11:43:25 sean-k-mooney ah ok i was going to say could you add a 1G swap file temporay to your devstack and see if it change behavior
11:43:35 sean-k-mooney if its in the ci then no point its not relatted
12:49:29 bauzas (late) reminder: final PTG day for nova sessions starting in 12 mins at https://www.openstack.org/ptg/rooms/newton
12:57:39 sean-k-mooney dansmith: for the health check you want me to support http over tcp ranther then just a tcp socket right. assuming yes im assuming if it make sense ot make this a real wsgi application or just use https://eventlet.net/doc/modules/wsgi.html#eventlet.wsgi.server to call a binay specifci health check function

Earlier   Later