| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-10-22 | |||
| 10:28:37 | sean-k-mooney[m] | if we make the config option a list of vnic_types to wait for on hard reboot we just do | |
| 10:28:59 | sean-k-mooney[m] | if vif.vnic_type in CONF.wait_on_reboot: … | |
| 10:29:42 | gibi | hm yeah that is also a way | |
| 10:29:59 | sean-k-mooney[m] | im not sure if we need to have different behavor for other vnic types like the cyborg ones | |
| 10:30:24 | sean-k-mooney[m] | or baremetal though thtat is used only by ironic | |
| 10:31:41 | sean-k-mooney[m] | i wouold have to look at the spec again but when we are using cyborg provided smart nics neutron still sends the events right? | |
| 10:35:06 | gibi | hm neutron seems to support accelerator-direct with mech_sriov, and direct means a VF so I assume there is plug time events | |
| 10:35:51 | sean-k-mooney[m] | i would assume so too but its not actully mentioned in https://specs.openstack.org/openstack/nova-specs/specs/xena/implemented/sriov-smartnic-support.html | |
| 10:38:00 | sean-k-mooney[m] | gibi so you could either limit this to the case we know work (normal,direct,vhost-user) or you could filet out the case we know wont work (direct-physical,acclerator-direct-physical) | |
| 10:38:36 | gibi | yeah | |
| 10:38:45 | gibi | as the config today needs to be opt-in | |
| 10:38:55 | gibi | I guess opting in to supported vnic types are better | |
| 10:39:20 | gibi | when you say vhost-user why is not enough to simply filter for vnic_type direct? | |
| 10:40:03 | sean-k-mooney[m] | in that case the list would be normal,direct,macvtap,acclerator-direct,vhost-user | |
| 10:40:16 | sean-k-mooney[m] | well direct shoud work right | |
| 10:40:47 | sean-k-mooney[m] | hardware offloaded ovs with ml2/ovs support direct and will send plug time events and the sriov nic agent should also | |
| 10:41:10 | sean-k-mooney[m] | and vhost-user should also work with ml2/ovs and ml2/odl | |
| 10:41:35 | sean-k-mooney[m] | the sriov nica agent should support macvtap plug time events | |
| 10:42:16 | sean-k-mooney[m] | ml2/ovs should also send them for vdpa | |
| 10:42:17 | gibi | for the deployer probably it is easier to just list vnic_types and not go into details like vhost-user | |
| 10:42:30 | sean-k-mooney[m] | vhost-user is a vnic type | |
| 10:42:40 | gibi | sean-k-mooney[m]: is it? | |
| 10:42:43 | sean-k-mooney[m] | yes | |
| 10:42:48 | sean-k-mooney[m] | its not a vif_type | |
| 10:43:22 | gibi | blob/6d8e830859cd4ac9708701b8e344fdc68cbcaebb/neutron/plugins/ml2/drivers/mech_sriov/mech_driver/mech_driver.py#L164 | |
| 10:43:28 | gibi | sorry | |
| 10:43:30 | gibi | wrong buffe4r | |
| 10:43:34 | gibi | https://github.com/openstack/neutron-lib/blob/f01b2e9025d33aeff3bf22ea2568bda036878819/neutron_lib/api/definitions/portbindings.py#L131 | |
| 10:43:38 | sean-k-mooney[m] | so i think if you want to hard code it just filter out direct-physical,baremetal and acclerator-direct-physical | |
| 10:43:39 | gibi | se here are the vnic_types | |
| 10:44:26 | gibi | I don't see vhost-user as vnic type in that list | |
| 10:45:17 | sean-k-mooney[m] | oh sorry maybe your right i have not looked at dpdk in 2 years or more | |
| 10:45:53 | sean-k-mooney[m] | i might be miss rememebvring let me check the ml2/driver but i guess its vnic_normal | |
| 10:46:04 | gibi | I thin it is mapped to normal yes | |
| 10:46:22 | gibi | anyhow I think we are in agreement to have this filtering based on vnic_type | |
| 10:46:33 | gibi | I think I will amend the current patch with that | |
| 10:46:47 | sean-k-mooney[m] | ya you are right it is | |
| 10:47:42 | sean-k-mooney[m] | so we should just skip waiting then for *-physical and baremetal | |
| 10:48:38 | sean-k-mooney[m] | looking at the other vnic_types i dont think vnic_type smartnic is used with ovs or odl | |
| 10:48:57 | gibi | OK, I will discuss this the the downstream folks to and see if they prefer a configurable vnic_type or they are OK with a hardcode | |
| 10:48:59 | sean-k-mooney[m] | i think that is used by ironic | |
| 10:49:09 | sean-k-mooney[m] | ack | |
| 10:49:20 | gibi | yes, smartnic is ironic afaik | |
| 10:49:29 | gibi | so we can filter out that too | |
| 10:49:49 | sean-k-mooney[m] | ya most likely | |
| 10:50:10 | sean-k-mooney[m] | you could make the config an exclude list and default to the set we know wont work | |
| 10:50:20 | sean-k-mooney[m] | actully no | |
| 10:50:28 | sean-k-mooney[m] | that would enable it by default which we do not want | |
| 10:51:48 | sean-k-mooney[m] | ok ill be afk for 20 mins or so chat to you later | |
| 10:52:01 | gibi | ack, thanks! | |
| 11:13:14 | frickler | kashyap: couple of more findings: a) no change with the Nehalem cpu settings patch from clarkb | |
| 11:13:30 | frickler | b) same issue with qemu-6.1 compiled from source | |
| 11:13:44 | kashyap | frickler: Hi | |
| 11:14:13 | frickler | c) the delta doesn'n really increase with large flavors, i.e. with 512M or 1G, the cirros process still stays at 600M | |
| 11:15:00 | kashyap | frickler: The Nehalem thing here is not relevant (unless you're using CentOS9). | |
| 11:15:14 | kashyap | frickler: Good to know that you've actually tested it w/ compiled with source | |
| 11:15:24 | frickler | the latter is likely why this issue isn't more widely seen. it just affects CIs that try to start a larger number of small instances | |
| 11:15:44 | frickler | ... in a limited memory environment | |
| 11:17:19 | kashyap | frickler: Right. So, this is TCG - this is not super amazingly well-tested upstream. Because a lot of folks use hardware accel. That said: | |
| 11:18:02 | kashyap | frickler: Can you please file an upstream QEMU bug here (do you have a GitLab account?) - https://gitlab.com/qemu-project/qemu/-/issues | |
| 11:19:19 | kashyap | frickler: That'll help me investigate the issue with a TCG dev. | |
| 11:21:53 | kashyap | Also, please include the bits you posted yesterday. (https://paste.opendev.org/raw/810150/) | |
| 11:22:25 | kashyap | frickler: I wonder if we can replicate this outside of OpenStack CI: like artificially triggering a script that'll start a ton of CirrOS instances? | |
| 11:23:01 | frickler | kashyap: I'll do the bug report, though likely not today, I'll let you know then | |
| 11:23:31 | kashyap | Thanks! Do mention the buggy version where you saw it first. And also the 6.1 compiled-from-source test. | |
| 11:23:34 | kashyap | It'll help with bisecting. | |
| 11:24:08 | frickler | kashyap: well I replicated with a local devstack deployment. I can also try to just create an instance with virt-manager | |
| 11:24:34 | kashyap | Yes, that'll be more preferable, if possible. | |
| 11:25:24 | frickler | kashyap: ok, thx for your feedback so far | |
| 11:25:55 | kashyap | No problem. These TCG bugs (if it is indeed a bug) are hard to suss out. | |
| 11:37:05 | sean-k-mooney1 | frickler: you are seeing this just with a normal boot right | |
| 11:37:15 | sean-k-mooney1 | you dont need to boot many vms to trigger it | |
| 11:37:27 | sean-k-mooney1 | you are seeing the large memory usage with just a singel instnace | |
| 11:37:44 | sean-k-mooney1 | so this should not be hard to replicate right | |
| 11:37:53 | kashyap | sean-k-mooney1: I don't think it's just one boot. He said "CIs that try to start a larger no. of small instances" | |
| 11:37:57 | sean-k-mooney1 | frickler: out of interest do you have swap avaiable in these hosts | |
| 11:38:11 | kashyap | sean-k-mooney1: Good question ;-) The "ghost of swap"... | |
| 11:38:29 | sean-k-mooney1 | kashyap: its only an issue for cis because those small instance that used to fit nolonger do | |
| 11:38:56 | sean-k-mooney1 | kashyap: when i first spoke to frickler about this i think they mentioned it hapens for any small vms created | |
| 11:39:46 | sean-k-mooney | i.e. in ci the vm used to take say 128mb for ram is now using 600 so we cant run 4 of them in parallel anymore | |
| 11:40:36 | sean-k-mooney | so what im wondiring is in environment without swap are we seeing more resident memory usage | |
| 11:40:42 | frickler | sean-k-mooney: yes, I see this with a single instance. the CI example is just where we noticed it first, with jobs OOMing with tempest running parallel tests | |
| 11:41:35 | sean-k-mooney | frickler: we had a very weird customer issue where we saw python process have very large resident memory usage when no swap was presnet but when it was allocated they did not have high memory usage and also did not use any swap | |
| 11:42:27 | sean-k-mooney | it was like have swap avaiabel stop the memory allcoator preallcoating the memory | |
| 11:42:59 | frickler | hmm, indeed I have no swap on my test host. but we do have swap enabled on CI instances | |
| 11:43:25 | sean-k-mooney | ah ok i was going to say could you add a 1G swap file temporay to your devstack and see if it change behavior | |
| 11:43:35 | sean-k-mooney | if its in the ci then no point its not relatted | |
| 12:49:29 | bauzas | (late) reminder: final PTG day for nova sessions starting in 12 mins at https://www.openstack.org/ptg/rooms/newton | |
| 12:57:39 | sean-k-mooney | dansmith: for the health check you want me to support http over tcp ranther then just a tcp socket right. assuming yes im assuming if it make sense ot make this a real wsgi application or just use https://eventlet.net/doc/modules/wsgi.html#eventlet.wsgi.server to call a binay specifci health check function | |
| 12:58:17 | dansmith | sean-k-mooney: yes http and I'd keep it uuber simple (so the latter) | |
| 12:58:45 | sean-k-mooney | ok | |
| 13:00:24 | sean-k-mooney | brb just going to make a coffee and ill join | |
| 13:02:23 | bauzas | nova session started | |
| 13:03:08 | dansmith | sean-k-mooney: out of curiosity, does haproxy support some sort of bare tcp socket health check/ | |
| 13:03:12 | dansmith | I would kinda expect not | |
| 13:03:44 | dansmith | I thought even systemd wanted http, but can use a script too | |
| 13:10:03 | sean-k-mooney | dansmith: i think i tcan but orginaly haproxy was not part of my orignal usecases | |
| 13:10:14 | dansmith | okay | |
| 13:11:04 | sean-k-mooney | i was orginally thining of this as a camand/contol interface with comand objects echanged more like the rpc bus | |
| 13:11:39 | sean-k-mooney | with nc as or nova-manage as the cli | |