| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-03-23 | |||
| 12:42:57 | sean-k-mooney | which sice we dont over subscibe means we could effectivly enforce min | |
| 12:43:10 | gibi | yes, that is possible here too | |
| 12:43:19 | sean-k-mooney | i dont think ovs support min pps | |
| 12:43:27 | sean-k-mooney | even for sriov that is a challange | |
| 12:45:02 | gibi | three are some differences with BW. The packet rate inventory seems to be directionless in nature | |
| 12:45:48 | gibi | and in case of OVS it more belongs to the softswitch level than to the bridge level | |
| 12:46:15 | gibi | and for SRIOV packet rate is not really a limiting factor | |
| 12:46:19 | gibi | afaik | |
| 12:46:23 | sean-k-mooney | the same is true for pps form an ovs point of view | |
| 12:46:40 | sean-k-mooney | bridges are meaningless for the most part | |
| 12:47:03 | sean-k-mooney | gibi: it kind fo still is | |
| 12:47:23 | sean-k-mooney | even with sriov 64b packets will limit the bandwith fo the nic | |
| 12:47:44 | sean-k-mooney | typeically they can handel 1 or 2 VF at that rate but not multiple | |
| 12:48:06 | gibi | sean-k-mooney: interesting, I did not know that | |
| 12:49:06 | gibi | so the pps will be the limit before the bandwidth if there is a lot of VFs and the packets are small? | |
| 12:49:16 | sean-k-mooney | normlaly the north south driction can handel full line rate at 64b packets for 10G thats about 14.4mpps | |
| 12:49:53 | sean-k-mooney | so it depens where the limit will come in would be vm to vm traffic on the same host | |
| 12:50:22 | sean-k-mooney | the other place its a proble is prot mirroing | |
| 12:50:35 | sean-k-mooney | our running vf in trusted mode | |
| 12:51:27 | sean-k-mooney | setting a vf to trusted mode consumes a lot of bandwith ignorign the security implications espcially if there are multipel of them | |
| 12:51:54 | sean-k-mooney | it forcs the nic to copy a subset of packets to multiple vfs | |
| 12:53:24 | gibi | thanks, these make sense. This also points to the direction that keeping bandwidth and packet rate inventory configurable as it is really deployment and traffic pattern dependent | |
| 12:53:27 | sean-k-mooney | gibi: you tend to hit the pcie bottelneck before the pps bottelneck in most cases but if you wnat to get full line rate with sriov and small packet i know our nfv folks had to a lot fo tuning | |
| 12:54:12 | gibi | good to know | |
| 12:55:16 | sean-k-mooney | https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/network_functions_virtualization_planning_and_configuration_guide/part-sriov-nfv-configuration#sect-configuring-sriov | |
| 12:55:42 | sean-k-mooney | they are using isolcpus=1-19,21-39 on the kernel in part 4 | |
| 12:55:49 | sean-k-mooney | 6.2.4 | |
| 12:57:13 | gibi | yeah we also separate tenat cpus from host cpus and from ovs cpus | |
| 12:57:20 | gibi | with isolcpu | |
| 12:57:40 | sean-k-mooney | really i dont think people should use that on non realtime hosts | |
| 12:57:59 | sean-k-mooney | its dprecated upstream in the kernel and its kind fo a blunt hammer | |
| 12:58:04 | gibi | isolcpu helps avouding host cpu load to affect the pinned VMs | |
| 12:58:15 | sean-k-mooney | in many cases you can achive the same thing without using it | |
| 12:58:23 | sean-k-mooney | it does yes | |
| 12:58:42 | sean-k-mooney | but people enable it and then deploy unpinned vms on the host | |
| 12:58:43 | gibi | cgroups would be the alternative I guess | |
| 12:59:01 | sean-k-mooney | yes tuned has support | |
| 12:59:29 | sean-k-mooney | we wanted to entirely remove supprot for isolcpus form our product and move people to use tuned instead | |
| 12:59:40 | sean-k-mooney | but we have not been able to convice the nfv dfg to do that yet | |
| 13:00:20 | sean-k-mooney | https://github.com/redhat-performance/tuned/blob/master/profiles/cpu-partitioning/cpu-partitioning-variables.conf | |
| 13:00:50 | sean-k-mooney | you can use tuned to configure the cgoups for you including isolating the cores form irqs | |
| 13:01:18 | sean-k-mooney | thats what we use by default for non realtime hosts now | |
| 13:03:44 | gibi | I see | |
| 13:04:55 | sean-k-mooney | in the cgroup way you can prevent the kernel schduling vms to the core indepently of balancing the vms on cores | |
| 13:05:03 | sean-k-mooney | isolcpus does both | |
| 13:05:28 | sean-k-mooney | so you can use the tuned/cgrop method with floating vms too not jsut pinned | |
| 13:06:22 | sean-k-mooney | but isolcpu does a few things that you cant quite do form userspace fully or at least with tuned at this point so they still use and support itn for nfv usecases | |
| 13:06:30 | gibi | I see. I think we only support pinned vms by default | |
| 13:06:48 | sean-k-mooney | that both complicates and simplcies your life :) | |
| 13:06:53 | gibi | yeah | |
| 13:06:55 | gibi | soo telco | |
| 13:07:01 | gibi | :) | |
| 13:07:04 | sean-k-mooney | complciates becasue numa and simplfies because only numa | |
| 13:07:28 | gibi | yeah | |
| 13:07:44 | sean-k-mooney | if you dont have to deal with peopel mixing numa and non numa vms its nice | |
| 13:08:25 | gibi | yes we sort of force numa with pinned cpus and huge pages | |
| 13:19:59 | openstackgerrit | Merged openstack/nova master: Dynamically archive FK related records in archive_deleted_rows https://review.opendev.org/c/openstack/nova/+/773834 | |
| 13:29:58 | bauzas | folks, if you need to discuss with kashyap, he's off IRC as he has an Internet issue with his ISP | |
| 13:31:08 | sean-k-mooney | i did have one thing that came up last night | |
| 13:32:06 | sean-k-mooney | stars here http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2021-03-22.log.html#t2021-03-22T19:34:19 | |
| 13:32:49 | sean-k-mooney | basically qemu change the cpu flags that are set in a specific cpu model and if you dont use version machine types that will break live migration | |
| 13:33:29 | sean-k-mooney | we may or may not be able to work around that in nova via the migration xml | |
| 13:34:25 | sean-k-mooney | but the end effect is that eypc-ibrs when the vm was booted on the source host nolgere results in the same cpu feature enabel if you boot a vm on the same host or a different one with the same libvirt xml | |
| 13:34:40 | sean-k-mooney | wehn you are using unversion machine types | |
| 13:34:40 | bauzas | sean-k-mooney: kashyap dunno when he will be back | |
| 13:35:08 | sean-k-mooney | its not supper urgent at least for me | |
| 13:35:58 | sean-k-mooney | but rouk has already confrim that setting an explit machine type in nova or using kashyap recent patch to allow removing feature flags both cannot fix the issue | |
| 13:36:12 | sean-k-mooney | since neither ake effect on live migration | |
| 13:36:53 | sean-k-mooney | it wont break use downstream since we use versioned machine types but it will break anyone that uses an unversioned machine type after a qemu update | |
| 13:37:22 | sean-k-mooney | that include our downstream customer if they set an unversion machine type in the nova config or in the glance image | |
| 14:30:47 | lyarwood | gibi / stephenfin ; https://review.opendev.org/c/openstack/nova/+/733627 - could either of you look at this sometime this week, wsgi fix when launched by mod_wsgi | |
| 14:33:23 | gibi | lyarwood: ack | |
| 14:33:27 | lyarwood | thanks | |
| 14:39:14 | stephenfin | done | |
| 14:43:45 | gibi | stephenfin won | |
| 14:54:49 | bauzas | gibi: taking the semaphore for bug triaging | |
| 14:54:57 | gibi | bauzas: given | |
| 14:55:07 | bauzas | I see very old open bugs that aren't triaged | |
| 14:55:19 | bauzas | like https://bugs.launchpad.net/nova/+bug/1463631 | |
| 14:55:19 | openstack | Launchpad bug 1463631 in grenade "60_nova/resources.sh:106:ping_check_public fails intermittently" [Undecided,Confirmed] | |
| 14:55:28 | bauzas | any reason to leave them in such state ? | |
| 15:13:59 | bauzas | lyarwood: hmm, very interesting corner case https://bugs.launchpad.net/nova/+bug/1920886 | |
| 15:14:00 | openstack | Launchpad bug 1920886 in OpenStack Compute (nova) "ImageNotFound error occurs after live migration" [Undecided,New] | |
| 15:22:37 | gibi | bauzas: https://bugs.launchpad.net/nova/+bug/1463631 has a not too old comment from lyarwood that he saw it again | |
| 15:22:38 | openstack | Launchpad bug 1463631 in grenade "60_nova/resources.sh:106:ping_check_public fails intermittently" [Undecided,Confirmed] | |
| 15:24:07 | lyarwood | sorry was flooded with pings downstream | |
| 15:25:33 | lyarwood | my god, a well written bug report | |
| 15:25:42 | lyarwood | I think I might faint | |
| 15:35:05 | lyarwood | gibi / bauzas ; re the grenade bug, yeah I couldn't make any sense of that, we could drop nova if you don't think we can help | |
| 15:38:03 | gibi | lyarwood: I've just run a logstash query http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Couldn't%20ping%20server%5C%22 and in the last 7 days we had 5 hits so this is active | |
| 15:39:19 | kashyap | bauzas: Thanks! My network has been super flaky; just reading the scrollback | |
| 15:39:25 | openstackgerrit | Lee Yarwood proposed openstack/nova master: compute: Reject requests to commit intermediary snapshot of an inactive instance https://review.opendev.org/c/openstack/nova/+/781138 | |
| 15:39:36 | kashyap | sean-k-mooney: Hi, looking at the chat log | |
| 15:40:21 | kashyap | Assuming I'm still connected here, I'm getting a "We’re having trouble finding that site." | |
| 15:45:31 | bauzas | lyarwood: ack, will put the grenade bug to be invalid for nova | |
| 15:46:33 | rouk | sean-k-mooney: so yeah, updating cpu_map nor machine type worked. | |
| 15:46:54 | rouk | im just building images with qemu 3.1 i guess now as the last ditch fix. | |
| 15:58:43 | openstackgerrit | Ruby Loo proposed openstack/nova master: Allow plus sign in flavor ids https://review.opendev.org/c/openstack/nova/+/782545 | |
| 16:09:31 | openstackgerrit | Ruby Loo proposed openstack/nova master: Allow plus sign in flavor ids https://review.opendev.org/c/openstack/nova/+/782545 | |
| 16:56:13 | openstackgerrit | Merged openstack/nova master: Initialize global data separately and run_once in WSGI app init https://review.opendev.org/c/openstack/nova/+/733627 | |