| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-03-22 | |||
| 16:34:58 | sean-k-mooney | melwitt: ack, my view is the proported optimisation was never valid or functional in any meaningful way and it was broken by design | |
| 16:35:54 | sean-k-mooney | i think alex agreed with the design part in his last comment but was unsure if the optimiasation acutlly provided a performance imporment | |
| 16:36:20 | manuvakery1 | Hi. what could be a acceptable load average on compute host. I can see its consistently between 30-40 when i stress the vm with same no of cores as compute host | |
| 16:36:38 | sean-k-mooney | and stephenfin was concerend about a regression in functionality and belive the optimisation may have been valid in some cases | |
| 16:37:16 | melwitt | ah, ok | |
| 16:37:35 | sean-k-mooney | if you have 40 cores then a load average of 40 means you are fully utilising the system and not over stressing the cpus | |
| 16:37:45 | sean-k-mooney | manuvakery1:^ | |
| 16:38:02 | sean-k-mooney | so a load average fo <= number of cores meens you are below or at capasity | |
| 16:38:19 | manuvakery1 | its a 48 core host | |
| 16:38:25 | sean-k-mooney | if you exceed it it means there is contention between prcoess to execute cpu instrucutions | |
| 16:38:39 | sean-k-mooney | manuvakery1: 30-40 is prefectly accpable in that case | |
| 16:38:52 | manuvakery1 | thanks sean-k-mooney | |
| 16:39:11 | sean-k-mooney | if the load avergae exceed core count it means the vms are under perferoming because they are cpu starved | |
| 16:39:36 | sean-k-mooney | that may or may not matter depening on your use case but i would not be concerned with your current values | |
| 16:43:58 | manuvakery1 | that means when using cpu over commit, I can expect high load average and my vms can under perform when all are trying to gather cpu . I am ok if my vms are performing little slow but don't want my host machine to be non responsive | |
| 16:43:59 | kashyap | stephenfin: I take it that when you move content, you're _only_ moving content -- or are you also mixing in little fix-ups? | |
| 16:44:13 | kashyap | stephenfin: E.g. I'm looking at the SEV guide | |
| 16:44:35 | stephenfin | I might fix spellings and messed with the structure but it general the content is the same. I tried to keep major reworks separate | |
| 16:44:41 | sean-k-mooney | manuvakery1: we generally recommend confinging the guest to run on a subset of host cores | |
| 16:44:54 | kashyap | Right; I see that you've also added hyperlinks where you can | |
| 16:45:06 | sean-k-mooney | manuvakery1: using vcpu_pin_set before train or cpu_share_set and cpu_dedicated_set after train | |
| 16:45:07 | kashyap | E.g. on line-17 I see you've added the link to _deploying-sev-capable-infrastructure | |
| 16:45:18 | sean-k-mooney | manuvakery1: that way you can ensure that the host os never locks up | |
| 16:45:24 | kashyap | stephenfin: Okay; figured as much - major rework separate. Thx | |
| 16:45:49 | sean-k-mooney | manuvakery1: our general recommendateion is to reserve at least the first core from each numa node or for the OS to use | |
| 16:46:15 | manuvakery1 | sean-k-mooney: ok. I will try that | |
| 16:46:31 | kashyap | stephenfin: Err, thinko above: it's not a link, but an "anchor". | |
| 16:46:58 | sean-k-mooney | manuvakery1: so assuming you have 2 sockets each wtih 12 cores and 24 hypertreads you weroul ideally reserved cores 0,12,24,36 | |
| 16:48:19 | sean-k-mooney | manuvakery1: before train that would be done with vcpu_pin_set=1-11,13-23,25-35,37-47 | |
| 16:48:50 | sean-k-mooney | after train cpu_shared_set=1-11,13-23,25-35,37-47 | |
| 16:49:37 | sean-k-mooney | vcpu_pin_set was in the [DEFAULT] sechtion and cpu_*_set are in the [compute] section of the nova.conf, this need to be set on each compute node | |
| 16:50:46 | manuvakery1 | sean-k-mooney: I am using train. I will see to it. Thanks for your response | |
| 17:18:39 | MrClayPole | -ll to ensure its no longer active? | |
| 17:18:39 | MrClayPole | We are troubleshooting and issue with our Openstack backup provider (Trilio) and our Storage vender/cinder driver (Zadara). As part of this troubleshooting we've been asked to disable iSCSI multipathing. I can see that this installed as part of the nova deployment in Openstack ansible from the mutipath-tools package. Is it OK to just stop and disable the multipathd services. Then run mutipath -F. Then check with multipath | |
| 17:34:09 | openstackgerrit | Merged openstack/osc-placement master: Include usage in 'inventory list', 'inventory show' https://review.opendev.org/c/openstack/osc-placement/+/743976 | |
| 17:55:29 | sean-k-mooney | lyarwood: whats the status of nova-grenade-multinode | |
| 17:55:51 | sean-k-mooney | did you have a patch to move that to v3? or am i just wishing you did | |
| 17:56:18 | sean-k-mooney | ah https://review.opendev.org/c/openstack/nova/+/778885 | |
| 17:56:47 | sean-k-mooney | is ^ going to happen this cycle? | |
| 18:19:49 | ricolin | stephenfin, hi about https://review.opendev.org/c/openstack/nova/+/781210 | |
| 18:21:35 | ricolin | I update some logs in comments, for what I can tell it only happen to bionic environment from aarch64. it disappear once I moved to focal | |
| 18:32:44 | openstackgerrit | Merged openstack/nova stable/victoria: Add config parameter 'live_migration_scheme' to live migration with tls guide https://review.opendev.org/c/openstack/nova/+/781211 | |
| 18:36:34 | stephenfin | ricolin: That looks like a bug. Can you open a bug on launchpad and I'll take a look tomorrow? | |
| 18:36:52 | stephenfin | ricolin: Referring to this http://paste.openstack.org/show/803788/ | |
| 18:47:20 | ricolin | stephenfin, thanks I will open one for both errors I found | |
| 19:06:36 | sean-k-mooney | that a python2 vs python3 issue i think | |
| 19:07:06 | sean-k-mooney | ricolin: master is not intended to run on bionic by the way | |
| 19:07:36 | sean-k-mooney | it might be compatiable but its not part of the offical testing runtimes anymore | |
| 19:08:04 | sean-k-mooney | we have 1/2 jobs that use it for reasons but the arm jobs should be on focal | |
| 19:09:08 | sean-k-mooney | ricolin: ussuri was the last release to use bionic | |
| 19:09:41 | sean-k-mooney | all jobs for victoria wallaby and master shoudl be useding 20.04/focal if they are ubuntu based | |
| 19:12:11 | sean-k-mooney | ricolin: is there a reason the aarch64 job was using bionic | |
| 19:14:06 | sean-k-mooney | ricolin: stephenfin we have seen issue with libs incorrectly monkeypactching codecs.py in the past | |
| 19:15:35 | sean-k-mooney | this looks like we are hitting https://github.com/openstack/nova/commit/b862f6ff35d1611d0d63623a6254fc889012bfb9 | |
| 19:16:20 | sean-k-mooney | where blockdiag was messing with codecs.getreader | |
| 19:34:19 | rouk | after updating to the latest stable/ussuri nova version, live migrations are broken due to cpu features being added that shouldnt be. | |
| 19:35:04 | rouk | i have merged the patches in master which allow disabling cpu features, but even that doesnt make migrations work. | |
| 19:35:30 | rouk | even after disabling these features, live migrate fights back with libvirt.libvirtError: operation failed: guest CPU doesn't match specification: extra features: npt,nrip-save | |
| 19:36:53 | rouk | how do i stop nova from adding features on migration and breaking things? | |
| 19:40:49 | rouk | seems like a pretty big breaking change that made it into a patch... | |
| 19:42:54 | sean-k-mooney | rouk: that is not a breaking change in nova | |
| 19:43:13 | sean-k-mooney | that is a breaking change in your kernel | |
| 19:43:31 | rouk | it only appeared after patching nova a week ago to stable/ussuri... | |
| 19:43:31 | sean-k-mooney | rouk: nova does not add those features | |
| 19:43:41 | rouk | and is present on old-kernel boxes. | |
| 19:44:01 | openstackgerrit | Merged openstack/nova stable/train: Prevent archiving of pci_devices records because of 'instance_uuid' https://review.opendev.org/c/openstack/nova/+/760978 | |
| 19:44:03 | sean-k-mooney | nova does not add those features | |
| 19:44:19 | sean-k-mooney | rouk: so i think you got some other change you did not intend | |
| 19:44:44 | rouk | it never used to, no. but these new changes that came in to "fix" live migrations, are adding features | |
| 19:45:06 | sean-k-mooney | which change are you refering too | |
| 19:45:16 | rouk | sec while i grab the commit | |
| 19:49:00 | sean-k-mooney | this ? https://github.com/openstack/nova/commit/b6c473159ec45e0aa715edd45cde28f77484a5f7 | |
| 19:49:24 | rouk | there was one a bit earlier | |
| 19:49:48 | sean-k-mooney | thre is no nova code to add those extrapsec explitly | |
| 19:49:57 | sean-k-mooney | *extra features | |
| 19:50:21 | sean-k-mooney | can you share how you have configured the cpu_mode/cpu_model/extra cpu flags | |
| 19:50:28 | sean-k-mooney | in your nova.conf | |
| 19:50:57 | rouk | i have model, i added extra flags to try and fix this, as i merged support for - syntax to remove features. | |
| 19:51:09 | rouk | but, just epyc-ibpb | |
| 19:51:35 | sean-k-mooney | are all you servers amd eypc ? | |
| 19:51:38 | rouk | yep. | |
| 19:52:31 | rouk | https://patchwork.kernel.org/project/qemu-devel/patch/20190121155051.5628-1-vkuznets@redhat.com/ | |
| 19:52:44 | rouk | which, this happened a while ago, which added these as features retoactively in qemu for the model. | |
| 19:52:55 | sean-k-mooney | thise appears to be the defintion of that model | |
| 19:52:56 | sean-k-mooney | http://paste.openstack.org/show/803793/ | |
| 19:53:17 | rouk | and then this got picked up by nova, which then added these as required features on migrate | |
| 19:53:36 | sean-k-mooney | ok so this is not a nova bug so | |
| 19:53:47 | sean-k-mooney | those requiremtns are coming form libvirt | |
| 19:54:03 | rouk | so how do we stop them from retoactively being added on migrate from nova? | |
| 19:54:12 | sean-k-mooney | did you update the qemu/libvirt version when you updated ussuir | |
| 19:54:29 | sean-k-mooney | have you confrim that is what is happening | |
| 19:54:36 | sean-k-mooney | do the vms actully have them? | |
| 19:54:52 | rouk | we were on 4.0+ (where this qemu change happened) the whole time during ussuri, migrations only broke recently. | |
| 19:54:52 | sean-k-mooney | if hte vm was hard rebooted it could have had teh feature exposted to it | |
| 19:55:06 | rouk | hard rebooting the vm does fix it, by adding the feature | |
| 19:55:10 | rouk | i dont want to reboot an entire cloud. | |
| 19:55:37 | sean-k-mooney | fair but do you have the nova fix i mentioned above | |
| 19:55:51 | sean-k-mooney | https://github.com/openstack/nova/commit/b6c473159ec45e0aa715edd45cde28f77484a5f7 | |
| 19:56:05 | rouk | yes, i am on stable/ussuri as of a week ago. | |
| 19:56:11 | rouk | built from git. | |