| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-02-03 | |||
| 12:09:17 | sean-k-mooney | bauzas: we are expictly only supporting this when you use cpu_dedicated_set right and not vcpu_pin_set | |
| 12:09:29 | bauzas | correct | |
| 12:09:48 | bauzas | well, checking my code | |
| 12:10:01 | sean-k-mooney | do we want to raise a config error like dansmith is doing in the stable uuid series | |
| 12:10:15 | sean-k-mooney | i.e. if you enable this but also have vcpu_pin_set defiend | |
| 12:10:39 | sean-k-mooney | or dont have cpu_dedicated_set defiend | |
| 12:10:58 | bauzas | good question, I need to think about it | |
| 12:11:14 | sean-k-mooney | there are extra edgecases that only come into play if you use vcpu_pin_set | |
| 12:11:18 | sean-k-mooney | which is why im asking | |
| 12:11:34 | bauzas | https://review.opendev.org/c/openstack/nova/+/868237/6/nova/virt/libvirt/cpu/api.py | |
| 12:11:37 | sean-k-mooney | specificaly related to cpu_thread_policy=isolate on a host with hyperthreading | |
| 12:11:55 | bauzas | here, we only check get_decicated_set() | |
| 12:12:15 | bauzas | but we try to look the numa_topo blob from the instance | |
| 12:12:47 | sean-k-mooney | ok so that is not going to block it | |
| 12:12:49 | bauzas | if the instance is not having a numa_topo blob, we could have an exception, shit | |
| 12:13:02 | sean-k-mooney | well we shoudl not have an expction | |
| 12:13:06 | bauzas | as I directly call the subfield from numa_topo | |
| 12:13:07 | sean-k-mooney | we should do nothing | |
| 12:13:12 | bauzas | what's the default ? | |
| 12:13:22 | bauzas | i need to check the object default values | |
| 12:13:38 | sean-k-mooney | oh you ment it will currently raise | |
| 12:13:39 | bauzas | for the standard instance typo | |
| 12:13:40 | sean-k-mooney | yes it will | |
| 12:13:45 | sean-k-mooney | the default is None | |
| 12:13:52 | bauzas | yeah | |
| 12:14:03 | bauzas | my code is wrong for standard non-numa instances | |
| 12:14:05 | sean-k-mooney | so right now this will casue an atribute error if there is no numa toplogy | |
| 12:14:10 | sean-k-mooney | yep | |
| 12:14:16 | bauzas | correct, I need to fix it | |
| 12:15:01 | sean-k-mooney | but also we shoudl add a check in init host in the driver to see if CONF.libvirt.cpu_power_management is enabled and cpu_dedicated_set is not defined | |
| 12:15:32 | sean-k-mooney | and raise an InvalidConfiguration exception | |
| 12:16:08 | bauzas | we haven't said it in the spec but that looks ok to me | |
| 12:16:37 | sean-k-mooney | well the old way of doing cpu pinning is deprecated for removal | |
| 12:17:24 | sean-k-mooney | and the old way ( if the vm has cpu_thread_policy=isolate and the host has hyperthread) claims 2 host cpus per guest cpu | |
| 12:17:39 | sean-k-mooney | bauzas: i think your code will handel that but if we want to allow that we need to test it | |
| 12:17:58 | bauzas | I don't want to support the old config | |
| 12:17:58 | sean-k-mooney | so either raise an error or we need to test that in a fucntional test later in the series | |
| 12:18:08 | sean-k-mooney | works for me | |
| 12:18:34 | bauzas | so I'd rather prefer to hardstop at startup if power management is set with legacy pinning config | |
| 12:18:53 | bauzas | hence me saying about the non-discussed in the spec but I'm OK | |
| 12:18:57 | sean-k-mooney | i would like to remove the old config and pining logic next cycle if we can find time to do it | |
| 12:19:28 | sean-k-mooney | bauzas: yep we didnt dicuss it on the spec since we generally for get about the legacy pinning | |
| 12:19:39 | bauzas | cool | |
| 12:19:41 | bauzas | no worries | |
| 12:19:43 | sean-k-mooney | it was ment to be remvoed 2 or 3 releases ago | |
| 12:20:51 | bauzas | np | |
| 12:23:34 | bauzas | sean-k-mooney: one last question, can we both have cpu_dedicated_set and vcpu_pin_set defined ? | |
| 12:23:54 | bauzas | or are they mutually exclusive ? | |
| 12:25:19 | sean-k-mooney | cpu_dedicated_set and vcpu_pin_set are mutally exclsive | |
| 12:25:25 | bauzas | ++ | |
| 12:25:38 | sean-k-mooney | although that is checked later in the code not in the config option definition | |
| 12:25:53 | sean-k-mooney | ones in compute and the others in default | |
| 12:27:02 | sean-k-mooney | bauzas: you can simpley check "if CONF.compute.cpu_dedicated_set is None and CONF.libvirt.cpu_power_management: raise" | |
| 12:30:03 | bauzas | that was my guess | |
| 12:32:10 | sean-k-mooney | im going to swap to downstream stuff for a while unless pinged | |
| 12:32:31 | sean-k-mooney | bauzas: can we try and land dansmith's stable uuid seriese again today | |
| 12:32:50 | bauzas | sean-k-mooney: yup, I can try to take a look again | |
| 12:33:25 | sean-k-mooney | cool im +2 all the way up. dan did some rebases and fix a minior bug so they lost the +2s they had before | |
| 14:23:45 | ralonsoh | sean-k-mooney, hi! Do you know if this is a known error? | |
| 14:23:46 | ralonsoh | https://zuul.opendev.org/t/openstack/build/0d9af8c4e906422a9e4a27b1b849f315 | |
| 14:23:55 | ralonsoh | This is the second recheck with this error | |
| 14:25:34 | sean-k-mooney | no thats not a know error | |
| 14:25:42 | sean-k-mooney | although the volume tests can be flaky | |
| 14:26:02 | sean-k-mooney | this looks unrealted | |
| 14:29:09 | sean-k-mooney | i think there may have been an OOM issue | |
| 14:29:28 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/0d9af8c4e906422a9e4a27b1b849f315/log/controller/logs/screen-memory_tracker.txt#4768 | |
| 14:29:38 | ralonsoh | right, thanks | |
| 14:30:23 | sean-k-mooney | look like nova api lost access to the db | |
| 14:30:51 | sean-k-mooney | so my guess is mariadb was killed | |
| 14:31:03 | sean-k-mooney | same for keystone it los db connection | |
| 14:31:06 | ralonsoh | that usually happens with a oom | |
| 14:31:23 | ralonsoh | did you try reducing the parallelism in the functional tests? | |
| 14:31:23 | sean-k-mooney | yep so we might need to add a little more swap to that job | |
| 14:31:37 | sean-k-mooney | its not the funcitonl tests its tempest | |
| 14:31:43 | ralonsoh | right, in tempest | |
| 14:33:20 | sean-k-mooney | its currently 4 which should be ok | |
| 14:33:36 | ralonsoh | we reduced some jobs to 3 or 2 | |
| 14:33:41 | ralonsoh | to avoid this issue | |
| 14:33:47 | sean-k-mooney | i think this is using the default swap of 1G we shoudl set it to 8 | |
| 14:34:01 | sean-k-mooney | we could reduce concernace too but i would try swap first | |
| 14:34:07 | ralonsoh | perfect | |
| 14:34:46 | sean-k-mooney | we crrently set concurancy here https://github.com/openstack/os-vif/blob/master/.zuul.yaml#L23 | |
| 14:35:58 | sean-k-mooney | you can add configure_swap_size: 8192 there as well | |
| 14:36:18 | sean-k-mooney | if that does not work then set concurancy to 3 | |
| 14:36:18 | ralonsoh | should I do in a separate patch? | |
| 14:36:26 | sean-k-mooney | ya lets make it a spereate patch | |
| 14:36:32 | ralonsoh | cool, give me 1 min | |
| 14:36:44 | sean-k-mooney | then we can just recheck yours if it passes once its merged | |
| 14:39:37 | opendevreview | Rodolfo Alonso proposed openstack/os-vif master: Increase the swap size to 8GB in tempest jobs https://review.opendev.org/c/openstack/os-vif/+/872655 | |
| 14:40:45 | sean-k-mooney | ok lets leave that run but we should be able to merge both by the end of the day all going well | |
| 14:41:14 | ralonsoh | that's perfect | |
| 15:39:39 | opendevreview | Elod Illes proposed openstack/nova stable/wallaby: [stable-only] Remove broken sdk job from wallaby https://review.opendev.org/c/openstack/nova/+/871798 | |
| 16:10:47 | spatel | sean-k-mooney did you ever see this error when trying to create vm1 - https://paste.opendev.org/show/bmAj7IrkhbRpMbe14zht/ | |
| 16:13:02 | sean-k-mooney | i have see it if the wsgi timeout or haproxy timeout is set too low and the services are under heavy load | |
| 16:13:36 | spatel | Hmm!! | |
| 16:14:09 | sean-k-mooney | you need to check if the request ever got to the nova api. if it did you need to check if the api responed before the connection closed and if it got to the loadbalance or not | |
| 16:14:10 | spatel | server load is 0.66 average | |
| 16:15:36 | spatel | I can do nova list etc.. which works fine. only vm creation choking up | |
| 16:15:48 | spatel | let me check nova api logs etc.. | |
| 16:17:18 | spatel | HAproxy showing all green health for nova members | |