| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-02-03 | |||
| 12:13:22 | bauzas | i need to check the object default values | |
| 12:13:38 | sean-k-mooney | oh you ment it will currently raise | |
| 12:13:39 | bauzas | for the standard instance typo | |
| 12:13:40 | sean-k-mooney | yes it will | |
| 12:13:45 | sean-k-mooney | the default is None | |
| 12:13:52 | bauzas | yeah | |
| 12:14:03 | bauzas | my code is wrong for standard non-numa instances | |
| 12:14:05 | sean-k-mooney | so right now this will casue an atribute error if there is no numa toplogy | |
| 12:14:10 | sean-k-mooney | yep | |
| 12:14:16 | bauzas | correct, I need to fix it | |
| 12:15:01 | sean-k-mooney | but also we shoudl add a check in init host in the driver to see if CONF.libvirt.cpu_power_management is enabled and cpu_dedicated_set is not defined | |
| 12:15:32 | sean-k-mooney | and raise an InvalidConfiguration exception | |
| 12:16:08 | bauzas | we haven't said it in the spec but that looks ok to me | |
| 12:16:37 | sean-k-mooney | well the old way of doing cpu pinning is deprecated for removal | |
| 12:17:24 | sean-k-mooney | and the old way ( if the vm has cpu_thread_policy=isolate and the host has hyperthread) claims 2 host cpus per guest cpu | |
| 12:17:39 | sean-k-mooney | bauzas: i think your code will handel that but if we want to allow that we need to test it | |
| 12:17:58 | bauzas | I don't want to support the old config | |
| 12:17:58 | sean-k-mooney | so either raise an error or we need to test that in a fucntional test later in the series | |
| 12:18:08 | sean-k-mooney | works for me | |
| 12:18:34 | bauzas | so I'd rather prefer to hardstop at startup if power management is set with legacy pinning config | |
| 12:18:53 | bauzas | hence me saying about the non-discussed in the spec but I'm OK | |
| 12:18:57 | sean-k-mooney | i would like to remove the old config and pining logic next cycle if we can find time to do it | |
| 12:19:28 | sean-k-mooney | bauzas: yep we didnt dicuss it on the spec since we generally for get about the legacy pinning | |
| 12:19:39 | bauzas | cool | |
| 12:19:41 | bauzas | no worries | |
| 12:19:43 | sean-k-mooney | it was ment to be remvoed 2 or 3 releases ago | |
| 12:20:51 | bauzas | np | |
| 12:23:34 | bauzas | sean-k-mooney: one last question, can we both have cpu_dedicated_set and vcpu_pin_set defined ? | |
| 12:23:54 | bauzas | or are they mutually exclusive ? | |
| 12:25:19 | sean-k-mooney | cpu_dedicated_set and vcpu_pin_set are mutally exclsive | |
| 12:25:25 | bauzas | ++ | |
| 12:25:38 | sean-k-mooney | although that is checked later in the code not in the config option definition | |
| 12:25:53 | sean-k-mooney | ones in compute and the others in default | |
| 12:27:02 | sean-k-mooney | bauzas: you can simpley check "if CONF.compute.cpu_dedicated_set is None and CONF.libvirt.cpu_power_management: raise" | |
| 12:30:03 | bauzas | that was my guess | |
| 12:32:10 | sean-k-mooney | im going to swap to downstream stuff for a while unless pinged | |
| 12:32:31 | sean-k-mooney | bauzas: can we try and land dansmith's stable uuid seriese again today | |
| 12:32:50 | bauzas | sean-k-mooney: yup, I can try to take a look again | |
| 12:33:25 | sean-k-mooney | cool im +2 all the way up. dan did some rebases and fix a minior bug so they lost the +2s they had before | |
| 14:23:45 | ralonsoh | sean-k-mooney, hi! Do you know if this is a known error? | |
| 14:23:46 | ralonsoh | https://zuul.opendev.org/t/openstack/build/0d9af8c4e906422a9e4a27b1b849f315 | |
| 14:23:55 | ralonsoh | This is the second recheck with this error | |
| 14:25:34 | sean-k-mooney | no thats not a know error | |
| 14:25:42 | sean-k-mooney | although the volume tests can be flaky | |
| 14:26:02 | sean-k-mooney | this looks unrealted | |
| 14:29:09 | sean-k-mooney | i think there may have been an OOM issue | |
| 14:29:28 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/0d9af8c4e906422a9e4a27b1b849f315/log/controller/logs/screen-memory_tracker.txt#4768 | |
| 14:29:38 | ralonsoh | right, thanks | |
| 14:30:23 | sean-k-mooney | look like nova api lost access to the db | |
| 14:30:51 | sean-k-mooney | so my guess is mariadb was killed | |
| 14:31:03 | sean-k-mooney | same for keystone it los db connection | |
| 14:31:06 | ralonsoh | that usually happens with a oom | |
| 14:31:23 | ralonsoh | did you try reducing the parallelism in the functional tests? | |
| 14:31:23 | sean-k-mooney | yep so we might need to add a little more swap to that job | |
| 14:31:37 | sean-k-mooney | its not the funcitonl tests its tempest | |
| 14:31:43 | ralonsoh | right, in tempest | |
| 14:33:20 | sean-k-mooney | its currently 4 which should be ok | |
| 14:33:36 | ralonsoh | we reduced some jobs to 3 or 2 | |
| 14:33:41 | ralonsoh | to avoid this issue | |
| 14:33:47 | sean-k-mooney | i think this is using the default swap of 1G we shoudl set it to 8 | |
| 14:34:01 | sean-k-mooney | we could reduce concernace too but i would try swap first | |
| 14:34:07 | ralonsoh | perfect | |
| 14:34:46 | sean-k-mooney | we crrently set concurancy here https://github.com/openstack/os-vif/blob/master/.zuul.yaml#L23 | |
| 14:35:58 | sean-k-mooney | you can add configure_swap_size: 8192 there as well | |
| 14:36:18 | sean-k-mooney | if that does not work then set concurancy to 3 | |
| 14:36:18 | ralonsoh | should I do in a separate patch? | |
| 14:36:26 | sean-k-mooney | ya lets make it a spereate patch | |
| 14:36:32 | ralonsoh | cool, give me 1 min | |
| 14:36:44 | sean-k-mooney | then we can just recheck yours if it passes once its merged | |
| 14:39:37 | opendevreview | Rodolfo Alonso proposed openstack/os-vif master: Increase the swap size to 8GB in tempest jobs https://review.opendev.org/c/openstack/os-vif/+/872655 | |
| 14:40:45 | sean-k-mooney | ok lets leave that run but we should be able to merge both by the end of the day all going well | |
| 14:41:14 | ralonsoh | that's perfect | |
| 15:39:39 | opendevreview | Elod Illes proposed openstack/nova stable/wallaby: [stable-only] Remove broken sdk job from wallaby https://review.opendev.org/c/openstack/nova/+/871798 | |
| 16:10:47 | spatel | sean-k-mooney did you ever see this error when trying to create vm1 - https://paste.opendev.org/show/bmAj7IrkhbRpMbe14zht/ | |
| 16:13:02 | sean-k-mooney | i have see it if the wsgi timeout or haproxy timeout is set too low and the services are under heavy load | |
| 16:13:36 | spatel | Hmm!! | |
| 16:14:09 | sean-k-mooney | you need to check if the request ever got to the nova api. if it did you need to check if the api responed before the connection closed and if it got to the loadbalance or not | |
| 16:14:10 | spatel | server load is 0.66 average | |
| 16:15:36 | spatel | I can do nova list etc.. which works fine. only vm creation choking up | |
| 16:15:48 | spatel | let me check nova api logs etc.. | |
| 16:17:18 | spatel | HAproxy showing all green health for nova members | |
| 16:18:25 | spatel | openstack compute service list also showing all service up | |
| 16:22:47 | spatel | This logs has nothing to do with this issue correct - https://paste.opendev.org/show/bhP6raKj9pFgARdlb0ES/ | |
| 16:25:08 | spatel | sean-k-mooney it works after i restarted all nova-* service | |
| 16:25:25 | spatel | do you think this is bug? I am running wallaby release | |
| 16:28:57 | dansmith | bauzas: you still around? | |
| 16:29:09 | bauzas | dansmith: yes | |
| 16:29:40 | dansmith | any chance you could look at the stable compute stuff today? it's +2d up to the top, most of what is left are additional checks and tests | |
| 16:29:51 | dansmith | I'm going to work on docs next week | |
| 16:30:44 | sean-k-mooney | spatel: maybe but without you debuging it and finding a root cause or an indication of the problem we dont dont ahve anything to go on | |
| 16:31:19 | bauzas | dansmith: I'm a bit short in time but I can try | |
| 16:31:23 | sean-k-mooney | spatel: i dont think that log is related | |
| 16:32:07 | spatel | sean-k-mooney i think better i should upgrade to Xena or yoga. | |
| 16:32:15 | spatel | wallaby is little behind now | |
| 16:32:25 | dansmith | bauzas: ack, well, if we have to recheck grind them, I just don't want to be racing next week.. and monday is a holiday for some folks | |
| 16:34:52 | bauzas | dansmith: I'm running up to finish to write functests on https://review.opendev.org/c/openstack/nova/+/868237/ but again, I'll try | |
| 16:35:20 | dansmith | okay, well, it's not that critical I guess | |
| 16:36:12 | dansmith | maybe I could bribe melwitt to have a look-see | |
| 16:40:27 | bauzas | sean-k-mooney: quick q, when reading https://docs.openstack.org/nova/latest/admin/cpu-topologies.html#configure-libvirt-pinning | |
| 16:40:56 | bauzas | sean-k-mooney: on my functest, I only did set cpu_dedicated_set but when trying to boot a regular non-pinned instance, it errors out | |