| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-02-03 | |||
| 12:33:25 | sean-k-mooney | cool im +2 all the way up. dan did some rebases and fix a minior bug so they lost the +2s they had before | |
| 14:23:45 | ralonsoh | sean-k-mooney, hi! Do you know if this is a known error? | |
| 14:23:46 | ralonsoh | https://zuul.opendev.org/t/openstack/build/0d9af8c4e906422a9e4a27b1b849f315 | |
| 14:23:55 | ralonsoh | This is the second recheck with this error | |
| 14:25:34 | sean-k-mooney | no thats not a know error | |
| 14:25:42 | sean-k-mooney | although the volume tests can be flaky | |
| 14:26:02 | sean-k-mooney | this looks unrealted | |
| 14:29:09 | sean-k-mooney | i think there may have been an OOM issue | |
| 14:29:28 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/0d9af8c4e906422a9e4a27b1b849f315/log/controller/logs/screen-memory_tracker.txt#4768 | |
| 14:29:38 | ralonsoh | right, thanks | |
| 14:30:23 | sean-k-mooney | look like nova api lost access to the db | |
| 14:30:51 | sean-k-mooney | so my guess is mariadb was killed | |
| 14:31:03 | sean-k-mooney | same for keystone it los db connection | |
| 14:31:06 | ralonsoh | that usually happens with a oom | |
| 14:31:23 | ralonsoh | did you try reducing the parallelism in the functional tests? | |
| 14:31:23 | sean-k-mooney | yep so we might need to add a little more swap to that job | |
| 14:31:37 | sean-k-mooney | its not the funcitonl tests its tempest | |
| 14:31:43 | ralonsoh | right, in tempest | |
| 14:33:20 | sean-k-mooney | its currently 4 which should be ok | |
| 14:33:36 | ralonsoh | we reduced some jobs to 3 or 2 | |
| 14:33:41 | ralonsoh | to avoid this issue | |
| 14:33:47 | sean-k-mooney | i think this is using the default swap of 1G we shoudl set it to 8 | |
| 14:34:01 | sean-k-mooney | we could reduce concernace too but i would try swap first | |
| 14:34:07 | ralonsoh | perfect | |
| 14:34:46 | sean-k-mooney | we crrently set concurancy here https://github.com/openstack/os-vif/blob/master/.zuul.yaml#L23 | |
| 14:35:58 | sean-k-mooney | you can add configure_swap_size: 8192 there as well | |
| 14:36:18 | sean-k-mooney | if that does not work then set concurancy to 3 | |
| 14:36:18 | ralonsoh | should I do in a separate patch? | |
| 14:36:26 | sean-k-mooney | ya lets make it a spereate patch | |
| 14:36:32 | ralonsoh | cool, give me 1 min | |
| 14:36:44 | sean-k-mooney | then we can just recheck yours if it passes once its merged | |
| 14:39:37 | opendevreview | Rodolfo Alonso proposed openstack/os-vif master: Increase the swap size to 8GB in tempest jobs https://review.opendev.org/c/openstack/os-vif/+/872655 | |
| 14:40:45 | sean-k-mooney | ok lets leave that run but we should be able to merge both by the end of the day all going well | |
| 14:41:14 | ralonsoh | that's perfect | |
| 15:39:39 | opendevreview | Elod Illes proposed openstack/nova stable/wallaby: [stable-only] Remove broken sdk job from wallaby https://review.opendev.org/c/openstack/nova/+/871798 | |
| 16:10:47 | spatel | sean-k-mooney did you ever see this error when trying to create vm1 - https://paste.opendev.org/show/bmAj7IrkhbRpMbe14zht/ | |
| 16:13:02 | sean-k-mooney | i have see it if the wsgi timeout or haproxy timeout is set too low and the services are under heavy load | |
| 16:13:36 | spatel | Hmm!! | |
| 16:14:09 | sean-k-mooney | you need to check if the request ever got to the nova api. if it did you need to check if the api responed before the connection closed and if it got to the loadbalance or not | |
| 16:14:10 | spatel | server load is 0.66 average | |
| 16:15:36 | spatel | I can do nova list etc.. which works fine. only vm creation choking up | |
| 16:15:48 | spatel | let me check nova api logs etc.. | |
| 16:17:18 | spatel | HAproxy showing all green health for nova members | |
| 16:18:25 | spatel | openstack compute service list also showing all service up | |
| 16:22:47 | spatel | This logs has nothing to do with this issue correct - https://paste.opendev.org/show/bhP6raKj9pFgARdlb0ES/ | |
| 16:25:08 | spatel | sean-k-mooney it works after i restarted all nova-* service | |
| 16:25:25 | spatel | do you think this is bug? I am running wallaby release | |
| 16:28:57 | dansmith | bauzas: you still around? | |
| 16:29:09 | bauzas | dansmith: yes | |
| 16:29:40 | dansmith | any chance you could look at the stable compute stuff today? it's +2d up to the top, most of what is left are additional checks and tests | |
| 16:29:51 | dansmith | I'm going to work on docs next week | |
| 16:30:44 | sean-k-mooney | spatel: maybe but without you debuging it and finding a root cause or an indication of the problem we dont dont ahve anything to go on | |
| 16:31:19 | bauzas | dansmith: I'm a bit short in time but I can try | |
| 16:31:23 | sean-k-mooney | spatel: i dont think that log is related | |
| 16:32:07 | spatel | sean-k-mooney i think better i should upgrade to Xena or yoga. | |
| 16:32:15 | spatel | wallaby is little behind now | |
| 16:32:25 | dansmith | bauzas: ack, well, if we have to recheck grind them, I just don't want to be racing next week.. and monday is a holiday for some folks | |
| 16:34:52 | bauzas | dansmith: I'm running up to finish to write functests on https://review.opendev.org/c/openstack/nova/+/868237/ but again, I'll try | |
| 16:35:20 | dansmith | okay, well, it's not that critical I guess | |
| 16:36:12 | dansmith | maybe I could bribe melwitt to have a look-see | |
| 16:40:27 | bauzas | sean-k-mooney: quick q, when reading https://docs.openstack.org/nova/latest/admin/cpu-topologies.html#configure-libvirt-pinning | |
| 16:40:56 | bauzas | sean-k-mooney: on my functest, I only did set cpu_dedicated_set but when trying to boot a regular non-pinned instance, it errors out | |
| 16:41:03 | bauzas | with NoValidHost | |
| 16:41:13 | bauzas | so I guess I also need to set cpu_shared_set ? | |
| 16:42:14 | bauzas | https://docs.openstack.org/nova/latest/configuration/config.html#compute.cpu_dedicated_set doesn't mention the dependency | |
| 16:43:11 | bauzas | oh wait, see my bug | |
| 16:44:58 | bauzas | my self.flags was setting None to cpu_shared_set | |
| 16:45:00 | bauzas | ffff | |
| 16:45:17 | sean-k-mooney | cpu_shared_set is for non pined core yes | |
| 16:45:24 | sean-k-mooney | or floating vms | |
| 16:45:37 | bauzas | I know tyhis | |
| 16:45:39 | sean-k-mooney | you only need to set it if cpu_dedicated_set is defiend | |
| 16:45:52 | bauzas | ok, both are dependent then | |
| 16:46:16 | bauzas | you can't just set cpu_dedicated_set without opt-in the cpu_shared_set | |
| 16:46:16 | sean-k-mooney | i dont wnat to go into the complexiyt i can explain it in detail next week if you like | |
| 16:46:28 | bauzas | no worries, it works | |
| 16:46:40 | sean-k-mooney | you can but if you dont set cpu_shared_set then the host can only be used for pinnded vms | |
| 16:46:40 | bauzas | it was a pebkac | |
| 16:46:59 | bauzas | I see | |
| 16:47:09 | bauzas | with the restricted set of dedicated cpus | |
| 16:47:24 | sean-k-mooney | yep | |
| 16:47:24 | bauzas | like, I have 10 cpus, I set 5 of them as dedicated and period. | |
| 16:47:32 | bauzas | that means that the 5 lefts won't be used | |
| 16:47:36 | bauzas | gotcha | |
| 16:47:43 | sean-k-mooney | ya they are reserved for the host | |
| 16:47:54 | sean-k-mooney | the ones that are not listed | |
| 16:48:30 | sean-k-mooney | so normally i would expect the first core in each socket to not be in either set | |
| 16:48:44 | sean-k-mooney | and then you devied up the rest depending on your workloads | |
| 16:50:36 | bauzas | cool enough | |
| 17:05:26 | opendevreview | Elod Illes proposed openstack/nova stable/yoga: ignore deleted server groups in validation https://review.opendev.org/c/openstack/nova/+/867989 | |
| 17:05:27 | opendevreview | Elod Illes proposed openstack/nova stable/yoga: add repoducer test for bug 1890244 https://review.opendev.org/c/openstack/nova/+/872663 | |
| 18:31:12 | opendevreview | Merged openstack/nova stable/yoga: Improving logging at '_allocate_mdevs'. https://review.opendev.org/c/openstack/nova/+/871414 | |
| 18:46:13 | opendevreview | Sylvain Bauza proposed openstack/nova master: enable cpus when an instance is spawning https://review.opendev.org/c/openstack/nova/+/868237 | |
| 19:50:56 | dvo-plv | Hello | |
| 19:52:10 | dvo-plv | I have a question about trait. I added new trait COMPUTE_NET_VIRTIO_PACKED. WHen I execute next command : openstack allocation candidate list --resource VCPU=1 I can see that one compute node has this trait and another does not | |
| 19:53:13 | sean-k-mooney | you did not request the trait in the query | |
| 19:53:36 | sean-k-mooney | so you will get back a list of all host that have 1 cpu core free | |
| 19:53:48 | dvo-plv | When I do live migration from compute node with new trait to the comptue without this trait, soes scheduler should check if requested compute node has this trait and forbid to execute migration ? | |
| 19:54:17 | sean-k-mooney | if and only if you have requested it in the flavor or implemnted that in your change correctly | |
| 19:54:49 | sean-k-mooney | you would ideally have a prefilter that add the trait request | |