| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-02-03 | |||
| 16:14:09 | sean-k-mooney | you need to check if the request ever got to the nova api. if it did you need to check if the api responed before the connection closed and if it got to the loadbalance or not | |
| 16:14:10 | spatel | server load is 0.66 average | |
| 16:15:36 | spatel | I can do nova list etc.. which works fine. only vm creation choking up | |
| 16:15:48 | spatel | let me check nova api logs etc.. | |
| 16:17:18 | spatel | HAproxy showing all green health for nova members | |
| 16:18:25 | spatel | openstack compute service list also showing all service up | |
| 16:22:47 | spatel | This logs has nothing to do with this issue correct - https://paste.opendev.org/show/bhP6raKj9pFgARdlb0ES/ | |
| 16:25:08 | spatel | sean-k-mooney it works after i restarted all nova-* service | |
| 16:25:25 | spatel | do you think this is bug? I am running wallaby release | |
| 16:28:57 | dansmith | bauzas: you still around? | |
| 16:29:09 | bauzas | dansmith: yes | |
| 16:29:40 | dansmith | any chance you could look at the stable compute stuff today? it's +2d up to the top, most of what is left are additional checks and tests | |
| 16:29:51 | dansmith | I'm going to work on docs next week | |
| 16:30:44 | sean-k-mooney | spatel: maybe but without you debuging it and finding a root cause or an indication of the problem we dont dont ahve anything to go on | |
| 16:31:19 | bauzas | dansmith: I'm a bit short in time but I can try | |
| 16:31:23 | sean-k-mooney | spatel: i dont think that log is related | |
| 16:32:07 | spatel | sean-k-mooney i think better i should upgrade to Xena or yoga. | |
| 16:32:15 | spatel | wallaby is little behind now | |
| 16:32:25 | dansmith | bauzas: ack, well, if we have to recheck grind them, I just don't want to be racing next week.. and monday is a holiday for some folks | |
| 16:34:52 | bauzas | dansmith: I'm running up to finish to write functests on https://review.opendev.org/c/openstack/nova/+/868237/ but again, I'll try | |
| 16:35:20 | dansmith | okay, well, it's not that critical I guess | |
| 16:36:12 | dansmith | maybe I could bribe melwitt to have a look-see | |
| 16:40:27 | bauzas | sean-k-mooney: quick q, when reading https://docs.openstack.org/nova/latest/admin/cpu-topologies.html#configure-libvirt-pinning | |
| 16:40:56 | bauzas | sean-k-mooney: on my functest, I only did set cpu_dedicated_set but when trying to boot a regular non-pinned instance, it errors out | |
| 16:41:03 | bauzas | with NoValidHost | |
| 16:41:13 | bauzas | so I guess I also need to set cpu_shared_set ? | |
| 16:42:14 | bauzas | https://docs.openstack.org/nova/latest/configuration/config.html#compute.cpu_dedicated_set doesn't mention the dependency | |
| 16:43:11 | bauzas | oh wait, see my bug | |
| 16:44:58 | bauzas | my self.flags was setting None to cpu_shared_set | |
| 16:45:00 | bauzas | ffff | |
| 16:45:17 | sean-k-mooney | cpu_shared_set is for non pined core yes | |
| 16:45:24 | sean-k-mooney | or floating vms | |
| 16:45:37 | bauzas | I know tyhis | |
| 16:45:39 | sean-k-mooney | you only need to set it if cpu_dedicated_set is defiend | |
| 16:45:52 | bauzas | ok, both are dependent then | |
| 16:46:16 | sean-k-mooney | i dont wnat to go into the complexiyt i can explain it in detail next week if you like | |
| 16:46:16 | bauzas | you can't just set cpu_dedicated_set without opt-in the cpu_shared_set | |
| 16:46:28 | bauzas | no worries, it works | |
| 16:46:40 | bauzas | it was a pebkac | |
| 16:46:40 | sean-k-mooney | you can but if you dont set cpu_shared_set then the host can only be used for pinnded vms | |
| 16:46:59 | bauzas | I see | |
| 16:47:09 | bauzas | with the restricted set of dedicated cpus | |
| 16:47:24 | bauzas | like, I have 10 cpus, I set 5 of them as dedicated and period. | |
| 16:47:24 | sean-k-mooney | yep | |
| 16:47:32 | bauzas | that means that the 5 lefts won't be used | |
| 16:47:36 | bauzas | gotcha | |
| 16:47:43 | sean-k-mooney | ya they are reserved for the host | |
| 16:47:54 | sean-k-mooney | the ones that are not listed | |
| 16:48:30 | sean-k-mooney | so normally i would expect the first core in each socket to not be in either set | |
| 16:48:44 | sean-k-mooney | and then you devied up the rest depending on your workloads | |
| 16:50:36 | bauzas | cool enough | |
| 17:05:26 | opendevreview | Elod Illes proposed openstack/nova stable/yoga: ignore deleted server groups in validation https://review.opendev.org/c/openstack/nova/+/867989 | |
| 17:05:27 | opendevreview | Elod Illes proposed openstack/nova stable/yoga: add repoducer test for bug 1890244 https://review.opendev.org/c/openstack/nova/+/872663 | |
| 18:31:12 | opendevreview | Merged openstack/nova stable/yoga: Improving logging at '_allocate_mdevs'. https://review.opendev.org/c/openstack/nova/+/871414 | |
| 18:46:13 | opendevreview | Sylvain Bauza proposed openstack/nova master: enable cpus when an instance is spawning https://review.opendev.org/c/openstack/nova/+/868237 | |
| 19:50:56 | dvo-plv | Hello | |
| 19:52:10 | dvo-plv | I have a question about trait. I added new trait COMPUTE_NET_VIRTIO_PACKED. WHen I execute next command : openstack allocation candidate list --resource VCPU=1 I can see that one compute node has this trait and another does not | |
| 19:53:13 | sean-k-mooney | you did not request the trait in the query | |
| 19:53:36 | sean-k-mooney | so you will get back a list of all host that have 1 cpu core free | |
| 19:53:48 | dvo-plv | When I do live migration from compute node with new trait to the comptue without this trait, soes scheduler should check if requested compute node has this trait and forbid to execute migration ? | |
| 19:54:17 | sean-k-mooney | if and only if you have requested it in the flavor or implemnted that in your change correctly | |
| 19:54:49 | sean-k-mooney | you would ideally have a prefilter that add the trait request | |
| 19:55:15 | dvo-plv | I set this trait in the flavor with next command openstack flavor set --property trait:COMPUTE_NET_VIRTIO_PACKED=required 3 | |
| 19:55:41 | sean-k-mooney | that will take effect for new vms but not existing ones | |
| 19:55:57 | sean-k-mooney | if you have that set however then it will be included with the placment request and enforced | |
| 19:56:17 | dvo-plv | Yes, I create flavor with this trait and after that create VM and try to do live migration | |
| 19:56:47 | sean-k-mooney | then placement should enable this to work as intended | |
| 19:57:04 | sean-k-mooney | the vm will only shcedule to a host with that trait including on migrations | |
| 19:58:34 | sean-k-mooney | this is part of this proposal correct https://review.opendev.org/c/openstack/nova-specs/+/868377/1/specs/2023.1/approved/virtio_packedring_configuration_support.rst | |
| 20:00:43 | dvo-plv | In the spec file you told that packed ring option should be handled in some specific way to be sure that vm with packed_ring qemu option will not migrate to the node without this feature support and i can not get this feature from the libvirt. I found that this feature is enabled only from libvirt 6.3. So if I migrate to the node with lower libvirt the next method _check_compatible_with_source_hypervisor will forbid to migrate | |
| 20:01:29 | sean-k-mooney | that is not a sufficent check unfortunatly | |
| 20:01:47 | sean-k-mooney | this is not a feature of libvirt its a qmeu one and that check would already be too late | |
| 20:02:28 | sean-k-mooney | what we need to do is have nova report the trait on all hosts that support this | |
| 20:03:09 | sean-k-mooney | and then either have the guest opt into this with a image property or flavor extra specs | |
| 20:03:27 | sean-k-mooney | we can then use that image property/extra spec to request the trait automatically | |
| 20:03:45 | sean-k-mooney | that is the simple approch | |
| 20:04:00 | sean-k-mooney | the harder approch is to figure out a way to auto enable this | |
| 20:04:32 | sean-k-mooney | i dont know a simple way to do that off the top of my head unfortunetly | |
| 20:05:20 | sean-k-mooney | its a bit late here but we should proably disucss this some more next week | |
| 20:05:30 | dvo-plv | another way, we can check qemu version, this feature was implemented from qemu 4.2 | |
| 20:06:24 | dvo-plv | or grep funtionality like in this port http://blog.vmsplice.net/2020/05/how-to-check-virtio-feature-bits-inside.html | |
| 20:07:06 | dvo-plv | Thank you, let's move it to next weekÑ” | |
| 20:07:06 | dvo-plv | Thank you, let's move it to next weekє | |
| 20:07:33 | sean-k-mooney | the issue is we need to make the scudleing work before we have selected any host | |
| 20:08:22 | sean-k-mooney | so in general we cannot check the qemu/libvirt verion at the schduling step | |
| 20:08:39 | dvo-plv | I tried to get this feature at this method static_traits in the nova/virt/libvirt/driver.py | |
| 20:09:18 | sean-k-mooney | the best i can think of would be to stash a flag in the instance_system_metadta to recored it was booted on a host that supported the pact format | |
| 20:09:36 | sean-k-mooney | and then include a request for the trait if that is found in the instance_system_metadata | |
| 20:09:54 | sean-k-mooney | that is proably the best we can do since we dont know if its used by the guest | |
| 20:10:31 | sean-k-mooney | the static_traits function is the correct plase to report the trait for the compute host | |
| 21:06:28 | gmann | bauzas: stephenfin: need one more review in this to unblock stable/wallaby gate (removing broken sdk job) https://review.opendev.org/c/openstack/nova/+/871798 | |
| 22:39:50 | melwitt | dansmith: I have look-see'd | |
| 22:40:31 | opendevreview | Merged openstack/nova stable/wallaby: [stable-only] Remove broken sdk job from wallaby https://review.opendev.org/c/openstack/nova/+/871798 | |
| 22:54:55 | dansmith | melwitt: um | |
| 22:55:04 | dansmith | I'm not sure you can past-tense-verb look-see | |
| 22:55:22 | dansmith | (but thanks :) | |
| 22:55:35 | melwitt | :) | |
| 22:56:49 | sean-k-mooney | its english there is always a way to do things | |
| 23:00:52 | sean-k-mooney | 023-02-03 23:00:25.664 41 ERROR nova.virt.driver [None req-80a00ce3-332e-4a98-be2e-d0a1bd9329d7 - - - - - -] Compute driver option required, but not specified | |
| 23:01:07 | sean-k-mooney | lol ok that explains why its not working | |