| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-07-03 | |||
| 18:27:08 | dansmith | I don't _think_ nova-next does a ton of weird config | |
| 18:27:40 | opendevreview | Masahito Muroi proposed openstack/nova-specs master: Add spec for the network id query in server list and server details https://review.opendev.org/c/openstack/nova-specs/+/938823 | |
| 18:28:17 | clarkb | yes its possible they use private ips ratehr than public ips | |
| 18:28:45 | clarkb | or listen only on the local ip address so connections for the floating ip are rejected | |
| 18:28:50 | dansmith | transport_url = rabbit://stackrabbit:secretrabbit@10.209.32.130:5672/ | |
| 18:28:58 | clarkb | that'll do it | |
| 18:29:20 | dansmith | okay so I don't think that's specific to nova-next here | |
| 18:32:09 | dansmith | well, it does make some choices about neutron so maybe it's wrapped up in that | |
| 18:51:04 | sean-k-mooney | its been a very long time but we used tc when i was at intel to injectup to 120ms of latency into the links between host | |
| 18:51:33 | sean-k-mooney | for the most part that seam to still be fine provide the clocks were in sync | |
| 18:51:34 | dansmith | yeah it's clearly not routable because of the private ip in this case | |
| 18:51:40 | dansmith | uep | |
| 18:52:05 | sean-k-mooney | ah its routing ya | |
| 18:52:40 | sean-k-mooney | that woudl make sense. i know we try and setbale a vxlan or gre mesh usign linux bridge in the multi node jobs | |
| 18:52:47 | sean-k-mooney | and hten use that for neutron tenant networking | |
| 18:52:57 | sean-k-mooney | but i dont know if we actuly route all traffic over that | |
| 18:53:16 | sean-k-mooney | i guess we are not or that is not working properly if we are | |
| 18:54:49 | dansmith | it's not that because this is just nova-compute unable to contact the mq on the controller node because it's not using its public ip | |
| 18:54:54 | dansmith | nothing to do with guest networking | |
| 18:55:14 | sean-k-mooney | what i ment was if we used the ip form that brdige network those might work | |
| 18:55:28 | sean-k-mooney | but ya in devstack we lookup the ip of the node by defualt | |
| 18:55:34 | sean-k-mooney | i dotn think we overried that in the jobs | |
| 18:55:46 | dansmith | right | |
| 18:56:09 | sean-k-mooney | we expct the vms to be on the same neutron network in the josb since nodes in any given nodeset are nto allwos to comefrom diffent providers | |
| 18:56:28 | sean-k-mooney | or at least they were not allowed to do tha tin the past so the josb depend on that | |
| 18:56:38 | sean-k-mooney | changign that would be a breakign change | |
| 18:56:57 | sean-k-mooney | not nessiarly a bad one but not one that our jobs can handel so it woudl need ot be opt in | |
| 20:33:26 | opendevreview | Brett Holman proposed openstack/nova master: Add DMI data to non-x86 architectures https://review.opendev.org/c/openstack/nova/+/953989 | |
| #openstack-nova - 2025-07-04 | |||
| 09:58:07 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Return 404 on bad project ID for 'os-quota-sets' https://review.opendev.org/c/openstack/nova/+/937125 | |
| 10:00:15 | stephenfin | gibi: sean-k-mooney: Can we get https://review.opendev.org/c/openstack/nova/+/952266 in? It's rather annoying when running unit tests 😅 | |
| 10:16:28 | gibi | +2 | |
| 10:17:39 | stephenfin | ty | |
| 10:51:51 | opendevreview | Stephen Finucane proposed openstack/nova master: WIP: api: Remove version controller split https://review.opendev.org/c/openstack/nova/+/954103 | |
| 10:51:51 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Deprecate v2 API https://review.opendev.org/c/openstack/nova/+/954102 | |
| 11:05:24 | sean-k-mooney | stephenfin: sure | |
| 11:25:13 | opendevreview | Biser Milanov proposed openstack/nova stable/2024.1: Hardcode the use of iothreads for KVM. https://review.opendev.org/c/openstack/nova/+/954113 | |
| 11:28:14 | opendevreview | Biser Milanov proposed openstack/nova stable/2024.1: Hardcode the use of iothreads for KVM. https://review.opendev.org/c/openstack/nova/+/954113 | |
| 11:29:42 | opendevreview | Biser Milanov proposed openstack/nova stable/2024.1: StorPool: Pass the instance UUID and device_name to os-brick https://review.opendev.org/c/openstack/nova/+/954115 | |
| 11:30:37 | sp-bmilanov | sean-k-mooney: Hi, you can ignore the backport chain : https://review.opendev.org/c/openstack/nova/+/954115 , it is not meant to actually be merged | |
| 11:30:56 | sean-k-mooney | ack well the master version is not appliable either | |
| 11:31:28 | sp-bmilanov | yep, I am aware there's an effort to go about adding iothreads another way | |
| 11:31:32 | sean-k-mooney | we are not gong to make this a per host config option | |
| 12:50:30 | sean-k-mooney | masahito: by the way even if we are pass the spec freeze i would encurrage you to work on a poc of the iothread and queue patches based on the current version fo hte spec, while we may not have time to do extensive reivew of it if you have a working implemation with tempest test ectra at the sart of the new cycle in october i think it would be reaosnbale ot avocate for | |
| 12:50:32 | sean-k-mooney | reiviewing and completign the overall functionatliy early in the cycle. | |
| 12:51:21 | sean-k-mooney | depending on how early it was done you may even have time to look at the next steps although that less likely to happen in time for 2026.1 | |
| 12:51:48 | gibi | fyi I see significant failure rate in our multi node gate due to rabbit is not accessible from compute1 https://bugs.launchpad.net/nova/+bug/2115980 It is started 3 days ago | |
| 12:52:04 | sean-k-mooney | gibi: ya so that a zuul issue | |
| 12:52:25 | gibi | is it a tracker for it from infra perspective? | |
| 12:52:33 | sean-k-mooney | gibi: tl;dr in the old impelation nodesets were not allowe to be provisioned form differnt providers | |
| 12:52:59 | sean-k-mooney | in the new one it can but that breaks our jobs beacue they use the local ips since it expect all the vms to be on the same neutron network | |
| 12:53:32 | sean-k-mooney | gibi: in terms of tracker i am not sure but dansmith and clarkb where talkign about it last night | |
| 12:53:44 | gibi | OK, reading back then... | |
| 12:53:46 | gibi | thanks | |
| 12:53:51 | sean-k-mooney | they know that his is happenign and i think they are looking to disbale that feature | |
| 12:54:20 | sean-k-mooney | but i dont know if that change has been doen yet, the openstack-infra channle is proably hte best plase to follow up | |
| 12:57:49 | gibi | ack, I've pinged infra about it | |
| 12:59:02 | sean-k-mooney | i belive this is basicly just a boolean we need to flip for our tenant. long term we coudl refactor the multi node role that we use in the devstack job to use the floating ips and streach the deploymnt over the wan but that is likely to cause other issues so i dont think that will pan out | |
| 13:01:28 | gibi | 14:59 < fungi> gibi: there's https://review.opendev.org/954064 which we should be auto-upgrading to within the next 24 hours | |
| 13:01:39 | gibi | so fix is in the pipe | |
| 13:01:59 | sean-k-mooney | cool | |
| 13:02:34 | fungi | yeah, we automatically upgrade our zuul servers through rolling restarts over the weekend | |
| 13:03:32 | sean-k-mooney | fungi: i woudl still suggest that it shoudl be posisble to disable the ablity to use multiple providers at the nodeset or tenant level | |
| 13:03:41 | fungi | booting a node in another provider for a multi-node job should only ever be a fallback in cases where the job would have otherwise been reported as a node failure | |
| 13:04:10 | fungi | the fact that it was happening in recent cases was a logic bug, which that change addresses | |
| 13:04:33 | sean-k-mooney | fungi: right but we know our multi node devstack jobs will never work in that case | |
| 13:04:59 | fungi | disabling the fallback would just mean always reporting node_failure in those cases rather than trying anyway and possibly failing for other reasons | |
| 13:04:59 | sean-k-mooney | so woudl it not be better to allow declaring if split providers can be tollerated in a job/build set level | |
| 13:05:30 | sean-k-mooney | right which may actully be a beter develoepr experince | |
| 13:05:49 | sean-k-mooney | it mean i dont havee to debug why it fail jsut to see it was mulit provider | |
| 13:06:05 | fungi | so the job would still fail, but yes maybe the benefit is that it returns sooner, doesn't waste as many resources, and gives a clearer failure reason for those | |
| 13:06:25 | sean-k-mooney | if we modify our devstack josb to alwasy use publicly routabel ips for all our networking that would be one thing | |
| 13:07:52 | sean-k-mooney | fungi: for what its worth i am happy the multi provider capablity was finally added i just wish is was more contolable | |
| 13:08:42 | fungi | yeah, but also the fact that two nodes boot in the same provider region doesn't guarantee low latency between them | |
| 13:09:56 | fungi | anyway, once we're running on the referenced change, the incidence of this particular failure should be on par with prior frequency of node_failure results in those jobs, which i hope was exceedingly rare to begin with | |
| 13:10:01 | sean-k-mooney | fungi: latency is not really the concern we do expect routableity between the nodes however using there local ips | |
| 13:10:47 | fungi | ah, yeah also booting in the same provider region doesn't guarantee that their local addresses can reach each other, though it's a relatively safe assumption | |
| 13:13:58 | sean-k-mooney | fungi: well that demped how you conifuged the provider | |
| 13:14:31 | sean-k-mooney | specificly you can configure the subnet and i think the network in the provider section | |
| 13:14:36 | fungi | right. in many providers we, as the users, aren't configuring that | |
| 13:14:44 | sean-k-mooney | to ensure that all node provided by that guarentee that | |
| 13:15:21 | sean-k-mooney | in which case we are assumign there is only one netowrk and that is beign used by default | |
| 13:15:28 | sean-k-mooney | which si ok but not ideal | |
| 13:15:52 | fungi | a lot of openstack public clouds do shared provider networks for the server ports, but yes those can generally all still reach each other even in cases where they allocate out of disjoint networks (they just bounce off a gateway address) | |
| 13:16:24 | opendevreview | Merged openstack/nova master: db: Resolve alembic deprecation warning https://review.opendev.org/c/openstack/nova/+/952266 | |
| 13:16:48 | sean-k-mooney | yep so our current multi jobs all depend on that today | |
| 13:18:04 | sean-k-mooney | ideally we woudl be able to express that on the nodeset definitoin "node_must_be_routeable" or somethign like that, i assuem we cant assuem all nodes have ipv6 yet | |
| 13:18:43 | sean-k-mooney | if we coudl we coudl have the exisitng multi node role that create the tunnel messh do that over ipv6 and then use the ipv4 addreses it provides for our jobs | |
| 13:19:14 | sean-k-mooney | i.e. instead of using the ip form the default route use the one for the multi node bridge | |
| 13:19:47 | sean-k-mooney | im referint ot https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/multi-node-bridge just ot be clear | |
| 13:20:35 | sean-k-mooney | if we could rely on that mesh working across providers we woudl not have to care for the most part as that vxlan tunnle should resolve any firewall issues we have | |
| 13:21:05 | sean-k-mooney | although runing ceph rbd over that likely wotn be a fun time | |
| 13:21:55 | sean-k-mooney | lets just see how things go once that patch lands | |
| 13:24:35 | fungi | another option is an early acceptance check in the job to confirm the nodes are able to reach one another satisfactorally over the interfaces it wants them to use, and then the nodeset can be discarded and the build automatically retried if not. we do something similar in a common role to make sure nodes are able to reach internet resources, for example | |
| 13:28:03 | sean-k-mooney | ya thats not a bad idea we coudl do it as a pre-playbook that would alow the job to potetally rescdhule to a diffetn node set | |
| 13:28:26 | sean-k-mooney | thats honestly the best of both worlds as we may be able to avoid recheckign | |
| 13:28:56 | sean-k-mooney | and if we do it as a role we coudl selectivly add it to the jobs that actully need it | |
| 13:29:25 | sean-k-mooney | that may end up beign all the multi node jobs but perhaps not | |
| 14:17:18 | opendevreview | Takashi Natsume proposed openstack/nova master: Update contributor guide for 2025.2 Flamingo https://review.opendev.org/c/openstack/nova/+/944603 | |
| 14:19:52 | gboutry | Hello nova, is there anything else required from this review (a part from reviews): https://review.opendev.org/c/openstack/nova/+/953737? I'd be quite interested for the backport in 2025.1 | |
| 14:27:34 | gibi | gboutry: no, I think we just waiting for reviews there | |