| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-07-03 | |||
| 17:33:54 | dansmith | I know the threads are useful for more than just blk, but if we can assume that threads mean you can do the blk queue that might be nice | |
| 17:33:56 | sean-k-mooney | we can we can also combine the spec if we really want ot at this point | |
| 17:34:01 | dansmith | just hate the trait explosion we get | |
| 17:34:52 | sean-k-mooney | orgially masahito goal was much more advance then the combidnaiton of these two spec, we can still evlove to that goal but the intent was to reduce the scope to soemthign achivable this cycle | |
| 17:36:23 | sean-k-mooney | dansmith: if we can do both this cycle i dont se why we would need 2 traits | |
| 17:37:07 | sean-k-mooney | so if we put the multi queue enhacement on top of the iotreads one when we are writign the code for this i think we can resue it to mean both | |
| 17:37:25 | dansmith | yeah | |
| 17:37:26 | sean-k-mooney | the multi queu change will now be very small so im not worried about that missing | |
| 17:37:54 | dansmith | I don't really see how we can get these both merged this cycle at this point | |
| 17:38:07 | dansmith | I'm out for the next ten days starting tomorrow | |
| 17:38:11 | dansmith | I dunno how others feel | |
| 18:12:24 | dansmith | gibi: just noticed you rechecking for rabbit reasons again... any chance we're mucking with things that are causing us to stall a real thread or something? | |
| 18:12:44 | dansmith | I mean, I realize I should know as well as you, but.. | |
| 18:12:53 | dansmith | are we seeing those same failures elsewhere? | |
| 18:13:28 | clarkb | dansmith: have a link to an example failure? | |
| 18:13:35 | clarkb | (I want to rule one thing out or in) | |
| 18:13:53 | dansmith | clarkb: https://review.opendev.org/c/openstack/nova/+/948079?tab=change-view-tab-header-zuul-results-summary | |
| 18:14:03 | dansmith | not sure which of those fails is the one he rechecked for | |
| 18:15:08 | dansmith | (man we have /got/ to squelch the verbose tracebacking from os-brick) | |
| 18:16:19 | clarkb | each of the three failures were multinode jobs and zuul-launcher (the new nodepool replacement) gave you mixed cloud provider nodes. Its only really supposed to do this when there is no other choice, but there was at least one bug causing it to do that too often that should be fixed now | |
| 18:16:39 | clarkb | those jobs ran ~5 hours ago so the fix for that particular problem should've been in place. | |
| 18:16:40 | dansmith | ah | |
| 18:16:55 | clarkb | But anyway if your multinode job can't handle that (due to ip addressing or latency or whatever) that could be a reason | |
| 18:19:27 | dansmith | so, nova officially doesn't support cross-site MQ, even though lots of people do it | |
| 18:19:40 | dansmith | redhat has strict latency requirements for when we allow people to do it | |
| 18:19:54 | dansmith | getting nodes from multiple providers would seem to make that kindof difficult | |
| 18:20:12 | clarkb | I've brought up that the mixed nodesets are still happening in #opendev as we're trying to track zuul-launcher behaviors that are unexpected or cause problems there | |
| 18:20:19 | clarkb | ya its not the default | |
| 18:20:36 | dansmith | ack | |
| 18:21:00 | clarkb | and I think the vast majority of the time it isn't happening that way (opendev's own multinode jobs testing deployment of our software is also sensitive to it and I noticed immediately when it was doing it too often. Since I think I'e only seen one failure on our side btu we also run fewer jobs than nova and have fewer changes) | |
| 18:24:36 | dansmith | in general, we're a lot more tolerant of it than I would have thought, so it's probably not really a problem in a lot of cases (and maybe good to be throwing some of it in there) | |
| 18:24:49 | dansmith | I'm just saying, it's definitely not what we expect to happen | |
| 18:25:40 | dansmith | looks like this is the fail he was referring to: https://zuul.opendev.org/t/openstack/build/66557ea92bae4e809c09526eba619c07/log/compute1/logs/screen-n-cpu.txt | |
| 18:25:42 | dansmith | which is just fully unable to talk to rabbit | |
| 18:25:59 | clarkb | which may be an issue of ip routing | |
| 18:26:26 | clarkb | depending no how the ip addrs are configured. By default we'd allow the traffic to cross the internet and get from point a to point b but you have to use the public ips on both sides which may be floating ips etc | |
| 18:26:47 | dansmith | and you think that our jobs might be misconfigured for that? | |
| 18:27:08 | dansmith | I don't _think_ nova-next does a ton of weird config | |
| 18:27:40 | opendevreview | Masahito Muroi proposed openstack/nova-specs master: Add spec for the network id query in server list and server details https://review.opendev.org/c/openstack/nova-specs/+/938823 | |
| 18:28:17 | clarkb | yes its possible they use private ips ratehr than public ips | |
| 18:28:45 | clarkb | or listen only on the local ip address so connections for the floating ip are rejected | |
| 18:28:50 | dansmith | transport_url = rabbit://stackrabbit:secretrabbit@10.209.32.130:5672/ | |
| 18:28:58 | clarkb | that'll do it | |
| 18:29:20 | dansmith | okay so I don't think that's specific to nova-next here | |
| 18:32:09 | dansmith | well, it does make some choices about neutron so maybe it's wrapped up in that | |
| 18:51:04 | sean-k-mooney | its been a very long time but we used tc when i was at intel to injectup to 120ms of latency into the links between host | |
| 18:51:33 | sean-k-mooney | for the most part that seam to still be fine provide the clocks were in sync | |
| 18:51:34 | dansmith | yeah it's clearly not routable because of the private ip in this case | |
| 18:51:40 | dansmith | uep | |
| 18:52:05 | sean-k-mooney | ah its routing ya | |
| 18:52:40 | sean-k-mooney | that woudl make sense. i know we try and setbale a vxlan or gre mesh usign linux bridge in the multi node jobs | |
| 18:52:47 | sean-k-mooney | and hten use that for neutron tenant networking | |
| 18:52:57 | sean-k-mooney | but i dont know if we actuly route all traffic over that | |
| 18:53:16 | sean-k-mooney | i guess we are not or that is not working properly if we are | |
| 18:54:49 | dansmith | it's not that because this is just nova-compute unable to contact the mq on the controller node because it's not using its public ip | |
| 18:54:54 | dansmith | nothing to do with guest networking | |
| 18:55:14 | sean-k-mooney | what i ment was if we used the ip form that brdige network those might work | |
| 18:55:28 | sean-k-mooney | but ya in devstack we lookup the ip of the node by defualt | |
| 18:55:34 | sean-k-mooney | i dotn think we overried that in the jobs | |
| 18:55:46 | dansmith | right | |
| 18:56:09 | sean-k-mooney | we expct the vms to be on the same neutron network in the josb since nodes in any given nodeset are nto allwos to comefrom diffent providers | |
| 18:56:28 | sean-k-mooney | or at least they were not allowed to do tha tin the past so the josb depend on that | |
| 18:56:38 | sean-k-mooney | changign that would be a breakign change | |
| 18:56:57 | sean-k-mooney | not nessiarly a bad one but not one that our jobs can handel so it woudl need ot be opt in | |
| 20:33:26 | opendevreview | Brett Holman proposed openstack/nova master: Add DMI data to non-x86 architectures https://review.opendev.org/c/openstack/nova/+/953989 | |
| #openstack-nova - 2025-07-04 | |||
| 09:58:07 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Return 404 on bad project ID for 'os-quota-sets' https://review.opendev.org/c/openstack/nova/+/937125 | |
| 10:00:15 | stephenfin | gibi: sean-k-mooney: Can we get https://review.opendev.org/c/openstack/nova/+/952266 in? It's rather annoying when running unit tests 😅 | |
| 10:16:28 | gibi | +2 | |
| 10:17:39 | stephenfin | ty | |
| 10:51:51 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Deprecate v2 API https://review.opendev.org/c/openstack/nova/+/954102 | |
| 10:51:51 | opendevreview | Stephen Finucane proposed openstack/nova master: WIP: api: Remove version controller split https://review.opendev.org/c/openstack/nova/+/954103 | |
| 11:05:24 | sean-k-mooney | stephenfin: sure | |
| 11:25:13 | opendevreview | Biser Milanov proposed openstack/nova stable/2024.1: Hardcode the use of iothreads for KVM. https://review.opendev.org/c/openstack/nova/+/954113 | |
| 11:28:14 | opendevreview | Biser Milanov proposed openstack/nova stable/2024.1: Hardcode the use of iothreads for KVM. https://review.opendev.org/c/openstack/nova/+/954113 | |
| 11:29:42 | opendevreview | Biser Milanov proposed openstack/nova stable/2024.1: StorPool: Pass the instance UUID and device_name to os-brick https://review.opendev.org/c/openstack/nova/+/954115 | |
| 11:30:37 | sp-bmilanov | sean-k-mooney: Hi, you can ignore the backport chain : https://review.opendev.org/c/openstack/nova/+/954115 , it is not meant to actually be merged | |
| 11:30:56 | sean-k-mooney | ack well the master version is not appliable either | |
| 11:31:28 | sp-bmilanov | yep, I am aware there's an effort to go about adding iothreads another way | |
| 11:31:32 | sean-k-mooney | we are not gong to make this a per host config option | |
| 12:50:30 | sean-k-mooney | masahito: by the way even if we are pass the spec freeze i would encurrage you to work on a poc of the iothread and queue patches based on the current version fo hte spec, while we may not have time to do extensive reivew of it if you have a working implemation with tempest test ectra at the sart of the new cycle in october i think it would be reaosnbale ot avocate for | |
| 12:50:32 | sean-k-mooney | reiviewing and completign the overall functionatliy early in the cycle. | |
| 12:51:21 | sean-k-mooney | depending on how early it was done you may even have time to look at the next steps although that less likely to happen in time for 2026.1 | |
| 12:51:48 | gibi | fyi I see significant failure rate in our multi node gate due to rabbit is not accessible from compute1 https://bugs.launchpad.net/nova/+bug/2115980 It is started 3 days ago | |
| 12:52:04 | sean-k-mooney | gibi: ya so that a zuul issue | |
| 12:52:25 | gibi | is it a tracker for it from infra perspective? | |
| 12:52:33 | sean-k-mooney | gibi: tl;dr in the old impelation nodesets were not allowe to be provisioned form differnt providers | |
| 12:52:59 | sean-k-mooney | in the new one it can but that breaks our jobs beacue they use the local ips since it expect all the vms to be on the same neutron network | |
| 12:53:32 | sean-k-mooney | gibi: in terms of tracker i am not sure but dansmith and clarkb where talkign about it last night | |
| 12:53:44 | gibi | OK, reading back then... | |
| 12:53:46 | gibi | thanks | |
| 12:53:51 | sean-k-mooney | they know that his is happenign and i think they are looking to disbale that feature | |
| 12:54:20 | sean-k-mooney | but i dont know if that change has been doen yet, the openstack-infra channle is proably hte best plase to follow up | |
| 12:57:49 | gibi | ack, I've pinged infra about it | |
| 12:59:02 | sean-k-mooney | i belive this is basicly just a boolean we need to flip for our tenant. long term we coudl refactor the multi node role that we use in the devstack job to use the floating ips and streach the deploymnt over the wan but that is likely to cause other issues so i dont think that will pan out | |
| 13:01:28 | gibi | 14:59 < fungi> gibi: there's https://review.opendev.org/954064 which we should be auto-upgrading to within the next 24 hours | |
| 13:01:39 | gibi | so fix is in the pipe | |
| 13:01:59 | sean-k-mooney | cool | |
| 13:02:34 | fungi | yeah, we automatically upgrade our zuul servers through rolling restarts over the weekend | |
| 13:03:32 | sean-k-mooney | fungi: i woudl still suggest that it shoudl be posisble to disable the ablity to use multiple providers at the nodeset or tenant level | |
| 13:03:41 | fungi | booting a node in another provider for a multi-node job should only ever be a fallback in cases where the job would have otherwise been reported as a node failure | |