| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-05-11 | |||
| 12:02:23 | sean-k-mooney | gibi: thanks for reviewing | |
| 12:08:01 | opendevreview | Merged openstack/os-vif master: remove focal based jobs https://review.opendev.org/c/openstack/os-vif/+/882755 | |
| 12:15:16 | bauzas | stable cores are needed for 2023.1 https://review.opendev.org/c/openstack/nova/+/882783 | |
| 12:15:16 | opendevreview | Balazs Gibizer proposed openstack/nova stable/2023.1: Bump nova-ceph-multstore timeout https://review.opendev.org/c/openstack/nova/+/882784 | |
| 12:16:12 | gibi | bauzas, dansmith, melwitt: The nova-ceph-mutlistore timeout bump is merged to master so I cherry-picked it to 2023.1 ^^ | |
| 12:16:37 | bauzas | gibi: done | |
| 12:16:51 | bauzas | gibi: sean-k-mooney: could you please look at https://review.opendev.org/c/openstack/nova/+/882783 ? | |
| 12:17:44 | sean-k-mooney | yep | |
| 12:18:39 | sean-k-mooney | elodilles: :) | |
| 12:18:49 | elodilles | done :) | |
| 12:18:53 | sean-k-mooney | bauzas: they are both now +2w'd | |
| 12:18:59 | gibi | \o/ | |
| 12:20:54 | sean-k-mooney | by the way i dont care that w is a letter can cant be past tense but +2w'd is definelty a thing :) and the 'd gets that point across | |
| 12:21:46 | sean-k-mooney | english sometime allows you to convay info in a way that would make tech writers cry | |
| 12:22:40 | sean-k-mooney | and im ok with that, we all know i mainly comunicate in seanspeak anyway :) | |
| 12:22:41 | elodilles | yepp, i also use EOL'd +2'd +W'd o:) | |
| 12:25:52 | elodilles | i don't remember where i saw that 1st time, but it can be understood, so i started to use that myself too | |
| 12:27:46 | sean-k-mooney | i dont think any one has ever objected but from a grammer rules point of view english allows you to syntasies past tense verbs this way. i just know my english teach would have been unhappy | |
| 12:28:22 | sean-k-mooney | with makeing an acronym both be a verb and have tence | |
| 12:28:50 | sean-k-mooney | ' is a very powerful thing :) | |
| 12:30:29 | elodilles | :) | |
| 12:43:31 | Uggla | @bauzas, gibi, if you can have a look at those small patches https://review.opendev.org/c/openstack/tempest/+/882822/1, https://review.opendev.org/c/openstack/tempest/+/882823/2 as I said tuesday that may help to find the bug. | |
| 12:54:37 | ykarel | sean-k-mooney, bauzas can you please revisit https://review.opendev.org/c/openstack/nova/+/868419 when get a chance | |
| 12:56:17 | opendevreview | Merged openstack/nova stable/2023.1: Enable use of service user token with admin context https://review.opendev.org/c/openstack/nova/+/882859 | |
| 13:23:36 | dansmith | gibi: thanks! | |
| 13:28:40 | opendevreview | Amit Uniyal proposed openstack/nova stable/wallaby: Have host look for CPU controller of cgroupsv2 location. https://review.opendev.org/c/openstack/nova/+/882939 | |
| 13:29:57 | bauzas | dansmith: thanks btw. for having +2d on my way | |
| 13:30:20 | bauzas | upstream CVE bugfixes are already merged by master and 2023.1 \o/ | |
| 13:32:06 | dansmith | bauzas: no problem | |
| 13:42:10 | opendevreview | Merged openstack/os-vif master: set default qos policy https://review.opendev.org/c/openstack/os-vif/+/881751 | |
| 13:52:53 | gibi | Uggla: done. I don't have +2 rights in tempest but those patches looks good | |
| 13:53:52 | Uggla | gibi, cool thx. | |
| 13:56:30 | dansmith | bauzas: gibi so just an update, based on my opensearch digging, I don't think we have seen any volume detach failures in the last week with the exception of cases where either ceph oomed or the guest had a kernel panic | |
| 13:56:50 | dansmith | not definitive for sure, but based on previous behavior, I think that's massively better | |
| 14:21:24 | opendevreview | Merged openstack/nova stable/2023.1: Revert "Debug Nova APIs call failures" https://review.opendev.org/c/openstack/nova/+/882783 | |
| 14:25:11 | bauzas | dansmith: bravo to you | |
| 14:25:34 | dansmith | also, the cve backport just failed on the backport validator | |
| 14:25:42 | dansmith | maybe just github not updated yet? | |
| 14:25:45 | opendevreview | Elod Illes proposed openstack/nova master: CI: fix backport validator for new branch naming https://review.opendev.org/c/openstack/nova/+/882956 | |
| 14:26:52 | elodilles | bauzas dansmith : we need this fix as new branch naming broke our validator :S ^^^ | |
| 14:27:05 | dansmith | ah | |
| 14:27:42 | dansmith | +2 | |
| 14:27:50 | elodilles | thx | |
| 14:27:53 | dansmith | elodilles: nice job thanks | |
| 14:28:28 | elodilles | i'll propose the backports if this is about to merge | |
| 14:28:34 | gibi | dansmith: nice results! | |
| 14:29:38 | dansmith | cinder is still running with validations disabled and so they're hitting all the ones we used to but we're not, so that's also a nice A/B comparison :) | |
| 14:48:22 | opendevreview | Oleksandr Klymenko proposed openstack/nova master: Host removed from AZ when service is manually disabled https://review.opendev.org/c/openstack/nova/+/882957 | |
| 15:05:01 | opendevreview | Elod Illes proposed openstack/nova stable/2023.1: CI: fix backport validator for new branch naming https://review.opendev.org/c/openstack/nova/+/882964 | |
| 15:06:15 | opendevreview | Elod Illes proposed openstack/nova stable/zed: CI: fix backport validator for new branch naming https://review.opendev.org/c/openstack/nova/+/882965 | |
| 15:07:26 | opendevreview | Elod Illes proposed openstack/nova stable/yoga: CI: fix backport validator for new branch naming https://review.opendev.org/c/openstack/nova/+/882966 | |
| 15:08:38 | opendevreview | Elod Illes proposed openstack/nova stable/xena: CI: fix backport validator for new branch naming https://review.opendev.org/c/openstack/nova/+/882967 | |
| 15:09:49 | opendevreview | Elod Illes proposed openstack/nova stable/wallaby: CI: fix backport validator for new branch naming https://review.opendev.org/c/openstack/nova/+/882968 | |
| 15:11:00 | opendevreview | Elod Illes proposed openstack/nova stable/victoria: CI: fix backport validator for new branch naming https://review.opendev.org/c/openstack/nova/+/882969 | |
| 15:12:10 | opendevreview | Elod Illes proposed openstack/nova stable/ussuri: CI: fix backport validator for new branch naming https://review.opendev.org/c/openstack/nova/+/882970 | |
| 15:13:22 | opendevreview | Elod Illes proposed openstack/nova stable/train: CI: fix backport validator for new branch naming https://review.opendev.org/c/openstack/nova/+/882971 | |
| 15:29:22 | opendevreview | Sylvain Bauza proposed openstack/nova stable/zed: Revert "Debug Nova APIs call failures" https://review.opendev.org/c/openstack/nova/+/882786 | |
| 16:17:57 | bauzas | folks, I'm also saying it loudly, tomorrow I'll be working and looking at specs | |
| 16:52:21 | dansmith | melwitt: the backport checker fix is going to fail on functional, that db table race thing | |
| 16:54:43 | melwitt | argh | |
| 16:54:58 | dansmith | is there a bug open for that/ | |
| 16:55:56 | melwitt | yes, sec | |
| 16:56:10 | dansmith | I dunno why zuul hasn't kicked it out so I can recheck it yet | |
| 16:56:30 | melwitt | I believe it's this one https://bugs.launchpad.net/nova/+bug/1946339 | |
| 16:57:14 | dansmith | hmm, similar at least | |
| 16:57:18 | melwitt | gibi has done a lot of work to improve the situation but it's a gnarly issue | |
| 16:58:16 | dansmith | okay yeah that's the same | |
| 16:58:20 | JayF | Hey; I re-proposed the Ironic sharding spec for this cycle about 2 weeks ago. It's not gotten any reviews. If anyone can take a look I'd appreciate it: https://review.opendev.org/c/openstack/nova-specs/+/881643 | |
| 16:58:50 | melwitt | there is a change I think we could do that might help but I haven't proposed it yet bc it hadn't been happening very often for a long time there | |
| 16:59:14 | dansmith | melwitt: okay gibi just commented on the bug that we've seen an. uptick recently | |
| 16:59:37 | melwitt | yeah, I just saw that too | |
| 17:12:49 | dansmith | fungi: there's something that has completed all our jobs in gate that we need to recheck, but it's just sitting there in the queue and I'm not sure why | |
| 17:13:24 | dansmith | it's already identified as failed and out of the queue, but not.. uh, reporting or whatever | |
| 17:13:25 | clarkb | dansmith: because the changes ahead of it haven't finished | |
| 17:13:37 | clarkb | only the first thing in the queue can report | |
| 17:13:48 | clarkb | doing so removes it from the queue and then the next item can be processed | |
| 17:13:54 | dansmith | I thought if nothing in front could have caused the failure on that job it would come out immediately | |
| 17:14:04 | clarkb | zuul doesn't have that information so can't do that | |
| 17:14:14 | dansmith | hmm, okay | |
| 17:14:25 | clarkb | by putting things in the same queue you are asserting a failure in one may be caused by the other | |
| 17:14:31 | clarkb | and zuul is operating on that knowledge | |
| 17:14:37 | dansmith | okay | |
| 17:15:03 | dansmith | but it already shows it as failed out (meaning the fork in the line) so I thought that was it saying it knows the things behind it no longer depend | |
| 17:15:41 | clarkb | correct the things behind it no longer depend on it. But the things ahead of it may be where the actual bug is | |
| 17:16:02 | clarkb | in that case you want to evict the broken stuff ahead and restart the things ehind | |
| 17:16:18 | dansmith | so the green checks behind this are based on skipping it or with it applied? | |
| 17:16:38 | clarkb | the green checks behind are based on skipping the one that has failed | |
| 17:16:41 | dansmith | I assume with it applied and they'll restart if it decides it was legit to kick it out? | |
| 17:16:47 | dansmith | hmm okay | |
| 17:16:50 | clarkb | the unknown is the not yet completed jobs ahead of it | |
| 17:16:55 | dansmith | so if it doesn't get kicked out they restart? | |
| 17:16:57 | fungi | but they'll all be tested again from scratch if something else ahead of all of those fails a job | |
| 17:17:49 | clarkb | you have 5 changes, 6th is a failure, then X behind. Zuul does not know if the failure was caused by the 5 changes at the front so it does not completely evict the 6th until it processes the 5 ahead of it | |
| 17:17:50 | fungi | until all changes ahead of the failing change merge successfully, zuul can't be sure that there's something wrong with that change | |
| 17:18:18 | dansmith | it's too bad we can't mark a job as isolated or something, because this is only running nova unit tests, but it's held up as if it has the same dependencies as something with a tempest (which is why the queue needs to be shared) | |
| 17:18:28 | dansmith | obviously not a very common case | |
| 17:19:12 | dansmith | fungi: ack, the fork in the graph makes it look to me like it's already "out" but yeah okay | |
| 17:19:36 | dansmith | "out of consideration" I should say | |
| 17:20:37 | dansmith | but yeah I guess I thought there was job affinity and not just place-in-the-queue | |
| 17:24:31 | fungi | right, if the failure were due to a change ahead of it in an oslo lib, the bug in that oslo change might fail on some other job which exposed the same bug through some other tests which aren't the nova unit test job | |