| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-09-13 | |||
| 17:22:47 | sean-k-mooney | ok but it has not release yet right | |
| 17:22:53 | JayF | I want it to land; but I don't want to break people and my confidence in this feature functioning is extremely low | |
| 17:22:57 | JayF | sean-k-mooney: That's been out since A | |
| 17:23:06 | JayF | sean-k-mooney: so I likely need to validate that broken behavior and fix it | |
| 17:23:25 | JayF | sean-k-mooney: or it's possible it's not broken; but I need to check that and I don't think RC is the time to be answering questions like that | |
| 17:24:23 | sean-k-mooney | ack | |
| 17:24:29 | dansmith | indeed. | |
| 17:24:41 | sean-k-mooney | it sound like there are enough things in flight that we wont knwo for a while :( | |
| 17:25:21 | opendevreview | Sylvain Bauza proposed openstack/nova master: Revert "Make compute node rebalance safter" https://review.opendev.org/c/openstack/nova/+/894944 | |
| 17:25:22 | opendevreview | Sylvain Bauza proposed openstack/nova master: Revert "Add nova-manage ironic-compute-node-move" https://review.opendev.org/c/openstack/nova/+/894945 | |
| 17:25:22 | opendevreview | Sylvain Bauza proposed openstack/nova master: Revert "Limit nodes by ironic shard key" https://review.opendev.org/c/openstack/nova/+/894946 | |
| 17:25:23 | opendevreview | Sylvain Bauza proposed openstack/nova master: Revert "Deprecate ironic.peer_list" https://review.opendev.org/c/openstack/nova/+/894947 | |
| 17:25:49 | dansmith | bauzas: we need to leave the deprecation in place, are you going to do a new one to deprecate with different wording? | |
| 17:26:35 | bauzas | dansmith: I thought JayF said "unplug the whole stack since we don't want to deprecate until shards exist" | |
| 17:26:54 | dansmith | bauzas: no, we should deprecate now, but not remove until shards is in place | |
| 17:27:02 | JayF | My opinion on deprecation is null; my opinion on *removal* is "after there has been at least one SLURP with shard+peer_list in a release together" | |
| 17:27:21 | dansmith | yup | |
| 17:27:48 | sean-k-mooney | +1 | |
| 17:27:50 | bauzas | o ok | |
| 17:27:55 | dansmith | bauzas: good with ninja approvals on the reverts right? | |
| 17:28:11 | bauzas | dansmith: I think it's even documented in our docs | |
| 17:28:11 | sean-k-mooney | i mean i can also just hit them | |
| 17:28:24 | sean-k-mooney | bauzas: it is but not really for this | |
| 17:28:34 | sean-k-mooney | the fast revert poicly does allow it however | |
| 17:28:40 | bauzas | well, this is a broken piece | |
| 17:28:50 | bauzas | fast reverts apply to this | |
| 17:29:02 | dansmith | bauzas: yep, just confirming.. did the first three | |
| 17:29:08 | bauzas | so | |
| 17:29:18 | bauzas | I need to look at the deprecation patch | |
| 17:29:21 | bauzas | to see the wordings | |
| 17:30:16 | dansmith | specifically we need to keep something like "Running multiple nova-compute processes that point at the same conductor group is now deprecated" | |
| 17:30:31 | dansmith | and the conf rename and the deprecation reason there | |
| 17:30:38 | dansmith | however we do that is fine | |
| 17:31:01 | bauzas | I just did read the whole patch | |
| 17:31:08 | bauzas | and I think I can abandon my revert | |
| 17:31:19 | bauzas | the deprecation just says the truth | |
| 17:33:11 | bauzas | dansmith: JayF: remind me my SLURP knowledge | |
| 17:33:22 | bauzas | if we say 'we deprecate in B' | |
| 17:33:41 | bauzas | that means we can't hardly remove in C, provided ops wouldn't have had the memo | |
| 17:33:44 | sean-k-mooney | if we deprecate in be we still have to keep it to have the deprecateion released in C | |
| 17:33:45 | dansmith | we also need to deprecate in C and then we can remove in E, assuming C has the shards as usable | |
| 17:33:57 | JayF | Yeah, that matches my understanding. | |
| 17:33:59 | sean-k-mooney | so first removal is D regardless of when we deprecate | |
| 17:34:11 | bauzas | yeah | |
| 17:34:16 | dansmith | right, we could remove in D | |
| 17:34:25 | dansmith | first slurp it can be missing from is E | |
| 17:34:30 | bauzas | we have a couple of deprecations this cycle | |
| 17:34:40 | bauzas | https://docs.openstack.org/releasenotes/nova/unreleased.html#deprecation-notes | |
| 17:34:48 | sean-k-mooney | yep which is fine | |
| 17:34:51 | bauzas | so I'll ensure we don't remove anything next cycle | |
| 17:35:01 | sean-k-mooney | we will be removing thing next cycle | |
| 17:35:03 | sean-k-mooney | just not those | |
| 17:35:27 | sean-k-mooney | we have deprecated things from A and older that we can remove | |
| 17:35:46 | sean-k-mooney | that is ok because A was a slurp | |
| 17:36:04 | sean-k-mooney | so if it was deprected in A we can remove if it was deprected in B we cant until D | |
| 17:36:26 | sean-k-mooney | ok im going to call it a day | |
| 17:36:30 | sean-k-mooney | o/ | |
| 17:37:56 | bauzas | JayF: dansmith: updated the cycle highlights in order to remove any occurrence of the shards https://review.opendev.org/c/openstack/releases/+/894213 | |
| 17:38:04 | bauzas | I'd appreciate if you could quickly review it | |
| 17:38:06 | dansmith | bauzas: waaaay ahead of you | |
| 17:38:52 | bauzas | :) | |
| 17:39:21 | bauzas | okay, I gonna call it a wrap too, but I'll poke around after dinner, since we have a couple of changes upfront in the gate | |
| 17:42:12 | opendevreview | Sylvain Bauza proposed openstack/nova master: Add a Bobcat prelude section https://review.opendev.org/c/openstack/nova/+/894940 | |
| 17:42:29 | dansmith | bauzas: I've seen this fail twice this morning already, not sure if something changed or not: https://b5ee1ed3653a458879e1-60fa9bbec8248937c3af4b3a8047f40b.ssl.cf2.rackcdn.com/893540/6/check/nova-live-migration/6e12356/testr_results.html | |
| 17:42:57 | bauzas | dansmith: yeah that's a known bug | |
| 17:43:13 | bauzas | https://bugs.launchpad.net/neutron/+bug/1940425 | |
| 17:43:32 | bauzas | it's old but its occurrences raised recently | |
| 17:43:45 | dansmith | bauzas: yeah, but I haven't seen it very much in the past few months, but several times today now | |
| 17:43:50 | dansmith | okay | |
| 17:44:27 | dansmith | critical but unfixed in neutron since 2021..eesh | |
| 17:44:29 | bauzas | I haven't pinged lajoskatona ralonsoh nd the other neutron cores | |
| 17:44:37 | bauzas | (yet) | |
| 17:45:20 | bauzas | dansmith: well, I guess they just don't check the number of critical bugs they have on a weekly basis, that's it :D | |
| 18:18:06 | gmann | dansmith: seems we need to put this whole things under cinder check https://review.opendev.org/c/openstack/nova/+/893540/6/roles/run-evacuate-hook/files/test_evacuate.sh#58 | |
| 18:18:22 | gmann | it is failing there https://zuul.opendev.org/t/openstack/build/5efc0aaf88874c45b2eb68d8e9cf4a0d/console | |
| 18:18:24 | dansmith | gmann: dammit | |
| 18:18:45 | dansmith | I was just going to fix one thing, see how it goes, fix another, etc | |
| 18:18:54 | dansmith | you keep pointing out all the problems and are making me look bad :) | |
| 18:29:53 | opendevreview | Dan Smith proposed openstack/nova master: Make our nova-ovs-hybrid-plug job omit cinder https://review.opendev.org/c/openstack/nova/+/893540 | |
| 18:43:42 | lajoskatona | dansmithm bauzas: I think we have now this bug for that issue: https://bugs.launchpad.net/neutron/+bug/2033887 and this is the patch series for that: https://review.opendev.org/q/Ifc2d37e2042fad43dd838821953defd99a5f8665 | |
| 18:45:41 | dansmith | lajoskatona: col | |
| 18:45:44 | dansmith | er cool even | |
| 18:54:02 | gmann | dansmith: you doing so much work and on gate stability makes you always better and not bad :) | |
| 18:54:24 | dansmith | heh | |
| 19:03:44 | bauzas | dansmith: fwiw, also seeing more kernel crashes like https://bb94d2825af897cfcd12-f6f1806a4829a343b7540be166a34ea9.ssl.cf5.rackcdn.com/860829/4/check/nova-next/bf91602/testr_results.html | |
| 19:04:05 | bauzas | again an already known issue but more failures | |
| 19:04:53 | dansmith | bauzas: always with no space left on device? | |
| 19:05:08 | bauzas | don't think so | |
| 19:05:18 | bauzas | I have to doublecheck tho | |
| 19:05:24 | dansmith | that's during boot, so it looks more like maybe a broken image creation or if it's on ceph, maybe a backend problem | |
| 19:06:05 | dansmith | oh volume boot, so yeah something wrong with the actual volume | |
| 19:06:21 | bauzas | yeah probably a ceph RC | |
| 19:06:26 | dansmith | it never mounted and thus couldn't pivot over and the ENOSPC comes from having not mounted something writable on the target | |
| 19:20:52 | bauzas | hmpfff | |
| 19:21:35 | bauzas | Sep 13 13:49:37.042893 np0035238841 nova-compute[40581]: FileNotFoundError: [Errno 2] No such file or directory: 'multipathd' | |
| 19:21:54 | dansmith | always logged by brick AFAIK :( | |
| 19:23:17 | bauzas | I wish I would be Neo 'operator, get me all the knowledge about chopters, err. volume bindings in OpenStack" | |
| 19:24:40 | bauzas | "Tank, I need a pilot program for B-212 helicopter. Hurry.� | |
| 19:24:40 | bauzas | "Tank, I need a pilot program for B-212 helicopter. Hurry.” | |
| 19:24:54 | bauzas | (found the right catchphrase) | |