| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-09-01 | |||
| 15:42:28 | sean-k-mooney | i cant do the second early out | |
| 15:42:58 | opendevreview | sean mooney proposed openstack/nova master: only attempt to clean up dangling bdm if cinder is installed https://review.opendev.org/c/openstack/nova/+/893502 | |
| 15:47:08 | sean-k-mooney | ok i need to run now so dansmith bauzas can ye take this from here. | |
| 15:47:20 | bauzas | ack | |
| 16:20:40 | bauzas | dansmith: can you please validate https://review.opendev.org/c/openstack/nova/+/893502 ? | |
| 16:23:48 | dansmith | bauzas: +W.. we might want to get a depends-on running against it in parallel to make sure it really solves the problem | |
| 16:23:59 | dansmith | was that a devstack base job that hit the original problem? | |
| 16:24:10 | bauzas | dansmith: yeah, some neutron ones | |
| 16:24:18 | bauzas | https://bugs.launchpad.net/tempest/+bug/2033752 | |
| 16:24:48 | dansmith | neutron ones or devstack ones? | |
| 16:25:11 | dansmith | ykarel: still around? | |
| 16:25:12 | bauzas | sorry https://zuul.openstack.org/job/neutron-ovn-base | |
| 16:25:47 | ykarel | dansmith, yes kind off | |
| 16:25:48 | bauzas | https://zuul.openstack.org/job/tempest-integrated-networking doesn't skip cinder, but his child above skips it | |
| 16:26:03 | dansmith | ykarel: I'll submit a DNM neutron patch to test | |
| 16:26:44 | dansmith | ykarel: https://review.opendev.org/c/openstack/neutron/+/893538 | |
| 16:26:51 | ykarel | dansmith, ack | |
| 16:27:33 | bauzas | I'll bail out now | |
| 16:27:39 | dansmith | bauzas: ack | |
| 16:27:41 | bauzas | long day for a long week | |
| 16:28:00 | dansmith | bauzas: puh lease man.. back for one week after gone for what, three? | |
| 16:28:11 | dansmith | I'm going to really enjoy that ONE day off I have next week ;) | |
| 16:28:15 | bauzas | 3 and a half :) | |
| 16:28:21 | dansmith | pssh | |
| 16:28:32 | bauzas | yeah, but going back on the ff week, lovin' <3 | |
| 16:29:04 | dansmith | bauzas: go on, get out of here.. I don't want to over stress your system :) | |
| 16:29:14 | bauzas | dansmith: anyway, enjoy your own timeoff :) | |
| 16:29:20 | dansmith | heh thanks | |
| 16:46:56 | auniyal | dansmith - https://bf80d1d7066e6c071f87-1fbe10cd6d23547f6092d75bc2b558ab.ssl.cf2.rackcdn.com/893502/6/check/nova-ovs-hybrid-plug/4042471/job-output.txt - "DBNonExistentDatabase" | |
| 16:47:37 | auniyal | and https://zuul.opendev.org/t/openstack/build/40424718c3524bb78ca22cb9d5293b6c/log/controller/logs/screen-n-cpu.txt#14946 - this one is oslo_service.periodic_task | |
| 16:48:21 | dansmith | auniyal: what about it? | |
| 16:48:49 | auniyal | nova-ovs-hybrid-plug job failed with POST fail | |
| 16:49:07 | auniyal | because devstack@c-bak.service | |
| 16:49:28 | dansmith | oh, sean changed the zuul yaml | |
| 16:49:31 | dansmith | I didn't even look at that | |
| 16:49:34 | auniyal | couldn't start | |
| 16:49:49 | dansmith | yeah we should just do that in a different patch | |
| 16:50:03 | dansmith | auniyal: can you update sean's patch to just not do the .zuul part and I'll +2+W it? | |
| 16:50:23 | dansmith | move the .zuul part to a second patch on top and we can go from there separate from fixing the gate | |
| 16:50:34 | auniyal | ack | |
| 16:50:46 | dansmith | auniyal: can you do it quick? if not I will, but it would be better if you do and I +W | |
| 16:51:00 | auniyal | yes I am doing | |
| 16:51:02 | dansmith | cool | |
| 16:51:32 | dansmith | ++ glad you were watching those jobs live, I didn't even pay attention to the .zuul changer | |
| 16:54:09 | dansmith | something else is re-enabling c-bak, which is why that happened: | |
| 16:54:11 | dansmith | 2023-09-01 16:19:08.457805 | compute1 | +++ enable_service c-bak | |
| 16:56:26 | opendevreview | Amit Uniyal proposed openstack/nova master: only attempt to clean up dangling bdm if cinder is installed https://review.opendev.org/c/openstack/nova/+/893502 | |
| 16:56:27 | opendevreview | Amit Uniyal proposed openstack/nova master: Moved zuul job update https://review.opendev.org/c/openstack/nova/+/893540 | |
| 16:57:26 | dansmith | gmann: can you tell us why this is not working? https://review.opendev.org/c/openstack/nova/+/893540/1/.zuul.yaml | |
| 16:57:45 | dansmith | gmann: c-bak ends up re-enabled but fails to start because the rest of cinder is disabled | |
| 16:58:24 | gmann | dansmith: checking | |
| 16:59:51 | dansmith | auniyal: ah, you didn't remove the comment from the commit message about modifying the job, can you fix that real quick? | |
| 17:00:01 | auniyal | yes | |
| 17:01:27 | sean-k-mooney | without the job config tweak we dont actuly knwo that this is fixed | |
| 17:01:46 | dansmith | sean-k-mooney: yes, we have a neutron DNM against it | |
| 17:01:48 | sean-k-mooney | but i see that was factored out into its own change | |
| 17:02:21 | sean-k-mooney | ok i would have prefered to keep them in the same patch btu sure | |
| 17:02:39 | gmann | dansmith: where you are seeing it failing? | |
| 17:02:43 | sean-k-mooney | as long as we eventually get a job without cinder working its fine | |
| 17:02:48 | dansmith | gmann: https://bf80d1d7066e6c071f87-1fbe10cd6d23547f6092d75bc2b558ab.ssl.cf2.rackcdn.com/893502/6/check/nova-ovs-hybrid-plug/4042471/job-output.txt | |
| 17:02:55 | dansmith | gmann: search for "noneextentdatabase" | |
| 17:03:14 | dansmith | sean-k-mooney: yeah, definitely need that, but just fixing that job won't even ensure that neutron is unblocked | |
| 17:03:24 | dansmith | sean-k-mooney: I've got https://review.opendev.org/c/openstack/neutron/+/893538 for that verification | |
| 17:03:52 | sean-k-mooney | cool well i was just checking back to see if there were any issues | |
| 17:03:57 | opendevreview | Amit Uniyal proposed openstack/nova master: only attempt to clean up dangling bdm if cinder is installed https://review.opendev.org/c/openstack/nova/+/893502 | |
| 17:03:58 | opendevreview | Amit Uniyal proposed openstack/nova master: zuul update https://review.opendev.org/c/openstack/nova/+/893540 | |
| 17:04:27 | sean-k-mooney | auniyal: why did you rebase those? | |
| 17:04:35 | dansmith | sean-k-mooney: to fix the commit message | |
| 17:04:51 | dansmith | sean-k-mooney: if we can figure out why it's not working in devstack, I'll fast-approve that job change or get gmann to help me, it just doesn't work as you had it | |
| 17:05:09 | dansmith | since it was part of the approved initial patch | |
| 17:05:13 | sean-k-mooney | it can wait till monday so if its not workign ill look at it then | |
| 17:05:23 | dansmith | ack | |
| 17:05:30 | gmann | dansmith: sean-k-mooney auniyal this is multinode job and we need to disable the cinder in subnode also | |
| 17:05:42 | dansmith | ahh | |
| 17:05:44 | sean-k-mooney | oh right | |
| 17:05:47 | gmann | neutron might be having example of those | |
| 17:06:05 | gmann | like this one https://github.com/openstack/neutron/blob/ed6023c3473585b4af6980404e7fe36ea152f980/zuul.d/tempest-multinode.yaml#L151 | |
| 17:06:18 | sean-k-mooney | its trivial | |
| 17:06:19 | dansmith | yeah | |
| 17:06:26 | sean-k-mooney | it just need to be in teh group vars section | |
| 17:06:27 | dansmith | I'll fix it | |
| 17:07:32 | opendevreview | Dan Smith proposed openstack/nova master: zuul update https://review.opendev.org/c/openstack/nova/+/893540 | |
| 17:08:09 | opendevreview | Dan Smith proposed openstack/nova master: Make our nova-ovs-hybrid-plug job omit cinder https://review.opendev.org/c/openstack/nova/+/893540 | |
| 17:08:32 | gmann | perfect, it should work now. will keep eyes on it | |
| 17:08:43 | dansmith | thanks gmann ;) | |
| 17:08:47 | sean-k-mooney | its slightly overkill but ya it will | |
| 17:09:16 | sean-k-mooney | the sub node only has cinder volume and not swift bits | |
| 17:09:25 | sean-k-mooney | but it does not hurt to be explcit | |
| 17:09:31 | dansmith | it clearly had c-bak | |
| 17:09:50 | sean-k-mooney | really i tought that was only on the contoler | |
| 17:09:57 | dansmith | that was the failure | |
| 17:10:11 | sean-k-mooney | in anycase it does not hurt to disabel them all explictly | |
| 17:10:23 | dansmith | c-bak was trying to start and didn't find the cinder database because it wasn't setup on the main node | |
| 17:10:38 | sean-k-mooney | ack we might have moved that at some point | |
| 17:11:53 | gmann | yeah, we do not enable swift in subnode by default but never know which child job doing might be doing/will do that so disable explicitly is better | |
| 17:12:59 | sean-k-mooney | ok im going to go mostly afk. i will skim the logs later tonight but otherwise ill cat to ye on tuesday? its a long weekend in the US right | |
| 17:13:06 | dansmith | yup | |
| 17:13:12 | sean-k-mooney | cool well enjoy | |
| 17:13:26 | dansmith | thanks, o/ | |
| 17:14:06 | gmann | sean-k-mooney: o/ enjoy weekend | |