Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2023-09-01
16:48:49 auniyal nova-ovs-hybrid-plug job failed with POST fail
16:49:07 auniyal because devstack@c-bak.service
16:49:28 dansmith oh, sean changed the zuul yaml
16:49:31 dansmith I didn't even look at that
16:49:34 auniyal couldn't start
16:49:49 dansmith yeah we should just do that in a different patch
16:50:03 dansmith auniyal: can you update sean's patch to just not do the .zuul part and I'll +2+W it?
16:50:23 dansmith move the .zuul part to a second patch on top and we can go from there separate from fixing the gate
16:50:34 auniyal ack
16:50:46 dansmith auniyal: can you do it quick? if not I will, but it would be better if you do and I +W
16:51:00 auniyal yes I am doing
16:51:02 dansmith cool
16:51:32 dansmith ++ glad you were watching those jobs live, I didn't even pay attention to the .zuul changer
16:54:09 dansmith something else is re-enabling c-bak, which is why that happened:
16:54:11 dansmith 2023-09-01 16:19:08.457805 | compute1 | +++ enable_service c-bak
16:56:26 opendevreview Amit Uniyal proposed openstack/nova master: only attempt to clean up dangling bdm if cinder is installed https://review.opendev.org/c/openstack/nova/+/893502
16:56:27 opendevreview Amit Uniyal proposed openstack/nova master: Moved zuul job update https://review.opendev.org/c/openstack/nova/+/893540
16:57:26 dansmith gmann: can you tell us why this is not working? https://review.opendev.org/c/openstack/nova/+/893540/1/.zuul.yaml
16:57:45 dansmith gmann: c-bak ends up re-enabled but fails to start because the rest of cinder is disabled
16:58:24 gmann dansmith: checking
16:59:51 dansmith auniyal: ah, you didn't remove the comment from the commit message about modifying the job, can you fix that real quick?
17:00:01 auniyal yes
17:01:27 sean-k-mooney without the job config tweak we dont actuly knwo that this is fixed
17:01:46 dansmith sean-k-mooney: yes, we have a neutron DNM against it
17:01:48 sean-k-mooney but i see that was factored out into its own change
17:02:21 sean-k-mooney ok i would have prefered to keep them in the same patch btu sure
17:02:39 gmann dansmith: where you are seeing it failing?
17:02:43 sean-k-mooney as long as we eventually get a job without cinder working its fine
17:02:48 dansmith gmann: https://bf80d1d7066e6c071f87-1fbe10cd6d23547f6092d75bc2b558ab.ssl.cf2.rackcdn.com/893502/6/check/nova-ovs-hybrid-plug/4042471/job-output.txt
17:02:55 dansmith gmann: search for "noneextentdatabase"
17:03:14 dansmith sean-k-mooney: yeah, definitely need that, but just fixing that job won't even ensure that neutron is unblocked
17:03:24 dansmith sean-k-mooney: I've got https://review.opendev.org/c/openstack/neutron/+/893538 for that verification
17:03:52 sean-k-mooney cool well i was just checking back to see if there were any issues
17:03:57 opendevreview Amit Uniyal proposed openstack/nova master: only attempt to clean up dangling bdm if cinder is installed https://review.opendev.org/c/openstack/nova/+/893502
17:03:58 opendevreview Amit Uniyal proposed openstack/nova master: zuul update https://review.opendev.org/c/openstack/nova/+/893540
17:04:27 sean-k-mooney auniyal: why did you rebase those?
17:04:35 dansmith sean-k-mooney: to fix the commit message
17:04:51 dansmith sean-k-mooney: if we can figure out why it's not working in devstack, I'll fast-approve that job change or get gmann to help me, it just doesn't work as you had it
17:05:09 dansmith since it was part of the approved initial patch
17:05:13 sean-k-mooney it can wait till monday so if its not workign ill look at it then
17:05:23 dansmith ack
17:05:30 gmann dansmith: sean-k-mooney auniyal this is multinode job and we need to disable the cinder in subnode also
17:05:42 dansmith ahh
17:05:44 sean-k-mooney oh right
17:05:47 gmann neutron might be having example of those
17:06:05 gmann like this one https://github.com/openstack/neutron/blob/ed6023c3473585b4af6980404e7fe36ea152f980/zuul.d/tempest-multinode.yaml#L151
17:06:18 sean-k-mooney its trivial
17:06:19 dansmith yeah
17:06:26 sean-k-mooney it just need to be in teh group vars section
17:06:27 dansmith I'll fix it
17:07:32 opendevreview Dan Smith proposed openstack/nova master: zuul update https://review.opendev.org/c/openstack/nova/+/893540
17:08:09 opendevreview Dan Smith proposed openstack/nova master: Make our nova-ovs-hybrid-plug job omit cinder https://review.opendev.org/c/openstack/nova/+/893540
17:08:32 gmann perfect, it should work now. will keep eyes on it
17:08:43 dansmith thanks gmann ;)
17:08:47 sean-k-mooney its slightly overkill but ya it will
17:09:16 sean-k-mooney the sub node only has cinder volume and not swift bits
17:09:25 sean-k-mooney but it does not hurt to be explcit
17:09:31 dansmith it clearly had c-bak
17:09:50 sean-k-mooney really i tought that was only on the contoler
17:09:57 dansmith that was the failure
17:10:11 sean-k-mooney in anycase it does not hurt to disabel them all explictly
17:10:23 dansmith c-bak was trying to start and didn't find the cinder database because it wasn't setup on the main node
17:10:38 sean-k-mooney ack we might have moved that at some point
17:11:53 gmann yeah, we do not enable swift in subnode by default but never know which child job doing might be doing/will do that so disable explicitly is better
17:12:59 sean-k-mooney ok im going to go mostly afk. i will skim the logs later tonight but otherwise ill cat to ye on tuesday? its a long weekend in the US right
17:13:06 dansmith yup
17:13:12 sean-k-mooney cool well enjoy
17:13:26 dansmith thanks, o/
17:14:06 gmann sean-k-mooney: o/ enjoy weekend
17:33:01 dansmith auniyal: devstack finished on the subnode this time
17:33:23 auniyal ack
17:34:33 auniyal job "openstacksdk-functional-devstack" is still running
17:35:07 dansmith I meant on the top patch to change the job
17:35:30 dansmith the one where we had to add the c-bak settings to the subnode
17:37:07 auniyal .zuul patch where `c-bak: false` is set
17:42:21 dansmith yeah
17:43:58 auniyal in here https://zuul.openstack.org/status#893540 - nova-ovs-hybrid-plug: failing and openstacksdk-functional-devstack: in-progress
17:44:06 auniyal unit and functional tests are passed
17:44:30 auniyal okay you meant deployment finished without issue
17:44:35 auniyal right ?
17:50:27 elodilles hi, nova team! fyi, train-eol patch has merged & the branches are deleted: https://paste.opendev.org/show/bVfeofByKRN0ZrNX779o/
17:50:37 auniyal this is not a regular thing right - https://zuul.opendev.org/t/openstack/build/475dab49cd33437a96ba81437779b875/log/controller/logs/screen-n-cpu.txt#14515
17:50:47 dansmith auniyal: yeah
17:52:09 gmann elodilles: thanks
17:52:17 elodilles gmann: np
21:05:16 dansmith gmann: this is a different failure: https://zuul.opendev.org/t/openstack/build/475dab49cd33437a96ba81437779b875
21:05:35 dansmith the one we were trying to resolve was where it tried to start c-bak on the subnode and failed
21:06:00 dansmith oh, but, it's running an evacuate test on a bfv instance which clearly won't work, I see
21:06:09 dansmith so we need to remove that post action or whatever
21:06:43 gmann dansmith: I mean the failure it had before for evacuate test
21:07:03 dansmith okay gotcha
21:07:18 gmann yeah, or separate the test with cinder availability or so?
21:09:43 opendevreview Merged openstack/nova master: only attempt to clean up dangling bdm if cinder is installed https://review.opendev.org/c/openstack/nova/+/893502
21:12:46 dansmith gmann: I don't think there's any reason we should be running that post test on the ovs job, I think it's just inherited and wasting time
21:13:47 dansmith I'll leave that to sean on monday
21:14:25 gmann dansmith: not sure, but it job definition explicitly says about it "Run move operations, reboot, and evacuation (via the same post-run hook
21:14:25 gmann as the nova-live-migration job) tests with the OVS network backend "
21:14:54 gmann I think we can avoid running bfv evacuate test if cinder disabled.
21:18:17 dansmith yep, that's what I meant and said in my comment
21:23:09 gmann yeah

Earlier   Later