| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-09-29 | |||
| 09:08:47 | lyarwood | so master grenade deploys a stable/xena env and then upgrades that to master | |
| 09:10:08 | bauzas | gibi: I stupidely forgot to +1 the RC2 patches yesterday before leaving | |
| 09:10:14 | bauzas | :facepalm: | |
| 09:10:29 | bauzas | now we need elodilles_pto but as his nick says, he's on PTO :) | |
| 09:12:17 | gibi | lyarwood: if we do this https://review.opendev.org/c/openstack/devstack/+/811518/2/lib/tempest then we need to do this as well https://review.opendev.org/c/openstack/grenade/+/811542/1/.zuul.yaml on stable. But I agre that we don't have to enable trunk on stable, skip is OK to me too. | |
| 09:12:59 | lyarwood | yeah I agree with the comment on the stable/wallaby change that we shouldn't be adding tests to stable this late on | |
| 09:13:01 | gibi | bauzas: I think other release cores can approve the RC2 we don't necessary need to wait for elodilles_pto. | |
| 09:13:12 | gibi | anyhow he is back tomorrow so no worries | |
| 09:14:13 | gibi | lyarwood: ack, we can discuss this with gmann when he is up | |
| 09:20:45 | lyarwood | kk | |
| 10:56:11 | opendevreview | Lee Yarwood proposed openstack/nova master: nova-manage: Ensure mountpoint is passed when updating attachment https://review.opendev.org/c/openstack/nova/+/811713 | |
| 11:17:24 | opendevreview | Lee Yarwood proposed openstack/nova master: nova-manage: Always get BDMs using get_by_volume_and_instance https://review.opendev.org/c/openstack/nova/+/811716 | |
| 11:26:10 | lyarwood | https://review.opendev.org/c/openstack/nova/+/811118 - cool the check queue is passing with the grenade fix | |
| 11:27:28 | gibi | \o/ | |
| 12:00:50 | mdbooth | 👋 Are we confident that https://review.opendev.org/c/openstack/devstack/+/811399 fixed the devstack issue on Ubuntu? My devstack-using CI job still seems to be failing: | |
| 12:00:51 | mdbooth | https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api-provider-openstack/1009/pull-cluster-api-provider-openstack-e2e-test/1443169418262614016/artifacts/logs/cloud-final.log | |
| 12:02:32 | mdbooth | I can see the following in the devstack output: "ProxyPass "/placement" "unix:/var/run/uwsgi/placement-api.socket|uwsgi://uwsgi-uds-placement-api" retry=0", which looks like it contains the fix. However I'm still seeing the borked GETs to placement | |
| 12:04:33 | mdbooth | I'm far from discounting some other issue in the environment, btw, just looking for confirmation or otherwise that we've seen this fix the issue. | |
| 12:05:09 | gibi | mdbooth: you logs shows the original error I can confirm as placement logs ""GET /placemen//resource_providers?in_" | |
| 12:05:21 | mdbooth | gibi: Right | |
| 12:06:07 | gibi | and I do see the ProxyPass you mentioned, and that is the correct one | |
| 12:06:21 | gibi | are you sure that your env using the ProxyPass that is logged? | |
| 12:06:23 | mdbooth | gibi: There's a git pull; git show up the top of that output which suggests the working directory is at "7f16f6d4 Fix uwsgi config for trailing slashes" | |
| 12:07:28 | mdbooth | gibi: i.e. Can I confirm that the apache config actually contains that ProxyPass? | |
| 12:07:54 | gibi | gibi: or that apache2 was restarted / reload after the setting is applied | |
| 12:08:22 | gibi | hm, I see an apache2 reload later | |
| 12:08:54 | mdbooth | This is running in cloud-init on initial boot. *However* it is running from an image that already contains devstack bits, so it's entirely possible there's dirty state there. | |
| 12:09:28 | mdbooth | Let me try to confirm that. I hadn't considered that it might not actually be updating the apache config. | |
| 12:09:54 | gibi | systemctl reload apache2 | |
| 12:09:54 | gibi | To activate the new configuration, you need to run: | |
| 12:09:54 | gibi | hm, sorry I only see logs that suggest the reload like | |
| 12:10:20 | gibi | but I don't see the reload itself | |
| 12:11:12 | gibi | so If you can log into the system then try accessing placment if it fails then check the ProxyPass confing then reload apache and check that placement is accessible now | |
| 12:11:38 | gibi | (I did this manually with in my local devstack so I do belive the fix helps at least in devstack) | |
| 12:12:27 | mdbooth | Unfortunately this is running in a CI system I don't have access to :( I have to debug via modifying the job and re-executing 😬 | |
| 12:12:45 | mdbooth | I'll see what I can confirm. Thanks! | |
| 12:13:59 | gibi | mdbooth: no problem, let us know if this still fails for you | |
| 12:18:14 | mdbooth | Ok, I can confirm that the image the machine is created from already contains dirty devstack state | |
| 12:18:28 | mdbooth | Including the broken ProxyPass directives | |
| 12:19:08 | mdbooth | So unless we consider it a bug that devstack doesn't update that, I'm guessing this is my problem? | |
| 12:20:30 | gibi | mdbooth: so you have both wrong and bad ProxyPass lines in the config file? | |
| 12:20:41 | gibi | :D | |
| 12:20:41 | gibi | I mean wrong and good | |
| 12:21:11 | mdbooth | So *before* pulling and executing a patched devstack I already have bad config present | |
| 12:21:21 | mdbooth | And running the patched devstack doesn't fix it | |
| 12:21:58 | gibi | I do see multiple ProxyPass lines locally too, I guess those are from multiple unstack.sh / stack.sh runs in my case | |
| 12:22:04 | lyarwood | yeah that isn't a devstack bug | |
| 12:22:07 | lyarwood | ./clean.sh first | |
| 12:22:38 | gibi | hm, if clean.sh removes it then I agree it is not a bug (I'm lazy to always run clean.sh but I should) | |
| 12:23:03 | mdbooth | lyarwood: Or more likely summon the arcane wizards of CI and update the 'preinstalled' image! | |
| 12:23:22 | lyarwood | yeah or that if it's already baked into the image | |
| 12:23:24 | gibi | mdbooth: yeah, for CI I suggest to always start from a clean image :) | |
| 12:23:26 | lyarwood | that's odd tbh | |
| 12:24:05 | mdbooth | Apparently it saves a ton of time, although I haven't personally measured it. I'm about to measure it, though :) | |
| 12:24:44 | gibi | not too long ago dansmith added parallelism to devstack stack.sh run that helped a lot with runtime | |
| 12:27:51 | lyarwood | yeah that we use across most jobs now AFAIK | |
| 12:28:02 | mdbooth | lyarwood: Is there a flag? | |
| 12:29:16 | gibi | there should be | |
| 12:29:19 | gibi | but is on by default | |
| 12:29:27 | mdbooth | Ok, cool | |
| 12:29:57 | lyarwood | yeah it's there by default in master and xena | |
| 12:29:58 | gibi | btw, ./clean.sh deletes all the config from /etc/apache2/sites-enabled/ except glance :D | |
| 12:30:06 | mdbooth | Ah, looks like it's DEVSTACK_PARALLEL, and it's not in victoria | |
| 12:30:12 | mdbooth | Perhaps I'll update | |
| 12:30:34 | lyarwood | why are you deploying victoria btw? | |
| 12:31:02 | mdbooth | lyarwood: Because nobody changed it | |
| 12:31:18 | lyarwood | oh fun | |
| 12:31:34 | gibi | :) | |
| 12:32:22 | mdbooth | Honestly this CI system is awesome, I'm not complaining. It installs devstack in a VM in GCE and runs tests against it, and until we hit this it was rock solid. | |
| 12:33:40 | mdbooth | Ok, I'm going to run again from a clean image instead of the preinstalled image to see how long that takes. Then I'm going to do the same again but against Xena to measure again. | |
| 12:33:45 | mdbooth | Thanks for all the help! | |
| 12:34:19 | gibi | happy to help | |
| 12:36:25 | gmann | bauzas: other than neutron-grenade. nova-ceph-multistore is broken on stable (https://bugs.launchpad.net/devstack-plugin-ceph/+bug/1945358) fixes are ready to merger - https://review.opendev.org/q/topic:%22bug%252F1945358%22+(status:open%20OR%20status:merged) | |
| 12:36:37 | gmann | gibi: lyarwood checking comments on grenade fixes, | |
| 12:39:16 | gibi | gmann: in short we are questioning whether we need to turn on trunk testing on stable branches | |
| 12:41:15 | gmann | gibi: lyarwood ok, I will say we should as Tempest master is used to test the stable/ussuri -> master and if we do not enable then we skip the trunk test for no reason. it was just a miss in enabling the extension in stable branches. | |
| 12:41:39 | lyarwood | gm | |
| 12:41:41 | lyarwood | ops sorry | |
| 12:42:06 | lyarwood | gmann: Yeah I appreciate that but if wasn't tested until now it seems a little odd to add it in for non-master branches | |
| 12:42:29 | lyarwood | gmann: the missing coverage was mostly around live migration right? | |
| 12:42:37 | gmann | and we can see it is passing in nova grenade jobs in stable so we do not need anythings extra than grenade fix | |
| 12:43:04 | gmann | lyarwood: yeah, live migration trunk tests. those are only tests for trunk currently in tempest | |
| 12:43:06 | lyarwood | gmann: okay well if it's passing and the neutron folks are okay with helping with any fallout that might appear in the coming days then I guess we can go ahead | |
| 12:43:44 | gmann | lyarwood: yeah, if neutron team is not comfortable or confident then we can just keep it like that. | |
| 12:53:46 | gmann | lyarwood: whether we need it for Xena or not, I remember master testing patch passed only when I added xena fix as depends-on here https://review.opendev.org/c/openstack/nova/+/811118 | |
| 12:54:08 | gmann | I think i added due to zuul job inventory but let me debug that what exactly happing here | |
| 12:54:14 | lyarwood | okay if that's the case lets just merge both | |
| 13:08:52 | gmann | lyarwood: zuul inventory seems to take job definition from master only https://zuul.opendev.org/t/openstack/build/01f10e2cb162450e912551026dde8f85/log/zuul-info/inventory.yaml#294-304 | |
| 13:09:08 | gmann | let me remove the xena fix as depends-on and then we can get more clarity | |
| 13:09:45 | opendevreview | Ghanshyam proposed openstack/nova master: DNM testing grenade neutron-trunk fix https://review.opendev.org/c/openstack/nova/+/811118 | |
| 13:16:32 | lyarwood | gmann: kk | |
| 13:55:54 | bauzas | sorry folks, I had issues with my computer | |
| 13:56:08 | bauzas | gmann: thanks, unfortunately gtk | |
| 13:57:37 | bauzas | looks like it's limbo | |
| 13:58:31 | gmann | nova grenade master fix is in gate https://review.opendev.org/c/openstack/grenade/+/811117 | |
| 13:58:57 | gmann | for enabling testing on stable or not is different things and we can continue discussion. | |
| 14:01:40 | dansmith | bauzas: gibi looks like the nova-yoga-ptg etherpad has been emptied | |
| 14:01:51 | bauzas | ORLY ? | |
| 14:02:21 | gibi | shit, I see | |
| 14:02:52 | bauzas | dansmith: this was really done 3 mins before | |