Posted	Nick	Remark
#openstack-nova - 2021-09-29
01:56:37	opendevreview	Federico Ressi proposed openstack/nova master: Debug Nova APIs call failures https://review.opendev.org/c/openstack/nova/+/806683
04:10:29	opendevreview	Ghanshyam proposed openstack/nova stable/victoria: DNM: Testing nova-grenade-multinode with neutron-trunk https://review.opendev.org/c/openstack/nova/+/811540
06:43:02	bauzas	good morning Nova
08:16:00	deke	Hi
08:16:48	deke	Has there been any discussion about implementing a MAAS driver for nova to enable ironic-like functionality for users who have deployed openstack with juju on MAAS?
08:36:13	bauzas	deke: I haven't heard anything about this
08:36:30	bauzas	deke: tbh, it would be a laaaarge discussion, right?
08:36:38	deke	yea it would
08:36:43	deke	it was just a thought I had
08:37:00	deke	if we are already deploying on top of a baremetal provisioning service
08:37:15	deke	then why deploy another baremetal provisioning service
08:39:35	bauzas	we are not saying "no" for a new virt driver
08:40:31	bauzas	but for having a new virt driver upstream, that means we need to discuss how to use it and how we could verify it by the CI
08:40:55	bauzas	that means large discussions during a lot of PTGs + making sure we at least have a third-party CI
08:42:49	lyarwood	dansmith: https://review.opendev.org/c/openstack/grenade/+/811117 - looks like you're the only remaining active core on grenade, can you ack this when you get online to unblock Nova's gate?
09:03:28	gibi	gmann: thanks for the patches!
09:05:12	bauzas	gmann: what gibi said, thanks for having worked on them
09:06:08	bauzas	so the xena and master changes for fixing the placement API endpoints are now merged, we only have the grenade issue left, right?
09:07:08	lyarwood	Yup I believe so
09:07:12	lyarwood	and I think we only need it in master
09:07:15	gibi	yupp thats my view too
09:07:17	lyarwood	I don't know why we've backported it tbh
09:07:53	gibi	lyarwood: you mean why we are enabling trunk testing on wallaby and back?
09:08:26	lyarwood	on stable/xena and backwards, I thought grenade used the current branch of itself and the previous branch for everything else for the initial deploy
09:08:47	lyarwood	so master grenade deploys a stable/xena env and then upgrades that to master
09:10:08	bauzas	gibi: I stupidely forgot to +1 the RC2 patches yesterday before leaving
09:10:14	bauzas	:facepalm:
09:10:29	bauzas	now we need elodilles_pto but as his nick says, he's on PTO :)
09:12:17	gibi	lyarwood: if we do this https://review.opendev.org/c/openstack/devstack/+/811518/2/lib/tempest then we need to do this as well https://review.opendev.org/c/openstack/grenade/+/811542/1/.zuul.yaml on stable. But I agre that we don't have to enable trunk on stable, skip is OK to me too.
09:12:59	lyarwood	yeah I agree with the comment on the stable/wallaby change that we shouldn't be adding tests to stable this late on
09:13:01	gibi	bauzas: I think other release cores can approve the RC2 we don't necessary need to wait for elodilles_pto.
09:13:12	gibi	anyhow he is back tomorrow so no worries
09:14:13	gibi	lyarwood: ack, we can discuss this with gmann when he is up
09:20:45	lyarwood	kk
10:56:11	opendevreview	Lee Yarwood proposed openstack/nova master: nova-manage: Ensure mountpoint is passed when updating attachment https://review.opendev.org/c/openstack/nova/+/811713
11:17:24	opendevreview	Lee Yarwood proposed openstack/nova master: nova-manage: Always get BDMs using get_by_volume_and_instance https://review.opendev.org/c/openstack/nova/+/811716
11:26:10	lyarwood	https://review.opendev.org/c/openstack/nova/+/811118 - cool the check queue is passing with the grenade fix
11:27:28	gibi	\o/
12:00:50	mdbooth	👋 Are we confident that https://review.opendev.org/c/openstack/devstack/+/811399 fixed the devstack issue on Ubuntu? My devstack-using CI job still seems to be failing:
12:00:51	mdbooth	https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api-provider-openstack/1009/pull-cluster-api-provider-openstack-e2e-test/1443169418262614016/artifacts/logs/cloud-final.log
12:02:32	mdbooth	I can see the following in the devstack output: "ProxyPass "/placement" "unix:/var/run/uwsgi/placement-api.socket|uwsgi://uwsgi-uds-placement-api" retry=0", which looks like it contains the fix. However I'm still seeing the borked GETs to placement
12:04:33	mdbooth	I'm far from discounting some other issue in the environment, btw, just looking for confirmation or otherwise that we've seen this fix the issue.
12:05:09	gibi	mdbooth: you logs shows the original error I can confirm as placement logs ""GET /placemen//resource_providers?in_"
12:05:21	mdbooth	gibi: Right
12:06:07	gibi	and I do see the ProxyPass you mentioned, and that is the correct one
12:06:21	gibi	are you sure that your env using the ProxyPass that is logged?
12:06:23	mdbooth	gibi: There's a git pull; git show up the top of that output which suggests the working directory is at "7f16f6d4 Fix uwsgi config for trailing slashes"
12:07:28	mdbooth	gibi: i.e. Can I confirm that the apache config actually contains that ProxyPass?
12:07:54	gibi	gibi: or that apache2 was restarted / reload after the setting is applied
12:08:22	gibi	hm, I see an apache2 reload later
12:08:54	mdbooth	This is running in cloud-init on initial boot. *However* it is running from an image that already contains devstack bits, so it's entirely possible there's dirty state there.
12:09:28	mdbooth	Let me try to confirm that. I hadn't considered that it might not actually be updating the apache config.
12:09:54	gibi	hm, sorry I only see logs that suggest the reload like
12:09:54	gibi	To activate the new configuration, you need to run:
12:09:54	gibi	systemctl reload apache2
12:10:20	gibi	but I don't see the reload itself
12:11:12	gibi	so If you can log into the system then try accessing placment if it fails then check the ProxyPass confing then reload apache and check that placement is accessible now
12:11:38	gibi	(I did this manually with in my local devstack so I do belive the fix helps at least in devstack)
12:12:27	mdbooth	Unfortunately this is running in a CI system I don't have access to :( I have to debug via modifying the job and re-executing 😬
12:12:45	mdbooth	I'll see what I can confirm. Thanks!
12:13:59	gibi	mdbooth: no problem, let us know if this still fails for you
12:18:14	mdbooth	Ok, I can confirm that the image the machine is created from already contains dirty devstack state
12:18:28	mdbooth	Including the broken ProxyPass directives
12:19:08	mdbooth	So unless we consider it a bug that devstack doesn't update that, I'm guessing this is my problem?
12:20:30	gibi	mdbooth: so you have both wrong and bad ProxyPass lines in the config file?
12:20:41	gibi	I mean wrong and good
12:20:41	gibi	:D
12:21:11	mdbooth	So *before* pulling and executing a patched devstack I already have bad config present
12:21:21	mdbooth	And running the patched devstack doesn't fix it
12:21:58	gibi	I do see multiple ProxyPass lines locally too, I guess those are from multiple unstack.sh / stack.sh runs in my case
12:22:04	lyarwood	yeah that isn't a devstack bug
12:22:07	lyarwood	./clean.sh first
12:22:38	gibi	hm, if clean.sh removes it then I agree it is not a bug (I'm lazy to always run clean.sh but I should)
12:23:03	mdbooth	lyarwood: Or more likely summon the arcane wizards of CI and update the 'preinstalled' image!
12:23:22	lyarwood	yeah or that if it's already baked into the image
12:23:24	gibi	mdbooth: yeah, for CI I suggest to always start from a clean image :)
12:23:26	lyarwood	that's odd tbh
12:24:05	mdbooth	Apparently it saves a ton of time, although I haven't personally measured it. I'm about to measure it, though :)
12:24:44	gibi	not too long ago dansmith added parallelism to devstack stack.sh run that helped a lot with runtime
12:27:51	lyarwood	yeah that we use across most jobs now AFAIK
12:28:02	mdbooth	lyarwood: Is there a flag?
12:29:16	gibi	there should be
12:29:19	gibi	but is on by default
12:29:27	mdbooth	Ok, cool
12:29:57	lyarwood	yeah it's there by default in master and xena
12:29:58	gibi	btw, ./clean.sh deletes all the config from /etc/apache2/sites-enabled/ except glance :D
12:30:06	mdbooth	Ah, looks like it's DEVSTACK_PARALLEL, and it's not in victoria
12:30:12	mdbooth	Perhaps I'll update
12:30:34	lyarwood	why are you deploying victoria btw?
12:31:02	mdbooth	lyarwood: Because nobody changed it
12:31:18	lyarwood	oh fun
12:31:34	gibi	:)
12:32:22	mdbooth	Honestly this CI system is awesome, I'm not complaining. It installs devstack in a VM in GCE and runs tests against it, and until we hit this it was rock solid.
12:33:40	mdbooth	Ok, I'm going to run again from a clean image instead of the preinstalled image to see how long that takes. Then I'm going to do the same again but against Xena to measure again.
12:33:45	mdbooth	Thanks for all the help!
12:34:19	gibi	happy to help
12:36:25	gmann	bauzas: other than neutron-grenade. nova-ceph-multistore is broken on stable (https://bugs.launchpad.net/devstack-plugin-ceph/+bug/1945358) fixes are ready to merger - https://review.opendev.org/q/topic:%22bug%252F1945358%22+(status:open%20OR%20status:merged)
12:36:37	gmann	gibi: lyarwood checking comments on grenade fixes,
12:39:16	gibi	gmann: in short we are questioning whether we need to turn on trunk testing on stable branches
12:41:15	gmann	gibi: lyarwood ok, I will say we should as Tempest master is used to test the stable/ussuri -> master and if we do not enable then we skip the trunk test for no reason. it was just a miss in enabling the extension in stable branches.

Earlier   Later