Posted	Nick	Remark
#openstack-nova - 2021-01-21
14:15:14	lyarwood	stephenfin: ack sorry didn't mean to spam
14:15:24	stephenfin	nope, all good
14:40:15	openstackgerrit	Dan Smith proposed openstack/nova master: Try harder to exclude encrypted volumes https://review.opendev.org/c/openstack/nova/+/771822
14:40:40	dansmith	lyarwood: we're also suddenly running cinder encrypted tests in that multistore job despite the blacklist regex, which are coming from the cinder tempest plugin
14:40:46	dansmith	lyarwood: ^ this tries to hit them too
14:43:04	sean-k-mooney	hehe i like the title
14:43:25	dansmith	I dunno why that has started failing all the sudden,
14:43:32	dansmith	like if a name changed and we started running those or something
14:43:46	dansmith	but they end up putting an image int he wrong glance repo, which doesn't work
14:44:46	dansmith	oh, heh
14:44:57	sean-k-mooney	https://github.com/openstack/cinder-tempest-plugin/commit/5ab3fef76283ca91b92fa130ba2685886bd8af68
14:44:58	dansmith	looks like there's already something in gate
14:45:00	sean-k-mooney	maybe that
14:45:10	dansmith	sean-k-mooney: yuuup
14:46:08	sean-k-mooney	so ya just new test added
14:47:38	dansmith	lyarwood: sorry, just saw yours from 1am.. I had half figured this out last night but ran out of time, thanks for grabbing that
14:47:52	dansmith	that makes three distinct and legit non-nova, non-glance issues with that job in a week :)
14:53:12	sean-k-mooney	well at least there is now testing for this so its still progress
14:53:26	sean-k-mooney	even if there are teething issues
14:59:12	dansmith	lyarwood: I assume that snuck in because we run the whole tempest-cinder-plugin on that job, but they don't gate on it.. so my first thought was just to not load their tempest plugin in there to prevent this from happening in the future.. thoughts on that?
14:59:44	dansmith	sean-k-mooney: yes, especially good when they're actual problems and not just flaky or unreliable tests
15:03:19	lyarwood	dansmith: morning, sorry was afk
15:03:51	lyarwood	dansmith: yeah correct, I was going to see if we could add this job into their gate tbh
15:03:56	dansmith	lyarwood: np, it took a night of sleep to realize that test was from a tempest plugin, thanks for nailing that whilst I slept
15:04:02	dansmith	lyarwood: that also works
15:04:50	lyarwood	so what's the deal with these project specific tempest plugins
15:05:01	lyarwood	from what I've seen most of this should live in core tempest anyway
15:05:03	dansmith	I dunno, asking for trouble :)
15:05:21	openstackgerrit	Balazs Gibizer proposed openstack/nova master: Replace blind retry with libvirt event waiting in detach https://review.opendev.org/c/openstack/nova/+/770246
15:05:29	lyarwood	hehe
15:05:52	lyarwood	I just don't get the need for loads of different test repos all doing the same time tbh, I'll go away and read up
15:05:54	dansmith	without co-gating it's just an attack vector
15:12:56	lyarwood	dansmith: multistore attempts to cover some edge usecases right?
15:21:06	dansmith	lyarwood: the whole arrangement in devstack is edgy yeah
15:21:49	lyarwood	dansmith: cool, so looking at the actual test that borked here this might be something we can fix on their side
15:22:16	lyarwood	dansmith: for whatever reason the test is creating a unique image for the instance to use
15:22:41	lyarwood	dansmith: when it really isn't required as the test is about cloning an encrypted volume attached to the instance
15:23:06	lyarwood	I'll submit a few changes to the cinder plugin before adding our job to their gate somehow
15:23:06	dansmith	yeah, and it creates it in the wrong glance store, which is the problem
15:23:18	lyarwood	we might be able to drop the blacklist around encrypted volumes after all
15:23:26	lyarwood	skiplist*
15:23:36	dansmith	brownlist
15:23:40	dansmith	like, turds
15:25:31	lyarwood	^_^
15:26:02	spatel	noonedeadpunk: In your designate example where is pool_id? http://paste.openstack.org/show/801787/
15:26:55	noonedeadpunk	o_O
15:27:01	spatel	i had issue that issue and when i put pool id then it works
15:27:08	spatel	what is the workaround for that?
15:27:47	spatel	does OSA generate random pool id ? (because default isn't valid)
15:28:43	spatel	sorry folks wrong channel
15:34:57	sean-k-mooney	i had issue with the pool id in kolla ansible too
15:35:37	sean-k-mooney	for me i had to both mess with the config and the db to eventulaly get it to work
15:36:00	sean-k-mooney	but i remmeber id did not jsut work for me when it was deployed
15:36:38	sean-k-mooney	i added
15:36:40	sean-k-mooney	[service:central]
15:36:43	sean-k-mooney	default_pool_id = 6a67bd3a8e654ddeb3675b8534899f16
15:36:45	sean-k-mooney	workers = 5
15:36:54	sean-k-mooney	and then i think i needed to edit the db to make it match
15:37:03	sean-k-mooney	it was strange
15:46:04	openstackgerrit	Lee Yarwood proposed openstack/nova master: zuul: Allow encryption tests in nova-ceph-multistore again https://review.opendev.org/c/openstack/nova/+/771837
15:46:09	lyarwood	dansmith: ^ hopefully this work
15:46:33	lyarwood	that links through to a fix and another change adding the job to gate of the tempest plugin
15:47:18	lyarwood	ah I think I also need to fix some core tempest tests
15:47:21	lyarwood	my bad
15:47:41	dansmith	yeah, there are core tests too, and that's why last night I was like "wtf, how is this just now breaking when nothing has changed here"
15:48:53	lyarwood	ack kk, I'll let this fail and then fix them up, it's likely the same bogus image creation copy pasted between tempest and the plugin tbh
15:49:21	dansmith	wouldn't be surprised, yeah
15:49:28	dansmith	which is further support for "why is this separate?"
15:49:44	lyarwood	yup
15:50:48	gibi	nova weekly meeting starts in 10 minutes in #openstack-meeting-3
15:50:55	bauzas	gibi: given the curfew we have at 6pm, I'll stay in the meeting until 5.30pm our time
15:54:52	gibi	bauzas: O
15:54:53	gibi	K
16:06:39	dansmith	hey sean-k-mooney, you like devstacking.. would you run this locally and compare before/after numbers? https://review.opendev.org/c/openstack/devstack/+/771505
16:07:57	sean-k-mooney	" know this is going to be controversial, but hear me out." L)
16:08:03	sean-k-mooney	* :)
16:08:08	sean-k-mooney	sure i can try that
16:08:20	dansmith	I actually maybe should remove that,
16:08:30	dansmith	because so far it has been super stable for me, and easy to debug things
16:08:45	dansmith	initial attempt interleaved all the parallel logging, which I knew would be hard to swallow for people
16:09:01	sean-k-mooney	we still have some native calls to osc by the way that dont use osc wrap
16:09:05	dansmith	but now it looks very clean, just potentially different orderings
16:09:14	sean-k-mooney	if we fixed up those it would also help
16:09:19	dansmith	sean-k-mooney: oscwrap is just for time accounting
16:09:23	dansmith	it doesn't make it any faster
16:09:38	dansmith	and it sets up an alias, so everything should be using it, AFAI
16:09:39	sean-k-mooney	waith i tought it used osc in itneractive mode via a socket
16:09:44	dansmith	no
16:09:50	sean-k-mooney	did we not merge danpbs prototype of that
16:09:53	sean-k-mooney	oh ok
16:10:32	dansmith	I tried doing that myself this week, but getting the return code is going to be non-trivial and we need that in a *lot* of places in devstack
16:10:55	sean-k-mooney	well ya ill pull that into one of my vms and let it run
16:11:01	dansmith	we do lots of single-threaded things in devstack, waiting on osc is just the dumbest one :)
16:11:12	dansmith	sean-k-mooney: ooh, thanks I'd definitely appreciate run in a vm
16:11:19	dansmith	I'm on baremetal
16:12:00	sean-k-mooney	well i ment run it in one of my openstack vms so that will be similar to the upstream ci runs
16:12:08	dansmith	that's cool
16:12:43	dansmith	performance in the actual nodepool workers varies so much from minute to minute it's hard to prove with linkable numbers that it's better
16:12:48	dansmith	so more anecdotal evidence would be good
16:14:03	dansmith	I'm also on spinning rust currently, and I feel like I could be getting better parallel performance by overlapping the db syncs if I wasn't

Earlier   Later