Posted	Nick	Remark
#openstack-nova - 2021-05-21
15:14:34	openstackgerrit	Balazs Gibizer proposed openstack/nova master: Fix RequestLevelParams persistence handling in RequestSpec https://review.opendev.org/c/openstack/nova/+/791502
15:14:35	openstackgerrit	Balazs Gibizer proposed openstack/nova master: [func test] create pps resource on OVS agent RP https://review.opendev.org/c/openstack/nova/+/787205
15:14:35	openstackgerrit	Balazs Gibizer proposed openstack/nova master: [func test] move port creation to the NeutronFixture https://review.opendev.org/c/openstack/nova/+/787206
15:14:36	openstackgerrit	Balazs Gibizer proposed openstack/nova master: [func test] refactor assertPortMatchesAllocation https://review.opendev.org/c/openstack/nova/+/792458
15:18:23	openstackgerrit	Balazs Gibizer proposed openstack/nova master: Add same_subtree field to RequestLevelParams https://review.opendev.org/c/openstack/nova/+/791503
15:19:41	openstackgerrit	Balazs Gibizer proposed openstack/nova master: Bump min placement microversion to 1.36 https://review.opendev.org/c/openstack/nova/+/791504
15:22:09	openstackgerrit	Balazs Gibizer proposed openstack/nova master: Support same_subtree in allocation_canadidate query https://review.opendev.org/c/openstack/nova/+/791505
15:23:35	openstackgerrit	Balazs Gibizer proposed openstack/nova master: Support the new port resource_request format https://review.opendev.org/c/openstack/nova/+/787208
15:23:36	openstackgerrit	Balazs Gibizer proposed openstack/nova master: Transfer RequestLevelParams from ports to scheduling https://review.opendev.org/c/openstack/nova/+/791506
15:27:34	openstackgerrit	Merged openstack/python-novaclient master: Change minversion of tox to 3.18.0 https://review.opendev.org/c/openstack/python-novaclient/+/791969
15:27:35	openstackgerrit	Merged openstack/python-novaclient master: Change minversion of tox to 3.18.0 https://review.opendev.org/c/openstack/python-novaclient/+/791969
17:58:48	openstackgerrit	Merged openstack/nova stable/wallaby: libvirt: Remove dead error handling code https://review.opendev.org/c/openstack/nova/+/788724
17:58:48	openstackgerrit	Merged openstack/nova stable/wallaby: libvirt: Remove dead error handling code https://review.opendev.org/c/openstack/nova/+/788724
19:02:12	openstackgerrit	Dmitrii Shcherbakov proposed openstack/nova-specs master: Introduce Transport Nodes https://review.opendev.org/c/openstack/nova-specs/+/787458
19:02:12	openstackgerrit	Dmitrii Shcherbakov proposed openstack/nova-specs master: Introduce Transport Nodes https://review.opendev.org/c/openstack/nova-specs/+/787458
#openstack-nova - 2021-05-22
05:30:22	openstackgerrit	Merged openstack/nova stable/wallaby: Move instance power state check to _detach_with_retry https://review.opendev.org/c/openstack/nova/+/788725
05:30:22	openstackgerrit	Merged openstack/nova stable/wallaby: Move instance power state check to _detach_with_retry https://review.opendev.org/c/openstack/nova/+/788725
17:47:20	openstackgerrit	Merged openstack/nova stable/wallaby: Consolidate device detach error handling https://review.opendev.org/c/openstack/nova/+/788726
17:47:20	openstackgerrit	Merged openstack/nova stable/wallaby: Consolidate device detach error handling https://review.opendev.org/c/openstack/nova/+/788726
20:54:06	openstackgerrit	Merged openstack/nova stable/train: Make _rebase_with_qemu_img() generic https://review.opendev.org/c/openstack/nova/+/780782
20:54:06	openstackgerrit	Merged openstack/nova stable/train: Make _rebase_with_qemu_img() generic https://review.opendev.org/c/openstack/nova/+/780782
#openstack-nova - 2021-05-23
07:23:45	openstackgerrit	Wenping Song proposed openstack/placement master: Remove unvalid test https://review.opendev.org/c/openstack/placement/+/792734
11:15:03	openstackgerrit	Merged openstack/nova stable/train: Rebase qcow2 images when unshelving an instance https://review.opendev.org/c/openstack/nova/+/785627
14:52:23	openstackgerrit	Merged openstack/nova stable/train: Update image_base_image_ref during rebuild. https://review.opendev.org/c/openstack/nova/+/787945
#openstack-nova - 2021-05-24
08:01:54	lyarwood	\o morning
08:28:45	lyarwood	stephenfin: would you mind hitting these again if you're around today https://review.opendev.org/q/topic:%22bug%252F1928063%22+(status:open%20OR%20status:merged)
09:03:55	kevinz	lyarwood: stephenfin: morning! Could you help to review this when you convenient? https://review.opendev.org/c/openstack/nova/+/763928, It is about live migration support on Arm64
10:21:18	openstackgerrit	Sylvain Bauza proposed openstack/nova-specs master: Add generic mdevs to Nova https://review.opendev.org/c/openstack/nova-specs/+/792796
11:41:40	openstackgerrit	Stephen Finucane proposed openstack/nova master: tests: Move libvirt-specific fixtures https://review.opendev.org/c/openstack/nova/+/790969
11:41:41	openstackgerrit	Stephen Finucane proposed openstack/nova master: tests: Add os-brick fixture https://review.opendev.org/c/openstack/nova/+/790970
11:41:41	openstackgerrit	Stephen Finucane proposed openstack/nova master: tests: Rename 'ImageBackendFixture' to 'LibvirtImageBackendFixture' https://review.opendev.org/c/openstack/nova/+/792353
11:41:42	openstackgerrit	Stephen Finucane proposed openstack/nova master: Create a fixture around fake_notifier https://review.opendev.org/c/openstack/nova/+/758446
11:41:43	openstackgerrit	Stephen Finucane proposed openstack/nova master: Use NotificationFixture for legacy notifications too https://review.opendev.org/c/openstack/nova/+/758448
11:41:44	openstackgerrit	Stephen Finucane proposed openstack/nova master: Test the NotificationFixture https://review.opendev.org/c/openstack/nova/+/758450
11:41:44	openstackgerrit	Stephen Finucane proposed openstack/nova master: Move fake_notifier impl under NotificationFixture https://review.opendev.org/c/openstack/nova/+/758451
11:41:45	openstackgerrit	Stephen Finucane proposed openstack/nova master: rpc: Mark attributes as private https://review.opendev.org/c/openstack/nova/+/792803
12:34:43	openstackgerrit	Merged openstack/nova master: tests: Move libvirt-specific fixtures https://review.opendev.org/c/openstack/nova/+/790969
12:35:26	openstackgerrit	Merged openstack/nova master: tests: Add os-brick fixture https://review.opendev.org/c/openstack/nova/+/790970
12:36:04	sean-k-mooney	stephenfin: are you rewriting all the test fixutures again :P
12:36:35	openstackgerrit	Merged openstack/nova master: tests: Rename 'ImageBackendFixture' to 'LibvirtImageBackendFixture' https://review.opendev.org/c/openstack/nova/+/792353
12:36:58	lyarwood	I didn't like backporting func tests anyway
12:37:08	lyarwood	/s
12:37:13	sean-k-mooney	:)
12:37:36	sean-k-mooney	we just need to get all our customer to deploy master at all times
12:37:39	sean-k-mooney	problem solved
12:39:44	lyarwood	* lyarwood nods
13:48:56	lyarwood	Random, as an admin I can't seem to list server events for a user defined instance?
13:49:03	lyarwood	$ openstack server event list srvtree-server1-rh5da6maeidh
13:49:03	lyarwood	No server with a name or ID of 'srvtree-server1-rh5da6maeidh' exists.
13:51:26	stephenfin	@lyarwood If you're in a different project then that won't work
13:51:36	lyarwood	even as the admin?
13:51:45	lyarwood	okay weird
13:51:52	lyarwood	I thought this worked previously
13:51:56	stephenfin	because the name-based search is done against the instances in the project
13:52:19	stephenfin	I doubt it. A couple of command have an '--all-projects' option just for this
13:52:51	stephenfin	* stephenfin double checks to make sure he didn't change anything recently, just in case
13:53:20	lyarwood	ah right I was likely using the UUID in the past
13:53:57	stephenfin	Nope, I added missing options but it's otherwise unchanged, save for some docs, since it was added in 2017
13:54:07	stephenfin	I'd say so
13:56:17	lyarwood	Yeah sorry, I've used UUIDs everywhere in the request-id presentation and that just threw me
14:35:49	lyarwood	test_volume_backed_live_migration keeps failing on master at the moment btw
14:49:44	lyarwood	Ah I see why, slow nodes and pre_live_migration is timing out
15:35:57	melwitt	stephenfin: were you planning to work on trying to remove eventlet or would you mind if I took a try at it? the ptg discussion on that occurred earlier than I had come online that day
15:36:57	stephenfin	melwitt: go for it. I am hoping to do something on it but it's a potential minefield that would benefit from many eyes
15:41:06	melwitt	stephenfin: cool, thanks. I've gone quite down the rabbit hole related to eventlet on some downstream bugs, so I've some ideas now (fortunately or unfortunately)
16:06:51	dansmith	melwitt: this is remove eventlet from api right?
16:07:44	melwitt	dansmith: and potentially everything else too
16:09:12	dansmith	melwitt: nothing else is threadsafe so I have a hard time imagining that being a thing
16:10:31	melwitt	tl;dr is there's a bad interaction and failure mode between gevent/eventlet and pymysql and mysqlconnector wherein if a green thread is killed before a connection is cleaned up, it leaves it in an inconsistent state and the next attempt to use a connection blows up
16:11:47	melwitt	I've spoken at length with zzzeek about this and my understanding is this can't be worked around or handled and that replacing our usage of eventlet with native threading or similar is the only way to avoid it
16:12:33	dansmith	is there a pointer to something to read about it?
16:12:51	melwitt	yeah, sec
16:15:01	dansmith	api and conductor going to native threads are doable I think without too much crazy,
16:15:23	dansmith	but I think compute will be a nightmare, but it also doesn't use the DB driver at all, so it should be immune
16:15:30	melwitt	dansmith: this comment contains the relevant references https://bugzilla.redhat.com/show_bug.cgi?id=1927994#c45 the rest of that bug has a lot of comments, most of which are private because I'm not sure they help bring any clarity. it's been a long discussion on there
16:16:01	melwitt	oh, the entire bug looks to be private /facepalm
16:16:14	melwitt	https://github.com/PyMySQL/PyMySQL/issues/234
16:16:19	melwitt	https://github.com/sqlalchemy/sqlalchemy/issues/3258
16:16:26	melwitt	https://github.com/PyMySQL/PyMySQL/issues/260
16:16:33	melwitt	those are the references ^
16:17:01	dansmith	thanks, I can read it at least
16:17:16	dansmith	so the assertion is that all openstack projects will have to undertake this conversion?
16:17:28	dansmith	that's a pretty big deal
16:18:11	melwitt	for whatever reason nova is the only one that seems affected by this, this error has not been found in any other service's logs in the same deployments that see it in nova often
16:19:16	dansmith	and only nova-api?
16:19:17	melwitt	and I think most of the appearance of the error is from back when we had the eventlet-based wsgi server for nova-api. my guess is that since moving away from that, our usage is much reduced and reduces the chances of hitting this
16:19:45	melwitt	no I have seen traces in nova-scheduler and nova-conductor as well
16:20:23	dansmith	okay, because nova-api already runs with a combination of native and green threads, so if it was just api, then that could be why,
16:20:45	dansmith	but if scheduler and conductor see it as well, I'm not sure why no other service would be affected
16:22:53	melwitt	yeah, I'm guessing it's because it becomes very rare when the only things using eventlet are the periodic tasks, timers/retries, and some bits of scatter/gather. the error shows up under high load and coexists with other various connection errors to the database
16:23:21	melwitt	well, maybe "very rare" is not a good way to put it, "more rare"
16:25:43	melwitt	I have searched for some way that nova does something different than any other service wrt to database access and found nothing
16:26:15	melwitt	the only lead we have so far is that nova uses eventlet more than any other services do (afaik so far)
16:26:17	sean-k-mooney	melwitt: i assume we have not had any downstream sqlalcamy updtes in rhel 7
16:27:21	sean-k-mooney	does this happen in modren openstack that was for 13 so queens
16:27:48	melwitt	sean-k-mooney: what do you mean, like version changes? I don't think so but zzzeek would have covered that
16:28:12	sean-k-mooney	yep or perhaps a backport that could have caused it
16:28:59	melwitt	sean-k-mooney: to your other question, I could find no bug reports for this for newer than 13
16:29:39	melwitt	that could either mean it went away or that it happens rarely enough that no one has bothered reporting it. not sure what to think
16:30:09	dansmith	melwitt: but nothing changed about conductor and scheduler that would account for them not showing the issue in later versions

Earlier   Later