Posted	Nick	Remark
#openstack-nova - 2019-04-10
19:57:03	melwitt	efried: oh sorry, I'm working on refreshing patches for an unrelated bug
19:57:30	melwitt	and ran into this problem of the RP will never be created again while the service is running
19:57:50	melwitt	because the func tests I'm working with are now failing because of it
19:58:27	efried	I mean, it makes sense, because we have the provider tree cache, and no reason to think it has expired, so we never go looking for the provider to notice it's gone so we never create it.
19:58:37	efried	This is what SIGHUP was supposed to be for.
19:59:06	melwitt	ok, so it's a known thing that if an operator deletes and creates a service with the same hostname, they're supposed to SIGHUP now
19:59:07	efried	And this would be a doc issue: "If you delete and recreate the service, SIGHUP (or restart ffs) your n-cpu process"
19:59:52	efried	Well, I'm saying if it's not documented as such, that would be my first proposed solution. I'm not positive it's going to fix the problem (even when SIGHUP works, which it still doesn't afaik) but it'd be the first thing I'd try.
19:59:58	mriedem	fwiw we already have a note about that in the API reference https://developer.openstack.org/api-ref/compute/?expanded=delete-compute-service-detail#delete-compute-service
20:00:02	melwitt	ok. I can simulate that in the test, but wanted to mention it in case it wasn't desired/expected
20:00:20	melwitt	thanks
20:00:53	efried	cool, the doc actually says to stop the thing, which makes freakin sense to me.
20:01:11	efried	Maybe I'm being too simplistic in my thinking
20:01:23	efried	but how does an operator expect "delete a thing but keep it running" to ever work?
20:01:36	melwitt	no, it's fine, I just saw a change and didn't have the context about it
20:02:14	melwitt	well, the delete it and create it again, so they expect it to be running after they create it, in this scenario
20:02:14	efried	sorry, tbc I'm not griping at you; I guess I'm kind of wtf-ing that there's a bug about this.
20:02:29	efried	Perhaps we should have delete kill the service
20:02:39	efried	I think that may be what cdent was suggesting in the above patch
20:03:05	melwitt	I can't remember why operators have done that, something to do with an upgrade. cfriesen ran into it before I think
20:03:36	efried	...or make the delete operation fail if the service is running, to force the operator to do it.
20:03:42	melwitt	(delete service, create service, for same hostname)
20:04:03	efried	Oh, I'm *sure* there's a good reason to do that ^
20:04:05	efried	:)
20:04:34	efried	but expecting the running process to survive intact and work properly through it...
20:05:09	efried	perhaps it used to work by pure luck and now they're facing RBB.
20:06:07	cfriesen	I think we probably hit it during upgrade...our upgrade mechanism was to delete the standby controller, then reinstall it with the new load (and the same hostname as before), then migrate the DB over and cut over to the new controller.
20:07:11	melwitt	cfriesen: that sounds like what I recall
20:07:48	cfriesen	hmm..that wouldn't delete the nova-compute service though.
20:08:36	melwitt	cfriesen: yeah this was deleting the nova-compute service for a host, then recreating it. deleting the nova-compute service record would kill the compute node record too
20:08:53	melwitt	and you're saying your upgrade routine would not do that
20:10:43	cfriesen	I don't think so...trying to find the context for https://bugs.launchpad.net/nova/+bug/1764556
20:10:45	openstack	Launchpad bug 1764556 in OpenStack Compute (nova) ""nova list" fails with exception.ServiceNotFound if service is deleted and has no UUID" [Medium,In progress] - Assigned to melanie witt (melwitt)
20:11:25	melwitt	cfriesen: it's ok, don't spend time on it. we were just talking about the whole service delete and create thing. behavior changed semi recently
20:11:42	melwitt	and I thought I remembered you might have done something like that in the past
20:11:52	cfriesen	okay, so it looks like for that bug we *did* delete a compute node while on Newton, then migrated to Pike, then created a new node with the same name. May have been an artificial test though, not actually required for upgrade.
20:12:06	cfriesen	upgraded to Pike, rather
20:12:12	melwitt	gotcha
20:16:13	mriedem	the delete compute service while it's still running was never really thought out i don't think
20:16:23	mriedem	we've only semi recently tried to shore up a lot of that nonsense
20:16:47	mriedem	it's only been noticed as more of a problem since we started building onto it with things like host mappings and resource providers
20:17:33	mriedem	i.e. this is semi recent https://github.com/openstack/nova/blob/d42a007425d9adb691134137e1e0b7dda356df62/nova/api/openstack/compute/services.py#L247
20:17:41	mriedem	and this https://github.com/openstack/nova/blob/d42a007425d9adb691134137e1e0b7dda356df62/nova/api/openstack/compute/services.py#L270
20:17:46	mriedem	and this https://github.com/openstack/nova/blob/d42a007425d9adb691134137e1e0b7dda356df62/nova/api/openstack/compute/services.py#L275
20:18:30	melwitt	ah yeah, I remember that first one
20:23:54	openstackgerrit	Merged openstack/nova stable/stein: doc: Capitalize keystone domain name https://review.openstack.org/650600
20:24:00	openstackgerrit	Merged openstack/nova stable/stein: Add functional regression test for bug 1669054 https://review.openstack.org/649319
20:24:01	openstack	bug 1669054 in OpenStack Compute (nova) stein "RequestSpec.ignore_hosts from resize is reused in subsequent evacuate" [Medium,In progress] https://launchpad.net/bugs/1669054 - Assigned to Matt Riedemann (mriedem)
20:24:08	openstackgerrit	Merged openstack/nova stable/stein: doc: Fix openstack CLI command https://review.openstack.org/648412
20:28:06	openstackgerrit	melanie witt proposed openstack/nova master: Add functional recreate test for bug 1764556 https://review.openstack.org/562041
20:28:07	openstack	bug 1764556 in OpenStack Compute (nova) ""nova list" fails with exception.ServiceNotFound if service is deleted and has no UUID" [Medium,In progress] https://launchpad.net/bugs/1764556 - Assigned to melanie witt (melwitt)
20:28:07	openstackgerrit	melanie witt proposed openstack/nova master: Add functional regression test for bug 1778305 https://review.openstack.org/582407
20:28:08	openstackgerrit	melanie witt proposed openstack/nova master: Don't generate service UUID for deleted services https://review.openstack.org/582408
20:28:09	openstack	bug 1778305 in OpenStack Compute (nova) "Nova may erronously look up service version of a deleted service, when hostname have been reused" [Medium,In progress] https://launchpad.net/bugs/1778305 - Assigned to melanie witt (melwitt)
20:31:46	openstackgerrit	Chris Friesen proposed openstack/nova stable/rocky: Add missing libvirt exception during device detach https://review.openstack.org/651637
20:34:12	openstackgerrit	Chris Friesen proposed openstack/nova stable/queens: Add missing libvirt exception during device detach https://review.openstack.org/651639
20:36:46	openstackgerrit	Chris Friesen proposed openstack/nova stable/pike: Add missing libvirt exception during device detach https://review.openstack.org/651642
20:54:17	openstackgerrit	Matt Riedemann proposed openstack/nova master: Use InstanceList.get_count_by_hosts when deleting a compute service https://review.openstack.org/651647
21:12:57	openstackgerrit	Matt Riedemann proposed openstack/nova master: Add archive_deleted_rows wrinkle to cross-cell functional test https://review.openstack.org/651650
21:15:09	openstackgerrit	Matt Riedemann proposed openstack/nova master: Add archive_deleted_rows wrinkle to cross-cell functional test https://review.openstack.org/651650
21:33:08	openstackgerrit	Matt Riedemann proposed openstack/nova-specs master: FUP for I68498afd481f7291a6102928d7999b4be49ded7a https://review.openstack.org/651653
21:47:09	openstackgerrit	Merged openstack/nova stable/stein: Override the 'get' method in DriverBlockDevice class https://review.openstack.org/647646
21:47:14	openstackgerrit	Merged openstack/nova stable/stein: Don't warn on network-vif-unplugged event during live migration https://review.openstack.org/650060
21:55:55	openstackgerrit	Merged openstack/nova stable/stein: libvirt: disconnect volume when encryption fails https://review.openstack.org/650931
22:01:33	openstackgerrit	Merged openstack/nova stable/stein: Move create of ComputeAPI object in websocketproxy https://review.openstack.org/649374
22:35:13	openstackgerrit	Colleen Murphy proposed openstack/nova stable/rocky: Move create of ComputeAPI object in websocketproxy https://review.openstack.org/649375
23:51:49	openstackgerrit	Artom Lifshitz proposed openstack/nova master: Revert "Wait for network-vif-plugged on resize revert" https://review.openstack.org/639396
23:51:49	openstackgerrit	Artom Lifshitz proposed openstack/nova master: [WIP] Revert resize: wait for external events in compute manager https://review.openstack.org/644881
#openstack-nova - 2019-04-11
00:34:03	openstackgerrit	Dakshina Ilangovan proposed openstack/nova-specs master: Resource Management Daemon - Base Enablement https://review.openstack.org/651130
01:39:41	openstackgerrit	ya.wang proposed openstack/nova-specs master: Expose auto converge and post copy https://review.openstack.org/651681
02:42:17	gmann	sean-k-mooney: artom : we are good on 648123 from microversion point of view. It is not breaking any API contract. Commented inline. https://review.openstack.org/#/c/648123/4
02:42:43	gmann	tempest schema test is passing which make sure the API contract verification
03:11:37	openstackgerrit	Boxiang Zhu proposed openstack/nova-specs master: Add host and hypervisor_hostname flag to create server https://review.openstack.org/645458
03:54:37	openstackgerrit	Merged openstack/nova master: Add test coverage for nova.privsep.libvirt. https://review.openstack.org/648616
04:06:22	openstackgerrit	Merged openstack/nova master: Add test coverage for nova.privsep.qemu. https://review.openstack.org/649191
04:16:07	openstackgerrit	Merged openstack/nova stable/stein: Do not persist RequestSpec.ignore_hosts https://review.openstack.org/649320
09:02:52	openstackgerrit	Theodoros Tsioutsias proposed openstack/nova-specs master: Add PENDING vm state https://review.openstack.org/648687
10:20:00	openstackgerrit	Chris Dent proposed openstack/nova master: Use update_provider_tree in vmware virt driver https://review.openstack.org/651615
10:29:42	openstackgerrit	Chris Dent proposed openstack/nova master: Delete the placement code https://review.openstack.org/618215
10:39:11	openstackgerrit	Merged openstack/nova-specs master: Spec: Use in_tree getting allocation candidates https://review.openstack.org/646029
11:46:38	openstackgerrit	Stephen Finucane proposed openstack/nova master: trivial: Remove dead nova.db functions https://review.openstack.org/649570
11:48:02	stephenfin	gibi: Wanna send this cleanup patch on its way? https://review.openstack.org/#/c/650018/
11:58:18	gibi	stephenfin: done
11:58:24	stephenfin	ta
12:03:01	artom	gmann, cheers! (if you're still awake/around)
12:39:03	cdent	aspiers: have you seen http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004779.html ?
12:39:41	aspiers	cdent: yes, it's on my TODO list :)
12:39:47	aspiers	still jetlagged from SUSECON
12:39:54	cdent	roger, thanks
12:40:15	aspiers	I saw that efried felt OK with the status quo though
12:40:37	openstackgerrit	Tetsuro Nakamura proposed openstack/nova master: Add in_tree field to RequestGroup object https://review.openstack.org/649534
12:40:37	aspiers	I'm inclined to agree that "don't do that" is a reasonable stance
12:40:37	openstackgerrit	Tetsuro Nakamura proposed openstack/nova master: Add get_compute_nodes_by_host_or_node() https://review.openstack.org/650877
12:40:38	openstackgerrit	Tetsuro Nakamura proposed openstack/nova master: Pass target host to RequestGroup.in_tree https://review.openstack.org/650878
12:40:38	openstackgerrit	Tetsuro Nakamura proposed openstack/nova master: Query `in_tree` to placement https://review.openstack.org/649535
12:40:44	sean-k-mooney	/query cdent
12:41:01	cdent	?
12:41:52	sean-k-mooney	i was going to pm you and had an extra space
12:41:54	sean-k-mooney	:)
12:42:54	aspiers	lucky you noticed before you said something rude in public ;-)

Earlier   Later