Posted	Nick	Remark
#openstack-nova - 2017-12-13
00:16:51	mriedem	but the instance shouldn't be in both the build_requests and instances table permanently
00:17:05	mriedem	the entry in build_requests should be temporary until we find a host for the instance during scheduling
00:17:41	mriedem	when listing instances, we start with the build_requests table and then move to the instances tables in the nova db
00:19:19	newOpenstacker	on newton we are 7 minor releases behind. current tag on nova newton is 14.0.10 while this setup is on 14.0.3.
00:20:09	mriedem	ok, definitely might have fixed the problem already where the build request should have been deleted
00:20:10	newOpenstacker	I'll have to check why the instance is stuck in building state. Its been that way for couple of days.
00:20:34	newOpenstacker	Ah ok. I'll see if we can move to the latest on newton
00:20:34	mriedem	looking at our paging code though, i don't think we're handling the marker properly if we find it in the build_requests table
00:20:59	newOpenstacker	Is there a bug already for this that I can monitor?
00:21:03	mriedem	normally when we find the marker we need to nix it so we don't look for that marker in the instances table if we have more room in our limit, but i don't see that happening
00:21:12	mriedem	i've never heard of the marker thing
00:22:43	mriedem	these are the fixes you're missing btw http://paste.openstack.org/show/628777/
00:23:13	mriedem	608105a Provide an online data migration to cleanup orphaned build requests
00:23:16	mriedem	could be just what you need
00:24:32	mriedem	release notes for newton if you're going to upgrade https://docs.openstack.org/releasenotes/nova/newton.html
00:25:51	newOpenstacker	OK. Thanks @mriedem I'll try to get those fixes
00:26:12	mriedem	i think that will fix the marker issue you're seeing too
00:26:19	mriedem	because we should never find the marker in two different places
00:27:09	newOpenstacker	Understood. So in this case removing those offending records from the database should fix it as well?
00:27:17	mriedem	that's my guess
00:27:33	newOpenstacker	Cool. Thanks @mriedem
00:27:36	mriedem	yw
00:30:03	mriedem	dansmith: melwitt: something to lose sleep over - shouldn't we set the marker to None if we find the marker in the build_requests table? https://github.com/openstack/nova/blob/master/nova/objects/build_request.py#L447
00:30:25	mriedem	just like here https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2426
00:30:58	dansmith	why?
00:31:05	dansmith	marker isn't used after that, right?
00:31:26	mriedem	yeah after build requests we page into the cells
00:31:34	dansmith	back in compute api
00:32:00	mriedem	yeah https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2372-L2378
00:32:19	mriedem	so i realize setting marker=None in the build request code wouldn't fix that, we'd have to pass a marker variable back
00:32:32	mriedem	here https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2356
00:32:50	dansmith	if we got anything back from build request list then the marker was in there, yeah?
00:32:59	openstackgerrit	Merged openstack/nova master: VMware: fix memory stats https://review.openstack.org/516634
00:33:27	dansmith	which I guess we don't check
00:33:34	mriedem	i don't think so
00:33:35	mriedem	https://github.com/openstack/nova/blob/master/nova/objects/build_request.py#L440-L457
00:33:39	mriedem	we happily return what we found
00:33:51	mriedem	i.e. we don't raise MarkerNotFound
00:34:08	mriedem	which seems buggish...
00:34:08	dansmith	hmm
00:34:09	dansmith	yeah
00:36:43	tetsuro	mriedem: I responsed to your comment in https://review.openstack.org/#/c/465160/
00:38:00	mriedem	https://bugs.launchpad.net/nova/+bug/1737856
00:38:01	openstack	Launchpad bug 1737856 in OpenStack Compute (nova) "Listing instances with a marker doesn't nix the marker if it's found in build_requests" [Undecided,Triaged]
00:40:33	mriedem	alright my eyes are going to fall out if i don't stop staring at this screen so away i go
00:43:43	openstackgerrit	Ghanshyam Mann proposed openstack/python-novaclient master: Optimize jobs run on novaclient https://review.openstack.org/527550
00:44:17	gmann	mriedem: for your morning ^^
00:44:31	gmann	its re on your comment - https://review.openstack.org/#/c/522099/5/playbooks/legacy/novaclient-dsvm-functional-neutron/run.yaml@2
01:54:08	openstackgerrit	Matt Riedemann proposed openstack/nova stable/pike: Unmap compute nodes when deleting host mapping https://review.openstack.org/527560
01:56:22	openstackgerrit	Jackie Truong proposed openstack/nova master: Implement certificate_utils https://review.openstack.org/479949
02:05:05	openstackgerrit	Takashi NATSUME proposed openstack/nova master: [placement] Add sending global request ID in delete https://review.openstack.org/525231
02:05:18	openstackgerrit	Takashi NATSUME proposed openstack/nova master: [placement] Add sending global request ID in delete (2) https://review.openstack.org/526263
02:06:04	openstackgerrit	Takashi NATSUME proposed openstack/nova master: [placement] Add sending global request ID in delete (3) https://review.openstack.org/526557
02:06:20	openstackgerrit	Takashi NATSUME proposed openstack/nova master: [placement] Add sending global request ID in post https://review.openstack.org/526823
02:06:33	openstackgerrit	Takashi NATSUME proposed openstack/nova master: Fix the order of target host checks https://review.openstack.org/526225
02:07:21	openstackgerrit	Takashi NATSUME proposed openstack/nova master: List/show all server migration types (1/2) https://review.openstack.org/430608
02:07:38	openstackgerrit	Takashi NATSUME proposed openstack/nova master: List/show all server migration types (2/2) https://review.openstack.org/459483
02:08:45	openstackgerrit	Takashi NATSUME proposed openstack/nova master: Add a warning in 'nova-manage cell_v2 delete_cell' https://review.openstack.org/513771
02:10:14	mriedem	gmann: thanks
02:32:13	openstackgerrit	Matt Riedemann proposed openstack/nova master: Raise MarkerNotFound if BuildRequestList.get_by_filters doesn't find marker https://review.openstack.org/527564
02:36:34	openstackgerrit	Takashi NATSUME proposed openstack/nova master: [placement] Add x-openstack-request-id in API ref https://review.openstack.org/523007
02:56:38	openstackgerrit	Merged openstack/nova master: Add quiesce and unquiesce in support matrix https://review.openstack.org/482390
03:25:21	mriedem	alex_xu: can you get this in? https://review.openstack.org/#/c/527440/ i'm seeing a lot of failed CI jobs from that.
03:41:20	openstackgerrit	Merged openstack/nova master: SchedulerReportClient._get_providers_in_tree https://review.openstack.org/520663
04:31:00	openstackgerrit	Merged openstack/nova stable/pike: Make TestRPC inherit from the base nova TestCase https://review.openstack.org/511842
04:31:12	openstackgerrit	Merged openstack/nova master: api-ref: Fix a description for 'guest_format' https://review.openstack.org/525928
05:30:34	openstackgerrit	Merged openstack/nova master: [placement] Add cache headers to placement api requests https://review.openstack.org/521640
05:35:19	openstackgerrit	Merged openstack/nova master: [placement] Add info about last-modified to contrib docs https://review.openstack.org/526084
05:52:17	openstackgerrit	Takashi NATSUME proposed openstack/nova master: Add a warning in 'nova-manage cell_v2 delete_cell' https://review.openstack.org/513771
06:17:57	openstackgerrit	Merged openstack/nova master: Remove the unused request_id filter from api-paste.ini https://review.openstack.org/526001
06:32:08	openstackgerrit	Yikun Jiang (Kero) proposed openstack/nova master: Some nit fix in multi_cell_list https://review.openstack.org/527597
07:11:25	openstackgerrit	Merged openstack/nova master: Refactor placement version check https://review.openstack.org/512497
07:18:46	openstackgerrit	Merged openstack/nova master: trivial: more suitable log in set_admin_password https://review.openstack.org/526592
07:35:57	openstackgerrit	OpenStack Proposal Bot proposed openstack/nova master: Imported Translations from Zanata https://review.openstack.org/524795
08:40:12	openstackgerrit	Tetsuro Nakamura proposed openstack/nova master: [libvirt] Add _set_vcpu_realtime_scheduler() https://review.openstack.org/527630
08:40:13	openstackgerrit	Tetsuro Nakamura proposed openstack/nova master: Add _set_vcpu_pinning() and _set_numa_memory() https://review.openstack.org/527631
09:03:15	openstackgerrit	Tetsuro Nakamura proposed openstack/nova master: Add _set_vcpu_pinning() and _set_numa_memory() https://review.openstack.org/527631
09:17:47	mdbooth	mnaser: melwitt Reading back scroll. Yes, a per-backend image cache was part of my plans both times I've submitted patches for a massive refactor of the imagebackend code.
09:18:38	mdbooth	You'll hit a few architectural issues getting it landed, though.
09:18:52	openstackgerrit	Yikun Jiang (Kero) proposed openstack/nova master: Add pagination and changes-since for instance-actions https://review.openstack.org/326326
09:21:39	mdbooth	Firstly, the image cache code is unnecessarily and messily embedded implicitly in each individual imagebackend already. All that code needs to be cleaned up. In itself that's not too hard, but the tests are also hugely problematic. Ultimately, fixing the tests to a point where it was possible to refactor the code sunk me twice. There are a huge number of tests in there, and in general the quality is rock bottom.
09:23:19	mdbooth	However, if you get to the point where you've unpicked the imagecache from the imagebackend code, you still have to handle both layouts.
09:23:58	mdbooth	Currently, the only source of truth for the layout of an instance is the config of the compute host is resides on.
09:24:54	mdbooth	So if we're to have a transitional period where we're altering the on-disk layout, which is what changing the image cache would amount to, we need to have a per-instance datastore for driver-specific data.
09:25:22	mdbooth	Incidentally, my plan was to add this to the BDM so it could actually be per-disk.
09:26:43	mdbooth	But if you can get there, yes the current design of the imagecache makes no sense
09:27:04	mdbooth	Well, no longer makes any sense
09:27:27	mdbooth	It should have been refactored long ago the first time we implemented an alternate backend
09:36:53	openstackgerrit	Yikun Jiang (Kero) proposed openstack/nova master: Add pagination and changes-since for instance-actions https://review.openstack.org/326326
09:41:06	kashyap	mdbooth: You might want to spell that documentation somewhere to whoever that is planning to go that route...
09:42:01	kashyap	Maybe you did in all those previous iterations. In that case, disregard me :-)
09:42:14	mdbooth	kashyap: I wrote the code :)
09:42:21	mdbooth	Then I threw it away.
09:42:25	mdbooth	The I wrote it again!
09:42:29	kashyap	mdbooth: Whoops
09:42:33	mdbooth	Then I threw that away, too.
09:42:44	kashyap	Hence the "sunk me twice", now that nice expression makes sense
09:42:59	kashyap	s/makes sense/makes doubly sense/
09:43:01	mdbooth	Each one took me a couple of months.
09:43:23	mdbooth	Throwing away 2 months of effort is good for the soul.

Earlier   Later