Posted	Nick	Remark
#openstack-nova - 2017-12-13
00:02:28	mriedem	what happens if you do the same thing with a straight curl request? i wonder if the the CLIs are trying to "help" somehow
00:03:17	newOpenstacker	ok. i'll try that and get back
00:06:11	mriedem	i'd point you at https://developer.openstack.org/api-guide/compute/paginated_collections.html but it's more confusing probably because the description and samples are all effed up
00:06:26	mriedem	saying it's paging over images but it's clearly servers, and the bottom sample has a self link with the wrong id
00:06:51	mriedem	heh, and the bug link on that page doesn't work
00:10:30	mriedem	https://bugs.launchpad.net/nova/+bug/1737854
00:10:31	openstack	Launchpad bug 1737854 in OpenStack Compute (nova) "Wrong content in "paginated collections" API guide page" [High,Confirmed]
00:11:07	newOpenstacker	yes, even with curl calls the behavior is the same
00:12:07	mriedem	hmm, not sure why with limit=1 and marker=x you'd get back x
00:12:16	mriedem	it should be x+1 in the instances table
00:12:29	mriedem	sorted on created_at,id in descending order
00:13:23	mriedem	i wonder if that instance is a build request or something...
00:13:40	mriedem	can you check if that instance is in the nova.instances table or the nova_api.build_requests table?
00:13:44	mriedem	based on the instance uuid
00:14:04	newOpenstacker	any particular column that is of interest here?
00:15:23	newOpenstacker	The instance is in nova.instances with vm_state as building and also present in the build_requests table
00:15:36	mriedem	ok that's likely the problem
00:15:44	mriedem	did that instance fail to build?
00:15:53	newOpenstacker	it got stuck in building state...
00:15:57	mriedem	i'm guessing the build request didn't get cleaned up
00:16:07	mriedem	the instance failed to build and the build request didn't get cleaned up for some reason
00:16:26	newOpenstacker	so if there are instances in building state then paginations don't work?
00:16:27	mriedem	do you have all of the latest newton fixes from stable/newton?
00:16:38	mriedem	no
00:16:41	mriedem	not saying that
00:16:49	newOpenstacker	checking the exact version
00:16:51	mriedem	but the instance shouldn't be in both the build_requests and instances table permanently
00:17:05	mriedem	the entry in build_requests should be temporary until we find a host for the instance during scheduling
00:17:41	mriedem	when listing instances, we start with the build_requests table and then move to the instances tables in the nova db
00:19:19	newOpenstacker	on newton we are 7 minor releases behind. current tag on nova newton is 14.0.10 while this setup is on 14.0.3.
00:20:09	mriedem	ok, definitely might have fixed the problem already where the build request should have been deleted
00:20:10	newOpenstacker	I'll have to check why the instance is stuck in building state. Its been that way for couple of days.
00:20:34	mriedem	looking at our paging code though, i don't think we're handling the marker properly if we find it in the build_requests table
00:20:34	newOpenstacker	Ah ok. I'll see if we can move to the latest on newton
00:20:59	newOpenstacker	Is there a bug already for this that I can monitor?
00:21:03	mriedem	normally when we find the marker we need to nix it so we don't look for that marker in the instances table if we have more room in our limit, but i don't see that happening
00:21:12	mriedem	i've never heard of the marker thing
00:22:43	mriedem	these are the fixes you're missing btw http://paste.openstack.org/show/628777/
00:23:13	mriedem	608105a Provide an online data migration to cleanup orphaned build requests
00:23:16	mriedem	could be just what you need
00:24:32	mriedem	release notes for newton if you're going to upgrade https://docs.openstack.org/releasenotes/nova/newton.html
00:25:51	newOpenstacker	OK. Thanks @mriedem I'll try to get those fixes
00:26:12	mriedem	i think that will fix the marker issue you're seeing too
00:26:19	mriedem	because we should never find the marker in two different places
00:27:09	newOpenstacker	Understood. So in this case removing those offending records from the database should fix it as well?
00:27:17	mriedem	that's my guess
00:27:33	newOpenstacker	Cool. Thanks @mriedem
00:27:36	mriedem	yw
00:30:03	mriedem	dansmith: melwitt: something to lose sleep over - shouldn't we set the marker to None if we find the marker in the build_requests table? https://github.com/openstack/nova/blob/master/nova/objects/build_request.py#L447
00:30:25	mriedem	just like here https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2426
00:30:58	dansmith	why?
00:31:05	dansmith	marker isn't used after that, right?
00:31:26	mriedem	yeah after build requests we page into the cells
00:31:34	dansmith	back in compute api
00:32:00	mriedem	yeah https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2372-L2378
00:32:19	mriedem	so i realize setting marker=None in the build request code wouldn't fix that, we'd have to pass a marker variable back
00:32:32	mriedem	here https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2356
00:32:50	dansmith	if we got anything back from build request list then the marker was in there, yeah?
00:32:59	openstackgerrit	Merged openstack/nova master: VMware: fix memory stats https://review.openstack.org/516634
00:33:27	dansmith	which I guess we don't check
00:33:34	mriedem	i don't think so
00:33:35	mriedem	https://github.com/openstack/nova/blob/master/nova/objects/build_request.py#L440-L457
00:33:39	mriedem	we happily return what we found
00:33:51	mriedem	i.e. we don't raise MarkerNotFound
00:34:08	dansmith	hmm
00:34:08	mriedem	which seems buggish...
00:34:09	dansmith	yeah
00:36:43	tetsuro	mriedem: I responsed to your comment in https://review.openstack.org/#/c/465160/
00:38:00	mriedem	https://bugs.launchpad.net/nova/+bug/1737856
00:38:01	openstack	Launchpad bug 1737856 in OpenStack Compute (nova) "Listing instances with a marker doesn't nix the marker if it's found in build_requests" [Undecided,Triaged]
00:40:33	mriedem	alright my eyes are going to fall out if i don't stop staring at this screen so away i go
00:43:43	openstackgerrit	Ghanshyam Mann proposed openstack/python-novaclient master: Optimize jobs run on novaclient https://review.openstack.org/527550
00:44:17	gmann	mriedem: for your morning ^^
00:44:31	gmann	its re on your comment - https://review.openstack.org/#/c/522099/5/playbooks/legacy/novaclient-dsvm-functional-neutron/run.yaml@2
01:54:08	openstackgerrit	Matt Riedemann proposed openstack/nova stable/pike: Unmap compute nodes when deleting host mapping https://review.openstack.org/527560
01:56:22	openstackgerrit	Jackie Truong proposed openstack/nova master: Implement certificate_utils https://review.openstack.org/479949
02:05:05	openstackgerrit	Takashi NATSUME proposed openstack/nova master: [placement] Add sending global request ID in delete https://review.openstack.org/525231
02:05:18	openstackgerrit	Takashi NATSUME proposed openstack/nova master: [placement] Add sending global request ID in delete (2) https://review.openstack.org/526263
02:06:04	openstackgerrit	Takashi NATSUME proposed openstack/nova master: [placement] Add sending global request ID in delete (3) https://review.openstack.org/526557
02:06:20	openstackgerrit	Takashi NATSUME proposed openstack/nova master: [placement] Add sending global request ID in post https://review.openstack.org/526823
02:06:33	openstackgerrit	Takashi NATSUME proposed openstack/nova master: Fix the order of target host checks https://review.openstack.org/526225
02:07:21	openstackgerrit	Takashi NATSUME proposed openstack/nova master: List/show all server migration types (1/2) https://review.openstack.org/430608
02:07:38	openstackgerrit	Takashi NATSUME proposed openstack/nova master: List/show all server migration types (2/2) https://review.openstack.org/459483
02:08:45	openstackgerrit	Takashi NATSUME proposed openstack/nova master: Add a warning in 'nova-manage cell_v2 delete_cell' https://review.openstack.org/513771
02:10:14	mriedem	gmann: thanks
02:32:13	openstackgerrit	Matt Riedemann proposed openstack/nova master: Raise MarkerNotFound if BuildRequestList.get_by_filters doesn't find marker https://review.openstack.org/527564
02:36:34	openstackgerrit	Takashi NATSUME proposed openstack/nova master: [placement] Add x-openstack-request-id in API ref https://review.openstack.org/523007
02:56:38	openstackgerrit	Merged openstack/nova master: Add quiesce and unquiesce in support matrix https://review.openstack.org/482390
03:25:21	mriedem	alex_xu: can you get this in? https://review.openstack.org/#/c/527440/ i'm seeing a lot of failed CI jobs from that.
03:41:20	openstackgerrit	Merged openstack/nova master: SchedulerReportClient._get_providers_in_tree https://review.openstack.org/520663
04:31:00	openstackgerrit	Merged openstack/nova stable/pike: Make TestRPC inherit from the base nova TestCase https://review.openstack.org/511842
04:31:12	openstackgerrit	Merged openstack/nova master: api-ref: Fix a description for 'guest_format' https://review.openstack.org/525928
05:30:34	openstackgerrit	Merged openstack/nova master: [placement] Add cache headers to placement api requests https://review.openstack.org/521640
05:35:19	openstackgerrit	Merged openstack/nova master: [placement] Add info about last-modified to contrib docs https://review.openstack.org/526084
05:52:17	openstackgerrit	Takashi NATSUME proposed openstack/nova master: Add a warning in 'nova-manage cell_v2 delete_cell' https://review.openstack.org/513771
06:17:57	openstackgerrit	Merged openstack/nova master: Remove the unused request_id filter from api-paste.ini https://review.openstack.org/526001
06:32:08	openstackgerrit	Yikun Jiang (Kero) proposed openstack/nova master: Some nit fix in multi_cell_list https://review.openstack.org/527597
07:11:25	openstackgerrit	Merged openstack/nova master: Refactor placement version check https://review.openstack.org/512497
07:18:46	openstackgerrit	Merged openstack/nova master: trivial: more suitable log in set_admin_password https://review.openstack.org/526592
07:35:57	openstackgerrit	OpenStack Proposal Bot proposed openstack/nova master: Imported Translations from Zanata https://review.openstack.org/524795

Earlier   Later