Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-12-12
22:28:59 mriedem would probably be good to have a tree structure sample like that in the api ref description to explain the scenarios
22:42:00 mnaser melwitt: https://wiki.openstack.org/wiki/Nova-Cells-v2#Open_Questions appended the discussion re cellsv2+ceph+images
22:42:04 mnaser thanks for putting it up there
22:43:36 melwitt mnaser: cool, thanks for adding to it. so, with my question I was wondering how does glance manage multiple locations then. AFAIK it can only connect to one ceph cluster at a time itself, but I'm pretty ignorant about glance
22:44:26 openstackgerrit Jackie Truong proposed openstack/nova master: Implement certificate_utils https://review.openstack.org/479949
22:44:45 mnaser melwitt: thats a good question, i think in this context, the nova <=> glance interactions, glance wont really ever talk to ceph because its just supplying locations
22:45:07 mnaser melwitt: now when you delete an image with multiple locations, some which are not accessible, what's going to happen...
22:45:11 mnaser another fun item to the list lol
22:45:33 melwitt mnaser: I see. so the workflow for the admin would be, store image in ceph cluster A, call glance api to add it to the DB and point at location A, and repeat
22:46:17 melwitt I saw that glance stores ceph credentials (one set) so that's what I was thinking of with the question
22:46:29 melwitt in its conf file I mean
22:46:41 mnaser yeah glance currently works with a single set of credentials, so the second location would manually be set (the API allows you up add/remove a location)
22:46:49 melwitt got it
22:47:16 mnaser so you can have some sort of cronjob that "replicates" stuff over across cluster but thats hacky and becomes confusing to the end user
22:47:41 mnaser and as you brought up, operations in glance might fail like deletes (unless you remove the location manually then delete but thats no fun for useres)
22:47:56 melwitt right
22:48:22 mnaser (this macbook keyboard is driving me insane with this weird things where it types 2 characters of the same letter when clicking it :()
22:48:47 melwitt O.o
22:49:00 mnaser (apple claimed it was a software bug.)
22:49:22 melwitt geesh, that's annoying
23:40:24 openstackgerrit Merged openstack/nova master: Remove direct usage of glance.generate_image_url https://review.openstack.org/511397
23:42:01 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Add nova-status check for ironic flavor migration https://review.openstack.org/527541
23:42:02 mriedem edleafe: jaypipes: dansmith: ^ finally got around to add a nova-status check for the ironic flavor migration stuff; needs tests but hopefully it does the thing we need
23:42:20 mriedem i also specifically call out the need for some jaypipes join-fu
23:55:42 newOpenstacker @here need some help with nova APIs. If I do openstack server list --limit 1, I get 1 instance and then if I try openstack server list --limit 1 --marker <instance uuid>, I get same instance again
23:56:25 newOpenstacker has anyone seen this before?
23:57:44 newOpenstacker There is another issue on this setup. We have ~1800 instances. I tried nova list --all-tenants --limit 1000. If I follow the same pattern as mentioned earlier I end up with 2200 instances
#openstack-nova - 2017-12-13
00:00:43 jaypipes mriedem: thanks matt
00:00:45 mriedem newOpenstacker: is this ocata, pike, queens (master)?
00:01:15 newOpenstacker Oh sorry I forgot to mention that. This is on newton currently
00:02:28 mriedem what happens if you do the same thing with a straight curl request? i wonder if the the CLIs are trying to "help" somehow
00:03:17 newOpenstacker ok. i'll try that and get back
00:06:11 mriedem i'd point you at https://developer.openstack.org/api-guide/compute/paginated_collections.html but it's more confusing probably because the description and samples are all effed up
00:06:26 mriedem saying it's paging over images but it's clearly servers, and the bottom sample has a self link with the wrong id
00:06:51 mriedem heh, and the bug link on that page doesn't work
00:10:30 mriedem https://bugs.launchpad.net/nova/+bug/1737854
00:10:31 openstack Launchpad bug 1737854 in OpenStack Compute (nova) "Wrong content in "paginated collections" API guide page" [High,Confirmed]
00:11:07 newOpenstacker yes, even with curl calls the behavior is the same
00:12:07 mriedem hmm, not sure why with limit=1 and marker=x you'd get back x
00:12:16 mriedem it should be x+1 in the instances table
00:12:29 mriedem sorted on created_at,id in descending order
00:13:23 mriedem i wonder if that instance is a build request or something...
00:13:40 mriedem can you check if that instance is in the nova.instances table or the nova_api.build_requests table?
00:13:44 mriedem based on the instance uuid
00:14:04 newOpenstacker any particular column that is of interest here?
00:15:23 newOpenstacker The instance is in nova.instances with vm_state as building and also present in the build_requests table
00:15:36 mriedem ok that's likely the problem
00:15:44 mriedem did that instance fail to build?
00:15:53 newOpenstacker it got stuck in building state...
00:15:57 mriedem i'm guessing the build request didn't get cleaned up
00:16:07 mriedem the instance failed to build and the build request didn't get cleaned up for some reason
00:16:26 newOpenstacker so if there are instances in building state then paginations don't work?
00:16:27 mriedem do you have all of the latest newton fixes from stable/newton?
00:16:38 mriedem no
00:16:41 mriedem not saying that
00:16:49 newOpenstacker checking the exact version
00:16:51 mriedem but the instance shouldn't be in both the build_requests and instances table permanently
00:17:05 mriedem the entry in build_requests should be temporary until we find a host for the instance during scheduling
00:17:41 mriedem when listing instances, we start with the build_requests table and then move to the instances tables in the nova db
00:19:19 newOpenstacker on newton we are 7 minor releases behind. current tag on nova newton is 14.0.10 while this setup is on 14.0.3.
00:20:09 mriedem ok, definitely might have fixed the problem already where the build request should have been deleted
00:20:10 newOpenstacker I'll have to check why the instance is stuck in building state. Its been that way for couple of days.
00:20:34 newOpenstacker Ah ok. I'll see if we can move to the latest on newton
00:20:34 mriedem looking at our paging code though, i don't think we're handling the marker properly if we find it in the build_requests table
00:20:59 newOpenstacker Is there a bug already for this that I can monitor?
00:21:03 mriedem normally when we find the marker we need to nix it so we don't look for that marker in the instances table if we have more room in our limit, but i don't see that happening
00:21:12 mriedem i've never heard of the marker thing
00:22:43 mriedem these are the fixes you're missing btw http://paste.openstack.org/show/628777/
00:23:13 mriedem 608105a Provide an online data migration to cleanup orphaned build requests
00:23:16 mriedem could be just what you need
00:24:32 mriedem release notes for newton if you're going to upgrade https://docs.openstack.org/releasenotes/nova/newton.html
00:25:51 newOpenstacker OK. Thanks @mriedem I'll try to get those fixes
00:26:12 mriedem i think that will fix the marker issue you're seeing too
00:26:19 mriedem because we should never find the marker in two different places
00:27:09 newOpenstacker Understood. So in this case removing those offending records from the database should fix it as well?
00:27:17 mriedem that's my guess
00:27:33 newOpenstacker Cool. Thanks @mriedem
00:27:36 mriedem yw
00:30:03 mriedem dansmith: melwitt: something to lose sleep over - shouldn't we set the marker to None if we find the marker in the build_requests table? https://github.com/openstack/nova/blob/master/nova/objects/build_request.py#L447
00:30:25 mriedem just like here https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2426
00:30:58 dansmith why?
00:31:05 dansmith marker isn't used after that, right?
00:31:26 mriedem yeah after build requests we page into the cells
00:31:34 dansmith back in compute api
00:32:00 mriedem yeah https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2372-L2378
00:32:19 mriedem so i realize setting marker=None in the build request code wouldn't fix that, we'd have to pass a marker variable back
00:32:32 mriedem here https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2356
00:32:50 dansmith if we got anything back from build request list then the marker was in there, yeah?
00:32:59 openstackgerrit Merged openstack/nova master: VMware: fix memory stats https://review.openstack.org/516634
00:33:27 dansmith which I guess we don't check
00:33:34 mriedem i don't think so
00:33:35 mriedem https://github.com/openstack/nova/blob/master/nova/objects/build_request.py#L440-L457
00:33:39 mriedem we happily return what we found
00:33:51 mriedem i.e. we don't raise MarkerNotFound
00:34:08 mriedem which seems buggish...
00:34:08 dansmith hmm
00:34:09 dansmith yeah
00:36:43 tetsuro mriedem: I responsed to your comment in https://review.openstack.org/#/c/465160/
00:38:00 mriedem https://bugs.launchpad.net/nova/+bug/1737856
00:38:01 openstack Launchpad bug 1737856 in OpenStack Compute (nova) "Listing instances with a marker doesn't nix the marker if it's found in build_requests" [Undecided,Triaged]

Earlier   Later