| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-12-12 | |||
| 22:08:17 | jroll | hiya dims | |
| 22:20:43 | mriedem | jaypipes: so with the in_tree filter, | |
| 22:20:51 | mriedem | is the value for that a parent provider? | |
| 22:20:59 | mriedem | like, leaf nodes would always return an empty list? | |
| 22:22:08 | mriedem | if if i have parent A and children B and C, if i query for in_tree=C, do i get B because it's in the "same tree"? | |
| 22:22:56 | jaypipes | mriedem: the value is the UUID of any resource provider. the returned results are all providers in that provider's tree. you can provider a child, a root, a grandchild, it doesn't matter. | |
| 22:23:32 | mriedem | ok that wasn't clear to me from the api ref description, "A UUID of a resource provider. The returned resource providers will be in the same “provider tree” as the specified provider." and looking at the tests used in https://review.openstack.org/#/c/521097/ | |
| 22:23:56 | jaypipes | mriedem: the tests specifically show that behaviour. | |
| 22:24:31 | jaypipes | mriedem: you can specify the child, the root or the grandchild as the in_tree parameter value and all the providers in the tree will be returned. | |
| 22:24:34 | mriedem | sorry i mean https://review.openstack.org/#/c/520663/9 | |
| 22:24:40 | mriedem | https://review.openstack.org/#/c/520663/9/nova/tests/functional/api/openstack/placement/test_report_client.py | |
| 22:24:49 | mriedem | that's not a parent/child relationship so it wasn't clear to me | |
| 22:26:29 | jaypipes | mriedem: oh, yeah, sorry, I didn't write that particular patch | |
| 22:26:58 | jaypipes | mriedem: eric is always passing the root (which happens to be the compute node UUID) | |
| 22:27:50 | mriedem | yeah see my questoin at https://review.openstack.org/#/c/520663/9/nova/tests/unit/scheduler/client/test_report.py | |
| 22:27:57 | mriedem | L1400 | |
| 22:28:59 | mriedem | would probably be good to have a tree structure sample like that in the api ref description to explain the scenarios | |
| 22:42:00 | mnaser | melwitt: https://wiki.openstack.org/wiki/Nova-Cells-v2#Open_Questions appended the discussion re cellsv2+ceph+images | |
| 22:42:04 | mnaser | thanks for putting it up there | |
| 22:43:36 | melwitt | mnaser: cool, thanks for adding to it. so, with my question I was wondering how does glance manage multiple locations then. AFAIK it can only connect to one ceph cluster at a time itself, but I'm pretty ignorant about glance | |
| 22:44:26 | openstackgerrit | Jackie Truong proposed openstack/nova master: Implement certificate_utils https://review.openstack.org/479949 | |
| 22:44:45 | mnaser | melwitt: thats a good question, i think in this context, the nova <=> glance interactions, glance wont really ever talk to ceph because its just supplying locations | |
| 22:45:07 | mnaser | melwitt: now when you delete an image with multiple locations, some which are not accessible, what's going to happen... | |
| 22:45:11 | mnaser | another fun item to the list lol | |
| 22:45:33 | melwitt | mnaser: I see. so the workflow for the admin would be, store image in ceph cluster A, call glance api to add it to the DB and point at location A, and repeat | |
| 22:46:17 | melwitt | I saw that glance stores ceph credentials (one set) so that's what I was thinking of with the question | |
| 22:46:29 | melwitt | in its conf file I mean | |
| 22:46:41 | mnaser | yeah glance currently works with a single set of credentials, so the second location would manually be set (the API allows you up add/remove a location) | |
| 22:46:49 | melwitt | got it | |
| 22:47:16 | mnaser | so you can have some sort of cronjob that "replicates" stuff over across cluster but thats hacky and becomes confusing to the end user | |
| 22:47:41 | mnaser | and as you brought up, operations in glance might fail like deletes (unless you remove the location manually then delete but thats no fun for useres) | |
| 22:47:56 | melwitt | right | |
| 22:48:22 | mnaser | (this macbook keyboard is driving me insane with this weird things where it types 2 characters of the same letter when clicking it :() | |
| 22:48:47 | melwitt | O.o | |
| 22:49:00 | mnaser | (apple claimed it was a software bug.) | |
| 22:49:22 | melwitt | geesh, that's annoying | |
| 23:40:24 | openstackgerrit | Merged openstack/nova master: Remove direct usage of glance.generate_image_url https://review.openstack.org/511397 | |
| 23:42:01 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add nova-status check for ironic flavor migration https://review.openstack.org/527541 | |
| 23:42:02 | mriedem | edleafe: jaypipes: dansmith: ^ finally got around to add a nova-status check for the ironic flavor migration stuff; needs tests but hopefully it does the thing we need | |
| 23:42:20 | mriedem | i also specifically call out the need for some jaypipes join-fu | |
| 23:55:42 | newOpenstacker | @here need some help with nova APIs. If I do openstack server list --limit 1, I get 1 instance and then if I try openstack server list --limit 1 --marker <instance uuid>, I get same instance again | |
| 23:56:25 | newOpenstacker | has anyone seen this before? | |
| 23:57:44 | newOpenstacker | There is another issue on this setup. We have ~1800 instances. I tried nova list --all-tenants --limit 1000. If I follow the same pattern as mentioned earlier I end up with 2200 instances | |
| #openstack-nova - 2017-12-13 | |||
| 00:00:43 | jaypipes | mriedem: thanks matt | |
| 00:00:45 | mriedem | newOpenstacker: is this ocata, pike, queens (master)? | |
| 00:01:15 | newOpenstacker | Oh sorry I forgot to mention that. This is on newton currently | |
| 00:02:28 | mriedem | what happens if you do the same thing with a straight curl request? i wonder if the the CLIs are trying to "help" somehow | |
| 00:03:17 | newOpenstacker | ok. i'll try that and get back | |
| 00:06:11 | mriedem | i'd point you at https://developer.openstack.org/api-guide/compute/paginated_collections.html but it's more confusing probably because the description and samples are all effed up | |
| 00:06:26 | mriedem | saying it's paging over images but it's clearly servers, and the bottom sample has a self link with the wrong id | |
| 00:06:51 | mriedem | heh, and the bug link on that page doesn't work | |
| 00:10:30 | mriedem | https://bugs.launchpad.net/nova/+bug/1737854 | |
| 00:10:31 | openstack | Launchpad bug 1737854 in OpenStack Compute (nova) "Wrong content in "paginated collections" API guide page" [High,Confirmed] | |
| 00:11:07 | newOpenstacker | yes, even with curl calls the behavior is the same | |
| 00:12:07 | mriedem | hmm, not sure why with limit=1 and marker=x you'd get back x | |
| 00:12:16 | mriedem | it should be x+1 in the instances table | |
| 00:12:29 | mriedem | sorted on created_at,id in descending order | |
| 00:13:23 | mriedem | i wonder if that instance is a build request or something... | |
| 00:13:40 | mriedem | can you check if that instance is in the nova.instances table or the nova_api.build_requests table? | |
| 00:13:44 | mriedem | based on the instance uuid | |
| 00:14:04 | newOpenstacker | any particular column that is of interest here? | |
| 00:15:23 | newOpenstacker | The instance is in nova.instances with vm_state as building and also present in the build_requests table | |
| 00:15:36 | mriedem | ok that's likely the problem | |
| 00:15:44 | mriedem | did that instance fail to build? | |
| 00:15:53 | newOpenstacker | it got stuck in building state... | |
| 00:15:57 | mriedem | i'm guessing the build request didn't get cleaned up | |
| 00:16:07 | mriedem | the instance failed to build and the build request didn't get cleaned up for some reason | |
| 00:16:26 | newOpenstacker | so if there are instances in building state then paginations don't work? | |
| 00:16:27 | mriedem | do you have all of the latest newton fixes from stable/newton? | |
| 00:16:38 | mriedem | no | |
| 00:16:41 | mriedem | not saying that | |
| 00:16:49 | newOpenstacker | checking the exact version | |
| 00:16:51 | mriedem | but the instance shouldn't be in both the build_requests and instances table permanently | |
| 00:17:05 | mriedem | the entry in build_requests should be temporary until we find a host for the instance during scheduling | |
| 00:17:41 | mriedem | when listing instances, we start with the build_requests table and then move to the instances tables in the nova db | |
| 00:19:19 | newOpenstacker | on newton we are 7 minor releases behind. current tag on nova newton is 14.0.10 while this setup is on 14.0.3. | |
| 00:20:09 | mriedem | ok, definitely might have fixed the problem already where the build request should have been deleted | |
| 00:20:10 | newOpenstacker | I'll have to check why the instance is stuck in building state. Its been that way for couple of days. | |
| 00:20:34 | mriedem | looking at our paging code though, i don't think we're handling the marker properly if we find it in the build_requests table | |
| 00:20:34 | newOpenstacker | Ah ok. I'll see if we can move to the latest on newton | |
| 00:20:59 | newOpenstacker | Is there a bug already for this that I can monitor? | |
| 00:21:03 | mriedem | normally when we find the marker we need to nix it so we don't look for that marker in the instances table if we have more room in our limit, but i don't see that happening | |
| 00:21:12 | mriedem | i've never heard of the marker thing | |
| 00:22:43 | mriedem | these are the fixes you're missing btw http://paste.openstack.org/show/628777/ | |
| 00:23:13 | mriedem | 608105a Provide an online data migration to cleanup orphaned build requests | |
| 00:23:16 | mriedem | could be just what you need | |
| 00:24:32 | mriedem | release notes for newton if you're going to upgrade https://docs.openstack.org/releasenotes/nova/newton.html | |
| 00:25:51 | newOpenstacker | OK. Thanks @mriedem I'll try to get those fixes | |
| 00:26:12 | mriedem | i think that will fix the marker issue you're seeing too | |
| 00:26:19 | mriedem | because we should never find the marker in two different places | |
| 00:27:09 | newOpenstacker | Understood. So in this case removing those offending records from the database should fix it as well? | |
| 00:27:17 | mriedem | that's my guess | |
| 00:27:33 | newOpenstacker | Cool. Thanks @mriedem | |
| 00:27:36 | mriedem | yw | |
| 00:30:03 | mriedem | dansmith: melwitt: something to lose sleep over - shouldn't we set the marker to None if we find the marker in the build_requests table? https://github.com/openstack/nova/blob/master/nova/objects/build_request.py#L447 | |
| 00:30:25 | mriedem | just like here https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2426 | |
| 00:30:58 | dansmith | why? | |
| 00:31:05 | dansmith | marker isn't used after that, right? | |
| 00:31:26 | mriedem | yeah after build requests we page into the cells | |
| 00:31:34 | dansmith | back in compute api | |