| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-12-12 | |||
| 22:49:00 | mnaser | (apple claimed it was a software bug.) | |
| 22:49:22 | melwitt | geesh, that's annoying | |
| 23:40:24 | openstackgerrit | Merged openstack/nova master: Remove direct usage of glance.generate_image_url https://review.openstack.org/511397 | |
| 23:42:01 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add nova-status check for ironic flavor migration https://review.openstack.org/527541 | |
| 23:42:02 | mriedem | edleafe: jaypipes: dansmith: ^ finally got around to add a nova-status check for the ironic flavor migration stuff; needs tests but hopefully it does the thing we need | |
| 23:42:20 | mriedem | i also specifically call out the need for some jaypipes join-fu | |
| 23:55:42 | newOpenstacker | @here need some help with nova APIs. If I do openstack server list --limit 1, I get 1 instance and then if I try openstack server list --limit 1 --marker <instance uuid>, I get same instance again | |
| 23:56:25 | newOpenstacker | has anyone seen this before? | |
| 23:57:44 | newOpenstacker | There is another issue on this setup. We have ~1800 instances. I tried nova list --all-tenants --limit 1000. If I follow the same pattern as mentioned earlier I end up with 2200 instances | |
| #openstack-nova - 2017-12-13 | |||
| 00:00:43 | jaypipes | mriedem: thanks matt | |
| 00:00:45 | mriedem | newOpenstacker: is this ocata, pike, queens (master)? | |
| 00:01:15 | newOpenstacker | Oh sorry I forgot to mention that. This is on newton currently | |
| 00:02:28 | mriedem | what happens if you do the same thing with a straight curl request? i wonder if the the CLIs are trying to "help" somehow | |
| 00:03:17 | newOpenstacker | ok. i'll try that and get back | |
| 00:06:11 | mriedem | i'd point you at https://developer.openstack.org/api-guide/compute/paginated_collections.html but it's more confusing probably because the description and samples are all effed up | |
| 00:06:26 | mriedem | saying it's paging over images but it's clearly servers, and the bottom sample has a self link with the wrong id | |
| 00:06:51 | mriedem | heh, and the bug link on that page doesn't work | |
| 00:10:30 | mriedem | https://bugs.launchpad.net/nova/+bug/1737854 | |
| 00:10:31 | openstack | Launchpad bug 1737854 in OpenStack Compute (nova) "Wrong content in "paginated collections" API guide page" [High,Confirmed] | |
| 00:11:07 | newOpenstacker | yes, even with curl calls the behavior is the same | |
| 00:12:07 | mriedem | hmm, not sure why with limit=1 and marker=x you'd get back x | |
| 00:12:16 | mriedem | it should be x+1 in the instances table | |
| 00:12:29 | mriedem | sorted on created_at,id in descending order | |
| 00:13:23 | mriedem | i wonder if that instance is a build request or something... | |
| 00:13:40 | mriedem | can you check if that instance is in the nova.instances table or the nova_api.build_requests table? | |
| 00:13:44 | mriedem | based on the instance uuid | |
| 00:14:04 | newOpenstacker | any particular column that is of interest here? | |
| 00:15:23 | newOpenstacker | The instance is in nova.instances with vm_state as building and also present in the build_requests table | |
| 00:15:36 | mriedem | ok that's likely the problem | |
| 00:15:44 | mriedem | did that instance fail to build? | |
| 00:15:53 | newOpenstacker | it got stuck in building state... | |
| 00:15:57 | mriedem | i'm guessing the build request didn't get cleaned up | |
| 00:16:07 | mriedem | the instance failed to build and the build request didn't get cleaned up for some reason | |
| 00:16:26 | newOpenstacker | so if there are instances in building state then paginations don't work? | |
| 00:16:27 | mriedem | do you have all of the latest newton fixes from stable/newton? | |
| 00:16:38 | mriedem | no | |
| 00:16:41 | mriedem | not saying that | |
| 00:16:49 | newOpenstacker | checking the exact version | |
| 00:16:51 | mriedem | but the instance shouldn't be in both the build_requests and instances table permanently | |
| 00:17:05 | mriedem | the entry in build_requests should be temporary until we find a host for the instance during scheduling | |
| 00:17:41 | mriedem | when listing instances, we start with the build_requests table and then move to the instances tables in the nova db | |
| 00:19:19 | newOpenstacker | on newton we are 7 minor releases behind. current tag on nova newton is 14.0.10 while this setup is on 14.0.3. | |
| 00:20:09 | mriedem | ok, definitely might have fixed the problem already where the build request should have been deleted | |
| 00:20:10 | newOpenstacker | I'll have to check why the instance is stuck in building state. Its been that way for couple of days. | |
| 00:20:34 | mriedem | looking at our paging code though, i don't think we're handling the marker properly if we find it in the build_requests table | |
| 00:20:34 | newOpenstacker | Ah ok. I'll see if we can move to the latest on newton | |
| 00:20:59 | newOpenstacker | Is there a bug already for this that I can monitor? | |
| 00:21:03 | mriedem | normally when we find the marker we need to nix it so we don't look for that marker in the instances table if we have more room in our limit, but i don't see that happening | |
| 00:21:12 | mriedem | i've never heard of the marker thing | |
| 00:22:43 | mriedem | these are the fixes you're missing btw http://paste.openstack.org/show/628777/ | |
| 00:23:13 | mriedem | 608105a Provide an online data migration to cleanup orphaned build requests | |
| 00:23:16 | mriedem | could be just what you need | |
| 00:24:32 | mriedem | release notes for newton if you're going to upgrade https://docs.openstack.org/releasenotes/nova/newton.html | |
| 00:25:51 | newOpenstacker | OK. Thanks @mriedem I'll try to get those fixes | |
| 00:26:12 | mriedem | i think that will fix the marker issue you're seeing too | |
| 00:26:19 | mriedem | because we should never find the marker in two different places | |
| 00:27:09 | newOpenstacker | Understood. So in this case removing those offending records from the database should fix it as well? | |
| 00:27:17 | mriedem | that's my guess | |
| 00:27:33 | newOpenstacker | Cool. Thanks @mriedem | |
| 00:27:36 | mriedem | yw | |
| 00:30:03 | mriedem | dansmith: melwitt: something to lose sleep over - shouldn't we set the marker to None if we find the marker in the build_requests table? https://github.com/openstack/nova/blob/master/nova/objects/build_request.py#L447 | |
| 00:30:25 | mriedem | just like here https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2426 | |
| 00:30:58 | dansmith | why? | |
| 00:31:05 | dansmith | marker isn't used after that, right? | |
| 00:31:26 | mriedem | yeah after build requests we page into the cells | |
| 00:31:34 | dansmith | back in compute api | |
| 00:32:00 | mriedem | yeah https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2372-L2378 | |
| 00:32:19 | mriedem | so i realize setting marker=None in the build request code wouldn't fix that, we'd have to pass a marker variable back | |
| 00:32:32 | mriedem | here https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2356 | |
| 00:32:50 | dansmith | if we got anything back from build request list then the marker was in there, yeah? | |
| 00:32:59 | openstackgerrit | Merged openstack/nova master: VMware: fix memory stats https://review.openstack.org/516634 | |
| 00:33:27 | dansmith | which I guess we don't check | |
| 00:33:34 | mriedem | i don't think so | |
| 00:33:35 | mriedem | https://github.com/openstack/nova/blob/master/nova/objects/build_request.py#L440-L457 | |
| 00:33:39 | mriedem | we happily return what we found | |
| 00:33:51 | mriedem | i.e. we don't raise MarkerNotFound | |
| 00:34:08 | dansmith | hmm | |
| 00:34:08 | mriedem | which seems buggish... | |
| 00:34:09 | dansmith | yeah | |
| 00:36:43 | tetsuro | mriedem: I responsed to your comment in https://review.openstack.org/#/c/465160/ | |
| 00:38:00 | mriedem | https://bugs.launchpad.net/nova/+bug/1737856 | |
| 00:38:01 | openstack | Launchpad bug 1737856 in OpenStack Compute (nova) "Listing instances with a marker doesn't nix the marker if it's found in build_requests" [Undecided,Triaged] | |
| 00:40:33 | mriedem | alright my eyes are going to fall out if i don't stop staring at this screen so away i go | |
| 00:43:43 | openstackgerrit | Ghanshyam Mann proposed openstack/python-novaclient master: Optimize jobs run on novaclient https://review.openstack.org/527550 | |
| 00:44:17 | gmann | mriedem: for your morning ^^ | |
| 00:44:31 | gmann | its re on your comment - https://review.openstack.org/#/c/522099/5/playbooks/legacy/novaclient-dsvm-functional-neutron/run.yaml@2 | |
| 01:54:08 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Unmap compute nodes when deleting host mapping https://review.openstack.org/527560 | |
| 01:56:22 | openstackgerrit | Jackie Truong proposed openstack/nova master: Implement certificate_utils https://review.openstack.org/479949 | |
| 02:05:05 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: [placement] Add sending global request ID in delete https://review.openstack.org/525231 | |
| 02:05:18 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: [placement] Add sending global request ID in delete (2) https://review.openstack.org/526263 | |
| 02:06:04 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: [placement] Add sending global request ID in delete (3) https://review.openstack.org/526557 | |
| 02:06:20 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: [placement] Add sending global request ID in post https://review.openstack.org/526823 | |
| 02:06:33 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Fix the order of target host checks https://review.openstack.org/526225 | |
| 02:07:21 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: List/show all server migration types (1/2) https://review.openstack.org/430608 | |
| 02:07:38 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: List/show all server migration types (2/2) https://review.openstack.org/459483 | |
| 02:08:45 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Add a warning in 'nova-manage cell_v2 delete_cell' https://review.openstack.org/513771 | |
| 02:10:14 | mriedem | gmann: thanks | |
| 02:32:13 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Raise MarkerNotFound if BuildRequestList.get_by_filters doesn't find marker https://review.openstack.org/527564 | |
| 02:36:34 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: [placement] Add x-openstack-request-id in API ref https://review.openstack.org/523007 | |
| 02:56:38 | openstackgerrit | Merged openstack/nova master: Add quiesce and unquiesce in support matrix https://review.openstack.org/482390 | |