| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-06-05 | |||
| 19:18:50 | openstack | Launchpad bug 1773945 in OpenStack Compute (nova) "nova client servers.list crashes with bad marker" [Undecided,New] - Assigned to Surya Seetharaman (tssurya) | |
| 19:18:55 | tssurya | couldn't figure out what the issue is | |
| 19:19:10 | tssurya | if any of you have ideas | |
| 19:19:54 | mriedem | i saw something really weird the other day with markers | |
| 19:19:57 | mriedem | with osc | |
| 19:20:03 | mriedem | which uses nova python API bindings | |
| 19:20:26 | tssurya | oh | |
| 19:20:56 | mriedem | i was creating some instances when rabbitmq dropped | |
| 19:21:19 | mriedem | and i think it might have been what takashi said in the bug report, "if the record of the marker VM instance exists in the cell | |
| 19:21:19 | mriedem | but the 'cell_mapping' in the InstanceMapping of the marker VM instance is null (None), | |
| 19:21:19 | mriedem | the issue occurs." | |
| 19:21:54 | tssurya | but this would happen only during schedule time right ? | |
| 19:22:04 | tssurya | I thought takashi was right too at first | |
| 19:22:07 | mriedem | rabbit died during scheduling | |
| 19:22:26 | tssurya | ah :D | |
| 19:22:39 | mriedem | the weird thing is when i was debugging the CLI, it was passing a marker | |
| 19:22:46 | mriedem | and i don't know where that marker was coming from | |
| 19:22:58 | dansmith | tssurya: is that bug not just that the marker gets deleted during the paging? | |
| 19:23:21 | dansmith | oh, no, we do read_deleted=yes it | |
| 19:23:29 | tssurya | yea | |
| 19:24:04 | dansmith | so maybe we should check if the marker has a null cell mapping and log that case/ | |
| 19:24:10 | dansmith | either way, not much we can do there | |
| 19:24:29 | tssurya | dansmith: yep I agree, | |
| 19:24:53 | tssurya | I just never thought there would be case without any BuildRequest and NULL InstanceMapping | |
| 19:25:09 | dansmith | or, we just scan every cell! (no not really) | |
| 19:27:06 | tssurya | dansmith: I also wanted to ask another thing about have a new option --cell-name to nova service-list ; how you would feel about this, but we could also do this in tomorrow's cells meeting if you are busy now | |
| 19:27:29 | dansmith | tssurya: I don't want that, no | |
| 19:27:46 | dansmith | tssurya: put them in an aggregate and have a way to list via aggregate or something | |
| 19:28:01 | tssurya | right I thought so :), just tried my luck :D | |
| 19:28:04 | dansmith | or use nova-manage | |
| 19:28:33 | tssurya | yep, thanks! | |
| 19:29:47 | dansmith | gdi, why does zuul hate me? | |
| 19:30:09 | mriedem | tssurya: so this should only be called if the client passes in a marker https://github.com/openstack/nova/blob/f902e0d5d87fb05207e4a7aca73d185775d43df2/nova/compute/instance_list.py#L56-L73 | |
| 19:30:27 | mriedem | which when i ran openstack --debug service list, i saw it was passing a marker, but i wasn't | |
| 19:30:33 | tssurya | mriedem: yes | |
| 19:30:38 | dansmith | oh, it's python3 that hates me | |
| 19:30:38 | mriedem | so is there something in novaclient that always gets a marker? | |
| 19:30:47 | tssurya | oh no, we pass a marker | |
| 19:31:04 | mriedem | we? | |
| 19:31:20 | tssurya | I mean a marker is passed only if we give it via the CLI | |
| 19:31:50 | tssurya | what you are saying seems strange, but I haven't tested that | |
| 19:32:00 | tssurya | tested/observed | |
| 19:32:13 | melwitt | does getting MarkerNotFound mean that a marker had to have been passed? because the bug reporter was using servers.list() python API and got MarkerNotFound | |
| 19:32:46 | mriedem | right, and when i was doing openstack server list, i wasn't passing a marker, | |
| 19:32:48 | tssurya | melwitt: that's what I thought so too, | |
| 19:32:58 | mriedem | but i ran it with debug and saw it was doing GET /servers/ with a marker | |
| 19:33:00 | melwitt | that would mean that somewhere in servers.list() novaclient is automatically passing a marker. I guess it would if it were paging on its own | |
| 19:33:20 | mriedem | novaclient does do that | |
| 19:33:41 | mriedem | until it doesn't get any more results | |
| 19:33:44 | melwitt | yeah, that's what I'm referring to, the fact that it does that if the list is > api limit | |
| 19:33:54 | mriedem | i only had 2 instances | |
| 19:33:59 | melwitt | oh | |
| 19:34:08 | mriedem | i think it will happen at any point if you have one of these messed up mappings | |
| 19:34:31 | tssurya | mriedem: hmm yea, true at some point even we have NULL cell_mappings | |
| 19:34:51 | mriedem | https://github.com/openstack/python-novaclient/blob/10.2.0/novaclient/v2/servers.py#L863 | |
| 19:34:58 | melwitt | I missed why/how that is normal | |
| 19:35:28 | melwitt | if an instance is in the middle of being scheduled, it shouldn't have an instance_mapping yet | |
| 19:36:22 | tssurya | melwitt: it will have an instance_mapping but with a NULL cell_mapping | |
| 19:36:34 | mriedem | we create the empty instance mapping in the api | |
| 19:36:49 | melwitt | oh, I see. so any instance in the middle of scheduling would have the null marker | |
| 19:37:05 | melwitt | didn't realize that | |
| 19:37:10 | mriedem | https://github.com/openstack/nova/blob/f902e0d5d87fb05207e4a7aca73d185775d43df2/nova/compute/api.py#L913 | |
| 19:37:16 | mriedem | an instance during scheduling will have a build request | |
| 19:37:30 | dansmith | right, and we page those out of BRs not out of the cell anyway | |
| 19:37:32 | mriedem | once we pick a host, we update the instance mapping for the cell and delete the build request | |
| 19:37:50 | mriedem | https://github.com/openstack/nova/blob/f902e0d5d87fb05207e4a7aca73d185775d43df2/nova/conductor/manager.py#L1236-L1239 | |
| 19:38:08 | melwitt | ah, thanks. I can never remember this | |
| 19:38:16 | tssurya | dansmith: yes which us why I was confused | |
| 19:38:19 | mriedem | i also want to say i remember a novaclient patch recently that changed the default limit/paging behavior when listing instances | |
| 19:38:19 | tssurya | is* | |
| 19:38:26 | mriedem | so maybe someone regressed something | |
| 19:38:53 | mriedem | https://review.openstack.org/#/c/534222/ | |
| 19:38:56 | openstackgerrit | Dan Smith proposed openstack/nova master: Change consecutive build failure limit to a weigher https://review.openstack.org/572195 | |
| 19:40:01 | mriedem | i guess i never commented on that, but i don't know why it was needed, | |
| 19:40:08 | mriedem | you can pass limit=-1 for no limit | |
| 19:40:58 | melwitt | yeah, the bug is saying the paging loop was ending prematurely for some reason | |
| 19:42:57 | mriedem | heh, "I'm not sure how this happened but I can't reproduce the problem now. It seems to work as expected with `--limit -1`." | |
| 19:43:12 | mriedem | from the author of the change, in the backport that was -1ed by me https://review.openstack.org/#/c/543968/ | |
| 19:43:22 | melwitt | guh | |
| 19:44:08 | mriedem | so before https://review.openstack.org/#/c/534222/ if you don't specify a limit, we'd get up to max_limit instances (default 1000) and then stop | |
| 19:44:22 | mriedem | now, we get up to max_limit instances, and then query again until we don't get any more results | |
| 19:44:43 | mriedem | which, if the marker for the 2nd query ends up being a busted instance mapping, nova list is perma broken | |
| 19:45:15 | mriedem | i kind of just want to revert that change... | |
| 19:45:48 | mriedem | if i create 10 instances and set CONF.api.max_limit to 5, and run nova list, i should get 5 instances, and if i run nova list --limit -1, i should get 10 | |
| 19:45:50 | melwitt | the old behavior wouldn't be able to return a non-existent marker? | |
| 19:46:18 | mriedem | if you have a broken instance mapping, we still have a problem on the server side, | |
| 19:46:31 | mriedem | but with what i was hitting when i hit this over the weekend, i couldn't get around it | |
| 19:46:40 | mriedem | because the last instance in my list was the busted marker | |
| 19:46:51 | mriedem | and i think i only had 2 instances | |
| 19:47:03 | tssurya | regarding the broken instance_mapping, we could just check if its broken and print a LOG or something | |
| 19:47:21 | mriedem | i don't even think i was able to delete the server w/o getting into the database | |
| 19:48:31 | mriedem | tssurya: we could....but that will reset your page which is not what the user expects either | |
| 19:48:40 | mriedem | really it should probably be a 500 | |
| 19:49:13 | tssurya | mriedem: hmm, maybe | |
| 19:49:27 | tssurya | yea since its broken | |
| 19:50:47 | melwitt | mriedem: oh, weird, I thought the default without passing --limit would have been -1 | |
| 19:51:00 | melwitt | (old behavior) | |
| 19:57:50 | openstackgerrit | Matt Riedemann proposed openstack/python-novaclient master: Revert "Fix listing of instances above API max_limit" https://review.openstack.org/572539 | |
| 19:59:49 | mgagne | mriedem: am I right in assuming the allocations healer requires cellsv2 ? | |
| 20:00:47 | mriedem | yes | |