| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-06-05 | |||
| 19:30:38 | mriedem | so is there something in novaclient that always gets a marker? | |
| 19:30:38 | dansmith | oh, it's python3 that hates me | |
| 19:30:47 | tssurya | oh no, we pass a marker | |
| 19:31:04 | mriedem | we? | |
| 19:31:20 | tssurya | I mean a marker is passed only if we give it via the CLI | |
| 19:31:50 | tssurya | what you are saying seems strange, but I haven't tested that | |
| 19:32:00 | tssurya | tested/observed | |
| 19:32:13 | melwitt | does getting MarkerNotFound mean that a marker had to have been passed? because the bug reporter was using servers.list() python API and got MarkerNotFound | |
| 19:32:46 | mriedem | right, and when i was doing openstack server list, i wasn't passing a marker, | |
| 19:32:48 | tssurya | melwitt: that's what I thought so too, | |
| 19:32:58 | mriedem | but i ran it with debug and saw it was doing GET /servers/ with a marker | |
| 19:33:00 | melwitt | that would mean that somewhere in servers.list() novaclient is automatically passing a marker. I guess it would if it were paging on its own | |
| 19:33:20 | mriedem | novaclient does do that | |
| 19:33:41 | mriedem | until it doesn't get any more results | |
| 19:33:44 | melwitt | yeah, that's what I'm referring to, the fact that it does that if the list is > api limit | |
| 19:33:54 | mriedem | i only had 2 instances | |
| 19:33:59 | melwitt | oh | |
| 19:34:08 | mriedem | i think it will happen at any point if you have one of these messed up mappings | |
| 19:34:31 | tssurya | mriedem: hmm yea, true at some point even we have NULL cell_mappings | |
| 19:34:51 | mriedem | https://github.com/openstack/python-novaclient/blob/10.2.0/novaclient/v2/servers.py#L863 | |
| 19:34:58 | melwitt | I missed why/how that is normal | |
| 19:35:28 | melwitt | if an instance is in the middle of being scheduled, it shouldn't have an instance_mapping yet | |
| 19:36:22 | tssurya | melwitt: it will have an instance_mapping but with a NULL cell_mapping | |
| 19:36:34 | mriedem | we create the empty instance mapping in the api | |
| 19:36:49 | melwitt | oh, I see. so any instance in the middle of scheduling would have the null marker | |
| 19:37:05 | melwitt | didn't realize that | |
| 19:37:10 | mriedem | https://github.com/openstack/nova/blob/f902e0d5d87fb05207e4a7aca73d185775d43df2/nova/compute/api.py#L913 | |
| 19:37:16 | mriedem | an instance during scheduling will have a build request | |
| 19:37:30 | dansmith | right, and we page those out of BRs not out of the cell anyway | |
| 19:37:32 | mriedem | once we pick a host, we update the instance mapping for the cell and delete the build request | |
| 19:37:50 | mriedem | https://github.com/openstack/nova/blob/f902e0d5d87fb05207e4a7aca73d185775d43df2/nova/conductor/manager.py#L1236-L1239 | |
| 19:38:08 | melwitt | ah, thanks. I can never remember this | |
| 19:38:16 | tssurya | dansmith: yes which us why I was confused | |
| 19:38:19 | tssurya | is* | |
| 19:38:19 | mriedem | i also want to say i remember a novaclient patch recently that changed the default limit/paging behavior when listing instances | |
| 19:38:26 | mriedem | so maybe someone regressed something | |
| 19:38:53 | mriedem | https://review.openstack.org/#/c/534222/ | |
| 19:38:56 | openstackgerrit | Dan Smith proposed openstack/nova master: Change consecutive build failure limit to a weigher https://review.openstack.org/572195 | |
| 19:40:01 | mriedem | i guess i never commented on that, but i don't know why it was needed, | |
| 19:40:08 | mriedem | you can pass limit=-1 for no limit | |
| 19:40:58 | melwitt | yeah, the bug is saying the paging loop was ending prematurely for some reason | |
| 19:42:57 | mriedem | heh, "I'm not sure how this happened but I can't reproduce the problem now. It seems to work as expected with `--limit -1`." | |
| 19:43:12 | mriedem | from the author of the change, in the backport that was -1ed by me https://review.openstack.org/#/c/543968/ | |
| 19:43:22 | melwitt | guh | |
| 19:44:08 | mriedem | so before https://review.openstack.org/#/c/534222/ if you don't specify a limit, we'd get up to max_limit instances (default 1000) and then stop | |
| 19:44:22 | mriedem | now, we get up to max_limit instances, and then query again until we don't get any more results | |
| 19:44:43 | mriedem | which, if the marker for the 2nd query ends up being a busted instance mapping, nova list is perma broken | |
| 19:45:15 | mriedem | i kind of just want to revert that change... | |
| 19:45:48 | mriedem | if i create 10 instances and set CONF.api.max_limit to 5, and run nova list, i should get 5 instances, and if i run nova list --limit -1, i should get 10 | |
| 19:45:50 | melwitt | the old behavior wouldn't be able to return a non-existent marker? | |
| 19:46:18 | mriedem | if you have a broken instance mapping, we still have a problem on the server side, | |
| 19:46:31 | mriedem | but with what i was hitting when i hit this over the weekend, i couldn't get around it | |
| 19:46:40 | mriedem | because the last instance in my list was the busted marker | |
| 19:46:51 | mriedem | and i think i only had 2 instances | |
| 19:47:03 | tssurya | regarding the broken instance_mapping, we could just check if its broken and print a LOG or something | |
| 19:47:21 | mriedem | i don't even think i was able to delete the server w/o getting into the database | |
| 19:48:31 | mriedem | tssurya: we could....but that will reset your page which is not what the user expects either | |
| 19:48:40 | mriedem | really it should probably be a 500 | |
| 19:49:13 | tssurya | mriedem: hmm, maybe | |
| 19:49:27 | tssurya | yea since its broken | |
| 19:50:47 | melwitt | mriedem: oh, weird, I thought the default without passing --limit would have been -1 | |
| 19:51:00 | melwitt | (old behavior) | |
| 19:57:50 | openstackgerrit | Matt Riedemann proposed openstack/python-novaclient master: Revert "Fix listing of instances above API max_limit" https://review.openstack.org/572539 | |
| 19:59:49 | mgagne | mriedem: am I right in assuming the allocations healer requires cellsv2 ? | |
| 20:00:47 | mriedem | yes | |
| 20:01:11 | mriedem | it relies on the cell mappings in the api db to find the instances in each cell | |
| 20:01:29 | mgagne | ok, just trying to figure out how I will be able to test without updating the whole planet or doing the actual work ;) | |
| 20:01:52 | mgagne | like, can I just import our prod database and run migrations and then the script for example. | |
| 20:02:16 | mgagne | (with some "minor" details to be addressed) | |
| 20:02:16 | mriedem | mgagne: you mean a backup test copy of your prod db? | |
| 20:03:10 | mriedem | ? | |
| 20:03:10 | mriedem | mgagne: what do you have for cell mappings right now? just cell0> | |
| 20:03:16 | mgagne | kind of. import somewhere, run release db migration to queens. then migrate somehow to cellsv2 (since it's required?) and run the healer with a dummy placement api. | |
| 20:03:27 | mgagne | nothing, I'm running mitaka. | |
| 20:04:11 | mgagne | trying to find a simpler way to get to queens with prod data | |
| 20:04:38 | mgagne | if cellsv2 is required, I will just find a way to make it happen. | |
| 20:05:12 | mriedem | mgagne: have you read through https://docs.openstack.org/nova/latest/user/cells.html#upgrade-with-cells-v1 ? | |
| 20:05:36 | mriedem | the cell mappings are going to be 1:1 with your child ells | |
| 20:05:38 | mriedem | *cells | |
| 20:05:44 | mriedem | plus cell0 | |
| 20:05:53 | mgagne | kind of but never went through it yet. I will read and see how far I can go. | |
| 20:06:08 | mriedem | you'll have num(child_cell) + 1 cell mappings in the api db | |
| 20:06:11 | mgagne | my main concern is with online data migration and maybe nova-compute exporting resources to placement ? | |
| 20:06:33 | mriedem | nova-compute will start reporting inventory and allocation information to placement in ocata | |
| 20:06:48 | mriedem | if you're going straight to queens, nova-compute will not be reporting allocation information to placement, the scheduler does that | |
| 20:06:50 | mgagne | right so I need to run the actual service at least once | |
| 20:07:05 | mriedem | in queens, nova-compute will report inventory to placement on startup | |
| 20:07:34 | mriedem | in queens, the allocations will not be automatically posted for existing instances, which is why you need the heal_allocations CLI | |
| 20:08:05 | mriedem | but before you can run that, you need (1) the cell mappings created for each of your cells (2) the instances mapped to the cells | |
| 20:08:26 | mgagne | yes, will create mapping | |
| 20:08:39 | mgagne | my concern is with resources being exported by compute node before. right? | |
| 20:09:00 | mriedem | i'm not sure what that means | |
| 20:09:12 | mriedem | the cell mappings and placement stuff are unrelated | |
| 20:09:14 | mgagne | I mean, you need providers to be created somehow first? | |
| 20:09:32 | mgagne | I'm not that familiar with placement terminology btw | |
| 20:09:37 | mriedem | yes, you'll need the compute nodes reporting into placement before heal_allocations can run | |
| 20:10:24 | mriedem | heal allocations will iterate the cell mappings looking for instances that have a host but don't have allocations in placement, and then attempt to post allocations to plcaement for the given instance and compute node - so yeah you need the compute nodes reporting as providers into placement first | |
| 20:10:48 | mgagne | ok, that's what I wanted to understand. got it | |
| 20:13:04 | mriedem | btw, i think you can still do all of the cells v2 stuff (cell mappings, host mappings, instance mappings) and placement stuff in queens with the caching scheduler still in place - we have an experimental ci job that does that | |
| 20:13:29 | mriedem | well, we have a cells v1 job that does that too | |