| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-06-05 | |||
| 19:42:57 | mriedem | heh, "I'm not sure how this happened but I can't reproduce the problem now. It seems to work as expected with `--limit -1`." | |
| 19:43:12 | mriedem | from the author of the change, in the backport that was -1ed by me https://review.openstack.org/#/c/543968/ | |
| 19:43:22 | melwitt | guh | |
| 19:44:08 | mriedem | so before https://review.openstack.org/#/c/534222/ if you don't specify a limit, we'd get up to max_limit instances (default 1000) and then stop | |
| 19:44:22 | mriedem | now, we get up to max_limit instances, and then query again until we don't get any more results | |
| 19:44:43 | mriedem | which, if the marker for the 2nd query ends up being a busted instance mapping, nova list is perma broken | |
| 19:45:15 | mriedem | i kind of just want to revert that change... | |
| 19:45:48 | mriedem | if i create 10 instances and set CONF.api.max_limit to 5, and run nova list, i should get 5 instances, and if i run nova list --limit -1, i should get 10 | |
| 19:45:50 | melwitt | the old behavior wouldn't be able to return a non-existent marker? | |
| 19:46:18 | mriedem | if you have a broken instance mapping, we still have a problem on the server side, | |
| 19:46:31 | mriedem | but with what i was hitting when i hit this over the weekend, i couldn't get around it | |
| 19:46:40 | mriedem | because the last instance in my list was the busted marker | |
| 19:46:51 | mriedem | and i think i only had 2 instances | |
| 19:47:03 | tssurya | regarding the broken instance_mapping, we could just check if its broken and print a LOG or something | |
| 19:47:21 | mriedem | i don't even think i was able to delete the server w/o getting into the database | |
| 19:48:31 | mriedem | tssurya: we could....but that will reset your page which is not what the user expects either | |
| 19:48:40 | mriedem | really it should probably be a 500 | |
| 19:49:13 | tssurya | mriedem: hmm, maybe | |
| 19:49:27 | tssurya | yea since its broken | |
| 19:50:47 | melwitt | mriedem: oh, weird, I thought the default without passing --limit would have been -1 | |
| 19:51:00 | melwitt | (old behavior) | |
| 19:57:50 | openstackgerrit | Matt Riedemann proposed openstack/python-novaclient master: Revert "Fix listing of instances above API max_limit" https://review.openstack.org/572539 | |
| 19:59:49 | mgagne | mriedem: am I right in assuming the allocations healer requires cellsv2 ? | |
| 20:00:47 | mriedem | yes | |
| 20:01:11 | mriedem | it relies on the cell mappings in the api db to find the instances in each cell | |
| 20:01:29 | mgagne | ok, just trying to figure out how I will be able to test without updating the whole planet or doing the actual work ;) | |
| 20:01:52 | mgagne | like, can I just import our prod database and run migrations and then the script for example. | |
| 20:02:16 | mgagne | (with some "minor" details to be addressed) | |
| 20:02:16 | mriedem | mgagne: you mean a backup test copy of your prod db? | |
| 20:03:10 | mriedem | ? | |
| 20:03:10 | mriedem | mgagne: what do you have for cell mappings right now? just cell0> | |
| 20:03:16 | mgagne | kind of. import somewhere, run release db migration to queens. then migrate somehow to cellsv2 (since it's required?) and run the healer with a dummy placement api. | |
| 20:03:27 | mgagne | nothing, I'm running mitaka. | |
| 20:04:11 | mgagne | trying to find a simpler way to get to queens with prod data | |
| 20:04:38 | mgagne | if cellsv2 is required, I will just find a way to make it happen. | |
| 20:05:12 | mriedem | mgagne: have you read through https://docs.openstack.org/nova/latest/user/cells.html#upgrade-with-cells-v1 ? | |
| 20:05:36 | mriedem | the cell mappings are going to be 1:1 with your child ells | |
| 20:05:38 | mriedem | *cells | |
| 20:05:44 | mriedem | plus cell0 | |
| 20:05:53 | mgagne | kind of but never went through it yet. I will read and see how far I can go. | |
| 20:06:08 | mriedem | you'll have num(child_cell) + 1 cell mappings in the api db | |
| 20:06:11 | mgagne | my main concern is with online data migration and maybe nova-compute exporting resources to placement ? | |
| 20:06:33 | mriedem | nova-compute will start reporting inventory and allocation information to placement in ocata | |
| 20:06:48 | mriedem | if you're going straight to queens, nova-compute will not be reporting allocation information to placement, the scheduler does that | |
| 20:06:50 | mgagne | right so I need to run the actual service at least once | |
| 20:07:05 | mriedem | in queens, nova-compute will report inventory to placement on startup | |
| 20:07:34 | mriedem | in queens, the allocations will not be automatically posted for existing instances, which is why you need the heal_allocations CLI | |
| 20:08:05 | mriedem | but before you can run that, you need (1) the cell mappings created for each of your cells (2) the instances mapped to the cells | |
| 20:08:26 | mgagne | yes, will create mapping | |
| 20:08:39 | mgagne | my concern is with resources being exported by compute node before. right? | |
| 20:09:00 | mriedem | i'm not sure what that means | |
| 20:09:12 | mriedem | the cell mappings and placement stuff are unrelated | |
| 20:09:14 | mgagne | I mean, you need providers to be created somehow first? | |
| 20:09:32 | mgagne | I'm not that familiar with placement terminology btw | |
| 20:09:37 | mriedem | yes, you'll need the compute nodes reporting into placement before heal_allocations can run | |
| 20:10:24 | mriedem | heal allocations will iterate the cell mappings looking for instances that have a host but don't have allocations in placement, and then attempt to post allocations to plcaement for the given instance and compute node - so yeah you need the compute nodes reporting as providers into placement first | |
| 20:10:48 | mgagne | ok, that's what I wanted to understand. got it | |
| 20:13:04 | mriedem | btw, i think you can still do all of the cells v2 stuff (cell mappings, host mappings, instance mappings) and placement stuff in queens with the caching scheduler still in place - we have an experimental ci job that does that | |
| 20:13:29 | mriedem | well, we have a cells v1 job that does that too | |
| 20:14:17 | mgagne | yea, that's my plan. getting to queens, import database and run the tool and see how it goes | |
| 20:14:23 | mriedem | i think once you get all of the allocations healed, you can drop the caching scheduler and disable cells v1 | |
| 20:15:24 | mriedem | would be ideal if you could even kick the tires with a single child cell test env to get used to this before doing it | |
| 20:17:32 | mgagne | yea. I guess so. I need to test with Ironic too... but I thought I needed cells v2 ? I'm not sure what you mean by "drop the caching scheduler and disable cells v1" | |
| 20:26:04 | mriedem | mgagne: the goal is to not using the caching scheduler or cells v1 since both are deprecated | |
| 20:26:25 | mriedem | you need cells v2 since ocata yes | |
| 20:26:36 | mgagne | right | |
| 20:26:44 | mriedem | the IronicHostManager is also deprecated since pike | |
| 20:27:15 | mriedem | placement + custom resource classes removes the need for the ironic host manager in the scheduler and the Exact* filters | |
| 20:27:23 | mriedem | but that's another thing you can deal with separately before rocky | |
| 20:28:19 | mriedem | melwitt: this is the last day for https://review.openstack.org/#/q/topic:bp/granular-placement-policy+status:open in the runway, has all +2s, could use another core to go through it | |
| 20:28:35 | mgagne | right. I'm trying to get in a state where I can test your patch with production data without spending 900 man hours to get there. I think with your help I got enough info to kick start the process. | |
| 20:30:18 | mgagne | 1) Create test environment with Nova Mitaka+CellsV1 2) Import production database 3) Upgrade to Nova Queens, with FFU or whatever process I will find. 4) Migrate to CellsV2 5) Test allocation healer 6) Profit | |
| 20:33:34 | mriedem | mgagne: yeah, 3 and 4 are going to get fuzzy | |
| 20:33:45 | mgagne | can't wait =) | |
| 20:33:46 | mriedem | there is a blocker migration in ocata that won't let you finish the api db sync until you have cell0 for example | |
| 20:34:01 | mriedem | i just mean, 3 and 4 are kind of co-dependent | |
| 20:34:05 | mgagne | good to know, will add that step | |
| 20:34:21 | mriedem | once you get to ocata+ you can also run the nova-status upgrade check CLI | |
| 20:34:23 | mgagne | ok, so ocata will require some steps for cellsv2 | |
| 20:34:24 | mriedem | to check your progress | |
| 20:34:27 | mriedem | yup | |
| 20:34:38 | mgagne | cool, can't wait to test that tool too | |
| 20:43:09 | melwitt | mriedem: ack | |
| 20:44:00 | mriedem | i'm +2 on the vmware live migration spec if another specs core wants to put it in https://review.openstack.org/#/c/299207/ - i'm ok with holding up the merge of the actual code until the multinode vmware 3rd party ci job shows up | |
| 20:44:53 | melwitt | that reminds me of the zvm spec amendment, still not approved | |
| 20:45:05 | melwitt | but that's not subject to spec freeze | |
| 20:45:15 | melwitt | I don't think, anyway | |
| 20:57:28 | mriedem | nova list --limit -1 works like a charm http://paste.openstack.org/show/722757/ | |
| 21:01:59 | melwitt | mriedem: kinda wonder why that's not the default. I guess so you have to opt into paging | |
| 21:07:27 | mriedem | frickler: are you happy with this now? https://review.openstack.org/#/c/312626/ | |
| 21:07:33 | mriedem | i see you've done quite a bit of review on it already | |
| 21:08:04 | mriedem | melwitt: idk, for a smaller cloud with <1000 instances it's not going to be a problem, | |
| 21:08:15 | mriedem | for a public cloud the admin probably wants to opt into getting back a million records | |
| 21:08:23 | melwitt | yeah, good point | |
| 21:12:19 | mriedem | anyway, that patch made around ~400 LOC change for something that wasn't even broken | |
| 21:12:35 | mriedem | no release note on the behavior change either | |
| 21:13:21 | melwitt | those are the best | |
| 21:24:05 | melwitt | mriedem: I proposed adding the nova-lvm job to pike (as experimental) and it passed (phew) https://review.openstack.org/572130 | |
| 21:24:36 | melwitt | the ocata one however is busted, but on a cinder problem AFAICT, not the nova regression/fix https://review.openstack.org/572132 | |
| 21:29:47 | mriedem | ok | |