| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-11-26 | |||
| 15:52:16 | mriedem | and i know you have a spec for filtering that way as well | |
| 15:52:39 | cdent | I abandoned that spec because people weren't sure that was a sufficient use case, which confused me | |
| 15:52:55 | mriedem | heh, well we have a use case right here | |
| 15:53:41 | mriedem | nova-status upgrade check already hits the placement REST API to check the version in _check_placement | |
| 15:53:45 | mriedem | anywhere | |
| 15:53:47 | mriedem | *anyway | |
| 15:54:01 | mriedem | we might want to undo https://review.openstack.org/#/c/617941/21/nova/tests/unit/cmd/test_status.py in a follow up | |
| 15:54:20 | mriedem | unless we remove that upgrade check, but that's another discussion | |
| 15:54:55 | cdent | well, we'll have to at least change the upgrade check, in which case the test will have to change so it does rather go together | |
| 15:55:11 | mriedem | sure, the test has to exist to change it though | |
| 15:55:14 | mriedem | unless you're agreeing with me | |
| 15:55:39 | cdent | I'm mostly agreeing with you, except for the part about it being another discussion | |
| 15:55:49 | cdent | if we're going to remove the command, then job done | |
| 15:56:24 | mriedem | it depends on what nova-compute does on startup - if it can't create a resource provider b/c placement isn't running or nova-compute isn't configured to talk to it, then we can probably remove the check | |
| 15:56:44 | mriedem | otherwise it's useful as a base install verification that you've got computes reporting in and resource providers for those computes | |
| 15:57:46 | mriedem | FFU makes me nervous about when we can remove these things... | |
| 15:57:57 | mriedem | b/c anyone can FFU from any release to another presumably | |
| 15:58:32 | mriedem | having said that, i'm not sure these would work for FFU'ers anyway b/c the placement api would likely be down | |
| 15:58:53 | mriedem | dansmith: is that correct? we can expect placement-api to be down during an FFU? | |
| 16:00:24 | sean-k-mooney | cdent: placement does not randomise allocation candiates by defualt correct. they are just retruned in db order? | |
| 16:00:48 | mriedem | https://docs.openstack.org/nova/latest/configuration/config.html#placement.randomize_allocation_candidates | |
| 16:02:53 | sean-k-mooney | mriedem: thanks ya its off by default. i was wondering if that could be related to this ml list post http://lists.openstack.org/pipermail/openstack-discuss/2018-November/000209.html | |
| 16:03:17 | sean-k-mooney | that said i woudl have expected the default weigher to kick in and spread the instances | |
| 16:04:08 | cdent | mriedem: yeah, I'm wondering if coupling nova-status to placement status is a good/safe idea? If they are supposed to be independently upgraded (for some value of "independent" maybe placement-status should do some kind of check? Except that we expect placement to upgrade first (usually). /me throws hands | |
| 16:05:02 | cdent | sean-k-mooney: that config item was left as not random by default so as to encoureage/allow pack, what would the weigher be doing? | |
| 16:05:15 | cdent | changing the config is certainly something they could at least try | |
| 16:05:32 | sean-k-mooney | cdent: the weighers used to spread by default | |
| 16:05:33 | cdent | my impression was that most of the system was biased towards packing to save $$ | |
| 16:05:46 | sean-k-mooney | cdent: i think they still do | |
| 16:06:46 | cdent | probably still worth trying the config setting just to see? | |
| 16:06:51 | mriedem | cdent: i left a note to self / open question in the check code so i can maybe come back to it some other time when it comes up, or is a more pressing decision that needs to be made | |
| 16:07:09 | sean-k-mooney | ya i was also gong to ask them to provide the nova.conf the schduler is useing not the compute node one | |
| 16:07:23 | cdent | good point sean-k-mooney | |
| 16:07:27 | cdent | mriedem: ✔ | |
| 16:07:55 | dansmith | mriedem: not just expect, but require | |
| 16:09:59 | mriedem | ok yeah then in that case, nova-status upgrade check will not work during FFU today | |
| 16:10:14 | mriedem | it will fail the check placement check since the API won't be up | |
| 16:10:48 | dansmith | I thought nova-status was to be db-only anyway? I guess not once placement is a different db, eh? | |
| 16:11:01 | mriedem | there is a check for the minimum placement version | |
| 16:11:10 | mriedem | otherwise it looks in the nova_api db yeah | |
| 16:11:31 | dansmith | so just have to make the api-bound check graceful I suppose | |
| 16:11:34 | mriedem | was discussing if we should change that to hit the placement API now, or just remove the placement-related checks | |
| 16:12:04 | mriedem | the upgrade checkers things were written before FFU was really a consideration i think | |
| 16:12:09 | mriedem | so i haven't put a ton of thought into it | |
| 16:12:46 | dansmith | well, I thought the plan was for that thing to be db-only anyway, so it wouldn't matter, but the scope has definitely gotten larger as we find more things for it to do | |
| 16:12:47 | mriedem | e.g. if we start saying that nova requires cinder >= 3.44 to remove our api compat code, presumably we'd want an upgrade check for that | |
| 16:12:56 | mriedem | yeah | |
| 16:13:26 | dansmith | well, if the api is up and you're not doing an ffu, it'd be nice to get that check, you just need to be super graceful I think | |
| 16:13:45 | dansmith | maybe a new category, peer to warning, error, for "manual check" or "unknown" ? | |
| 16:13:57 | dansmith | like "I would check this for you, but I can't so be sure you check it" | |
| 16:13:58 | mriedem | i was thinking it'd just be a warning | |
| 16:14:04 | mriedem | warning is already "this might be a problem, but i'm not sure" | |
| 16:14:13 | dansmith | do the deployment things hork on warning? | |
| 16:14:15 | dansmith | if not, then cool | |
| 16:14:31 | mriedem | do the deployment things run the upgrade checkers... heh | |
| 16:14:33 | mriedem | osa runs it | |
| 16:14:36 | mriedem | i don't think tripleo does | |
| 16:14:45 | dansmith | owalsh_: ? | |
| 16:14:54 | mriedem | btw, i ask this at least once every 2 weeks in here :) | |
| 16:15:08 | dansmith | you said you didn't know :) | |
| 16:15:21 | mriedem | rhetorical | |
| 16:15:28 | mriedem | i know tripleo doesn't run it | |
| 16:15:32 | dansmith | okay | |
| 16:15:53 | mriedem | http://codesearch.openstack.org/?q=nova-status&i=nope&files=&repos= | |
| 16:15:59 | mriedem | grenade, kolla-ansible and osa | |
| 16:16:27 | mriedem | so i was going to say, | |
| 16:16:48 | mriedem | unless/until someone comes along saying, "hey this doesn't work for me during FFU" i don't have much motivation to care about making it graceful in those cases | |
| 16:17:04 | dansmith | :/ | |
| 16:17:19 | dansmith | does it stack trace now? | |
| 16:17:28 | mriedem | if placement is down? | |
| 16:17:31 | mriedem | it will return a failure | |
| 16:17:33 | dansmith | yeah | |
| 16:17:43 | dansmith | oh you mean it just reports error instead of your proposed warning? | |
| 16:17:50 | mriedem | https://github.com/openstack/nova/blob/master/nova/cmd/status.py#L230 | |
| 16:17:52 | mriedem | yes | |
| 16:18:15 | mriedem | there is the guantlet of ksa exceptions, | |
| 16:18:22 | mriedem | so without testing it in devstack i'm not exactly sure | |
| 16:18:23 | dansmith | okay | |
| 16:18:45 | dansmith | well, if it just reports error that's reasonable enough I think | |
| 16:19:22 | mriedem | that likely means any tooling running it during an FFU should ignore the results, which makes me wonder why even run it during FFU | |
| 16:19:57 | mriedem | but yeah, i just don't have the energy to figure out what our official process/stance is for upgrade checkers during FFU :) | |
| 16:20:03 | mriedem | especially that now it's a community wide goal | |
| 16:20:11 | mriedem | and FFU is still very nebulous to me | |
| 16:20:26 | dansmith | well, | |
| 16:20:37 | dansmith | the checking of things that have to be done before moving on is pretty critical to FFU | |
| 16:21:00 | mriedem | my most basic understanding is take all control plane services down, and roll through each release running data migrations and schema migrations | |
| 16:21:06 | dansmith | I think everyone right now is just doing it very manually, including the deployment projects | |
| 16:21:45 | dansmith | lyarwood might be a good person to ask about this | |
| 16:22:30 | dansmith | lyarwood: mschuppert: the question is why tripleo doesn't run nova-status during any upgrade, including ffu, even if just to collect/log the status | |
| 16:23:26 | dansmith | and/or I guess: if/do tripleo people use it whilst trying to get a particular N->M transition working, and then just not run it programmatically for everyone, assuming they have the steps perfected? | |
| 16:23:39 | lyarwood | dansmith / mriedem ; no reason, I did push an example up for the upgrades team a while ago and asked them to take it forward but I assume they just didn't follow up | |
| 16:24:17 | lyarwood | this came up again at PTG, didn't we create tripleo bugs to track this during S? | |
| 16:24:58 | lyarwood | https://bugs.launchpad.net/tripleo/+bug/1777060 | |
| 16:24:58 | openstack | Launchpad bug 1777060 in tripleo "nova-status should be used during deployment and upgrades" [High,New] - Assigned to Lee Yarwood (lyarwood) | |
| 16:25:34 | dansmith | cool | |
| 16:28:07 | mriedem | until we actually have any kind of FFU ci testing it's also hard for me to care a ton about stuff like this | |
| 16:28:22 | mriedem | i mean, i don't want to lose sleep over it | |
| 16:28:30 | mriedem | when i have so many other things i can lose sleep over | |
| 16:29:54 | dansmith | we could easily just run it during grenade before we bring things back up and log the output right? | |
| 16:30:23 | mriedem | we do run nova-status upgrade check during grenade | |