| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-11-26 | |||
| 16:00:48 | mriedem | https://docs.openstack.org/nova/latest/configuration/config.html#placement.randomize_allocation_candidates | |
| 16:02:53 | sean-k-mooney | mriedem: thanks ya its off by default. i was wondering if that could be related to this ml list post http://lists.openstack.org/pipermail/openstack-discuss/2018-November/000209.html | |
| 16:03:17 | sean-k-mooney | that said i woudl have expected the default weigher to kick in and spread the instances | |
| 16:04:08 | cdent | mriedem: yeah, I'm wondering if coupling nova-status to placement status is a good/safe idea? If they are supposed to be independently upgraded (for some value of "independent" maybe placement-status should do some kind of check? Except that we expect placement to upgrade first (usually). /me throws hands | |
| 16:05:02 | cdent | sean-k-mooney: that config item was left as not random by default so as to encoureage/allow pack, what would the weigher be doing? | |
| 16:05:15 | cdent | changing the config is certainly something they could at least try | |
| 16:05:32 | sean-k-mooney | cdent: the weighers used to spread by default | |
| 16:05:33 | cdent | my impression was that most of the system was biased towards packing to save $$ | |
| 16:05:46 | sean-k-mooney | cdent: i think they still do | |
| 16:06:46 | cdent | probably still worth trying the config setting just to see? | |
| 16:06:51 | mriedem | cdent: i left a note to self / open question in the check code so i can maybe come back to it some other time when it comes up, or is a more pressing decision that needs to be made | |
| 16:07:09 | sean-k-mooney | ya i was also gong to ask them to provide the nova.conf the schduler is useing not the compute node one | |
| 16:07:23 | cdent | good point sean-k-mooney | |
| 16:07:27 | cdent | mriedem: ✔ | |
| 16:07:55 | dansmith | mriedem: not just expect, but require | |
| 16:09:59 | mriedem | ok yeah then in that case, nova-status upgrade check will not work during FFU today | |
| 16:10:14 | mriedem | it will fail the check placement check since the API won't be up | |
| 16:10:48 | dansmith | I thought nova-status was to be db-only anyway? I guess not once placement is a different db, eh? | |
| 16:11:01 | mriedem | there is a check for the minimum placement version | |
| 16:11:10 | mriedem | otherwise it looks in the nova_api db yeah | |
| 16:11:31 | dansmith | so just have to make the api-bound check graceful I suppose | |
| 16:11:34 | mriedem | was discussing if we should change that to hit the placement API now, or just remove the placement-related checks | |
| 16:12:04 | mriedem | the upgrade checkers things were written before FFU was really a consideration i think | |
| 16:12:09 | mriedem | so i haven't put a ton of thought into it | |
| 16:12:46 | dansmith | well, I thought the plan was for that thing to be db-only anyway, so it wouldn't matter, but the scope has definitely gotten larger as we find more things for it to do | |
| 16:12:47 | mriedem | e.g. if we start saying that nova requires cinder >= 3.44 to remove our api compat code, presumably we'd want an upgrade check for that | |
| 16:12:56 | mriedem | yeah | |
| 16:13:26 | dansmith | well, if the api is up and you're not doing an ffu, it'd be nice to get that check, you just need to be super graceful I think | |
| 16:13:45 | dansmith | maybe a new category, peer to warning, error, for "manual check" or "unknown" ? | |
| 16:13:57 | dansmith | like "I would check this for you, but I can't so be sure you check it" | |
| 16:13:58 | mriedem | i was thinking it'd just be a warning | |
| 16:14:04 | mriedem | warning is already "this might be a problem, but i'm not sure" | |
| 16:14:13 | dansmith | do the deployment things hork on warning? | |
| 16:14:15 | dansmith | if not, then cool | |
| 16:14:31 | mriedem | do the deployment things run the upgrade checkers... heh | |
| 16:14:33 | mriedem | osa runs it | |
| 16:14:36 | mriedem | i don't think tripleo does | |
| 16:14:45 | dansmith | owalsh_: ? | |
| 16:14:54 | mriedem | btw, i ask this at least once every 2 weeks in here :) | |
| 16:15:08 | dansmith | you said you didn't know :) | |
| 16:15:21 | mriedem | rhetorical | |
| 16:15:28 | mriedem | i know tripleo doesn't run it | |
| 16:15:32 | dansmith | okay | |
| 16:15:53 | mriedem | http://codesearch.openstack.org/?q=nova-status&i=nope&files=&repos= | |
| 16:15:59 | mriedem | grenade, kolla-ansible and osa | |
| 16:16:27 | mriedem | so i was going to say, | |
| 16:16:48 | mriedem | unless/until someone comes along saying, "hey this doesn't work for me during FFU" i don't have much motivation to care about making it graceful in those cases | |
| 16:17:04 | dansmith | :/ | |
| 16:17:19 | dansmith | does it stack trace now? | |
| 16:17:28 | mriedem | if placement is down? | |
| 16:17:31 | mriedem | it will return a failure | |
| 16:17:33 | dansmith | yeah | |
| 16:17:43 | dansmith | oh you mean it just reports error instead of your proposed warning? | |
| 16:17:50 | mriedem | https://github.com/openstack/nova/blob/master/nova/cmd/status.py#L230 | |
| 16:17:52 | mriedem | yes | |
| 16:18:15 | mriedem | there is the guantlet of ksa exceptions, | |
| 16:18:22 | mriedem | so without testing it in devstack i'm not exactly sure | |
| 16:18:23 | dansmith | okay | |
| 16:18:45 | dansmith | well, if it just reports error that's reasonable enough I think | |
| 16:19:22 | mriedem | that likely means any tooling running it during an FFU should ignore the results, which makes me wonder why even run it during FFU | |
| 16:19:57 | mriedem | but yeah, i just don't have the energy to figure out what our official process/stance is for upgrade checkers during FFU :) | |
| 16:20:03 | mriedem | especially that now it's a community wide goal | |
| 16:20:11 | mriedem | and FFU is still very nebulous to me | |
| 16:20:26 | dansmith | well, | |
| 16:20:37 | dansmith | the checking of things that have to be done before moving on is pretty critical to FFU | |
| 16:21:00 | mriedem | my most basic understanding is take all control plane services down, and roll through each release running data migrations and schema migrations | |
| 16:21:06 | dansmith | I think everyone right now is just doing it very manually, including the deployment projects | |
| 16:21:45 | dansmith | lyarwood might be a good person to ask about this | |
| 16:22:30 | dansmith | lyarwood: mschuppert: the question is why tripleo doesn't run nova-status during any upgrade, including ffu, even if just to collect/log the status | |
| 16:23:26 | dansmith | and/or I guess: if/do tripleo people use it whilst trying to get a particular N->M transition working, and then just not run it programmatically for everyone, assuming they have the steps perfected? | |
| 16:23:39 | lyarwood | dansmith / mriedem ; no reason, I did push an example up for the upgrades team a while ago and asked them to take it forward but I assume they just didn't follow up | |
| 16:24:17 | lyarwood | this came up again at PTG, didn't we create tripleo bugs to track this during S? | |
| 16:24:58 | lyarwood | https://bugs.launchpad.net/tripleo/+bug/1777060 | |
| 16:24:58 | openstack | Launchpad bug 1777060 in tripleo "nova-status should be used during deployment and upgrades" [High,New] - Assigned to Lee Yarwood (lyarwood) | |
| 16:25:34 | dansmith | cool | |
| 16:28:07 | mriedem | until we actually have any kind of FFU ci testing it's also hard for me to care a ton about stuff like this | |
| 16:28:22 | mriedem | i mean, i don't want to lose sleep over it | |
| 16:28:30 | mriedem | when i have so many other things i can lose sleep over | |
| 16:29:54 | dansmith | we could easily just run it during grenade before we bring things back up and log the output right? | |
| 16:30:23 | mriedem | we do run nova-status upgrade check during grenade | |
| 16:30:36 | dansmith | but not when *everything* is down right? | |
| 16:30:39 | dansmith | only during nova-upgrade? | |
| 16:30:56 | mriedem | http://git.openstack.org/cgit/openstack-dev/grenade/tree/projects/60_nova/upgrade.sh#n88 | |
| 16:31:02 | dansmith | right, | |
| 16:31:05 | mriedem | we specifically bring placement up before running the check | |
| 16:31:11 | dansmith | right | |
| 16:31:19 | dansmith | and other projects before us would be up (i.e. keystone) | |
| 16:31:42 | mriedem | yeah i mean i could run it before starting placement | |
| 16:31:43 | mriedem | and see it fail | |
| 16:31:49 | dansmith | actually verify preupgrade might run with nothing | |
| 16:32:01 | dansmith | sorry verify_noapi preupgrade | |
| 16:33:41 | dansmith | https://review.openstack.org/620104 | |
| 16:34:00 | dansmith | mriedem: anyway, don't lose sleep over it, I'll check in on that later to see how it goes | |
| 16:34:11 | mriedem | ack thanks | |
| 16:45:14 | openstackgerrit | Adam Spiers proposed openstack/nova-specs master: Add spec for libvirt driver launching AMD SEV-encrypted instances https://review.openstack.org/609779 | |
| 16:59:53 | mriedem | i guess we never documented anywhere officially that we only support n-1 computes.. | |
| 16:59:57 | mriedem | even though it comes up every so often | |
| 17:00:55 | sean-k-mooney | compute older then n-1 may work in some cases however we just dont test them | |
| 17:01:45 | mriedem | yes i know that. | |
| 17:01:59 | mriedem | what i'm asking is, didn't we ever document this because the last time it came up, i thought we said someone would document it. | |