| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-26 | |||
| 13:19:36 | cdent | bauzas: Is there only one at this point? | |
| 13:20:13 | bauzas | cdent: yup, I fixed the functional issue | |
| 13:20:22 | bauzas | cdent: now, we still have the grenade problem | |
| 13:20:35 | bauzas | that's the blocking issue | |
| 13:20:41 | cdent | did mriedem option #4 not work (I haven't had a chance to check yet)? | |
| 13:20:59 | bauzas | which option, sorry? | |
| 13:21:40 | bauzas | oh, you mean the email ? | |
| 13:22:30 | cdent | yeah | |
| 13:23:53 | bauzas | well, #4 is not working yet | |
| 13:24:32 | bauzas | but I agree with mriedem about trying to have #4 and then if not working, doing #3 | |
| 13:24:42 | bauzas | but we need to have a consensus before | |
| 13:25:24 | cdent | I think 4 is the best choice | |
| 13:25:30 | cdent | (assuming we can make it work) | |
| 13:26:48 | bauzas | well, tbh, I hate the forced_host field | |
| 13:26:51 | bauzas | cdent: ^ | |
| 13:26:58 | bauzas | because it's silly | |
| 13:27:20 | bauzas | rather I prefer to use the requested_destination like we did for live-migrating and evacuate, ie. calling all the filters | |
| 13:27:28 | bauzas | but... | |
| 13:27:32 | johnthetubaguy | bauzas: so that option #3, I have a slightly different version that could help? | |
| 13:27:43 | cdent | Yeah, I understand there's quite a lot of history on force_host | |
| 13:27:49 | cdent | (social history) | |
| 13:27:55 | bauzas | if we don't find a solution for #4 today, then the scheduler calling the placement couldn't be merged by Ocata | |
| 13:28:25 | johnthetubaguy | bauzas: if we check for the min_compute service version, and fallback to [] if everything has been upgraded, and the full list of old computenodes if everything hasn't yet been upgraded? | |
| 13:28:29 | bauzas | johnthetubaguy: the fact is that we verify the forced field *after* getting the list of nodes | |
| 13:28:43 | bauzas | which is weirdo | |
| 13:28:57 | bauzas | because if you want to force to a destination, you don't care about others | |
| 13:29:01 | johnthetubaguy | oh, so the problem is different to what I am thinking it is, I thought it was just placement not being present | |
| 13:29:13 | bauzas | johnthetubaguy: nope, we fixed that by my grenade job | |
| 13:29:25 | johnthetubaguy | OK, so this is a different error now? | |
| 13:29:28 | johnthetubaguy | I missed that | |
| 13:29:33 | openstackgerrit | Andrey Kurilin proposed openstack/python-novaclient: [cli] Use environment variable for token https://review.openstack.org/425647 | |
| 13:29:40 | bauzas | johnthetubaguy: yup, see mriedem's email from this early morning | |
| 13:29:54 | bauzas | johnthetubaguy: we have a tempest test that's blocking us for a very specific behaviour | |
| 13:30:12 | bauzas | I'll actually respond to mriedem | |
| 13:30:26 | johnthetubaguy | yeah, I was reading that and it seemed to say different | |
| 13:30:29 | bauzas | because I think the current forced_hosts thing is already weirdo | |
| 13:30:52 | cdent | could/should we reorder when force_host happens? | |
| 13:31:09 | johnthetubaguy | so its a test thats confirm we can build on "both" nodes I guess? | |
| 13:31:11 | bauzas | cdent: or just accepting [] for forced_hosts that's it :) | |
| 13:31:14 | johnthetubaguy | is thats whats failing here? | |
| 13:31:32 | bauzas | johnthetubaguy: it's failing because it forces to a destination using the --az hack that I *HATE* | |
| 13:32:07 | bauzas | johnthetubaguy: given the subnode is not yet providing its resources to placement, the placement call returns [] which fails later when looking at forced_hosts | |
| 13:32:25 | sfinucan | johnthetubaguy, raj_singh: could you enlighten me as to the purpose of these three lines https://github.com/openstack/nova/blob/master/nova/conf/neutron.py#L116-L118 | |
| 13:32:30 | johnthetubaguy | right, so thats a bigger issue though, we would want to the subnode to be providing its resources already, I assume? | |
| 13:32:39 | bauzas | johnthetubaguy: that's what we ask | |
| 13:32:58 | sfinucan | I'm guessing they refer to the various version of the neutron API and we only need one, but that's a guess | |
| 13:33:00 | sfinucan | https://github.com/openstack/watcher/blob/695ddf8ae7a2dccff4c62a28821888b7d49e2338/watcher/conf/_opts.py#L58 | |
| 13:33:06 | bauzas | johnthetubaguy: in theory, you have to configure your computes to talk to the placement service even if they're Newton | |
| 13:33:26 | bauzas | or you'd not get them when calling the placement | |
| 13:33:37 | johnthetubaguy | sfinucan: its what is done for neutron, it describes all the setting you can have for the password keystoneauth settings (i.e register_auth_conf_options), AFAIK | |
| 13:36:24 | cdent | bauzas: I thought the point of option 4 was that was causing the subnode to have placement in newton | |
| 13:36:35 | bauzas | cdent: yup | |
| 13:37:05 | cdent | so will that fix the force_host test? | |
| 13:38:27 | johnthetubaguy | cdent: thats what I was assuming too, feel like I am missing a bit | |
| 13:38:57 | cdent | johnthetubaguy: yeah, exactly, I've not a clear hi-res picture of the situation | |
| 13:38:58 | johnthetubaguy | unless the newton compute nodes are not providing placement with enough information, because of the new APIs we added this cycle? | |
| 13:39:32 | johnthetubaguy | certainly feel like the is a big piece I am missing here | |
| 13:39:37 | johnthetubaguy | there | |
| 13:40:50 | openstackgerrit | Stephen Finucane proposed openstack/nova: Use a service account to make vendordata requests. https://review.openstack.org/415597 | |
| 13:40:51 | openstackgerrit | Stephen Finucane proposed openstack/nova: Cache boot time roles for vendordata. https://review.openstack.org/417780 | |
| 13:40:51 | openstackgerrit | Stephen Finucane proposed openstack/nova: Optionally make dynamic vendordata failures fatal. https://review.openstack.org/416391 | |
| 13:41:57 | sfinucan | johnthetubaguy: Appears those v2password, v3password lines refer to (keystone?) plugins https://github.com/openstack/keystoneauth/blob/master/keystoneauth1/loading/conf.py#L46 | |
| 13:42:13 | johnthetubaguy | sfinucan: yeah, the auth config is pluggable | |
| 13:42:14 | sfinucan | Blindly copied and pasted and ran genconfig without issue | |
| 13:42:21 | sfinucan | ...so I think we're good | |
| 13:42:34 | johnthetubaguy | sfinucan: its consistent now, which I guess is good | |
| 13:42:58 | sfinucan | So long as it's consistently correct :) | |
| 13:43:38 | johnthetubaguy | I am OK with consistently wrong, over in-consistently wrong, but maybe I am in a funny mood | |
| 13:45:41 | cdent | bauzas, johnthetubaguy so looking at that grenade run, it appears that placement was not configured on the old side. Was the change in option 4 supposed to cause that? | |
| 13:46:56 | bauzas | cdent: I'm writing an email | |
| 13:46:59 | bauzas | cdent: you'll see my thoughts | |
| 13:47:00 | cdent | k | |
| 13:47:15 | bauzas | after thinking more of that, I think we hit the bigger problem that johnthetubaguy said | |
| 13:47:33 | johnthetubaguy | I think its time for option 3 | |
| 13:47:35 | bauzas | which is that we need to ask operators to modify their compute configuration before upgrading | |
| 13:47:58 | johnthetubaguy | but if we limit it only for during an upgrade, that feels better than raw option 3, like option 3.11 or something | |
| 13:48:03 | bauzas | johnthetubaguy: could you please reply to my email I'm sending in 2 mins ? | |
| 13:48:25 | johnthetubaguy | bauzas: sure | |
| 13:49:26 | cdent | I wish I understood why it is hard to set the config on the subnode | |
| 13:50:33 | bauzas | johnthetubaguy: cdent: so I just sent my email | |
| 13:50:46 | bauzas | tl;dr: I think we have 2 problems, one being larger than other | |
| 13:51:17 | bauzas | the fact that if you force a destination it fails is to me a corner case that can easily be tackled if we make the scheduler more defensive | |
| 13:51:25 | bauzas | but | |
| 13:52:12 | bauzas | it raises the main issue that it requires nova.conf to be updated for all your computes before you upgrade to Ocata, and that's a larger action item than the one we stated in the upgrade process http://docs.openstack.org/developer/nova/upgrade.html#rolling-upgrade-process | |
| 13:52:26 | bauzas | where we only ask for modifying the upgrade_levels flag | |
| 13:52:51 | bauzas | and even that | |
| 13:53:30 | bauzas | cdent: johnthetubaguywe all supposed that a rolling upgrade means that you can only focus on upgrading your controller without touching the computes (unless just upgrading them to the latest stable release) | |
| 13:53:34 | bauzas | johnthetubaguy: $ | |
| 13:53:46 | bauzas | snap, I'm wearing gloves | |
| 13:53:55 | bauzas | that's cold here, but not a reason | |
| 13:54:58 | cdent | We've known for a few months that there will be config change on the compute nodes, it's part of why we made the upgrade ready script, isn't it? | |
| 13:56:23 | bauzas | cdent: tbh, I never identified that it would require a nova.conf change on old computes | |
| 13:57:01 | edleafe | bauzas: what change is required? | |
| 13:57:03 | cdent | we've said for a long time that newton computes will need to run placement before the ocata upgrade | |
| 13:57:20 | cdent | in order ot set inventory | |
| 13:57:33 | cdent | and in order to set inventory you have to have the [placement] section in nova.conf | |
| 13:57:55 | cdent | edleafe: it contains the auth bits to do placement service discovery | |
| 13:57:56 | bauzas | edleafe: what cdent just said | |
| 13:58:48 | bauzas | cdent: are we sure we are going to ask operators to modify all their computes nova.conf before upgrading ? | |
| 13:58:55 | edleafe | cdent: but isn't that what "have to run the placement service" would entail? | |