| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-19 | |||
| 15:06:20 | mriedem | the question is what if the minimum available isn't new enough for what you need? | |
| 15:06:21 | bauzas | but I'd prefer to have our reportclient having that | |
| 15:07:05 | mriedem | now that i'm thinking about this, don't people usually upgrade the api last? | |
| 15:07:05 | bauzas | mriedem: I think the safe_connect deco should say sorry then | |
| 15:07:21 | mriedem | yeah but what does that mean for the filter scheduler? | |
| 15:07:25 | openstackgerrit | Sam Betts proposed openstack/nova: Change order of _cleanup_deploy and _unprovision in Ironic virt https://review.openstack.org/422678 | |
| 15:07:57 | bauzas | mriedem: it would mean that you'd have to run the Ocata placement service before upgrading nova-scheduler | |
| 15:08:10 | bauzas | EmilienM: ^ | |
| 15:09:00 | EmilienM | bauzas: that kind of orchestration is not easy to do, specially when you deploy in production | |
| 15:09:09 | bauzas | I know | |
| 15:09:40 | mriedem | bauzas: but that also means upgrading nova-api to ocata doesn't it? unless you package them as separate services | |
| 15:10:52 | mriedem | do i need to start a ML thread to think this through? | |
| 15:10:53 | bauzas | mriedem: that indeed depends on how it is packed | |
| 15:10:59 | bauzas | mriedem: that looks important | |
| 15:11:09 | bauzas | and we're 1 week away from FF | |
| 15:11:13 | bauzas | which makes me nervous | |
| 15:11:39 | mriedem | well let me start a thread first and we can stew in there | |
| 15:12:33 | mriedem | johnthetubaguy: keep me honest about the order of upgrade, with n-api coming last | |
| 15:12:55 | mriedem | it seems this changes every time i talk about it, might be n-cond -> n-api -> n-cpus | |
| 15:13:05 | bauzas | in general yes | |
| 15:13:26 | mriedem | http://docs.openstack.org/developer/nova/upgrade.html | |
| 15:13:32 | mriedem | " It is safest to start nova-conductor first and nova-api last." | |
| 15:13:35 | bauzas | we did the modification with dan's change so that newton computes auto-heal once they find a placement service | |
| 15:13:47 | mriedem | yeah i'm not worried about n-cpu | |
| 15:13:59 | bauzas | but we haven't tell'm what version of placement they should expect | |
| 15:15:30 | johnthetubaguy | mriedem: so it coming last is kind of a new thing | |
| 15:15:40 | johnthetubaguy | mriedem: we used to do it first, then n-cpu last | |
| 15:16:40 | johnthetubaguy | mriedem: so I should be clearer, it used to say turn off all non n-cpu, turn on the new ones, then restart the new n-cpu processes | |
| 15:17:02 | mriedem | but we've never been clear on n-api or n-sch having an order, | |
| 15:17:04 | mriedem | now it would | |
| 15:17:09 | mriedem | depending on how things are packaged | |
| 15:17:13 | mriedem | although, | |
| 15:17:26 | johnthetubaguy | yeah, I think before we were meaning, everything off, then everything back on | |
| 15:17:36 | mriedem | you could roll the new code, you just don't restart n-api until the end of the control services | |
| 15:17:42 | mriedem | you could still restart placement | |
| 15:18:15 | johnthetubaguy | oh, I see what you mean now | |
| 15:18:27 | johnthetubaguy | if you run placement on the same as n-api | |
| 15:18:32 | johnthetubaguy | its going to be really late | |
| 15:18:33 | bauzas | of course | |
| 15:18:58 | bauzas | but that still means we have an hard dependency on the restart conditions between placement and scheduler | |
| 15:19:13 | bauzas | you have to have the placement service be upgraded before n-sch | |
| 15:19:30 | johnthetubaguy | oh, scheduler hard depends on a new API microversion? | |
| 15:19:35 | bauzas | yup | |
| 15:19:40 | johnthetubaguy | dang | |
| 15:19:48 | jaypipes | johnthetubaguy: yep, absolutely. | |
| 15:19:53 | bauzas | because the resources field is a 1.4 microversion | |
| 15:20:16 | johnthetubaguy | its almost like it would be easier if its considered a separate project, that gets upgrade before nova | |
| 15:20:22 | bauzas | johnthetubaguy: jaypipes: proof here http://logs.openstack.org/61/417961/10/check/gate-tempest-dsvm-cells-ubuntu-xenial/8a974f7/logs/screen-n-sch.txt.gz?level=TRACE#_2017-01-19_14_17_38_980 | |
| 15:20:34 | bauzas | johnthetubaguy: too soon IMO | |
| 15:20:56 | bauzas | we're already having most of our operators trailing with deploying a placement service for ocata | |
| 15:21:01 | johnthetubaguy | bauzas: probably | |
| 15:21:14 | bauzas | we can't expect them to package and manage a totally separate service right now | |
| 15:21:25 | bauzas | even for pike | |
| 15:21:35 | johnthetubaguy | feels like we should really support N and N-1 placement | |
| 15:21:36 | edleafe | johnthetubaguy: separate project, eh? Now there's a novel approach! | |
| 15:21:39 | edleafe | :) | |
| 15:21:58 | bauzas | honestly, I don't want to go that way | |
| 15:22:07 | bauzas | we already claim to support N-1 services | |
| 15:22:13 | mriedem | "we can't expect them to package and manage a totally separate service right now" ? | |
| 15:22:15 | mriedem | really? | |
| 15:22:21 | bauzas | so we should support the placement service being n-1 | |
| 15:22:39 | mriedem | placement has been available since newton, it's already out there for the packaging before upgrading to ocata, | |
| 15:22:41 | mriedem | so i don't see a problem there | |
| 15:23:02 | bauzas | mriedem: you know that most of the time, people don't care unless we break them | |
| 15:23:16 | mriedem | then i guess they should care? | |
| 15:23:19 | mriedem | lest be broken? | |
| 15:23:26 | mriedem | like, | |
| 15:23:40 | mriedem | the ocata release notes are going to a big fat YOU NEED TO UPGRADE PLACEMENT BEFORE SCHEDULER | |
| 15:23:41 | johnthetubaguy | so there is a way you can fix up your release N, so you get a smooth roll to N+1 | |
| 15:23:47 | mriedem | if you don't read that and upgrade scheduler anyway, then sorry | |
| 15:23:54 | johnthetubaguy | thats the important bit | |
| 15:24:17 | bauzas | mriedem: correct me if I'm wrong but we always assume that we can deal with N-1 versions of our components | |
| 15:24:30 | johnthetubaguy | from the summit the majority still turn the old one off, and then turn the new one on | |
| 15:24:38 | bauzas | mriedem: if so, why couldn't we assume that the scheduler has to assume a n-1 placement service ? | |
| 15:24:44 | johnthetubaguy | bauzas: its more because we never defined an upgrade order | |
| 15:24:56 | johnthetubaguy | its certainly more usual to do that | |
| 15:25:02 | mriedem | n-1 for computes definitely | |
| 15:25:11 | mriedem | n-1 for scheduler probably, | |
| 15:25:16 | mriedem | because n-sch is pretty standalone | |
| 15:25:28 | mriedem | n-1 for n-api probably, because it's just new microversions | |
| 15:25:40 | mriedem | but n-cond needs to be first because it deals with, well, conducting | |
| 15:26:17 | jaypipes | bhagyashris: hey there. | |
| 15:26:17 | bauzas | mriedem: my thoughts are : | |
| 15:26:35 | bauzas | mriedem: getting a response from the placement service is an hard requirement | |
| 15:26:40 | bauzas | because we say it's mandatory | |
| 15:26:56 | bauzas | so, getting something like EndpointNotFound should be an hard stop | |
| 15:26:57 | bauzas | but | |
| 15:27:18 | bauzas | if we get something like a microversion not supported, I'd treat that like a soft fail | |
| 15:27:25 | bauzas | and do the old way | |
| 15:27:34 | mriedem | i just posted to the ML | |
| 15:27:36 | mriedem | i have to run to scrum | |
| 15:28:04 | mriedem | vladikr: i'd like to tease the unit tests out a bit in https://review.openstack.org/#/c/405954/ | |
| 15:28:44 | johnthetubaguy | bauzas: supporting N-1 feels like the normal thing to do, but isn't that a lot of extra code to maintain? | |
| 15:28:57 | bauzas | johnthetubaguy: not really | |
| 15:29:13 | bauzas | johnthetubaguy: we're getting the exceptions from keystone | |
| 15:29:48 | bauzas | if I'm getting a 400, that means I'm running an old microversion on the placement side | |
| 15:29:57 | vladikr | mriedem, sure, will add that, thanks | |
| 15:30:59 | johnthetubaguy | bauzas: its a bit tempting to see how bad it would be to support 1.0 and the latest | |
| 15:31:12 | bauzas | johnthetubaguy: in https://review.openstack.org/#/c/417961/10/nova/scheduler/host_manager.py@660, the or [] makes a problem with calling the placement service returning a novalidhosts | |
| 15:31:54 | bauzas | johnthetubaguy: I could just try to verify in the upcall whether I'm getting a 400 or something else and treat that differently | |