| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-19 | |||
| 15:12:55 | mriedem | it seems this changes every time i talk about it, might be n-cond -> n-api -> n-cpus | |
| 15:13:05 | bauzas | in general yes | |
| 15:13:26 | mriedem | http://docs.openstack.org/developer/nova/upgrade.html | |
| 15:13:32 | mriedem | " It is safest to start nova-conductor first and nova-api last." | |
| 15:13:35 | bauzas | we did the modification with dan's change so that newton computes auto-heal once they find a placement service | |
| 15:13:47 | mriedem | yeah i'm not worried about n-cpu | |
| 15:13:59 | bauzas | but we haven't tell'm what version of placement they should expect | |
| 15:15:30 | johnthetubaguy | mriedem: so it coming last is kind of a new thing | |
| 15:15:40 | johnthetubaguy | mriedem: we used to do it first, then n-cpu last | |
| 15:16:40 | johnthetubaguy | mriedem: so I should be clearer, it used to say turn off all non n-cpu, turn on the new ones, then restart the new n-cpu processes | |
| 15:17:02 | mriedem | but we've never been clear on n-api or n-sch having an order, | |
| 15:17:04 | mriedem | now it would | |
| 15:17:09 | mriedem | depending on how things are packaged | |
| 15:17:13 | mriedem | although, | |
| 15:17:26 | johnthetubaguy | yeah, I think before we were meaning, everything off, then everything back on | |
| 15:17:36 | mriedem | you could roll the new code, you just don't restart n-api until the end of the control services | |
| 15:17:42 | mriedem | you could still restart placement | |
| 15:18:15 | johnthetubaguy | oh, I see what you mean now | |
| 15:18:27 | johnthetubaguy | if you run placement on the same as n-api | |
| 15:18:32 | johnthetubaguy | its going to be really late | |
| 15:18:33 | bauzas | of course | |
| 15:18:58 | bauzas | but that still means we have an hard dependency on the restart conditions between placement and scheduler | |
| 15:19:13 | bauzas | you have to have the placement service be upgraded before n-sch | |
| 15:19:30 | johnthetubaguy | oh, scheduler hard depends on a new API microversion? | |
| 15:19:35 | bauzas | yup | |
| 15:19:40 | johnthetubaguy | dang | |
| 15:19:48 | jaypipes | johnthetubaguy: yep, absolutely. | |
| 15:19:53 | bauzas | because the resources field is a 1.4 microversion | |
| 15:20:16 | johnthetubaguy | its almost like it would be easier if its considered a separate project, that gets upgrade before nova | |
| 15:20:22 | bauzas | johnthetubaguy: jaypipes: proof here http://logs.openstack.org/61/417961/10/check/gate-tempest-dsvm-cells-ubuntu-xenial/8a974f7/logs/screen-n-sch.txt.gz?level=TRACE#_2017-01-19_14_17_38_980 | |
| 15:20:34 | bauzas | johnthetubaguy: too soon IMO | |
| 15:20:56 | bauzas | we're already having most of our operators trailing with deploying a placement service for ocata | |
| 15:21:01 | johnthetubaguy | bauzas: probably | |
| 15:21:14 | bauzas | we can't expect them to package and manage a totally separate service right now | |
| 15:21:25 | bauzas | even for pike | |
| 15:21:35 | johnthetubaguy | feels like we should really support N and N-1 placement | |
| 15:21:36 | edleafe | johnthetubaguy: separate project, eh? Now there's a novel approach! | |
| 15:21:39 | edleafe | :) | |
| 15:21:58 | bauzas | honestly, I don't want to go that way | |
| 15:22:07 | bauzas | we already claim to support N-1 services | |
| 15:22:13 | mriedem | "we can't expect them to package and manage a totally separate service right now" ? | |
| 15:22:15 | mriedem | really? | |
| 15:22:21 | bauzas | so we should support the placement service being n-1 | |
| 15:22:39 | mriedem | placement has been available since newton, it's already out there for the packaging before upgrading to ocata, | |
| 15:22:41 | mriedem | so i don't see a problem there | |
| 15:23:02 | bauzas | mriedem: you know that most of the time, people don't care unless we break them | |
| 15:23:16 | mriedem | then i guess they should care? | |
| 15:23:19 | mriedem | lest be broken? | |
| 15:23:26 | mriedem | like, | |
| 15:23:40 | mriedem | the ocata release notes are going to a big fat YOU NEED TO UPGRADE PLACEMENT BEFORE SCHEDULER | |
| 15:23:41 | johnthetubaguy | so there is a way you can fix up your release N, so you get a smooth roll to N+1 | |
| 15:23:47 | mriedem | if you don't read that and upgrade scheduler anyway, then sorry | |
| 15:23:54 | johnthetubaguy | thats the important bit | |
| 15:24:17 | bauzas | mriedem: correct me if I'm wrong but we always assume that we can deal with N-1 versions of our components | |
| 15:24:30 | johnthetubaguy | from the summit the majority still turn the old one off, and then turn the new one on | |
| 15:24:38 | bauzas | mriedem: if so, why couldn't we assume that the scheduler has to assume a n-1 placement service ? | |
| 15:24:44 | johnthetubaguy | bauzas: its more because we never defined an upgrade order | |
| 15:24:56 | johnthetubaguy | its certainly more usual to do that | |
| 15:25:02 | mriedem | n-1 for computes definitely | |
| 15:25:11 | mriedem | n-1 for scheduler probably, | |
| 15:25:16 | mriedem | because n-sch is pretty standalone | |
| 15:25:28 | mriedem | n-1 for n-api probably, because it's just new microversions | |
| 15:25:40 | mriedem | but n-cond needs to be first because it deals with, well, conducting | |
| 15:26:17 | jaypipes | bhagyashris: hey there. | |
| 15:26:17 | bauzas | mriedem: my thoughts are : | |
| 15:26:35 | bauzas | mriedem: getting a response from the placement service is an hard requirement | |
| 15:26:40 | bauzas | because we say it's mandatory | |
| 15:26:56 | bauzas | so, getting something like EndpointNotFound should be an hard stop | |
| 15:26:57 | bauzas | but | |
| 15:27:18 | bauzas | if we get something like a microversion not supported, I'd treat that like a soft fail | |
| 15:27:25 | bauzas | and do the old way | |
| 15:27:34 | mriedem | i just posted to the ML | |
| 15:27:36 | mriedem | i have to run to scrum | |
| 15:28:04 | mriedem | vladikr: i'd like to tease the unit tests out a bit in https://review.openstack.org/#/c/405954/ | |
| 15:28:44 | johnthetubaguy | bauzas: supporting N-1 feels like the normal thing to do, but isn't that a lot of extra code to maintain? | |
| 15:28:57 | bauzas | johnthetubaguy: not really | |
| 15:29:13 | bauzas | johnthetubaguy: we're getting the exceptions from keystone | |
| 15:29:48 | bauzas | if I'm getting a 400, that means I'm running an old microversion on the placement side | |
| 15:29:57 | vladikr | mriedem, sure, will add that, thanks | |
| 15:30:59 | johnthetubaguy | bauzas: its a bit tempting to see how bad it would be to support 1.0 and the latest | |
| 15:31:12 | bauzas | johnthetubaguy: in https://review.openstack.org/#/c/417961/10/nova/scheduler/host_manager.py@660, the or [] makes a problem with calling the placement service returning a novalidhosts | |
| 15:31:54 | bauzas | johnthetubaguy: I could just try to verify in the upcall whether I'm getting a 400 or something else and treat that differently | |
| 15:32:23 | bauzas | that would mean that operators would necessary have to run the placement service in Ocata, but they could be allowed to run the newton release of it | |
| 15:33:04 | johnthetubaguy | bauzas: well, that have to update placement before the turn on any new n-cpu I guess? | |
| 15:33:09 | bhagyashris | jaypipes: I want to discuss few points regarding the resource provider: 1. As the palcement apis are called at the scheduler side using the patch [1] https://review.openstack.org/#/c/417961 which basically updates the allocations of the resources in placement databse . | |
| 15:33:42 | jaypipes | bhagyashris: gimme ten minutes to read back on IRC, k? :) | |
| 15:34:28 | jaypipes | bauzas: FYI, Fuel definitely isn't CD :) | |
| 15:34:33 | bauzas | johnthetubaguy: I think it would be acceptable to get back to calling the DB if we get a 400 for knowing the list of compute nodes | |
| 15:35:16 | bauzas | johnthetubaguy: but we could make a stronger contract, like checking the max n-cpu version | |
| 15:35:26 | bhagyashris | jaypipes: so is there need to enable the periodic task (update_available_resource()) at nova compute side? | |
| 15:35:30 | bauzas | johnthetubaguy: and stop to be gentle if we find an ocata n-cpu | |
| 15:36:10 | bhagyashris | jaypipes: ok. np. | |
| 15:36:15 | bauzas | that way, the upgrade path would be n-cond, <other services incl. sched>, n-cpu | |
| 15:36:16 | johnthetubaguy | bauzas: yeah, that full solution is the one that sounds like a lot of work, although it sounds more correct | |
| 15:36:27 | bauzas | I can try to draft it | |
| 15:39:20 | gibi | mriedem: hi! I cannot participate on the team meeting today but I updated the notification subteam part of the agenda with some highlights | |
| 15:44:19 | bauzas | mriedem: responded to your thread with a possibility | |
| 15:46:43 | openstackgerrit | Gábor Antal proposed openstack/nova: Transform instance.live_migration_rollback notification https://review.openstack.org/402124 | |
| 15:48:22 | mriedem | gibi: thanks | |
| 15:49:11 | mriedem | bhagyashris: update_available_resource is a required periodic task on the computes yes | |