| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-10 | |||
| 14:34:37 | cdent | I've seen that quote before somewhere. | |
| 14:35:48 | bauzas | meh, we should just mention the issue page for k8s, period. | |
| 14:35:53 | eglynn | cdent: in option #1, I would envisage the placement API being rolled out, but not depended on, in the ocata GA ... the forcing function would be the post-ocata roll-forward to the new scheduler decision making in the post-ocata release | |
| 14:36:23 | cdent | An important thing to remember here is that the use of the placement api in ocata is just filtering the available nodes, it is not "scheduling decisions". | |
| 14:36:33 | cdent | That's a perhaps subtle but important distinction | |
| 14:37:10 | eglynn | cdent: fair enough ... put another way, the scheduler logic without which placement decisions become impossible | |
| 14:37:10 | cdent | the filer sheduler will now filter 10 instead of 1000 nodes (or whatever) | |
| 14:37:16 | bauzas | cdent: the confusion I could see would be, some deployers restart their computes for some reason, then they begin reporting their inventories | |
| 14:37:32 | bauzas | we would only provide destinations based on a very small subset | |
| 14:38:02 | bauzas | cdent: yeah, that's the problem : what if only 10 nodes are registered by having their inventories ? | |
| 14:38:31 | bauzas | surely the upgrade status script would canary fail the situation, but that's something happening somewhere during the upgrade | |
| 14:39:01 | mriedem | eglynn: if you're comfortable with that cutover in an ocata point release (which i'm not) how is it you're not comfortable with rolling out placement support in your product in a newton point release? | |
| 14:39:52 | mriedem | anyway, i'm going to be late for a job interview so i need to head out of here, as fun as this is, i'll have to pick it up in the ML later | |
| 14:40:31 | eglynn | mriedem: problem is orchestrating the major version upgrade (newton->ocata) in such a way as to include the pre-upgrade step or running placement API, restarting computes, waiting for placement to warm-up before proceeding in the normal way with the rest of the upgrade | |
| 14:40:37 | dansmith | I wish this conversation wasn't happening now, | |
| 14:40:45 | dansmith | since I have to leave in just a few minutes | |
| 14:40:59 | eglynn | ... the pre-upgrade step *of running placement API ... | |
| 14:41:00 | dansmith | but newton/ocata was the optional/required seam, | |
| 14:41:22 | dansmith | and while it could have been messaged better, I think we'll have the same set of surprised people if we make the seam ocata/pike | |
| 14:41:40 | dansmith | doing it in a point release is not reasonable (at all) IMHO, so we just kick this can down the road one more if we do that | |
| 14:41:58 | edleafe | dansmith: +1 | |
| 14:42:11 | dansmith | if that's what we have to do, then it's what we have to do, but I think it just delays realizing scheduler gains and I think we'll have the same argument in +6mo | |
| 14:42:15 | dansmith | that's MHO | |
| 14:43:16 | andreykurilin | mriedem: did you resolve an issue with novaclient's gates? :) | |
| 14:44:06 | cdent | presumably one option, not entirely friendly but existing, is we can just put the seam in the upstream where it needs to be and the distros can cope in whatever fashion they want? | |
| 14:44:21 | diana_clarke | andreykurilin: The last comment from Kevin appears to solve the mystery: https://review.openstack.org/#/c/418190/ | |
| 14:44:25 | cdent | I state this not to say I'm in favor of it or anything, but just to list it as one of the options. | |
| 14:45:39 | dansmith | cdent: it's a big seam... people will unpatch it if they have to of course | |
| 14:45:53 | dansmith | cdent: the other thing to remember, is that if you haven't setup placement before the upgrade, | |
| 14:46:06 | eglynn | dansmith: IMO the main problem here is that there's an emergent aspect of the cut-over (the requirement to deploy placement and restart computes in *advance* of doing the main upgrade) | |
| 14:46:08 | dansmith | then you really just have a few extra minutes of downtime | |
| 14:46:17 | andreykurilin | diana_clarke: lol. my patch to requirements fixed an issue (I tried to fix another one) | |
| 14:46:22 | cdent | dansmith: Yeah, just trying to enumerate the several facets of reality because without all the facets its hard to see. | |
| 14:46:26 | dansmith | while placement fills and if you just keep your maintenance window open to fill that, then it's not as big of a deployment issue | |
| 14:46:46 | dansmith | eglynn: that's only if you really need to minimize downtime | |
| 14:46:48 | cdent | dansmith: yeah, that's what confuses me, I'm still unsure why this is a big deal, since we're sort of self headling | |
| 14:47:17 | dansmith | eglynn: if you don't, and I assert anyone running, say, tripleo does not obsess over 120 seconds of downtime, then you don't need to pre-deploy it | |
| 14:47:29 | eglynn | the "sort of" in "sort of self healing" makes me very nervous :) | |
| 14:47:37 | dansmith | cdent: yeah | |
| 14:47:48 | edleafe | cdent: any single node is ~1 minute to self-heal; a huge deployment may take much longer | |
| 14:47:49 | dansmith | eglynn: we're exactly as self-healing with placement as we are with resource-tracker | |
| 14:47:51 | dansmith | more so actually | |
| 14:47:59 | cdent | eglynn: heh. I don't like the term self healing in any context, really. It's more: the info is updated every 60s | |
| 14:48:04 | cdent | which results in good info | |
| 14:48:10 | cdent | so it isn't that a failure was healed | |
| 14:48:37 | andreykurilin | mriedem: ok, it looks like I lied (unwittingly) you yesterday and I knew the root of novaclient's failures :)) | |
| 14:49:10 | eglynn | dansmith: the 120s of downtime would just be the inability to schedule new instances while placement get warmed up with reported inventories & allocations? | |
| 14:49:42 | dansmith | eglynn: yep, and you could just keep your api down to prevent failed schedules if you want | |
| 14:49:43 | eglynn | (assuming all the compute were SIGHUP'd before the "real" rolling upgrade of computes kicks off) | |
| 14:49:45 | cdent | (when bauzas's pending code is done, we can test this) | |
| 14:50:35 | openstackgerrit | Gábor Antal proposed openstack/nova: Transform instance.volume_attach notification https://review.openstack.org/401992 | |
| 14:50:48 | dansmith | eglynn: not sure what you mean about SIGHUP | |
| 14:51:44 | dansmith | you should be able to roll ocata services in any order and have them start reporting whenever the placement service becomes available, | |
| 14:51:48 | dansmith | no SIGHUP required, AFAIK | |
| 14:51:49 | eglynn | dansmith: bauzas reckoned the nova-compute services would have to restarted (even if still possibly based on Newton code) in order to re-trigger their reporting into the placement API | |
| 14:52:01 | dansmith | I think we dropped the try-once thing, but if not we could certainly tweak that | |
| 14:52:25 | eglynn | dansmith: a-ha, k that would simplify | |
| 14:52:36 | dansmith | eglynn: newton ones because they try not to obsess if the catalog was missing the placement endpoint I think | |
| 14:52:37 | bauzas | dansmith: the safe_connect decorator prevents us to retry | |
| 14:52:49 | dansmith | eglynn: but *that* is something we could tweak in a point release for newton, | |
| 14:52:56 | dansmith | and if it's not gone from ocata, it needs to be, | |
| 14:53:08 | dansmith | so when you rolled those computes you'd be good once they're upgraded, regardless of the ordering | |
| 14:53:16 | bauzas | I mean https://github.com/openstack/nova/blob/180e6340a595ec047c59365465f36fed7a669ec3/nova/scheduler/client/report.py#L40-L67 | |
| 14:53:37 | dansmith | bauzas: yeah, we have to pull most of that out for ocata anyway | |
| 14:53:40 | dansmith | ocata nodes *should* keep obsessing | |
| 14:53:53 | bauzas | dansmith: but newton computes would still stop reporting ? | |
| 14:53:58 | bauzas | dansmith: I mean | |
| 14:54:11 | bauzas | dansmith: if we say we would run the placement service before upgrading | |
| 14:54:19 | bauzas | and then the computes would self-heal | |
| 14:54:34 | eglynn | dansmith: yeah, if we could get rid of the try-once on a point release for newton, and quantify the warm-up time / downtime to populate placement data, then we could have a path forward here | |
| 14:54:35 | bauzas | dansmith: that wouldn't work for newton computes because of that decorator, right? | |
| 14:55:07 | openstackgerrit | Gábor Antal proposed openstack/nova: Transform instance.volume_detach notification https://review.openstack.org/408676 | |
| 14:55:08 | eglynn | bauzas: IIUC dansmith was suggesting that the try-once logic is removed in a point release of newton | |
| 14:55:10 | bauzas | oh, if we remove that in a point release, then there is a path indeed | |
| 14:55:14 | eglynn | yep | |
| 14:55:44 | bauzas | but that would mean you couldn't upgrade from 14.0.0 straight to 15.0.1 | |
| 14:55:46 | bauzas | oops | |
| 14:55:49 | dansmith | I really have to go now | |
| 14:55:50 | bauzas | to 15.0.0 rather | |
| 14:55:57 | cdent | thanks dansmith | |
| 14:56:06 | bauzas | you'd have to apply the point release before upgrading to Ocata | |
| 14:56:20 | bauzas | that's a reasonable trade-off but that requires documenting | |
| 14:56:31 | bauzas | dansmith: go off, we can discuss that later | |
| 14:57:42 | cdent | bauzas: is that only need to apply the point release prior to upgrading if you were concerned about limiting any downtime for launching new instances as much as possible? Or always ned to apply the point release? | |
| 14:58:40 | bauzas | cdent: if you wanna do rolling upgrade of your computes, I guess the point release upgrade is a necessary prereq | |
| 14:59:54 | bauzas | cdent: because if you don't apply the point release, then once you upgrade your scheduler, it won't prevent you to pick any newton node | |
| 15:00:38 | cdent | "won't" or "will"? | |
| 15:00:49 | bauzas | oh oops will | |
| 15:01:05 | cdent | yeah, what I'm asking is: for some people, presumably that's not a problem | |
| 15:01:16 | cdent | new instances would only go on upgraded nodes | |
| 15:01:27 | cdent | existing instances would stay wherever they happened to be | |
| 15:01:41 | bauzas | not only new instances | |
| 15:01:42 | cdent | for a small number of seconds zero nodes would available | |
| 15:01:56 | bauzas | if you migrate, then good luck | |
| 15:02:18 | cdent | good luck why? | |
| 15:02:44 | bauzas | the number of seconds where you have zero nodes is the duration you take for upgrading your first compute to ocata | |
| 15:32:57 | markus_z | lyarwood_: That's probably closed, isn't it? https://bugs.launchpad.net/openstack-gate/+bug/1646127 | |
| 15:32:57 | openstack | Launchpad bug 1646127 in OpenStack-Gate "gate-grenade-dsvm-neutron-multinode-live-migration-nv failing on stable/newton due to devstack not supporting xenial during Mitaka" [Undecided,In progress] - Assigned to Lee Yarwood (lyarwood) | |
| 15:33:39 | lyarwood_ | markus_z: hey, yeah that should be closed now | |
| 15:34:29 | markus_z | Cool. I simply stumbled over it while looking for another thing. | |