| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-10 | |||
| 08:18:27 | openstackgerrit | melanie witt proposed openstack/nova: Make Quotas object favor the API database https://review.openstack.org/410945 | |
| 08:19:33 | openstackgerrit | Huan Xie proposed openstack/nova: XenAPI: Fix vif plug problem during VM rescue/unrescue https://review.openstack.org/413469 | |
| 08:24:17 | openstackgerrit | Zhenyu Zheng proposed openstack/nova: Support tag instances when boot https://review.openstack.org/394321 | |
| 08:30:16 | openstackgerrit | Béla Vancsics proposed openstack/nova: Transform instance.trigger_crash_dump notification https://review.openstack.org/396225 | |
| 08:31:58 | openstackgerrit | Matt Riedemann proposed openstack/python-novaclient: Fix functional tests to deal with multiple networks https://review.openstack.org/418190 | |
| 08:40:18 | openstackgerrit | Zhenyu Zheng proposed openstack/nova: Add sort_key white list for server list/detail https://review.openstack.org/415142 | |
| 08:40:56 | openstackgerrit | Zhenyu Zheng proposed openstack/nova: Add new policy for server list/detail with all_tenants https://review.openstack.org/415330 | |
| 08:44:59 | openstackgerrit | Alex Xu proposed openstack/nova: [POC][placement] Add Trait and TraitList objects https://review.openstack.org/376199 | |
| 08:47:52 | openstackgerrit | Zhenyu Zheng proposed openstack/nova: Check destination_type when booting with bdm provided https://review.openstack.org/402372 | |
| 08:48:08 | openstackgerrit | Zhenyu Zheng proposed openstack/nova: Don't delete neutron port when attach failed https://review.openstack.org/403568 | |
| 08:51:21 | alex_xu | Kevin_Zheng: ping | |
| 08:51:38 | alex_xu | Kevin_Zheng: I guess you miss the comment in the main page of https://review.openstack.org/#/c/408571/ | |
| 08:52:01 | alex_xu | Kevin_Zheng: "we should make the strip extra properties method to support patternProperties https://github.com/openstack/nova/blob/master/nova/api/validation/__init__.py#L113" | |
| 08:53:46 | Kevin_Zheng | let me check | |
| 08:58:18 | Kevin_Zheng | what do you mean by this? | |
| 08:58:34 | Kevin_Zheng | OK, I just got it | |
| 08:58:56 | Kevin_Zheng | How, do you have any suggestions? | |
| 08:59:19 | alex_xu | Kevin_Zheng: even patternProperties enable the parameter prefix with '_', but it is still stripped out | |
| 08:59:57 | alex_xu | Kevin_Zheng: we shouldn't strip the parameters which match the regex in the patterProperties, so we need to support that at https://github.com/openstack/nova/blob/master/nova/api/validation/__init__.py#L113 | |
| 09:17:52 | andreykurilin | mriedem: hi! I wonder if you found an issue yesturday. Did you?) | |
| 14:28:47 | eglynn | bearing in mind that the placement service landed after newton-3, IIRC | |
| 14:28:52 | eglynn | (and was uncertain up to the last, IIRC) | |
| 14:28:58 | eglynn | meaning that it would have been well-nigh impossible for deployment tooling to include support for it in newton-derived distros | |
| 14:29:00 | mriedem | eglynn: we were running CI on it well before then | |
| 14:29:33 | bauzas | mriedem: you know that we had problems with us communicating the placement API | |
| 14:29:38 | bauzas | mriedem: in the relnotes | |
| 14:29:41 | eglynn | deployment tooling can't start supporting stuff that still effectively WIP until after feature freeze | |
| 14:30:01 | bauzas | mriedem: mostly because our prelude section wasn't included in the 14.0.0 release but in a dot release | |
| 14:30:09 | bauzas | (thanks to reno) | |
| 14:30:29 | bauzas | so, most people deploying Newton hadn't paid much attention to that fresh new service | |
| 14:31:20 | cdent | So what are the realistic options from the deployment tooling standpoint eglynn ? | |
| 14:31:34 | cdent | (as in, how do we move forward, from your perspective) | |
| 14:32:40 | mriedem | we had everything in place by FF | |
| 14:32:50 | eglynn | cdent: 1. support a fallback to the old resource tracker data in ocata, then roll forward to based scheduling decisions on the placement API in first minor point release after ocata | |
| 14:33:21 | mriedem | we can't do that in a stable point release | |
| 14:33:23 | eglynn | cdent: 2. revert the scheduler cut-over patch downstream-only in ocata GA, then roll forward as in option #1 | |
| 14:33:26 | cdent | how then to _force_ people to run placement, as that's the opposite side of the concern, as far as I can tell | |
| 14:34:10 | mriedem | cdent: i guess the argument is it's still optional in ocata, with warnings and other communication that it's going to be required in pike, for realz | |
| 14:34:25 | mriedem | at which point everyone will have given up on placement and gone to work on k8s | |
| 14:34:26 | mriedem | :) | |
| 14:34:37 | cdent | I've seen that quote before somewhere. | |
| 14:35:48 | bauzas | meh, we should just mention the issue page for k8s, period. | |
| 14:35:53 | eglynn | cdent: in option #1, I would envisage the placement API being rolled out, but not depended on, in the ocata GA ... the forcing function would be the post-ocata roll-forward to the new scheduler decision making in the post-ocata release | |
| 14:36:23 | cdent | An important thing to remember here is that the use of the placement api in ocata is just filtering the available nodes, it is not "scheduling decisions". | |
| 14:36:33 | cdent | That's a perhaps subtle but important distinction | |
| 14:37:10 | cdent | the filer sheduler will now filter 10 instead of 1000 nodes (or whatever) | |
| 14:37:10 | eglynn | cdent: fair enough ... put another way, the scheduler logic without which placement decisions become impossible | |
| 14:37:16 | bauzas | cdent: the confusion I could see would be, some deployers restart their computes for some reason, then they begin reporting their inventories | |
| 14:37:32 | bauzas | we would only provide destinations based on a very small subset | |
| 14:38:02 | bauzas | cdent: yeah, that's the problem : what if only 10 nodes are registered by having their inventories ? | |
| 14:38:31 | bauzas | surely the upgrade status script would canary fail the situation, but that's something happening somewhere during the upgrade | |
| 14:39:01 | mriedem | eglynn: if you're comfortable with that cutover in an ocata point release (which i'm not) how is it you're not comfortable with rolling out placement support in your product in a newton point release? | |
| 14:39:52 | mriedem | anyway, i'm going to be late for a job interview so i need to head out of here, as fun as this is, i'll have to pick it up in the ML later | |
| 14:40:31 | eglynn | mriedem: problem is orchestrating the major version upgrade (newton->ocata) in such a way as to include the pre-upgrade step or running placement API, restarting computes, waiting for placement to warm-up before proceeding in the normal way with the rest of the upgrade | |
| 14:40:37 | dansmith | I wish this conversation wasn't happening now, | |
| 14:40:45 | dansmith | since I have to leave in just a few minutes | |
| 14:40:59 | eglynn | ... the pre-upgrade step *of running placement API ... | |
| 14:41:00 | dansmith | but newton/ocata was the optional/required seam, | |
| 14:41:22 | dansmith | and while it could have been messaged better, I think we'll have the same set of surprised people if we make the seam ocata/pike | |
| 14:41:40 | dansmith | doing it in a point release is not reasonable (at all) IMHO, so we just kick this can down the road one more if we do that | |
| 14:41:58 | edleafe | dansmith: +1 | |
| 14:42:11 | dansmith | if that's what we have to do, then it's what we have to do, but I think it just delays realizing scheduler gains and I think we'll have the same argument in +6mo | |
| 14:42:15 | dansmith | that's MHO | |
| 14:43:16 | andreykurilin | mriedem: did you resolve an issue with novaclient's gates? :) | |
| 14:44:06 | cdent | presumably one option, not entirely friendly but existing, is we can just put the seam in the upstream where it needs to be and the distros can cope in whatever fashion they want? | |
| 14:44:21 | diana_clarke | andreykurilin: The last comment from Kevin appears to solve the mystery: https://review.openstack.org/#/c/418190/ | |
| 14:44:25 | cdent | I state this not to say I'm in favor of it or anything, but just to list it as one of the options. | |
| 14:45:39 | dansmith | cdent: it's a big seam... people will unpatch it if they have to of course | |
| 14:45:53 | dansmith | cdent: the other thing to remember, is that if you haven't setup placement before the upgrade, | |
| 14:46:06 | eglynn | dansmith: IMO the main problem here is that there's an emergent aspect of the cut-over (the requirement to deploy placement and restart computes in *advance* of doing the main upgrade) | |
| 14:46:08 | dansmith | then you really just have a few extra minutes of downtime | |
| 14:46:17 | andreykurilin | diana_clarke: lol. my patch to requirements fixed an issue (I tried to fix another one) | |
| 14:46:22 | cdent | dansmith: Yeah, just trying to enumerate the several facets of reality because without all the facets its hard to see. | |
| 14:46:26 | dansmith | while placement fills and if you just keep your maintenance window open to fill that, then it's not as big of a deployment issue | |
| 14:46:46 | dansmith | eglynn: that's only if you really need to minimize downtime | |
| 14:46:48 | cdent | dansmith: yeah, that's what confuses me, I'm still unsure why this is a big deal, since we're sort of self headling | |
| 14:47:17 | dansmith | eglynn: if you don't, and I assert anyone running, say, tripleo does not obsess over 120 seconds of downtime, then you don't need to pre-deploy it | |
| 14:47:29 | eglynn | the "sort of" in "sort of self healing" makes me very nervous :) | |
| 14:47:37 | dansmith | cdent: yeah | |
| 14:47:48 | edleafe | cdent: any single node is ~1 minute to self-heal; a huge deployment may take much longer | |
| 14:47:49 | dansmith | eglynn: we're exactly as self-healing with placement as we are with resource-tracker | |
| 14:47:51 | dansmith | more so actually | |
| 14:47:59 | cdent | eglynn: heh. I don't like the term self healing in any context, really. It's more: the info is updated every 60s | |
| 14:48:04 | cdent | which results in good info | |
| 14:48:10 | cdent | so it isn't that a failure was healed | |
| 14:48:37 | andreykurilin | mriedem: ok, it looks like I lied (unwittingly) you yesterday and I knew the root of novaclient's failures :)) | |
| 14:49:10 | eglynn | dansmith: the 120s of downtime would just be the inability to schedule new instances while placement get warmed up with reported inventories & allocations? | |
| 14:49:42 | dansmith | eglynn: yep, and you could just keep your api down to prevent failed schedules if you want | |
| 14:49:43 | eglynn | (assuming all the compute were SIGHUP'd before the "real" rolling upgrade of computes kicks off) | |
| 14:49:45 | cdent | (when bauzas's pending code is done, we can test this) | |
| 14:50:35 | openstackgerrit | Gábor Antal proposed openstack/nova: Transform instance.volume_attach notification https://review.openstack.org/401992 | |
| 14:50:48 | dansmith | eglynn: not sure what you mean about SIGHUP | |
| 14:51:44 | dansmith | you should be able to roll ocata services in any order and have them start reporting whenever the placement service becomes available, | |
| 14:51:48 | dansmith | no SIGHUP required, AFAIK | |
| 14:51:49 | eglynn | dansmith: bauzas reckoned the nova-compute services would have to restarted (even if still possibly based on Newton code) in order to re-trigger their reporting into the placement API | |
| 14:52:01 | dansmith | I think we dropped the try-once thing, but if not we could certainly tweak that | |
| 14:52:25 | eglynn | dansmith: a-ha, k that would simplify | |
| 14:52:36 | dansmith | eglynn: newton ones because they try not to obsess if the catalog was missing the placement endpoint I think | |
| 14:52:37 | bauzas | dansmith: the safe_connect decorator prevents us to retry | |
| 14:52:49 | dansmith | eglynn: but *that* is something we could tweak in a point release for newton, | |