| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-26 | |||
| 14:48:05 | dansmith | I'm pretty sure it does | |
| 14:48:09 | johnthetubaguy | so we might need one step worse... | |
| 14:48:19 | johnthetubaguy | the placement data will be incomplete | |
| 14:48:33 | johnthetubaguy | so we could use the crutch all the time, until the upgrade has completed | |
| 14:48:39 | dansmith | johnthetubaguy: right you'd have to favor the RT data until the time when you switch | |
| 14:48:43 | openstackgerrit | Sergey Nikitin proposed openstack/nova: Change parent of diagnostics objects to NovaObject class https://review.openstack.org/394480 | |
| 14:48:44 | openstackgerrit | Sergey Nikitin proposed openstack/nova: Standardization of VM diagnostics info API. https://review.openstack.org/355540 | |
| 14:48:46 | openstackgerrit | Sergey Nikitin proposed openstack/nova: Completed implementation of instance diagnostics for Xen https://review.openstack.org/399613 | |
| 14:49:02 | dansmith | johnthetubaguy: the downside is that it makes the switch to different data non-obvious, so people may not know why things are suddenly being different, | |
| 14:49:08 | dansmith | just because a lab monkey upgraded the last one | |
| 14:49:15 | johnthetubaguy | yeah | |
| 14:49:38 | bauzas | easy peasy question, can someone tell me the minimum service version for Ocata computes ? | |
| 14:49:57 | bauzas | we don't comment that in the SERVICE_VERSION_HISTORY dict | |
| 14:50:02 | dansmith | johnthetubaguy: we could log all-caps WARNING messages whilst using the crutch :) | |
| 14:50:04 | johnthetubaguy | bauzas: technically, you might want to bump that, so you know everyone has your patch | |
| 14:50:06 | dansmith | bauzas: yes, it's easy | |
| 14:50:18 | bauzas | johnthetubaguy: we don't need that | |
| 14:50:29 | cdent | okay, I think I'm caught up with you dansmith: the scheduler is the one that checks if all the compute nodes are up to date, and then makes a choice on what type of scheduling it does, yeah? | |
| 14:50:35 | bauzas | johnthetubaguy: all computes are already able to talk to placement since newton | |
| 14:50:49 | bauzas | johnthetubaguy: I just want to detect a fresh code | |
| 14:50:49 | dansmith | bauzas: https://github.com/openstack/nova/blob/stable/newton/nova/objects/service.py#L92 | |
| 14:50:59 | dansmith | cdent: yes | |
| 14:50:59 | cdent | bauzas: but only if they have the config... | |
| 14:51:12 | bauzas | cdent: but we say they need to upgrade to ocata, right? | |
| 14:51:27 | johnthetubaguy | so does a compute node die if it doesn't have placement configured? | |
| 14:51:29 | bauzas | dansmith: okay, so I need to set a new version | |
| 14:51:34 | bauzas | johnthetubaguy: nope | |
| 14:51:38 | dansmith | will ocata computes fail to start if the placement config is not done? | |
| 14:51:39 | cdent | bauzas: yes, but earlier you were complaining that making people change their configs to upgrade to ocata was in violation? | |
| 14:51:52 | dansmith | bauzas: I dunno what a new version would signal, TBH | |
| 14:52:21 | snikitin | sfinucan: Hi! thank you for comments! I updated https://review.openstack.org/#/c/399613 | |
| 14:52:38 | dansmith | bauzas: if they don't die, then maybe we should make them die and make that a new service version, and then check for that version in the scheduler | |
| 14:52:41 | bauzas | dansmith: good question about nova computes refusing to start | |
| 14:52:51 | bauzas | dansmith: I agree with that plan | |
| 14:52:54 | johnthetubaguy | cdent: its not as bad, because the old compute could have had that same config, but its sure borderline | |
| 14:52:57 | dansmith | I think it's what johnthetubaguy was saying too | |
| 14:52:59 | bauzas | dansmith: that's the move defensive one | |
| 14:53:12 | johnthetubaguy | yeah, make it obvious things are bad | |
| 14:53:22 | snikitin | alex_xu: Hi Alex! I got two +2 on this patch https://review.openstack.org/#/c/394480/ can I get a +W, please? | |
| 14:53:41 | dansmith | johnthetubaguy: do you think it's too borderline? | |
| 14:53:49 | bauzas | but we said placement is mandatory for ocata, right? | |
| 14:54:02 | dansmith | johnthetubaguy: I feel like we've already agreed we're going to make this decision | |
| 14:54:03 | dansmith | right | |
| 14:54:03 | bauzas | so that means you have to access it when restarting your services | |
| 14:54:04 | dansmith | what bauzas said | |
| 14:54:05 | dansmith | and we're logging on newton | |
| 14:54:28 | johnthetubaguy | dansmith: I think I am OK because we get a massive gain, so we stick with it | |
| 14:54:31 | bauzas | here, we're just trying to allow newton computes that are untouched to still be able to get instances on them | |
| 14:54:58 | dansmith | johnthetubaguy: but really, it was optional in newton, required in ocata, and this is just enforcing that no? | |
| 14:55:21 | bauzas | dansmith: to summarize, if I'm bumping the service version now, that would mean I'm sure the service is upgraded to ocata | |
| 14:55:49 | johnthetubaguy | dansmith: I think the key bit that makes me OK, is they just need to update the config on the old system before they do the upgrade, if they haven't done that | |
| 14:55:54 | mriedem | o/ | |
| 14:56:05 | dansmith | bauzas: you don't need to bump it now, but if you're going to make the computes refuse to start, I'd call that a service version.. otherwise you're adding a number that doesn't have any real meaning | |
| 14:56:06 | bauzas | now, the only question would be : should we silently soft-fail and do nothing with the risk for operators that they could loose capacity if they misconfigure ? | |
| 14:56:19 | johnthetubaguy | big fire, always the big fire | |
| 14:56:24 | dansmith | johnthetubaguy: right, and they could have done that in newton months ago if they wanted to be proactive | |
| 14:56:29 | bauzas | dansmith: the thing is that we haven't bumped the version in Ocata | |
| 14:56:36 | dansmith | bauzas: not at all? | |
| 14:56:36 | johnthetubaguy | dansmith: yeah, I think thats what makes me OK with it | |
| 14:56:52 | bauzas | dansmith: I'm still seeing version 15 in my local master branch | |
| 14:56:59 | dansmith | mriedem: read back to my "let me summarize" comment to cdent | |
| 14:57:03 | mriedem | what are we doing with a min service version check? | |
| 14:57:04 | dansmith | bauzas: then yes you need to add one | |
| 14:57:12 | johnthetubaguy | yeah, just add one | |
| 14:57:23 | bauzas | on it | |
| 14:57:23 | dansmith | mriedem: thinking we will add a fallback to RT data _if_ the computes are not all upgraded | |
| 14:57:27 | dansmith | mriedem: in the scheduler | |
| 14:57:38 | bauzas | and we hard-fail, as johnthetubaguy said | |
| 14:57:39 | dansmith | mriedem: which will let us get past grenade, and give the deployers a little longer grace period | |
| 14:58:15 | dansmith | mriedem: once they upgrade all their computes, scheduler will stop looking at RT data | |
| 14:58:29 | dansmith | mriedem: which means if they only change config during the upgrade they will still be fine | |
| 14:58:55 | dansmith | mriedem: whereas before, we had to make them configure their newton computes before starting the controller upgrade | |
| 14:59:25 | mriedem | we only fallback if we don't get any RPs *and* minimum compute isn't new enough right? | |
| 14:59:26 | sfinucan | claudiub|2: I looked at that and it seems fine, but I think one of Moshe's questions still stands | |
| 14:59:41 | dansmith | mriedem: no we have to always fallback if computes are old | |
| 14:59:54 | mriedem | so don't even check placement until then? | |
| 14:59:56 | johnthetubaguy | otherwise you get incomplete data, and the test we want to pass will still fail | |
| 14:59:57 | sfinucan | ...and I don't think I can approve until one of ye proves it's not a problem :) | |
| 15:00:02 | dansmith | mriedem: because otherwise after one compute reports RPs you'd hard switch and have a 1-compute cloud | |
| 15:00:46 | mriedem | johnthetubaguy: the test would pass if we would fallback to computenodelist.get_all if there were 0 RPs, but yes i understand we'd have a 1 compute cloud for awhile | |
| 15:00:57 | mriedem | fwiw we talked about that weeks ago and people thought that was ok and a small window | |
| 15:01:04 | johnthetubaguy | mriedem: I was thinking the force host thing, but yeah | |
| 15:01:18 | dansmith | mriedem: I think that was about waiting for the 60-second timer, | |
| 15:01:27 | dansmith | mriedem: but it could be weeks if you upgrade one compute as a test | |
| 15:01:43 | dansmith | mriedem: in this scenario | |
| 15:03:56 | mriedem | dansmith: was refering to this http://lists.openstack.org/pipermail/openstack-dev/2016-December/109060.html | |
| 15:04:16 | sfinucan | snikitin: Done and done | |
| 15:04:57 | mriedem | but whatever, that was before we had a blocking issue with grenade | |
| 15:04:58 | dansmith | mriedem: right but that was a thing that is triggered by doing the controller upgrade, right? this would be a thing triggered by the first compute upgrade | |
| 15:05:09 | sfinucan | claudiub|2: Call me a fool, but wasn't this kind of request what we were implementing qualitative resources for? https://review.openstack.org/#/c/420960 | |
| 15:06:19 | snikitin | sfinucan, thanks! | |
| 15:06:27 | dansmith | mriedem: meaning, I think before we were assuming 1. Configure computes for placement, 2. upgrade controllers, 3. wait 60s for data to filter before scheduling works properly | |
| 15:06:29 | sfinucan | johnthetubaguy: would it be too cheeky of me to +2 this, given that I addressed comments on it? https://review.openstack.org/#/c/415597 | |
| 15:06:32 | claudiub|2 | sfinucan: sure, but that is being used for the ImageMetaProps object. that still has to exist | |
| 15:06:40 | dansmith | mriedem: now, it would be 1. upgrade controllers, 2. upgrade one compute, 3. stuck until you upgrade more computes | |
| 15:07:49 | sfinucan | claudiub|2: Ah, that's how the instance would request those properties while resource providers is for hosts publishing that information | |
| 15:08:01 | claudiub|2 | yep | |
| 15:08:06 | sfinucan | gotcha | |
| 15:08:35 | mriedem | dansmith: maybe, i know there have been several threads on this, and basically at this point those don't really matter :) | |
| 15:08:48 | dansmith | aye | |