| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-26 | |||
| 14:39:00 | bauzas | I'll at least work on trying to have the force_hosts thing to accept an empty list | |
| 14:39:06 | johnthetubaguy | is that without any builds happening? | |
| 14:39:07 | dansmith | johnthetubaguy: I have been yeah | |
| 14:39:24 | cdent | johnthetubaguy: yes, that's after the builds are done and change has frozen | |
| 14:39:37 | johnthetubaguy | dansmith: don't get me wrong, I would love to do a clean switch, but I would rather that than not land it | |
| 14:39:42 | cdent | but it can be cut _way_ back with a few changes. halfed with a bug fix that's already in progress | |
| 14:40:05 | johnthetubaguy | cdent: that sounds like a worthy aim | |
| 14:40:44 | cdent | in order to make one fix, I/someone needs to understand why tracked_instances in the resource tracker is being cleared on every periodic job | |
| 14:40:46 | dansmith | cdent: okay, but also, we can scale that horizontally | |
| 14:41:02 | cdent | if that clear() doesn't happen, the number of requests can drop, alot | |
| 14:41:14 | cdent | dansmith: yeah, which is nicely easy | |
| 14:41:17 | dansmith | johnthetubaguy: the problem with that is that I think people will rely on that crutch | |
| 14:41:30 | cdent | yeah, that crutch is something we've been wanting to avoid | |
| 14:41:34 | dansmith | johnthetubaguy: but, if we check service_version and only do the fallback if there are old nodes, maybe that's a thing | |
| 14:41:45 | johnthetubaguy | dansmith: thats the one I was meaning | |
| 14:41:50 | dansmith | johnthetubaguy: that way once they've updated everything then they're stuck | |
| 14:41:51 | johnthetubaguy | dansmith: only during upgrade, allow the fallback | |
| 14:42:04 | johnthetubaguy | we should totally not fallback once we are upgraded | |
| 14:42:07 | dansmith | cdent: I know, I really want to avoid the crutch | |
| 14:42:19 | bauzas | we all want to avoid the crutch | |
| 14:42:29 | bauzas | (whatever the word means for a non-English speaking person) | |
| 14:42:31 | dansmith | but, if we give them a crutch that is half cut through so that it breaks once they walk out the door | |
| 14:43:04 | johnthetubaguy | dansmith: thats the one | |
| 14:43:18 | johnthetubaguy | dansmith: like trolley wheels that lock on if you steal the trolley | |
| 14:43:19 | bauzas | johnthetubaguy: dansmith: I can propose something about a conditional branch verifying the service version | |
| 14:43:36 | bauzas | and if you upgrade all your computes, then good luck | |
| 14:43:40 | johnthetubaguy | bauzas: thats what I suggested on the ML anyways (at least thats what I tried to suggest) | |
| 14:44:21 | dansmith | johnthetubaguy: by trolley you mean "shopping cart" right? | |
| 14:44:25 | bauzas | johnthetubaguy: well, I mentioned that also in my email as a conclusion, but I said we turned down the idea | |
| 14:44:41 | bauzas | johnthetubaguy: if we revise that to something acceptable, then I can surely implement it | |
| 14:44:49 | bauzas | but I'd love to see some quorum here | |
| 14:44:49 | dansmith | cdent: what do you think about that? | |
| 14:45:11 | dansmith | cdent: I'll summarize | |
| 14:45:21 | dansmith | cdent: we can detect if all the computes have been upgraded or not, | |
| 14:45:43 | dansmith | cdent: and so we could fall back on the crutch until that point, and then go scorched-earth after that and refuse to look at anything other than placement | |
| 14:46:09 | dansmith | cdent: which would solve our current problem and not let anyone (reasonably) keep from using placement in ocata | |
| 14:46:29 | dansmith | most people that don't do full-on live upgrades would have to deploy placement immediately (i.e. the tripleo types) | |
| 14:46:31 | cdent | so each resource tracker would say "are all my peers up to date with me?" and if not, they'd use the old method of handling inventory? | |
| 14:46:37 | dansmith | no, | |
| 14:46:51 | dansmith | computes would all do placement and RT all the time like they do now, | |
| 14:47:00 | dansmith | but scheduler would refuse to look at the RT data once all the computes are upgraded | |
| 14:47:30 | dansmith | so it would just be a change in bauzas' patch | |
| 14:47:31 | cdent | but the problem we're having right now is computes that _may_ be upgraded, but haven't changed their configuration? | |
| 14:47:32 | bauzas | lemme fire a patch ip | |
| 14:47:40 | bauzas | so cdent can see | |
| 14:47:45 | bauzas | I mean a diff pastebin | |
| 14:47:53 | cdent | I understand the change, I'm just not clear if it covers all the issues | |
| 14:48:05 | dansmith | I'm pretty sure it does | |
| 14:48:09 | johnthetubaguy | so we might need one step worse... | |
| 14:48:19 | johnthetubaguy | the placement data will be incomplete | |
| 14:48:33 | johnthetubaguy | so we could use the crutch all the time, until the upgrade has completed | |
| 14:48:39 | dansmith | johnthetubaguy: right you'd have to favor the RT data until the time when you switch | |
| 14:48:43 | openstackgerrit | Sergey Nikitin proposed openstack/nova: Change parent of diagnostics objects to NovaObject class https://review.openstack.org/394480 | |
| 14:48:44 | openstackgerrit | Sergey Nikitin proposed openstack/nova: Standardization of VM diagnostics info API. https://review.openstack.org/355540 | |
| 14:48:46 | openstackgerrit | Sergey Nikitin proposed openstack/nova: Completed implementation of instance diagnostics for Xen https://review.openstack.org/399613 | |
| 14:49:02 | dansmith | johnthetubaguy: the downside is that it makes the switch to different data non-obvious, so people may not know why things are suddenly being different, | |
| 14:49:08 | dansmith | just because a lab monkey upgraded the last one | |
| 14:49:15 | johnthetubaguy | yeah | |
| 14:49:38 | bauzas | easy peasy question, can someone tell me the minimum service version for Ocata computes ? | |
| 14:49:57 | bauzas | we don't comment that in the SERVICE_VERSION_HISTORY dict | |
| 14:50:02 | dansmith | johnthetubaguy: we could log all-caps WARNING messages whilst using the crutch :) | |
| 14:50:04 | johnthetubaguy | bauzas: technically, you might want to bump that, so you know everyone has your patch | |
| 14:50:06 | dansmith | bauzas: yes, it's easy | |
| 14:50:18 | bauzas | johnthetubaguy: we don't need that | |
| 14:50:29 | cdent | okay, I think I'm caught up with you dansmith: the scheduler is the one that checks if all the compute nodes are up to date, and then makes a choice on what type of scheduling it does, yeah? | |
| 14:50:35 | bauzas | johnthetubaguy: all computes are already able to talk to placement since newton | |
| 14:50:49 | dansmith | bauzas: https://github.com/openstack/nova/blob/stable/newton/nova/objects/service.py#L92 | |
| 14:50:49 | bauzas | johnthetubaguy: I just want to detect a fresh code | |
| 14:50:59 | cdent | bauzas: but only if they have the config... | |
| 14:50:59 | dansmith | cdent: yes | |
| 14:51:12 | bauzas | cdent: but we say they need to upgrade to ocata, right? | |
| 14:51:27 | johnthetubaguy | so does a compute node die if it doesn't have placement configured? | |
| 14:51:29 | bauzas | dansmith: okay, so I need to set a new version | |
| 14:51:34 | bauzas | johnthetubaguy: nope | |
| 14:51:38 | dansmith | will ocata computes fail to start if the placement config is not done? | |
| 14:51:39 | cdent | bauzas: yes, but earlier you were complaining that making people change their configs to upgrade to ocata was in violation? | |
| 14:51:52 | dansmith | bauzas: I dunno what a new version would signal, TBH | |
| 14:52:21 | snikitin | sfinucan: Hi! thank you for comments! I updated https://review.openstack.org/#/c/399613 | |
| 14:52:38 | dansmith | bauzas: if they don't die, then maybe we should make them die and make that a new service version, and then check for that version in the scheduler | |
| 14:52:41 | bauzas | dansmith: good question about nova computes refusing to start | |
| 14:52:51 | bauzas | dansmith: I agree with that plan | |
| 14:52:54 | johnthetubaguy | cdent: its not as bad, because the old compute could have had that same config, but its sure borderline | |
| 14:52:57 | dansmith | I think it's what johnthetubaguy was saying too | |
| 14:52:59 | bauzas | dansmith: that's the move defensive one | |
| 14:53:12 | johnthetubaguy | yeah, make it obvious things are bad | |
| 14:53:22 | snikitin | alex_xu: Hi Alex! I got two +2 on this patch https://review.openstack.org/#/c/394480/ can I get a +W, please? | |
| 14:53:41 | dansmith | johnthetubaguy: do you think it's too borderline? | |
| 14:53:49 | bauzas | but we said placement is mandatory for ocata, right? | |
| 14:54:02 | dansmith | johnthetubaguy: I feel like we've already agreed we're going to make this decision | |
| 14:54:03 | bauzas | so that means you have to access it when restarting your services | |
| 14:54:03 | dansmith | right | |
| 14:54:04 | dansmith | what bauzas said | |
| 14:54:05 | dansmith | and we're logging on newton | |
| 14:54:28 | johnthetubaguy | dansmith: I think I am OK because we get a massive gain, so we stick with it | |
| 14:54:31 | bauzas | here, we're just trying to allow newton computes that are untouched to still be able to get instances on them | |
| 14:54:58 | dansmith | johnthetubaguy: but really, it was optional in newton, required in ocata, and this is just enforcing that no? | |
| 14:55:21 | bauzas | dansmith: to summarize, if I'm bumping the service version now, that would mean I'm sure the service is upgraded to ocata | |
| 14:55:49 | johnthetubaguy | dansmith: I think the key bit that makes me OK, is they just need to update the config on the old system before they do the upgrade, if they haven't done that | |
| 14:55:54 | mriedem | o/ | |
| 14:56:05 | dansmith | bauzas: you don't need to bump it now, but if you're going to make the computes refuse to start, I'd call that a service version.. otherwise you're adding a number that doesn't have any real meaning | |