| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-24 | |||
| 10:09:28 | tojuvone | Vitrage need to differentiate host fault from planned maintenance, to have proper alarms | |
| 10:09:47 | mdbooth | johnthetubaguy: I'm joking, if we need to revert lets do it. | |
| 10:09:57 | mdbooth | However, I really would like to get that merged | |
| 10:09:59 | johnthetubaguy | mdbooth: I think they did the revert already | |
| 10:10:03 | mdbooth | Ok, cool | |
| 10:10:29 | johnthetubaguy | yeah, I think the emergency bit is done already | |
| 10:10:56 | mdbooth | We don't run the NFS job by default, which is probably how that slipped through | |
| 10:11:13 | bauwser | tojuvone: so, to understand correctly the NFV usecase, Vitrage emits alarms, right? | |
| 10:11:20 | mdbooth | And, as you say, I only did the more thorough manual testing on the next patch in the series. | |
| 10:11:37 | bauwser | tojuvone: for each "node" that is impacted by some notification | |
| 10:11:48 | mdbooth | I'll line that up today | |
| 10:12:21 | bauwser | tojuvone: now, once those deduced alarms are raised, which tool will trigger actions on those ? | |
| 10:12:29 | tojuvone | bauwser, Its RCA raising alarms on each level where needed to | |
| 10:12:40 | bauwser | tojuvone: that I got it too | |
| 10:12:48 | bauwser | the output is basically a set of alarms | |
| 10:12:58 | bauwser | with linkages between thoses | |
| 10:12:59 | tojuvone | bauwser, so here planned mainteancne instead | |
| 10:13:33 | tojuvone | bauwser, at least in fault situation it could force down host or what else would be configured | |
| 10:13:48 | tojuvone | bauwser, but it is not intended to make workflows to migrate | |
| 10:13:56 | bauwser | *what* would do that ? | |
| 10:14:02 | bauwser | Vitrage? | |
| 10:14:24 | bauwser | ie. what executes the action plan ? | |
| 10:14:30 | openstackgerrit | OpenStack Proposal Bot proposed openstack/python-novaclient master: Imported Translations from Zanata https://review.openstack.org/449041 | |
| 10:14:48 | bauwser | tojuvone: is this something which is already present today | |
| 10:14:49 | bauwser | ? | |
| 10:15:03 | bauwser | tojuvone: because my main concern on Nova is that we need to understand the missing gaps | |
| 10:15:30 | bauwser | tojuvone: ie. what is specifically lacking on our side so that a specific usecase could be fulfilled | |
| 10:15:40 | tojuvone | bauwser, I was planning to make poc with Craton upon planned mainteancne | |
| 10:16:09 | bauwser | tojuvone: the main problem I have with your spec is that I don't see yet a clear picture on the overall workflow, and what's missing in that workflow on the nova side | |
| 10:16:36 | tojuvone | bauwser, I am not sure somebody doing this in fully open source | |
| 10:16:55 | bauwser | well, I'm not exactly concerned by that | |
| 10:17:13 | bauwser | but I need to understand why we can't just disable services or force hosts down | |
| 10:17:27 | bauwser | because I don't have the clear workflow picture in my mind | |
| 10:17:40 | bauwser | for example, say the forced_down series you implemented | |
| 10:18:18 | tojuvone | bauwser, In Nova it is much to me to see it in servers API | |
| 10:18:21 | bauwser | it was clear to me that there was an usecase where 3rd-party tools (opensource or not) could be identifying before our SG API that a host is down and could accelerate the logic by having a specific bit to call | |
| 10:19:03 | tojuvone | that is fault management | |
| 10:19:18 | tojuvone | here you have planned maintenance | |
| 10:19:23 | bauwser | correct, I'm not saying your spec is that | |
| 10:19:45 | bauwser | sure, but what's missing in nova for that ? | |
| 10:19:54 | bauwser | you can disable a service | |
| 10:20:10 | bauwser | you can migrate all the existing instances of that host out | |
| 10:20:20 | tojuvone | the planned window is missing | |
| 10:20:21 | bauwser | without having user-visible impacy | |
| 10:20:39 | bauwser | you mean "disable me this service in 1 hour" ? | |
| 10:20:54 | tojuvone | yes | |
| 10:21:03 | tojuvone | and what you can do in that windwo | |
| 10:21:19 | bauwser | say you have a planned maintenance in one hour | |
| 10:21:28 | bauwser | what do you want to have on the nova side? | |
| 10:21:40 | bauwser | ie. what do you want to do with the scheduler and your instances ? | |
| 10:21:50 | tojuvone | and to have the rolling thing with that window | |
| 10:21:50 | bauwser | not speaking of the proposed implementation | |
| 10:22:08 | bauwser | rather about what you wanna do as an operator during that period *before* the maintenance | |
| 10:22:28 | tojuvone | yes, for now we can forget what is proposed currently as implementation | |
| 10:22:40 | bauwser | for example | |
| 10:22:53 | bauwser | let's speak ITIL for a spec | |
| 10:22:55 | bauwser | sec | |
| 10:23:17 | tojuvone | as a user, application, you want to do something | |
| 10:23:19 | bauwser | your change manager gave you a change window that will happen in 1 hour because you need to upgrade the kernel | |
| 10:23:26 | johnthetubaguy | bauwser: I think we just need to focus on the use cases in the spec | |
| 10:23:27 | bauwser | on a specific host | |
| 10:23:39 | johnthetubaguy | I am current adding a patch on top of tojuvone's spec to see if I understand them correctly | |
| 10:23:48 | johnthetubaguy | I think we should stay out of implementation detail right now | |
| 10:23:54 | bauwser | johnthetubaguy: those usecases are way too-high level for me | |
| 10:24:03 | bauwser | johnthetubaguy: I agree | |
| 10:24:13 | bauwser | johnthetubaguy: I want to understand the crux of the problem | |
| 10:24:16 | bauwser | without speaking code | |
| 10:24:37 | bauwser | ie. I want to discuss about what is a "planned maintenance" and what actions are needed to be done | |
| 10:24:43 | johnthetubaguy | yeah, I am trying to write that out now, to see if I understand what I think tojuvone is trying to say | |
| 10:24:52 | bauwser | for example | |
| 10:24:57 | johnthetubaguy | (this is based on how I know we wanted this to work for public cloud) | |
| 10:25:15 | johnthetubaguy | well, we built this system using server metadata | |
| 10:25:15 | bauwser | "* As an admin I want to have an external tool to handle hosts and their maintenance:" is mentioned in that spec, but waaaaays too much than just nova | |
| 10:25:36 | tojuvone | yes, that is used before ;) | |
| 10:25:58 | johnthetubaguy | it will be many, many different tools doing lots of this different operations, the key bit is the Nova interactions they all have in common, I am trying to write those down right now | |
| 10:26:14 | bauwser | okay, tojuvone could you please explain me what is a "planned maintenance" ? | |
| 10:26:16 | mdbooth | johnthetubaguy: https://review.openstack.org/#/c/439520/ | |
| 10:26:29 | mdbooth | I don't understand your comment about the rebase. What's the context there? | |
| 10:26:41 | openstackgerrit | John Garbutt proposed openstack/nova master: compute: Move detach logic from manager into driver BDM https://review.openstack.org/439520 | |
| 10:27:00 | johnthetubaguy | mdbooth: so I expected rebase to fail once that patch I linked to lands on master | |
| 10:27:16 | johnthetubaguy | mdbooth: 437665 I think it was | |
| 10:27:21 | mdbooth | johnthetubaguy: IIRC there was no conflict between the 2 | |
| 10:27:36 | openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/nova master: Read Neutron port 'binding_profile' during boot https://review.openstack.org/449257 | |
| 10:27:41 | johnthetubaguy | mdbooth: so there was on my system, which is kinda freaky | |
| 10:27:52 | johnthetubaguy | mdbooth: I must have fat fingered something | |
| 10:27:55 | mdbooth | Ok, probably means there was, then :) | |
| 10:28:06 | tojuvone | I have used "planned maitenance", to high light it does not happen because sudden fault | |
| 10:28:16 | mdbooth | But when lyarwood separated them he didn't lose review votes | |
| 10:28:16 | johnthetubaguy | so its on master, and it just rebased cleanly, so maybe we are OK | |
| 10:28:24 | mdbooth | Suggesting there was no difference | |
| 10:28:29 | johnthetubaguy | mdbooth: ah, good point, that usually means no difference | |
| 10:28:52 | johnthetubaguy | I may have tried to cherry pick ontop of an old version of its-self or something dumb | |
| 10:29:02 | johnthetubaguy | oh, well, seems fine now | |
| 10:29:08 | bauwser | tojuvone: I'm just commenting your spec | |
| 10:29:08 | mdbooth | Meh, if the other one's already in the gate it's no problem | |
| 10:29:14 | bauwser | tojuvone: I think I get a better picture | |
| 10:29:26 | tojuvone | bauwser, ok :) | |
| 10:29:44 | bauwser | tojuvone: #1 you wanna signal some hosts for a maintenance and prevent instances booting on them | |
| 10:29:57 | bauwser | #2 you wanna test those hosts during that maintenance | |
| 10:30:16 | bauwser | #3 you wanna only schedule instances to maintened hosts | |