| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-26 | |||
| 14:56:06 | bauzas | now, the only question would be : should we silently soft-fail and do nothing with the risk for operators that they could loose capacity if they misconfigure ? | |
| 14:56:19 | johnthetubaguy | big fire, always the big fire | |
| 14:56:24 | dansmith | johnthetubaguy: right, and they could have done that in newton months ago if they wanted to be proactive | |
| 14:56:29 | bauzas | dansmith: the thing is that we haven't bumped the version in Ocata | |
| 14:56:36 | dansmith | bauzas: not at all? | |
| 14:56:36 | johnthetubaguy | dansmith: yeah, I think thats what makes me OK with it | |
| 14:56:52 | bauzas | dansmith: I'm still seeing version 15 in my local master branch | |
| 14:56:59 | dansmith | mriedem: read back to my "let me summarize" comment to cdent | |
| 14:57:03 | mriedem | what are we doing with a min service version check? | |
| 14:57:04 | dansmith | bauzas: then yes you need to add one | |
| 14:57:12 | johnthetubaguy | yeah, just add one | |
| 14:57:23 | bauzas | on it | |
| 14:57:23 | dansmith | mriedem: thinking we will add a fallback to RT data _if_ the computes are not all upgraded | |
| 14:57:27 | dansmith | mriedem: in the scheduler | |
| 14:57:38 | bauzas | and we hard-fail, as johnthetubaguy said | |
| 14:57:39 | dansmith | mriedem: which will let us get past grenade, and give the deployers a little longer grace period | |
| 14:58:15 | dansmith | mriedem: once they upgrade all their computes, scheduler will stop looking at RT data | |
| 14:58:29 | dansmith | mriedem: which means if they only change config during the upgrade they will still be fine | |
| 14:58:55 | dansmith | mriedem: whereas before, we had to make them configure their newton computes before starting the controller upgrade | |
| 14:59:25 | mriedem | we only fallback if we don't get any RPs *and* minimum compute isn't new enough right? | |
| 14:59:26 | sfinucan | claudiub|2: I looked at that and it seems fine, but I think one of Moshe's questions still stands | |
| 14:59:41 | dansmith | mriedem: no we have to always fallback if computes are old | |
| 14:59:54 | mriedem | so don't even check placement until then? | |
| 14:59:56 | johnthetubaguy | otherwise you get incomplete data, and the test we want to pass will still fail | |
| 14:59:57 | sfinucan | ...and I don't think I can approve until one of ye proves it's not a problem :) | |
| 15:00:02 | dansmith | mriedem: because otherwise after one compute reports RPs you'd hard switch and have a 1-compute cloud | |
| 15:00:46 | mriedem | johnthetubaguy: the test would pass if we would fallback to computenodelist.get_all if there were 0 RPs, but yes i understand we'd have a 1 compute cloud for awhile | |
| 15:00:57 | mriedem | fwiw we talked about that weeks ago and people thought that was ok and a small window | |
| 15:01:04 | johnthetubaguy | mriedem: I was thinking the force host thing, but yeah | |
| 15:01:18 | dansmith | mriedem: I think that was about waiting for the 60-second timer, | |
| 15:01:27 | dansmith | mriedem: but it could be weeks if you upgrade one compute as a test | |
| 15:01:43 | dansmith | mriedem: in this scenario | |
| 15:03:56 | mriedem | dansmith: was refering to this http://lists.openstack.org/pipermail/openstack-dev/2016-December/109060.html | |
| 15:04:16 | sfinucan | snikitin: Done and done | |
| 15:04:57 | mriedem | but whatever, that was before we had a blocking issue with grenade | |
| 15:04:58 | dansmith | mriedem: right but that was a thing that is triggered by doing the controller upgrade, right? this would be a thing triggered by the first compute upgrade | |
| 15:05:09 | sfinucan | claudiub|2: Call me a fool, but wasn't this kind of request what we were implementing qualitative resources for? https://review.openstack.org/#/c/420960 | |
| 15:06:19 | snikitin | sfinucan, thanks! | |
| 15:06:27 | dansmith | mriedem: meaning, I think before we were assuming 1. Configure computes for placement, 2. upgrade controllers, 3. wait 60s for data to filter before scheduling works properly | |
| 15:06:29 | sfinucan | johnthetubaguy: would it be too cheeky of me to +2 this, given that I addressed comments on it? https://review.openstack.org/#/c/415597 | |
| 15:06:32 | claudiub|2 | sfinucan: sure, but that is being used for the ImageMetaProps object. that still has to exist | |
| 15:06:40 | dansmith | mriedem: now, it would be 1. upgrade controllers, 2. upgrade one compute, 3. stuck until you upgrade more computes | |
| 15:07:49 | sfinucan | claudiub|2: Ah, that's how the instance would request those properties while resource providers is for hosts publishing that information | |
| 15:08:01 | claudiub|2 | yep | |
| 15:08:06 | sfinucan | gotcha | |
| 15:08:35 | mriedem | dansmith: maybe, i know there have been several threads on this, and basically at this point those don't really matter :) | |
| 15:08:48 | dansmith | aye | |
| 15:08:51 | mriedem | i'm just reading back through the responses from the ML from earlier today | |
| 15:09:13 | sfinucan | snikitin: If you could take a look at https://review.openstack.org/#/c/405954/ (and the follow up) today, I'd appreciate it. Think you should know enough about that code to comment | |
| 15:09:47 | johnthetubaguy | sfinucan: usually I would say no, depends how you feel I guess. | |
| 15:10:40 | sfinucan | johnthetubaguy: Sweet - I'll go for it so. It looks good and would be good to get it in | |
| 15:10:51 | snikitin | sfinucan, sure | |
| 15:11:01 | bauzas | dansmith: question, what should I hard-fail ? | |
| 15:11:34 | bauzas | dansmith: just that the placement section is empty, or that we aren't able to call the placement API by doing a query? | |
| 15:11:54 | bauzas | dansmith: I'd favor a real test call | |
| 15:12:02 | dansmith | bauzas: just that the config isn't there I think | |
| 15:12:07 | dansmith | bauzas: doing a real call in the middle of init_host to check that and failing would be terrible | |
| 15:12:16 | dansmith | bauzas: because if it fails for some legit reason, computes wouldn't start | |
| 15:12:24 | bauzas | dansmith: okay | |
| 15:13:25 | mriedem | what do we need to touch init_host in the compute for? | |
| 15:13:53 | bauzas | mriedem: for preventing ocata computes to start if they haven't filled in their placement section | |
| 15:14:02 | dansmith | mriedem: to drive home that placement config is required | |
| 15:14:10 | dansmith | mriedem: i.e. like anything else, there's not much point in starting if we don't have sufficient config to do anything | |
| 15:14:29 | claudiub|2 | sfinucan: so, VFs / PFs won't show up as assignable PCI devices, rather those are requested from Hyper-V, and Hyper-V will then assign them to the VMs. | |
| 15:14:49 | sfinucan | claudiub|2: as what? VIFs? | |
| 15:14:53 | johnthetubaguy | bauzas: +1 what dansmith said, that sounds right | |
| 15:15:15 | bauzas | johnthetubaguy: dansmith: and it would raise an exception if not CONF.placement? | |
| 15:15:33 | mriedem | ok so the grenade case should be ok b/c newton subnode will be running and ocata compute will have config so will start | |
| 15:15:42 | dansmith | mriedem: right | |
| 15:15:43 | mriedem | but scheduler will use the old RT data because of the newton compute | |
| 15:15:47 | dansmith | yes | |
| 15:16:00 | claudiub|2 | sfinucan: yeah. The vif has a property - if it SR-IOV is enabled for it or not. by default it is false. | |
| 15:16:02 | dansmith | bauzas: yes, or whatever will arrest init_host cleanly | |
| 15:16:13 | bauzas | dansmith: that's my question | |
| 15:16:17 | bauzas | dansmith: should we log and stop ? | |
| 15:16:26 | bauzas | or should we more like "sorry dude" | |
| 15:16:36 | dansmith | I dunno what that means | |
| 15:16:40 | bauzas | heh | |
| 15:16:43 | bauzas | I mean | |
| 15:16:45 | bauzas | two choices | |
| 15:16:58 | bauzas | #1 log an ERROR and return | |
| 15:17:07 | bauzas | #2 raise an exception | |
| 15:17:16 | bauzas | I'd favor #1 | |
| 15:17:28 | dansmith | bauzas: that won't stop compute from starting | |
| 15:17:38 | dansmith | that will just leave it further hamstrung | |
| 15:17:53 | bauzas | okay, lemme see how init_host is working | |
| 15:18:19 | mriedem | an unhandled exception from init_host will kill the nova-compute process | |
| 15:18:22 | mriedem | which i think is the goal | |
| 15:18:54 | mriedem | meaning you upgraded the code to ocata but didn't configure it properly, | |
| 15:18:57 | mriedem | so you need to get paged | |
| 15:18:59 | dansmith | right, only an exception will stop it from what I see | |
| 15:19:08 | bauzas | okay, that's my question | |
| 15:19:15 | bauzas | because it's threaded ? | |
| 15:19:22 | dansmith | bauzas: so just raise an exception, which will be ugly, and we could maybe clean that up in service.py after the fact to make it a little less ugly on the eyes | |
| 15:19:23 | bauzas | I mean gt | |
| 15:19:30 | dansmith | nothing related to threading | |
| 15:19:33 | bauzas | anyway | |
| 15:19:37 | bauzas | let's do the exception | |
| 15:19:46 | bauzas | no time for introspecting the caller :) | |
| 15:20:31 | mriedem | i guess https://review.openstack.org/#/c/425524/ didn't work huh | |