| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-03-25 | |||
| 18:30:47 | sean-k-mooney | jhinman: if you have issue im also on the #openstack-kolla channel so feel free to ping me there not to bother the nova folks. if its not specific feel free to ping me here | |
| 18:53:40 | openstackgerrit | sean mooney proposed openstack/nova-specs master: add spec for image metadata prefiltering https://review.openstack.org/647578 | |
| 18:56:49 | mnaser | would it be ok to add https://review.openstack.org/#/c/645352/ to the rc2 list? and https://review.openstack.org/#/c/641907/ ? | |
| 18:59:30 | sean-k-mooney | mnaser: oh that the fix for the null issue you were looking at last week? | |
| 19:01:22 | mnaser | sean-k-mooney: correct, the first one is that, the second one is Oslo.service doing a full restart of the service instead of reload, which breaks some assumptions nova does | |
| 19:01:29 | mnaser | which means reloading to refresh info leaves you in a broken state | |
| 19:02:35 | sean-k-mooney | mnaser: right i brifly skimed that thread on the mailing list | |
| 19:02:54 | sean-k-mooney | oh so that was how this was getting to None in the first place | |
| 19:03:39 | mnaser | sean-k-mooney: yeah I could see it in the DB going from "NULL" (sql null) to "null" (son null) | |
| 19:03:40 | sean-k-mooney | or rather how self._bdm_obj.connection_info was | |
| 19:03:42 | mnaser | json* | |
| 19:08:19 | sean-k-mooney | mnaser: by the way on the olso service thing my intiall reaction was make oslos reload not call stop and start but i have no idea what other effect that would have | |
| 19:08:42 | sean-k-mooney | mnaser: but that is more have sane names for things then because it definetly the right thing to do | |
| 19:10:34 | openstackgerrit | sean mooney proposed openstack/nova-specs master: re add for train https://review.openstack.org/647581 | |
| 19:15:08 | openstackgerrit | sean mooney proposed openstack/nova-specs master: forward ported form stien https://review.openstack.org/647582 | |
| 19:17:42 | sean-k-mooney | mriedem: do ^ need to go back through full review or do can they be fast approved. i litrally just copied them updated the history footer? i know we are still focusing on RC2 but im hoping we can land that code before the ptg or m1 | |
| 19:18:20 | sean-k-mooney | mriedem: i know we want to talk about openlab for the numa testing at the ptg but just said i would ask | |
| 19:19:47 | efried | sean-k-mooney: If they were approved in stein, and they're verbatim (except the footer and s/stein/train/i) then they can be fast approved. | |
| 19:20:24 | sean-k-mooney | efried: ya they are | |
| 19:20:29 | efried | sean-k-mooney: However, I would request that you make some reference to the actual blueprint being proposed in the commit title and message body. | |
| 19:20:42 | melwitt | mnaser: are they regressions? I think we're only doing regression fixes for RC2 | |
| 19:20:46 | efried | sean-k-mooney: those titles are going to kill me | |
| 19:20:57 | sean-k-mooney | sure :) | |
| 19:21:04 | efried | thanks | |
| 19:21:07 | sean-k-mooney | i was just going for dinner and wanted to push something | |
| 19:21:15 | efried | cool | |
| 19:21:21 | efried | if you need to run, I can edit and fast approve. | |
| 19:21:27 | sean-k-mooney | ill fix them later. should i add them to the metting aggenda? | |
| 19:21:45 | melwitt | sean-k-mooney: instructions, you need the Previously-approved in the commit message https://specs.openstack.org/openstack/nova-specs/readme.html#previously-approved-specifications | |
| 19:22:02 | sean-k-mooney | melwitt ah perfect thanks | |
| 19:22:16 | efried | sean-k-mooney: You mean for the PTG? Unless there's something more to be discussed on them, I wouldn't think so. | |
| 19:22:47 | efried | sean-k-mooney: I guess they will come up in a "what do we prioritize for train" discussion, but that'll be fed by whatever's in the train approved blueprint list at the time anyway. | |
| 19:22:56 | sean-k-mooney | efried: no i wasnt sure if i was ment to add fast approavl spec to the nova team meeting agenda | |
| 19:23:01 | sean-k-mooney | ill just do what the doc says | |
| 19:23:30 | efried | oh, yeah, that won't be necessary. I (or someone) will push them as soon as they conform to what melwitt linked above. | |
| 19:24:01 | sean-k-mooney | efried: cool in that case ill fix them up and upload them later tonight | |
| 19:24:08 | efried | sean-k-mooney: ++ | |
| 19:25:00 | mnaser | melwitt: no both are long standing upgrade bugs afaik so yeah | |
| 19:29:09 | mriedem | mnaser: i've got a todo to try and recreate that first one with an in-tree functional test, which shouldn't take me long | |
| 19:29:33 | mriedem | idk about that oslo.service one - note that would require a release freeze exception on oslo.service and a minimum required version of oslo.service bump in nova (maybe that'd be optional though) | |
| 19:32:19 | mnaser | well, I think we need to come to consensus on if the issue is in Oslo service or nova. | |
| 19:32:37 | mnaser | Imho it’s Oslo service cause then what’s the point of sighup if you’re gonna stop and start... | |
| 19:32:53 | dansmith | mnaser: I've lost context here, but is there anything nova can do to _not_ be fully torn down and reset? | |
| 19:33:07 | dansmith | because I thought not, based on how we get started and managed from oslo | |
| 19:34:16 | dansmith | as it is, a "systemctl restart nova-compute" is much better than the behavior we're getting from HUP, IMHO | |
| 19:34:37 | mnaser | dansmith: based on my understanding it looks like Oslo service never really provides an API to handle HUP signals. | |
| 19:35:01 | dansmith | mnaser: you mean it grabs HUP itself, but doesn't do what a HUP signal should really do | |
| 19:35:09 | dansmith | and, if so, I agree :) | |
| 19:35:48 | mnaser | Exactly. It calls reload but only after calling stop and start after so that’s a bit useless at that point. | |
| 19:36:00 | dansmith | right | |
| 19:36:21 | mnaser | I think the most portable potential fix is adding a keyword argument to the stop() function call which includes sighup=bool | |
| 19:36:26 | dansmith | that doesn't seem like a reasonable status quo to me, which means I think it's an oslo.service bug, not a "nova should be doing something different" scenario | |
| 19:36:36 | efried | I agree with that | |
| 19:36:49 | mnaser | Yes. I think Nova is doing the right thing as well. | |
| 19:36:53 | dansmith | efried: agree with which? | |
| 19:37:05 | mnaser | Nothing except reload() should be called on SIGHUP. | |
| 19:37:07 | efried | dansmith: `it's an oslo.service bug, not a "nova should be doing something different" scenario` | |
| 19:37:12 | dansmith | efried: ack | |
| 19:37:21 | efried | And we've got stuff documented that is supposed to happen to nova on HUP | |
| 19:37:27 | dansmith | right | |
| 19:37:34 | efried | which currently does happen, but is worthless because it destroys the process | |
| 19:37:44 | dansmith | yarp | |
| 19:38:12 | efried | stuff I have the confidence to say in IRC, but not on the ML | |
| 19:38:21 | efried | but I have been watching that thread with interest | |
| 19:38:31 | mnaser | So my review here solved this but “changed” existing behaviour ... https://review.openstack.org/#/c/641907 | |
| 19:38:42 | mnaser | So is it a bug fix or behaviour change. Up for debate. | |
| 19:39:04 | efried | it can be both | |
| 19:39:27 | dansmith | the only argument not to do that would be to keep the existing behavior because it's always been broken, but that leaves us unable to resolve our issue | |
| 19:39:33 | mnaser | But it looks like nova is a heavy user of sighup and I can’t get other projects to talk about their usage. But it looks like neutron is broken because it makes the same (norms) assumption | |
| 19:39:33 | dansmith | which does not seem reasonable | |
| 19:40:01 | efried | Zane's comment is valid, though. The first time I heard of SIGHUP for openstack it was in the context of mutable config options. So if the "fix" makes some config options not mutable, that's a problem. | |
| 19:40:22 | dansmith | efried: I don't think it has anything to do with that | |
| 19:40:38 | dansmith | the only thing that would be impacted are things that you only look at on service start | |
| 19:40:57 | efried | like RPC settings? | |
| 19:40:58 | dansmith | however, those kinda by definition shouldn't be mutable, or you need to specifically handle the case where they have changed | |
| 19:41:13 | dansmith | efried: right, db connection and rpc connection have always been "these can't be mutable" | |
| 19:41:27 | dansmith | because it's unlikely people are going to handle them properly (as exposed here) | |
| 19:41:38 | efried | Okay. Can you respond to Zane in https://review.openstack.org/#/c/641907 ? | |
| 19:42:12 | dansmith | I can, but I don't really want to.. he's also not asserting that it makes sense to say that, | |
| 19:42:20 | dansmith | he's just coming up with some guess | |
| 19:42:38 | dansmith | oh, | |
| 19:42:53 | dansmith | I see his latest one | |
| 19:45:04 | dansmith | there, I said things | |
| 19:45:20 | mnaser | I'll try to keep following up on things and get it to land, I just don't know how back portable these things can be | |
| 19:55:40 | openstackgerrit | Chris Dent proposed openstack/nova master: Delete the placement code https://review.openstack.org/618215 | |
| 20:12:00 | mriedem | mnaser: there should definitely at least be a "fixes" release note in that oslo.service change if it's changing behavior but fixing a long-standing bug in HUP handling | |
| 20:12:10 | mriedem | related to your "I just don't know how back portable these things can be" comment | |
| 20:12:28 | mriedem | generally don't want to backport changes in behavior, but if we have a good reason to, let's make sure there is a release note | |
| 20:15:11 | mnaser | mriedem: ok, ill wait for zane's comments and try to write up a release note that explains how critical (and how it was pretty much broken in every project I've inspected) | |
| 20:17:33 | mriedem | mnaser: dansmith: efried: i've also dumped some thoughts in there | |
| 20:17:43 | mriedem | trying to compromise but give operators a switch | |
| 20:17:50 | mriedem | which could also be backportable | |
| 20:19:46 | mriedem | essentially add a [workarounds] style option to oslo.service | |
| 20:42:55 | openstackgerrit | Merged openstack/os-vif master: Drop testtools from test-requirements.txt https://review.openstack.org/647286 | |
| 20:57:09 | mriedem | mnaser: i wrote a functional test for bug 1821244 but can't recreate the issue | |
| 20:57:13 | openstack | bug 1821244 in OpenStack Compute (nova) "Failed volume creation can result in invalid `connection_info` field" [Medium,In progress] https://launchpad.net/bugs/1821244 - Assigned to Mohammed Naser (mnaser) | |
| 20:57:26 | efried | mriedem: Some bp questions if you have a mo | |
| 20:57:37 | mriedem | mnaser: i'll post it quick and see if you can poke holes in the test | |
| 20:57:47 | mriedem | efried: i have to run real quick, but should be back before 5 | |
| 20:58:13 | efried | mriedem: k, I'll leave this here for you: | |