| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-02-15 | |||
| 18:56:18 | efried | Yup, I'm looking. We shouldn't be able to get here except by really bad timing. | |
| 18:56:29 | efried | TheJulia: And you said it happened more than once? | |
| 18:56:39 | dansmith | more than once in the logs even | |
| 18:56:49 | dansmith | on each sync | |
| 18:56:58 | jroll | could also be a problem with our multinode, whether it's queens+queens or queens+master | |
| 18:57:00 | TheJulia | efried: we have _not_ tried to recheck in case it was some process fluke | |
| 18:57:22 | efried | No, this simply shouldn't happen, still looking... | |
| 19:00:12 | efried | Okay, this is a *name* conflict, not a *uuid* conflict. This means we somehow got two RPs with different UUIDs but the same name. | |
| 19:00:26 | efried | and the name is a UUID, which is nice and confusing. | |
| 19:00:35 | cfriesen | mriedem: with respect to https://review.openstack.org/#/c/544748/ and my proposed change https://review.openstack.org/#/c/525253/. In our case it passes scheduling but then fails for whatever reason on the compute node. As such, the proposed fix is not sufficient because we would not end up calling _bury_in_cell0(). | |
| 19:00:36 | efried | I thought johnthetubaguy was banging his head against this last Fall. | |
| 19:00:46 | jroll | efried: the name would be the ironic uuid, right? | |
| 19:01:01 | efried | jroll: I'm not an expert there, but yeah, something like that. | |
| 19:01:08 | dansmith | efried: I wonder if we detected the ironic node go away and come back and we're trying to recreate it but never deleted it? | |
| 19:01:14 | dansmith | jroll: yes | |
| 19:01:32 | jroll | I seem to remember this being john's head banging https://review.openstack.org/#/c/508555/ | |
| 19:02:11 | jroll | but there shouldn't be a rebalance happening | |
| 19:02:46 | efried | jroll: Beat me to it, yeah, that's the patch I was thinking of. | |
| 19:03:01 | efried | but there was more to it than that, I thought. | |
| 19:03:32 | jroll | not sure | |
| 19:04:15 | openstackgerrit | Jay Pipes proposed openstack/nova-specs master: mirror nova host aggregates to placement API https://review.openstack.org/545057 | |
| 19:04:22 | jroll | the title on the bug certainly looks like this problem: https://bugs.launchpad.net/nova/+bug/1714248 | |
| 19:04:24 | openstack | Launchpad bug 1714248 in OpenStack Compute (nova) pike "Compute node HA for ironic doesn't work due to the name duplication of Resource Provider " [High,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 19:04:33 | efried | There was another patch where we mucked with one of the rt update methods to look for things with different names. | |
| 19:04:47 | dansmith | jroll: if ironic was down and the two nova computes notice it is back at different times maybe they could be disagreeing briefly on who owns what? | |
| 19:05:11 | jroll | dansmith: yeah, trying to track down if it's long enough to trigger a rebalance | |
| 19:05:58 | jroll | servicegroup api tells us | |
| 19:06:02 | melwitt | mriedem: FYI, digging into the difference between the cleanup volumes vs ports bugs today | |
| 19:06:18 | melwitt | based on your comments in the patch | |
| 19:08:50 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: DNM: Test LM with encrypted volumes https://review.openstack.org/545093 | |
| 19:09:38 | jroll | dansmith: ah yep, the subnode n-cpu considers itself down at this point, I believe http://logs.openstack.org/50/544750/8/check/ironic-grenade-dsvm-multinode-multitenant/5713fb8/logs/subnode-2/screen-n-cpu.txt.gz#_Feb_15_17_59_05_738069 | |
| 19:10:06 | jroll | ironic is unreachable for like 5 minutes | |
| 19:10:22 | jroll | or rather a full resource tracker run and then some | |
| 19:11:19 | openstackgerrit | Jackie Truong proposed openstack/python-novaclient master: Microversion 2.61 - Add trusted_image_certificates https://review.openstack.org/500396 | |
| 19:12:05 | mriedem | cfriesen: then what you have here https://review.openstack.org/#/c/525253/1/nova/conductor/manager.py doesn't help you | |
| 19:12:10 | mriedem | cfriesen: so i'm confused | |
| 19:12:42 | mriedem | cfriesen: is https://review.openstack.org/#/c/528385/ what you are looking for? | |
| 19:12:56 | dansmith | jroll: yeah, but with johnthetubaguy's reuse-compute-node patch I would think this wouldn't be a problem right? | |
| 19:12:58 | jroll | aaaand we have some problems deleting RPs: http://logs.openstack.org/50/544750/8/check/ironic-grenade-dsvm-multinode-multitenant/5713fb8/logs/screen-n-cpu.txt.gz#_Feb_15_17_58_22_418875 | |
| 19:13:00 | jroll | (wtf) | |
| 19:13:22 | jroll | dansmith: I would think so too, just confirming there is likely a rebalance, so they could be disagreeing | |
| 19:13:32 | dansmith | yeah | |
| 19:13:50 | dansmith | jroll: ah, that 503 during deleting is weird | |
| 19:14:11 | jroll | dansmith: indeed | |
| 19:14:20 | dansmith | jroll: do you guys have to restart apache? | |
| 19:14:39 | TheJulia | dansmith: we do | |
| 19:14:44 | dansmith | okay | |
| 19:14:45 | dansmith | also | |
| 19:14:47 | TheJulia | we update the configuration to load a vhost | |
| 19:15:02 | TheJulia | we also shutdown services at 17:55 for the upgrade, nova would have remained running | |
| 19:15:09 | dansmith | if placement is crashing the same way as conductor, maybe placement is dead under apache, hence the 503? | |
| 19:15:12 | jroll | good lord keystone db migrations are spammy | |
| 19:15:13 | dansmith | until you restart? | |
| 19:15:20 | TheJulia | so 17:58 is when everything is down | |
| 19:15:31 | dansmith | thus nova never got to delete the RPs | |
| 19:15:55 | TheJulia | I think the restart is during the ironic upgrade, checking to see when it actually occured | |
| 19:16:09 | dansmith | right, but if placement started crashing during the upgrade of packages, | |
| 19:16:21 | dansmith | which is when nova would have noticed ironic went away and tried to delete RPs or something, | |
| 19:16:24 | jroll | OH | |
| 19:16:28 | jroll | keystone is upgrading there | |
| 19:16:35 | jroll | and so placement can't validate the token | |
| 19:16:36 | dansmith | and then you restart apache.. | |
| 19:16:38 | dansmith | ohh | |
| 19:16:43 | jroll | http://logs.openstack.org/50/544750/8/check/ironic-grenade-dsvm-multinode-multitenant/5713fb8/logs/screen-placement-api.txt.gz#_Feb_15_17_58_22_463228 | |
| 19:17:02 | mriedem | melwitt: i'm likely also going to start writing a functional test for the case that we delete a build request for a bfv instance during local delete, because we aren't cleaning up volumes there either | |
| 19:17:08 | dansmith | and you get a bug, and you get a bug, and you get a bug... | |
| 19:17:21 | dansmith | jroll: good catch | |
| 19:17:28 | melwitt | mriedem: ack | |
| 19:17:50 | TheJulia | http://logs.openstack.org/50/544750/8/check/ironic-grenade-dsvm-multinode-multitenant/5713fb8/logs/grenade.sh.txt.gz#_2018-02-15_18_00_28_322 is when we restart apache | |
| 19:17:53 | dansmith | jroll: surely you're not intending to have keystone be upgradingin this task right? | |
| 19:17:59 | dansmith | s/task/job/ | |
| 19:18:30 | jroll | dansmith: I think we just haven't cared either way in the past | |
| 19:18:44 | jroll | keystone's one of those hand-wavy things to me, it's always there, don't care what version | |
| 19:18:46 | dansmith | jroll: do you care now? :P | |
| 19:18:49 | jroll | heh | |
| 19:21:37 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add admin guide doc on volume multiattach support https://review.openstack.org/544090 | |
| 19:21:48 | TheJulia | jroll: we could just skip restarting apache and fire up the local ironic-api, however then we're not forcing traffic through an older API endpoint which changes the scenario, although we would still have the rpc version pin | |
| 19:21:56 | mriedem | bauzas: since you helped review the multiattach series, can you check out ^ so we can backport that for queens RC2? | |
| 19:22:23 | jroll | TheJulia: eh, I'd rather not | |
| 19:22:28 | jroll | also these times aren't lining up :/ | |
| 19:22:32 | TheJulia | actually, other services will still likely need to restart things | |
| 19:22:44 | TheJulia | so we shouldn't try to avoid restarting apache | |
| 19:22:59 | jroll | oh it does line up, okay | |
| 19:23:16 | jroll | TheJulia: maybe we um, try to make apache not need 66 seconds to restart | |
| 19:23:18 | TheJulia | heh, 28 apache restarts in the grenade log | |
| 19:23:27 | jroll | jesus | |
| 19:23:31 | dansmith | wow | |
| 19:23:33 | dansmith | that's impressive | |
| 19:23:57 | mriedem | the edge people said openstack needed to be slimmed down | |
| 19:24:13 | TheJulia | yeah.... | |
| 19:24:15 | cfriesen | mriedem: I think you're right, I got messed up with which patch was fixing what. :) the fixes at https://review.openstack.org/#/c/528385 and https://review.openstack.org/#/c/340614/ look like they might do the trick. | |
| 19:24:31 | mriedem | cfriesen: cool | |
| 19:24:36 | mriedem | it is confusing | |
| 19:24:55 | mriedem | cfriesen: the point of creating the bdms in cell0 also was so that the local delete in the api can remove them | |
| 19:25:06 | mriedem | and thus avoid us having nova-compute, nova-api AND nova-conductor doing volume cleanup | |
| 19:25:49 | TheJulia | jroll: what makes you think 66 seconds? | |
| 19:26:27 | jroll | TheJulia: this line to the next one: http://logs.openstack.org/50/544750/8/check/ironic-grenade-dsvm-multinode-multitenant/5713fb8/logs/screen-keystone.txt.gz#_Feb_15_17_57_44_538878 | |
| 19:26:31 | jroll | it isn't 66 seconds to restart | |
| 19:26:48 | jroll | nor is it an apache thing | |
| 19:26:50 | jroll | but we shut down keystone for the entire keystone upgrade | |