Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-11-21
23:06:22 melwitt noted
23:06:44 melwitt ok, I see. yeah, I think sean-k-mooney mentioned that too but I didn't understand it at the time
23:06:46 mriedem if the uuids are all synced, compute restarts, creates the compute node with the same uuid and finds the provider already exists with that uuid
23:07:18 mriedem not so lucky with libvirt though
23:07:30 mriedem b/c the uuid will be unique per compute node record on the same host
23:07:46 mriedem granted our code that checks to see if the provider already exists could be smarter
23:08:23 mriedem around here https://opendev.org/openstack/nova/src/tag/19.0.0/nova/scheduler/client/report.py#L568
23:08:34 melwitt ok, hm that last part is interesting because that sounds like our second case (non-ironic)
23:08:40 mriedem if we found a provider with the same name we should probably just use it
23:09:10 melwitt yeah. I think fixing this would kill two birds with one stone
23:09:21 melwitt ironic and non-ironic, unless I'm missing something
23:09:33 melwitt *fixing it in that way
23:09:37 mriedem we can also easily find a provider with the same name using GET /resource_providers?name=<name>
23:09:43 mriedem if we find out, use it
23:09:46 mriedem *one
23:09:54 melwitt aye
23:09:57 mriedem maybe
23:10:04 mriedem i mean there could be dragons there too i'm not thinking about
23:10:06 melwitt that would be an extra call tho
23:10:21 mriedem only if you 409
23:10:25 melwitt right
23:10:31 melwitt oh
23:10:33 mriedem because clearly we aren't getting this back when we hit this name_conflict = 'Conflicting resource provider name:'
23:10:35 melwitt I yeah
23:10:58 mriedem or maybe we are getting something like that but the message changed in placement, idk
23:11:02 mriedem see the todo from efried
23:11:42 melwitt ok
23:12:07 melwitt so maybe I can try a patch for this and get efried to look for dragons
23:12:32 efried (I haven't been following the conversation, and need to split real soon, but sure, add me to a patch)
23:12:37 melwitt because this looks like it could be a small backportable change that would save us in both the ironic and non-ironic cases
23:15:09 melwitt thanks for the help on this
23:15:14 dansmith I'm not sure how I feel about reusing the provider by name
23:15:21 melwitt ah dammit
23:15:25 dansmith we're kinda making that a primary key because the name has to be unique,
23:15:33 dansmith but if you think about compute nodes in split brain,
23:15:38 dansmith they're going to fight over the same provider
23:15:48 dansmith and with conductor groups in ironic,
23:16:07 dansmith or two ironics and separate computes,
23:16:18 dansmith if you ended up with two nodes of the same name, nova isn't going to know they're different and is going to fight over them
23:16:29 dansmith and by fight, I mean silently overwrite inventory
23:16:43 dansmith if that provider has allocations, things are going to get all messed (the eff) up I think
23:17:10 dansmith moving the ironic node id to be the provider id is the right move (which we've already done)
23:17:32 dansmith so I dunno, maybe that means for <rocky, the name hack works, but... it's kinda contrary to the point of the provider id
23:17:45 melwitt what's the right move in the non-ironic case then?
23:17:45 dansmith melwitt: I think I hinted at this in my email on the internal thread about this
23:18:01 dansmith for the service delete?
23:18:08 dansmith that's probably even worse really,
23:18:17 melwitt I assume it's service delete. I don't know for sure how they got into that state
23:18:34 dansmith because if you end up with two computes with the same name due to some dns or dhcp breakage, they'll take over each others' providers (and allocations) which would be helzabad
23:18:37 melwitt that thread spun off from the ironic bz
23:19:22 dansmith you will have other issues if you have a name clash, obviously, but if you mix allocations from two computes together, or have one go negative because it's a smaller compute, just... hard to debug and fix
23:19:45 dansmith so anyway, I dunno
23:20:01 dansmith not saying don't do it.. glad we don't have to do it on master, but I'm not super confident that it's a great idea for <rocky either
23:20:02 melwitt ok... sigh.. so with the moving node id to be provider id, would that not require a migration step if we were able to backport to queens?
23:20:17 dansmith that only affects ironic
23:20:50 melwitt yeah sorry. going back to the ironic thing again, to fix that one in that way, would it require a migration step?
23:21:26 dansmith as I said on that thread, you'd have to make sure all the computes rolled to that change atomically, and yeah I dunno how the allocations get or got moved with that when we transitioned,
23:21:35 dansmith but not reasonable for a backport either way
23:21:49 melwitt I can and will publish the workaround steps but given the number of customers hitting it, I dunno, thinking about trying the backport
23:21:52 dansmith reversing the order of provider delete works around the ironic issue doesn't it?
23:22:01 melwitt yeah it does
23:22:25 dansmith that's reasonable, backporting the node uuid thing is not reasonable I think
23:22:38 melwitt ok. got it
23:22:55 melwitt thanks
23:24:15 melwitt the non-ironic thing I think need more information because I don't know how it got into the state. and no idea how to workaround bc they can't migrate any instances away from it because all migrations fail with ResourceProviderCreationFailed
23:24:48 melwitt and there's no heal_allocations so they can't delete the allocations and RP and restore allocations
23:24:58 dansmith well, they can
23:25:02 dansmith using osc-placement
23:25:14 dansmith I mean, you can script that for them
23:25:24 dansmith or backport heal allocations, that's much less scary I think
23:25:43 melwitt yeah, that's what I'm thinking
23:25:47 dansmith heal allocations will just fix it so you can delete everything, let it create and then heal right?
23:25:51 melwitt osc-placement I didn't see a way to update a RP with a different uuid
23:25:56 dansmith you mean the nova-manage healer thing right?
23:26:02 melwitt I do yeah
23:26:26 dansmith i.e. the sunday morning public access TV version of nova.. BAH HALED!
23:26:49 melwitt lol
23:26:58 dansmith melwitt: I mean with osc-placement you can create and delete allocations, IIRC, so you can save them off, then re-add them after it creates the provider afresh
23:27:06 melwitt I wish the command were healer, that would be more fun
23:27:20 melwitt dansmith: oh geez. yes, that's true
23:27:49 dansmith https://makeameme.org/meme/come-to-me-551mb7
23:27:49 melwitt I didn't even think about that, guh
23:28:06 melwitt lol oh I wish we could put that on the docs page for it
23:28:28 efried mriedem, sean-k-mooney, gibi: http://lists.openstack.org/pipermail/openstack-discuss/2019-November/011044.html
23:28:36 efried and I'm out
23:29:17 melwitt yeah, I'm gonna just backport heal_allocations instead because I'm just imagining support doing that on a one-by-one instance basis. oof
23:29:36 melwitt er, one by one RP
23:30:00 dansmith melwitt: you could, oh I dunno, write them a script :)
23:30:14 dansmith but whatever :)
23:30:19 melwitt oh
23:30:39 dansmith osc is like lovin up on some scriptability amirite?
23:30:45 melwitt lol
23:30:48 melwitt sure.
23:31:04 dansmith anyway, I'm very happy to leave that decision and work to you
23:31:09 dansmith just sayin'
23:31:22 melwitt don't worry, they're all assigned to e
23:31:25 melwitt *me
23:32:14 melwitt unfortunately for them
23:32:51 dansmith praise be
23:47:47 mriedem i haven't read all the way back, but regarding "if you ended up with two nodes of the same name, nova isn't going to know they're different and is going to fight over them" - with ironic that's not possible (since rocky) since there is a unique index on the compute_nodes.uuid

Earlier   Later