| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-11-21 | |||
| 23:15:14 | dansmith | I'm not sure how I feel about reusing the provider by name | |
| 23:15:21 | melwitt | ah dammit | |
| 23:15:25 | dansmith | we're kinda making that a primary key because the name has to be unique, | |
| 23:15:33 | dansmith | but if you think about compute nodes in split brain, | |
| 23:15:38 | dansmith | they're going to fight over the same provider | |
| 23:15:48 | dansmith | and with conductor groups in ironic, | |
| 23:16:07 | dansmith | or two ironics and separate computes, | |
| 23:16:18 | dansmith | if you ended up with two nodes of the same name, nova isn't going to know they're different and is going to fight over them | |
| 23:16:29 | dansmith | and by fight, I mean silently overwrite inventory | |
| 23:16:43 | dansmith | if that provider has allocations, things are going to get all messed (the eff) up I think | |
| 23:17:10 | dansmith | moving the ironic node id to be the provider id is the right move (which we've already done) | |
| 23:17:32 | dansmith | so I dunno, maybe that means for <rocky, the name hack works, but... it's kinda contrary to the point of the provider id | |
| 23:17:45 | melwitt | what's the right move in the non-ironic case then? | |
| 23:17:45 | dansmith | melwitt: I think I hinted at this in my email on the internal thread about this | |
| 23:18:01 | dansmith | for the service delete? | |
| 23:18:08 | dansmith | that's probably even worse really, | |
| 23:18:17 | melwitt | I assume it's service delete. I don't know for sure how they got into that state | |
| 23:18:34 | dansmith | because if you end up with two computes with the same name due to some dns or dhcp breakage, they'll take over each others' providers (and allocations) which would be helzabad | |
| 23:18:37 | melwitt | that thread spun off from the ironic bz | |
| 23:19:22 | dansmith | you will have other issues if you have a name clash, obviously, but if you mix allocations from two computes together, or have one go negative because it's a smaller compute, just... hard to debug and fix | |
| 23:19:45 | dansmith | so anyway, I dunno | |
| 23:20:01 | dansmith | not saying don't do it.. glad we don't have to do it on master, but I'm not super confident that it's a great idea for <rocky either | |
| 23:20:02 | melwitt | ok... sigh.. so with the moving node id to be provider id, would that not require a migration step if we were able to backport to queens? | |
| 23:20:17 | dansmith | that only affects ironic | |
| 23:20:50 | melwitt | yeah sorry. going back to the ironic thing again, to fix that one in that way, would it require a migration step? | |
| 23:21:26 | dansmith | as I said on that thread, you'd have to make sure all the computes rolled to that change atomically, and yeah I dunno how the allocations get or got moved with that when we transitioned, | |
| 23:21:35 | dansmith | but not reasonable for a backport either way | |
| 23:21:49 | melwitt | I can and will publish the workaround steps but given the number of customers hitting it, I dunno, thinking about trying the backport | |
| 23:21:52 | dansmith | reversing the order of provider delete works around the ironic issue doesn't it? | |
| 23:22:01 | melwitt | yeah it does | |
| 23:22:25 | dansmith | that's reasonable, backporting the node uuid thing is not reasonable I think | |
| 23:22:38 | melwitt | ok. got it | |
| 23:22:55 | melwitt | thanks | |
| 23:24:15 | melwitt | the non-ironic thing I think need more information because I don't know how it got into the state. and no idea how to workaround bc they can't migrate any instances away from it because all migrations fail with ResourceProviderCreationFailed | |
| 23:24:48 | melwitt | and there's no heal_allocations so they can't delete the allocations and RP and restore allocations | |
| 23:24:58 | dansmith | well, they can | |
| 23:25:02 | dansmith | using osc-placement | |
| 23:25:14 | dansmith | I mean, you can script that for them | |
| 23:25:24 | dansmith | or backport heal allocations, that's much less scary I think | |
| 23:25:43 | melwitt | yeah, that's what I'm thinking | |
| 23:25:47 | dansmith | heal allocations will just fix it so you can delete everything, let it create and then heal right? | |
| 23:25:51 | melwitt | osc-placement I didn't see a way to update a RP with a different uuid | |
| 23:25:56 | dansmith | you mean the nova-manage healer thing right? | |
| 23:26:02 | melwitt | I do yeah | |
| 23:26:26 | dansmith | i.e. the sunday morning public access TV version of nova.. BAH HALED! | |
| 23:26:49 | melwitt | lol | |
| 23:26:58 | dansmith | melwitt: I mean with osc-placement you can create and delete allocations, IIRC, so you can save them off, then re-add them after it creates the provider afresh | |
| 23:27:06 | melwitt | I wish the command were healer, that would be more fun | |
| 23:27:20 | melwitt | dansmith: oh geez. yes, that's true | |
| 23:27:49 | dansmith | https://makeameme.org/meme/come-to-me-551mb7 | |
| 23:27:49 | melwitt | I didn't even think about that, guh | |
| 23:28:06 | melwitt | lol oh I wish we could put that on the docs page for it | |
| 23:28:28 | efried | mriedem, sean-k-mooney, gibi: http://lists.openstack.org/pipermail/openstack-discuss/2019-November/011044.html | |
| 23:28:36 | efried | and I'm out | |
| 23:29:17 | melwitt | yeah, I'm gonna just backport heal_allocations instead because I'm just imagining support doing that on a one-by-one instance basis. oof | |
| 23:29:36 | melwitt | er, one by one RP | |
| 23:30:00 | dansmith | melwitt: you could, oh I dunno, write them a script :) | |
| 23:30:14 | dansmith | but whatever :) | |
| 23:30:19 | melwitt | oh | |
| 23:30:39 | dansmith | osc is like lovin up on some scriptability amirite? | |
| 23:30:45 | melwitt | lol | |
| 23:30:48 | melwitt | sure. | |
| 23:31:04 | dansmith | anyway, I'm very happy to leave that decision and work to you | |
| 23:31:09 | dansmith | just sayin' | |
| 23:31:22 | melwitt | don't worry, they're all assigned to e | |
| 23:31:25 | melwitt | *me | |
| 23:32:14 | melwitt | unfortunately for them | |
| 23:32:51 | dansmith | praise be | |
| 23:47:47 | mriedem | i haven't read all the way back, but regarding "if you ended up with two nodes of the same name, nova isn't going to know they're different and is going to fight over them" - with ironic that's not possible (since rocky) since there is a unique index on the compute_nodes.uuid | |
| 23:49:52 | melwitt | yeah we were talking queens and non-ironic as well | |
| 23:50:14 | melwitt | I was trying to get together a game plan for each case | |
| 23:51:05 | mriedem | well, just give this to whoever? https://docs.openstack.org/nova/latest/admin/troubleshooting/orphaned-allocations.html | |
| 23:51:25 | mriedem | i wrote that b/c of all of this | |
| 23:51:46 | melwitt | we don't have heal_allocations yet. but I'm gonna backport it after this convo | |
| 23:51:49 | mriedem | help with my service delete backports | |
| 23:51:50 | melwitt | in queens | |
| 23:51:58 | eandersson | Awesome | |
| 23:51:58 | mriedem | and then review my wip thing which is the last piece | |
| 23:52:08 | mriedem | eandersson: i think she's talking about internal to rhosp | |
| 23:52:09 | mriedem | so you're sol | |
| 23:52:35 | eandersson | awh | |
| 23:52:38 | melwitt | yeah, I have to get these things unwedged first | |
| 23:52:56 | mriedem | sylvain has this audit thing he's been working on for about 3 years as well | |
| 23:53:00 | melwitt | and I don't know for sure whether service delete caused the issue, but I think there's a fair chance | |
| 23:53:03 | mriedem | between contract negotiations and skiing | |
| 23:53:24 | eandersson | tbh we are almost always backporting these things ourselfs, but much better to have it officially backported :p | |
| 23:53:37 | mriedem | eandersson: you know you could propose backports *upstream* | |
| 23:53:49 | eandersson | So much work to do, so little time :D | |
| 23:54:02 | mriedem | or at least be like, "can you guys backport x because that would be gr8 lol" | |
| 23:54:04 | eandersson | I am still top1 of lines contributed for U :D | |
| 23:54:11 | melwitt | yeah, just hang on, I got involved with these bugs within the last couple of weeks. I didn't understand them before | |
| 23:54:15 | mriedem | if operators speak up about needing shit we usually jump on it a bit faster | |
| 23:54:43 | mriedem | did you know belmiro has a dedicated line to dansmith's office? | |
| 23:55:10 | eandersson | I tend to backport things, but a lot of the nova backports are high complexity due to the number of changes between master and stable branches. | |
| 23:55:56 | eandersson | So a lot of my effort goes into projects I already understand (e.g. designate, senlin etc). | |
| 23:56:04 | eandersson | But I do try to report them here =] | |
| 23:56:06 | melwitt | all of the compute/service/host/node/RP/allocation intermingling stuff has not been my area of expertise | |
| 23:56:26 | mriedem | it's confusing | |
| 23:57:00 | melwitt | now that I have some idea wtf is going on, sure I will help with the backports and all that | |
| 23:57:34 | eandersson | Sometimes I wish I wasn't a manager. Would give me more time behind the keyboard. =] | |