Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-11-21
23:17:10 dansmith moving the ironic node id to be the provider id is the right move (which we've already done)
23:17:32 dansmith so I dunno, maybe that means for <rocky, the name hack works, but... it's kinda contrary to the point of the provider id
23:17:45 melwitt what's the right move in the non-ironic case then?
23:17:45 dansmith melwitt: I think I hinted at this in my email on the internal thread about this
23:18:01 dansmith for the service delete?
23:18:08 dansmith that's probably even worse really,
23:18:17 melwitt I assume it's service delete. I don't know for sure how they got into that state
23:18:34 dansmith because if you end up with two computes with the same name due to some dns or dhcp breakage, they'll take over each others' providers (and allocations) which would be helzabad
23:18:37 melwitt that thread spun off from the ironic bz
23:19:22 dansmith you will have other issues if you have a name clash, obviously, but if you mix allocations from two computes together, or have one go negative because it's a smaller compute, just... hard to debug and fix
23:19:45 dansmith so anyway, I dunno
23:20:01 dansmith not saying don't do it.. glad we don't have to do it on master, but I'm not super confident that it's a great idea for <rocky either
23:20:02 melwitt ok... sigh.. so with the moving node id to be provider id, would that not require a migration step if we were able to backport to queens?
23:20:17 dansmith that only affects ironic
23:20:50 melwitt yeah sorry. going back to the ironic thing again, to fix that one in that way, would it require a migration step?
23:21:26 dansmith as I said on that thread, you'd have to make sure all the computes rolled to that change atomically, and yeah I dunno how the allocations get or got moved with that when we transitioned,
23:21:35 dansmith but not reasonable for a backport either way
23:21:49 melwitt I can and will publish the workaround steps but given the number of customers hitting it, I dunno, thinking about trying the backport
23:21:52 dansmith reversing the order of provider delete works around the ironic issue doesn't it?
23:22:01 melwitt yeah it does
23:22:25 dansmith that's reasonable, backporting the node uuid thing is not reasonable I think
23:22:38 melwitt ok. got it
23:22:55 melwitt thanks
23:24:15 melwitt the non-ironic thing I think need more information because I don't know how it got into the state. and no idea how to workaround bc they can't migrate any instances away from it because all migrations fail with ResourceProviderCreationFailed
23:24:48 melwitt and there's no heal_allocations so they can't delete the allocations and RP and restore allocations
23:24:58 dansmith well, they can
23:25:02 dansmith using osc-placement
23:25:14 dansmith I mean, you can script that for them
23:25:24 dansmith or backport heal allocations, that's much less scary I think
23:25:43 melwitt yeah, that's what I'm thinking
23:25:47 dansmith heal allocations will just fix it so you can delete everything, let it create and then heal right?
23:25:51 melwitt osc-placement I didn't see a way to update a RP with a different uuid
23:25:56 dansmith you mean the nova-manage healer thing right?
23:26:02 melwitt I do yeah
23:26:26 dansmith i.e. the sunday morning public access TV version of nova.. BAH HALED!
23:26:49 melwitt lol
23:26:58 dansmith melwitt: I mean with osc-placement you can create and delete allocations, IIRC, so you can save them off, then re-add them after it creates the provider afresh
23:27:06 melwitt I wish the command were healer, that would be more fun
23:27:20 melwitt dansmith: oh geez. yes, that's true
23:27:49 dansmith https://makeameme.org/meme/come-to-me-551mb7
23:27:49 melwitt I didn't even think about that, guh
23:28:06 melwitt lol oh I wish we could put that on the docs page for it
23:28:28 efried mriedem, sean-k-mooney, gibi: http://lists.openstack.org/pipermail/openstack-discuss/2019-November/011044.html
23:28:36 efried and I'm out
23:29:17 melwitt yeah, I'm gonna just backport heal_allocations instead because I'm just imagining support doing that on a one-by-one instance basis. oof
23:29:36 melwitt er, one by one RP
23:30:00 dansmith melwitt: you could, oh I dunno, write them a script :)
23:30:14 dansmith but whatever :)
23:30:19 melwitt oh
23:30:39 dansmith osc is like lovin up on some scriptability amirite?
23:30:45 melwitt lol
23:30:48 melwitt sure.
23:31:04 dansmith anyway, I'm very happy to leave that decision and work to you
23:31:09 dansmith just sayin'
23:31:22 melwitt don't worry, they're all assigned to e
23:31:25 melwitt *me
23:32:14 melwitt unfortunately for them
23:32:51 dansmith praise be
23:47:47 mriedem i haven't read all the way back, but regarding "if you ended up with two nodes of the same name, nova isn't going to know they're different and is going to fight over them" - with ironic that's not possible (since rocky) since there is a unique index on the compute_nodes.uuid
23:49:52 melwitt yeah we were talking queens and non-ironic as well
23:50:14 melwitt I was trying to get together a game plan for each case
23:51:05 mriedem well, just give this to whoever? https://docs.openstack.org/nova/latest/admin/troubleshooting/orphaned-allocations.html
23:51:25 mriedem i wrote that b/c of all of this
23:51:46 melwitt we don't have heal_allocations yet. but I'm gonna backport it after this convo
23:51:49 mriedem help with my service delete backports
23:51:50 melwitt in queens
23:51:58 eandersson Awesome
23:51:58 mriedem and then review my wip thing which is the last piece
23:52:08 mriedem eandersson: i think she's talking about internal to rhosp
23:52:09 mriedem so you're sol
23:52:35 eandersson awh
23:52:38 melwitt yeah, I have to get these things unwedged first
23:52:56 mriedem sylvain has this audit thing he's been working on for about 3 years as well
23:53:00 melwitt and I don't know for sure whether service delete caused the issue, but I think there's a fair chance
23:53:03 mriedem between contract negotiations and skiing
23:53:24 eandersson tbh we are almost always backporting these things ourselfs, but much better to have it officially backported :p
23:53:37 mriedem eandersson: you know you could propose backports *upstream*
23:53:49 eandersson So much work to do, so little time :D
23:54:02 mriedem or at least be like, "can you guys backport x because that would be gr8 lol"
23:54:04 eandersson I am still top1 of lines contributed for U :D
23:54:11 melwitt yeah, just hang on, I got involved with these bugs within the last couple of weeks. I didn't understand them before
23:54:15 mriedem if operators speak up about needing shit we usually jump on it a bit faster
23:54:43 mriedem did you know belmiro has a dedicated line to dansmith's office?
23:55:10 eandersson I tend to backport things, but a lot of the nova backports are high complexity due to the number of changes between master and stable branches.
23:55:56 eandersson So a lot of my effort goes into projects I already understand (e.g. designate, senlin etc).
23:56:04 eandersson But I do try to report them here =]
23:56:06 melwitt all of the compute/service/host/node/RP/allocation intermingling stuff has not been my area of expertise
23:56:26 mriedem it's confusing
23:57:00 melwitt now that I have some idea wtf is going on, sure I will help with the backports and all that
23:57:34 eandersson Sometimes I wish I wasn't a manager. Would give me more time behind the keyboard. =]
#openstack-nova - 2019-11-22
00:25:42 mriedem i've seen your minions in here
00:26:04 mriedem but yes TPS reports are important
00:42:12 mriedem artom: i have a gift for you
00:42:38 mriedem and it's not even this https://www.youtube.com/watch?v=GqY3bS6v6Wc
00:43:59 artom Back pocket then?
00:46:09 mriedem no, this: https://review.opendev.org/695609
00:47:09 mriedem oh gdi
00:47:25 artom I think you missed 'git add <new file>'
00:47:35 mriedem yup, pushing
00:47:56 mriedem gerritbot must also be dead

Earlier   Later