| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-01-13 | |||
| 15:13:59 | bauzas | sean-k-mooney: oki doki, hold your brace | |
| 15:14:50 | TheJulia | sean-k-mooney: I didn't create that and I've never seen it before | |
| 15:15:05 | sean-k-mooney | oh its the spec for https://review.opendev.org/c/openstack/nova/+/813897 no? | |
| 15:15:16 | TheJulia | sean-k-mooney: I've been treating the issue as a bug but I've been beyond slammed as of recent | |
| 15:16:20 | sean-k-mooney | ok we might want to put this intout the compute team backlog downstream and take it over to land it next cycle | |
| 15:16:44 | sean-k-mooney | given we are chaign the api behavior and rebalaicne semantic upstream we were considering it a feature | |
| 15:17:27 | TheJulia | well, that is what it calls for | |
| 15:17:31 | TheJulia | at least, a glance at the spec | |
| 15:17:43 | TheJulia | the bug can be solved by just doing the needful and not changing the API | |
| 15:18:08 | sean-k-mooney | its not changing the api syntatically | |
| 15:18:18 | sean-k-mooney | i guess the sidefffect is not nessisaly visable | |
| 15:18:28 | sean-k-mooney | so you could arge it snot changing it semantically too | |
| 15:18:49 | TheJulia | yeah, essentially invisible unless someone goes looking for it and even then it could have been a migration in a vm context for all anyone looking at the api knows | |
| 15:19:08 | sean-k-mooney | if other are ok with this as a pure bugfix and we change your initall patch to look at the dissabled state not up/down then i coudl buy that | |
| 15:19:54 | TheJulia | I think it was already doing that, but I've litterally forgotten exactly what is in the patch since I've not been able to pull it back up since November | |
| 15:20:22 | sean-k-mooney | ack i think you didnt in v1 but ill adming i simarly dont know whats in v5 | |
| 15:20:30 | TheJulia | or disabled was contextually redundant to the data on hand or something like that | |
| 15:21:03 | sean-k-mooney | well one of the concers we hat was making sure this only happend if the operator opted in | |
| 15:21:12 | TheJulia | I was also going for keeping it as light weight on the DB as possible given multi-thousand node clusters will make things cry regardless | |
| 15:21:15 | sean-k-mooney | and the way to do tha was to delete the failed service or disable it | |
| 15:21:40 | TheJulia | Yeah, and I think the operator opt-in is where I never got into | |
| 15:22:06 | TheJulia | opt-in in general is kind of crap for our end users since they need to be aware and otherwise they are exposed to... not great behavior otherwise | |
| 15:22:40 | sean-k-mooney | right but arbitary chaning the isntace.host effectivly at anytime is not somethign we wanted to allow either | |
| 15:22:59 | sean-k-mooney | we were more or less ok with it in a failure or mantaince event | |
| 15:23:27 | TheJulia | it is not actually that arbitrary, it would be during a hash ring rebalance which is not an arbitrary event in the first place | |
| 15:24:00 | TheJulia | which would have been triggered by a failure or maintenance event | |
| 15:24:43 | sean-k-mooney | true im a little worried about what happens in the case of a network partion but ill try and review this again and see wht athe current patch does | |
| 15:24:46 | TheJulia | maybe we're using the same words with different causes in mind | |
| 15:25:16 | TheJulia | computes don't directly interact with the db do they? | |
| 15:25:25 | TheJulia | in terms of open socket to it ? | |
| 15:25:26 | sean-k-mooney | no they connect via the conductor | |
| 15:25:51 | sean-k-mooney | so they make rpcs which the conductor the exectues against the db | |
| 15:25:58 | TheJulia | so whatever has conductor connectivity would win and ultimately the partitioned cluster would cease to really function but still be able to carry the requests if they get them via another route | |
| 15:26:26 | TheJulia | that is as long as it doesn't hit the db | |
| 15:26:30 | sean-k-mooney | well the case i was thinking of is one of the compute cant conenct to rabbit teperaly and the heartbeat expires | |
| 15:26:33 | TheJulia | at which point deadlocked compute process | |
| 15:26:34 | sean-k-mooney | it gets marked as down | |
| 15:26:45 | sean-k-mooney | you starte to reblance and it heartbeats and is marked as up | |
| 15:27:00 | TheJulia | well, the object access is over rabbit yes? | |
| 15:27:01 | sean-k-mooney | we dont want to continually reblance in that case when its flapping | |
| 15:27:36 | TheJulia | wouldn't the process just halt on the db queries through rabbit in that case? | |
| 15:27:52 | TheJulia | since they are remoted objects | |
| 15:28:04 | sean-k-mooney | it depnes on what happening | |
| 15:28:36 | sean-k-mooney | btu the compute serivce if the rabbit connection drop woudl continue runnign normally and then if it needed to do somethign with the conductor it would retry | |
| 15:29:42 | sean-k-mooney | if the connection to rabbit is restablished the service will continute to work as normal | |
| 15:29:54 | TheJulia | yes, but if we're pulling a list of things or even a reference of an object through rabbit to satisfy even the existing logic, then wouldn't that compute be stuck at that point anyway until rabbit connectivity is re-established and new requests can be picked up off the message bus? | |
| 15:30:54 | TheJulia | yes, I think we're on the same about connectivity with rabbit | |
| 15:30:59 | TheJulia | err, on the same page | |
| 15:31:18 | sean-k-mooney | so ideally if this drops and reconnects in the space of say 5 seconds | |
| 15:31:27 | sean-k-mooney | that shoudl not trigger a rebalnce | |
| 15:31:51 | sean-k-mooney | with the patch as wrttien i think we coudl get unlucky in that case yes? | |
| 15:31:54 | TheJulia | no, is the alive balance a remote object save to the db every 30 seconds? | |
| 15:32:18 | sean-k-mooney | am yes i think that is the default interval | |
| 15:32:39 | TheJulia | so loosing connectivity briefly wouldn't cause the machine to drop from the list returned from the db | |
| 15:32:50 | TheJulia | unless the conductor is watching like a hawk and overriding the table contents | |
| 15:33:04 | sean-k-mooney | in the gneral case yes. if its a littel more protacted then it would | |
| 15:33:24 | TheJulia | yes, but at which point that compute is out of service or unable to be used | |
| 15:34:10 | sean-k-mooney | yep | |
| 15:34:11 | TheJulia | I know in ironic that we don't drop it from the working list until after 3 failures have occured, but I'd need to check the code in nova | |
| 15:34:24 | sean-k-mooney | i think its similar | |
| 15:34:39 | TheJulia | which means it wouldn't be disqualified I think until 90 seconds have actually passed | |
| 15:34:39 | sean-k-mooney | we dont mark it ad down until multiple heart beats are missed | |
| 15:34:42 | sean-k-mooney | i think its also 3 | |
| 15:34:52 | sean-k-mooney | ya | |
| 15:35:00 | sean-k-mooney | how long does rebalancing typically take | |
| 15:35:17 | sean-k-mooney | is it second or minutes? i assume the former | |
| 15:35:39 | TheJulia | seconds for the actual mapping at worst, the actual updates and iteration through is the painful part | |
| 15:35:58 | sean-k-mooney | without using disables as a distiubed lock | |
| 15:36:11 | sean-k-mooney | how do you prevent the compute when it compes back form racing iwth the reblance | |
| 15:36:19 | sean-k-mooney | that was the other usecase for it | |
| 15:36:25 | TheJulia | I don't think we've ever measured a startup rebalance on a large site in ironic but I've generally heard of a couple minutes | |
| 15:37:15 | TheJulia | sean-k-mooney: the code, as I remember it would check to see if the prior compute is back and then breakout of the rebalanance apply loop if so | |
| 15:37:50 | TheJulia | since surely in a little bit, it was going to undo some of what it just did since the hash ring is back to what it was prior to the failure | |
| 15:38:08 | sean-k-mooney | that makes sense | |
| 15:38:26 | TheJulia | a "oh, the universe just changed on us. stop what we're doing!" | |
| 15:38:31 | TheJulia | check | |
| 15:38:54 | sean-k-mooney | yep but it need to also roleback so liekly everthing need to be in one big transaction | |
| 15:39:07 | sean-k-mooney | so we either get the rebalanced sate or the old state | |
| 15:43:48 | TheJulia | a giant transaction would acutally harm the ability to understand what is going on because it also checks current record state so if that is effectively hiding in a giant transaction which will take a minute or two to commit, then we're introducing ourselves to more issues | |
| 15:44:15 | sean-k-mooney | it should not take minutes to commit | |
| 15:44:32 | sean-k-mooney | we shoudl just compute the desired endstate then comiit the change in one trasaction | |
| 15:44:47 | TheJulia | a couple thousand specific column field updates in a heavily used table? | |
| 15:44:48 | sean-k-mooney | you just need to use a case statement to update the instance.host on all instance in one go | |
| 15:45:02 | TheJulia | eh... then again performance has changed drastically since the days I did that often manually | |
| 15:46:03 | sean-k-mooney | so i think its just one colume on one db table with many rows | |
| 15:46:11 | sean-k-mooney | that shoudl be quick | |
| 15:46:40 | TheJulia | eh, last time I did something like that manually like 4 thousand rows in a 100k row table it was a couple minutes, but again, that was like a decade ago | |
| 15:47:48 | TheJulia | Anyway, I'd need to double check a few things but a transaction, at least to me seems more harmful in that the compute node's rebalance is hidden state so we could get a bunch of conflicting transactions pile up, all fail past the first one, and trash to try and correct some of it as time goes on consistency wise | |
| 15:49:41 | sean-k-mooney | well if we dont have a transaction you will need to role back the partially reblanced state | |
| 15:49:52 | sean-k-mooney | manually right | |
| 15:50:07 | sean-k-mooney | or trigger a full rebalnace again | |
| 15:50:18 | sean-k-mooney | to even out the load on the compute services | |
| 15:50:45 | opendevreview | Merged openstack/nova-specs master: add per process healthcheck spec https://review.opendev.org/c/openstack/nova-specs/+/821279 | |
| 15:52:32 | TheJulia | well... a transaction could work in try to generate the updates and then only check if things have changed before committing it if things have not changed, but I'd need to dig into the dbapi internals since to do so likely sounds like rpc functionality would be required to batch it all up properly outside of the normal transactions oslo_db drives | |
| 15:52:40 | TheJulia | since we're doing it one at a time instead of a bulk change | |
| 15:54:03 | sean-k-mooney | we would need to have a db method for this yes not just loop over the isntance set the filed and call save | |
| 15:54:09 | TheJulia | I guess I'm worried about trying to overthink it and then over engineer it, or if a model of eventual consistency is a happier place to be. I'm very much on the eventual consistency mindset since for ironic, it *really* doesn't matter which node proxies the request as long as it is online | |
| 15:54:51 | TheJulia | it all matters for rpc request routing ultimately | |
| 15:54:52 | sean-k-mooney | i wonder if for ironic it would be better to have a single shared topic queue | |
| 15:55:16 | TheJulia | what would that change/gain us in this situation? | |