| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-01-13 | |||
| 14:54:07 | sean-k-mooney | regardign the versioning | |
| 14:54:24 | sean-k-mooney | if it woudl make you feel better i can just bump the version everytime we add a posible check | |
| 14:54:31 | sean-k-mooney | e.g. the minior version | |
| 14:54:38 | sean-k-mooney | that would make it more consitnet | |
| 14:54:52 | gibi | I'm good any ways iff the rules of bumping is clear | |
| 14:54:52 | sean-k-mooney | i was jsut goign to have the keys of the checks sub dictionay unversioned | |
| 14:55:01 | gibi | that is fine | |
| 14:55:03 | gibi | then lets state taht | |
| 14:55:05 | gibi | that | |
| 14:55:15 | gibi | checks is unversioned the rest is semver | |
| 14:55:39 | sean-k-mooney | well that the thing i want the value of the checks to be versioned | |
| 14:55:42 | sean-k-mooney | just not the keys | |
| 14:55:55 | sean-k-mooney | so you can rely on the format of the checks | |
| 14:56:05 | sean-k-mooney | but the set of names is open | |
| 14:56:39 | gibi | aah | |
| 14:56:47 | sean-k-mooney | if we want to version everything in the responce the only delta form what i have now is the set of check names would be versioned too | |
| 14:56:57 | sean-k-mooney | which i can do but im not sure what all of them are yet | |
| 14:57:03 | gibi | then I would vote for versioning the set of check names too | |
| 14:57:12 | sean-k-mooney | ok | |
| 14:57:27 | sean-k-mooney | ill make that update and fix the typo | |
| 14:57:38 | gibi | OK, thanks | |
| 14:57:42 | gibi | I will +2 it | |
| 15:08:46 | opendevreview | sean mooney proposed openstack/nova-specs master: add per process healthcheck spec https://review.opendev.org/c/openstack/nova-specs/+/821279 | |
| 15:08:55 | sean-k-mooney | gibi: there you go ^ | |
| 15:09:23 | sean-k-mooney | if we are generally aligned on that i might adress any other nits if there are some in a followup | |
| 15:10:00 | gibi | sean-k-mooney: I'm +2 | |
| 15:10:01 | gibi | thank you | |
| 15:10:14 | sean-k-mooney | no worries :) | |
| 15:10:37 | sean-k-mooney | i proably shoudl start implementing that soon likely next week | |
| 15:10:56 | bauzas | ++ | |
| 15:11:09 | bauzas | if nothing controversial, gibi please +2/+W it then | |
| 15:11:21 | bauzas | or I can jab this spec again | |
| 15:11:47 | sean-k-mooney | bauzas: if you can that would be great | |
| 15:12:42 | gibi | I will go back to it before I leave today and +W it if bauzas didn't yet | |
| 15:13:14 | sean-k-mooney | TheJulia: given today is spec freeze are you plannign to rework https://review.opendev.org/c/openstack/nova-specs/+/815789 | |
| 15:13:59 | bauzas | sean-k-mooney: oki doki, hold your brace | |
| 15:14:50 | TheJulia | sean-k-mooney: I didn't create that and I've never seen it before | |
| 15:15:05 | sean-k-mooney | oh its the spec for https://review.opendev.org/c/openstack/nova/+/813897 no? | |
| 15:15:16 | TheJulia | sean-k-mooney: I've been treating the issue as a bug but I've been beyond slammed as of recent | |
| 15:16:20 | sean-k-mooney | ok we might want to put this intout the compute team backlog downstream and take it over to land it next cycle | |
| 15:16:44 | sean-k-mooney | given we are chaign the api behavior and rebalaicne semantic upstream we were considering it a feature | |
| 15:17:27 | TheJulia | well, that is what it calls for | |
| 15:17:31 | TheJulia | at least, a glance at the spec | |
| 15:17:43 | TheJulia | the bug can be solved by just doing the needful and not changing the API | |
| 15:18:08 | sean-k-mooney | its not changing the api syntatically | |
| 15:18:18 | sean-k-mooney | i guess the sidefffect is not nessisaly visable | |
| 15:18:28 | sean-k-mooney | so you could arge it snot changing it semantically too | |
| 15:18:49 | TheJulia | yeah, essentially invisible unless someone goes looking for it and even then it could have been a migration in a vm context for all anyone looking at the api knows | |
| 15:19:08 | sean-k-mooney | if other are ok with this as a pure bugfix and we change your initall patch to look at the dissabled state not up/down then i coudl buy that | |
| 15:19:54 | TheJulia | I think it was already doing that, but I've litterally forgotten exactly what is in the patch since I've not been able to pull it back up since November | |
| 15:20:22 | sean-k-mooney | ack i think you didnt in v1 but ill adming i simarly dont know whats in v5 | |
| 15:20:30 | TheJulia | or disabled was contextually redundant to the data on hand or something like that | |
| 15:21:03 | sean-k-mooney | well one of the concers we hat was making sure this only happend if the operator opted in | |
| 15:21:12 | TheJulia | I was also going for keeping it as light weight on the DB as possible given multi-thousand node clusters will make things cry regardless | |
| 15:21:15 | sean-k-mooney | and the way to do tha was to delete the failed service or disable it | |
| 15:21:40 | TheJulia | Yeah, and I think the operator opt-in is where I never got into | |
| 15:22:06 | TheJulia | opt-in in general is kind of crap for our end users since they need to be aware and otherwise they are exposed to... not great behavior otherwise | |
| 15:22:40 | sean-k-mooney | right but arbitary chaning the isntace.host effectivly at anytime is not somethign we wanted to allow either | |
| 15:22:59 | sean-k-mooney | we were more or less ok with it in a failure or mantaince event | |
| 15:23:27 | TheJulia | it is not actually that arbitrary, it would be during a hash ring rebalance which is not an arbitrary event in the first place | |
| 15:24:00 | TheJulia | which would have been triggered by a failure or maintenance event | |
| 15:24:43 | sean-k-mooney | true im a little worried about what happens in the case of a network partion but ill try and review this again and see wht athe current patch does | |
| 15:24:46 | TheJulia | maybe we're using the same words with different causes in mind | |
| 15:25:16 | TheJulia | computes don't directly interact with the db do they? | |
| 15:25:25 | TheJulia | in terms of open socket to it ? | |
| 15:25:26 | sean-k-mooney | no they connect via the conductor | |
| 15:25:51 | sean-k-mooney | so they make rpcs which the conductor the exectues against the db | |
| 15:25:58 | TheJulia | so whatever has conductor connectivity would win and ultimately the partitioned cluster would cease to really function but still be able to carry the requests if they get them via another route | |
| 15:26:26 | TheJulia | that is as long as it doesn't hit the db | |
| 15:26:30 | sean-k-mooney | well the case i was thinking of is one of the compute cant conenct to rabbit teperaly and the heartbeat expires | |
| 15:26:33 | TheJulia | at which point deadlocked compute process | |
| 15:26:34 | sean-k-mooney | it gets marked as down | |
| 15:26:45 | sean-k-mooney | you starte to reblance and it heartbeats and is marked as up | |
| 15:27:00 | TheJulia | well, the object access is over rabbit yes? | |
| 15:27:01 | sean-k-mooney | we dont want to continually reblance in that case when its flapping | |
| 15:27:36 | TheJulia | wouldn't the process just halt on the db queries through rabbit in that case? | |
| 15:27:52 | TheJulia | since they are remoted objects | |
| 15:28:04 | sean-k-mooney | it depnes on what happening | |
| 15:28:36 | sean-k-mooney | btu the compute serivce if the rabbit connection drop woudl continue runnign normally and then if it needed to do somethign with the conductor it would retry | |
| 15:29:42 | sean-k-mooney | if the connection to rabbit is restablished the service will continute to work as normal | |
| 15:29:54 | TheJulia | yes, but if we're pulling a list of things or even a reference of an object through rabbit to satisfy even the existing logic, then wouldn't that compute be stuck at that point anyway until rabbit connectivity is re-established and new requests can be picked up off the message bus? | |
| 15:30:54 | TheJulia | yes, I think we're on the same about connectivity with rabbit | |
| 15:30:59 | TheJulia | err, on the same page | |
| 15:31:18 | sean-k-mooney | so ideally if this drops and reconnects in the space of say 5 seconds | |
| 15:31:27 | sean-k-mooney | that shoudl not trigger a rebalnce | |
| 15:31:51 | sean-k-mooney | with the patch as wrttien i think we coudl get unlucky in that case yes? | |
| 15:31:54 | TheJulia | no, is the alive balance a remote object save to the db every 30 seconds? | |
| 15:32:18 | sean-k-mooney | am yes i think that is the default interval | |
| 15:32:39 | TheJulia | so loosing connectivity briefly wouldn't cause the machine to drop from the list returned from the db | |
| 15:32:50 | TheJulia | unless the conductor is watching like a hawk and overriding the table contents | |
| 15:33:04 | sean-k-mooney | in the gneral case yes. if its a littel more protacted then it would | |
| 15:33:24 | TheJulia | yes, but at which point that compute is out of service or unable to be used | |
| 15:34:10 | sean-k-mooney | yep | |
| 15:34:11 | TheJulia | I know in ironic that we don't drop it from the working list until after 3 failures have occured, but I'd need to check the code in nova | |
| 15:34:24 | sean-k-mooney | i think its similar | |
| 15:34:39 | sean-k-mooney | we dont mark it ad down until multiple heart beats are missed | |
| 15:34:39 | TheJulia | which means it wouldn't be disqualified I think until 90 seconds have actually passed | |
| 15:34:42 | sean-k-mooney | i think its also 3 | |
| 15:34:52 | sean-k-mooney | ya | |
| 15:35:00 | sean-k-mooney | how long does rebalancing typically take | |