| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-11-26 | |||
| 16:29:45 | sean-k-mooney | currently if you enable the discover host perodic task in more then one schdluer it can get duplict key error form the db | |
| 16:29:47 | bauzas | are we speaking of concurrent nova-scheduler services that need to be HA active-active for X reasons ? | |
| 16:29:54 | sean-k-mooney | as 2 process can race to create the mappings | |
| 16:29:59 | sean-k-mooney | leading to errors in the logs | |
| 16:30:04 | sean-k-mooney | we dont actully supprot that today | |
| 16:30:09 | bauzas | I think we always said that nova-scheduler has to be active-passive | |
| 16:30:13 | sean-k-mooney | but our documention on that is kind of lacking | |
| 16:30:17 | sean-k-mooney | no | |
| 16:30:26 | bauzas | I pretty bet we documented it | |
| 16:30:28 | sean-k-mooney | the schdluer has been supproted in active active for a very long time | |
| 16:30:31 | bauzas | nevre | |
| 16:30:35 | sean-k-mooney | yes | |
| 16:31:03 | tkajinam | as far as I can tell Tripleo in the past deployed it in all controllers | |
| 16:31:12 | bauzas | with placement, we thought that we /could/ run it active-active but there were reasons not to | |
| 16:31:21 | sean-k-mooney | nope | |
| 16:31:26 | bauzas | tkajinam: which was a bug that we raised a couple of times | |
| 16:31:33 | sean-k-mooney | downstream its been active active since like 16 maybe before | |
| 16:31:41 | bauzas | and I think TripleO changed it to A-P | |
| 16:31:49 | bauzas | for that exact reason | |
| 16:31:53 | sean-k-mooney | nope | |
| 16:31:56 | tkajinam | no | |
| 16:32:22 | sean-k-mooney | ok well i think we need a longer discussion on this RFE request | |
| 16:32:36 | sean-k-mooney | likely a spec and we probly dont have time to complete it in epoxy | |
| 16:32:50 | sean-k-mooney | but we shoudl dicuss this more async | |
| 16:33:22 | s3rj1k | no prob, thanks sean-k-mooney for taking a lead on explaining | |
| 16:34:28 | bauzas | I have to admit that none of that tribal knowledge is written in https://docs.openstack.org/nova/latest/admin/scheduling.html | |
| 16:34:43 | sean-k-mooney | its also not in the config option | |
| 16:35:01 | sean-k-mooney | i left my inital feedback on the bug when i traged it as opion | |
| 16:35:30 | sean-k-mooney | i dint make it as invlaid as i tought we shoudl atelast dicuss it more widely first | |
| 16:36:01 | bauzas | for now, we should document that active-passive HA configuration for sure | |
| 16:36:16 | sean-k-mooney | for the perodic only | |
| 16:36:19 | bauzas | because indeed, we know that there is no eventual consistency betwen schedulers | |
| 16:36:27 | sean-k-mooney | the schduler shoudl be generally deploy active active | |
| 16:36:42 | bauzas | that's your opinion :) | |
| 16:36:44 | sean-k-mooney | but also the perodic has perfomance issuues | |
| 16:36:50 | sean-k-mooney | bauzas: its waht we use in our product | |
| 16:36:58 | sean-k-mooney | and what almost all instller do by defult | |
| 16:37:42 | bauzas | https://specs.openstack.org/openstack/nova-specs/specs/abandoned/parallel-scheduler.html | |
| 16:37:53 | tkajinam | yeah > almost all installer do by default | |
| 16:38:11 | sean-k-mooney | that a diffent proposal | |
| 16:38:36 | bauzas | I litterally quote the first sentence of that spec : | |
| 16:38:42 | bauzas | "If you running two nova-scheduler processes they race each other, they don’t find out about each others choices until the DB gets updated by the nova-compute resource tracker. This has lead to many deployments opting for an Active/Passive HA setup for the nova-scheduler process." | |
| 16:39:10 | tkajinam | people may not prefer using act-act for simplicity and avoid clustering mechanism to implement active-passive. | |
| 16:39:24 | tkajinam | without large warning :-P | |
| 16:39:31 | sean-k-mooney | bauzas: that does not really apply as of placement | |
| 16:39:55 | sean-k-mooney | bauzas: i woudl condier it to be very incorrect advice to deocument that active active is not supported | |
| 16:40:50 | gibi | yeah the goal of placement to shrink the race window between parallel schedulers | |
| 16:41:07 | gibi | it is a solved problem for those resources that are tracked in placement | |
| 16:41:13 | bauzas | I don't disagree with the fact that HA active-active schedulers is a problem to solve | |
| 16:41:30 | gibi | for those that are not tracked there, the compute manager has a lock around claim to prevent overallocation | |
| 16:41:36 | gibi | and we have alternatives to reschedule | |
| 16:41:50 | bauzas | gibi: exactly, hence the A/P mechanism | |
| 16:41:57 | gibi | no this is A A | |
| 16:42:10 | gibi | the only A P problem is in the periodic discovery | |
| 16:42:13 | bauzas | in the very early times, we were considering reschedules as a way to address the problem | |
| 16:42:38 | bauzas | we stopped that tenet by wanting to reduce the reschedules, leading to indeed a broader problem | |
| 16:43:10 | gibi | we reduced reschedules with placement | |
| 16:43:16 | bauzas | originally, the scheduler wasn't intended to provide an exact solution | |
| 16:43:21 | gibi | and we improved reschedules with alterntive generation | |
| 16:43:30 | bauzas | right, which is why we never solved that problem | |
| 16:43:46 | bauzas | we reduced the scope of reschedules, that's it | |
| 16:43:49 | sean-k-mooney | we solved it to the point that we recomemnd active active as the defualt | |
| 16:43:50 | gibi | in a distributes system you have limits what you can solve exactly | |
| 16:44:11 | gibi | I agree with sean-k-mooney we can recomend A A | |
| 16:44:19 | gibi | actually OSP 18 does A A A | |
| 16:44:43 | sean-k-mooney | right our product does not supprot active passive but i belive that was true in 17 as well | |
| 16:44:43 | gibi | (or as many As as you want :D) | |
| 16:44:45 | bauzas | A A A is OK to me with resources tracked by placement | |
| 16:45:16 | sean-k-mooney | anyway perhasp we should move on? | |
| 16:45:23 | bauzas | agreed | |
| 16:45:28 | sean-k-mooney | we can talk about this more but proably dont need to in the meeting | |
| 16:45:32 | bauzas | and agreed on the fact we need a spec | |
| 16:45:43 | bauzas | but maybe the solution is to add more resources to placement | |
| 16:46:01 | sean-k-mooney | well that is the general direction anyway | |
| 16:46:04 | bauzas | or consider this as a non-solvable problem and accepting reschedules as a caveart | |
| 16:46:07 | bauzas | caveat | |
| 16:46:11 | sean-k-mooney | but that does not adress the reporte problem | |
| 16:46:17 | gibi | on the proposal of a distributed discover I can suggest to do the discover outside of a scheduler periodic to avoid the race | |
| 16:46:19 | sean-k-mooney | nova-audit would | |
| 16:46:28 | bauzas | anyway, moving on | |
| 16:46:31 | sean-k-mooney | gibi: yes its very diffent | |
| 16:46:41 | sean-k-mooney | bauzas: ack so i had one quick topic | |
| 16:46:41 | bauzas | s3rj1k: fancy writing a spec ? | |
| 16:46:47 | bauzas | sean-k-mooney: shoot | |
| 16:46:51 | s3rj1k | gibi: similar issue would be with CLI, check out RFE | |
| 16:47:05 | s3rj1k | bauzas: will do | |
| 16:47:12 | sean-k-mooney | so last week i raised adding rodolfo to os-vif core | |
| 16:47:23 | sean-k-mooney | i sent a mail to the list and no one objected | |
| 16:47:24 | gibi | s3rj1k: I mean if you control to only run the discover from a single CLI session at a time then I assume there is no race | |
| 16:47:37 | sean-k-mooney | so if there is no other objection here i will proceed with that after the call. | |
| 16:48:01 | s3rj1k | gibi: yes, need external control on how CLI gets run | |
| 16:48:16 | s3rj1k | lets move on, yes | |
| 16:48:31 | gibi | sean-k-mooney: no objection on my side | |
| 16:48:51 | bauzas | sean-k-mooney: no objections indeed | |
| 16:49:17 | sean-k-mooney | ack so that is all i had | |
| 16:49:36 | tkajinam | I have no objections but +1 :-) (I'm not a core, though) | |
| 16:50:18 | sean-k-mooney | ill send a mail to the list and then ill add them after that | |
| 16:50:37 | sean-k-mooney | jsut to keep a record of it beyond this meeting | |
| 16:51:59 | bauzas | ++ | |
| 16:52:07 | bauzas | okay, then I think we're done for today | |