| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-11-26 | |||
| 16:17:28 | fwiesel | Sorry , I am here | |
| 16:17:31 | bauzas | ah | |
| 16:17:41 | bauzas | anything to raise from your side ? | |
| 16:17:59 | fwiesel | There was a regression in oslo.utils (master) and I have created a change to fix it: https://review.opendev.org/c/openstack/oslo.utils/+/936247 | |
| 16:18:23 | fwiesel | Hopefully the builds will back to the two failures and I will tackle these then. | |
| 16:18:26 | fwiesel | That's from my side | |
| 16:18:28 | sean-k-mooney | ah is that related to removign netifaces | |
| 16:18:55 | bauzas | okay, gtk | |
| 16:18:57 | bauzas | thanks | |
| 16:19:16 | tkajinam | get that in | |
| 16:19:23 | bauzas | nice, thanks tkajinam | |
| 16:19:24 | tkajinam | fwiesel, if you need a new release with the fix early then ping me | |
| 16:19:36 | tkajinam | once that is merged | |
| 16:20:01 | fwiesel | tkajinam: Thanks, I'll let you know | |
| 16:20:39 | bauzas | cool | |
| 16:20:47 | bauzas | then moving to the last item from the agenda | |
| 16:20:54 | bauzas | #topic Open discussion | |
| 16:21:01 | bauzas | anything in the agenda, so anything anyone ? | |
| 16:21:11 | s3rj1k | there is this https://bugs.launchpad.net/nova/+bug/2089386 | |
| 16:21:34 | sean-k-mooney | i have one followup form last week too | |
| 16:21:48 | sean-k-mooney | lets start with s3rj1k topic | |
| 16:22:29 | bauzas | ok, s3rj1k, shoot | |
| 16:23:30 | s3rj1k | idea is to allow for host discovery to be concurrent, both cli and internal using distributed locking | |
| 16:24:25 | sean-k-mooney | so perhaps i can provide some context | |
| 16:24:46 | s3rj1k | thi mostly needed for k8s like envs where discovery is run in multiple places | |
| 16:24:47 | sean-k-mooney | s3rj1k is interested in using the discover hsost perodic in a ha env | |
| 16:25:09 | bauzas | s3rj1k: I think that topic requires a proper discussion that can't be done during a meeting | |
| 16:25:18 | sean-k-mooney | currently we require that if you use the perodic its enabled on at most one host | |
| 16:25:29 | sean-k-mooney | they would like to adress that pain point | |
| 16:25:35 | bauzas | if we want to discuss about the design, it has to be an async conversation that has to be in a proper formatted document | |
| 16:26:04 | bauzas | that's the reason why we introduced our specification program for those kind of feature requests | |
| 16:26:14 | s3rj1k | bauzas: spec? or rfe is enough for this time? | |
| 16:26:30 | sean-k-mooney | so this would defneitlly be a spec if you were going to work on it | |
| 16:26:36 | bauzas | s3rj1k: are you familiar to the specs writing or do you need guidance ? | |
| 16:27:03 | s3rj1k | bauzas: done one for neutron, so all ok | |
| 16:27:11 | sean-k-mooney | i think before going that far however s3rj1k wanted some intiall feedback on is this in scope of nova to fix | |
| 16:27:44 | bauzas | sean-k-mooney: well, I'm not sure we have a quorum today for such design discussion | |
| 16:28:10 | bauzas | if that was something before the PTG, we would have said "sure, just add that to the PTG and we'll discuss it" | |
| 16:28:24 | sean-k-mooney | thats still an option | |
| 16:28:37 | sean-k-mooney | i suggested that s3rj1k bring it here to advertise that it exist | |
| 16:28:47 | bauzas | honestly, I haven't yet formally written the nova deadlines for Epoxy but we're already running short in tome | |
| 16:28:49 | sean-k-mooney | and then start eithe r a mailing list or spec dicussion after that | |
| 16:28:52 | bauzas | time* | |
| 16:29:19 | bauzas | what exact problem are we trying to solve then ? | |
| 16:29:45 | sean-k-mooney | currently if you enable the discover host perodic task in more then one schdluer it can get duplict key error form the db | |
| 16:29:47 | bauzas | are we speaking of concurrent nova-scheduler services that need to be HA active-active for X reasons ? | |
| 16:29:54 | sean-k-mooney | as 2 process can race to create the mappings | |
| 16:29:59 | sean-k-mooney | leading to errors in the logs | |
| 16:30:04 | sean-k-mooney | we dont actully supprot that today | |
| 16:30:09 | bauzas | I think we always said that nova-scheduler has to be active-passive | |
| 16:30:13 | sean-k-mooney | but our documention on that is kind of lacking | |
| 16:30:17 | sean-k-mooney | no | |
| 16:30:26 | bauzas | I pretty bet we documented it | |
| 16:30:28 | sean-k-mooney | the schdluer has been supproted in active active for a very long time | |
| 16:30:31 | bauzas | nevre | |
| 16:30:35 | sean-k-mooney | yes | |
| 16:31:03 | tkajinam | as far as I can tell Tripleo in the past deployed it in all controllers | |
| 16:31:12 | bauzas | with placement, we thought that we /could/ run it active-active but there were reasons not to | |
| 16:31:21 | sean-k-mooney | nope | |
| 16:31:26 | bauzas | tkajinam: which was a bug that we raised a couple of times | |
| 16:31:33 | sean-k-mooney | downstream its been active active since like 16 maybe before | |
| 16:31:41 | bauzas | and I think TripleO changed it to A-P | |
| 16:31:49 | bauzas | for that exact reason | |
| 16:31:53 | sean-k-mooney | nope | |
| 16:31:56 | tkajinam | no | |
| 16:32:22 | sean-k-mooney | ok well i think we need a longer discussion on this RFE request | |
| 16:32:36 | sean-k-mooney | likely a spec and we probly dont have time to complete it in epoxy | |
| 16:32:50 | sean-k-mooney | but we shoudl dicuss this more async | |
| 16:33:22 | s3rj1k | no prob, thanks sean-k-mooney for taking a lead on explaining | |
| 16:34:28 | bauzas | I have to admit that none of that tribal knowledge is written in https://docs.openstack.org/nova/latest/admin/scheduling.html | |
| 16:34:43 | sean-k-mooney | its also not in the config option | |
| 16:35:01 | sean-k-mooney | i left my inital feedback on the bug when i traged it as opion | |
| 16:35:30 | sean-k-mooney | i dint make it as invlaid as i tought we shoudl atelast dicuss it more widely first | |
| 16:36:01 | bauzas | for now, we should document that active-passive HA configuration for sure | |
| 16:36:16 | sean-k-mooney | for the perodic only | |
| 16:36:19 | bauzas | because indeed, we know that there is no eventual consistency betwen schedulers | |
| 16:36:27 | sean-k-mooney | the schduler shoudl be generally deploy active active | |
| 16:36:42 | bauzas | that's your opinion :) | |
| 16:36:44 | sean-k-mooney | but also the perodic has perfomance issuues | |
| 16:36:50 | sean-k-mooney | bauzas: its waht we use in our product | |
| 16:36:58 | sean-k-mooney | and what almost all instller do by defult | |
| 16:37:42 | bauzas | https://specs.openstack.org/openstack/nova-specs/specs/abandoned/parallel-scheduler.html | |
| 16:37:53 | tkajinam | yeah > almost all installer do by default | |
| 16:38:11 | sean-k-mooney | that a diffent proposal | |
| 16:38:36 | bauzas | I litterally quote the first sentence of that spec : | |
| 16:38:42 | bauzas | "If you running two nova-scheduler processes they race each other, they don’t find out about each others choices until the DB gets updated by the nova-compute resource tracker. This has lead to many deployments opting for an Active/Passive HA setup for the nova-scheduler process." | |
| 16:39:10 | tkajinam | people may not prefer using act-act for simplicity and avoid clustering mechanism to implement active-passive. | |
| 16:39:24 | tkajinam | without large warning :-P | |
| 16:39:31 | sean-k-mooney | bauzas: that does not really apply as of placement | |
| 16:39:55 | sean-k-mooney | bauzas: i woudl condier it to be very incorrect advice to deocument that active active is not supported | |
| 16:40:50 | gibi | yeah the goal of placement to shrink the race window between parallel schedulers | |
| 16:41:07 | gibi | it is a solved problem for those resources that are tracked in placement | |
| 16:41:13 | bauzas | I don't disagree with the fact that HA active-active schedulers is a problem to solve | |
| 16:41:30 | gibi | for those that are not tracked there, the compute manager has a lock around claim to prevent overallocation | |
| 16:41:36 | gibi | and we have alternatives to reschedule | |
| 16:41:50 | bauzas | gibi: exactly, hence the A/P mechanism | |
| 16:41:57 | gibi | no this is A A | |
| 16:42:10 | gibi | the only A P problem is in the periodic discovery | |
| 16:42:13 | bauzas | in the very early times, we were considering reschedules as a way to address the problem | |
| 16:42:38 | bauzas | we stopped that tenet by wanting to reduce the reschedules, leading to indeed a broader problem | |