| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-11-02 | |||
| 16:23:54 | bauzas | elodilles: hah, I warned about it in the channel :p | |
| 16:24:06 | sean-k-mooney[m] | are the detach issue due to the qemu version we have in bionic | |
| 16:24:28 | sean-k-mooney[m] | i assume train is not on focal? | |
| 16:24:36 | elodilles | yes, train is on bionic | |
| 16:24:50 | elodilles | (just like ussuri) | |
| 16:26:34 | bauzas | hmmm, technically, Train is EM | |
| 16:26:53 | sean-k-mooney[m] | im somewhat tempeted to same maybe move it to centos 8 or focal but we could disable the volume tests | |
| 16:26:59 | sean-k-mooney[m] | yes it is | |
| 16:27:11 | bauzas | I'd rather prefer us fixing the gate issues rather than reducing the test coverage, but this depends on any actions we can take | |
| 16:27:24 | bauzas | so, let's be pragmatic | |
| 16:28:17 | sean-k-mooney[m] | well the first question would be does train have gibis event based witing patch or is it still using the retry loop | |
| 16:28:39 | bauzas | gibi's patch isn't merged yet, right? | |
| 16:28:44 | bauzas | could it help ? | |
| 16:28:57 | sean-k-mooney[m] | the only options reallly to fi this are change the qemu verions or backport gibis patch | |
| 16:29:10 | bauzas | (I'll have to leave gibi chair in the next 2 mins but dansmith has a point I'm interested in) | |
| 16:29:37 | dansmith | I also have to go sooner | |
| 16:29:43 | bauzas | sean-k-mooney[m]: we can try to backport gibi's patch and see whether that helps | |
| 16:29:45 | dansmith | maybe we could swap open and libvirt? | |
| 16:29:52 | bauzas | dansmith: I'll | |
| 16:29:59 | gibi | I don't think there is anything in the libvirt topic | |
| 16:30:02 | gibi | lyarwood is out now | |
| 16:30:09 | dansmith | okay | |
| 16:30:21 | bauzas | okay, elodilles I'll propose to wait for gibi's patch to land in master and then be backported | |
| 16:30:32 | gibi | bauzas: it is backported til wallaby | |
| 16:30:33 | elodilles | bauzas: ack | |
| 16:30:34 | bauzas | and punt the decision to reduce the test coverage once we get better ideas | |
| 16:30:41 | gibi | if we are talking about https://review.opendev.org/q/topic:bug/1882521 | |
| 16:31:09 | bauzas | gibi: then we need to backport it down to train | |
| 16:31:13 | gibi | I don't think it will be a piece of cake to bring that back train | |
| 16:31:20 | gibi | *to train | |
| 16:31:32 | bauzas | gibi: (apologies I confused with the vnic types waiting patch) | |
| 16:31:59 | bauzas | I have to leave, but can we hold this one discussion and go straight to dansmith's point | |
| 16:32:01 | bauzas | ? | |
| 16:32:09 | gibi | anyhow we can take that outside when lyarwood is back | |
| 16:32:09 | bauzas | so I and dansmith can leave | |
| 16:32:21 | gibi | lets go to that | |
| 16:32:25 | bauzas | #topic Sub/related team Highlights | |
| 16:32:27 | bauzas | nothing to tell | |
| 16:32:32 | bauzas | #topic Open discussion | |
| 16:32:37 | bauzas | Bring default overcommit ratios into sanity (dansmith / yuriys) | |
| 16:32:39 | dansmith | So, I think we all know the default 16x cpu overcommit default is insane | |
| 16:32:41 | yuriys | exciting | |
| 16:32:54 | dansmith | we've got reports that some operators are USING those defaults because they think we're recommending them | |
| 16:32:59 | sean-k-mooney[m] | yes it is | |
| 16:33:04 | yuriys | I am so used to Slack and Discord for drop a paragraph level of communication, so pardon all the incoming spam! I prewrote stuff. | |
| 16:33:04 | bauzas | hah | |
| 16:33:08 | dansmith | yuriys is here to offer guidance and help work on this, | |
| 16:33:21 | dansmith | but I think we need to move those defaults to something sane, both in code and update the docs | |
| 16:33:28 | bauzas | I guess this can be workload dependent, right? | |
| 16:33:29 | sean-k-mooney[m] | it basically should not be set over 4x | |
| 16:33:39 | yuriys | 4:1 for cpu , 1:1 for mem. | |
| 16:33:44 | sean-k-mooney[m] | yep | |
| 16:33:48 | bauzas | I think we started to document things based on workloads | |
| 16:33:56 | yuriys | yes | |
| 16:33:58 | dansmith | bauzas: it's completely workload dependent, but we should not be recommending really anything, and thus I think the default needs to be closer to 1:1 with docs saying why you may increase it (or not) | |
| 16:33:58 | bauzas | but we never achieved this | |
| 16:34:01 | yuriys | it's VERY use case specific | |
| 16:34:17 | sean-k-mooney[m] | it is | |
| 16:34:26 | bauzas | I'm referring to https://docs.openstack.org/nova/latest/user/feature-classification.html | |
| 16:34:28 | yuriys | Ideally the documentation is restructured on how to scale up these over commits to match the desired use case, density and performance and start at sane values/defaults. Engineers can then template out the necessary config values after they've gotten to know system capabilities. Instead of working backwards from chaos and mayhem, giving future admins the opportunity to reach desired state through scaling up from a stable syste | |
| 16:34:28 | yuriys | hould be the goal of arch design. | |
| 16:35:07 | dansmith | so can we agree that we'll change the defaults to something that seems reasonable, and modify the docs that just say "these are the defaults" to have big flashing warnings that defaults will never be universal in this case? | |
| 16:35:09 | sean-k-mooney[m] | the 16:1 number originally was assuming webhosting as the main usecase or similar workloads | |
| 16:35:13 | dansmith | specifically: https://docs.openstack.org/arch-design/design-compute/design-compute-overcommit.html | |
| 16:35:19 | sean-k-mooney[m] | that does not fit with how openstack is typicaly used | |
| 16:35:28 | bauzas | dansmith: this sounds a reasonable change | |
| 16:35:45 | dansmith | cool, specless bp or bug? | |
| 16:36:00 | bauzas | my only worries would go on how this is wired at the object level but we can take the opportunity to lift this | |
| 16:36:12 | sean-k-mooney[m] | 4:1 for cpu is the highist ratio i would general consider usable in production | |
| 16:36:32 | bauzas | dansmith: there are a few upgrade concerns with placement as IIRC this is set by the model itself | |
| 16:36:32 | dansmith | I think we already moved towards something better when we moved the defaults to placement right? but we need to do something | |
| 16:36:54 | sean-k-mooney[m] | well we have the inital allcoation ratios now | |
| 16:36:56 | yuriys | i said 4 to be reasonable haha, i think i've set about 3:1 with nova-scheduler and placement randomization elements. | |
| 16:36:59 | dansmith | bauzas: yeah I think placement now has explicit defaults right? | |
| 16:37:02 | bauzas | I'm pretty sure we have some default value in the placement DB that says "16" | |
| 16:37:03 | sean-k-mooney[m] | but it still default to 16 | |
| 16:37:07 | dansmith | sean-k-mooney[m]: right | |
| 16:37:17 | sean-k-mooney[m] | i think we can decrease inial to 4 for cpu an 1 for memory | |
| 16:37:22 | dansmith | so I think we can just move those and reno that operators who took those defualts years ago should change them likely | |
| 16:37:39 | sean-k-mooney[m] | +1 | |
| 16:37:39 | bauzas | I don't wanna go procedural | |
| 16:37:51 | bauzas | so a specless BP could work for me but, | |
| 16:37:57 | bauzas | we need renos | |
| 16:38:01 | dansmith | sean-k-mooney[m]: yeah I think 4:1 CPU and 1:1 memory is fine for a default, we might need up to up for devstack I guess but that's where insane defaults should be :) | |
| 16:38:03 | sean-k-mooney[m] | yep i was thinking the same | |
| 16:38:10 | bauzas | + we need to ensure we consider the DB impact before | |
| 16:38:15 | dansmith | cool, specless bp and renos.. sounds good | |
| 16:38:37 | bauzas | if that becomes debatable in the reviews, we could go drafting more | |
| 16:38:42 | sean-k-mooney[m] | bauzas: i dont think there will be any | |
| 16:38:59 | bauzas | but here, we're talking of changing defaults, not changing existing deployments | |
| 16:39:00 | sean-k-mooney[m] | if we are just changing the initial values it wont affect existing RPs | |
| 16:39:02 | dansmith | yeah I think it'll be straightforward, but we can always revise the plan if needed | |
| 16:39:07 | dansmith | right | |
| 16:39:18 | bauzas | OK, looks to me we have a plan | |
| 16:39:24 | dansmith | #micdrop | |
| 16:39:40 | bauzas | #agreed changing overcommit CPU ratio to <16.0 can be a specless BP | |
| 16:39:49 | bauzas | yuriys: typey typey | |
| 16:40:16 | bauzas | and ping me on IRC once you have the Launchpad BP up so I can approve it | |
| 16:40:29 | gibi | OK | |
| 16:40:37 | gibi | is there anything else for today? | |