| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-03-14 | |||
| 16:46:29 | bauzas | example here https://review.opendev.org/c/openstack/nova/+/874664 | |
| 16:47:02 | bauzas | dansmith: but that means we would need to backport the devstack change too right? | |
| 16:47:12 | dansmith | that's why I said "this in devstack" | |
| 16:47:21 | dansmith | meaning you need it in the devstack branc you're running on | |
| 16:47:25 | elodilles | the nova patch is at least part of stable/2023.1 :) | |
| 16:47:27 | dansmith | which is kinda :/ | |
| 16:48:13 | sean-k-mooney | so the memory patch proably shoudl be backported in devstack | |
| 16:48:14 | bauzas | yeah | |
| 16:48:23 | sean-k-mooney | becuase i think that might be useful on other stable branches | |
| 16:48:28 | bauzas | https://review.opendev.org/c/openstack/devstack/+/873646/3/lib/databases/mysql I understand dansmith's concerns | |
| 16:48:52 | bauzas | but it looks to me mysql doesn't bubble up the memory | |
| 16:48:54 | dansmith | I don't have specific concerns | |
| 16:49:08 | dansmith | if it's really that impactful, it's probably worth it, | |
| 16:49:18 | bauzas | do we have some mysql monitoring in devstack ? | |
| 16:49:19 | dansmith | it's just that it takes the devstack patch on each branch, plus job changes to enable | |
| 16:49:31 | dansmith | bauzas: we have my performance.json which has the info in it | |
| 16:49:56 | dansmith | gmann specifically didn't want to enable by default until bobcat (understandable) | |
| 16:50:02 | dansmith | so backporting to stable is kinda the opposite of that :) | |
| 16:50:14 | bauzas | yeah, I can understand | |
| 16:50:24 | bauzas | we shouldn't default this to all the jovs | |
| 16:50:26 | bauzas | obs | |
| 16:50:31 | bauzas | damn, jobs even | |
| 16:50:32 | sean-k-mooney | why not | |
| 16:50:38 | dansmith | the concerns are that it could slow down mysql and introduce other performance regressions that manifest as failures | |
| 16:50:44 | dansmith | it hasn't seemed to have done that in practice, | |
| 16:50:46 | sean-k-mooney | did we see a change in the job execution time | |
| 16:50:57 | sean-k-mooney | right | |
| 16:51:00 | dansmith | but the thought was to minimize the risk, in an already risky environment | |
| 16:51:05 | bauzas | sean-k-mooney: because of the fact we don't really monitor mysqld runs | |
| 16:51:12 | bauzas | in the logs | |
| 16:51:15 | dansmith | bauzas: what does that mean? | |
| 16:51:15 | sean-k-mooney | and reducing memory pressure might actully reduce swapping and speed up the job | |
| 16:51:51 | dansmith | bauzas: I think we do monitor it plenty, it's just that it's a pretty core function and breaking it could have lots and lots of wide impacts, both obvious and non-obvious | |
| 16:51:52 | bauzas | dansmith: correct me if I'm wrong, but do we trace the mysql performance in the logs, you said we have that in performance.yaml | |
| 16:52:06 | bauzas | I should take a look at this file | |
| 16:52:10 | bauzas | (tbh) | |
| 16:52:17 | dansmith | performance.json | |
| 16:52:30 | dansmith | there's a lot of data in there, but memory is probably the only relevant bit | |
| 16:52:39 | sean-k-mooney | we dont messure query reponce time as far as i know but we moditor memory usage | |
| 16:52:45 | dansmith | I'm just saying I don't know what "monitor mysqld runs" means in this context | |
| 16:53:16 | dansmith | yeah, no query time logging, but that would need to be done in aggregate to have any sort of meaningful result I think | |
| 16:53:29 | sean-k-mooney | as long as we are not seeing errors form teh services (timeouts) or longer overall job runs that all we really need to know | |
| 16:53:30 | dansmith | *and* it's time-based which is nearly impossible to compare across runs in the gate | |
| 16:54:01 | dansmith | sean-k-mooney: right, well, not having it on by default in master yet, we don't really have that large of a sample | |
| 16:54:13 | dansmith | we have a few jobs that are already atypical opt-ed into it | |
| 16:54:15 | bauzas | dansmith: I'll look at what we get from performance.yaml | |
| 16:54:31 | dansmith | anyway, I was good to go default on, but there's definitely risk so we just have to keep that in mind | |
| 16:54:48 | dansmith | bauzas: please stop saying performance.yaml :) | |
| 16:54:56 | dansmith | it's performance.json dammit :D | |
| 16:55:00 | bauzas | my question was more about the fact that given we have less large temporary tables and innodb pool sizes, I'd love to see some mysql insights about botyh | |
| 16:55:14 | bauzas | oh f, you're right | |
| 16:55:17 | bauzas | pardon my YAML | |
| 16:55:28 | sean-k-mooney | ... | |
| 16:55:38 | bauzas | anyway, we're quite at the end of the meeting | |
| 16:55:58 | bauzas | #topic Open discussion | |
| 16:56:10 | bauzas | the agenda is free from any item | |
| 16:56:16 | bauzas | so, anything to say ? | |
| 16:57:22 | bauzas | looks not | |
| 16:57:29 | bauzas | thanks all | |
| 16:57:34 | bauzas | #endmeeting | |
| 16:57:34 | opendevmeet | Meeting ended Tue Mar 14 16:57:34 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | |
| 16:57:34 | opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2023/nova.2023-03-14-16.00.html | |
| 16:57:34 | opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2023/nova.2023-03-14-16.00.txt | |
| 16:57:34 | opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2023/nova.2023-03-14-16.00.log.html | |
| 16:59:23 | sean-k-mooney | bauzas: for what its worth i remember turnign mysql for low memory foot print many years ago when i first started workign on openstack | |
| 16:59:44 | sean-k-mooney | there used to be a blog or doc that told you how to do it | |
| 16:59:59 | sean-k-mooney | i dont know where that was cause its been years but this type of change is not new to me | |
| 17:00:12 | bauzas | sean-k-mooney: AFAIR, you can use mysql's information schema tables to get the recommended pool sizes | |
| 17:00:31 | bauzas | but I was a mysql operator a decade ago and my brain is fried now | |
| 17:00:47 | sean-k-mooney | probably but i dont really have concern with the value that are bing used | |
| 17:00:57 | sean-k-mooney | i think we used to do this in the intel nfv ci | |
| 17:01:16 | sean-k-mooney | to reduce memoryusage because we were deploying with ovs-dpdk | |
| 17:01:23 | bauzas | for the temp tables, well that's less of a concern | |
| 17:01:40 | bauzas | iirc, if the tmp tables aren't large enough, this just goes on disk | |
| 17:01:53 | sean-k-mooney | no it was for the pools as well | |
| 17:02:01 | bauzas | so this becomes an I/O performance question | |
| 17:02:11 | sean-k-mooney | right i dont think it will be in our usage | |
| 17:02:29 | sean-k-mooney | we will see but i would prefer to see by turning this on by default in all the jobs | |
| 17:02:34 | sean-k-mooney | and seeign if we see any regressions | |
| 17:03:25 | bauzas | so, I double-checked and yeah the pools are for caching | |
| 17:03:32 | sean-k-mooney | yep | |
| 17:03:46 | bauzas | so in case the pools aren't large enough, this is just a disk io performance concern as well | |
| 17:03:53 | sean-k-mooney | both of result sets and quey execution plans i belive | |
| 17:04:18 | sean-k-mooney | bauzas: yep but we are already swappign to disk in several jobs | |
| 17:04:30 | sean-k-mooney | if we can reduce that it likely will improve perfromce over all | |
| 17:04:57 | bauzas | yeah, now I refresh my mind | |
| 17:05:01 | sean-k-mooney | basically what im saying is any perfromance hit form the mysql turning will likely be offset by reduced memory pressure over all | |
| 17:05:13 | bauzas | I remember we can call the innodb engine to get the performance metrics | |
| 18:11:26 | auniyal | this :( https://zuul.openstack.org/status#864055, its always fail on gate | |
| #openstack-nova - 2023-03-15 | |||
| 05:37:58 | opendevreview | Amit Uniyal proposed openstack/nova master: Disconnecting volume from the compute host https://review.opendev.org/c/openstack/nova/+/877446 | |
| 05:57:45 | opendevreview | Rajesh Tailor proposed openstack/nova master: Fix duplicate cell creation with same name https://review.opendev.org/c/openstack/nova/+/876940 | |
| 10:18:07 | opendevreview | Rajesh Tailor proposed openstack/nova master: Fix case-sensitivity for metadata keys https://review.opendev.org/c/openstack/nova/+/873901 | |
| 10:33:35 | opendevreview | Fabian Wiesel proposed openstack/nova master: Add more password generation options https://review.opendev.org/c/openstack/nova/+/865669 | |
| 12:00:05 | opendevreview | Justas Poderys proposed openstack/nova-specs master: Add support for Napatech LinkVirt SmartNICs https://review.opendev.org/c/openstack/nova-specs/+/859290 | |
| 12:10:32 | opendevreview | Justas Poderys proposed openstack/nova-specs master: Add support for Napatech LinkVirt SmartNICs https://review.opendev.org/c/openstack/nova-specs/+/859290 | |
| 13:51:41 | dvo-plv | sean-k-mooney: Hello, we have updated spec file for our blueprint https://review.opendev.org/c/openstack/nova-specs/+/859290/8. Could you please review it, when you will have time | |
| 13:53:32 | dvo-plv | Also we have another blueprint https://review.opendev.org/c/openstack/nova-specs/+/868377 which wait for review process, pay attention to it, when it will be possible, please. | |
| 13:55:44 | sean-k-mooney | dvo-plv: i had some pendign comment i forgot to submit in gerrit for https://review.opendev.org/c/openstack/nova-specs/+/868377 | |
| 13:56:49 | sean-k-mooney | ill try and take a look at the other one today or tormorrow | |
| 13:57:08 | sean-k-mooney | i will be on off on firday for st patricks day just an fyi | |