Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2023-03-14
16:41:27 bauzas elodilles: your time
16:41:40 elodilles well, nothing special
16:41:49 elodilles i mean, not so many patches merge
16:41:59 elodilles but as far as i see:
16:42:09 elodilles #info stable gates seem to be OK - though it's hard to merge patches due to intermittent failures
16:42:16 elodilles #info stable branch status / gate failures tracking etherpad: https://etherpad.opendev.org/p/nova-stable-branch-ci
16:42:31 elodilles that's all :X
16:43:15 dansmith a bunch of the things we've fixed haven't been backported to stable,
16:43:15 bauzas elodilles: dansmith: afaik, we haven't backported the mysqld memory reduction changes into stable branches ?
16:43:21 dansmith so I expect that will remain challenging
16:43:22 bauzas hah, jinx
16:43:40 bauzas we could work on that
16:43:48 dansmith I haven't been monitoring the mysql memory thing, but it seems like that _has_ helped yeah/
16:44:27 opendevreview Alexey Stupnikov proposed openstack/nova master: Don't remove cached base images for failed resize ops https://review.opendev.org/c/openstack/nova/+/877410
16:44:31 dansmith there was a potential for negative impacts, but I've heard no complaints
16:45:19 elodilles dansmith: do you have a topic set for those patches? so that I could see whether some could be backported?
16:45:21 bauzas mmmh
16:45:56 dansmith elodilles: this in devstack plus the flag enabled: https://review.opendev.org/c/openstack/devstack/+/873646
16:46:26 elodilles dansmith: thanks, i'll have a look
16:46:29 bauzas example here https://review.opendev.org/c/openstack/nova/+/874664
16:47:02 bauzas dansmith: but that means we would need to backport the devstack change too right?
16:47:12 dansmith that's why I said "this in devstack"
16:47:21 dansmith meaning you need it in the devstack branc you're running on
16:47:25 elodilles the nova patch is at least part of stable/2023.1 :)
16:47:27 dansmith which is kinda :/
16:48:13 sean-k-mooney so the memory patch proably shoudl be backported in devstack
16:48:14 bauzas yeah
16:48:23 sean-k-mooney becuase i think that might be useful on other stable branches
16:48:28 bauzas https://review.opendev.org/c/openstack/devstack/+/873646/3/lib/databases/mysql I understand dansmith's concerns
16:48:52 bauzas but it looks to me mysql doesn't bubble up the memory
16:48:54 dansmith I don't have specific concerns
16:49:08 dansmith if it's really that impactful, it's probably worth it,
16:49:18 bauzas do we have some mysql monitoring in devstack ?
16:49:19 dansmith it's just that it takes the devstack patch on each branch, plus job changes to enable
16:49:31 dansmith bauzas: we have my performance.json which has the info in it
16:49:56 dansmith gmann specifically didn't want to enable by default until bobcat (understandable)
16:50:02 dansmith so backporting to stable is kinda the opposite of that :)
16:50:14 bauzas yeah, I can understand
16:50:24 bauzas we shouldn't default this to all the jovs
16:50:26 bauzas obs
16:50:31 bauzas damn, jobs even
16:50:32 sean-k-mooney why not
16:50:38 dansmith the concerns are that it could slow down mysql and introduce other performance regressions that manifest as failures
16:50:44 dansmith it hasn't seemed to have done that in practice,
16:50:46 sean-k-mooney did we see a change in the job execution time
16:50:57 sean-k-mooney right
16:51:00 dansmith but the thought was to minimize the risk, in an already risky environment
16:51:05 bauzas sean-k-mooney: because of the fact we don't really monitor mysqld runs
16:51:12 bauzas in the logs
16:51:15 sean-k-mooney and reducing memory pressure might actully reduce swapping and speed up the job
16:51:15 dansmith bauzas: what does that mean?
16:51:51 dansmith bauzas: I think we do monitor it plenty, it's just that it's a pretty core function and breaking it could have lots and lots of wide impacts, both obvious and non-obvious
16:51:52 bauzas dansmith: correct me if I'm wrong, but do we trace the mysql performance in the logs, you said we have that in performance.yaml
16:52:06 bauzas I should take a look at this file
16:52:10 bauzas (tbh)
16:52:17 dansmith performance.json
16:52:30 dansmith there's a lot of data in there, but memory is probably the only relevant bit
16:52:39 sean-k-mooney we dont messure query reponce time as far as i know but we moditor memory usage
16:52:45 dansmith I'm just saying I don't know what "monitor mysqld runs" means in this context
16:53:16 dansmith yeah, no query time logging, but that would need to be done in aggregate to have any sort of meaningful result I think
16:53:29 sean-k-mooney as long as we are not seeing errors form teh services (timeouts) or longer overall job runs that all we really need to know
16:53:30 dansmith *and* it's time-based which is nearly impossible to compare across runs in the gate
16:54:01 dansmith sean-k-mooney: right, well, not having it on by default in master yet, we don't really have that large of a sample
16:54:13 dansmith we have a few jobs that are already atypical opt-ed into it
16:54:15 bauzas dansmith: I'll look at what we get from performance.yaml
16:54:31 dansmith anyway, I was good to go default on, but there's definitely risk so we just have to keep that in mind
16:54:48 dansmith bauzas: please stop saying performance.yaml :)
16:54:56 dansmith it's performance.json dammit :D
16:55:00 bauzas my question was more about the fact that given we have less large temporary tables and innodb pool sizes, I'd love to see some mysql insights about botyh
16:55:14 bauzas oh f, you're right
16:55:17 bauzas pardon my YAML
16:55:28 sean-k-mooney ...
16:55:38 bauzas anyway, we're quite at the end of the meeting
16:55:58 bauzas #topic Open discussion
16:56:10 bauzas the agenda is free from any item
16:56:16 bauzas so, anything to say ?
16:57:22 bauzas looks not
16:57:29 bauzas thanks all
16:57:34 opendevmeet Log: https://meetings.opendev.org/meetings/nova/2023/nova.2023-03-14-16.00.log.html
16:57:34 opendevmeet Minutes (text): https://meetings.opendev.org/meetings/nova/2023/nova.2023-03-14-16.00.txt
16:57:34 opendevmeet Minutes: https://meetings.opendev.org/meetings/nova/2023/nova.2023-03-14-16.00.html
16:57:34 opendevmeet Meeting ended Tue Mar 14 16:57:34 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
16:57:34 bauzas #endmeeting
16:59:23 sean-k-mooney bauzas: for what its worth i remember turnign mysql for low memory foot print many years ago when i first started workign on openstack
16:59:44 sean-k-mooney there used to be a blog or doc that told you how to do it
16:59:59 sean-k-mooney i dont know where that was cause its been years but this type of change is not new to me
17:00:12 bauzas sean-k-mooney: AFAIR, you can use mysql's information schema tables to get the recommended pool sizes
17:00:31 bauzas but I was a mysql operator a decade ago and my brain is fried now
17:00:47 sean-k-mooney probably but i dont really have concern with the value that are bing used
17:00:57 sean-k-mooney i think we used to do this in the intel nfv ci
17:01:16 sean-k-mooney to reduce memoryusage because we were deploying with ovs-dpdk
17:01:23 bauzas for the temp tables, well that's less of a concern
17:01:40 bauzas iirc, if the tmp tables aren't large enough, this just goes on disk
17:01:53 sean-k-mooney no it was for the pools as well
17:02:01 bauzas so this becomes an I/O performance question
17:02:11 sean-k-mooney right i dont think it will be in our usage
17:02:29 sean-k-mooney we will see but i would prefer to see by turning this on by default in all the jobs
17:02:34 sean-k-mooney and seeign if we see any regressions
17:03:25 bauzas so, I double-checked and yeah the pools are for caching
17:03:32 sean-k-mooney yep

Earlier   Later