Posted	Nick	Remark
#openstack-nova - 2023-03-14
16:32:49	bauzas	as I don't see a lot of topics yet in the ptg etherpad
16:33:00	bauzas	(well, I actually wrote the majority of them)
16:33:26	bauzas	also, if you want to discuss with other teams, please add your topics in the cross-project agenda
16:33:43	bauzas	and then I'll ping the other PTLs in order to find some time
16:34:09	bauzas	as a reminder, I asked for 4 days of 4 hours
16:34:47	bauzas	but if the agenda remains quite empty as it is, I'll probably unbook the Friday rooms
16:35:26	sean-k-mooney	if it was in person i would suggest using the firday slot for an unconfernce/hackaton
16:35:38	sean-k-mooney	but i kind of hate the idea of doing that virutally
16:35:44	bauzas	don't get me on that direction :)
16:35:52	sean-k-mooney	so sure
16:36:04	sean-k-mooney	we could keep the slot and decied durign the ptg
16:36:09	bauzas	I mean, after 3 vPTGs, I'm quite done with them
16:36:12	sean-k-mooney	i.e. if there is a topic we ant to come back too
16:36:23	bauzas	surely
16:36:44	bauzas	and I'm pretty sure people will add topics on the last time like every cycle..
16:36:56	bauzas	(tbc, I'm also doing this game)
16:37:19	bauzas	but even with that, I just feel Friday will be either off or hackathon
16:37:45	bauzas	fwiw, my main priorities for the beginning of Bobcat is to review some series we accepted
16:37:56	bauzas	like the manila one, which I'll probably help too
16:38:00	sean-k-mooney	if we have noting to do on firday im fine with just say "thanks for coming folks" and doing a spec review day or something
16:38:13	bauzas	sure
16:38:20	bauzas	that's an idea
16:38:23	bauzas	anyway
16:39:18	bauzas	fwiw, the ptg main agenda is at the moment not large https://ptg.opendev.org/ptg.html
16:39:37	bauzas	hopefully projects will book their rooms next week
16:40:25	bauzas	(one thing I also think we miss with *virtual* PTGs is those kind of cross-project large discussions we had before)
16:40:48	bauzas	but meh, moving on
16:41:09	bauzas	#topic Review priorities
16:41:14	bauzas	#link https://review.opendev.org/q/status:open+(project:openstack/nova+OR+project:openstack/placement+OR+project:openstack/os-traits+OR+project:openstack/os-resource-classes+OR+project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/osc-placement)+(label:Review-Priority%252B1+OR+label:Review-Priority%252B2)
16:41:18	bauzas	#info As a reminder, cores eager to review changes can +1 to indicate their interest, +2 for committing to the review
16:41:25	bauzas	#topic Stable Branches
16:41:27	bauzas	elodilles: your time
16:41:40	elodilles	well, nothing special
16:41:49	elodilles	i mean, not so many patches merge
16:41:59	elodilles	but as far as i see:
16:42:09	elodilles	#info stable gates seem to be OK - though it's hard to merge patches due to intermittent failures
16:42:16	elodilles	#info stable branch status / gate failures tracking etherpad: https://etherpad.opendev.org/p/nova-stable-branch-ci
16:42:31	elodilles	that's all :X
16:43:15	bauzas	elodilles: dansmith: afaik, we haven't backported the mysqld memory reduction changes into stable branches ?
16:43:15	dansmith	a bunch of the things we've fixed haven't been backported to stable,
16:43:21	dansmith	so I expect that will remain challenging
16:43:22	bauzas	hah, jinx
16:43:40	bauzas	we could work on that
16:43:48	dansmith	I haven't been monitoring the mysql memory thing, but it seems like that _has_ helped yeah/
16:44:27	opendevreview	Alexey Stupnikov proposed openstack/nova master: Don't remove cached base images for failed resize ops https://review.opendev.org/c/openstack/nova/+/877410
16:44:31	dansmith	there was a potential for negative impacts, but I've heard no complaints
16:45:19	elodilles	dansmith: do you have a topic set for those patches? so that I could see whether some could be backported?
16:45:21	bauzas	mmmh
16:45:56	dansmith	elodilles: this in devstack plus the flag enabled: https://review.opendev.org/c/openstack/devstack/+/873646
16:46:26	elodilles	dansmith: thanks, i'll have a look
16:46:29	bauzas	example here https://review.opendev.org/c/openstack/nova/+/874664
16:47:02	bauzas	dansmith: but that means we would need to backport the devstack change too right?
16:47:12	dansmith	that's why I said "this in devstack"
16:47:21	dansmith	meaning you need it in the devstack branc you're running on
16:47:25	elodilles	the nova patch is at least part of stable/2023.1 :)
16:47:27	dansmith	which is kinda :/
16:48:13	sean-k-mooney	so the memory patch proably shoudl be backported in devstack
16:48:14	bauzas	yeah
16:48:23	sean-k-mooney	becuase i think that might be useful on other stable branches
16:48:28	bauzas	https://review.opendev.org/c/openstack/devstack/+/873646/3/lib/databases/mysql I understand dansmith's concerns
16:48:52	bauzas	but it looks to me mysql doesn't bubble up the memory
16:48:54	dansmith	I don't have specific concerns
16:49:08	dansmith	if it's really that impactful, it's probably worth it,
16:49:18	bauzas	do we have some mysql monitoring in devstack ?
16:49:19	dansmith	it's just that it takes the devstack patch on each branch, plus job changes to enable
16:49:31	dansmith	bauzas: we have my performance.json which has the info in it
16:49:56	dansmith	gmann specifically didn't want to enable by default until bobcat (understandable)
16:50:02	dansmith	so backporting to stable is kinda the opposite of that :)
16:50:14	bauzas	yeah, I can understand
16:50:24	bauzas	we shouldn't default this to all the jovs
16:50:26	bauzas	obs
16:50:31	bauzas	damn, jobs even
16:50:32	sean-k-mooney	why not
16:50:38	dansmith	the concerns are that it could slow down mysql and introduce other performance regressions that manifest as failures
16:50:44	dansmith	it hasn't seemed to have done that in practice,
16:50:46	sean-k-mooney	did we see a change in the job execution time
16:50:57	sean-k-mooney	right
16:51:00	dansmith	but the thought was to minimize the risk, in an already risky environment
16:51:05	bauzas	sean-k-mooney: because of the fact we don't really monitor mysqld runs
16:51:12	bauzas	in the logs
16:51:15	dansmith	bauzas: what does that mean?
16:51:15	sean-k-mooney	and reducing memory pressure might actully reduce swapping and speed up the job
16:51:51	dansmith	bauzas: I think we do monitor it plenty, it's just that it's a pretty core function and breaking it could have lots and lots of wide impacts, both obvious and non-obvious
16:51:52	bauzas	dansmith: correct me if I'm wrong, but do we trace the mysql performance in the logs, you said we have that in performance.yaml
16:52:06	bauzas	I should take a look at this file
16:52:10	bauzas	(tbh)
16:52:17	dansmith	performance.json
16:52:30	dansmith	there's a lot of data in there, but memory is probably the only relevant bit
16:52:39	sean-k-mooney	we dont messure query reponce time as far as i know but we moditor memory usage
16:52:45	dansmith	I'm just saying I don't know what "monitor mysqld runs" means in this context
16:53:16	dansmith	yeah, no query time logging, but that would need to be done in aggregate to have any sort of meaningful result I think
16:53:29	sean-k-mooney	as long as we are not seeing errors form teh services (timeouts) or longer overall job runs that all we really need to know
16:53:30	dansmith	*and* it's time-based which is nearly impossible to compare across runs in the gate
16:54:01	dansmith	sean-k-mooney: right, well, not having it on by default in master yet, we don't really have that large of a sample
16:54:13	dansmith	we have a few jobs that are already atypical opt-ed into it
16:54:15	bauzas	dansmith: I'll look at what we get from performance.yaml
16:54:31	dansmith	anyway, I was good to go default on, but there's definitely risk so we just have to keep that in mind
16:54:48	dansmith	bauzas: please stop saying performance.yaml :)
16:54:56	dansmith	it's performance.json dammit :D
16:55:00	bauzas	my question was more about the fact that given we have less large temporary tables and innodb pool sizes, I'd love to see some mysql insights about botyh

Earlier   Later