Posted	Nick	Remark
#openstack-nova - 2020-10-30
14:02:05	tosky	gibi: unless you have other topics of course (the list is huuuge); in case, feel free to ping me
14:02:28	gibi	tosky: if you can join now then I think we can get to this in about 10 minutes
14:03:43	tosky	gibi: oh, jumping in
14:03:48	gibi	tosky: ack
14:11:01	kashyap	gibi: /me was participating in KVM Forum BoFs and sessions; so been away here
14:11:19	kashyap	(Too many things happening at the same time...)
14:11:24	gibi	kashyap: ack
14:13:44	gibi	kashyap: I think we haven't encountered any topics from you yet
14:14:17	kashyap	gibi: The main thing I have is the long-running Secure Boot spec -- for repproval -- I put it on the back burner due to other things
14:14:28	kashyap	So it should be fairly uncontroversial
14:15:35	kashyap	We have various infra bits in-tree, and in-flight. (Aside: I have another conflict in an hour or so ... if I'm not around, please leave questions for me in the Etherpad)
14:15:40	kashyap	(I'll come back and answer)
14:17:24	gibi	kashyap: OK. I it good if we bring the secure boot up right after the current break?
14:20:42	kashyap	gibi: Current break is how long? I have a hard-break at 16:20 CET
14:21:26	kashyap	I can pay more attention back here again at 17:30 CET
14:21:58	gibi	we are back at 14:26 UTC
14:22:03	gibi	so in like 4 minutes
14:23:08	kashyap	Ah, sure; that works
14:23:23	kashyap	gibi: But I feel like I'm cheating by moving a topic up
14:23:29	kashyap	So feel free to say "no"
14:24:06	kashyap	(I don't mean, I literally moved it on the Etherpad :))
14:25:08	gibi	kashyap: not cheating, it is dynamic scheduling of the topic based on availability :)
14:25:23	kashyap	Hehe, reframing, is it
14:25:26	kashyap	Thanks a lot!
14:26:06	gibi	I moved your topic up in the etherpd
14:27:02	openstackgerrit	Lee Yarwood proposed openstack/nova master: zuul: Merge nova-evacuate into nova-multinode-live-migration https://review.opendev.org/755309
14:36:39	sean-k-mooney	im having issue with zoom ill try and rejion as soon as i can
14:42:33	gibi	melwitt, dansmith: fyi, we are delaying the NUMA discussion on the nova PTG so that you can join. let me know if we should not wait
14:43:07	dansmith	gibi: I'm triple-booked today
14:43:13	dansmith	gibi: this is numa and placement?
14:43:16	gibi	dansmith: yes
14:44:00	dansmith	ack, well, I'm in the camp of doing this in placement of course
14:44:18	dansmith	I think melwitt is too, so she can probably just say everything twice to pretend I'm there :)
14:44:37	gibi	dansmith: OK. thanks.
14:55:51	lyarwood	mgoddard: https://www.openstack.org/ptg/rooms/liberty - about to talk about https://bugs.launchpad.net/nova/+bug/1895848 if you're free btw
14:55:51	openstack	Launchpad bug 1895848 in OpenStack Compute (nova) "Migration and evacuation fails with encrypted volumes" [Medium,Confirmed] - Assigned to Lee Yarwood (lyarwood)
14:56:03	lyarwood	mgoddard: apologies for the late ping, failed at scrolling down.
14:56:23	lyarwood	mgoddard: https://etherpad.opendev.org/p/nova-wallaby-ptg is the pad btw
14:56:35	mgoddard	lyarwood: o/
14:56:40	mgoddard	just joined. Thanks for the ping
14:56:53	lyarwood	np
14:56:59	lyarwood	just finishing the previous topic
15:05:33	melwitt	gibi: cool, thanks for doing that! I'm online now
15:05:49	gibi	melwitt: o/ cool then we will take NUMA soon
15:06:32	bauzas	johnthetubaguy: https://github.com/openstack/nova/blob/master/nova/scheduler/request_filter.py#L122
15:06:43	bauzas	we log a bit of infor
15:20:11	bauzas	fedora's oomearly--
15:20:35	bauzas	zoom-- for eating all my RAM
15:23:02	stephenfin	melwitt: you're super echo'y
15:23:33	melwitt	stephenfin: argh ok, checking my headset settings
15:24:16	melwitt	hm looks like it's all set right
15:24:37	lyarwood	actually sounds really cool, I wouldn't worry
15:26:52	melwitt	sounds cool xD heh
15:44:41	lyarwood	gmann: https://review.opendev.org/#/c/755525/ - can you remove your -W on this btw, the `fix` landed in stable/victoria
15:45:01	openstackgerrit	Lee Yarwood proposed openstack/nova stable/ussuri: libvirt: Increase incremental and max sleep time during device detach https://review.opendev.org/757306
15:45:57	gmann	lyarwood: done,
15:53:38	lyarwood	gmann: thanks
16:19:46	johnsom	Hi Nova community. The Octavia team is seeing some strange behavior with libvirtd. https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c77/759973/3/check/octavia-v2-dsvm-scenario/c77fe63/controller/logs/libvirt/libvirtd_log.txt
16:19:52	johnsom	Warning, that is a 136MB log file
16:20:08	johnsom	It seems to go into some sort of tight loop:
16:20:26	lyarwood	ops, opening that in chrome was a mistake
16:20:30	johnsom	https://www.irccloud.com/pastebin/c6NcVeeF/
16:20:54	johnsom	Almost 90% of that log file is the above
16:20:59	lyarwood	kashyap: ^ if you have time
16:21:05	lyarwood	ring any bells?
16:21:38	johnsom	Eventually it seems to finally exit. The issue we have is while it's stuck in this loop other VMs are not going ACTIVE in nova, so causes all sorts of timeouts.
16:21:39	kashyap	lyarwood: Just a sec; lost in two other convo threads :-(
16:22:15	lyarwood	kashyap: np
16:22:26	kashyap	johnsom: As a quick question - in what scenario do you see that?
16:22:33	lyarwood	johnsom: might be useful to write that up in a nova bug that kashyap can follow up with
16:22:37	lyarwood	ops sorry
16:22:38	kashyap	johnsom: That "End of file" ... simply means libvirt lost connection to QEMU
16:22:52	johnsom	It appears that nova was trying to delete the instance
16:22:55	kashyap	lyarwood: No worries; a bug also helps - as a record
16:23:45	johnsom	Here is a chunk of the log right before the loop:
16:23:48	johnsom	https://www.irccloud.com/pastebin/LRErlmPf/
16:26:35	kashyap	johnsom: Yeah, but that doesn't tell us what events led to that (they're in the 130MB log file). :-) Is this reproducible?
16:27:12	johnsom	It is reproducible, but intermittent. Not every job triggers it.
16:28:05	gibi	#nova now refresh connection_info / avoid storing stale connection_info in Nova
16:28:54	johnsom	We are seeing it while we run our scenario test suites.
16:30:06	kashyap	johnsom: A write-up in a bug would be nice to track. Along w/ a description of a test case that's hitting it. (I'm almost out of neurons tonight, and my mind now has the "thousand yard stare")
16:31:10	johnsom	Yeah, I'm writing one up. I will link to the long, but I don't think I should attach it given it's size, so it will expire off the object store.
16:31:18	johnsom	I will paste in a few snippets.
16:31:56	kashyap	You can xz-compress the log; as the CI one might get cleaned up any moment
16:32:35	kashyap	johnsom: Also, paste-bins expore; please add relevant snippets as a _text_ file. Because, LaunchPad, annoyingly enough, breaks formatting
16:32:46	kashyap	Thanks :)
16:33:12	johnsom	Ok
16:40:17	kashyap	johnsom: (Sorry for being pedanctic.)
16:40:38	johnsom	Oh no worries. I want to make sure you get what you need to help us out.
16:54:51	openstack	Launchpad bug 1902276 in OpenStack Compute (nova) "libvirtd going into a tight loop causing instances to not transition to ACTIVE" [Undecided,New]
16:54:51	johnsom	kashyap https://bugs.launchpad.net/nova/+bug/1902276
16:59:59	kashyap	johnsom: Thanks; another question - so the instances are not coming up online?
17:02:11	kashyap	johnsom: Disregard me; you actually say that "eventually things go back to normal"
17:03:15	johnsom	Yep
17:10:24	kashyap	johnsom: I'm a bit baffled with it all :) I asked a couple of libvirt devs to see if the lead in and the end of the loop makes any further sense to them
17:10:43	johnsom	Ok, thank you
17:15:46	kashyap	johnsom: Haha, you write: "We seen this regularly, but it's intermittent" — now, which is it?
17:16:19	johnsom	kashyap We see it on jobs daily, but it's not every run
17:16:54	johnsom	Maybe that makes more sense? grin
17:18:10	johnsom	It's been a long week, words are hard. grin

Earlier   Later