| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-10-30 | |||
| 14:23:08 | kashyap | Ah, sure; that works | |
| 14:23:23 | kashyap | gibi: But I feel like I'm cheating by moving a topic up | |
| 14:23:29 | kashyap | So feel free to say "no" | |
| 14:24:06 | kashyap | (I don't mean, I literally moved it on the Etherpad :)) | |
| 14:25:08 | gibi | kashyap: not cheating, it is dynamic scheduling of the topic based on availability :) | |
| 14:25:23 | kashyap | Hehe, reframing, is it | |
| 14:25:26 | kashyap | Thanks a lot! | |
| 14:26:06 | gibi | I moved your topic up in the etherpd | |
| 14:27:02 | openstackgerrit | Lee Yarwood proposed openstack/nova master: zuul: Merge nova-evacuate into nova-multinode-live-migration https://review.opendev.org/755309 | |
| 14:36:39 | sean-k-mooney | im having issue with zoom ill try and rejion as soon as i can | |
| 14:42:33 | gibi | melwitt, dansmith: fyi, we are delaying the NUMA discussion on the nova PTG so that you can join. let me know if we should not wait | |
| 14:43:07 | dansmith | gibi: I'm triple-booked today | |
| 14:43:13 | dansmith | gibi: this is numa and placement? | |
| 14:43:16 | gibi | dansmith: yes | |
| 14:44:00 | dansmith | ack, well, I'm in the camp of doing this in placement of course | |
| 14:44:18 | dansmith | I think melwitt is too, so she can probably just say everything twice to pretend I'm there :) | |
| 14:44:37 | gibi | dansmith: OK. thanks. | |
| 14:55:51 | openstack | Launchpad bug 1895848 in OpenStack Compute (nova) "Migration and evacuation fails with encrypted volumes" [Medium,Confirmed] - Assigned to Lee Yarwood (lyarwood) | |
| 14:55:51 | lyarwood | mgoddard: https://www.openstack.org/ptg/rooms/liberty - about to talk about https://bugs.launchpad.net/nova/+bug/1895848 if you're free btw | |
| 14:56:03 | lyarwood | mgoddard: apologies for the late ping, failed at scrolling down. | |
| 14:56:23 | lyarwood | mgoddard: https://etherpad.opendev.org/p/nova-wallaby-ptg is the pad btw | |
| 14:56:35 | mgoddard | lyarwood: o/ | |
| 14:56:40 | mgoddard | just joined. Thanks for the ping | |
| 14:56:53 | lyarwood | np | |
| 14:56:59 | lyarwood | just finishing the previous topic | |
| 15:05:33 | melwitt | gibi: cool, thanks for doing that! I'm online now | |
| 15:05:49 | gibi | melwitt: o/ cool then we will take NUMA soon | |
| 15:06:32 | bauzas | johnthetubaguy: https://github.com/openstack/nova/blob/master/nova/scheduler/request_filter.py#L122 | |
| 15:06:43 | bauzas | we log a bit of infor | |
| 15:20:11 | bauzas | fedora's oomearly-- | |
| 15:20:35 | bauzas | zoom-- for eating all my RAM | |
| 15:23:02 | stephenfin | melwitt: you're super echo'y | |
| 15:23:33 | melwitt | stephenfin: argh ok, checking my headset settings | |
| 15:24:16 | melwitt | hm looks like it's all set right | |
| 15:24:37 | lyarwood | actually sounds really cool, I wouldn't worry | |
| 15:26:52 | melwitt | sounds cool xD heh | |
| 15:44:41 | lyarwood | gmann: https://review.opendev.org/#/c/755525/ - can you remove your -W on this btw, the `fix` landed in stable/victoria | |
| 15:45:01 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/ussuri: libvirt: Increase incremental and max sleep time during device detach https://review.opendev.org/757306 | |
| 15:45:57 | gmann | lyarwood: done, | |
| 15:53:38 | lyarwood | gmann: thanks | |
| 16:19:46 | johnsom | Hi Nova community. The Octavia team is seeing some strange behavior with libvirtd. https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c77/759973/3/check/octavia-v2-dsvm-scenario/c77fe63/controller/logs/libvirt/libvirtd_log.txt | |
| 16:19:52 | johnsom | Warning, that is a 136MB log file | |
| 16:20:08 | johnsom | It seems to go into some sort of tight loop: | |
| 16:20:26 | lyarwood | ops, opening that in chrome was a mistake | |
| 16:20:30 | johnsom | https://www.irccloud.com/pastebin/c6NcVeeF/ | |
| 16:20:54 | johnsom | Almost 90% of that log file is the above | |
| 16:20:59 | lyarwood | kashyap: ^ if you have time | |
| 16:21:05 | lyarwood | ring any bells? | |
| 16:21:38 | johnsom | Eventually it seems to finally exit. The issue we have is while it's stuck in this loop other VMs are not going ACTIVE in nova, so causes all sorts of timeouts. | |
| 16:21:39 | kashyap | lyarwood: Just a sec; lost in two other convo threads :-( | |
| 16:22:15 | lyarwood | kashyap: np | |
| 16:22:26 | kashyap | johnsom: As a quick question - in what scenario do you see that? | |
| 16:22:33 | lyarwood | johnsom: might be useful to write that up in a nova bug that kashyap can follow up with | |
| 16:22:37 | lyarwood | ops sorry | |
| 16:22:38 | kashyap | johnsom: That "End of file" ... simply means libvirt lost connection to QEMU | |
| 16:22:52 | johnsom | It appears that nova was trying to delete the instance | |
| 16:22:55 | kashyap | lyarwood: No worries; a bug also helps - as a record | |
| 16:23:45 | johnsom | Here is a chunk of the log right before the loop: | |
| 16:23:48 | johnsom | https://www.irccloud.com/pastebin/LRErlmPf/ | |
| 16:26:35 | kashyap | johnsom: Yeah, but that doesn't tell us what events led to that (they're in the 130MB log file). :-) Is this reproducible? | |
| 16:27:12 | johnsom | It is reproducible, but intermittent. Not every job triggers it. | |
| 16:28:05 | gibi | #nova now refresh connection_info / avoid storing stale connection_info in Nova | |
| 16:28:54 | johnsom | We are seeing it while we run our scenario test suites. | |
| 16:30:06 | kashyap | johnsom: A write-up in a bug would be nice to track. Along w/ a description of a test case that's hitting it. (I'm almost out of neurons tonight, and my mind now has the "thousand yard stare") | |
| 16:31:10 | johnsom | Yeah, I'm writing one up. I will link to the long, but I don't think I should attach it given it's size, so it will expire off the object store. | |
| 16:31:18 | johnsom | I will paste in a few snippets. | |
| 16:31:56 | kashyap | You can xz-compress the log; as the CI one might get cleaned up any moment | |
| 16:32:35 | kashyap | johnsom: Also, paste-bins expore; please add relevant snippets as a _text_ file. Because, LaunchPad, annoyingly enough, breaks formatting | |
| 16:32:46 | kashyap | Thanks :) | |
| 16:33:12 | johnsom | Ok | |
| 16:40:17 | kashyap | johnsom: (Sorry for being pedanctic.) | |
| 16:40:38 | johnsom | Oh no worries. I want to make sure you get what you need to help us out. | |
| 16:54:51 | johnsom | kashyap https://bugs.launchpad.net/nova/+bug/1902276 | |
| 16:54:51 | openstack | Launchpad bug 1902276 in OpenStack Compute (nova) "libvirtd going into a tight loop causing instances to not transition to ACTIVE" [Undecided,New] | |
| 16:59:59 | kashyap | johnsom: Thanks; another question - so the instances are not coming up online? | |
| 17:02:11 | kashyap | johnsom: Disregard me; you actually say that "eventually things go back to normal" | |
| 17:03:15 | johnsom | Yep | |
| 17:10:24 | kashyap | johnsom: I'm a bit baffled with it all :) I asked a couple of libvirt devs to see if the lead in and the end of the loop makes any further sense to them | |
| 17:10:43 | johnsom | Ok, thank you | |
| 17:15:46 | kashyap | johnsom: Haha, you write: "We seen this regularly, but it's intermittent" — now, which is it? | |
| 17:16:19 | johnsom | kashyap We see it on jobs daily, but it's not every run | |
| 17:16:54 | johnsom | Maybe that makes more sense? grin | |
| 17:18:10 | johnsom | It's been a long week, words are hard. grin | |
| 17:18:11 | kashyap | Ah, yes. That makes more sense | |
| 17:18:24 | kashyap | But something seems fishy; what changed so that things "go back to normal" | |
| 17:18:33 | kashyap | To be continued ... next week :) | |
| 17:18:34 | johnsom | Yeah, I have no idea | |
| 17:20:43 | johnsom | It repeats 276713 times in the log, so not a nice even number really. | |
| 18:10:35 | melwitt | dansmith: train backport of cherry pick check fix if you get a moment https://review.opendev.org/759118 | |
| 22:02:10 | openstackgerrit | Merged openstack/nova stable/train: Follow up for cherry-pick check for merge patch https://review.opendev.org/759118 | |
| 22:22:45 | openstackgerrit | melanie witt proposed openstack/nova stable/stein: Follow up for cherry-pick check for merge patch https://review.opendev.org/760673 | |
| #openstack-nova - 2020-10-31 | |||
| 06:31:41 | openstackgerrit | Merged openstack/nova master: Fix virsh domifstat to get vhostuser vif statistics https://review.opendev.org/757448 | |
| 07:38:23 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Update contributor guide for Wallaby https://review.opendev.org/754427 | |
| 08:14:33 | openstackgerrit | Merged openstack/nova master: Add regression test for bug #1899649 https://review.opendev.org/757893 | |
| 08:14:33 | openstack | bug 1899649 in OpenStack Compute (nova) "Volume marked as available after a failure to build" [Undecided,In progress] https://launchpad.net/bugs/1899649 - Assigned to Lee Yarwood (lyarwood) | |
| 08:26:21 | openstackgerrit | zhanghao proposed openstack/nova stable/victoria: Fix virsh domifstat to get vhostuser vif statistics https://review.opendev.org/760684 | |
| 08:28:38 | openstackgerrit | Jorhson Deng proposed openstack/nova master: limit the instance to backup while task state is not None https://review.opendev.org/760685 | |
| 10:54:34 | noonedeadpunk | hey, everyone. still having issue with isolated aggregates here... I've just got instances spawned on the isolated aggregate without any traits provided. and I ythink I have like 1-2 vms being spawned that way per month, so it's not something that I can reproduce but somehow that happens | |
| 10:56:20 | noonedeadpunk | nothing to look at in logs, but I don't have debug enabled since it prod env.... | |
| 11:20:22 | openstackgerrit | Merged openstack/nova master: objects: Fix issue in exception type https://review.opendev.org/756069 | |