| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-09-23 | |||
| 12:15:50 | openstackgerrit | Federico Ressi proposed openstack/nova master: Requires default python3-devel package on CentOS https://review.opendev.org/752073 | |
| 12:16:25 | openstackgerrit | Federico Ressi proposed openstack/nova master: Requires default python3-devel package on CentOS https://review.opendev.org/752073 | |
| 12:17:43 | openstackgerrit | Federico Ressi proposed openstack/nova master: Requires default python3-devel package on CentOS 8 https://review.opendev.org/752073 | |
| 12:18:13 | openstackgerrit | Federico Ressi proposed openstack/nova master: Log stack trace when fails launching a defined domain https://review.opendev.org/750652 | |
| 12:19:11 | brinzhang0 | sean-k-mooney: yeah, you are right, I can run it by "~/.local/bin/pre-commit run -v -a | |
| 12:19:12 | brinzhang0 | " | |
| 12:38:27 | kashyap | sean-k-mooney: Right, the exception is not the right thing, I did it for discussion, as I noted it in the commit message :) | |
| 12:51:40 | openstackgerrit | Merged openstack/nova master: test_evacuate.sh: Stop using libvirt-bin https://review.opendev.org/753267 | |
| 13:09:52 | lyarwood | gibi: ack looking | |
| 13:21:53 | lyarwood | gibi: tbh that looks more like a canonical support bug than an actual bug in Nova | |
| 13:21:59 | lyarwood | gibi: the change isn't related | |
| 13:24:02 | bauzas | the Focal saga again ? | |
| 13:24:09 | lyarwood | nope | |
| 13:24:19 | bauzas | lyarwood: you should ask a second payslip from Canonical, y'know :p | |
| 13:25:33 | gibi | lyarwood: thanks lyarwood for checking it | |
| 13:26:56 | lyarwood | ah actually it might be a libvirt issue after all | |
| 13:27:06 | lyarwood | that bug points to https://bugs.launchpad.net/nova/+bug/1885418 that has a trace | |
| 13:27:06 | openstack | Launchpad bug 1885418 in OpenStack Compute (nova) "libvirt.libvirtError: unable to verify existence of block copy target: Permission denied" [Undecided,Incomplete] | |
| 13:27:13 | lyarwood | I had asked the reporter for more details a while ago but never got any | |
| 13:32:09 | lyarwood | argh why is the gate useless this week of all weeks | |
| 13:41:06 | bauzas | lyarwood: tbh /me remembers old times when we were two days before the RC1 cut and the gate was like 5 days off | |
| 13:41:14 | bauzas | just because of all the changes | |
| 13:41:24 | bauzas | now, this is too easy | |
| 14:39:18 | lyarwood | gibi / bauzas ; https://bugs.launchpad.net/nova/+bug/1896617 updated, I can't reproduce it using devstack on Focal or F32. I've marked it as incomplete and asked for a reproducer against devstack before we look again. | |
| 14:39:18 | openstack | lyarwood: Error: Could not gather data from Launchpad for bug #1896617 (https://launchpad.net/bugs/1896617). The error has been logged | |
| 14:40:13 | lyarwood | https://bugs.launchpad.net/nova/+bug/1885418 also marked as a duplicate | |
| 14:40:13 | openstack | lyarwood: Error: Could not gather data from Launchpad for bug #1885418 (https://launchpad.net/bugs/1885418). The error has been logged | |
| 14:40:13 | gibi | lyarwood: much appreciated! | |
| 14:43:10 | artom | Yeah, Launchpad seems to have 'sploded | |
| 14:44:04 | artom | Ah no, it's back | |
| 14:48:25 | lyarwood | fun so https://6f4d572707e259c12736-93a8141b9f45324a3b85293f1fcfbf55.ssl.cf1.rackcdn.com/753581/2/check/devstack-plugin-ceph-tempest-fedora-latest/8cc45c2/job-output.txt is ceph not py3 only yet? | |
| 14:48:28 | artom | Weird, I have debug=True in nova.conf, but oslo_messaging appears to be only logging at INFO | |
| 14:48:38 | artom | Is there a different conf option that I'm not setting? | |
| 14:49:14 | lyarwood | there's a default level for specific libs but I thought DEBUG replaced that | |
| 14:49:14 | lyarwood | it's in oslo.log however | |
| 14:50:05 | lyarwood | artom: https://github.com/openstack/oslo.log/blob/51324b276a8a5f69847d3a17fa89924dabad4759/oslo_log/_options.py#L19-L31 | |
| 14:50:24 | lyarwood | artom: and https://github.com/openstack/oslo.log/blob/51324b276a8a5f69847d3a17fa89924dabad4759/oslo_log/_options.py#L175-L178 | |
| 14:50:53 | lyarwood | that reminds me I wanted to set os-brick to DEBUG downstream | |
| 14:52:34 | artom | lyarwood, aha, thanks | |
| 15:48:20 | openstackgerrit | Lee Yarwood proposed openstack/nova master: zuul: Introduce nova-evacuate https://review.opendev.org/744883 | |
| 16:06:36 | lyarwood | gibi++ | |
| 22:15:01 | openstackgerrit | Merged openstack/nova master: libvirt: Bump MIN_{LIBVIRT,QEMU}_VERSION and NEXT_MIN_{LIBVIRT,QEMU}_VERSION https://review.opendev.org/746981 | |
| 22:30:37 | lyarwood | \o/ | |
| 22:58:12 | lyarwood | melwitt: https://review.opendev.org/#/c/747978/ - would you mind looking at this if you have time today | |
| #openstack-nova - 2020-09-24 | |||
| 03:00:09 | openstackgerrit | Rui Zang proposed openstack/nova master: DO NOT REVIEW https://review.opendev.org/753919 | |
| 04:34:55 | openstackgerrit | Hemanth Nakkina proposed openstack/nova master: Update pci stat pools based on PCI device changes https://review.opendev.org/749175 | |
| 07:33:32 | gibi | I have to be offline for the next 2-3 hours but I will be back after lunch (UTC 11) | |
| 08:26:33 | openstackgerrit | Lucas Alvares Gomes proposed openstack/nova master: DO NOT REVIEW: Test OVN devstack module https://review.opendev.org/748226 | |
| 09:14:40 | openstackgerrit | Lee Yarwood proposed openstack/nova master: zuul: Replace nova-live-migration with zuulv3 jobs https://review.opendev.org/752557 | |
| 09:15:53 | lyarwood | stephenfin: https://review.opendev.org/#/c/744883 - would you mind hitting this back into the gate | |
| 09:16:03 | stephenfin | sure, looking | |
| 09:16:59 | lyarwood | had to be rebased to pickup the test-evacuate.sh change as the original live-migration job is still being used until https://review.opendev.org/752557 lands | |
| 09:17:15 | stephenfin | yup, makes sense | |
| 09:17:18 | stephenfin | +W | |
| 09:35:15 | lyarwood | bauzas: https://review.opendev.org/#/c/753281/ - Morning would you mind landing this in stable/ussuri please? | |
| 09:45:38 | CeeMac | lyarwood: morning :) I'm still trying to get my head around the process involved in the cinder retype initiated nova swap_volume. Is nova calling libvirt drive-mirror (or blockdev-mirror) to perform the sync, or is it relying on cinder copying/syncing the data to the new volume before doing an attachment swap? Trying to work out at which point the swap_volume job failure could be affecting the instance | |
| 09:46:11 | CeeMac | i've been reading through the driver.py and qemu block job docs | |
| 09:46:33 | CeeMac | but I lack a lot of context unfortunately | |
| 09:47:21 | CeeMac | i'm also trying to work out if any of the operations rely on / prefer the qemu quest agent and connection being present | |
| 10:06:54 | lyarwood | CeeMac: Morning sorry was afk | |
| 10:07:42 | lyarwood | CeeMac: so we call the https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainBlockRebase API at the moment (if libvirt is < 6.0.0) | |
| 10:08:25 | lyarwood | CeeMac: AFAIK we don't use the guest agent at all during the process | |
| 10:09:40 | lyarwood | CeeMac: in terms of https://bugs.launchpad.net/nova/+bug/1896621 the issue is with parallel requests to retype multiple attachments to the same instance AFAICT | |
| 10:09:40 | openstack | Launchpad bug 1896621 in OpenStack Compute (nova) "instance corrupted after volume retype" [High,Confirmed] - Assigned to Lee Yarwood (lyarwood) | |
| 10:09:52 | lyarwood | CeeMac: virDomainBlockRebase is disk specific so that isn't an issue | |
| 10:10:11 | lyarwood | CeeMac: but the final thing we do when swapping disks is to update the persistent config for the overall domain | |
| 10:10:32 | lyarwood | CeeMac: AFAICT this fails when other calls to virDomainBlockRebase and thus QEMU block jobs are still active | |
| 10:12:03 | lyarwood | CeeMac: I was going to throw an instance wide lock on the compute _swap_volume call to avoid this as I can reproduce the failure with libvirt > 6.0.0 where we now use the https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainBlockCopy libvirt API | |
| 10:19:54 | lyarwood | CeeMac: I'll try to find some time this afternoon to post a WIP | |
| 10:21:17 | lyarwood | CeeMac: I'll also try to work out if we can recover from this in your env, I've got a feeling it might be possible if we can work out the details of the new volume and attach it somehow to the instance | |
| 10:47:42 | brinzhang_ | stephenfin: thanks, yes, I missed sth, updated, and replied in the patch :D | |
| 10:58:19 | CeeMac | lyarwood: sorry, was just in a long meeting. thanks for the notes, I'll do some more reading! :) | |
| 10:59:48 | CeeMac | lyarwood: so, i wonder if it is the domain persistent config update. Or if it is just a direct result of the multiple block jobs, I need to stand up a disposable environment to validate if this affects all instances or specific ones | |
| 11:01:21 | CeeMac | lyarwood: for reference, i seem to have the most issue with Microsoft Active Directory Controllers. They really don't seem to like a lot of the normal operations. removing/recreating the instance from the same boot volume causes the same level of corruption. As does deleting the instance and recreating it from a nova snapshot. | |
| 11:02:31 | CeeMac | lyarwood: I had to delete the affected instance in the production environment, but I think I still have a corrupt instance in my dev area I could hold off on killing and rebuilding | |
| 11:15:14 | CeeMac | lyarwood: so if I'm understanding the docs correctly, you're using VIR_DOMAIN_BLOCK_REBASE_REUSE_EXT with VIR_DOMAIN_BLOCK_REBASE_COPY_DEV to point the rebase job at a volume device that has already been created and syning the content across to it? | |
| 11:22:19 | lyarwood | CeeMac: yes, I believe some cinder-volume backends clone things in the background so we don't end up copying everything across | |
| 11:22:34 | lyarwood | CeeMac: just the delta but backends like the LVM/iSCSI backend don't do this | |
| 11:23:03 | lyarwood | CeeMac: we then switch the volumes around within Cinder | |
| 11:23:45 | lyarwood | CeeMac: I think that's likely led to the corruption as we are pointing at the old likely unmapped volume | |
| 11:23:51 | lyarwood | CeeMac: but I'll need to reproduce to confirm | |
| 11:24:22 | CeeMac | that would make sense for the case where I've been able to log on to the instance fine but it broke after a reboot | |
| 11:25:01 | CeeMac | i'll have a poke around the configurations of one of the corrupt instances see if I can compare a dumpxml of attached volumes against expected attached volumes | |
| 11:54:21 | gibi | cores, I see the libvirt cleanup https://review.opendev.org/#/q/topic:bump-libvirt-qemu-victoria and the live migration / evacuation part of the zuul migration https://review.opendev.org/#/q/topic:native-zuulv3-migration+project:openstack/nova still actively pushed before RC1 | |
| 11:55:03 | gibi | is there anything else that I need to keep track of before / when I update the RC1 release patch? | |
| 11:55:19 | gibi | are we OK if they are not made it to RC1? | |
| 11:55:46 | gibi | I think the libvirt bump is merged just the cleanup is open so that is OK if we are not able to merge it before RC1 | |
| 11:55:55 | gibi | but what about the zuul migration? | |
| 11:56:05 | gibi | can we not merge them before RC1? | |
| 11:56:10 | gibi | gmann, lyarwood: ^^ | |
| 11:58:16 | lyarwood | gibi: I'm fine with the cleanups not landing | |
| 11:58:26 | lyarwood | gibi: for zuulv3 I'll defer to gmann | |
| 11:58:33 | gibi | thanks, noted | |
| 11:58:49 | gibi | I would like to make the RC1 right after the weekly meeting if possible | |
| 11:59:03 | gibi | but if things are on the gate then I think we can wait for them | |
| 12:00:52 | lyarwood | gibi: yeah I'd only wait for the zuulv3 stuff and only if gmann confirms that we need to tbh | |
| 12:01:18 | gibi | ack | |
| 12:01:19 | gibi | thanks | |
| 12:06:04 | kashyap | lyarwood: Where can I update this DevStack job using F31 (which is EOL) to F32? - https://zuul.opendev.org/t/vexxhost/job/devstack-platform-fedora-latest-virt-preview# | |
| 12:06:19 | kashyap | I'm wading through different CI / DevStack repos, thought I'd ask here first | |