| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-09-23 | |||
| 11:44:25 | brinzhang0 | but in my env I used Python 3.6.9 (default, Jul 17 2020, 12:50:27) | |
| 11:45:03 | kashyap | sean-k-mooney: Right; on deprecation and reno (added in the TODO in the commit) | |
| 11:45:12 | kashyap | sean-k-mooney: Also, one more thing before I go AFK for a bit: | |
| 11:46:46 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: [WIP] libvirt: Remove support for floppy drives https://review.opendev.org/753588 | |
| 11:47:28 | kashyap | sean-k-mooney: It is about the versioned object bump when removing / deprecating supported values for hw_floppy_bus | |
| 11:47:37 | kashyap | (In nova/objects/image_meta.py) | |
| 11:47:48 | kashyap | I also mentioned it in the commit message above | |
| 11:48:56 | sean-k-mooney | you can also use a floppy disk dirve as a parmater for hw_disk_bus or hw_rescue_bus | |
| 11:49:19 | sean-k-mooney | we cant raise raise exception.EmulatedFloppyDriveIsDeprecated() in wallby | |
| 11:49:29 | sean-k-mooney | this removes support it does not depcate it | |
| 11:50:04 | sean-k-mooney | we have to supprot them for one more cycle before we stop generating vms that can use floppy disk drives | |
| 11:58:41 | openstack | Launchpad bug 1896617 in OpenStack Compute (nova) "Creation of image (or live snapshot) from the existing VM fails if libvirt-image-backend is configured to qcow2 starting from Ussuri" [Undecided,New] | |
| 11:58:41 | gibi | lyarwood: hi! if you have some spare time, could you look at https://bugs.launchpad.net/nova/+bug/1896617 ? the reporter refers to your change https://review.opendev.org/#/c/696834/ so I assume you have better context than me | |
| 12:07:33 | brinzhang0 | sean-k-mooney: I saw https://review.opendev.org/#/c/702453/, that I can run "pip install --user pre-commit", but cannot run pre-commit command, do you have some suggestion? | |
| 12:07:45 | brinzhang0 | # pre-commit install --allow-missing-config | |
| 12:07:46 | brinzhang0 | -bash: pre-commit: command not found | |
| 12:13:35 | sean-k-mooney | brinzhang0: it may not be in you path by defualt | |
| 12:14:19 | sean-k-mooney | when you do a user install it proably put it in ~/.local/bin | |
| 12:14:31 | sean-k-mooney | not all distos default to putting that in your path | |
| 12:14:31 | brinzhang0 | maybe yes, I try to use the docs of https://pre-commit.com/#pre-commit-configyaml---top-level to have a try | |
| 12:15:04 | sean-k-mooney | i manually adde it to my path in my bashrc | |
| 12:15:50 | openstackgerrit | Federico Ressi proposed openstack/nova master: Requires default python3-devel package on CentOS https://review.opendev.org/752073 | |
| 12:16:25 | openstackgerrit | Federico Ressi proposed openstack/nova master: Requires default python3-devel package on CentOS https://review.opendev.org/752073 | |
| 12:17:43 | openstackgerrit | Federico Ressi proposed openstack/nova master: Requires default python3-devel package on CentOS 8 https://review.opendev.org/752073 | |
| 12:18:13 | openstackgerrit | Federico Ressi proposed openstack/nova master: Log stack trace when fails launching a defined domain https://review.opendev.org/750652 | |
| 12:19:11 | brinzhang0 | sean-k-mooney: yeah, you are right, I can run it by "~/.local/bin/pre-commit run -v -a | |
| 12:19:12 | brinzhang0 | " | |
| 12:38:27 | kashyap | sean-k-mooney: Right, the exception is not the right thing, I did it for discussion, as I noted it in the commit message :) | |
| 12:51:40 | openstackgerrit | Merged openstack/nova master: test_evacuate.sh: Stop using libvirt-bin https://review.opendev.org/753267 | |
| 13:09:52 | lyarwood | gibi: ack looking | |
| 13:21:53 | lyarwood | gibi: tbh that looks more like a canonical support bug than an actual bug in Nova | |
| 13:21:59 | lyarwood | gibi: the change isn't related | |
| 13:24:02 | bauzas | the Focal saga again ? | |
| 13:24:09 | lyarwood | nope | |
| 13:24:19 | bauzas | lyarwood: you should ask a second payslip from Canonical, y'know :p | |
| 13:25:33 | gibi | lyarwood: thanks lyarwood for checking it | |
| 13:26:56 | lyarwood | ah actually it might be a libvirt issue after all | |
| 13:27:06 | openstack | Launchpad bug 1885418 in OpenStack Compute (nova) "libvirt.libvirtError: unable to verify existence of block copy target: Permission denied" [Undecided,Incomplete] | |
| 13:27:06 | lyarwood | that bug points to https://bugs.launchpad.net/nova/+bug/1885418 that has a trace | |
| 13:27:13 | lyarwood | I had asked the reporter for more details a while ago but never got any | |
| 13:32:09 | lyarwood | argh why is the gate useless this week of all weeks | |
| 13:41:06 | bauzas | lyarwood: tbh /me remembers old times when we were two days before the RC1 cut and the gate was like 5 days off | |
| 13:41:14 | bauzas | just because of all the changes | |
| 13:41:24 | bauzas | now, this is too easy | |
| 14:39:18 | openstack | lyarwood: Error: Could not gather data from Launchpad for bug #1896617 (https://launchpad.net/bugs/1896617). The error has been logged | |
| 14:39:18 | lyarwood | gibi / bauzas ; https://bugs.launchpad.net/nova/+bug/1896617 updated, I can't reproduce it using devstack on Focal or F32. I've marked it as incomplete and asked for a reproducer against devstack before we look again. | |
| 14:40:13 | gibi | lyarwood: much appreciated! | |
| 14:40:13 | openstack | lyarwood: Error: Could not gather data from Launchpad for bug #1885418 (https://launchpad.net/bugs/1885418). The error has been logged | |
| 14:40:13 | lyarwood | https://bugs.launchpad.net/nova/+bug/1885418 also marked as a duplicate | |
| 14:43:10 | artom | Yeah, Launchpad seems to have 'sploded | |
| 14:44:04 | artom | Ah no, it's back | |
| 14:48:25 | lyarwood | fun so https://6f4d572707e259c12736-93a8141b9f45324a3b85293f1fcfbf55.ssl.cf1.rackcdn.com/753581/2/check/devstack-plugin-ceph-tempest-fedora-latest/8cc45c2/job-output.txt is ceph not py3 only yet? | |
| 14:48:28 | artom | Weird, I have debug=True in nova.conf, but oslo_messaging appears to be only logging at INFO | |
| 14:48:38 | artom | Is there a different conf option that I'm not setting? | |
| 14:49:14 | lyarwood | it's in oslo.log however | |
| 14:49:14 | lyarwood | there's a default level for specific libs but I thought DEBUG replaced that | |
| 14:50:05 | lyarwood | artom: https://github.com/openstack/oslo.log/blob/51324b276a8a5f69847d3a17fa89924dabad4759/oslo_log/_options.py#L19-L31 | |
| 14:50:24 | lyarwood | artom: and https://github.com/openstack/oslo.log/blob/51324b276a8a5f69847d3a17fa89924dabad4759/oslo_log/_options.py#L175-L178 | |
| 14:50:53 | lyarwood | that reminds me I wanted to set os-brick to DEBUG downstream | |
| 14:52:34 | artom | lyarwood, aha, thanks | |
| 15:48:20 | openstackgerrit | Lee Yarwood proposed openstack/nova master: zuul: Introduce nova-evacuate https://review.opendev.org/744883 | |
| 16:06:36 | lyarwood | gibi++ | |
| 22:15:01 | openstackgerrit | Merged openstack/nova master: libvirt: Bump MIN_{LIBVIRT,QEMU}_VERSION and NEXT_MIN_{LIBVIRT,QEMU}_VERSION https://review.opendev.org/746981 | |
| 22:30:37 | lyarwood | \o/ | |
| 22:58:12 | lyarwood | melwitt: https://review.opendev.org/#/c/747978/ - would you mind looking at this if you have time today | |
| #openstack-nova - 2020-09-24 | |||
| 03:00:09 | openstackgerrit | Rui Zang proposed openstack/nova master: DO NOT REVIEW https://review.opendev.org/753919 | |
| 04:34:55 | openstackgerrit | Hemanth Nakkina proposed openstack/nova master: Update pci stat pools based on PCI device changes https://review.opendev.org/749175 | |
| 07:33:32 | gibi | I have to be offline for the next 2-3 hours but I will be back after lunch (UTC 11) | |
| 08:26:33 | openstackgerrit | Lucas Alvares Gomes proposed openstack/nova master: DO NOT REVIEW: Test OVN devstack module https://review.opendev.org/748226 | |
| 09:14:40 | openstackgerrit | Lee Yarwood proposed openstack/nova master: zuul: Replace nova-live-migration with zuulv3 jobs https://review.opendev.org/752557 | |
| 09:15:53 | lyarwood | stephenfin: https://review.opendev.org/#/c/744883 - would you mind hitting this back into the gate | |
| 09:16:03 | stephenfin | sure, looking | |
| 09:16:59 | lyarwood | had to be rebased to pickup the test-evacuate.sh change as the original live-migration job is still being used until https://review.opendev.org/752557 lands | |
| 09:17:15 | stephenfin | yup, makes sense | |
| 09:17:18 | stephenfin | +W | |
| 09:35:15 | lyarwood | bauzas: https://review.opendev.org/#/c/753281/ - Morning would you mind landing this in stable/ussuri please? | |
| 09:45:38 | CeeMac | lyarwood: morning :) I'm still trying to get my head around the process involved in the cinder retype initiated nova swap_volume. Is nova calling libvirt drive-mirror (or blockdev-mirror) to perform the sync, or is it relying on cinder copying/syncing the data to the new volume before doing an attachment swap? Trying to work out at which point the swap_volume job failure could be affecting the instance | |
| 09:46:11 | CeeMac | i've been reading through the driver.py and qemu block job docs | |
| 09:46:33 | CeeMac | but I lack a lot of context unfortunately | |
| 09:47:21 | CeeMac | i'm also trying to work out if any of the operations rely on / prefer the qemu quest agent and connection being present | |
| 10:06:54 | lyarwood | CeeMac: Morning sorry was afk | |
| 10:07:42 | lyarwood | CeeMac: so we call the https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainBlockRebase API at the moment (if libvirt is < 6.0.0) | |
| 10:08:25 | lyarwood | CeeMac: AFAIK we don't use the guest agent at all during the process | |
| 10:09:40 | openstack | Launchpad bug 1896621 in OpenStack Compute (nova) "instance corrupted after volume retype" [High,Confirmed] - Assigned to Lee Yarwood (lyarwood) | |
| 10:09:40 | lyarwood | CeeMac: in terms of https://bugs.launchpad.net/nova/+bug/1896621 the issue is with parallel requests to retype multiple attachments to the same instance AFAICT | |
| 10:09:52 | lyarwood | CeeMac: virDomainBlockRebase is disk specific so that isn't an issue | |
| 10:10:11 | lyarwood | CeeMac: but the final thing we do when swapping disks is to update the persistent config for the overall domain | |
| 10:10:32 | lyarwood | CeeMac: AFAICT this fails when other calls to virDomainBlockRebase and thus QEMU block jobs are still active | |
| 10:12:03 | lyarwood | CeeMac: I was going to throw an instance wide lock on the compute _swap_volume call to avoid this as I can reproduce the failure with libvirt > 6.0.0 where we now use the https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainBlockCopy libvirt API | |
| 10:19:54 | lyarwood | CeeMac: I'll try to find some time this afternoon to post a WIP | |
| 10:21:17 | lyarwood | CeeMac: I'll also try to work out if we can recover from this in your env, I've got a feeling it might be possible if we can work out the details of the new volume and attach it somehow to the instance | |
| 10:47:42 | brinzhang_ | stephenfin: thanks, yes, I missed sth, updated, and replied in the patch :D | |
| 10:58:19 | CeeMac | lyarwood: sorry, was just in a long meeting. thanks for the notes, I'll do some more reading! :) | |
| 10:59:48 | CeeMac | lyarwood: so, i wonder if it is the domain persistent config update. Or if it is just a direct result of the multiple block jobs, I need to stand up a disposable environment to validate if this affects all instances or specific ones | |
| 11:01:21 | CeeMac | lyarwood: for reference, i seem to have the most issue with Microsoft Active Directory Controllers. They really don't seem to like a lot of the normal operations. removing/recreating the instance from the same boot volume causes the same level of corruption. As does deleting the instance and recreating it from a nova snapshot. | |
| 11:02:31 | CeeMac | lyarwood: I had to delete the affected instance in the production environment, but I think I still have a corrupt instance in my dev area I could hold off on killing and rebuilding | |
| 11:15:14 | CeeMac | lyarwood: so if I'm understanding the docs correctly, you're using VIR_DOMAIN_BLOCK_REBASE_REUSE_EXT with VIR_DOMAIN_BLOCK_REBASE_COPY_DEV to point the rebase job at a volume device that has already been created and syning the content across to it? | |
| 11:22:19 | lyarwood | CeeMac: yes, I believe some cinder-volume backends clone things in the background so we don't end up copying everything across | |
| 11:22:34 | lyarwood | CeeMac: just the delta but backends like the LVM/iSCSI backend don't do this | |
| 11:23:03 | lyarwood | CeeMac: we then switch the volumes around within Cinder | |