| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-09-23 | |||
| 14:49:14 | lyarwood | there's a default level for specific libs but I thought DEBUG replaced that | |
| 14:49:14 | lyarwood | it's in oslo.log however | |
| 14:50:05 | lyarwood | artom: https://github.com/openstack/oslo.log/blob/51324b276a8a5f69847d3a17fa89924dabad4759/oslo_log/_options.py#L19-L31 | |
| 14:50:24 | lyarwood | artom: and https://github.com/openstack/oslo.log/blob/51324b276a8a5f69847d3a17fa89924dabad4759/oslo_log/_options.py#L175-L178 | |
| 14:50:53 | lyarwood | that reminds me I wanted to set os-brick to DEBUG downstream | |
| 14:52:34 | artom | lyarwood, aha, thanks | |
| 15:48:20 | openstackgerrit | Lee Yarwood proposed openstack/nova master: zuul: Introduce nova-evacuate https://review.opendev.org/744883 | |
| 16:06:36 | lyarwood | gibi++ | |
| 22:15:01 | openstackgerrit | Merged openstack/nova master: libvirt: Bump MIN_{LIBVIRT,QEMU}_VERSION and NEXT_MIN_{LIBVIRT,QEMU}_VERSION https://review.opendev.org/746981 | |
| 22:30:37 | lyarwood | \o/ | |
| 22:58:12 | lyarwood | melwitt: https://review.opendev.org/#/c/747978/ - would you mind looking at this if you have time today | |
| #openstack-nova - 2020-09-24 | |||
| 03:00:09 | openstackgerrit | Rui Zang proposed openstack/nova master: DO NOT REVIEW https://review.opendev.org/753919 | |
| 04:34:55 | openstackgerrit | Hemanth Nakkina proposed openstack/nova master: Update pci stat pools based on PCI device changes https://review.opendev.org/749175 | |
| 07:33:32 | gibi | I have to be offline for the next 2-3 hours but I will be back after lunch (UTC 11) | |
| 08:26:33 | openstackgerrit | Lucas Alvares Gomes proposed openstack/nova master: DO NOT REVIEW: Test OVN devstack module https://review.opendev.org/748226 | |
| 09:14:40 | openstackgerrit | Lee Yarwood proposed openstack/nova master: zuul: Replace nova-live-migration with zuulv3 jobs https://review.opendev.org/752557 | |
| 09:15:53 | lyarwood | stephenfin: https://review.opendev.org/#/c/744883 - would you mind hitting this back into the gate | |
| 09:16:03 | stephenfin | sure, looking | |
| 09:16:59 | lyarwood | had to be rebased to pickup the test-evacuate.sh change as the original live-migration job is still being used until https://review.opendev.org/752557 lands | |
| 09:17:15 | stephenfin | yup, makes sense | |
| 09:17:18 | stephenfin | +W | |
| 09:35:15 | lyarwood | bauzas: https://review.opendev.org/#/c/753281/ - Morning would you mind landing this in stable/ussuri please? | |
| 09:45:38 | CeeMac | lyarwood: morning :) I'm still trying to get my head around the process involved in the cinder retype initiated nova swap_volume. Is nova calling libvirt drive-mirror (or blockdev-mirror) to perform the sync, or is it relying on cinder copying/syncing the data to the new volume before doing an attachment swap? Trying to work out at which point the swap_volume job failure could be affecting the instance | |
| 09:46:11 | CeeMac | i've been reading through the driver.py and qemu block job docs | |
| 09:46:33 | CeeMac | but I lack a lot of context unfortunately | |
| 09:47:21 | CeeMac | i'm also trying to work out if any of the operations rely on / prefer the qemu quest agent and connection being present | |
| 10:06:54 | lyarwood | CeeMac: Morning sorry was afk | |
| 10:07:42 | lyarwood | CeeMac: so we call the https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainBlockRebase API at the moment (if libvirt is < 6.0.0) | |
| 10:08:25 | lyarwood | CeeMac: AFAIK we don't use the guest agent at all during the process | |
| 10:09:40 | lyarwood | CeeMac: in terms of https://bugs.launchpad.net/nova/+bug/1896621 the issue is with parallel requests to retype multiple attachments to the same instance AFAICT | |
| 10:09:40 | openstack | Launchpad bug 1896621 in OpenStack Compute (nova) "instance corrupted after volume retype" [High,Confirmed] - Assigned to Lee Yarwood (lyarwood) | |
| 10:09:52 | lyarwood | CeeMac: virDomainBlockRebase is disk specific so that isn't an issue | |
| 10:10:11 | lyarwood | CeeMac: but the final thing we do when swapping disks is to update the persistent config for the overall domain | |
| 10:10:32 | lyarwood | CeeMac: AFAICT this fails when other calls to virDomainBlockRebase and thus QEMU block jobs are still active | |
| 10:12:03 | lyarwood | CeeMac: I was going to throw an instance wide lock on the compute _swap_volume call to avoid this as I can reproduce the failure with libvirt > 6.0.0 where we now use the https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainBlockCopy libvirt API | |
| 10:19:54 | lyarwood | CeeMac: I'll try to find some time this afternoon to post a WIP | |
| 10:21:17 | lyarwood | CeeMac: I'll also try to work out if we can recover from this in your env, I've got a feeling it might be possible if we can work out the details of the new volume and attach it somehow to the instance | |
| 10:47:42 | brinzhang_ | stephenfin: thanks, yes, I missed sth, updated, and replied in the patch :D | |
| 10:58:19 | CeeMac | lyarwood: sorry, was just in a long meeting. thanks for the notes, I'll do some more reading! :) | |
| 10:59:48 | CeeMac | lyarwood: so, i wonder if it is the domain persistent config update. Or if it is just a direct result of the multiple block jobs, I need to stand up a disposable environment to validate if this affects all instances or specific ones | |
| 11:01:21 | CeeMac | lyarwood: for reference, i seem to have the most issue with Microsoft Active Directory Controllers. They really don't seem to like a lot of the normal operations. removing/recreating the instance from the same boot volume causes the same level of corruption. As does deleting the instance and recreating it from a nova snapshot. | |
| 11:02:31 | CeeMac | lyarwood: I had to delete the affected instance in the production environment, but I think I still have a corrupt instance in my dev area I could hold off on killing and rebuilding | |
| 11:15:14 | CeeMac | lyarwood: so if I'm understanding the docs correctly, you're using VIR_DOMAIN_BLOCK_REBASE_REUSE_EXT with VIR_DOMAIN_BLOCK_REBASE_COPY_DEV to point the rebase job at a volume device that has already been created and syning the content across to it? | |
| 11:22:19 | lyarwood | CeeMac: yes, I believe some cinder-volume backends clone things in the background so we don't end up copying everything across | |
| 11:22:34 | lyarwood | CeeMac: just the delta but backends like the LVM/iSCSI backend don't do this | |
| 11:23:03 | lyarwood | CeeMac: we then switch the volumes around within Cinder | |
| 11:23:45 | lyarwood | CeeMac: I think that's likely led to the corruption as we are pointing at the old likely unmapped volume | |
| 11:23:51 | lyarwood | CeeMac: but I'll need to reproduce to confirm | |
| 11:24:22 | CeeMac | that would make sense for the case where I've been able to log on to the instance fine but it broke after a reboot | |
| 11:25:01 | CeeMac | i'll have a poke around the configurations of one of the corrupt instances see if I can compare a dumpxml of attached volumes against expected attached volumes | |
| 11:54:21 | gibi | cores, I see the libvirt cleanup https://review.opendev.org/#/q/topic:bump-libvirt-qemu-victoria and the live migration / evacuation part of the zuul migration https://review.opendev.org/#/q/topic:native-zuulv3-migration+project:openstack/nova still actively pushed before RC1 | |
| 11:55:03 | gibi | is there anything else that I need to keep track of before / when I update the RC1 release patch? | |
| 11:55:19 | gibi | are we OK if they are not made it to RC1? | |
| 11:55:46 | gibi | I think the libvirt bump is merged just the cleanup is open so that is OK if we are not able to merge it before RC1 | |
| 11:55:55 | gibi | but what about the zuul migration? | |
| 11:56:05 | gibi | can we not merge them before RC1? | |
| 11:56:10 | gibi | gmann, lyarwood: ^^ | |
| 11:58:16 | lyarwood | gibi: I'm fine with the cleanups not landing | |
| 11:58:26 | lyarwood | gibi: for zuulv3 I'll defer to gmann | |
| 11:58:33 | gibi | thanks, noted | |
| 11:58:49 | gibi | I would like to make the RC1 right after the weekly meeting if possible | |
| 11:59:03 | gibi | but if things are on the gate then I think we can wait for them | |
| 12:00:52 | lyarwood | gibi: yeah I'd only wait for the zuulv3 stuff and only if gmann confirms that we need to tbh | |
| 12:01:18 | gibi | ack | |
| 12:01:19 | gibi | thanks | |
| 12:06:04 | kashyap | lyarwood: Where can I update this DevStack job using F31 (which is EOL) to F32? - https://zuul.opendev.org/t/vexxhost/job/devstack-platform-fedora-latest-virt-preview# | |
| 12:06:19 | kashyap | I'm wading through different CI / DevStack repos, thought I'd ask here first | |
| 12:06:46 | lyarwood | kashyap: I'm trying to do that https://review.opendev.org/#/c/750292/ already | |
| 12:06:52 | kashyap | Oh, good that I asked | |
| 12:07:06 | lyarwood | kashyap: https://review.opendev.org/#/q/topic:f32 for the full set of changes required FWIW | |
| 12:07:32 | kashyap | lyarwood: I was just fixing the conflicts here and rebasing the CI job (https://review.opendev.org/#/c/708701/) | |
| 12:07:39 | kashyap | lyarwood: Sweet | |
| 12:08:23 | kashyap | Ah, phew; I already was thinking about 'd-i-b', I see that it's already merged | |
| 12:09:49 | sean-k-mooney | speaking of dib i need to talk to infra or in openstack-dib about alpine support | |
| 12:10:08 | sean-k-mooney | i have started on it but im hitting some issues | |
| 12:10:37 | kashyap | One would think using a image would be a simple "drag-n-drop" of a pre-built image cache (a.k.a. virt-builder) | |
| 12:10:55 | kashyap | ... instead of this kind of a 'diff' across eight files: https://review.opendev.org/#/c/737217/ | |
| 12:11:02 | kashyap | But, that ship has sailed eight years ago ;-) | |
| 12:11:20 | sean-k-mooney | yep i really dislike virt-builder so im happy we use dib | |
| 12:11:26 | kashyap | sean-k-mooney: *Alpine*? I don't think it is one of the distros that matter | |
| 12:11:31 | kashyap | :D | |
| 12:11:39 | sean-k-mooney | kashyap: im trying to replace cirros | |
| 12:11:47 | kashyap | sean-k-mooney: What is your reason for dislike of 'virt-builder'? I think it does a solid job for its purpose | |
| 12:11:50 | sean-k-mooney | with a maintianed disto | |
| 12:11:55 | kashyap | And has saved my times countless times | |
| 12:12:14 | kashyap | It's well-documented, deadly simple to use, and it does many smart-and-useful things. | |
| 12:12:19 | kashyap | I don't see a single reason to dislike it :D | |
| 12:12:22 | sean-k-mooney | kashyap: i really dont like its workflow and it had rather poor supprot for non rpm distros orginailly | |
| 12:12:39 | sean-k-mooney | i want a declaritive interface which si what the element provide | |
| 12:12:53 | kashyap | sean-k-mooney: Even for non-RPM distros, it even lets you create Ubuntu and Debian images | |
| 12:13:09 | sean-k-mooney | kashyap: right but i hae always had issue trying to use it | |
| 12:13:11 | kashyap | Its main purpose is to allow you creating / modifying images _quickly_ | |
| 12:13:29 | kashyap | What exact issue? I'm surprised such a simple tool fails for you | |
| 12:13:35 | sean-k-mooney | for me its never been quick and i dont find its docs that easy to folow | |
| 12:14:00 | sean-k-mooney | the first road block whas always been user managment | |
| 12:14:16 | sean-k-mooney | injection of ssh keys ectra | |
| 12:14:38 | sean-k-mooney | the package customisation is not that bad but it used to not work well behind proxies | |
| 12:14:39 | kashyap | - First, on quickness: it uses minimal kickstart for RPM-based, and similar for Debian-based | |
| 12:14:46 | kashyap | It can't get quicker than what the mirrors allow | |
| 12:14:51 | sean-k-mooney | so it was a pain to get working behind the intel firewall | |