| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-09-24 | |||
| 10:08:25 | lyarwood | CeeMac: AFAIK we don't use the guest agent at all during the process | |
| 10:09:40 | openstack | Launchpad bug 1896621 in OpenStack Compute (nova) "instance corrupted after volume retype" [High,Confirmed] - Assigned to Lee Yarwood (lyarwood) | |
| 10:09:40 | lyarwood | CeeMac: in terms of https://bugs.launchpad.net/nova/+bug/1896621 the issue is with parallel requests to retype multiple attachments to the same instance AFAICT | |
| 10:09:52 | lyarwood | CeeMac: virDomainBlockRebase is disk specific so that isn't an issue | |
| 10:10:11 | lyarwood | CeeMac: but the final thing we do when swapping disks is to update the persistent config for the overall domain | |
| 10:10:32 | lyarwood | CeeMac: AFAICT this fails when other calls to virDomainBlockRebase and thus QEMU block jobs are still active | |
| 10:12:03 | lyarwood | CeeMac: I was going to throw an instance wide lock on the compute _swap_volume call to avoid this as I can reproduce the failure with libvirt > 6.0.0 where we now use the https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainBlockCopy libvirt API | |
| 10:19:54 | lyarwood | CeeMac: I'll try to find some time this afternoon to post a WIP | |
| 10:21:17 | lyarwood | CeeMac: I'll also try to work out if we can recover from this in your env, I've got a feeling it might be possible if we can work out the details of the new volume and attach it somehow to the instance | |
| 10:47:42 | brinzhang_ | stephenfin: thanks, yes, I missed sth, updated, and replied in the patch :D | |
| 10:58:19 | CeeMac | lyarwood: sorry, was just in a long meeting. thanks for the notes, I'll do some more reading! :) | |
| 10:59:48 | CeeMac | lyarwood: so, i wonder if it is the domain persistent config update. Or if it is just a direct result of the multiple block jobs, I need to stand up a disposable environment to validate if this affects all instances or specific ones | |
| 11:01:21 | CeeMac | lyarwood: for reference, i seem to have the most issue with Microsoft Active Directory Controllers. They really don't seem to like a lot of the normal operations. removing/recreating the instance from the same boot volume causes the same level of corruption. As does deleting the instance and recreating it from a nova snapshot. | |
| 11:02:31 | CeeMac | lyarwood: I had to delete the affected instance in the production environment, but I think I still have a corrupt instance in my dev area I could hold off on killing and rebuilding | |
| 11:15:14 | CeeMac | lyarwood: so if I'm understanding the docs correctly, you're using VIR_DOMAIN_BLOCK_REBASE_REUSE_EXT with VIR_DOMAIN_BLOCK_REBASE_COPY_DEV to point the rebase job at a volume device that has already been created and syning the content across to it? | |
| 11:22:19 | lyarwood | CeeMac: yes, I believe some cinder-volume backends clone things in the background so we don't end up copying everything across | |
| 11:22:34 | lyarwood | CeeMac: just the delta but backends like the LVM/iSCSI backend don't do this | |
| 11:23:03 | lyarwood | CeeMac: we then switch the volumes around within Cinder | |
| 11:23:45 | lyarwood | CeeMac: I think that's likely led to the corruption as we are pointing at the old likely unmapped volume | |
| 11:23:51 | lyarwood | CeeMac: but I'll need to reproduce to confirm | |
| 11:24:22 | CeeMac | that would make sense for the case where I've been able to log on to the instance fine but it broke after a reboot | |
| 11:25:01 | CeeMac | i'll have a poke around the configurations of one of the corrupt instances see if I can compare a dumpxml of attached volumes against expected attached volumes | |
| 11:54:21 | gibi | cores, I see the libvirt cleanup https://review.opendev.org/#/q/topic:bump-libvirt-qemu-victoria and the live migration / evacuation part of the zuul migration https://review.opendev.org/#/q/topic:native-zuulv3-migration+project:openstack/nova still actively pushed before RC1 | |
| 11:55:03 | gibi | is there anything else that I need to keep track of before / when I update the RC1 release patch? | |
| 11:55:19 | gibi | are we OK if they are not made it to RC1? | |
| 11:55:46 | gibi | I think the libvirt bump is merged just the cleanup is open so that is OK if we are not able to merge it before RC1 | |
| 11:55:55 | gibi | but what about the zuul migration? | |
| 11:56:05 | gibi | can we not merge them before RC1? | |
| 11:56:10 | gibi | gmann, lyarwood: ^^ | |
| 11:58:16 | lyarwood | gibi: I'm fine with the cleanups not landing | |
| 11:58:26 | lyarwood | gibi: for zuulv3 I'll defer to gmann | |
| 11:58:33 | gibi | thanks, noted | |
| 11:58:49 | gibi | I would like to make the RC1 right after the weekly meeting if possible | |
| 11:59:03 | gibi | but if things are on the gate then I think we can wait for them | |
| 12:00:52 | lyarwood | gibi: yeah I'd only wait for the zuulv3 stuff and only if gmann confirms that we need to tbh | |
| 12:01:18 | gibi | ack | |
| 12:01:19 | gibi | thanks | |
| 12:06:04 | kashyap | lyarwood: Where can I update this DevStack job using F31 (which is EOL) to F32? - https://zuul.opendev.org/t/vexxhost/job/devstack-platform-fedora-latest-virt-preview# | |
| 12:06:19 | kashyap | I'm wading through different CI / DevStack repos, thought I'd ask here first | |
| 12:06:46 | lyarwood | kashyap: I'm trying to do that https://review.opendev.org/#/c/750292/ already | |
| 12:06:52 | kashyap | Oh, good that I asked | |
| 12:07:06 | lyarwood | kashyap: https://review.opendev.org/#/q/topic:f32 for the full set of changes required FWIW | |
| 12:07:32 | kashyap | lyarwood: I was just fixing the conflicts here and rebasing the CI job (https://review.opendev.org/#/c/708701/) | |
| 12:07:39 | kashyap | lyarwood: Sweet | |
| 12:08:23 | kashyap | Ah, phew; I already was thinking about 'd-i-b', I see that it's already merged | |
| 12:09:49 | sean-k-mooney | speaking of dib i need to talk to infra or in openstack-dib about alpine support | |
| 12:10:08 | sean-k-mooney | i have started on it but im hitting some issues | |
| 12:10:37 | kashyap | One would think using a image would be a simple "drag-n-drop" of a pre-built image cache (a.k.a. virt-builder) | |
| 12:10:55 | kashyap | ... instead of this kind of a 'diff' across eight files: https://review.opendev.org/#/c/737217/ | |
| 12:11:02 | kashyap | But, that ship has sailed eight years ago ;-) | |
| 12:11:20 | sean-k-mooney | yep i really dislike virt-builder so im happy we use dib | |
| 12:11:26 | kashyap | sean-k-mooney: *Alpine*? I don't think it is one of the distros that matter | |
| 12:11:31 | kashyap | :D | |
| 12:11:39 | sean-k-mooney | kashyap: im trying to replace cirros | |
| 12:11:47 | kashyap | sean-k-mooney: What is your reason for dislike of 'virt-builder'? I think it does a solid job for its purpose | |
| 12:11:50 | sean-k-mooney | with a maintianed disto | |
| 12:11:55 | kashyap | And has saved my times countless times | |
| 12:12:14 | kashyap | It's well-documented, deadly simple to use, and it does many smart-and-useful things. | |
| 12:12:19 | kashyap | I don't see a single reason to dislike it :D | |
| 12:12:22 | sean-k-mooney | kashyap: i really dont like its workflow and it had rather poor supprot for non rpm distros orginailly | |
| 12:12:39 | sean-k-mooney | i want a declaritive interface which si what the element provide | |
| 12:12:53 | kashyap | sean-k-mooney: Even for non-RPM distros, it even lets you create Ubuntu and Debian images | |
| 12:13:09 | sean-k-mooney | kashyap: right but i hae always had issue trying to use it | |
| 12:13:11 | kashyap | Its main purpose is to allow you creating / modifying images _quickly_ | |
| 12:13:29 | kashyap | What exact issue? I'm surprised such a simple tool fails for you | |
| 12:13:35 | sean-k-mooney | for me its never been quick and i dont find its docs that easy to folow | |
| 12:14:00 | sean-k-mooney | the first road block whas always been user managment | |
| 12:14:16 | sean-k-mooney | injection of ssh keys ectra | |
| 12:14:38 | sean-k-mooney | the package customisation is not that bad but it used to not work well behind proxies | |
| 12:14:39 | kashyap | - First, on quickness: it uses minimal kickstart for RPM-based, and similar for Debian-based | |
| 12:14:46 | kashyap | It can't get quicker than what the mirrors allow | |
| 12:14:51 | sean-k-mooney | so it was a pain to get working behind the intel firewall | |
| 12:15:42 | kashyap | - Second, the "docs" are quite readable too (very few open source tools can claim that): https://libguestfs.org/virt-builder.1.html | |
| 12:15:57 | kashyap | For Fedora users, I wrote this dead-simple guide some years ago - https://developer.fedoraproject.org/tools/virt-builder/about.html | |
| 12:16:25 | sean-k-mooney | i can take a look but i have read the docs and i dont find them that useful | |
| 12:16:44 | sean-k-mooney | i tried virt builder before dib | |
| 12:17:04 | sean-k-mooney | i only tried dib after i could not get it to do what i wanted | |
| 12:17:17 | kashyap | sean-k-mooney: User management is also easy: | |
| 12:17:18 | kashyap | chage -d 0 stack | |
| 12:17:18 | kashyap | useradd -m -p "" -G wheel stack | |
| 12:17:18 | kashyap | # Create the user account. | |
| 12:17:18 | kashyap | virt-builder [...] --firstboot-command ' | |
| 12:17:20 | kashyap | chmod 0755 /home/stack | |
| 12:17:23 | kashyap | mkdir -m 0755 /home/stack/.ssh' | |
| 12:17:44 | sean-k-mooney | how do you do that declaritvly with out writing a bash script | |
| 12:17:54 | kashyap | sean-k-mooney: I know we're going deep into a non-Nova topic, but 'd-i-b' had the terrible practise of using root overall | |
| 12:18:27 | kashyap | sean-k-mooney: We discussed it when the tool was being written at that time; and pointed to better alternatives ... but they went NIH | |
| 12:18:52 | sean-k-mooney | it use a chroot | |
| 12:19:00 | sean-k-mooney | so root is not system root | |
| 12:19:09 | sean-k-mooney | but ya not a nova topic lets drop it | |
| 12:20:30 | kashyap | sean-k-mooney: It used to use system root, and we discussed the security implications at length. HP at that time wasn't convinced. | |
| 12:20:51 | kashyap | (Yeah, non-Nova, but indirectly very useful topic for Nova Infra :D) | |
| 12:27:35 | kashyap | sean-k-mooney: Not to belabour, but the "diskimage-builder" was so terribly unsafe, at that time the libguestfs folks went and created a "safe wrapper" for it, 'virt-dib': https://manpages.debian.org/testing/libguestfs-tools/virt-dib.1.en.html | |
| 12:33:30 | sean-k-mooney | i see well i still dont think we should replace it with virt-install | |
| 12:33:40 | sean-k-mooney | *virt-builder | |
| 12:34:12 | sean-k-mooney | regardless of the implementaion the design is more approtiate for a git based declaritve workflow IMO | |
| 12:34:15 | CeeMac | lyarwood: quick question, when the domain persistence config is recreated following a successful swap_volume operation, would this generate any new SSIDs against or within the config? | |
| 12:34:29 | sean-k-mooney | which is why i wrote https://github.com/intel-orchestration-software/dib-elements a few year ago using it instead of virt builder | |
| 12:40:57 | kashyap | sean-k-mooney: Yeah, yeah ... I wasn't arguing we shouldn't replace it now (too tedious, and can't justify the effort). The purpose of both tools are different | |
| 12:56:53 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: WIP: nova-next: Start testing the 'q35' machine type https://review.opendev.org/708701 | |