Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-09-24
10:21:17 lyarwood CeeMac: I'll also try to work out if we can recover from this in your env, I've got a feeling it might be possible if we can work out the details of the new volume and attach it somehow to the instance
10:47:42 brinzhang_ stephenfin: thanks, yes, I missed sth, updated, and replied in the patch :D
10:58:19 CeeMac lyarwood: sorry, was just in a long meeting. thanks for the notes, I'll do some more reading! :)
10:59:48 CeeMac lyarwood: so, i wonder if it is the domain persistent config update. Or if it is just a direct result of the multiple block jobs, I need to stand up a disposable environment to validate if this affects all instances or specific ones
11:01:21 CeeMac lyarwood: for reference, i seem to have the most issue with Microsoft Active Directory Controllers. They really don't seem to like a lot of the normal operations. removing/recreating the instance from the same boot volume causes the same level of corruption. As does deleting the instance and recreating it from a nova snapshot.
11:02:31 CeeMac lyarwood: I had to delete the affected instance in the production environment, but I think I still have a corrupt instance in my dev area I could hold off on killing and rebuilding
11:15:14 CeeMac lyarwood: so if I'm understanding the docs correctly, you're using VIR_DOMAIN_BLOCK_REBASE_REUSE_EXT with VIR_DOMAIN_BLOCK_REBASE_COPY_DEV to point the rebase job at a volume device that has already been created and syning the content across to it?
11:22:19 lyarwood CeeMac: yes, I believe some cinder-volume backends clone things in the background so we don't end up copying everything across
11:22:34 lyarwood CeeMac: just the delta but backends like the LVM/iSCSI backend don't do this
11:23:03 lyarwood CeeMac: we then switch the volumes around within Cinder
11:23:45 lyarwood CeeMac: I think that's likely led to the corruption as we are pointing at the old likely unmapped volume
11:23:51 lyarwood CeeMac: but I'll need to reproduce to confirm
11:24:22 CeeMac that would make sense for the case where I've been able to log on to the instance fine but it broke after a reboot
11:25:01 CeeMac i'll have a poke around the configurations of one of the corrupt instances see if I can compare a dumpxml of attached volumes against expected attached volumes
11:54:21 gibi cores, I see the libvirt cleanup https://review.opendev.org/#/q/topic:bump-libvirt-qemu-victoria and the live migration / evacuation part of the zuul migration https://review.opendev.org/#/q/topic:native-zuulv3-migration+project:openstack/nova still actively pushed before RC1
11:55:03 gibi is there anything else that I need to keep track of before / when I update the RC1 release patch?
11:55:19 gibi are we OK if they are not made it to RC1?
11:55:46 gibi I think the libvirt bump is merged just the cleanup is open so that is OK if we are not able to merge it before RC1
11:55:55 gibi but what about the zuul migration?
11:56:05 gibi can we not merge them before RC1?
11:56:10 gibi gmann, lyarwood: ^^
11:58:16 lyarwood gibi: I'm fine with the cleanups not landing
11:58:26 lyarwood gibi: for zuulv3 I'll defer to gmann
11:58:33 gibi thanks, noted
11:58:49 gibi I would like to make the RC1 right after the weekly meeting if possible
11:59:03 gibi but if things are on the gate then I think we can wait for them
12:00:52 lyarwood gibi: yeah I'd only wait for the zuulv3 stuff and only if gmann confirms that we need to tbh
12:01:18 gibi ack
12:01:19 gibi thanks
12:06:04 kashyap lyarwood: Where can I update this DevStack job using F31 (which is EOL) to F32? - https://zuul.opendev.org/t/vexxhost/job/devstack-platform-fedora-latest-virt-preview#
12:06:19 kashyap I'm wading through different CI / DevStack repos, thought I'd ask here first
12:06:46 lyarwood kashyap: I'm trying to do that https://review.opendev.org/#/c/750292/ already
12:06:52 kashyap Oh, good that I asked
12:07:06 lyarwood kashyap: https://review.opendev.org/#/q/topic:f32 for the full set of changes required FWIW
12:07:32 kashyap lyarwood: I was just fixing the conflicts here and rebasing the CI job (https://review.opendev.org/#/c/708701/)
12:07:39 kashyap lyarwood: Sweet
12:08:23 kashyap Ah, phew; I already was thinking about 'd-i-b', I see that it's already merged
12:09:49 sean-k-mooney speaking of dib i need to talk to infra or in openstack-dib about alpine support
12:10:08 sean-k-mooney i have started on it but im hitting some issues
12:10:37 kashyap One would think using a image would be a simple "drag-n-drop" of a pre-built image cache (a.k.a. virt-builder)
12:10:55 kashyap ... instead of this kind of a 'diff' across eight files: https://review.opendev.org/#/c/737217/
12:11:02 kashyap But, that ship has sailed eight years ago ;-)
12:11:20 sean-k-mooney yep i really dislike virt-builder so im happy we use dib
12:11:26 kashyap sean-k-mooney: *Alpine*? I don't think it is one of the distros that matter
12:11:31 kashyap :D
12:11:39 sean-k-mooney kashyap: im trying to replace cirros
12:11:47 kashyap sean-k-mooney: What is your reason for dislike of 'virt-builder'? I think it does a solid job for its purpose
12:11:50 sean-k-mooney with a maintianed disto
12:11:55 kashyap And has saved my times countless times
12:12:14 kashyap It's well-documented, deadly simple to use, and it does many smart-and-useful things.
12:12:19 kashyap I don't see a single reason to dislike it :D
12:12:22 sean-k-mooney kashyap: i really dont like its workflow and it had rather poor supprot for non rpm distros orginailly
12:12:39 sean-k-mooney i want a declaritive interface which si what the element provide
12:12:53 kashyap sean-k-mooney: Even for non-RPM distros, it even lets you create Ubuntu and Debian images
12:13:09 sean-k-mooney kashyap: right but i hae always had issue trying to use it
12:13:11 kashyap Its main purpose is to allow you creating / modifying images _quickly_
12:13:29 kashyap What exact issue? I'm surprised such a simple tool fails for you
12:13:35 sean-k-mooney for me its never been quick and i dont find its docs that easy to folow
12:14:00 sean-k-mooney the first road block whas always been user managment
12:14:16 sean-k-mooney injection of ssh keys ectra
12:14:38 sean-k-mooney the package customisation is not that bad but it used to not work well behind proxies
12:14:39 kashyap - First, on quickness: it uses minimal kickstart for RPM-based, and similar for Debian-based
12:14:46 kashyap It can't get quicker than what the mirrors allow
12:14:51 sean-k-mooney so it was a pain to get working behind the intel firewall
12:15:42 kashyap - Second, the "docs" are quite readable too (very few open source tools can claim that): https://libguestfs.org/virt-builder.1.html
12:15:57 kashyap For Fedora users, I wrote this dead-simple guide some years ago - https://developer.fedoraproject.org/tools/virt-builder/about.html
12:16:25 sean-k-mooney i can take a look but i have read the docs and i dont find them that useful
12:16:44 sean-k-mooney i tried virt builder before dib
12:17:04 sean-k-mooney i only tried dib after i could not get it to do what i wanted
12:17:17 kashyap sean-k-mooney: User management is also easy:
12:17:18 kashyap virt-builder [...] --firstboot-command '
12:17:18 kashyap # Create the user account.
12:17:18 kashyap useradd -m -p "" -G wheel stack
12:17:18 kashyap chage -d 0 stack
12:17:20 kashyap chmod 0755 /home/stack
12:17:23 kashyap mkdir -m 0755 /home/stack/.ssh'
12:17:44 sean-k-mooney how do you do that declaritvly with out writing a bash script
12:17:54 kashyap sean-k-mooney: I know we're going deep into a non-Nova topic, but 'd-i-b' had the terrible practise of using root overall
12:18:27 kashyap sean-k-mooney: We discussed it when the tool was being written at that time; and pointed to better alternatives ... but they went NIH
12:18:52 sean-k-mooney it use a chroot
12:19:00 sean-k-mooney so root is not system root
12:19:09 sean-k-mooney but ya not a nova topic lets drop it
12:20:30 kashyap sean-k-mooney: It used to use system root, and we discussed the security implications at length. HP at that time wasn't convinced.
12:20:51 kashyap (Yeah, non-Nova, but indirectly very useful topic for Nova Infra :D)
12:27:35 kashyap sean-k-mooney: Not to belabour, but the "diskimage-builder" was so terribly unsafe, at that time the libguestfs folks went and created a "safe wrapper" for it, 'virt-dib': https://manpages.debian.org/testing/libguestfs-tools/virt-dib.1.en.html
12:33:30 sean-k-mooney i see well i still dont think we should replace it with virt-install
12:33:40 sean-k-mooney *virt-builder
12:34:12 sean-k-mooney regardless of the implementaion the design is more approtiate for a git based declaritve workflow IMO
12:34:15 CeeMac lyarwood: quick question, when the domain persistence config is recreated following a successful swap_volume operation, would this generate any new SSIDs against or within the config?
12:34:29 sean-k-mooney which is why i wrote https://github.com/intel-orchestration-software/dib-elements a few year ago using it instead of virt builder
12:40:57 kashyap sean-k-mooney: Yeah, yeah ... I wasn't arguing we shouldn't replace it now (too tedious, and can't justify the effort). The purpose of both tools are different
12:56:53 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: WIP: nova-next: Start testing the 'q35' machine type https://review.opendev.org/708701
12:57:45 kashyap Cc: lyarwood --^ In my above modif, I've nuked the "check" queue altogether, as I see that the 'virt-preview' job is now added to the experimental queue upstream
13:00:03 kashyap (Also see that I've added your F32 support as a Depends-On. Let me know if I messed up anything in the above job config :-))
13:02:17 lyarwood kashyap: yeah you can't do that
13:02:25 lyarwood kashyap: you need to leave a single job in the check queue
13:02:37 lyarwood kashyap: just move the virt-preview job there
13:02:53 kashyap lyarwood: Aah, okay. Noted. Lemme edit :)
13:03:28 kashyap lyarwood: Aside: Sorry, but your "You can't do that" reminded me of a certain president and his conversation with an Australian journalist ;-)
13:09:50 kashyap lyarwood: BTW, also should at least one job be present in the 'gate' queue?

Earlier   Later