Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-09-23
13:32:09 lyarwood argh why is the gate useless this week of all weeks
13:41:06 bauzas lyarwood: tbh /me remembers old times when we were two days before the RC1 cut and the gate was like 5 days off
13:41:14 bauzas just because of all the changes
13:41:24 bauzas now, this is too easy
14:39:18 openstack lyarwood: Error: Could not gather data from Launchpad for bug #1896617 (https://launchpad.net/bugs/1896617). The error has been logged
14:39:18 lyarwood gibi / bauzas ; https://bugs.launchpad.net/nova/+bug/1896617 updated, I can't reproduce it using devstack on Focal or F32. I've marked it as incomplete and asked for a reproducer against devstack before we look again.
14:40:13 gibi lyarwood: much appreciated!
14:40:13 openstack lyarwood: Error: Could not gather data from Launchpad for bug #1885418 (https://launchpad.net/bugs/1885418). The error has been logged
14:40:13 lyarwood https://bugs.launchpad.net/nova/+bug/1885418 also marked as a duplicate
14:43:10 artom Yeah, Launchpad seems to have 'sploded
14:44:04 artom Ah no, it's back
14:48:25 lyarwood fun so https://6f4d572707e259c12736-93a8141b9f45324a3b85293f1fcfbf55.ssl.cf1.rackcdn.com/753581/2/check/devstack-plugin-ceph-tempest-fedora-latest/8cc45c2/job-output.txt is ceph not py3 only yet?
14:48:28 artom Weird, I have debug=True in nova.conf, but oslo_messaging appears to be only logging at INFO
14:48:38 artom Is there a different conf option that I'm not setting?
14:49:14 lyarwood it's in oslo.log however
14:49:14 lyarwood there's a default level for specific libs but I thought DEBUG replaced that
14:50:05 lyarwood artom: https://github.com/openstack/oslo.log/blob/51324b276a8a5f69847d3a17fa89924dabad4759/oslo_log/_options.py#L19-L31
14:50:24 lyarwood artom: and https://github.com/openstack/oslo.log/blob/51324b276a8a5f69847d3a17fa89924dabad4759/oslo_log/_options.py#L175-L178
14:50:53 lyarwood that reminds me I wanted to set os-brick to DEBUG downstream
14:52:34 artom lyarwood, aha, thanks
15:48:20 openstackgerrit Lee Yarwood proposed openstack/nova master: zuul: Introduce nova-evacuate https://review.opendev.org/744883
16:06:36 lyarwood gibi++
22:15:01 openstackgerrit Merged openstack/nova master: libvirt: Bump MIN_{LIBVIRT,QEMU}_VERSION and NEXT_MIN_{LIBVIRT,QEMU}_VERSION https://review.opendev.org/746981
22:30:37 lyarwood \o/
22:58:12 lyarwood melwitt: https://review.opendev.org/#/c/747978/ - would you mind looking at this if you have time today
#openstack-nova - 2020-09-24
03:00:09 openstackgerrit Rui Zang proposed openstack/nova master: DO NOT REVIEW https://review.opendev.org/753919
04:34:55 openstackgerrit Hemanth Nakkina proposed openstack/nova master: Update pci stat pools based on PCI device changes https://review.opendev.org/749175
07:33:32 gibi I have to be offline for the next 2-3 hours but I will be back after lunch (UTC 11)
08:26:33 openstackgerrit Lucas Alvares Gomes proposed openstack/nova master: DO NOT REVIEW: Test OVN devstack module https://review.opendev.org/748226
09:14:40 openstackgerrit Lee Yarwood proposed openstack/nova master: zuul: Replace nova-live-migration with zuulv3 jobs https://review.opendev.org/752557
09:15:53 lyarwood stephenfin: https://review.opendev.org/#/c/744883 - would you mind hitting this back into the gate
09:16:03 stephenfin sure, looking
09:16:59 lyarwood had to be rebased to pickup the test-evacuate.sh change as the original live-migration job is still being used until https://review.opendev.org/752557 lands
09:17:15 stephenfin yup, makes sense
09:17:18 stephenfin +W
09:35:15 lyarwood bauzas: https://review.opendev.org/#/c/753281/ - Morning would you mind landing this in stable/ussuri please?
09:45:38 CeeMac lyarwood: morning :) I'm still trying to get my head around the process involved in the cinder retype initiated nova swap_volume. Is nova calling libvirt drive-mirror (or blockdev-mirror) to perform the sync, or is it relying on cinder copying/syncing the data to the new volume before doing an attachment swap? Trying to work out at which point the swap_volume job failure could be affecting the instance
09:46:11 CeeMac i've been reading through the driver.py and qemu block job docs
09:46:33 CeeMac but I lack a lot of context unfortunately
09:47:21 CeeMac i'm also trying to work out if any of the operations rely on / prefer the qemu quest agent and connection being present
10:06:54 lyarwood CeeMac: Morning sorry was afk
10:07:42 lyarwood CeeMac: so we call the https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainBlockRebase API at the moment (if libvirt is < 6.0.0)
10:08:25 lyarwood CeeMac: AFAIK we don't use the guest agent at all during the process
10:09:40 openstack Launchpad bug 1896621 in OpenStack Compute (nova) "instance corrupted after volume retype" [High,Confirmed] - Assigned to Lee Yarwood (lyarwood)
10:09:40 lyarwood CeeMac: in terms of https://bugs.launchpad.net/nova/+bug/1896621 the issue is with parallel requests to retype multiple attachments to the same instance AFAICT
10:09:52 lyarwood CeeMac: virDomainBlockRebase is disk specific so that isn't an issue
10:10:11 lyarwood CeeMac: but the final thing we do when swapping disks is to update the persistent config for the overall domain
10:10:32 lyarwood CeeMac: AFAICT this fails when other calls to virDomainBlockRebase and thus QEMU block jobs are still active
10:12:03 lyarwood CeeMac: I was going to throw an instance wide lock on the compute _swap_volume call to avoid this as I can reproduce the failure with libvirt > 6.0.0 where we now use the https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainBlockCopy libvirt API
10:19:54 lyarwood CeeMac: I'll try to find some time this afternoon to post a WIP
10:21:17 lyarwood CeeMac: I'll also try to work out if we can recover from this in your env, I've got a feeling it might be possible if we can work out the details of the new volume and attach it somehow to the instance
10:47:42 brinzhang_ stephenfin: thanks, yes, I missed sth, updated, and replied in the patch :D
10:58:19 CeeMac lyarwood: sorry, was just in a long meeting. thanks for the notes, I'll do some more reading! :)
10:59:48 CeeMac lyarwood: so, i wonder if it is the domain persistent config update. Or if it is just a direct result of the multiple block jobs, I need to stand up a disposable environment to validate if this affects all instances or specific ones
11:01:21 CeeMac lyarwood: for reference, i seem to have the most issue with Microsoft Active Directory Controllers. They really don't seem to like a lot of the normal operations. removing/recreating the instance from the same boot volume causes the same level of corruption. As does deleting the instance and recreating it from a nova snapshot.
11:02:31 CeeMac lyarwood: I had to delete the affected instance in the production environment, but I think I still have a corrupt instance in my dev area I could hold off on killing and rebuilding
11:15:14 CeeMac lyarwood: so if I'm understanding the docs correctly, you're using VIR_DOMAIN_BLOCK_REBASE_REUSE_EXT with VIR_DOMAIN_BLOCK_REBASE_COPY_DEV to point the rebase job at a volume device that has already been created and syning the content across to it?
11:22:19 lyarwood CeeMac: yes, I believe some cinder-volume backends clone things in the background so we don't end up copying everything across
11:22:34 lyarwood CeeMac: just the delta but backends like the LVM/iSCSI backend don't do this
11:23:03 lyarwood CeeMac: we then switch the volumes around within Cinder
11:23:45 lyarwood CeeMac: I think that's likely led to the corruption as we are pointing at the old likely unmapped volume
11:23:51 lyarwood CeeMac: but I'll need to reproduce to confirm
11:24:22 CeeMac that would make sense for the case where I've been able to log on to the instance fine but it broke after a reboot
11:25:01 CeeMac i'll have a poke around the configurations of one of the corrupt instances see if I can compare a dumpxml of attached volumes against expected attached volumes
11:54:21 gibi cores, I see the libvirt cleanup https://review.opendev.org/#/q/topic:bump-libvirt-qemu-victoria and the live migration / evacuation part of the zuul migration https://review.opendev.org/#/q/topic:native-zuulv3-migration+project:openstack/nova still actively pushed before RC1
11:55:03 gibi is there anything else that I need to keep track of before / when I update the RC1 release patch?
11:55:19 gibi are we OK if they are not made it to RC1?
11:55:46 gibi I think the libvirt bump is merged just the cleanup is open so that is OK if we are not able to merge it before RC1
11:55:55 gibi but what about the zuul migration?
11:56:05 gibi can we not merge them before RC1?
11:56:10 gibi gmann, lyarwood: ^^
11:58:16 lyarwood gibi: I'm fine with the cleanups not landing
11:58:26 lyarwood gibi: for zuulv3 I'll defer to gmann
11:58:33 gibi thanks, noted
11:58:49 gibi I would like to make the RC1 right after the weekly meeting if possible
11:59:03 gibi but if things are on the gate then I think we can wait for them
12:00:52 lyarwood gibi: yeah I'd only wait for the zuulv3 stuff and only if gmann confirms that we need to tbh
12:01:18 gibi ack
12:01:19 gibi thanks
12:06:04 kashyap lyarwood: Where can I update this DevStack job using F31 (which is EOL) to F32? - https://zuul.opendev.org/t/vexxhost/job/devstack-platform-fedora-latest-virt-preview#
12:06:19 kashyap I'm wading through different CI / DevStack repos, thought I'd ask here first
12:06:46 lyarwood kashyap: I'm trying to do that https://review.opendev.org/#/c/750292/ already
12:06:52 kashyap Oh, good that I asked
12:07:06 lyarwood kashyap: https://review.opendev.org/#/q/topic:f32 for the full set of changes required FWIW
12:07:32 kashyap lyarwood: I was just fixing the conflicts here and rebasing the CI job (https://review.opendev.org/#/c/708701/)
12:07:39 kashyap lyarwood: Sweet
12:08:23 kashyap Ah, phew; I already was thinking about 'd-i-b', I see that it's already merged
12:09:49 sean-k-mooney speaking of dib i need to talk to infra or in openstack-dib about alpine support
12:10:08 sean-k-mooney i have started on it but im hitting some issues
12:10:37 kashyap One would think using a image would be a simple "drag-n-drop" of a pre-built image cache (a.k.a. virt-builder)
12:10:55 kashyap ... instead of this kind of a 'diff' across eight files: https://review.opendev.org/#/c/737217/
12:11:02 kashyap But, that ship has sailed eight years ago ;-)
12:11:20 sean-k-mooney yep i really dislike virt-builder so im happy we use dib
12:11:26 kashyap sean-k-mooney: *Alpine*? I don't think it is one of the distros that matter
12:11:31 kashyap :D
12:11:39 sean-k-mooney kashyap: im trying to replace cirros
12:11:47 kashyap sean-k-mooney: What is your reason for dislike of 'virt-builder'? I think it does a solid job for its purpose
12:11:50 sean-k-mooney with a maintianed disto
12:11:55 kashyap And has saved my times countless times
12:12:14 kashyap It's well-documented, deadly simple to use, and it does many smart-and-useful things.

Earlier   Later