Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-09-25
12:24:30 lyarwood at least that's my understanding
12:24:35 elod lyarwood: the pep8 tools/check-cherry-picks.sh fails without the tag
12:26:02 lyarwood ah my bad sorry
12:26:09 lyarwood I forgot that we had that check in there
12:27:22 elod yes, it's good most of the cases, but sometimes makes our life harder :)
12:28:29 elod at least the fix is just a commit message update, so not that painful :)
12:29:42 lyarwood yup true
12:44:44 openstack Launchpad bug 1896621 in OpenStack Compute (nova) "instance corrupted after volume retype" [High,Confirmed] - Assigned to Lee Yarwood (lyarwood)
12:44:44 lyarwood CeeMac: so I can reproduce the race in https://bugs.launchpad.net/nova/+bug/1896621
12:44:57 lyarwood CeeMac: looking at the corruption now, I assume serials are going to change once I reboot
12:45:35 CeeMac lyarwood: glad you were able to reproduce it
12:46:30 lyarwood CeeMac: so in that case the serials are correct after a hard reboot
12:46:45 lyarwood CeeMac: actually did you use --hard in the reboot command?
12:46:55 lyarwood CeeMac: I wonder if the issue is with a soft reboot here
12:47:27 CeeMac Lyarwood well
12:47:30 lyarwood oh wow the volume hasn't rolled back
12:48:06 CeeMac lyarwood: in one instance the server restarted itself and then powered itself off
12:48:07 lyarwood the instance is pointing to the original volume but cinder thinks that has retyped/migrated to the new type
12:48:21 CeeMac lyarwood: another one I soft rebooted
12:49:17 CeeMac lyarwood: is that visible somewhere?
12:49:49 CeeMac lyarwood: when I checked openstack volume show it had migration state error but original volume id
12:50:25 CeeMac And the disk mappings seemed correct in /dev/disk/by-path matching lun id etc
12:50:33 lyarwood CeeMac: ah sorry I was looking at the wrong volume
12:50:40 lyarwood CeeMac: the first volume is in that state
12:50:57 lyarwood CeeMac: migration status error and listed against the original type
12:51:08 CeeMac lyarwood: right
12:58:43 lyarwood CeeMac: yeah I can't make this fail and change the disk serials
12:58:53 lyarwood CeeMac: I'll update the bug with some ideas
12:59:40 CeeMac Thanks lyarwood
13:08:13 lyarwood CeeMac: was there much of a gap between the failure and the instances being rebooted?
13:08:45 lyarwood CeeMac: I think I see the issue now, we don't actually tell libvirt to revert back to the original volumes within the domain itself
13:09:04 lyarwood CeeMac: if we fail late when dumping the domain XML
13:20:23 CeeMac lyarwood: in the instance that broke itself its hard to say as it rebooted and powered itself off.
13:21:02 CeeMac lyarwood: one of the other instances there was a day or two before I manually rebooted
13:25:35 CeeMac lyarwood: sounds like you're on to something
13:25:58 lyarwood CeeMac: kk, I think the issue is that libvirt leaves the new volume connected after this failure, the instance keeps writing to the new volume until we rip out the volume from underneath it likely sending the disk into a read-only state within the instance. By the time the instance reboots it's out of sync with the original volume that is now reconnected and that's where the failure comes
13:25:59 lyarwood from.
13:26:46 CeeMac lyarwood: during the swap_volume process you mean?
13:26:50 lyarwood let me see if I can reproduce this on master with a Fedora VM so I can debug this a little more and prove that's the case
13:27:04 CeeMac lyarwood: so the new device is attached for a period until the job fails and is reverted?
13:27:06 lyarwood CeeMac: so we fail *after* the swap volume operation has completed
13:27:23 lyarwood CeeMac: yeah I believe so
13:27:46 lyarwood CeeMac: there's nothing in the failure path within Nova that would revert the active domain configuration AFAICT
13:33:35 openstackgerrit Merged openstack/nova master: libvirt: make mdev types name attribute be optional https://review.opendev.org/753574
13:34:34 CeeMac lyarwood: that could explain it. Appreciate you looking into this so diligently :)
13:47:39 openstackgerrit Takashi Natsume proposed openstack/nova master: Remove six.moves https://review.opendev.org/727224
13:47:55 openstackgerrit Takashi Natsume proposed openstack/nova master: Remove six.iteritems/itervalues/iterkeys https://review.opendev.org/727757
13:48:46 openstackgerrit Takashi Natsume proposed openstack/nova master: Remove six.byte2int/int2byte https://review.opendev.org/727777
13:49:12 openstackgerrit Takashi Natsume proposed openstack/nova master: Remove six.binary_type/integer_types/string_types https://review.opendev.org/728094
13:50:26 openstackgerrit Takashi Natsume proposed openstack/nova master: Remove six.text_type (1/2) https://review.opendev.org/728109
13:54:41 openstackgerrit Takashi Natsume proposed openstack/nova master: Add placeholder migrations for Victoria backports https://review.opendev.org/754396
14:05:18 openstackgerrit Takashi Natsume proposed openstack/nova master: Remove six.text_type (2/2) https://review.opendev.org/728117
14:16:52 openstackgerrit Stephen Finucane proposed openstack/nova stable/victoria: libvirt: make mdev types name attribute be optional https://review.opendev.org/754401
14:17:00 stephenfin bauzas: ^
14:18:37 lyarwood stephenfin: do we want to land that in stable before we are certain we don't want another rc?
14:19:06 stephenfin um, I don't understand the question
14:19:13 gibi if we merge someting meaningful to stable/victoria now then we have to cut RC2 I guess
14:19:13 stephenfin are you saying we should wait to land this in 22.0.1?
14:19:17 stephenfin ah
14:19:19 stephenfin gotcha
14:19:35 stephenfin then no, that's not critical IMO
14:19:47 stephenfin I just know bauzas was eager for it to land at some point
14:20:00 gibi yeah I would only land critical things to stable/v at the moment
14:20:06 stephenfin makes sense
14:21:47 gibi also landing things on master should be limited to make a possible critical bug backport easy
14:50:59 melwitt lyarwood: hey, did you miss adding a NOTE here for the backport conflicts? https://review.opendev.org/747978
14:52:08 lyarwood melwitt: looking
14:52:53 lyarwood melwitt: oh is this the pike one
14:53:18 melwitt yeah, I wasn't sure if a NOTE was left out intentionally or not
14:53:34 lyarwood melwitt: I didn't write them up for this sorry, I posted it as a courtesy for someone
14:53:58 lyarwood that and the Newton patch attached to the bug
14:54:06 melwitt ah gotcha
15:20:58 openstackgerrit Takashi Natsume proposed openstack/nova master: Update contributor guide for Wallaby https://review.opendev.org/754427
15:26:28 openstackgerrit melanie witt proposed openstack/nova stable/pike: libvirt: Provide VIR_MIGRATE_PARAM_PERSIST_XML during live migration https://review.opendev.org/747978
15:34:19 openstackgerrit Merged openstack/os-vif stable/victoria: Update .gitreview for stable/victoria https://review.opendev.org/750924
15:41:14 openstackgerrit Merged openstack/nova master: Use absolute path during qemu img rebase https://review.opendev.org/739246
15:41:21 openstackgerrit Merged openstack/nova stable/ussuri: Move revert resize under semaphore https://review.opendev.org/751354
15:41:32 openstackgerrit Merged openstack/nova master: libvirt: Remove MIN_LIBVIRT_FILE_BACKED_DISCARD_VERSION https://review.opendev.org/746982
15:49:23 openstackgerrit George Melikov proposed openstack/nova master: Instance snapshot: fallback to usual upload on ImageNotAuthorized https://review.opendev.org/754442
16:00:41 openstackgerrit melanie witt proposed openstack/nova stable/pike: Lowercase ironic driver hash ring and ignore case in cache https://review.opendev.org/723055
16:00:42 openstackgerrit melanie witt proposed openstack/nova stable/pike: Include only required fields in ironic node cache https://review.opendev.org/754444
16:02:27 openstackgerrit melanie witt proposed openstack/nova stable/pike: Include only required fields in ironic node cache https://review.opendev.org/754444
16:02:28 openstackgerrit melanie witt proposed openstack/nova stable/pike: Lowercase ironic driver hash ring and ignore case in cache https://review.opendev.org/723055
16:13:08 stephenfin melwitt: Care to review this functional reproducer for me, please? https://review.opendev.org/#/c/749713/
16:13:41 melwitt stephenfin: will do
16:13:45 stephenfin thanks :)
16:14:08 stephenfin lyarwood: If you're still about, this is a follow-up to a series you looked at https://review.opendev.org/#/c/749319/
16:15:53 lyarwood stephenfin: yup still here, if I don't get to today I'll make sure I do on Monday
16:16:02 stephenfin Good enough for me. Thanks
16:23:01 openstackgerrit Stephen Finucane proposed openstack/nova master: neutron: Remove references to out-of-tree 'qos-queue' extension https://review.opendev.org/754448
16:23:01 openstackgerrit Stephen Finucane proposed openstack/nova master: Rework how we check for neutron extensions https://review.opendev.org/705792
16:24:28 openstackgerrit Merged openstack/python-novaclient stable/ussuri: Add a cleanup for a server in a functional test https://review.opendev.org/743756
16:28:34 openstackgerrit Stephen Finucane proposed openstack/nova master: objects: Don't use generic 'Field' container https://review.opendev.org/738239
16:28:35 openstackgerrit Stephen Finucane proposed openstack/nova master: objects: Use imports instead of type aliases https://review.opendev.org/738018
16:28:35 openstackgerrit Stephen Finucane proposed openstack/nova master: objects: Remove unnecessary type aliases, exceptions https://review.opendev.org/738240
16:28:36 openstackgerrit Stephen Finucane proposed openstack/nova master: WIP: objects: Add type hints https://review.opendev.org/738020
16:28:36 openstackgerrit Stephen Finucane proposed openstack/nova master: objects: Remove wrappers around ovo mixins https://review.opendev.org/738019
16:31:44 lyarwood CeeMac: took a while but I've confirmed what I said before is what is happening btw, writing it up now.
16:32:27 openstackgerrit Stephen Finucane proposed openstack/nova master: Remove use of pkg_resources https://review.opendev.org/740661
16:35:00 CeeMac lyarwood: good work, thanks

Earlier   Later