Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-09-25
11:18:01 CeeMac or, which version of libvirt was the patch introduced in may be a better question kashyap
11:22:07 kashyap CeeMac: It is fixed in libvirt v5.0.0 and above
11:22:13 lyarwood CeeMac: https://review.opendev.org/#/q/I0c52917a5555a70c4973f37dea1aebf878dd73b4 - I fixed that years ago, sorry I missed that kashyap suggested that
11:22:14 CeeMac ah
11:22:30 CeeMac i'm on v4
11:22:38 CeeMac which explains why i still see it
11:22:47 kashyap OH, yes; that too - the 'mirror' element patch from lyarwood
11:23:52 CeeMac lyarwood: should that not be present in rocky anyhow if it was introduced around ocata? (presumably)?
11:24:06 lyarwood CeeMac: yes my change should be in stable/rocky
11:24:09 kashyap Yes, it should be in Rocky
11:24:47 CeeMac hmm
11:24:57 CeeMac then i'm still confused how I'm gettting it :/
11:25:01 kashyap lyarwood: Unrelated - I can't reproduce the Focal / DEVICE_DEL thing for the life of me :-( Maybe my chops for setting up synthetic workload are bad
11:25:11 CeeMac unless it would still require libvirt v5?
11:25:22 kashyap CeeMac: Hmm, for "lulz", can you try with libvirt-v5, if you can afford to?
11:25:42 CeeMac kashyap: you make that sound so easy :D
11:25:55 kashyap I know; I'm a terribly bad person for phrasing it that way :D
11:26:28 CeeMac is it a straight forward upgrade? or are there bits and bobs that plug in that would be affected
11:26:53 CeeMac i have a test environment which i have these issues in that i also got in prod, that is on the same version (running OSA)
11:27:04 CeeMac i was planning an upgrade from rocky to train anyhow
11:27:17 CeeMac not sure which libvirt comes down with that, will check
11:27:28 openstackgerrit Balazs Gibizer proposed openstack/nova master: Reproduce bug 1896463 in func env https://review.opendev.org/754100
11:27:28 openstack bug 1896463 in OpenStack Compute (nova) ussuri "evacuation failed: Port update failed : Unable to correlate PCI slot " [Low,Triaged] https://launchpad.net/bugs/1896463
11:27:35 kashyap CeeMac: Good question; when upgrading libvirt (assuming you're not using any container-based deployments):
11:27:56 CeeMac no containers kashyap
11:28:23 kashyap CeeMac: ... if you also see a QEMU update being pulled in: then you need to cold-reboot (stop + start) the instances for the libvirt to pick up the new QEMU
11:29:02 kashyap (If there's no QEMU update; then 'just' restarting the libvirtd will suffice - yes, your instance will stay intact during libvirtd restart)
11:29:20 CeeMac its like you've been asked these questions before :D
11:30:23 CeeMac just need to find out if powercycling the other instances that have failed retype on their attached volumes also breaks the instance, then I'll ponder options for upgrades etc
11:30:29 openstackgerrit kuang congxian proposed openstack/nova master: Replace six with python 3 style code in nova/test https://review.opendev.org/754354
11:31:19 kashyap CeeMac: Yes, very many times. Also some important bits here: https://kashyapc.fedorapeople.org/Effective-Virtual-CPU-Configuration-in-Nova-Berlin2018.pdf
11:31:59 kashyap CeeMac: (Slightly updated variant of the above, from Jan-2020: https://kashyapc.fedorapeople.org/Secure-CPU-config-for-KVM-based-guests-RHT_TechDay_Brussels_2020.pdf)
11:50:24 CeeMac kashyap: thanks, appreciate it
11:51:44 kashyap CeeMac: I mentioned the above because you also need to take into account "security" :D - (thanks to the CPU hardware flaws that flew by, and are still flowing by)
11:52:14 CeeMac soooo many security flaws in the last few years!
11:53:19 kashyap Yea, are you seeing the timeline slide? ;-)
11:55:41 CeeMac didn't look at it yet, was just thinking it terms of meltdown/spectre variants in general
11:56:54 CeeMac which I now see is on the slide :D
12:20:51 openstackgerrit Elod Illes proposed openstack/nova stable/victoria: [stable-only] Update .gitreview for stable/victoria https://review.opendev.org/754319
12:20:52 openstackgerrit Elod Illes proposed openstack/nova stable/victoria: [stable-only] Update TOX_CONSTRAINTS_FILE for stable/victoria https://review.opendev.org/754320
12:22:30 openstackgerrit Harshavardhan Metla proposed openstack/nova master: Updated the weights section of Compute schedulers in nova https://review.opendev.org/754365
12:22:50 lyarwood elod: the stable-only tags aren't technically required there at the moment as it isn't a full stable branch just yet
12:22:54 lyarwood elod: until GA
12:23:19 lyarwood elod: we will use the stable/$release branch to cut another rc tag if required
12:23:34 lyarwood etc
12:24:30 lyarwood at least that's my understanding
12:24:35 elod lyarwood: the pep8 tools/check-cherry-picks.sh fails without the tag
12:26:02 lyarwood ah my bad sorry
12:26:09 lyarwood I forgot that we had that check in there
12:27:22 elod yes, it's good most of the cases, but sometimes makes our life harder :)
12:28:29 elod at least the fix is just a commit message update, so not that painful :)
12:29:42 lyarwood yup true
12:44:44 lyarwood CeeMac: so I can reproduce the race in https://bugs.launchpad.net/nova/+bug/1896621
12:44:44 openstack Launchpad bug 1896621 in OpenStack Compute (nova) "instance corrupted after volume retype" [High,Confirmed] - Assigned to Lee Yarwood (lyarwood)
12:44:57 lyarwood CeeMac: looking at the corruption now, I assume serials are going to change once I reboot
12:45:35 CeeMac lyarwood: glad you were able to reproduce it
12:46:30 lyarwood CeeMac: so in that case the serials are correct after a hard reboot
12:46:45 lyarwood CeeMac: actually did you use --hard in the reboot command?
12:46:55 lyarwood CeeMac: I wonder if the issue is with a soft reboot here
12:47:27 CeeMac Lyarwood well
12:47:30 lyarwood oh wow the volume hasn't rolled back
12:48:06 CeeMac lyarwood: in one instance the server restarted itself and then powered itself off
12:48:07 lyarwood the instance is pointing to the original volume but cinder thinks that has retyped/migrated to the new type
12:48:21 CeeMac lyarwood: another one I soft rebooted
12:49:17 CeeMac lyarwood: is that visible somewhere?
12:49:49 CeeMac lyarwood: when I checked openstack volume show it had migration state error but original volume id
12:50:25 CeeMac And the disk mappings seemed correct in /dev/disk/by-path matching lun id etc
12:50:33 lyarwood CeeMac: ah sorry I was looking at the wrong volume
12:50:40 lyarwood CeeMac: the first volume is in that state
12:50:57 lyarwood CeeMac: migration status error and listed against the original type
12:51:08 CeeMac lyarwood: right
12:58:43 lyarwood CeeMac: yeah I can't make this fail and change the disk serials
12:58:53 lyarwood CeeMac: I'll update the bug with some ideas
12:59:40 CeeMac Thanks lyarwood
13:08:13 lyarwood CeeMac: was there much of a gap between the failure and the instances being rebooted?
13:08:45 lyarwood CeeMac: I think I see the issue now, we don't actually tell libvirt to revert back to the original volumes within the domain itself
13:09:04 lyarwood CeeMac: if we fail late when dumping the domain XML
13:20:23 CeeMac lyarwood: in the instance that broke itself its hard to say as it rebooted and powered itself off.
13:21:02 CeeMac lyarwood: one of the other instances there was a day or two before I manually rebooted
13:25:35 CeeMac lyarwood: sounds like you're on to something
13:25:58 lyarwood CeeMac: kk, I think the issue is that libvirt leaves the new volume connected after this failure, the instance keeps writing to the new volume until we rip out the volume from underneath it likely sending the disk into a read-only state within the instance. By the time the instance reboots it's out of sync with the original volume that is now reconnected and that's where the failure comes
13:25:59 lyarwood from.
13:26:46 CeeMac lyarwood: during the swap_volume process you mean?
13:26:50 lyarwood let me see if I can reproduce this on master with a Fedora VM so I can debug this a little more and prove that's the case
13:27:04 CeeMac lyarwood: so the new device is attached for a period until the job fails and is reverted?
13:27:06 lyarwood CeeMac: so we fail *after* the swap volume operation has completed
13:27:23 lyarwood CeeMac: yeah I believe so
13:27:46 lyarwood CeeMac: there's nothing in the failure path within Nova that would revert the active domain configuration AFAICT
13:33:35 openstackgerrit Merged openstack/nova master: libvirt: make mdev types name attribute be optional https://review.opendev.org/753574
13:34:34 CeeMac lyarwood: that could explain it. Appreciate you looking into this so diligently :)
13:47:39 openstackgerrit Takashi Natsume proposed openstack/nova master: Remove six.moves https://review.opendev.org/727224
13:47:55 openstackgerrit Takashi Natsume proposed openstack/nova master: Remove six.iteritems/itervalues/iterkeys https://review.opendev.org/727757
13:48:46 openstackgerrit Takashi Natsume proposed openstack/nova master: Remove six.byte2int/int2byte https://review.opendev.org/727777
13:49:12 openstackgerrit Takashi Natsume proposed openstack/nova master: Remove six.binary_type/integer_types/string_types https://review.opendev.org/728094
13:50:26 openstackgerrit Takashi Natsume proposed openstack/nova master: Remove six.text_type (1/2) https://review.opendev.org/728109
13:54:41 openstackgerrit Takashi Natsume proposed openstack/nova master: Add placeholder migrations for Victoria backports https://review.opendev.org/754396
14:05:18 openstackgerrit Takashi Natsume proposed openstack/nova master: Remove six.text_type (2/2) https://review.opendev.org/728117
14:16:52 openstackgerrit Stephen Finucane proposed openstack/nova stable/victoria: libvirt: make mdev types name attribute be optional https://review.opendev.org/754401
14:17:00 stephenfin bauzas: ^
14:18:37 lyarwood stephenfin: do we want to land that in stable before we are certain we don't want another rc?

Earlier   Later