| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-09-25 | |||
| 11:07:50 | CeeMac | lyarwood: no worries, enjoy your lunch :) | |
| 11:16:31 | CeeMac | kashyap: was this resolved in a paticular release do you know? is there some patch i can/should apply for rocky? | |
| 11:18:01 | CeeMac | or, which version of libvirt was the patch introduced in may be a better question kashyap | |
| 11:22:07 | kashyap | CeeMac: It is fixed in libvirt v5.0.0 and above | |
| 11:22:13 | lyarwood | CeeMac: https://review.opendev.org/#/q/I0c52917a5555a70c4973f37dea1aebf878dd73b4 - I fixed that years ago, sorry I missed that kashyap suggested that | |
| 11:22:14 | CeeMac | ah | |
| 11:22:30 | CeeMac | i'm on v4 | |
| 11:22:38 | CeeMac | which explains why i still see it | |
| 11:22:47 | kashyap | OH, yes; that too - the 'mirror' element patch from lyarwood | |
| 11:23:52 | CeeMac | lyarwood: should that not be present in rocky anyhow if it was introduced around ocata? (presumably)? | |
| 11:24:06 | lyarwood | CeeMac: yes my change should be in stable/rocky | |
| 11:24:09 | kashyap | Yes, it should be in Rocky | |
| 11:24:47 | CeeMac | hmm | |
| 11:24:57 | CeeMac | then i'm still confused how I'm gettting it :/ | |
| 11:25:01 | kashyap | lyarwood: Unrelated - I can't reproduce the Focal / DEVICE_DEL thing for the life of me :-( Maybe my chops for setting up synthetic workload are bad | |
| 11:25:11 | CeeMac | unless it would still require libvirt v5? | |
| 11:25:22 | kashyap | CeeMac: Hmm, for "lulz", can you try with libvirt-v5, if you can afford to? | |
| 11:25:42 | CeeMac | kashyap: you make that sound so easy :D | |
| 11:25:55 | kashyap | I know; I'm a terribly bad person for phrasing it that way :D | |
| 11:26:28 | CeeMac | is it a straight forward upgrade? or are there bits and bobs that plug in that would be affected | |
| 11:26:53 | CeeMac | i have a test environment which i have these issues in that i also got in prod, that is on the same version (running OSA) | |
| 11:27:04 | CeeMac | i was planning an upgrade from rocky to train anyhow | |
| 11:27:17 | CeeMac | not sure which libvirt comes down with that, will check | |
| 11:27:28 | openstack | bug 1896463 in OpenStack Compute (nova) ussuri "evacuation failed: Port update failed : Unable to correlate PCI slot " [Low,Triaged] https://launchpad.net/bugs/1896463 | |
| 11:27:28 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Reproduce bug 1896463 in func env https://review.opendev.org/754100 | |
| 11:27:35 | kashyap | CeeMac: Good question; when upgrading libvirt (assuming you're not using any container-based deployments): | |
| 11:27:56 | CeeMac | no containers kashyap | |
| 11:28:23 | kashyap | CeeMac: ... if you also see a QEMU update being pulled in: then you need to cold-reboot (stop + start) the instances for the libvirt to pick up the new QEMU | |
| 11:29:02 | kashyap | (If there's no QEMU update; then 'just' restarting the libvirtd will suffice - yes, your instance will stay intact during libvirtd restart) | |
| 11:29:20 | CeeMac | its like you've been asked these questions before :D | |
| 11:30:23 | CeeMac | just need to find out if powercycling the other instances that have failed retype on their attached volumes also breaks the instance, then I'll ponder options for upgrades etc | |
| 11:30:29 | openstackgerrit | kuang congxian proposed openstack/nova master: Replace six with python 3 style code in nova/test https://review.opendev.org/754354 | |
| 11:31:19 | kashyap | CeeMac: Yes, very many times. Also some important bits here: https://kashyapc.fedorapeople.org/Effective-Virtual-CPU-Configuration-in-Nova-Berlin2018.pdf | |
| 11:31:59 | kashyap | CeeMac: (Slightly updated variant of the above, from Jan-2020: https://kashyapc.fedorapeople.org/Secure-CPU-config-for-KVM-based-guests-RHT_TechDay_Brussels_2020.pdf) | |
| 11:50:24 | CeeMac | kashyap: thanks, appreciate it | |
| 11:51:44 | kashyap | CeeMac: I mentioned the above because you also need to take into account "security" :D - (thanks to the CPU hardware flaws that flew by, and are still flowing by) | |
| 11:52:14 | CeeMac | soooo many security flaws in the last few years! | |
| 11:53:19 | kashyap | Yea, are you seeing the timeline slide? ;-) | |
| 11:55:41 | CeeMac | didn't look at it yet, was just thinking it terms of meltdown/spectre variants in general | |
| 11:56:54 | CeeMac | which I now see is on the slide :D | |
| 12:20:51 | openstackgerrit | Elod Illes proposed openstack/nova stable/victoria: [stable-only] Update .gitreview for stable/victoria https://review.opendev.org/754319 | |
| 12:20:52 | openstackgerrit | Elod Illes proposed openstack/nova stable/victoria: [stable-only] Update TOX_CONSTRAINTS_FILE for stable/victoria https://review.opendev.org/754320 | |
| 12:22:30 | openstackgerrit | Harshavardhan Metla proposed openstack/nova master: Updated the weights section of Compute schedulers in nova https://review.opendev.org/754365 | |
| 12:22:50 | lyarwood | elod: the stable-only tags aren't technically required there at the moment as it isn't a full stable branch just yet | |
| 12:22:54 | lyarwood | elod: until GA | |
| 12:23:19 | lyarwood | elod: we will use the stable/$release branch to cut another rc tag if required | |
| 12:23:34 | lyarwood | etc | |
| 12:24:30 | lyarwood | at least that's my understanding | |
| 12:24:35 | elod | lyarwood: the pep8 tools/check-cherry-picks.sh fails without the tag | |
| 12:26:02 | lyarwood | ah my bad sorry | |
| 12:26:09 | lyarwood | I forgot that we had that check in there | |
| 12:27:22 | elod | yes, it's good most of the cases, but sometimes makes our life harder :) | |
| 12:28:29 | elod | at least the fix is just a commit message update, so not that painful :) | |
| 12:29:42 | lyarwood | yup true | |
| 12:44:44 | openstack | Launchpad bug 1896621 in OpenStack Compute (nova) "instance corrupted after volume retype" [High,Confirmed] - Assigned to Lee Yarwood (lyarwood) | |
| 12:44:44 | lyarwood | CeeMac: so I can reproduce the race in https://bugs.launchpad.net/nova/+bug/1896621 | |
| 12:44:57 | lyarwood | CeeMac: looking at the corruption now, I assume serials are going to change once I reboot | |
| 12:45:35 | CeeMac | lyarwood: glad you were able to reproduce it | |
| 12:46:30 | lyarwood | CeeMac: so in that case the serials are correct after a hard reboot | |
| 12:46:45 | lyarwood | CeeMac: actually did you use --hard in the reboot command? | |
| 12:46:55 | lyarwood | CeeMac: I wonder if the issue is with a soft reboot here | |
| 12:47:27 | CeeMac | Lyarwood well | |
| 12:47:30 | lyarwood | oh wow the volume hasn't rolled back | |
| 12:48:06 | CeeMac | lyarwood: in one instance the server restarted itself and then powered itself off | |
| 12:48:07 | lyarwood | the instance is pointing to the original volume but cinder thinks that has retyped/migrated to the new type | |
| 12:48:21 | CeeMac | lyarwood: another one I soft rebooted | |
| 12:49:17 | CeeMac | lyarwood: is that visible somewhere? | |
| 12:49:49 | CeeMac | lyarwood: when I checked openstack volume show it had migration state error but original volume id | |
| 12:50:25 | CeeMac | And the disk mappings seemed correct in /dev/disk/by-path matching lun id etc | |
| 12:50:33 | lyarwood | CeeMac: ah sorry I was looking at the wrong volume | |
| 12:50:40 | lyarwood | CeeMac: the first volume is in that state | |
| 12:50:57 | lyarwood | CeeMac: migration status error and listed against the original type | |
| 12:51:08 | CeeMac | lyarwood: right | |
| 12:58:43 | lyarwood | CeeMac: yeah I can't make this fail and change the disk serials | |
| 12:58:53 | lyarwood | CeeMac: I'll update the bug with some ideas | |
| 12:59:40 | CeeMac | Thanks lyarwood | |
| 13:08:13 | lyarwood | CeeMac: was there much of a gap between the failure and the instances being rebooted? | |
| 13:08:45 | lyarwood | CeeMac: I think I see the issue now, we don't actually tell libvirt to revert back to the original volumes within the domain itself | |
| 13:09:04 | lyarwood | CeeMac: if we fail late when dumping the domain XML | |
| 13:20:23 | CeeMac | lyarwood: in the instance that broke itself its hard to say as it rebooted and powered itself off. | |
| 13:21:02 | CeeMac | lyarwood: one of the other instances there was a day or two before I manually rebooted | |
| 13:25:35 | CeeMac | lyarwood: sounds like you're on to something | |
| 13:25:58 | lyarwood | CeeMac: kk, I think the issue is that libvirt leaves the new volume connected after this failure, the instance keeps writing to the new volume until we rip out the volume from underneath it likely sending the disk into a read-only state within the instance. By the time the instance reboots it's out of sync with the original volume that is now reconnected and that's where the failure comes | |
| 13:25:59 | lyarwood | from. | |
| 13:26:46 | CeeMac | lyarwood: during the swap_volume process you mean? | |
| 13:26:50 | lyarwood | let me see if I can reproduce this on master with a Fedora VM so I can debug this a little more and prove that's the case | |
| 13:27:04 | CeeMac | lyarwood: so the new device is attached for a period until the job fails and is reverted? | |
| 13:27:06 | lyarwood | CeeMac: so we fail *after* the swap volume operation has completed | |
| 13:27:23 | lyarwood | CeeMac: yeah I believe so | |
| 13:27:46 | lyarwood | CeeMac: there's nothing in the failure path within Nova that would revert the active domain configuration AFAICT | |
| 13:33:35 | openstackgerrit | Merged openstack/nova master: libvirt: make mdev types name attribute be optional https://review.opendev.org/753574 | |
| 13:34:34 | CeeMac | lyarwood: that could explain it. Appreciate you looking into this so diligently :) | |
| 13:47:39 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.moves https://review.opendev.org/727224 | |
| 13:47:55 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.iteritems/itervalues/iterkeys https://review.opendev.org/727757 | |
| 13:48:46 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.byte2int/int2byte https://review.opendev.org/727777 | |
| 13:49:12 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.binary_type/integer_types/string_types https://review.opendev.org/728094 | |
| 13:50:26 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.text_type (1/2) https://review.opendev.org/728109 | |
| 13:54:41 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Add placeholder migrations for Victoria backports https://review.opendev.org/754396 | |
| 14:05:18 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.text_type (2/2) https://review.opendev.org/728117 | |
| 14:16:52 | openstackgerrit | Stephen Finucane proposed openstack/nova stable/victoria: libvirt: make mdev types name attribute be optional https://review.opendev.org/754401 | |