| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-09-25 | |||
| 10:54:11 | CeeMac | kashyap: lyarwood thanks, i'll take a look | |
| 10:54:22 | kashyap | CeeMac: Ah-ha, here it goes; I recall debugging it on the list: http://lists.openstack.org/pipermail/openstack-dev/2016-October/105158.html | |
| 10:54:23 | CeeMac | lyarwood: yes, the ones that have been affected the worst so far | |
| 10:54:32 | lyarwood | CeeMac: and the actual corruption you're seeing is that they can't boot after a reboot? Does Windows even load? | |
| 10:55:02 | CeeMac | lyarwood: correct, and no, the AD DB appears to have been corrupted which causes a BSOD | |
| 10:55:20 | CeeMac | lyarwood: tried all of the startup recovery options for DSRM etc, no dice | |
| 10:55:31 | kashyap | CeeMac: Also, BTW, "one should not see that error anymore"(TM), because a bug that came out of debugging problem was fixed: https://bugzilla.redhat.com/show_bug.cgi?id=1382165 | |
| 10:55:31 | openstack | bugzilla.redhat.com bug 1382165 in libvirt "virDomainGetBlockJobInfo: Adjust job reporting based on QEMU stats & the "ready" field of `query-block-jobs`" [Unspecified,Closed: nextrelease] - Assigned to pkrempa | |
| 10:55:55 | CeeMac | lyarwood: I've discovered a few other instances with failed retype jobs, I'm going to try reboot them see if they're broken too or not | |
| 10:55:58 | lyarwood | CeeMac: is the AD DB on the same volume as the OS? | |
| 10:56:07 | CeeMac | lyarwood: it is not | |
| 10:56:18 | CeeMac | kashyap: thanks, i'll go read that too! | |
| 10:56:31 | lyarwood | CeeMac: right, I wonder if this is due to the disk serial changing during the retype | |
| 10:59:46 | CeeMac | lyarwood: thats what I was wondering too. And this seems to be the case for recreating a DC from its original disks, or from a snapshot | |
| 11:00:15 | CeeMac | lyarwood: although I haven't yet validated if the same corruption occurs, but the end result is absolutely the same. | |
| 11:04:34 | lyarwood | CeeMac: is it not possible to point at a different disk for the AD DB to confirm this has happened and also recover? | |
| 11:04:56 | lyarwood | CeeMac: or just list the disks in Windows before and after a retype | |
| 11:05:07 | lyarwood | CeeMac: looking at the code things should persist in the DB | |
| 11:05:25 | lyarwood | https://github.com/openstack/nova/blob/2931516b75ab1eda8deefec6910efee82a0a7ae2/nova/compute/manager.py#L7299-L7314 - we use the return from a call to c-api here to determine what the volume_id and serial should be | |
| 11:05:35 | lyarwood | https://github.com/openstack/nova/blob/2931516b75ab1eda8deefec6910efee82a0a7ae2/nova/compute/manager.py#L7364-L7389 - then stash it in our db here | |
| 11:07:37 | CeeMac | lyarwood: the problem is if the AD DB corrupts either the server reboots itself and won't start back up, or stays online but you can't log in to it because the authentication is tied to the AD DB. | |
| 11:07:50 | CeeMac | lyarwood: no worries, enjoy your lunch :) | |
| 11:16:31 | CeeMac | kashyap: was this resolved in a paticular release do you know? is there some patch i can/should apply for rocky? | |
| 11:18:01 | CeeMac | or, which version of libvirt was the patch introduced in may be a better question kashyap | |
| 11:22:07 | kashyap | CeeMac: It is fixed in libvirt v5.0.0 and above | |
| 11:22:13 | lyarwood | CeeMac: https://review.opendev.org/#/q/I0c52917a5555a70c4973f37dea1aebf878dd73b4 - I fixed that years ago, sorry I missed that kashyap suggested that | |
| 11:22:14 | CeeMac | ah | |
| 11:22:30 | CeeMac | i'm on v4 | |
| 11:22:38 | CeeMac | which explains why i still see it | |
| 11:22:47 | kashyap | OH, yes; that too - the 'mirror' element patch from lyarwood | |
| 11:23:52 | CeeMac | lyarwood: should that not be present in rocky anyhow if it was introduced around ocata? (presumably)? | |
| 11:24:06 | lyarwood | CeeMac: yes my change should be in stable/rocky | |
| 11:24:09 | kashyap | Yes, it should be in Rocky | |
| 11:24:47 | CeeMac | hmm | |
| 11:24:57 | CeeMac | then i'm still confused how I'm gettting it :/ | |
| 11:25:01 | kashyap | lyarwood: Unrelated - I can't reproduce the Focal / DEVICE_DEL thing for the life of me :-( Maybe my chops for setting up synthetic workload are bad | |
| 11:25:11 | CeeMac | unless it would still require libvirt v5? | |
| 11:25:22 | kashyap | CeeMac: Hmm, for "lulz", can you try with libvirt-v5, if you can afford to? | |
| 11:25:42 | CeeMac | kashyap: you make that sound so easy :D | |
| 11:25:55 | kashyap | I know; I'm a terribly bad person for phrasing it that way :D | |
| 11:26:28 | CeeMac | is it a straight forward upgrade? or are there bits and bobs that plug in that would be affected | |
| 11:26:53 | CeeMac | i have a test environment which i have these issues in that i also got in prod, that is on the same version (running OSA) | |
| 11:27:04 | CeeMac | i was planning an upgrade from rocky to train anyhow | |
| 11:27:17 | CeeMac | not sure which libvirt comes down with that, will check | |
| 11:27:28 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Reproduce bug 1896463 in func env https://review.opendev.org/754100 | |
| 11:27:28 | openstack | bug 1896463 in OpenStack Compute (nova) ussuri "evacuation failed: Port update failed : Unable to correlate PCI slot " [Low,Triaged] https://launchpad.net/bugs/1896463 | |
| 11:27:35 | kashyap | CeeMac: Good question; when upgrading libvirt (assuming you're not using any container-based deployments): | |
| 11:27:56 | CeeMac | no containers kashyap | |
| 11:28:23 | kashyap | CeeMac: ... if you also see a QEMU update being pulled in: then you need to cold-reboot (stop + start) the instances for the libvirt to pick up the new QEMU | |
| 11:29:02 | kashyap | (If there's no QEMU update; then 'just' restarting the libvirtd will suffice - yes, your instance will stay intact during libvirtd restart) | |
| 11:29:20 | CeeMac | its like you've been asked these questions before :D | |
| 11:30:23 | CeeMac | just need to find out if powercycling the other instances that have failed retype on their attached volumes also breaks the instance, then I'll ponder options for upgrades etc | |
| 11:30:29 | openstackgerrit | kuang congxian proposed openstack/nova master: Replace six with python 3 style code in nova/test https://review.opendev.org/754354 | |
| 11:31:19 | kashyap | CeeMac: Yes, very many times. Also some important bits here: https://kashyapc.fedorapeople.org/Effective-Virtual-CPU-Configuration-in-Nova-Berlin2018.pdf | |
| 11:31:59 | kashyap | CeeMac: (Slightly updated variant of the above, from Jan-2020: https://kashyapc.fedorapeople.org/Secure-CPU-config-for-KVM-based-guests-RHT_TechDay_Brussels_2020.pdf) | |
| 11:50:24 | CeeMac | kashyap: thanks, appreciate it | |
| 11:51:44 | kashyap | CeeMac: I mentioned the above because you also need to take into account "security" :D - (thanks to the CPU hardware flaws that flew by, and are still flowing by) | |
| 11:52:14 | CeeMac | soooo many security flaws in the last few years! | |
| 11:53:19 | kashyap | Yea, are you seeing the timeline slide? ;-) | |
| 11:55:41 | CeeMac | didn't look at it yet, was just thinking it terms of meltdown/spectre variants in general | |
| 11:56:54 | CeeMac | which I now see is on the slide :D | |
| 12:20:51 | openstackgerrit | Elod Illes proposed openstack/nova stable/victoria: [stable-only] Update .gitreview for stable/victoria https://review.opendev.org/754319 | |
| 12:20:52 | openstackgerrit | Elod Illes proposed openstack/nova stable/victoria: [stable-only] Update TOX_CONSTRAINTS_FILE for stable/victoria https://review.opendev.org/754320 | |
| 12:22:30 | openstackgerrit | Harshavardhan Metla proposed openstack/nova master: Updated the weights section of Compute schedulers in nova https://review.opendev.org/754365 | |
| 12:22:50 | lyarwood | elod: the stable-only tags aren't technically required there at the moment as it isn't a full stable branch just yet | |
| 12:22:54 | lyarwood | elod: until GA | |
| 12:23:19 | lyarwood | elod: we will use the stable/$release branch to cut another rc tag if required | |
| 12:23:34 | lyarwood | etc | |
| 12:24:30 | lyarwood | at least that's my understanding | |
| 12:24:35 | elod | lyarwood: the pep8 tools/check-cherry-picks.sh fails without the tag | |
| 12:26:02 | lyarwood | ah my bad sorry | |
| 12:26:09 | lyarwood | I forgot that we had that check in there | |
| 12:27:22 | elod | yes, it's good most of the cases, but sometimes makes our life harder :) | |
| 12:28:29 | elod | at least the fix is just a commit message update, so not that painful :) | |
| 12:29:42 | lyarwood | yup true | |
| 12:44:44 | lyarwood | CeeMac: so I can reproduce the race in https://bugs.launchpad.net/nova/+bug/1896621 | |
| 12:44:44 | openstack | Launchpad bug 1896621 in OpenStack Compute (nova) "instance corrupted after volume retype" [High,Confirmed] - Assigned to Lee Yarwood (lyarwood) | |
| 12:44:57 | lyarwood | CeeMac: looking at the corruption now, I assume serials are going to change once I reboot | |
| 12:45:35 | CeeMac | lyarwood: glad you were able to reproduce it | |
| 12:46:30 | lyarwood | CeeMac: so in that case the serials are correct after a hard reboot | |
| 12:46:45 | lyarwood | CeeMac: actually did you use --hard in the reboot command? | |
| 12:46:55 | lyarwood | CeeMac: I wonder if the issue is with a soft reboot here | |
| 12:47:27 | CeeMac | Lyarwood well | |
| 12:47:30 | lyarwood | oh wow the volume hasn't rolled back | |
| 12:48:06 | CeeMac | lyarwood: in one instance the server restarted itself and then powered itself off | |
| 12:48:07 | lyarwood | the instance is pointing to the original volume but cinder thinks that has retyped/migrated to the new type | |
| 12:48:21 | CeeMac | lyarwood: another one I soft rebooted | |
| 12:49:17 | CeeMac | lyarwood: is that visible somewhere? | |
| 12:49:49 | CeeMac | lyarwood: when I checked openstack volume show it had migration state error but original volume id | |
| 12:50:25 | CeeMac | And the disk mappings seemed correct in /dev/disk/by-path matching lun id etc | |
| 12:50:33 | lyarwood | CeeMac: ah sorry I was looking at the wrong volume | |
| 12:50:40 | lyarwood | CeeMac: the first volume is in that state | |
| 12:50:57 | lyarwood | CeeMac: migration status error and listed against the original type | |
| 12:51:08 | CeeMac | lyarwood: right | |
| 12:58:43 | lyarwood | CeeMac: yeah I can't make this fail and change the disk serials | |
| 12:58:53 | lyarwood | CeeMac: I'll update the bug with some ideas | |
| 12:59:40 | CeeMac | Thanks lyarwood | |
| 13:08:13 | lyarwood | CeeMac: was there much of a gap between the failure and the instances being rebooted? | |
| 13:08:45 | lyarwood | CeeMac: I think I see the issue now, we don't actually tell libvirt to revert back to the original volumes within the domain itself | |
| 13:09:04 | lyarwood | CeeMac: if we fail late when dumping the domain XML | |