Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-09-24
17:58:51 lyarwood ~.
18:37:23 openstackgerrit Merged openstack/nova stable/ussuri: Ensure source compute is up when confirming a resize https://review.opendev.org/748369
18:52:03 openstackgerrit sean mooney proposed openstack/nova master: Fix unplugging VIF when migrate/resize VM https://review.opendev.org/751642
18:52:04 openstackgerrit sean mooney proposed openstack/nova master: libvirt: delegate ovs plug to os-vif https://review.opendev.org/602432
20:01:07 openstackgerrit Merged openstack/nova stable/rocky: libvirt: Do not reference VIR_ERR_DEVICE_MISSING when libvirt is < v4.1.0 https://review.opendev.org/747361
21:37:24 openstackgerrit melanie witt proposed openstack/nova stable/ussuri: Test for disabling greendns https://review.opendev.org/754220
#openstack-nova - 2020-09-25
00:26:53 openstackgerrit norman shen proposed openstack/nova stable/rocky: libvirt:driver:Disallow AIO=native when 'O_DIRECT' is not available https://review.opendev.org/747612
03:29:57 openstackgerrit Jorhson Deng proposed openstack/nova master: add the decorator revert function for attach_volume and detach_volume https://review.opendev.org/754258
08:40:32 openstackgerrit OpenStack Release Bot proposed openstack/nova stable/victoria: Update .gitreview for stable/victoria https://review.opendev.org/754319
08:40:38 openstackgerrit OpenStack Release Bot proposed openstack/nova stable/victoria: Update TOX_CONSTRAINTS_FILE for stable/victoria https://review.opendev.org/754320
08:40:43 openstackgerrit OpenStack Release Bot proposed openstack/nova master: Update master for stable/victoria https://review.opendev.org/754321
08:40:49 openstackgerrit OpenStack Release Bot proposed openstack/nova master: Add Python3 wallaby unit tests https://review.opendev.org/754322
08:41:18 stephenfin hurrah \o/
08:41:21 stephenfin gibi++
08:47:47 gibi yepp, RC1 is out
09:05:36 CeeMac lyarwood: Morning. I've been trying to investigate the state of play with some other instances that were having their volumes retyped and I see a lot of errors. I found an instance with 'clean' volumes and tried to do a retype on the first volume attached. This failed with a different error "libvirtError: Requested operation is not valid: pivot of disk 'vda' requires an active copy job" which I have seen in the
09:05:36 CeeMac logs from some of the other tasks as I've been investigating
09:05:49 CeeMac lyarwood: full error log output: http://paste.openstack.org/show/798374/
09:07:26 CeeMac lyarwood: looking at the cinder-volume logs the retype completes successfully then goes through an error clean up and deletes the temporary volume again. http://paste.openstack.org/show/798375/
09:08:08 CeeMac so I'm not sure if there is something else more seriously wrong that is linked to the corruption, or if this a separate issue
09:08:54 CeeMac I'm going to enable debug on cinder-volume and try another one to see if there is any better info on what is happening.
09:10:01 CeeMac if anyone else feels like chipping in, don't be shy :) This is perplexing me somewhat
09:23:49 openstackgerrit Merged openstack/python-novaclient stable/victoria: Update .gitreview for stable/victoria https://review.opendev.org/750932
09:26:07 openstackgerrit Merged openstack/python-novaclient stable/victoria: Update TOX_CONSTRAINTS_FILE for stable/victoria https://review.opendev.org/750933
10:11:06 lyarwood CeeMac: morning sorry was AFK
10:12:15 lyarwood CeeMac: We would need to see the libvirtd logs associated with that rebase tbh
10:14:10 CeeMac lyarwood: how would I go about enabling logging for libvirt?
10:14:31 lyarwood kashyap: did you have a public doc on enabling libvirt debugging anywhere? :)
10:33:01 openstackgerrit Harshavardhan Metla proposed openstack/nova stable/rocky: Updated the weights section of Compute schedulers in nova https://review.opendev.org/754349
10:51:53 kashyap lyarwood: Sure: https://kashyapc.fedorapeople.org/virt/openstack/request-nova-libvirt-qemu-debug-logs.txt
10:52:05 kashyap lyarwood: It my most referred public "doc" :D
10:52:40 kashyap lyarwood: If you have 'virt-admin', take that route, instead (also documented above)
10:53:38 lyarwood CeeMac: ^
10:53:41 lyarwood kashyap: thanks
10:53:57 lyarwood CeeMac: also I've been thinking about your issues, these are Windows Domain Controllers right?
10:54:01 kashyap CeeMac: Oh, that "pivot" error brings back bad memories :D
10:54:11 CeeMac kashyap: lyarwood thanks, i'll take a look
10:54:22 kashyap CeeMac: Ah-ha, here it goes; I recall debugging it on the list: http://lists.openstack.org/pipermail/openstack-dev/2016-October/105158.html
10:54:23 CeeMac lyarwood: yes, the ones that have been affected the worst so far
10:54:32 lyarwood CeeMac: and the actual corruption you're seeing is that they can't boot after a reboot? Does Windows even load?
10:55:02 CeeMac lyarwood: correct, and no, the AD DB appears to have been corrupted which causes a BSOD
10:55:20 CeeMac lyarwood: tried all of the startup recovery options for DSRM etc, no dice
10:55:31 kashyap CeeMac: Also, BTW, "one should not see that error anymore"(TM), because a bug that came out of debugging problem was fixed: https://bugzilla.redhat.com/show_bug.cgi?id=1382165
10:55:31 openstack bugzilla.redhat.com bug 1382165 in libvirt "virDomainGetBlockJobInfo: Adjust job reporting based on QEMU stats & the "ready" field of `query-block-jobs`" [Unspecified,Closed: nextrelease] - Assigned to pkrempa
10:55:55 CeeMac lyarwood: I've discovered a few other instances with failed retype jobs, I'm going to try reboot them see if they're broken too or not
10:55:58 lyarwood CeeMac: is the AD DB on the same volume as the OS?
10:56:07 CeeMac lyarwood: it is not
10:56:18 CeeMac kashyap: thanks, i'll go read that too!
10:56:31 lyarwood CeeMac: right, I wonder if this is due to the disk serial changing during the retype
10:59:46 CeeMac lyarwood: thats what I was wondering too. And this seems to be the case for recreating a DC from its original disks, or from a snapshot
11:00:15 CeeMac lyarwood: although I haven't yet validated if the same corruption occurs, but the end result is absolutely the same.
11:04:34 lyarwood CeeMac: is it not possible to point at a different disk for the AD DB to confirm this has happened and also recover?
11:04:56 lyarwood CeeMac: or just list the disks in Windows before and after a retype
11:05:07 lyarwood CeeMac: looking at the code things should persist in the DB
11:05:25 lyarwood https://github.com/openstack/nova/blob/2931516b75ab1eda8deefec6910efee82a0a7ae2/nova/compute/manager.py#L7299-L7314 - we use the return from a call to c-api here to determine what the volume_id and serial should be
11:05:35 lyarwood https://github.com/openstack/nova/blob/2931516b75ab1eda8deefec6910efee82a0a7ae2/nova/compute/manager.py#L7364-L7389 - then stash it in our db here
11:07:37 CeeMac lyarwood: the problem is if the AD DB corrupts either the server reboots itself and won't start back up, or stays online but you can't log in to it because the authentication is tied to the AD DB.
11:07:50 CeeMac lyarwood: no worries, enjoy your lunch :)
11:16:31 CeeMac kashyap: was this resolved in a paticular release do you know? is there some patch i can/should apply for rocky?
11:18:01 CeeMac or, which version of libvirt was the patch introduced in may be a better question kashyap
11:22:07 kashyap CeeMac: It is fixed in libvirt v5.0.0 and above
11:22:13 lyarwood CeeMac: https://review.opendev.org/#/q/I0c52917a5555a70c4973f37dea1aebf878dd73b4 - I fixed that years ago, sorry I missed that kashyap suggested that
11:22:14 CeeMac ah
11:22:30 CeeMac i'm on v4
11:22:38 CeeMac which explains why i still see it
11:22:47 kashyap OH, yes; that too - the 'mirror' element patch from lyarwood
11:23:52 CeeMac lyarwood: should that not be present in rocky anyhow if it was introduced around ocata? (presumably)?
11:24:06 lyarwood CeeMac: yes my change should be in stable/rocky
11:24:09 kashyap Yes, it should be in Rocky
11:24:47 CeeMac hmm
11:24:57 CeeMac then i'm still confused how I'm gettting it :/
11:25:01 kashyap lyarwood: Unrelated - I can't reproduce the Focal / DEVICE_DEL thing for the life of me :-( Maybe my chops for setting up synthetic workload are bad
11:25:11 CeeMac unless it would still require libvirt v5?
11:25:22 kashyap CeeMac: Hmm, for "lulz", can you try with libvirt-v5, if you can afford to?
11:25:42 CeeMac kashyap: you make that sound so easy :D
11:25:55 kashyap I know; I'm a terribly bad person for phrasing it that way :D
11:26:28 CeeMac is it a straight forward upgrade? or are there bits and bobs that plug in that would be affected
11:26:53 CeeMac i have a test environment which i have these issues in that i also got in prod, that is on the same version (running OSA)
11:27:04 CeeMac i was planning an upgrade from rocky to train anyhow
11:27:17 CeeMac not sure which libvirt comes down with that, will check
11:27:28 openstackgerrit Balazs Gibizer proposed openstack/nova master: Reproduce bug 1896463 in func env https://review.opendev.org/754100
11:27:28 openstack bug 1896463 in OpenStack Compute (nova) ussuri "evacuation failed: Port update failed : Unable to correlate PCI slot " [Low,Triaged] https://launchpad.net/bugs/1896463
11:27:35 kashyap CeeMac: Good question; when upgrading libvirt (assuming you're not using any container-based deployments):
11:27:56 CeeMac no containers kashyap
11:28:23 kashyap CeeMac: ... if you also see a QEMU update being pulled in: then you need to cold-reboot (stop + start) the instances for the libvirt to pick up the new QEMU
11:29:02 kashyap (If there's no QEMU update; then 'just' restarting the libvirtd will suffice - yes, your instance will stay intact during libvirtd restart)
11:29:20 CeeMac its like you've been asked these questions before :D
11:30:23 CeeMac just need to find out if powercycling the other instances that have failed retype on their attached volumes also breaks the instance, then I'll ponder options for upgrades etc
11:30:29 openstackgerrit kuang congxian proposed openstack/nova master: Replace six with python 3 style code in nova/test https://review.opendev.org/754354
11:31:19 kashyap CeeMac: Yes, very many times. Also some important bits here: https://kashyapc.fedorapeople.org/Effective-Virtual-CPU-Configuration-in-Nova-Berlin2018.pdf
11:31:59 kashyap CeeMac: (Slightly updated variant of the above, from Jan-2020: https://kashyapc.fedorapeople.org/Secure-CPU-config-for-KVM-based-guests-RHT_TechDay_Brussels_2020.pdf)
11:50:24 CeeMac kashyap: thanks, appreciate it
11:51:44 kashyap CeeMac: I mentioned the above because you also need to take into account "security" :D - (thanks to the CPU hardware flaws that flew by, and are still flowing by)
11:52:14 CeeMac soooo many security flaws in the last few years!
11:53:19 kashyap Yea, are you seeing the timeline slide? ;-)
11:55:41 CeeMac didn't look at it yet, was just thinking it terms of meltdown/spectre variants in general
11:56:54 CeeMac which I now see is on the slide :D
12:20:51 openstackgerrit Elod Illes proposed openstack/nova stable/victoria: [stable-only] Update .gitreview for stable/victoria https://review.opendev.org/754319
12:20:52 openstackgerrit Elod Illes proposed openstack/nova stable/victoria: [stable-only] Update TOX_CONSTRAINTS_FILE for stable/victoria https://review.opendev.org/754320
12:22:30 openstackgerrit Harshavardhan Metla proposed openstack/nova master: Updated the weights section of Compute schedulers in nova https://review.opendev.org/754365

Earlier   Later