| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-09-24 | |||
| 16:31:33 | gmann | i think opensuse 15.1. has qemu 3.1.1.1 | |
| 16:31:59 | sean-k-mooney | 15.2 has 4.2.0 | |
| 16:32:05 | gmann | yeah | |
| 16:32:05 | gmann | https://software.opensuse.org/package/qemu? | |
| 16:32:05 | sean-k-mooney | and tumbelweed is on 5.1 | |
| 16:32:17 | sean-k-mooney | i was looking at https://repology.org/project/qemu/versions | |
| 16:32:45 | sean-k-mooney | frickler: so idealy to fix the suse job it woudl go to 15.2 | |
| 16:33:13 | gmann | yeah, we need to move the distro version | |
| 16:34:21 | gibi | frickler, sean-k-mooney, gmann: filed a bug for tracking https://bugs.launchpad.net/nova/+bug/1897118 | |
| 16:34:21 | openstack | Launchpad bug 1897118 in OpenStack Compute (nova) "nova-compute does not start in devstack-platform-opensuse-15 job due to < 4.0.0 qemu version" [Undecided,New] | |
| 16:35:08 | gibi | I have to leave for today | |
| 16:35:47 | gibi | see you tomorrow | |
| 16:36:00 | sean-k-mooney | looking at https://github.com/openstack/devstack/blob/master/doc/source/overview.rst#base-os susue is not on the list of maintained plathforms anymore | |
| 16:36:20 | sean-k-mooney | still if its an easy fix then there is no harm | |
| 16:36:58 | gmann | sean-k-mooney: i am not sure how easy, 15.2 is not in supported version yet so it depends on how new version work- https://github.com/openstack/devstack/blob/5aa38f51b3dd0660a0622aecd65937d3c56eedc2/stack.sh#L224 | |
| 16:38:23 | sean-k-mooney | gmann: right but devstack only offically support the latest current version of suse | |
| 16:39:09 | gmann | added devstack in bug | |
| 16:39:29 | sean-k-mooney | so in theory it should have swaped when 15.2 was released like we do for fedora when a new version comes out | |
| 16:39:58 | gmann | yeah, | |
| 16:40:19 | gmann | need more active maintainer for opensuse if needed. | |
| 16:40:28 | sean-k-mooney | the first step will be ensuring dib can build 15.2 | |
| 16:40:41 | sean-k-mooney | then if it can update devstack to support it | |
| 16:40:56 | sean-k-mooney | that or use the tumbelweed job instead | |
| 16:41:17 | sean-k-mooney | * /the/a/ | |
| 17:56:58 | openstackgerrit | Merged openstack/nova stable/ussuri: test_evacuate.sh: Stop using libvirt-bin https://review.opendev.org/753281 | |
| 17:58:51 | lyarwood | ~. | |
| 18:37:23 | openstackgerrit | Merged openstack/nova stable/ussuri: Ensure source compute is up when confirming a resize https://review.opendev.org/748369 | |
| 18:52:03 | openstackgerrit | sean mooney proposed openstack/nova master: Fix unplugging VIF when migrate/resize VM https://review.opendev.org/751642 | |
| 18:52:04 | openstackgerrit | sean mooney proposed openstack/nova master: libvirt: delegate ovs plug to os-vif https://review.opendev.org/602432 | |
| 20:01:07 | openstackgerrit | Merged openstack/nova stable/rocky: libvirt: Do not reference VIR_ERR_DEVICE_MISSING when libvirt is < v4.1.0 https://review.opendev.org/747361 | |
| 21:37:24 | openstackgerrit | melanie witt proposed openstack/nova stable/ussuri: Test for disabling greendns https://review.opendev.org/754220 | |
| #openstack-nova - 2020-09-25 | |||
| 00:26:53 | openstackgerrit | norman shen proposed openstack/nova stable/rocky: libvirt:driver:Disallow AIO=native when 'O_DIRECT' is not available https://review.opendev.org/747612 | |
| 03:29:57 | openstackgerrit | Jorhson Deng proposed openstack/nova master: add the decorator revert function for attach_volume and detach_volume https://review.opendev.org/754258 | |
| 08:40:32 | openstackgerrit | OpenStack Release Bot proposed openstack/nova stable/victoria: Update .gitreview for stable/victoria https://review.opendev.org/754319 | |
| 08:40:38 | openstackgerrit | OpenStack Release Bot proposed openstack/nova stable/victoria: Update TOX_CONSTRAINTS_FILE for stable/victoria https://review.opendev.org/754320 | |
| 08:40:43 | openstackgerrit | OpenStack Release Bot proposed openstack/nova master: Update master for stable/victoria https://review.opendev.org/754321 | |
| 08:40:49 | openstackgerrit | OpenStack Release Bot proposed openstack/nova master: Add Python3 wallaby unit tests https://review.opendev.org/754322 | |
| 08:41:18 | stephenfin | hurrah \o/ | |
| 08:41:21 | stephenfin | gibi++ | |
| 08:47:47 | gibi | yepp, RC1 is out | |
| 09:05:36 | CeeMac | lyarwood: Morning. I've been trying to investigate the state of play with some other instances that were having their volumes retyped and I see a lot of errors. I found an instance with 'clean' volumes and tried to do a retype on the first volume attached. This failed with a different error "libvirtError: Requested operation is not valid: pivot of disk 'vda' requires an active copy job" which I have seen in the | |
| 09:05:36 | CeeMac | logs from some of the other tasks as I've been investigating | |
| 09:05:49 | CeeMac | lyarwood: full error log output: http://paste.openstack.org/show/798374/ | |
| 09:07:26 | CeeMac | lyarwood: looking at the cinder-volume logs the retype completes successfully then goes through an error clean up and deletes the temporary volume again. http://paste.openstack.org/show/798375/ | |
| 09:08:08 | CeeMac | so I'm not sure if there is something else more seriously wrong that is linked to the corruption, or if this a separate issue | |
| 09:08:54 | CeeMac | I'm going to enable debug on cinder-volume and try another one to see if there is any better info on what is happening. | |
| 09:10:01 | CeeMac | if anyone else feels like chipping in, don't be shy :) This is perplexing me somewhat | |
| 09:23:49 | openstackgerrit | Merged openstack/python-novaclient stable/victoria: Update .gitreview for stable/victoria https://review.opendev.org/750932 | |
| 09:26:07 | openstackgerrit | Merged openstack/python-novaclient stable/victoria: Update TOX_CONSTRAINTS_FILE for stable/victoria https://review.opendev.org/750933 | |
| 10:11:06 | lyarwood | CeeMac: morning sorry was AFK | |
| 10:12:15 | lyarwood | CeeMac: We would need to see the libvirtd logs associated with that rebase tbh | |
| 10:14:10 | CeeMac | lyarwood: how would I go about enabling logging for libvirt? | |
| 10:14:31 | lyarwood | kashyap: did you have a public doc on enabling libvirt debugging anywhere? :) | |
| 10:33:01 | openstackgerrit | Harshavardhan Metla proposed openstack/nova stable/rocky: Updated the weights section of Compute schedulers in nova https://review.opendev.org/754349 | |
| 10:51:53 | kashyap | lyarwood: Sure: https://kashyapc.fedorapeople.org/virt/openstack/request-nova-libvirt-qemu-debug-logs.txt | |
| 10:52:05 | kashyap | lyarwood: It my most referred public "doc" :D | |
| 10:52:40 | kashyap | lyarwood: If you have 'virt-admin', take that route, instead (also documented above) | |
| 10:53:38 | lyarwood | CeeMac: ^ | |
| 10:53:41 | lyarwood | kashyap: thanks | |
| 10:53:57 | lyarwood | CeeMac: also I've been thinking about your issues, these are Windows Domain Controllers right? | |
| 10:54:01 | kashyap | CeeMac: Oh, that "pivot" error brings back bad memories :D | |
| 10:54:11 | CeeMac | kashyap: lyarwood thanks, i'll take a look | |
| 10:54:22 | kashyap | CeeMac: Ah-ha, here it goes; I recall debugging it on the list: http://lists.openstack.org/pipermail/openstack-dev/2016-October/105158.html | |
| 10:54:23 | CeeMac | lyarwood: yes, the ones that have been affected the worst so far | |
| 10:54:32 | lyarwood | CeeMac: and the actual corruption you're seeing is that they can't boot after a reboot? Does Windows even load? | |
| 10:55:02 | CeeMac | lyarwood: correct, and no, the AD DB appears to have been corrupted which causes a BSOD | |
| 10:55:20 | CeeMac | lyarwood: tried all of the startup recovery options for DSRM etc, no dice | |
| 10:55:31 | kashyap | CeeMac: Also, BTW, "one should not see that error anymore"(TM), because a bug that came out of debugging problem was fixed: https://bugzilla.redhat.com/show_bug.cgi?id=1382165 | |
| 10:55:31 | openstack | bugzilla.redhat.com bug 1382165 in libvirt "virDomainGetBlockJobInfo: Adjust job reporting based on QEMU stats & the "ready" field of `query-block-jobs`" [Unspecified,Closed: nextrelease] - Assigned to pkrempa | |
| 10:55:55 | CeeMac | lyarwood: I've discovered a few other instances with failed retype jobs, I'm going to try reboot them see if they're broken too or not | |
| 10:55:58 | lyarwood | CeeMac: is the AD DB on the same volume as the OS? | |
| 10:56:07 | CeeMac | lyarwood: it is not | |
| 10:56:18 | CeeMac | kashyap: thanks, i'll go read that too! | |
| 10:56:31 | lyarwood | CeeMac: right, I wonder if this is due to the disk serial changing during the retype | |
| 10:59:46 | CeeMac | lyarwood: thats what I was wondering too. And this seems to be the case for recreating a DC from its original disks, or from a snapshot | |
| 11:00:15 | CeeMac | lyarwood: although I haven't yet validated if the same corruption occurs, but the end result is absolutely the same. | |
| 11:04:34 | lyarwood | CeeMac: is it not possible to point at a different disk for the AD DB to confirm this has happened and also recover? | |
| 11:04:56 | lyarwood | CeeMac: or just list the disks in Windows before and after a retype | |
| 11:05:07 | lyarwood | CeeMac: looking at the code things should persist in the DB | |
| 11:05:25 | lyarwood | https://github.com/openstack/nova/blob/2931516b75ab1eda8deefec6910efee82a0a7ae2/nova/compute/manager.py#L7299-L7314 - we use the return from a call to c-api here to determine what the volume_id and serial should be | |
| 11:05:35 | lyarwood | https://github.com/openstack/nova/blob/2931516b75ab1eda8deefec6910efee82a0a7ae2/nova/compute/manager.py#L7364-L7389 - then stash it in our db here | |
| 11:07:37 | CeeMac | lyarwood: the problem is if the AD DB corrupts either the server reboots itself and won't start back up, or stays online but you can't log in to it because the authentication is tied to the AD DB. | |
| 11:07:50 | CeeMac | lyarwood: no worries, enjoy your lunch :) | |
| 11:16:31 | CeeMac | kashyap: was this resolved in a paticular release do you know? is there some patch i can/should apply for rocky? | |
| 11:18:01 | CeeMac | or, which version of libvirt was the patch introduced in may be a better question kashyap | |
| 11:22:07 | kashyap | CeeMac: It is fixed in libvirt v5.0.0 and above | |
| 11:22:13 | lyarwood | CeeMac: https://review.opendev.org/#/q/I0c52917a5555a70c4973f37dea1aebf878dd73b4 - I fixed that years ago, sorry I missed that kashyap suggested that | |
| 11:22:14 | CeeMac | ah | |
| 11:22:30 | CeeMac | i'm on v4 | |
| 11:22:38 | CeeMac | which explains why i still see it | |
| 11:22:47 | kashyap | OH, yes; that too - the 'mirror' element patch from lyarwood | |
| 11:23:52 | CeeMac | lyarwood: should that not be present in rocky anyhow if it was introduced around ocata? (presumably)? | |
| 11:24:06 | lyarwood | CeeMac: yes my change should be in stable/rocky | |
| 11:24:09 | kashyap | Yes, it should be in Rocky | |
| 11:24:47 | CeeMac | hmm | |
| 11:24:57 | CeeMac | then i'm still confused how I'm gettting it :/ | |
| 11:25:01 | kashyap | lyarwood: Unrelated - I can't reproduce the Focal / DEVICE_DEL thing for the life of me :-( Maybe my chops for setting up synthetic workload are bad | |
| 11:25:11 | CeeMac | unless it would still require libvirt v5? | |
| 11:25:22 | kashyap | CeeMac: Hmm, for "lulz", can you try with libvirt-v5, if you can afford to? | |
| 11:25:42 | CeeMac | kashyap: you make that sound so easy :D | |