| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-09-24 | |||
| 15:06:47 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add tests for 'AggregateInstanceExtraSpecsFilter' https://review.opendev.org/754116 | |
| 15:33:28 | tosky | stephenfin: re placement PTL, couldn't the nova candidate (I assume there is just one :) also apply for placement PTL? It would be a "personal union" of the crown | |
| 15:33:54 | stephenfin | tosky: entirely possible. That's a very valid option 3 | |
| 15:50:25 | gibi | nova meeting starts in 10 minutes in #openstack-meeting-3 | |
| 15:54:22 | openstackgerrit | Balazs Gibizer proposed openstack/nova-specs master: Move implemented spec https://review.opendev.org/754124 | |
| 15:54:35 | openstackgerrit | Balazs Gibizer proposed openstack/nova-specs master: Move implemented specs https://review.opendev.org/754124 | |
| 16:24:33 | frickler | did nova recently raise the bar on qemu? looks like opensuse is failing with "nova.exception.InternalError: Nova requires QEMU version 4.0.0 or greater." https://a5f2733c1907b1f26b90-5593d50c131879f6a486eeedbad80e3c.ssl.cf5.rackcdn.com/743800/14/check/devstack-platform-opensuse-15/91eeaf7/controller/logs/screen-n-cpu.txt | |
| 16:26:14 | gibi | frickler: yes we did bumped min version | |
| 16:26:50 | gibi | frickler: https://review.opendev.org/#/c/746981/ | |
| 16:27:48 | frickler | gibi: ah, o.k., so I guess nobody checked opensuse? I mean, personally I don't care, we can just drop that devstack job | |
| 16:28:28 | gibi | yeah it seems nobody checked the opensuse job | |
| 16:29:12 | gibi | as far as I see it is a non-voting job that was not run for the nova patch | |
| 16:30:02 | gibi | frickler: I will open a bug to track this if you haven't done that yet | |
| 16:30:27 | frickler | gibi: I've asked the suse contacts I know in #openstack-infra, feel free to create a bug, too | |
| 16:30:51 | sean-k-mooney | frickler: the latest version of opensuse shoudl have the new minium | |
| 16:31:03 | sean-k-mooney | but that does not mean that job is using the latest version | |
| 16:31:33 | gmann | i think opensuse 15.1. has qemu 3.1.1.1 | |
| 16:31:59 | sean-k-mooney | 15.2 has 4.2.0 | |
| 16:32:05 | gmann | yeah | |
| 16:32:05 | gmann | https://software.opensuse.org/package/qemu? | |
| 16:32:05 | sean-k-mooney | and tumbelweed is on 5.1 | |
| 16:32:17 | sean-k-mooney | i was looking at https://repology.org/project/qemu/versions | |
| 16:32:45 | sean-k-mooney | frickler: so idealy to fix the suse job it woudl go to 15.2 | |
| 16:33:13 | gmann | yeah, we need to move the distro version | |
| 16:34:21 | gibi | frickler, sean-k-mooney, gmann: filed a bug for tracking https://bugs.launchpad.net/nova/+bug/1897118 | |
| 16:34:21 | openstack | Launchpad bug 1897118 in OpenStack Compute (nova) "nova-compute does not start in devstack-platform-opensuse-15 job due to < 4.0.0 qemu version" [Undecided,New] | |
| 16:35:08 | gibi | I have to leave for today | |
| 16:35:47 | gibi | see you tomorrow | |
| 16:36:00 | sean-k-mooney | looking at https://github.com/openstack/devstack/blob/master/doc/source/overview.rst#base-os susue is not on the list of maintained plathforms anymore | |
| 16:36:20 | sean-k-mooney | still if its an easy fix then there is no harm | |
| 16:36:58 | gmann | sean-k-mooney: i am not sure how easy, 15.2 is not in supported version yet so it depends on how new version work- https://github.com/openstack/devstack/blob/5aa38f51b3dd0660a0622aecd65937d3c56eedc2/stack.sh#L224 | |
| 16:38:23 | sean-k-mooney | gmann: right but devstack only offically support the latest current version of suse | |
| 16:39:09 | gmann | added devstack in bug | |
| 16:39:29 | sean-k-mooney | so in theory it should have swaped when 15.2 was released like we do for fedora when a new version comes out | |
| 16:39:58 | gmann | yeah, | |
| 16:40:19 | gmann | need more active maintainer for opensuse if needed. | |
| 16:40:28 | sean-k-mooney | the first step will be ensuring dib can build 15.2 | |
| 16:40:41 | sean-k-mooney | then if it can update devstack to support it | |
| 16:40:56 | sean-k-mooney | that or use the tumbelweed job instead | |
| 16:41:17 | sean-k-mooney | * /the/a/ | |
| 17:56:58 | openstackgerrit | Merged openstack/nova stable/ussuri: test_evacuate.sh: Stop using libvirt-bin https://review.opendev.org/753281 | |
| 17:58:51 | lyarwood | ~. | |
| 18:37:23 | openstackgerrit | Merged openstack/nova stable/ussuri: Ensure source compute is up when confirming a resize https://review.opendev.org/748369 | |
| 18:52:03 | openstackgerrit | sean mooney proposed openstack/nova master: Fix unplugging VIF when migrate/resize VM https://review.opendev.org/751642 | |
| 18:52:04 | openstackgerrit | sean mooney proposed openstack/nova master: libvirt: delegate ovs plug to os-vif https://review.opendev.org/602432 | |
| 20:01:07 | openstackgerrit | Merged openstack/nova stable/rocky: libvirt: Do not reference VIR_ERR_DEVICE_MISSING when libvirt is < v4.1.0 https://review.opendev.org/747361 | |
| 21:37:24 | openstackgerrit | melanie witt proposed openstack/nova stable/ussuri: Test for disabling greendns https://review.opendev.org/754220 | |
| #openstack-nova - 2020-09-25 | |||
| 00:26:53 | openstackgerrit | norman shen proposed openstack/nova stable/rocky: libvirt:driver:Disallow AIO=native when 'O_DIRECT' is not available https://review.opendev.org/747612 | |
| 03:29:57 | openstackgerrit | Jorhson Deng proposed openstack/nova master: add the decorator revert function for attach_volume and detach_volume https://review.opendev.org/754258 | |
| 08:40:32 | openstackgerrit | OpenStack Release Bot proposed openstack/nova stable/victoria: Update .gitreview for stable/victoria https://review.opendev.org/754319 | |
| 08:40:38 | openstackgerrit | OpenStack Release Bot proposed openstack/nova stable/victoria: Update TOX_CONSTRAINTS_FILE for stable/victoria https://review.opendev.org/754320 | |
| 08:40:43 | openstackgerrit | OpenStack Release Bot proposed openstack/nova master: Update master for stable/victoria https://review.opendev.org/754321 | |
| 08:40:49 | openstackgerrit | OpenStack Release Bot proposed openstack/nova master: Add Python3 wallaby unit tests https://review.opendev.org/754322 | |
| 08:41:18 | stephenfin | hurrah \o/ | |
| 08:41:21 | stephenfin | gibi++ | |
| 08:47:47 | gibi | yepp, RC1 is out | |
| 09:05:36 | CeeMac | lyarwood: Morning. I've been trying to investigate the state of play with some other instances that were having their volumes retyped and I see a lot of errors. I found an instance with 'clean' volumes and tried to do a retype on the first volume attached. This failed with a different error "libvirtError: Requested operation is not valid: pivot of disk 'vda' requires an active copy job" which I have seen in the | |
| 09:05:36 | CeeMac | logs from some of the other tasks as I've been investigating | |
| 09:05:49 | CeeMac | lyarwood: full error log output: http://paste.openstack.org/show/798374/ | |
| 09:07:26 | CeeMac | lyarwood: looking at the cinder-volume logs the retype completes successfully then goes through an error clean up and deletes the temporary volume again. http://paste.openstack.org/show/798375/ | |
| 09:08:08 | CeeMac | so I'm not sure if there is something else more seriously wrong that is linked to the corruption, or if this a separate issue | |
| 09:08:54 | CeeMac | I'm going to enable debug on cinder-volume and try another one to see if there is any better info on what is happening. | |
| 09:10:01 | CeeMac | if anyone else feels like chipping in, don't be shy :) This is perplexing me somewhat | |
| 09:23:49 | openstackgerrit | Merged openstack/python-novaclient stable/victoria: Update .gitreview for stable/victoria https://review.opendev.org/750932 | |
| 09:26:07 | openstackgerrit | Merged openstack/python-novaclient stable/victoria: Update TOX_CONSTRAINTS_FILE for stable/victoria https://review.opendev.org/750933 | |
| 10:11:06 | lyarwood | CeeMac: morning sorry was AFK | |
| 10:12:15 | lyarwood | CeeMac: We would need to see the libvirtd logs associated with that rebase tbh | |
| 10:14:10 | CeeMac | lyarwood: how would I go about enabling logging for libvirt? | |
| 10:14:31 | lyarwood | kashyap: did you have a public doc on enabling libvirt debugging anywhere? :) | |
| 10:33:01 | openstackgerrit | Harshavardhan Metla proposed openstack/nova stable/rocky: Updated the weights section of Compute schedulers in nova https://review.opendev.org/754349 | |
| 10:51:53 | kashyap | lyarwood: Sure: https://kashyapc.fedorapeople.org/virt/openstack/request-nova-libvirt-qemu-debug-logs.txt | |
| 10:52:05 | kashyap | lyarwood: It my most referred public "doc" :D | |
| 10:52:40 | kashyap | lyarwood: If you have 'virt-admin', take that route, instead (also documented above) | |
| 10:53:38 | lyarwood | CeeMac: ^ | |
| 10:53:41 | lyarwood | kashyap: thanks | |
| 10:53:57 | lyarwood | CeeMac: also I've been thinking about your issues, these are Windows Domain Controllers right? | |
| 10:54:01 | kashyap | CeeMac: Oh, that "pivot" error brings back bad memories :D | |
| 10:54:11 | CeeMac | kashyap: lyarwood thanks, i'll take a look | |
| 10:54:22 | kashyap | CeeMac: Ah-ha, here it goes; I recall debugging it on the list: http://lists.openstack.org/pipermail/openstack-dev/2016-October/105158.html | |
| 10:54:23 | CeeMac | lyarwood: yes, the ones that have been affected the worst so far | |
| 10:54:32 | lyarwood | CeeMac: and the actual corruption you're seeing is that they can't boot after a reboot? Does Windows even load? | |
| 10:55:02 | CeeMac | lyarwood: correct, and no, the AD DB appears to have been corrupted which causes a BSOD | |
| 10:55:20 | CeeMac | lyarwood: tried all of the startup recovery options for DSRM etc, no dice | |
| 10:55:31 | kashyap | CeeMac: Also, BTW, "one should not see that error anymore"(TM), because a bug that came out of debugging problem was fixed: https://bugzilla.redhat.com/show_bug.cgi?id=1382165 | |
| 10:55:31 | openstack | bugzilla.redhat.com bug 1382165 in libvirt "virDomainGetBlockJobInfo: Adjust job reporting based on QEMU stats & the "ready" field of `query-block-jobs`" [Unspecified,Closed: nextrelease] - Assigned to pkrempa | |
| 10:55:55 | CeeMac | lyarwood: I've discovered a few other instances with failed retype jobs, I'm going to try reboot them see if they're broken too or not | |
| 10:55:58 | lyarwood | CeeMac: is the AD DB on the same volume as the OS? | |
| 10:56:07 | CeeMac | lyarwood: it is not | |
| 10:56:18 | CeeMac | kashyap: thanks, i'll go read that too! | |
| 10:56:31 | lyarwood | CeeMac: right, I wonder if this is due to the disk serial changing during the retype | |
| 10:59:46 | CeeMac | lyarwood: thats what I was wondering too. And this seems to be the case for recreating a DC from its original disks, or from a snapshot | |
| 11:00:15 | CeeMac | lyarwood: although I haven't yet validated if the same corruption occurs, but the end result is absolutely the same. | |
| 11:04:34 | lyarwood | CeeMac: is it not possible to point at a different disk for the AD DB to confirm this has happened and also recover? | |
| 11:04:56 | lyarwood | CeeMac: or just list the disks in Windows before and after a retype | |
| 11:05:07 | lyarwood | CeeMac: looking at the code things should persist in the DB | |
| 11:05:25 | lyarwood | https://github.com/openstack/nova/blob/2931516b75ab1eda8deefec6910efee82a0a7ae2/nova/compute/manager.py#L7299-L7314 - we use the return from a call to c-api here to determine what the volume_id and serial should be | |
| 11:05:35 | lyarwood | https://github.com/openstack/nova/blob/2931516b75ab1eda8deefec6910efee82a0a7ae2/nova/compute/manager.py#L7364-L7389 - then stash it in our db here | |
| 11:07:37 | CeeMac | lyarwood: the problem is if the AD DB corrupts either the server reboots itself and won't start back up, or stays online but you can't log in to it because the authentication is tied to the AD DB. | |
| 11:07:50 | CeeMac | lyarwood: no worries, enjoy your lunch :) | |
| 11:16:31 | CeeMac | kashyap: was this resolved in a paticular release do you know? is there some patch i can/should apply for rocky? | |