| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-01-04 | |||
| 02:39:40 | Kevin_Zheng | Thanks, problem solved | |
| 02:39:53 | Kevin_Zheng | did we mentioned this in any docs? | |
| 02:41:10 | mriedem | it should be in the install guide | |
| 02:42:22 | mriedem | oh, well, https://docs.openstack.org/nova/latest/install/controller-install-ubuntu.html#install-and-configure-components | |
| 02:42:25 | melwitt | mriedem: re: that multiattach thing, did you see this bug? https://bugzilla.redhat.com/show_bug.cgi?id=1378242 based on that it looks like there needs to be share-rw=on property set under |
|
| 02:42:26 | openstack | bugzilla.redhat.com bug 1378242 in libvirt "QEMU image file locking (libvirt)" [Unspecified,On_qa] - Assigned to pkrempa | |
| 02:42:40 | mriedem | Kevin_Zheng: that install guide works because it's configuring nova.conf to set the [database] to the cell1 db | |
| 02:42:56 | mriedem | Kevin_Zheng: that install guide was written before the superconductor mode stuff that dansmith did in devstack | |
| 02:43:04 | mriedem | where the controllers are pointed at cell0 | |
| 02:43:22 | mriedem | Kevin_Zheng: we also have https://docs.openstack.org/nova/latest/user/cells.html#setup-of-cells-v2 | |
| 02:44:12 | mriedem | it's not very clear, but there is a note in there too | |
| 02:44:13 | mriedem | "At this point, the API database can now find the cell database, and further commands will attempt to look inside. If this is a completely fresh database (such as if you’re adding a cell, or if this is a new deployment), then you will need to run nova-manage db sync on it to initialize the schema." | |
| 02:44:31 | mriedem | Kevin_Zheng: might be a good FAQs entry https://docs.openstack.org/nova/latest/user/cells.html#faqs | |
| 02:44:43 | mriedem | melwitt: nope never seen that | |
| 02:44:59 | Kevin_Zheng | yeah, | |
| 02:45:07 | Kevin_Zheng | I will add it later | |
| 02:45:40 | mriedem | melwitt: not mentioned in the domain xml docs at all https://libvirt.org/formatdomain.html | |
| 02:45:41 | mriedem | :( | |
| 02:47:27 | mriedem | and by the looks of when this was 'fixed' i'm guessing we'd need super modern versions of qemu to use this | |
| 02:47:45 | melwitt | yeah, I was just thinking the same | |
| 02:48:15 | mriedem | well, so much for multiattach in queens | |
| 02:48:21 | melwitt | this all looks fairly recent, which would explain the absence in the docs | |
| 02:48:22 | mriedem | wah wah | |
| 02:48:34 | mriedem | yeah | |
| 02:48:35 | mriedem | commit 860a3c4bea1d24773d8a495f213d5de3ac48a462 Author: Peter Krempa <pkrempa@redhat.com> Date: Wed Nov 15 15:02:58 2017 +0100 | |
| 02:50:22 | mriedem | we could have conditional logic around the version of qemu being used, but that sucks, | |
| 02:50:36 | mriedem | and i don't know if like, qemu < 2.10 works, and then is broken until qemu >= x | |
| 02:50:47 | mriedem | so you'd have a middle ground where things just don't work | |
| 02:53:08 | melwitt | what do you mean, like if this was a regression there could be a window in the middle where things don't work? | |
| 02:53:13 | mriedem | yeah | |
| 02:53:27 | mriedem | we didn't have this problem around ~newton when i wrote the original tempest test for multiattach | |
| 02:53:40 | mriedem | you could do the 2 attachments fine, it's just that nova didn't orchestrate the detach properly | |
| 02:53:45 | melwitt | oh :( | |
| 02:53:48 | mriedem | now we're in a case with newer qemu where the 2nd attach fails | |
| 02:54:07 | openstackgerrit | Chen Hanxiao proposed openstack/nova master: log test: use fixtures.StandardLogging in setUp https://review.openstack.org/531065 | |
| 02:54:29 | mriedem | so apparently fixed in libvirt-3.9.0-3.el7 | |
| 02:54:53 | mriedem | we are testing against 3.6.0 | |
| 02:55:34 | mriedem | https://bugzilla.redhat.com/show_bug.cgi?id=1378242#c13 | |
| 02:55:35 | openstack | bugzilla.redhat.com bug 1378242 in libvirt "QEMU image file locking (libvirt)" [Unspecified,On_qa] - Assigned to pkrempa | |
| 02:55:38 | melwitt | and this is the qemu bug that the libvirt bug was cloned from https://bugzilla.redhat.com/show_bug.cgi?id=1378241 | |
| 02:55:38 | openstack | bugzilla.redhat.com bug 1378241 in qemu-kvm-rhev "QEMU image file locking" [Unspecified,Verified] - Assigned to famz | |
| 02:55:38 | mriedem | same thing we're hitting | |
| 02:56:21 | melwitt | that says qemu-kvm-rhev though. | |
| 02:57:11 | mriedem | yeah so that caused the bug we had with 2.10 where we were hitting the same issue with qemu-img info | |
| 02:57:17 | mriedem | and had to start using the force flag for that | |
| 02:57:24 | mriedem | https://bugs.launchpad.net/nova/+bug/1718295 | |
| 02:57:26 | openstack | Launchpad bug 1718295 in OpenStack Compute (nova) "Live migration fails with qemu-img >= 2.10: "Failed to get shared "write" lock\nIs another process using the image?"" [High,Fix released] - Assigned to Sean Dague (sdague) | |
| 02:57:48 | mriedem | so now our support matrix is likely something like, | |
| 02:57:55 | mriedem | 1. do it the old way if qemu<2.10, else | |
| 02:58:05 | mriedem | 2. do it the new way if qemu>=2.10 AND libvirt>=3.9.0 | |
| 02:58:09 | mriedem | else | |
| 02:58:12 | mriedem | 3. shart your pants | |
| 02:58:31 | mriedem | and only 3 is testable in our current CI env :( | |
| 02:58:31 | melwitt | sigh, so this has some history | |
| 03:01:14 | clarkb | didnt nova implement its own lock wrapper for that? | |
| 03:01:29 | mriedem | for what? | |
| 03:01:33 | openstackgerrit | Eli Qiao proposed openstack/nova master: Api-guide: Add Block Device Mapping https://review.openstack.org/522084 | |
| 03:03:28 | clarkb | mriedem: for the lack of working locks | |
| 03:03:40 | clarkb | there wa sa while wrapper script thing iirc | |
| 03:04:42 | bjhuangr | mriedem: hi Matt, do you have a chance to review https://review.openstack.org/#/c/523387/ ? Thanks in advance | |
| 03:05:45 | mriedem | bjhuangr: not right now sorry | |
| 03:05:51 | mriedem | clarkb: sounds like something different | |
| 03:06:04 | mriedem | melwitt: looking at https://github.com/libvirt/libvirt/commit/28907b0043fbf71085a798372ab9c816ba043b93 it actually looks like that went into libvirt 3.10 | |
| 03:06:26 | mriedem | and it's unclear to me if we'd actually have to put something in the disk config xml or if libvirt just handles that for us | |
| 03:09:13 | melwitt | oh, hm. I wasn't sure about that either | |
| 03:10:20 | melwitt | looking at this, it looks like libvirt handles it. but not sure | |
| 03:10:32 | mriedem | yeah that's what i'm wondering | |
| 03:10:41 | mriedem | based on https://bugzilla.redhat.com/show_bug.cgi?id=1378242#c14 | |
| 03:10:42 | openstack | bugzilla.redhat.com bug 1378242 in libvirt "QEMU image file locking (libvirt)" [Unspecified,On_qa] - Assigned to pkrempa | |
| 03:11:14 | mriedem | i think peter is saying, you need qemu 2.10 with the new lock stuff and the fixes mentioned in comment 10 to libvirt | |
| 03:11:32 | mriedem | so if qemu>=2.10, libvirt must be >=3.10 (i think) | |
| 03:11:52 | mriedem | says it was fixed in libvirt-3.9.0-3.el7 but maybe that's an rpm package version with the patch backported? | |
| 03:12:02 | mriedem | because https://github.com/libvirt/libvirt/commit/28907b0043fbf71085a798372ab9c816ba043b93 says 3.10 | |
| 03:12:03 | melwitt | that's how I interpret it too | |
| 03:12:26 | mriedem | suck | |
| 03:12:50 | mriedem | i wonder if i could swindle jamespage and the gang to backport that to libvirt 3.6.0 :) | |
| 03:13:06 | melwitt | yeah, I'm honestly surprised this is all so recent | |
| 03:13:41 | melwitt | with all of the multiattach talk that's been happening for years, I never thought there could be a blocker at the qemu and libvirt level | |
| 03:14:38 | mriedem | sweet revenge | |
| 03:15:02 | melwitt | and the "fixed-in-version 3.9" with the commit in 3.10 doesn't make sense to me either | |
| 03:15:07 | mriedem | did you notice the guy complaining in comment 14 is using oracle RAC? | |
| 03:15:17 | mriedem | which is like the #1 use case for multiattach and why oracle wants it in nova so badly | |
| 03:15:41 | mriedem | comment 13 i should say | |
| 03:15:53 | mriedem | stvnoyes: fyi as this is all going to impact multiattach support in nova in queens ^ | |
| 03:16:08 | melwitt | what a mess | |
| 03:16:10 | mriedem | stvnoyes: if you're looking for something to do, we need some testing for the multiattach patches using qemu 2.10 and libvirt 3.1.0 | |
| 03:16:12 | mriedem | *3.10 | |
| 03:19:06 | mriedem | oh ha | |
| 03:19:07 | mriedem | lyarwood: https://bugzilla.redhat.com/show_bug.cgi?id=1415250 | |
| 03:19:09 | openstack | bugzilla.redhat.com bug 1415250 in openstack-nova "QEMU image file locking (RHOS)" [High,Post] - Assigned to lyarwood | |
| 03:19:10 | mriedem | melwitt: ^ | |
| 03:20:15 | melwitt | oh hello friends | |
| 03:21:33 | mriedem | left a comment because i like to comment in bz's | |
| 03:21:37 | mriedem | feels like i'm part of the team | |
| 03:21:50 | mriedem | ooo cc'ed to markmc@redhat.com | |
| 03:22:16 | mriedem | and eglynn@redhat.com, dansmith@redhat.com, | |
| 03:22:17 | mriedem | heh | |
| 03:23:45 | melwitt | :) | |
| 03:43:18 | openstackgerrit | Matt Riedemann proposed openstack/nova master: [libvirt] Allow multiple volume attachments https://review.openstack.org/267587 | |
| 03:43:19 | openstackgerrit | Matt Riedemann proposed openstack/nova master: [api] Allow multi-attach in compute api https://review.openstack.org/271047 | |