| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-01-04 | |||
| 02:25:16 | Kevin_Zheng | due to https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L497 | |
| 02:25:31 | openstackgerrit | Lance Bragstad proposed openstack/nova master: Simplify logic in get_enforcer https://review.openstack.org/531008 | |
| 02:25:48 | Kevin_Zheng | the field could not be added to cell1 db | |
| 02:27:05 | mriedem | Kevin_Zheng: that nova-manage code is really old | |
| 02:27:17 | mriedem | if the bdm uuid change broke that, we should have seen it in CI | |
| 02:27:19 | mriedem | since devstack runs this | |
| 02:27:38 | mriedem | the cell1 sync happens here https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L513 | |
| 02:27:45 | mriedem | after running the schema migrations for cell0 | |
| 02:28:46 | Kevin_Zheng | Hmm, but my db didn't got updated | |
| 02:28:53 | mriedem | is your nova.conf correct? | |
| 02:30:34 | Kevin_Zheng | I will check, but I have been using this for sometime, it should be correct. | |
| 02:32:11 | Kevin_Zheng | Hmm... in config file for api service, the database connection should be cell0 db, correct? | |
| 02:35:52 | mriedem | that's what we have in http://logs.openstack.org/58/526258/3/check/tempest-full/d133d1f/controller/logs/etc/nova/ | |
| 02:36:04 | mriedem | but devstack will sync using the cell1 conf too | |
| 02:36:11 | mriedem | which has the cell1 database in the [database] section | |
| 02:36:48 | mriedem | https://github.com/openstack-dev/devstack/blob/master/lib/nova#L707 | |
| 02:37:08 | mriedem | this runs it for cell0 https://github.com/openstack-dev/devstack/blob/master/lib/nova#L711 | |
| 02:37:29 | Kevin_Zheng | ah, I see | |
| 02:37:48 | mriedem | this was never implemented to hit all cells https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L479 | |
| 02:37:53 | mriedem | i had a patch for it but it must be abandoned | |
| 02:38:22 | mriedem | https://review.openstack.org/#/c/420973/ | |
| 02:39:26 | Kevin_Zheng | yeah, | |
| 02:39:36 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix up formatting for deprecate-api-extensions-policies release note https://review.openstack.org/531061 | |
| 02:39:40 | Kevin_Zheng | Thanks, problem solved | |
| 02:39:53 | Kevin_Zheng | did we mentioned this in any docs? | |
| 02:41:10 | mriedem | it should be in the install guide | |
| 02:42:22 | mriedem | oh, well, https://docs.openstack.org/nova/latest/install/controller-install-ubuntu.html#install-and-configure-components | |
| 02:42:25 | melwitt | mriedem: re: that multiattach thing, did you see this bug? https://bugzilla.redhat.com/show_bug.cgi?id=1378242 based on that it looks like there needs to be share-rw=on property set under |
|
| 02:42:26 | openstack | bugzilla.redhat.com bug 1378242 in libvirt "QEMU image file locking (libvirt)" [Unspecified,On_qa] - Assigned to pkrempa | |
| 02:42:40 | mriedem | Kevin_Zheng: that install guide works because it's configuring nova.conf to set the [database] to the cell1 db | |
| 02:42:56 | mriedem | Kevin_Zheng: that install guide was written before the superconductor mode stuff that dansmith did in devstack | |
| 02:43:04 | mriedem | where the controllers are pointed at cell0 | |
| 02:43:22 | mriedem | Kevin_Zheng: we also have https://docs.openstack.org/nova/latest/user/cells.html#setup-of-cells-v2 | |
| 02:44:12 | mriedem | it's not very clear, but there is a note in there too | |
| 02:44:13 | mriedem | "At this point, the API database can now find the cell database, and further commands will attempt to look inside. If this is a completely fresh database (such as if you’re adding a cell, or if this is a new deployment), then you will need to run nova-manage db sync on it to initialize the schema." | |
| 02:44:31 | mriedem | Kevin_Zheng: might be a good FAQs entry https://docs.openstack.org/nova/latest/user/cells.html#faqs | |
| 02:44:43 | mriedem | melwitt: nope never seen that | |
| 02:44:59 | Kevin_Zheng | yeah, | |
| 02:45:07 | Kevin_Zheng | I will add it later | |
| 02:45:40 | mriedem | melwitt: not mentioned in the domain xml docs at all https://libvirt.org/formatdomain.html | |
| 02:45:41 | mriedem | :( | |
| 02:47:27 | mriedem | and by the looks of when this was 'fixed' i'm guessing we'd need super modern versions of qemu to use this | |
| 02:47:45 | melwitt | yeah, I was just thinking the same | |
| 02:48:15 | mriedem | well, so much for multiattach in queens | |
| 02:48:21 | melwitt | this all looks fairly recent, which would explain the absence in the docs | |
| 02:48:22 | mriedem | wah wah | |
| 02:48:34 | mriedem | yeah | |
| 02:48:35 | mriedem | commit 860a3c4bea1d24773d8a495f213d5de3ac48a462 Author: Peter Krempa <pkrempa@redhat.com> Date: Wed Nov 15 15:02:58 2017 +0100 | |
| 02:50:22 | mriedem | we could have conditional logic around the version of qemu being used, but that sucks, | |
| 02:50:36 | mriedem | and i don't know if like, qemu < 2.10 works, and then is broken until qemu >= x | |
| 02:50:47 | mriedem | so you'd have a middle ground where things just don't work | |
| 02:53:08 | melwitt | what do you mean, like if this was a regression there could be a window in the middle where things don't work? | |
| 02:53:13 | mriedem | yeah | |
| 02:53:27 | mriedem | we didn't have this problem around ~newton when i wrote the original tempest test for multiattach | |
| 02:53:40 | mriedem | you could do the 2 attachments fine, it's just that nova didn't orchestrate the detach properly | |
| 02:53:45 | melwitt | oh :( | |
| 02:53:48 | mriedem | now we're in a case with newer qemu where the 2nd attach fails | |
| 02:54:07 | openstackgerrit | Chen Hanxiao proposed openstack/nova master: log test: use fixtures.StandardLogging in setUp https://review.openstack.org/531065 | |
| 02:54:29 | mriedem | so apparently fixed in libvirt-3.9.0-3.el7 | |
| 02:54:53 | mriedem | we are testing against 3.6.0 | |
| 02:55:34 | mriedem | https://bugzilla.redhat.com/show_bug.cgi?id=1378242#c13 | |
| 02:55:35 | openstack | bugzilla.redhat.com bug 1378242 in libvirt "QEMU image file locking (libvirt)" [Unspecified,On_qa] - Assigned to pkrempa | |
| 02:55:38 | mriedem | same thing we're hitting | |
| 02:55:38 | openstack | bugzilla.redhat.com bug 1378241 in qemu-kvm-rhev "QEMU image file locking" [Unspecified,Verified] - Assigned to famz | |
| 02:55:38 | melwitt | and this is the qemu bug that the libvirt bug was cloned from https://bugzilla.redhat.com/show_bug.cgi?id=1378241 | |
| 02:56:21 | melwitt | that says qemu-kvm-rhev though. | |
| 02:57:11 | mriedem | yeah so that caused the bug we had with 2.10 where we were hitting the same issue with qemu-img info | |
| 02:57:17 | mriedem | and had to start using the force flag for that | |
| 02:57:24 | mriedem | https://bugs.launchpad.net/nova/+bug/1718295 | |
| 02:57:26 | openstack | Launchpad bug 1718295 in OpenStack Compute (nova) "Live migration fails with qemu-img >= 2.10: "Failed to get shared "write" lock\nIs another process using the image?"" [High,Fix released] - Assigned to Sean Dague (sdague) | |
| 02:57:48 | mriedem | so now our support matrix is likely something like, | |
| 02:57:55 | mriedem | 1. do it the old way if qemu<2.10, else | |
| 02:58:05 | mriedem | 2. do it the new way if qemu>=2.10 AND libvirt>=3.9.0 | |
| 02:58:09 | mriedem | else | |
| 02:58:12 | mriedem | 3. shart your pants | |
| 02:58:31 | melwitt | sigh, so this has some history | |
| 02:58:31 | mriedem | and only 3 is testable in our current CI env :( | |
| 03:01:14 | clarkb | didnt nova implement its own lock wrapper for that? | |
| 03:01:29 | mriedem | for what? | |
| 03:01:33 | openstackgerrit | Eli Qiao proposed openstack/nova master: Api-guide: Add Block Device Mapping https://review.openstack.org/522084 | |
| 03:03:28 | clarkb | mriedem: for the lack of working locks | |
| 03:03:40 | clarkb | there wa sa while wrapper script thing iirc | |
| 03:04:42 | bjhuangr | mriedem: hi Matt, do you have a chance to review https://review.openstack.org/#/c/523387/ ? Thanks in advance | |
| 03:05:45 | mriedem | bjhuangr: not right now sorry | |
| 03:05:51 | mriedem | clarkb: sounds like something different | |
| 03:06:04 | mriedem | melwitt: looking at https://github.com/libvirt/libvirt/commit/28907b0043fbf71085a798372ab9c816ba043b93 it actually looks like that went into libvirt 3.10 | |
| 03:06:26 | mriedem | and it's unclear to me if we'd actually have to put something in the disk config xml or if libvirt just handles that for us | |
| 03:09:13 | melwitt | oh, hm. I wasn't sure about that either | |
| 03:10:20 | melwitt | looking at this, it looks like libvirt handles it. but not sure | |
| 03:10:32 | mriedem | yeah that's what i'm wondering | |
| 03:10:41 | mriedem | based on https://bugzilla.redhat.com/show_bug.cgi?id=1378242#c14 | |
| 03:10:42 | openstack | bugzilla.redhat.com bug 1378242 in libvirt "QEMU image file locking (libvirt)" [Unspecified,On_qa] - Assigned to pkrempa | |
| 03:11:14 | mriedem | i think peter is saying, you need qemu 2.10 with the new lock stuff and the fixes mentioned in comment 10 to libvirt | |
| 03:11:32 | mriedem | so if qemu>=2.10, libvirt must be >=3.10 (i think) | |
| 03:11:52 | mriedem | says it was fixed in libvirt-3.9.0-3.el7 but maybe that's an rpm package version with the patch backported? | |
| 03:12:02 | mriedem | because https://github.com/libvirt/libvirt/commit/28907b0043fbf71085a798372ab9c816ba043b93 says 3.10 | |
| 03:12:03 | melwitt | that's how I interpret it too | |
| 03:12:26 | mriedem | suck | |
| 03:12:50 | mriedem | i wonder if i could swindle jamespage and the gang to backport that to libvirt 3.6.0 :) | |
| 03:13:06 | melwitt | yeah, I'm honestly surprised this is all so recent | |